G. Warren Nutter, 55, a professor of economics at the University of Virginia and a former assistant secretary of defense for national security affairs, died of cancer Monday at his home in Charlottesville.

As a scholar and teacher, Prof. Nutter apecialized in the Soviet Union and the economics of defense. He believed in a strong defense posture for the United States and in the superiority of free enterprise over other economic systems.

He took these views with him when he joined the Nixon administration as assistant secretary of defense, a post he held from 1969 until 1973. He was an outspoken supporter of a continued U.S. stand in Vietnam, and a critic of the diplomatic methods of former secretary of state Henry A. Kissinger.

In March 1975, less than two months before the Communist capture of Saigon, Prof. Nutter contributed to series of articles in The Washington Post on whether this country should continue to provide aid to South Vietnam.

He argued that aid should be continued, and that part of the reason derived from "what we owe ourselves."

"What we owe ourselves depends on how much difference it makes to live in a disintegrating world order. For nothing will diminish our ability to wield a stabilizing influence in world affairs so much as a well deserved reputation for infidelity and unreliability in time of trouble,"

Later in 1975, Prof. Nutter published a book called "Kissinger's Grand Design." The book was endorsed by Melvin R. Laird, a secretary of defense in the first Nixon administration who wrote the foreword to it, and it attacked Kissinger's policy of detente with the Soviet Union.

Prof. Nutter argued that foreign policy in the United States must "rest on consensus" and that a consensus can be reached only if policy is "fully revealed and openly discussed." He said that the "personalistic, secretive, mysterious" diplomatic style of Kissinger made this process impossible.

Prof. Nutter had been on the faculty of the University of Virginia since 1956, and he returned to it after his government service. He also was an adjunce scholar of the American Enterprise Institute, which frequently espouses a conservative point of view on public questions, and director of its James Madison Center in Charlottesville.

At the time of his death, Prof. Nutter was a member of the board of visitors of The Citadel in South Carolina and of the advisory comittee on international studies at the Hoover Institute at Stanford University.

Prof. Nutter was born in Toledo, Ohio. He earned bachelor's, master's and doctoral degrees at the University of Chicago. He served in the Army in World War II, then taught at Lawrence College in Appleton, Wis., and Yale University before moving to the University of Virginia, where he headed the department of economics at one time.

His other books included "The Strange World of Ivan Ivanov," which sharply criticized the Soviet Union, "The Extent of Enterprise Monopoly in the United States," and "Growth of Government in the West."

Survivors include his wife, the former Jane Couch, of the home in Charlottesville; two daughters, Jane, of Charlottesville, and Anne E. Nones, of Minneapolis; two sons, Coleman A., of Merrick, N.Y., and William W., of Arlington; a brother, Coleman, of Naperville, Ill/., and six grandchildren.

The family suggests that expressions of sympathy be in the form of contributions to the Charlottesville-Albemarle Rescue Squad, Charlottesville, or to a charity of one's choice.