Budget director David A. Stockman came under blistering attack yesterday for suggesting that the military cares more about its $18 billion-a-year retirement program than about the nation's security, as Defense Secretary Caspar W. Weinberger charged that he "defamed" military people and veterans' groups urged that he be fired.

At a Senate Armed Services Committee hearing, Chairman Barry Goldwater (R-Ariz.) called Stockman's remarks, made at a Senate budget hearing Tuesday, "about as distasteful as anything I've heard coming out of this administration or any other." Goldwater called the retirement plan a deserved "silver rung at the end of the ladder" of military service.

The Veterans of Foreign Wars, meanwhile, sent President Reagan a telegram demanding that "the 4-F draft-dodging Stockman" be removed from his job and his statement repudiated by the White House.

Stockman was a divinity student during the Vietnam war and had a 4-D draft deferment.

At the White House, spokesman Larry Speakes said carefully that "the president is deeply appreciative of the sacrifice and contributions of the men and women in uniform. I point out to you that Stockman . . . said he was expressing a personal opinion probably not shared by the president. Other than that, I will have no comment. Try as you might, you will not get a comment."

White House chief of staff Donald T. Regan said Stockman's job "is not in jeopardy." He compared the Office of Management and Budget director to "a bull in the ring" who "lost his cool under pressure."

Stockman, whose candor on budget matters has stirred controversy in the past, had no comment. But his spokesman said the director "in no way intended to impugn the patriotism or devotion" of the military but was "expressing personal frustration with bureaucratic resistance -- not peculiar to the Department of Defense -- to reform."

The budget director in each of the last several years has tried with mixed success to make the Pentagon shoulder a larger share of the deficit-reduction burden.

At Tuesday's budget hearing, Stockman took issue with lawmakers who merely complain that defense spending is too high and urged them instead to propose cuts in specific programs. Then he singled out the retirement program as so bloated that it is a "a scandal" and added that "institutional forces in the military are more concerned about protecting their retirement benefits than they are about protecting the security of the American people."

Stockman predicted at the time that his remarks probably would land him in hot water, and the quick reaction bore him out.

Weinberger said military personnel "gave up any self-interest or sordid motives" by volunteering for service, adding that their retirement system "is fair but by no means lavish." Reagan's proposed 1986 budget does include a one-year delay in cost-of-living increases for retirees.

"I believe that the contribution that these dedicated leaders have made to this nation over many years of service should not be defamed," Weinberger said.

On the Senate Armed Services Committee, ranking minority member Sam Nunn (D-Ga.) said "the people in uniform, in my opinion, most of them don't go into the military, or serve in the military, or risk their lives for this country because of the retirement system. They do so because they love this country and what it stands for."

Rep. Charles E. Bennett (D-Fla.), a member of the House Armed Services Committee, branded Stockman's remarks "reprehensible." He called on the budget director to apologize "for impugning the patriotism of those who have given so much to our country."

Stockman received some support, however, from Senate Majority Leader Robert J. Dole (R-Kan.) and House Minority Leader Robert H. Michel (R-Ill.). They did not endorse his language but said the underlying call to reform the military retirement program has merit. Dole said Stockman was "very forthcoming in letting us know it's not a very easy thing we have to do. We're going to have to look at all those sacred cows."

Michel said "there's been no better person for the figures over these last four years than Dave Stockman" and that it would do no harm for Congress to examine "long-range" changes in retirement benefits.

A number of members of Congress, including new House Armed Services Committee Chairman Les Aspin (D-Wis.), have urged overhaul of the retirement system in the past. A study urging changes is also before Weinberger; he has not acted on it.

In addition to the Veterans of Foreign Wars, two other military groups -- the Retired Officers Association and the Reserve Officers Association -- sent telegrams to Reagan protesting Stockman's statement.