Budget director David A. Stockman opened fire on the military retirement system yesterday, charging that "institutional forces in the military are more concerned about protecting their retirement benefits than they are about protecting the security of the American people."

Stockman's broadside against the $18.3 billion military pension program came as a surprise during an otherwise sturdy defense of President Reagan's day-old $973.7 billion fiscal 1986 budget request before the Senate Budget Committee.

Defending Reagan's military spending proposals even though he unsuccessfully fought to trim them within the White House, the Office of Management and Budget chief said Congress should be proposing specific program cuts instead of broad-brush approaches such as spending freezes if it wants to achieve economies at the Pentagon.

Then, as committee members listened with stunned, rapt attention, he offered the military pension program as an example of something that could go on the budget chopping block, although he conceded that he had failed to win Reagan over to his point of view.

"It's a scandal; it's an outrage," Stockman said of the military retirement system and the way it is protected by the Pentagon hierarchy.

Pounding home his charge that the military puts its retirement ahead of the nation's security interests, he said, "When push comes to shove, they'll give up on security before they'll give up on retirement . . . . I'll probably be in hot water for saying it, but I'm going to say it because it's about time it was said."

Asked whether Reagan shared his view of the retirement system, he said, "I don't think so. I only stated a personal opinion."

The argument was raised within the administration that cutbacks in military retirement would harm recruitment efforts, Stockman said. "That argument doesn't persuade me. If you have to spend a half-million or a million over a lifetime to recruit someone, you'd better find a different way to recruit . . . ," he added.

From committee members, who often have found a lot to argue with the budget chief, there was little direct response. But Chairman Pete V. Domenici (R-N.M.) generally praised Stockman's performance.

Sen. Donald W. Riegle Jr. (D-Mich.), a frequent critic of Stockman, complimented his candor but suggested that he must first convince his own boss. "The person you have to persuade first is the president, the man you work for," Riegle said.

Asked later about Stockman's comments, Senate Majority Leader Robert J. Dole (R-Kan.) said, "I might have said it differently, but I do think it's an area we have to address."

Defense Secretary Caspar W. Weinberger also was asked about the comments as he left a House Armed Services Committee hearing. He responded that he believed that the president was standing by his budget proposal, which includes a one-year delay in cost-of-living increases for military retirees to save $491 million.

"The military retirement provisions that are in the president's budget are proper, and we have no indication that the president has in any way deviated from them," Weinberger said.

Asked about Stockman's use of the word "scandal," Weinberger said: "There's no scandal I know of."

Last night, the Defense Department issued a statement that "the military retirement provisions in the president's budget are entirely proper and are supported by this department."

The military retirement system is under review by Senate Republicans seeking to improve on Reagan's projected deficit reductions, but it is acknowledged to be one of the tougher issues on the agenda and no specific proposals for changing it have been aired publicly.

Stockman's testimony virtually assured that the system will undergo further -- and probably closer -- scrutiny as the congressional budget and deficit-reduction plan is developed.

As his parting shot, Stockman said: "I hope some of you up here who think the military budget is too big will call in the Joint Chiefs of Staff and ask them what they're going to put in the kitty by supplying a reasonable, modern, military retirement reform plan . . . . "

On defense spending in general, which Reagan seeks to increase by 13 percent next year while cutting most domestic programs by about 10 percent, committee members of both parties agreed that the Pentagon will get less than it wants.

The possibility of 3 percent growth after inflation, which is about half what the administration wants, is increasingly discussed. Domenici yesterday raised the possibility of such a statutory limitation for three years.

"We will have doubled the defense budget in five years," he said. "I just simply believe that we have enough buffer in our defense spending that this year we cannot give the president the defense request he recommends."

Domenici took issue with administration contentions that the military budget is being cut, saying it is being boosted by $30 billion over current spending.

"Even by Washington standards, to call such an increase a cut is stretching things a bit," he added.

Stockman conceded that he had argued within the White House that the Pentagon budget could be cut and implied it again yesterday, but he defended it nonetheless as the amount Reagan deemed necessary to protect national security.

Moreover, he said, the proposed military budget does little more than reflect continuation of weapons programs and other policies approved by Congress in the past four years. Many of those now advocating a freeze had voted for these built-in spending increases, he added.

Stockman said some freeze advocates are playing "Rip Van Winkle" by waking up after several years of supporting defense spending increases and suddenly deciding that the military budget is too big.

If Congress starts putting specific items on the tabl, negotiations with the administration on further defense cuts may be possible, Stockman implied.

"When you put some of that stuff on the table, with numbers on it, then maybe you can get a debate. Maybe some of those who have a different viewpoint downtown would have something to talk about as well," he said.