Office of Management and Budget Director David A. Stockman's swipe at the $18.3 billion military retirement system program during congressional testimony last week was indeed off the cuff, administration sources said this week. But Stockman's frustration over the issue goes back a decade and stems in part from the defeats dealt him by the Pentagon in the internal budget wars of 1981 and 1982.

Stockman tacitly acknowledged those defeats during deliberations on the current budget. Last fall, sources said, he didn't push for an overhaul of the military retirement program; he merely listed it as one area to examine in the context of his unsuccessful quest for deep defense spending cuts. President Reagan's fiscal 1986 budget does call for a freeze in the program's annual cost-of-living adjustment.

One OMB source said this week that Stockman regrets any implication that the defenders of the pension system were unpatriotic. But even though the remarks were disavowed by the president, the budget director is unrepentant about his basic points: that the system needs to be cut back and that the Pentagon brass won't face the issue. The latter problem is illustrated, one source said, by Defense Secretary Caspar W. Weinberger's failure to take action on reform suggestions by his own study group.

Other administration sources say that, during Reagan's first term, the Pentagon staged successful preemptive strikes on all pension-trimming proposals. "Every time he'd even start thinking about it," said a source, "the Joint Chiefs of Staff would rush to put a statement in The Army Times saying the system would never never never ever be cut . . . . We could talk about weapons systems, we could talk about personnel levels, but we couldn't even talk about that pensions." REG REVIEW II . . .

Executive branch agencies are scheduled to submit their first "budget" of proposed regulatory actions next Tuesday, kicking off the first stage of OMB's expanded regulatory review process.

Robert P. Bedell, deputy administrator of the Office of Information and Regulatory Affairs, said the first month probably will be consumed with first-time housekeeping tasks -- such as establishing a uniform definition of "significant regulatory action" from agency to agency. But there also may be some substantive set-tos between OMB and the agencies about the need for particular regulations.

The time for the serious head-knocking will come in April, when OIRA sends agencies a list of what it considers to be appropriate regulations -- the regulatory equivalent of the budget passbacks usually sent out in November.

In May, the first "regulatory program" document covering all executive branch agencies except the independent regulatory commissions is scheduled for publication. Until then, Bedell said, the OMB will treat the regulatory program with the same secrecy afforded budget deliberations. THE HAIR-SHIRT SET . . .

After cuffing around most other agencies' budgets, OMB this year demurely proposed a 1 percent reduction in its 1986 budget authority. That translates to a $450,000 cut and total budget authority of $38.4 million.

OMB spokesman Edwin L. Dale Jr. said the reductions reflect the administration-proposed 5 percent cut in civilian pay and a 10 percent cut in administrative overhead expenses.

Top OMB officials generally have felt pressure to keep the agency's budget down to avoid charges of hypocrisy when they attack other budgets. "We always have to wear a hair shirt," said one high-level career official.