Defense Secretary Caspar W. Weinberger has warned that even if Congress approves the Gramm-Rudman proposal to balance the federal budget, President Reagan has not agreed to cut military spending to meet any "rigid formula" for reducing the deficit.

Weinberger's public statements that defense spending must not be impaired by Gramm-Rudman reflect private concern he has expressed in recent days that the military budget could eventually be seriously restrained if the legislation is approved by Congress.

Although Reagan has strongly backed the legislation, Weinberger, Secretary of State George P. Shultz and national security affairs adviser Robert C. McFarlane reportedly are among top administration officials who have questioned the wisdom and workability of the proposal.

House and Senate conferees are continuing efforts to reach agreement on the legislation, which requires elimination of the deficit by 1991.

In his endorsements of Gramm-Rudman, Reagan has repeatedly offered the caveat that "we'll continue spending what is necessary" for defense.

But Weinberger has warned privately that despite Reagan's pledge, the legislation may force cuts in the military expansion for the rest of Reagan's term. He made that concern public this week.

In an interview with the conservative weekly Human Events, Weinberger said, "We can't have our defense and our security policy be a total prisoner of a rigid formula designed to reduce the budget." He predicted that Reagan "would not feel required to make reductions in defense."

"The president wants to -- and would have to -- retain the authority to request the amounts for defense that the world situation and the threat to the United States requires. And when he first spoke about the Gramm-Rudman amendment, he did not interpret it as restraining or restricting or reducing the amounts that he thought necessary to request for defense in view of whatever world situation existed at the time," he said in the interview.

An administration official familiar with Weinberger's concerns said he is particularly worried that Senate-passed amendments to the proposal -- made after Reagan's decision to endorse the legislation -- could lead to significant military spending cuts. As part of an earlier budget compromise, Congress has supposedly committed itself to 3 percent growth in military spending above inflation each year for the next two years.

"The fact is, Gramm-Rudman is going to be a disaster for defense, and Weinberger knows that," the official said.

Officials said Weinberger criticized the proposal this week in a White House meeting with Republican congressional leaders and that there are indications Shultz is dubious about the proposal.

White House officials say they are aware of Weinberger's complaints. But these aides claim that Gramm-Rudman will focus budget cuts on domestic spending and that whatever military reductions that are made would otherwise have been made by Congress.

The proposal would require Reagan and Congress to meet deficit targets starting at $180 billion this year and declining about $36 billion each year, to zero in 1991. If the targets are not met, the president would be required to cut spending across-the-board in certain areas, including parts of the defense budget, in order to meet the deficit target. Social Security would be exempt from the automatic cuts.

The plan is already having an impact as new Office of Management and Budget Director James C. Miller III prepares the fiscal 1987 budget Reagan must submit to Congress in January, the beginning of an election year.

The Senate-passed version of Gramm-Rudman would require the president to submit a deficit no lower than $144 billion, compared with $180 billion for the current year. White House chief of staff Donald T. Regan has asked Miller to come up with a budget that meets the $144 billion target. Miller has estimated that if Gramm-Rudman triggered the automatic cuts, they would be split about evenly between military and domestic spending; Weinberger has said defense would suffer more than half the burden.

Specifically, Miller is attempting to preserve a 3 percent increase above inflation in military spending, without tax increases or cuts in Social Security benefits.

To meet the target:

*Reagan must hold off pending "budget-busting" legislation, such as the farm bill, that is threatening to send the deficit billions of dollars higher than projected in the budget compromise of August.

*Reagan must ask Congress in an election year for all the drastic budget cuts he sought last year in domestic spending -- termination of Amtrak, revenue sharing and the Small Business Administration, among other things.

*Reagan, at minimum, will have to find another $10 billion to $15 billion in domestic spending cuts, according to the estimate of a senior budget policy-maker.