House Democrats considering legislation to force a balanced federal budget by 1991 yesterday demanded an explanation of Defense Secretary Caspar W. Weinberger's assertion that President Reagan would not be bound by a "rigid formula" calling for deep cuts in defense spending to meet the bill's deficit-reduction targets.

Reps. Richard A. Gephardt (D-Mo.) and David R. Obey (D-Wis.) said they have asked that administration officials, possibly including Weinberger, be called before the House-Senate conference committee considering the measure.

If those officials maintain that Congress cannot dictate to the president where to make spending cuts mandated by the legislation, Gephardt said, "I think this whole thing is deeply flawed."

While stopping short of predicting defeat of the politically popular measure, Obey said such a claim by the administration would "drive a hell of a lot of nails" into its prospects in the House.

Gephardt and Obey are members of the conference committee and chairmen of two of the four task forces established by House conferees to consider details of the legislation.

The bill as passed by the Senate calls for the deficit to be cut by $36 billion each year until the budget is balanced in 1991. It requires that the president submit and Congress adopt annual budgets meeting these targets.

If Congress fails in this, the president would be required to impose across-the-board reductions in most federal programs, except Social Security, to bring the budget within the deficit targets.

Reagan has endorsed the legislation, but Weinberger, reflecting administration concern that automatic, across-the-board cuts would strike particularly hard at the Pentagon budget, this week intensified doubts about how the measure would work.

In an interview with the conservative weekly Human Events, Weinberger said, "We can't have our defense and our security policy be a total prisoner of a rigid formula designed to reduce the budget."

When Reagan endorsed the legislation, Weinberger said, "he did not interpret it as restraining or restricting or reducing the amounts that he thought necessary to request for defense in view of whatever world situation existed at the time. He can't have his constitutional authority constrained to the point where his only consideration is the attempt to reduce deficits."

Although Weinberger referred only to presidential budget requests, Gephardt and Obey said the interview suggested that Reagan, invoking his authority as commander-in-chief, might seek to ignore congressional instructions that across-the-board budget cuts include defense.

They said they wanted to question Attorney General Edwin Meese III, budget director James C. Miller III and possibly Weinberger on this point.

"One of the critical points is the question of presidential discretion," Gephardt said. The conference committee, he noted, is determined to draft legislation granting the president virtually no leeway in determining where to make cuts.

"If there develops a good deal of doubt on that issue, I think it the deficit-reduction legislation is in deep water with both Republicans and Democrats," Gephardt said. "The question is, can the president change the priorities set by Congress and get away with it."

Sen. Bob Packwood (R-Ore.), the conference committee chairman, earlier in the week strongly disputed suggestions that the defense budget could be sheltered from automatic cuts.

"As far as we are concerned, defense is in the pot and we have no intention of leaving it out," he said. "If it is not in, then there will be no [deficit-reduction] process."