Legislation to force a balanced federal budget within five years stalled last night as renewed White House concerns over the measure's likely impact on defense spending forced a new round of negotiations and postponement of a House-Senate conference committee meeting that was expected to approve the legislation.

Despite the White House misgivings, House Democratic leaders involved in the negotiations predicted that remaining problems would be resolved and the measure would be enacted by Congress.

"I don't think there are insurmountable issues, but they are important to resolve before we go to conference," House Majority Whip Thomas S. Foley (D-Wash.) said.

He said that top House and Senate negotiators, who continued to meet last night, still had an "agreement in principle" on the legislation and hoped to present a final, compromise version of the measure to the full conference committee today.

Congress, entering what it hopes is its last week of work for the year, is poised to take action on two far-reaching domestic initiatives -- the mandated balanced budget and a major overhaul of the tax system. Both are scheduled for a series of key votes this week.

Details of the last-minute snag on the balanced-budget measure were not disclosed, but they clearly stemmed in large part from renewed expressions of White House alarm over the prospect of deep cuts in the defense budget under the measure's automatic spending-reduction provisions.

Earlier yesterday, White House spokesman Larry Speakes voiced this concern, saying the administration wanted to be "absolutely certain" that the automatic cuts would not strike harder at defense than at domestic programs.

"We would like to reduce the hit on defense as much as possible so we can go forward with our security buildup," he said.

Office of Management and Budget Director James C. Miller III said some of President Reagan's advisers have "expressed some serious reservations about the reduction for defense that would be almost inevitable."

Miller said the legislation, as approved in principle late Friday, could result in a $5.8 billion cut in defense spending this fiscal year. Miller had estimated the defense cut at $2.9 billion as late as Friday, but officials said new calculations were made by the budget office over the weekend reflecting the congressional agreement.

White House officials said they were seeking further changes last night to protect defense spending. Defense Secretary Caspar W. Weinberger telephoned White House chief of staff Donald T. Regan yesterday to voice objections to the size of the defense-spending cut, officials said.

Speakes also indicated administration dissatisfaction with a compromise struck by the congressional negotiators in which the administration was given flexibility to allocate cuts within specific program categories of the military budget for next year only. He said the administration wanted to continue working with the conference committee, "particularly on the question of flexibility on defense."

Speakes described the White House problems with the bill as "more than fine-tuning" but said Reagan had not decided whether to sign or veto it. He indicated that the White House was still uncertain about the exact size of the overall spending cut that might be required this year.

One senior administration official, asked whether the administration's problems with the bill are serious, responded, "You bet . . . ."

But while the bill was stalled, at least momentarily, conference committee members remained optimistic about its chances of passage.

"I can't possibly visualize that you can blow it up at this point based on one of these technocratic arguments," said Rep. Marvin Leath (D-Tex.).

The compromise version of the bill, which was hammered out by a handful of top Senate and House negotiators from the committee, sets fixed targets for the federal deficit over the next five years.

Under the measure, the deficit, which is now hovering at about $200 billion for this year, would be reduced each year until the budget is balanced in fiscal year 1991.

Automatic, across-the-board cuts in most federal programs would be imposed to reach the targets if Congress exceeded the deficit limits in any year.

Social Security benefits and several antipoverty programs would be exempt from automatic cuts, while Medicare and other health programs would be subject only to limited cuts in the compromise version.

Other spending reductions necessary to meet the deficit targets are to fall evenly on domestic programs and the defense budget, according to the conference committee draft of the legislation.

If enacted by Congress and signed by President Reagan, the measure would have an immediate impact on the federal government. It sets a deficit target for the current fiscal year of $172 billion, about $20 billion less than projections of the actual deficit. Even after reducing the size of the first mandatory spending cut, because the current fiscal year is already under way, the bill would require up to about $12 billion in budget cuts to be imposed by March 1.

Moreover, the mandatory deficit target set for the fiscal 1987 budget that Reagan will submit to Congress early next year is $144 billion, a figure that will require substantial reductions from current spending levels.

The legislation includes no provisions for tax increases, but there is a widespread feeling in Congress that the president will be forced to accept a tax increase if the mandatory deficit targets are to be reached without seriously reducing defense spending and crippling a number of popular domestic programs.

The balanced-budget legislation was at the top of the agenda yesterday as Congress began its most critical week of the year in terms of budget and spending issues.

House leaders are aiming to adjourn for the year on Friday, while the Senate, according to Majority Leader Robert J. Dole (R-Kan.), still has "a little glimmer" of hope of doing the same.

But before the end of the week, Congress must pass, at a minimum, legislation raising the national debt ceiling as well as an omnibus spending measure to fund government agencies through the rest of the current fiscal year.

The balanced-budget proposal is an amendment to legislation that would raise the debt ceiling to more than $2 trillion.

The deadline for its enactment is midnight Wednesday, when a temporary debt-ceiling extension passed by Congress last month expires. Failure to lift the debt ceiling could lead to an unprecedented government default by the end of the week as the government, stripped of its authority to borrow, would be unable to meet its obligations.

The $498 billion omnibus spending bill must be passed by midnight Thursday, when current stopgap funding authority for the government expires.

The so-called continuing resolution is needed to fund government agencies for which Congress has not passed regular appropriations bills for the fiscal year that began Oct. 1.

So far, Congress has passed seven of 13 appropriations measures. Four have been signed by Reagan, one has been vetoed and two are awaiting presidential action.

Reagan, contending that a version of the spending measure passed earlier by the House and the version the Senate began debating yesterday both exceed congressional budget targets, has threatened to veto the legislation. Congressional leaders dispute this, maintaining that both versions are within overall congressional budget targets.

In addition to the two fiscal deadlines Congress faces this week, the House is expected to vote Wednesday on tax-reform legislation, while a separate House-Senate conference committee continues work on so-called budget "reconciliation" legislation.

The reconciliation measure is designed to make permanent spending cuts and revenue increases sufficient to cut deficits by $20 billion next year and by a total of between $78 billion and $85 billion over the next three years.