Congress, flailing for ways to balance President Carter's budget for fiscal 1981, yesterday bumped into its own spending ceiling for the current fiscal year and had to cancel all further action on key spending bills.

The crunch immediately delayed floor action on two major pieces of legislation -- a Senate vote on a $197 million trade adjustment assistance act, and House action on an $8 billion foreign aid bill that contains assistance for Nicaragua.

The budget problem is expected to stymie action on a spate of key money bills in both houses for at least four more weeks, and possibly even longer if the two sides cannot agree quickly on how to resolve the situation.

Among the proposals apt to be caught in the crunch are renewal of the food-stamp program for the poor and President Carter's new plan to reinstate draft registration, which requires a $20 million appropriation.

The squeeze came as the Senate approved legislation that, ironically, would restrict the availability of extended unemployment benefits by repealing a plan that has offered them in all states whenever joblessness nationally rose sharply.

The vote, an overwhelming 89 to 3, came after Sen. Jacob K. Javits (R-N.Y.) complained the lawmakers were hurting the very group that most likely would suffer most from the current round of budget cutting.

Carter had requested the cutback as part of a group of money saving proposals in his budget.

The halt to congressional action on money bills came after the Congressional Budget Office reported that Congress had gone $10 billion over the $547.6 billion spending ceiling it approved last fall -- the first time it has exceeded its targets.

Under the new congressional budget process, the lawmakers cannot approve further spending bills until they agree either to cut spending elsewhere or pass a third budget resolution that raises the $547.6 billion ceiling.

Any spending bill brought up after the ceiling has been breached can be blocked by a point of order.

Congressional sources said yesterday it may be weeks before Congress can bail out of its bind. The House Budget Committee is not scheduled to consider a third resolution until March 18. The Senate will start a week later.

Moreover, Sen. Edmund S. Muskie (D-Maine), chairman of the Senate Budget Committee, has hinted that his panel may try to propose spending cuts instead, rather than following the House in seeking to raise the spending ceiling.

Any difference could delay agreement for weeks.

Yesterday, Muskie joined House Budget Committee Chairman Robert N. Giaimo (D-Conn.) in serving notice that he will urge his panel to propose a balanced budget for fiscal 1981 in the new resolution the lawmakers will consider this spring.

Meanwhile, Giaimo told House members he will schedule hearings early next week to consider proposals by various factions in the House for cutting spending in both fiscal 1980 and 1981 in an effort to reach a consensus.

Despite the budget-balancing fever in both houses congressional leaders have been concerned that the lawmakers would split badly over how to make the necessary cuts. Several groups have been meeting privately to work out a compromise.

Some of those strains showed up yesterday as Republicans on the Senate Budget Committee unveiled their own set of proposals for balancing the fiscal 1981 budget, concentrating entirely on domestic programs without touching defense.

Meanwhile, Rep. David R. Obey (D-Wis.), chairman of the House Democratic Study Group, warned that liberals would not agree to any budget-balancing move that did not include significant reductions in military outlays.

The breach of the $547.6 billion spending limit yesterday marked the first time in the history of the five-year-old budget process that Congress has exceeded its spending ceiling. All other budgets have stayed within target.

The excess was not the result of any particular legislation, but simply the consequence of higher-than-expected inflation and soaring interest rates, both of which have bloated government outlays.

Carter ran into the same situation last December in revising estimates of the fiscal 1980 budget situation when he discovered that higher inflation and interest rates had boosted his budget by $16 billion.

However, Carter has no formal ceiling to restrict his budget actions. The rule blocking further action on spending bills after the ceiling has been breached is the principal mechanism for enforcing Congress' spending limits.

The crunch cut off action on the draft registration legislation just as the White House was preparing to rescue the bill from an earlier defeat in a House Appropriations subcommittee.

The subcommittee had killed the measure on a 6-to-6 vote, but officials were confident of restoring it in full committee today. As of now, however, that effort appears to be moot.

The measure the Senate approved on unemployment benefits would save an estimated $351 million in fiscal 1980 spending, both by restricting the use of extended benefits and by tightening requirements for other jobless aid.

Under current law, the federal government offers all states an extra 13 weeks of jobless benefits for their unemployed workers whenever the insured unemployment rate nationally reaches 4.5 percent. It now is 3.05 percent.

The White House had argued that using this national average as a trigger-point was wasteful because it channeled benefits to states that did not have unemployment levels as well as those that did.

The extra 13 weeks' benefits would come on top of the basic 26-week package now offered to all states.

The action came as the administration continued its efforts to cut back the fiscal 1980 and 1981 budgets, with further meetings between key officials and top congressional leaders.

The White House disclosed yesterday that Carter had invited key senators and representatives to a private dinner late Monday night to discuss the budget situation. Those present reported no new developments.

Meanwhile, some lawmakers reported that top Carter officials were continuing to stand firm against a tax cut for 1980, holding out hope of a reduction in 1981 provided Congress balances the budget.

The fact that Congress bumped into its own budget ceiling was not unexpected. It was only the timing that came as a surprise. Some Capitol Hill leaders though they had several weeks.

The lawmakers missed one chance to avoid a budget crunch when the House refused last fall to enact several billion dollars worth of money-saving proposals that Congress had endorsed in an earlier tentative budget resolution.

Separately, the House Ways and Means Committee shied away from the budget scrap yesterday, postponing a fight over cutbacks in health benefits until the dust clears over the current situation.