The Senate flinched from the prospect of spending self-discipline last night.

It approved a $547.6 million federal budget for this fiscal year, after first dropping an enforcement provision it had voted earlier to keep committees from surpassing that total.

The House had refused to accept that "reconciliation" provison, and Senate leaders felt they had no choice but to cave in.

The Senate vote -- 57 to 20 -- leaves unanswered what Congress will do if, as expected, it starts to exceed its self-imposed spending ceiling. w

One alternative would be simply to yield to its spending instincts and raise the ceiling, as has been done before. But Senate budget leaders yesterday warned they would resist that this year, and they put nonbinding language in the budget resolution to that effect.

If Congress fails to raise the ceiling, the first spending bill to exceed it will be subject to a point of order on grounds it violates the budget rules.

But there also may be parliamentary means of dodging the point-of-order provision, so Congress conceivably could end up busting its budget next year without having to admit it had done so. Budget leaders would fight such a subterfuge, however.

The original reconciliation provision the Senate earlier adopted in a show of self-discipline would have avoided these future problems by instructing various committees now to cut back spending bills already passed or being considered. House Republicans favored this, but House Democrats balked -- and defeated it.

Under the budget resolution, which still must go back to the House for a last vote, the projected deficit for the fiscal year that began Oct. 1 is $29.8 billion. Both houses anxious to prove themselves conscientious on the spending issue had struggled to keep the deficit below $30 billion.

The Senate originally did this by cutting domestic programs in favor of defense spending, which it raised. The House went the other way. In the end, both houses got their way: both the defense and domestic budget figures rose. The Budget committees held down the deficit mainly by raising their revenue estimates, on the grounds that inflation would increase taxes, just as it does everhing else.

Senate budget leaders warned yesterday that without the orderly reconciliation procedure, some high-priority spending could end up being "crowded out" of the budget under a pont of order. Without reconciliation, such crowding-out could be random, affecting whichever spending bills come to the floor last.

One reason budget experts expect Congress to surpass its new spending totals is that the House voted Thursday to kill President Carter's hospital cost containment plan. They had assumed this measure would save more than $500 million in Medicare and other health outlays.