The lame-duck 96th Congress fell back from the brink of adjournment last night as a rebellious Senate rejected a 17 percent congressional pay raise that the House approved only hours before without so much as a roll-call vote.

The dispute had grown so intractable by 10:45 p.m. that Congress simply threw up its hands and went home, deferring adjournment until tomorrow at the earliest.

Senate rejection of the more than $10,000-a-year pay hike came depite an appeal for it by President-elect Ronald Reagan and predictions by Senate leaders of both parties that it would pass if the House approved it.

So adamant was the Senate against the pay hike, which would also have raised the pay of 34,000 high-level bureaucrats, that it initially voted 57 to 14 to reject a mandatory omnibus spending bill to which the pay raise had been attached.

Disapproval of the spending resolution was considered the worst of all possible worlds because the resolution is needed to prevent many government agencies from running out of money by midnight tomorrow. But the agencies will face the same risk of having to close down if Congress cannot resolve the pay dispute -- and thereby complete action on the spending measure -- by that same deadline.

"I've never seen a time when the Senate had such trouble killing a snake," said Senate Minority Leader Howard H. Baker Jr. (R-Tenn.) as he joined Majority leader Robert C. Byrd (D-W.Va.) in begging the Senate to reconsider so the pay issue could be resolved without undoing all the previous work on the spending bill.

The Senate finally agreed, 41 to 29, to reconsider, and then voted 62 to 8 to insist on defeat of the pay raise although not the entire spending measure.

But when the pay issue was lobbed back to the House, the House reponded with a two-pronged counterattack. It voted to go back to conference with the Senate on the pay issue and, at the same time, offered the Senate a choice that was bound to make the senators squirm. The choice was: win on the pay raise issue but drop all or nearly all 148 spending add-on amendments that senators insisted in tacking onto the spending package.

This second line of attack involved a reneging on concessions that the House had made earlier in hopes of reaching final agreement and going home, including a retreat from its demand for stricter restrictions on abortions.

As the evening wore on, Republicans in the House whiled away their time by singing Christmas carols.

Earlier in the day, both houses had given up some of their pet causes in hopes of reaching final agreement on the spending measure and leaving town before the night was over.

The House abandoned its fight for stricter controls on Medicaid-financed abortions and agreed to Senate demands to permit government compensation for abortions in cases of rape and incest as well as risk to the life of the woman.

The Senate, bowing to stiff House resistance, agreed to give up its proposal for severance pay for Senate staffers who lost jobs because of last month's elections.

Earlier in the week, both houses had consented to drop language that would have banned the Justice Department from intervening in school desegration suits involving busing, thereby heading off a confrontation with President Carter. Carter had vowed to veto the whole spending measure -- as he did a similar bill yesterday -- if it included the anti-busing clause.

But they had left in the bill many of the special-interest spending ornaments that caused the measure to be ridiculed by members of both houses as a "Christmas tree." These baubles ranged from $2.7 milliion to bail out the Lake Placid Olympic committee to $150,000 to fight the asparagus aphid and $100,000 for pea research.

The proposed pay raise, boosting congressional salaries from $60,652.50 to $70,900 a year starting Jan. 1 suddenly came to life at the tail end of a House-Senate conference on the spending resolution Thursday night.

With anonymous proxies voted by Senate Minority Whip Ted Stevens (R-Alaska), the conference voted for the pay raise even though both the House and Senate had previously voted in favor of retaining an existing freeze on top level government salaries.

The conferee's action to lift the cap also had the effect of giving pay raises to high-ranking executive branch employees, whose salaries are tied by law to congressional pay limits.

Their scheduled pay increases had been held hostage to congressional reluctance to incur voters' wrath by voting to raise their own salaries. The conferees, reportedly encouraged by some leaders of both houses, apparently hoped to spare their colleagues this pain by locking the pay increase into the mandatory spending bill.

The action had the effect of shielding House members from a separate roll-call vote on the pay issue, but it did not give the same protection to senators.

In the House, a proposal to send the resolution back to the conferees for general reworking, which could have included dropping the pay raise, failed 172 to 71 in an unrecorded talley in which members stood to indicate their votes.

When Rep. Robert E. Bauman (R-Md.) requested a recorded roll-call vote, he failed to get the necessary and generally routine 44 supporters -- one-fifth of a quorum of the House -- to join him in demanding a roll call. So the resolution passed without members having to make their positions public.

But when the spending resolution, with the pay raise locked into it, reached the Senate floor, senators exploded in indignation.

Sen. Howard M. Metzenbaum (D-Ohio) threatened "extended debate" and Sen. Barry Goldwater (R-Ariz.) promised to vote against the whole package saying "it's the only way I can go home next week with safety. Otherwise I might end up in Iran."

Baker said Republicans had pushed the pay raise largedly because Reagan wanted it. "The Reagan administration feels that unless they can get the pay cap lifted, they're going to have a tough time staffing the administration," said Baker after talking personally with Reagan about it.

Until the pay issue is resolved, there can be no final action on the spending resolution, which continues spending authority for all government agencies for which appropriations bills have not been passed and signed into law. Agencies that fall into this category include such large departments as State, Justice, Commerce, Labor, and Health and Human Resources.

Previous stopgap spending authority, approved by Congress last fall after its appropriations bills bogged down, expires at midnight tomorrow. The new authority runs through June 28.