The House last night passed a compromise version of the nickel-a-gallon increase in the gasoline tax and adjourned until next year, leaving up to the Senate the question of when and how to close out the 97th Congress.

In the Senate, however, conservative Republican Jesse Helms (N.C.) immediately picked up his filibuster against the administration-backed bill. In a test of wills with Majority Leader Howard H. Baker Jr. (R-Tenn.) and most of the rest of the Senate, Helms first said he was prepared to talk past Christmas to block the tax increase.

Later he relented and it was agreed that the Senate would recess until votes on cloture and then the bill itself Thursday morning.

But his refusal to permit a vote and adjournment last night produced an extraordinary departure from the normally elaborate politeness on the Senate floor.

"Seldom have I seen a more obdurate, more obnoxious performance," said Sen. Alan Simpson (R-Wyo.), and it appeared as if Helms, leader of the New Right and chairman of the Agriculture Committee, had cut his last ties with many even in his own party, almost isolating himself in the Senate.

Baker's problem was partly that senators wanted to go home; Minority Leader Robert C. Byrd (D-W.Va.) at one point suggested that he give up. But Baker persisted and filed his cloture petition.

On cloture Thursday, 60 votes will be required to shut off debate. If these are forthcoming, there will be a simple majority vote on the bill itself.

Up in the air along with the highway bill were two other major bills: an administration-supported farm bill to pay farmers with grain if they would curtail production next year, and the remaining part of President Reagan's Caribbean Basin initiative.

The House passed the gasoline tax-highway repair bill 187 to 80 after conferees worked from noon to nightfall to craft a compromise between competing House and Senate versions.

The gasoline tax would rise from the present 4 cents a gallon to 9 cents effective next April 1.

The bill also would eliminate federal taxes on passenger car tires but, in what turned out to be its most controversial provisions in conference, would raise federal user fees for heavy trucks.

The maximum tax on such trucks would rise to $1,900 by 1988.

The House had sought slightly more than that, while the Senate, pressed hard by the trucking industry, had asked for much less.

To compensate truckers, the bill also would increase the maximum allowable lengths and weights of trucks using the highways.

Transportation Secretary Drew Lewis said the administration was satisfied with the compromise, which would raise about $5.5 billion a year.

The compromise also included a stretch-out of existing federal unemployment compensation.

Workers would be eligible for an extra two to six weeks of benefits. That and the 320,000 jobs the administration said would come from the highway repairs to be financed by the 5-cent rise in the gasoline tax would be the only anti-recession programs to come out of the lame-duck session.

But others, including administration economists, have said that the highway bill would cost the economy more jobs than it would produce. And Helms and his fellow filibusterers have said that a recession is the wrong time for a tax increase in any case.

The conference report was also opposed by several environmental and consumer groups because it overruled state limits on truck size and weight.

"This bill will cause major safety problems by allowing double tractor trailers on the highways, which are now outlawed by several states," said Louise Dunlap of the Environmental Policy Center.

She said a recent Missouri referendum limited truck size to 73,000 pounds, while the bill allows 80,000 pounds.

Among Washington area legislators, Reps. Michael D. Barnes (D-Md.) and Steny H. Hoyer (D-Md.) voted for the compromise, as did Frank R. Wolf (R-Va.). Reps. Marjorie S. Holt (R-Md.) and Stan Parris (R-Va.) voted against.

The farm bill, a payment-in-kind measure intended to curtail production and increase prices, and another bill pushed by the Reagan administration, the Caribbean Basin initiative, faced possible fights on the Senate floor.

Sen. Paul E. Tsongas (D-Mass.) said he would refuse to allow consideration of the farm bill, claiming that it would be an unwarranted concession to Agriculture Committee Chairman Helms and the filibusterers.

Another group of Democratic senators, backed by organized labor, vowed to oppose the Caribbean Basin plan, which would ease trade and tariff terms with most Latin American countries. Labor said it would take jobs away from U.S. workers. Ranking Finance Committee Democrat Russell B. Long (La.) called it "just a lousy bill."

The White House, which had the bill on its list of high-priority measures for the lame-duck session, attacked the labor effort to kill it. "We think it is unfortunate that organized labor is opposing the bill," said White House spokesman Larry Speakes. "We believe their argument that it would cost jobs in the United States does not hold water."

At the beginning of yesterday's gasoline tax conference, before the lawmakers adjourned to a private room to hash out their disputes, they agreed to several provisions not directly related to the main thrust of the bill.

One would allow tax deductions for conventions held on American-flag cruise ships.

Another dispute between the House and Senate concerned the question of how states would share the revenues raised from the new taxes. One of those questions was cleared up before the conference.

The final bill proposes that all states would for the first time get back at least 85 cents in federal highway grants for every dollar they contributed.

While this provision provided a floor for how much money each state would get, sticking points remained over how the money would be divided for specific programs.

These battles pitted the larger, less populous states against the smaller, more urban states.

The current formulas for allocating highway monies are based on the number of miles of roads in a state, which has always been favorable to the large rural states. The House had attempted to change that in its bill. But in conference the Senate prevailed, keeping the existing formula in most of the new programs.

While the conferees bargained in the Longworth House Office Building all afternoon, the House chamber was empty and the Senate virtually deserted. So many House members had disappeared by mid-afternoon that there was some question as to whether a quorum would be present last night, but enough congressmen, 259, were found.

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) held his ritual news conference shortly before noon, saying he did not consider the lame-duck session a waste of time and that he was confident the gasoline tax and Caribbean Basin initiative would be approved.

Of Congress' inability to pass a jobs bill in the face of a veto threat by Reagan, O'Neill said, Reagan "believes a compromise is when we give him 80 percent of what he wants and he gives us nothing."

The House then opened for business for the day, and, after spending a brief time praising lame ducks and lamenting their legislative failures, went into recess at the call of the chair. The call came six hours later, and by that time many members had slipped out of town.

Before taking up the gasoline tax bill, the House approved a Senate-passed measure to slow down the review of Social Security disability cases, which had thrown thousands of people off the rolls across the country.

In 1980, Congress had ordered states to increase their reviews of disability cases in an effort to weed out unqualified recipients.

"It was a legitimate program that ran amok," said James M. Shannon (D-Mass.). "A lot of people were unfairly knocked off the rolls. This bill will slow down the process and allow them to collect benefits during their appeals."