Administration-backed legislation to raise the federal gasoline tax by a nickel a gallon suffered a potentially fatal setback last night as Senate Republican leaders pulled the bill off the floor in the face of a filibuster against it by party conservatives.

An aide to the leaders said they were doing so to pave the way for quick action on stopgap funding for the government that is needed to avert a shutdown of most departments and agencies after current funding expires at midnight.

After hours of fruitless efforts to end the filibuster, Majority Leader Howard H. Baker Jr. (R-Tenn.) told a weary, frustrated and testy Senate that he was reluctantly shelving the measure but not necessarily permanently.

And Senate Finance Committee Chairman Robert J. Dole (R-Kan.) told reporters he wants the Senate to stay in session over the Christmas holidays to finish the bill, and believes that President Reagan agrees. He said chances for passage would be "better than even" if Reagan were asked Congress to approve the bill before adjourning.

But other sources said that passage is now doubtful because government funding is no longer being held hostage for passage of the gas-tax increase. "It cast a lot of doubt that the tax increase will pass," said a leadership aide.

The increase, which would raise the gas tax from 4 to 9 cents a gallon on April 1, was proposed by Baker and House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) and endorsed by Reagan as a way of financing $5.5 billion in job-creating highway, bridge and transit work.

The bill, which the House passed last week, was to have been the centerpiece of the lame-duck session of Congress, which is now expected to achieve little more than passage of a continuing resolution to keep the government operating until next spring, or possibly later.

In other action on the continuing resolution, the Senate last night voted, 57 to 41, an additional $475 million in aid to Israel.

Still in doubt is whether Reagan may veto the stopgap measure because the bill, as it stands now, includes money for public works jobs that he does not want.

Sources said all but four Senate Republicans--Don Nickles (Okla.), Jesse Helms (N.C.), Gordon J. Humphrey (N.H.) and John P. East (N.C.) -- agreed to put the gas-tax bill aside if an arrangement for final passage could not be nailed down by mid-evening.

Then, according to a leadership source, Helms, Nickles and Humphrey indicated they would continue the filibuster, and Humphrey "indicated he would stay until Christmas and didn't care if the government closed."

Once the gas-tax bill was withdrawn, the Senate immediately launched into what was expected to be an all-night session on the continuing resolution. The swift movement improved -- but did not guarantee -- chances that Congress will finish the stopgap funding measure before government offices are scheduled to reopen Monday.

Earlier in the day, the Senate failed to choke off the filibuster against the gas-tax increase and Republican filibusterers spurned a bid by Democrats for a mutual agreement to drop all delaying amendments.

The initial effort to break the talkathon collapsed yesterday morning as liberal Democrats who wanted to add money to the bill for unemployment benefits joined with conservative Republicans who are opposed to the gas-tax increase to defeat a so-called cloture motion to limit debate.

The motion fell 12 votes short of the 60 needed for passage.

By 93 to 4, the Democrats eventually won Senate approval of a bipartisan compromise that would provide about $540 million for two to six weeks of extended unemployment benefits, depending in part on a state's jobless rate. Virginia and the District of Columbia would get two extra weeks; Maryland would get four or six additional weeks.

In hopes of easing truckers' opposition to the bill, the Senate also voted to stretch out and lighten the burden of new excise levies on heavy trucks, including reducing the proposed per-truck tax limit from $1,600 to $1,200. The current tax is about $240.

But the Senate still remained paralyzed through most of the day, and Baker vowed early in the afternoon to keep it in session all night if necessary.

He said he would order nonstop deliberations until the Senate finished the gas tax and took up the huge "continuing resolution" to fund the government after existing spending authority for most departments and agencies expires at 12:01 a.m. Saturday.

"We've gotten hold of a tar baby," Baker told the Senate at one point of almost total legislative gridlock. "I don't doubt we've got a sticky issue and we've got sticky senators."

As of early evening yesterday, Baker, using the threat of a government shutdown as leverage for passing the gas-tax increase, was refusing to drop the stalled tax measure and move on to the continuing resolution.

"We will stay here and finish even it it means they the filibusterers decide to close down the government," said Baker's press secretary, Tom Griscom.

At the White House, deputy press secretary Larry Speakes said the Office of Management and Budget was sending shutdown directives to government offices for use if the stopgap funding bill -- which is also threatened by a presidential veto -- is not approved and signed into law by midnight.

Vital services would be continued, Speakes said.

The resolution faces a possible veto because it includes money for jobs creation--$5.4 billion in the House-approved version and $1.2 billion in an alternative approved Wednesday by the Senate Appropriations Committee -- that Reagan doesn't want.

Despite earlier indications that Reagan might accept some jobs money, Speakes said yesterday that "there has been no talk of compromise."

Even the scaled-down Senate measure "is the type of bill that we do not favor," he said.

Congressional sources said privately that they believe a veto is a foregone conclusion, meaning that, even with resolution of the gas-tax dispute, time would be extremely short for final action on the continuing resolution before government offices are scheduled to reopen Monday.

Only fiscal 1983 appropriations for the legislative branch, military construction, the Department of Housing and Urban Development and some independent agencies have been signed into law so far, meaning that everything else has to be funded through the continuing resolution.

Only two or three other money bills are expected to be enacted within the next few days.

Most of the government was forced to shut down for a day a year ago until Reagan and Congress resolved a dispute over spending levels in a continuing resolution.

This year the problems are more complex, although some lawmakers were saying yesterday that frustrations are running so high that Congress may decide at any time to compromise on the continuing resolution and go home.

The impasse over unemployment insurance arose when the Democrats, taking the Senate Republican leadership by surprise, succeeded on a 50-to-47 vote in blocking an effort to shelve their proposal for additional unemployment benefits for workers who have exhausted their jobless aid.

Six Republicans, mostly from states with high unemployment, joined most Democrats in the vote.

When Republican leaders tried to block a final vote on the issue before cloture, which would have prevented further votes on such proposals, the Democrats decided the only way they could win on unemployment insurance was to vote against limiting debate.

All but seven of them did so, along with conservative Republican opponents of the gas tax and some liberal Republicans.