Back at work after a fruitless 17 1/2 hour session that had bleary-eyed senators snoozing on cloakroom sofas and responding to roll calls well past midnight, the Senate yesterday moved closer -- but not close enough -- to ending a filibuster on the administration's proposed oil tax.

Meanwhile, a behind-the-scenes effort to arrive at a compromise on the size of the tax almost succeeded, only to fizzle at the last minute.

Negotiations were to continue today before a third attempt is made to end the filibuster. "We're close but we're still not there," said Sen. Henry M. Jackson (D-Wash.).

The administration reportedly was displeased at the suggested compromise, which would have produced substantially less money than the Senate indicated in a test vote on Wednesday that it might be willing to approve. It was that vote that triggered the filibuster.

In its second cloture vote in as many days, the Senate came within four votes of the 60 necessary to limit debate on the measure, which would tax the proceeds from the president's gradual lifting of price controls on crude oil.

This was three votes more than cloture got Wednesday. "Counting absentees and possible switches, cloture -- limiting the Senate to 100 hours of debate -- might be invoked today or tomorrow.

"I'm encouraged by the vote," said Senate Majority Leader Robert C. Byrd (D-W.Va.). " . . . I think it means the Senate wants to see a final decision on this measure."

The dispute over the size of the tax involves a minimum tax on various categories of oil that are thus far exempt under the Senate's watered-down version of the administration's tax proposal. The administration and the tough-tax bloc in the Senate want to apply a levy of 20 percent to both newly discovered and expensive-to-produce oil, which would bring the overall yield of the Senate tax to at least $185 billion by 1990. Many Republicians and oil-state senators want no tax -- or at least a lower tax -- on newly discovered oil, and are pushing to keep the overall figure below $185 billion. The near-compromise was reported to be about $173 billion.

Late yesterday, four Republican senators who apparently never give up offered to vote for cloture in exchange for a tax cut of $25 billion to $34 billion in each of the next three years. The four were Sens. William V. Roth (Del.), David Durenberger (Minn.), Rudy Boschwitz (Minn.) and Gordon J. Humphrey (N.H.). The Senate's Democratic majority has several times rejected similar GOP attempts to attach general tax cuts to the oil bill.

Complicating the situation is a move by Sen. John C. Danforth (R-Mo.) to divert to the federal treasury billions of dollars that oil-producing states such as Louisiana anticipate when royalties and state severance tax revenues skyrocket because of price decontrol. Senate Finance Committee Chairman Russell B. Long (D-La.) has not taken kindly to the idea and, because of it, is opposing cloture on the very bill that his committee put together.

Danforth's emphasis has shifted from taxing state oil royalties to forcing oil producers to pay a higher federal tax if oil states profit from decontrol by increasing their severance tax take. This has not mollified Long.

The first proposal is "unconstitutinal . . . in three different ways" he told reporters yesterday. The second, he added, is simply "ridiculous."

Long, although he is at the center of the present imbroglio, was among the fittest of the senators on their second filibuster day because he went home early. "All I thought I could do is encourage them by sittin' around," he explained, smiling.

Although the marathon session that ended shortly after 3:30 a.m. produced little in the way of legislation, it left behind a few stacks of ruffled feathers.

Brushing aside the custom that senators address each other only as distinguished gentlemen, Sen. Lowell P. Weicker (R-Conn.) accused Majority Leader Byrd of indulging in "mealy-mouthed threats" in telling senators they would be spending the bulk of their Christmas holiday at their desks if they didn't act swiftly on the tax bill.

Criticizing Byrd's leadership, Wiecker accused him of a "propensity . . . not to pull the U.S. Senate together but to make sure it stands at odds." Weicker said Byrd has a "niggling" approach to leadership that avoids major issues.

Weicker's comments were immediately challenged by Sen. Bob Dole (R-Kan.), the minority's floor leader on the tax bill, who praised Byrd and said he was in "total disagreement with the distinguished gentlemen from Connecticut."

Later, Byrd, backed up by a straight party-line vote, succeeded in blocking Republicans from forcing a second cloture vote at 1 a.m., rather than during normal working hours. This rankled some Republican senators who charged that Byrd was tampering with established Senate procedures and vowed that he would regrt his acton.