The Senate broke its deadlock on the administration's proposed oil tax yesterday, ending a three-day filibuster and pointing the way toward speedy approval of the huge bill, perhaps this weekend.

The breakthrough on the tax bill came after oil-state and consumer-state leaders agreed in intensive behind-the-scenes negotiations on a compromise that would raise about $178 billion by 1990. The tax would be levied on the extra money oil producers will make from President Carter's decision to phase out federal price controls on crude oil.

So welcome was the compromise to the filibuster-weary senators that it breezed through the Senate without argument on a vote of 78 to 13.

The tax breakthrough occurred as, separately, House and Senate conferees reached tentative agreement on a $20 billion synthetic fuels production program proposed by the president. [Details on Page A6]. The two actions gave Carter one of his best days in recent memory on energy in Congress.

The anticipated oil tax yield of $178 billion over the next decade if $40 billion more than the Senate Finance Committee recommended and about $7 billion short of the goal set by Senate Democratic leaders -- repesenting a substantial victory for the White House in its drive to toughen the Senate version of the tax.

"It's significantly better than the original bill; it's significantly lower than the House bill," said White House press secretary Jody Powell.

"We'll continue to struggle for the average American," he said, indicating the administration would press the conferees to settle closer to the House version.

The tax as proposed by the president and passed by the House would recapture for the government about half of the so-called "windfall" profits from decontrol. The Finance Committee bill would have recaptured about a fourth. Yesterday's compromise would take away about a third, leaving the oil companies with two-thirds.

The House's earlier approval of a tax that would raise an estimated $277 billion by 1990, as the president proposed, virtually assures a final House-Senate compromise in excess of $200 billion, perhaps in time to reach the president's desk by Christmas.

The breakthrough, after more than four weeks of slow-moving Senate action, came in the form of an agreement to impose a so-called "minimum tax" of 10 percent on newly drilled oil, along with a 20 percent tax on so-called heavy oil and oil produced by expensive chemical methods. In each case the tax would fall only on the proceeds from decontrol.

The Finance Committee had proposed that these categories of oil be exempt from the tax. Republicans and oil-state senators held out to keep newly discovered oil, especially, tax free. But consumer-state senators, supported by the administration, wanted a tax of 20 percent on both these forms of oil and refused to budge below 10 percent, although they finally relented by allowing a higher base price threshold for imposition of the levy.

While the vote for the compromise was lopsided, a later vote on the minimum tax concept as a whole was closer, 52 to 38. The minimum levy would raise about $23 billion on top of the $155 billion that the Senate already approved in expanding the Finance Committee's draft.

The end of the filibuster followed a vote earlier in the day that indicated strongly that the Senate would have invoked debate-limiting cloture Monday.

While Senate Majority Leader Robert C. Byrd Jr. (D-W.Va.) fell four votes short of the 60 necessary to impose cloture in his third try to end the filibuster, at least four absent senators had voted previously for cloture and were expected to be on hand Monday, according to Byrd. The near-certainty of cloture, along with the Senate's general weariness and the approach of the Christmas recess, were factors in hastening the compromise, which had seemed near on several occasions, only to slip away at the last minute.

After a round of mutual congratulations on the Senate floor, Byrd told his collegues they will be holding a rare Saturday session today in hopes of completing action on the bill and sending it to conference by tonight. If the Senate completes action today or early next week, there is a chance that Congress will finish action on the measure by Dec. 21 or Dec. 22, thus avoiding the mid-holiday sessions that Byrd has threatened if the bill is not wrapped up by then. Congress also has the bill to aid the Chrysler Corp. on its agenda before adjournment.

The only major obstacle to passage of the bill appears to be a politically explosive proposal by Sen. John C. Danforth (R-Mo.) to apply the tax to oil royalties earned by states from state-owned lands. Senate Finance Committee Chairman Russell B. Long (D-La.), whose state could lose more than $1 billion over the next decade if the proposal were to pass, has indicated he would filibuster the bill to death if Danforth's amendment were approved. But Senate sources have said Danforth probably does not have the votes to get his proposal approved.

Byrd picked up four votes in his move to invoke cloture, three from switches and one from the return of an absentee senator. But he lost three other votes because of absences and one from a switch, balancing out to the same 56 votes he had last Thursday.

Sen. Bob Packwood (R-Ore.), a former cloture supporter, switched his vote yesterday to protest a maneuver by Byrd earlier in the week to schedule the second cloture vote at the Democratic majority's convenience. Packwood protested that Byrd was paving the way for a trampling of minority rights in the Senate.

Among the absentees on the last two votes was Sen. Edward M. Kennedy (D-Mass.), who was campaigning in California. Byrd indicated all the absentees, presumably including Kennedy, could have been expected to be on hand Monday if their presence had been required to invoke cloture.

Senatorial irritability after five weeks of wrangling over the bill surfaced again yesterdy as the Democrats stalled the cloture vote to accommodate tardy colleagues. At one point, Minority Whip Ted Stevens (R-Alaska) inquired acerbically if they were waiting for Kennedy.

Byrd's claim to have enough votes to get cloture Monday was based on the fact that four of yesterday's absentees -- Kennedy, Charles Mathias (R-Md.), Herman E. Talmadge (D-Ga.) and Frank Church (D-Idaho) -- previously had voted to cut off debate. Byrd, apparently anticipating more vote switches as well, said Monday's vote likely would have been 61 or 62 for cloture.

Among Washington area senators Harry F. Byrd Jr. (Ind.-Va.) have switched from the anticloture to pro-closture side, leaving only John W. Warner (R-Va.) opposed to shutting off debate.