Senate liberals, threatening to bring action on the tax cut bill to a halt yesterday, forced oil-state senators to give up an attempt to more than double the tax breaks for the oil industry in the administration bill.

An agreement, reached after a day of intense negotiation and a back-fired parliamentary maneuver by the Finance Committee chairman, cleared the way for Senate completion of the bill by a deadline of next Wednesday.

Under terms of the arrangement, Finance Committee Chairman Robert J. Dole (R-Kan.) agreed to withdraw an amendment that would have provided total tax breaks of $40.8 billion through 1990 and would have amounted to a significant dilution of the oil profits tax.

In return, liberal Democratic senators agreed to end a filibuster that had gone on for less than 24 hours. In a key test vote initiated by Dole, the opponents of the pro-oil amendment demonstrated that they had the votes to prevent cloture and block action on the legislation indefinitely.

Although a victory for northern Democratic senators, the Senate action cuts two ways in the convoluted political logic of the tax fight.

The two key swing groups in the House are the northeastern Republicans, known as the "gypsy moths," and the southern conservative Democrats, who have earned the nicknames of "boll weeivls" or "railbirds."

Among the gypsy moths, there had been growing complaints about the Dole-led effort to boost the tax break for the oil industry, and a number of these Republicans were threatening to defect to the Democratic bill or to vote against both partisan measures. "They are selling out to the oil industry to get the votes of the boll weevils," Rep. Silvio Conte (R-Mass.) coimplained before the Senate action about both the GOP and the Democratic bills. The decision of the Senate to back off from the Dole amendment is likely to increase GOP support for the Reagan bill.

Among the southern Democrats, however, the Democratic House Ways and Means Committee bill has becoming increasingly attractive, in large part because of provisions favorable to the oil industry that were added at 2:30 a.m. yesterday when the panel completed its version of the legislation.

Although oil interests are stronger in the Senate than in the House, the intense efforts by the House Democratic leadership to pull dissident southern Democrats back into the fold has resulted in a situation where the House bill is considerably more favorable to the industry than the Senate bill is.

Preliminary estimates by the Joint Committee on Taxation show that for the period from 1982 to 1986, the oil industry breaks would be $6.6 billion under the Senate bill and $7.1 billion under the House bill. Because of the complexities in calculating the cost of the House provisions, final estimates of the spread between the two bills are expected to show an even larger figure for the House.

Rep. Charles Wilson (D-Tex.), one of the principal negotiators for southern Democratic votes in behalf of the Democratic leadership, declared, "I will state categorically that we have the Republicans beat on this bill." Of the 29 Democrats who defected in the last budget vote to the Reagan administration, Democratic strategists claim they have received loyalty commitments on the tax bill from 12, acording to one estimate, up to 17, according to another.

These estimates are, however, highly tneuous because the Reagan administration has not begun in earnest the arm-twisting bargaining that proved highly successful in the efforts to gain approval of the Republican version of the budget, containing about $40 billion in budget cuts.

Adding to the complexity is the possibility that a number of Democratic liberals may refuse to vote for any bill because of the host of special-interest amendments that have been added to each. "We had the chance to be the party of frugality and low deficits, and we've blown it," Rep. Toby Moffett (D-Conn.), a leader of the liberal faction, said.

Today, Republicans organized by Rep. Stanford Parris (R-Va.) are to announce a $400,000 television and radio campaign in behalf of the Reagan tax bill. The commercials, which are scheduled to start in the Washington area today, include highly negative attacks on the Democratic measure, which the Gop'' has dubbed "the speaker's bill," in an effort to capitalize on what it believes is widespread public hostility in southern states to House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.).