House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) told the Carter administration yesterday that the House is cool to a 50-cent-a-gallon gasoline tax increase, as the Senate cleared the way for a major vote on the administration's oil "windfall profits" tax bill.

Sources said O'Neill told Treasury Secretary G. William Miller and presidential advisers Charles L. Schultze and Stuart Elizenstat that the House has never favored stiff gas tax increases. A 50-cent increase or gas rationing is under consideration by the administration to help curb consumption.

Meanwhile, the gas tax issue surfaced in the presidential race. Rep. John B. Anderson (Ill.), a Republican contender, introduced a bill to raise the gas tax, now four cents, by 50 cents and use most of the proceeds to provide Social Security tax relief.

Anderson conceded that some people would have to move closer to their jobs if his plan is adopted but said that "this is a dramatic way to immediately reduce demand and I think to send a very important signal to OPEC . . . that this country means business."

In the Senate, at least a test vote was scheduled tomorrow on a major administration-backed proposal to increase revenues expected from the oil tax bill by setting minimum levies for various categories of oil that have been exempted from the overall tax.

The idea is to get the tax's anticipated yield over the next decade closer to a goal of $185 billion than the bill's present provisions, which would produce about $155 billion. This would still be far short of the $277 billion tax approved by the House, but it would strengthen the administration's hand in seeking the strongest possible tax from a House-Senate conference on the hill.

Senate Majority Leader Robert C. Byrd (D-W.Va.) filed a cloture petition last night to limit debate and reduce the number of extraneous amendments to the tax bill. A vote on the petition is scheduled tomorrow.

Byrd's goal is to finish the bill by the end of the week so work on Chrysler Corp. relief legislation can be finished before Congress quits for Christmas. Indicating impatience with the Senate's progress, Byrd said he might call the Senate back into session Dec. 27 if it has not finished work before the Christmas recess, set to begin Dec. 21 or 22.

During slow-moving action yesterday, the Senate rejected another effort, backed mainly by Republicans, to write some overall fiscal restraint into the oil tax bill.

This proposal, defeated 40 to 43, would have limited federal spending to a fixed percentage of the nation's gross national product, roughly similar to a bill that House Budget Committee Chairman Robert N. Giamo (D-Conn.) is pushing.

The Senate earlier rejected, by five votes, another Republican proposal to limit tax revenues to a percentage of the nation's output, a plan critics said would guarantee huge budget deficits.

The proposal's chief sponsor, Sen. William V. Roth (R-Del.), argued that the amendment would reduce federal spending by $12 billion in 1981, producing a budget surplus of about $25 billion to be used for either tax cuts or debt reduction.

Although the latest proposal would limit spending as opposed to tax revenue, Senate Budget Committee Chairman Edmund S. Muskie (D-Maine) argued that it announced to "vigilante" budget-making. He said the ceiling for 1981 was not high enough to accommodate defense spending increases supported by Republicans. For future years, he said, the spending limits were higher than those anticipated by the Budget Committee.

The Senate also rejected, 56 to 29, a proposal to let oil producers escape the windfall profits tax if they plow their proceeds from oil price decontrol back into energy-producing business.

The proposal, by Sen. James A. McClure (R-Idaho), would have imposed a tax of 90 percent on all profits above the oil industry's average rate of return, but would have exempted producers from paying it if they reinvested the money in energy exploration and development. Critics said McClure's proposal would have gutted the tax bill.

At least one other "plowback" proposal is pending.

The Senate also approved favorable tax treatment for oil produced from Alaska's Cook Inlet, but rejected a more ambitious "plowback" proposal for Alaska.

Meanwhile, conferees on legislation to create an energy mobilization board, which is designed to cut red tape on energy projects, met inconclusively for two hours as they squabbled over the board's makeup. A synthetic fuels bill conference, where conferees quickly agreed Friday to board outlines of a bill that calls for a $20 billion program administered by a new energy security corporation, will meet again today.