The oil industry lost a first test vote on the windfall profits tax yesterday as the Senate began what promises to be lengthy debate on the controversial proposal.

By 53 to 32, the Senate rejected a surprise move by oil-state senators to reduce the levy the House has voted to impose on the so-called windfall profits that oil companies will reap from the administration's decontrol of oil prices.

The vote was largely symbolic because it would not affect the Senate Finance Committee's already scaled down version of the tax. That is the version which along with scores of amendments, is pending before the Senate.

But it appeared to give little encouragement to those who would alter the Finance Committee bill, which Chairman Russell B. Long (D-La.) is attempting to sell to the Senate as a "balanced" compromise.

The committee's bill would recapture an estimated $138 billion of the nearly $1 trillion that oil companies are expected to receive from decontrol over the next 10 years. The House bill would take in about $277 billion. The proposal the Senate rejected yesterday would have trimmed the House bill's anticipated revenues to about $240 billion.

Yesterday's vote also gave oil-state lawmakers a chance to twit the Carter administration, which is urging the Senate to increase the bite of its proposed tax to bring it more in line with the House bill.

In proposing to cut the House-approved tax from 60 percent to 50 percent on most categories of oil, Sen. David L. Boren (D-Okla.) reminded the Senate that the administration's intitial proposal was for a 50 percent levy.

"The president was right then," said Boren, and a 50 percent tax "makes even more sense today. It's an opportunity," he added, to the amusement of his colleagues, "to support the president of the United States."

Sen. Howard M. Metzenbaum (D-Ohio), one of the Senate's strongest critics of oil price decontrol and of the watered-down committee measure, saw another possible motive behind the vote. It gave wavering senators a chance to "make a record against being too good to the oil companies," by voting now against the Boren proposal and later supporting the Finance Committee's bill.

For Long, an oil-state senator who supported Boren, the vote also showed that there are those who would reduce the tax as well as those who would increase it -- all the better for the committee bill's reputation as a compromise.

In opening arguments on behalf of the committee proposal, Long characterized it as "the largest tax increase ever levied on a single American industry." At the same time he said it was a "compromise between revenue considerations and the need to provide the proper production incentives" for the oil industry.

Along the same lines, Sen. William V. Roth (R-Del.), a Finance Committee backer of the bill, described it as a "compromise between those who would want to kick the hell out of Big Oil and those who want to get in bed with Big Oil."

On the other hand, Metzenbaum said the central issue should be President Carter's decision to decontrol oil prices. He described it as "by far the biggest giveaway in American history" and a probable disincentive to expanded oil production because companies stand to rake in more profits by waiting until the tax phases out. Unlike the House bill, the Senate proposal would phase out the tax over a number of years.

Boren characterized his proposals as a clearcut choice between increasing oil production or increasing taxes. Even at his lower rate, said Boren, it would still be "larger than any tax increase in history."

Among Washington area senators, Harry F. Byrd Jr. (Ind.-Va.) and John W. Warner (R.-Va.) voted for the lower rate, while Paul S. Sarbanes (D-Md.) voted against it. Charles Mathias (R-Md.) did not vote.