The Republican-controlled Senate yesterday rejected two Democratic deficit-reduction proposals that included tax increases to help fend off the deep domestic-spending cuts contained in a plan worked out between Senate Republicans and President Reagan.

But the voting pattern indicated the possibility of support across party lines for some combination of tax increases, limits on cost-of-living adjustments of Social Security benefits and major restraint in defense spending.

It showed that, depending on other elements of an overall deficit-reduction plan, there were at least six Republican votes for tax increases and 29 Democratic votes for a cutback in Social Security benefit increases. Republicans have generally opposed tax increases, while Democrats have opposed tampering with Social Security.

However, any compromise along these lines would probably run into stiff opposition from Reagan. He has expressed strong opposition to tax increases and further cuts in his original defense proposal and agreed reluctantly to a curtailment of cost-of-living increases for Social Security. The Social Security change is contained in the compromise he worked out with GOP senators.

In preparation for drafting a final compromise, Senate Majority Leader Robert J. Dole (R-Kan.) has indicated willingness to negotiate with Democrats but has said he will not propose anything that does not have White House support.

In yesterday's action, the Senate rejected, on a 63-to-35 bipartisan vote, a plan by Sens. Lawton Chiles (D-Fla.) and Ernest F. Hollings (D-S.C.) that would have limited Social Security benefit increases in addition to raising taxes and restraining defense spending.

On a more strictly party-line vote of 54 to 43, it then turned down a plan from Minority Leader Robert C. Byrd (D-W.Va.) that generally paralleled the Chiles-Hollings plan but would have retained full cost-of-living increases for Social Security.

Byrd's plan also included slightly more defense spending and less of a tax increase than the Chiles-Hollings plan.

Brushing aside White House and Senate leadership opposition to tax increases, six Republicans -- Mark Andrews (N.D.), Rudy Boschwitz (Minn.), William S. Cohen (Maine), Mark O. Hatfield (Ore.), Nancy Landon Kassebaum (Kan.) and Charles McC. Mathias Jr. (Md.) -- voted with 29 Democrats for the Chiles-Hollings plan.

Mathias was the only Republican to vote for the Byrd plan, however. Four Democrats voted against it: Alan J. Dixon (Ill.), Hollings, Sam Nunn (Ga.) and William Proxmire (Wis.).

Rejection of the Democratic plans came as GOP leaders indicated that they, too, lacked the votes to pass their plan, although they contended that they were only a vote or two shy of a majority.

With the outcome apparently hinging on one or two votes, prospects for final action this week were complicated as a second Republican senator, Pete Wilson (Calif.), was hospitalized. Wilson had an appendectomy; Sen. John P. East (R-N.C.) has been hospitalized since April 20 with an acute thyroid deficiency, according to his office.

"That makes it difficult . . . . We're a little short-handed," Dole said, conceding early in the day he had no more than 48 or 49 votes for the Republican plan and does not plan to put it to a vote until he has a majority.

However, Dole indicated later he still hopes for a final vote today. "If we can finish Thursday , we ought to finish," he said. "And when I say finish, I mean we win."

In an effort to nail down the votes of farm-state senators for the Republican plan, a compromise was worked out late in the day to expand foreign sales of grain and restore funds for some agricultural programs, including soil conservation and crop insurance. There was also an agreement to restore all but $200 million of the proposed cuts in guaranteed student loans.

Farm programs had been regarded as perhaps the major obstacle to a compromise. The votes of Farm Belt senators could be decisive in tipping the scales toward the Republican plan.

The Byrd and Chiles-Hollings plans would have cut deficits by more than the Republican plan over three years, even while dropping or modifying some of the proposed GOP cuts in domestic programs ranging from Amtrak to rural electrification.

They would have achieved this largely by tax increases. Byrd would have raised taxes $61 billion over three years, mainly by closing corporate tax "loopholes" and retaining cigarette levies that are due to expire this year. Chiles and Hollings would have raised taxes $72 billion over the same period, possibily by limiting tax breaks to corporations and wealthy individuals. Both plans ruled out increases in individual tax rates.

Social Security was the key difference between the two plans, an issue that sharply divided the Senate Democratic caucus and prevented development of a single party alternative.

Byrd proposed payment of full cost-of-living increases, while Chiles and Hollings advocated a six-month freeze in benefits.

The Senate earlier voted to drop a Republican proposal to cut the inflation payments by roughly half over three years.

Dole has indicated that he may include a one-year freeze in an eventual compromise.

In arguing against the Democratic plans, Republican leaders zeroed in on their tax-increase proposals but avoided closing the door to any revenue measures.

"If we can do it without raising taxes, it ought to be done," Dole said. "I happen to believe 'last-resort' on taxes is a good thing," said Budget Committee Chairman Pete V. Domenici (R-N.M.), warning against resorting to taxes unless all else fails.

Among Washington area senators, Mathias voted for both Democratic plans, John W. Warner (R-Va.) and Paul S. Trible (R-Va.) voted against both while Paul S. Sarbanes (D-Md.) voted for the Byrd plan and against the Chiles-Hollings plan