A rebellious Senate yesterday rejected President Reagan's request for large new increases in defense spending as it continued to dismantle his deficit-reduction compromise with GOP leaders.

The defense-spending cutback was approved by voice vote. Earlier, on a key procedural motion, the Senate had voted 51 to 48 against Reagan's request.

Defying the White House and its Republican leadership for the second consecutive day, the Senate voted to allow defense spending to increase only for inflation next year.

Reagan originally requested an after-inflation defense-spending increase of nearly 6 percent for next year. He later compromised on 3 percent above inflation in his deficit-reduction accord with Senate GOP leaders.

The Senate also agreed to a far smaller increase than Reagan wanted for the subsequent two years. It approved 3 percent after-inflation increase for fiscal 1987 and 1988; Reagan sought more than 8 percent.

Even with the nearly $18 billion cut in defense outlays over the three-year period, military spending would continue to rise, mostly because of large increases approved during Reagan's first term.

But yesterday's vote marked an apparent turning point in support for major defense-spending increases by the Republican-controlled Senate, which generally has given Reagan most of what he requested for defense.

Appropriations Committee Chairman Mark O. Hatfield (R-Ore.), who with Sen. Charles E. Grassley (R-Iowa) proposed the cutback, called the vote "very historic." Sen. Edward M. Kennedy (D-Mass.) called it a "watershed" vote.

As in Wednesday's vote to scuttle a proposed cutback in cost-of-living increases for Social Security benefits, Republican votes were decisive in rejecting the president's position.

Twelve Republicans, including conservatives, crossed party lines to vote with a majority of Democrats for the defense cutback. Only eight Democrats, mostly southern conservatives, voted against the cutback.

Senate GOP leaders, who had expressed fear that a major defense-spending cut would jeopardize Reagan's support for the whole package, indicated that they would try to convince the president that his position could not be sustained in Congress.

"Both of the White House positions on defense and Social Security have failed, and I hope the president understands we made the best effort," Majority Leader Robert J. Dole (R-Kan.) said.

"What we're doing is emphasizing on a daily basis to the White House we don't have the votes to hold the package together," Sen. William S. Cohen (R-Maine) said.

The White House, Defense Secretary Caspar W. Weinberger and lobbyists for defense contractors made an all-out effort to persuade wavering senators of both parties to stick with Reagan's request.

White House chief of staff Donald T. Regan made last-minute phone calls from Bonn, where Reagan is attending the economic summit. The president had been expected to make calls, but it could not be determined whether any senators received them before the vote.

After the preliminary roll call, Dole and White House lobbyists failed to persuade several Republican defectors to switch votes. "It's now 5 after midnight in Bonn, and it's a little late for phone calls . . . ," Dole said, proposing to resolve the issue by a voice vote rather than risking further erosion on another roll call.

Acknowledging the Social Security and defense votes as setbacks, Dole expressed increasing optimism about prospects for a bipartisan compromise, cheered in part by disarray among Democrats about strategy and substance in dealing with alternative proposals, including Medicare and Medicaid modifications to come up today.

Dole vowed again yesterday to return after the voting on individual items in the deficit-reduction plan with a further compromise, probably along bipartisan lines, that would accomplish the original goal.

The plan, as negotiated by Republican leaders with the White House, would cut current budget deficits of more than $200 billion by half over the next three years, largely by big new cuts in domestic spending.

Much of the argument against the plan has been that defense spending did not bear its fair share of sacrifice, an argument frequently expressed during two weeks of debate on the plan.

As the Senate met into the night, it approved, 79 to 17, a resolution endorsing enactment of minimum taxes on corporations and individuals to help reduce tax rates as part of tax reform. The move was seen as an effort by beleaguered Republican leaders to counter mounting pressure for a minimum corporate tax proposed by Democrats to help reduce deficits.

It also voted, 80 to 18, to retain full cost-of-living increases for civil service and military pensions, as long as the Senate refuses to approve cutbacks in inflation adjustments for Social Security.

This was done to give equal treatment to all federal pension plans.

In scuttling the proposed cutbacks in Social Security and other retirement benefits, the Senate lost $34 billion from its three-year deficit reduction target, only partially offset by the $17.7 billion in additional defense savings.

During yesterday's defense debate, there was little talk of specific cuts in the Pentagon budget, prompting complaints from Armed Services Committee Chairman Barry Goldwater (R-Ariz.) and Budget Committee Chairman Pete V. Domenici (R-N.M.).

Instead, the focus was mostly on curtailing defense spending in general in order to strengthen the economy through deficit reductions and clamp down on Pentagon spending excesses.

"We've been here two days on the defense proposal , and everyone says we're spending too much on toilet seats," Domenici said, referring to high-priced equipment purchased for airplanes. "Why don't we talk about what our free nation needs to maintain the peace?"

Goldwater said he repeatedly has asked colleagues to suggest what might be cut from defense spending and has received no response. "When we begin looking at places we might cut the budget, let's think of national security ," he said. "Let's think of our country, let's not think of getting reelected . . . . "

Sen. Howard M. Metzenbaum (D-Ohio) promptly took the floor to suggest that defense contractors' tax bills would be one place to look for savings.

Goldwater's reference to political considerations in the defense vote was an ironic commentary on how back-home pressures on lawmakers have shifted over the last five years on defense spending as they are whipsawed by mounting deficits, an uneven economic recovery and Pentagon spending excesses.

While pressures were strong to increase defense spending when Reagan took office in 1981, senators now say voters are demanding cutbacks, with the savings to be used to reduce the deficit or increase domestic spending. This talk, they say, is especially strong among lawmakers from economically distressed areas, such as the Farm Belt.

Among those voting yesterday to cut back the military buildup yesterday were conservative Republicans James Abdnor (S.D.) and Don Nickles (Okla.), both of whom are up for reelection in potentially difficult races.