The Republican-controlled Senate Budget Committee voted 20 to 0 yesterday to reject President Reagan's budget, but within hours the White House and Senate Republican leaders reached a compromise that the committee approved.

The vote to reject Reagan's budget was the most powerful repudiation so far of his high-deficit tax and spending program for fiscal 1983 and beyond.

But the day ended with what had eluded Congress and the White House in three months of budget wrangling: a possible starting point for working out a final compromise between Reagan and Congress.

The plan, negotiated at the Capitol shortly after the committee vote rejecting Reagan's budget, was presented to the Budget Committee last night by Chairman Pete V. Domenici (R-N.M.), who said the president "will endorse and support this proposal."

But committee Democrats immediately criticized a provision that anticipates $40 billion in unspecified savings from Social Security over the next three years. They indicated that the plan lacks the bipartisan support that congressional leaders were trying to arrange in earlier negotiations.

The proposal was finally approved by the committee on a party-line vote of 11 to 9, although the committee is to return today to consider amendments on individual items.

The plan, which is to go before the full Senate later this month, calls for $77 billion in deficit reductions for fiscal 1983, including $20 billion in tax increases, leaving a projected deficit of $105 billion for next year. With $414 billion in tax increases and spending cuts over three years, it would reduce the deficit to $42 billion by 1985.

By contrast, congressional budget experts say Reagan's budget would produce a deficit of $132 billion next year, even if Congress approved all of the president's controversial proposed spending cuts. Congress has indicated strongly that it would not.

Key elements of the plan include a one-year pay freeze for military and civilian government employes, although readiness-related pay increases for the military would be allowed. For civilian government employes, a pay increase of 4 percent would be allowed in fiscal 1984 and 1985.

Reagan's big defense buildup would be cut by $5 billion next year, for a total of $22 billion over three years, less than Domenici had proposed. Domestic appropriations would be frozen at 1982 levels through 1985, and pension programs, other than Social Security, and veterans' benefits would be frozen for a year.

Other benefit entitlement programs would be cut by $33 billion over three years, starting with $7 billion next year.

The plan generally parallels one offered earlier by Domenici, with some major differences, including a substantial reduction in the amount of tax increases that would be required by 1985 to bring down deficits and help ease pressure on interest rates. There also would be more domestic spending cuts than Domenici proposed, but, even with them, the deficit would be higher than was proposed either by Domenici or by Sen. Ernest F. Hollings (D-S.C.) in a budget alternative that he proposed earlier in the day.

As worked out by White House chief of staff James A. Baker III and Office of Management and Budget Director David A. Stockman in a meeting with Domenici and Senate Majority Leader Howard H. Baker Jr. (R-Tenn.), the plan would raise taxes by $95 billion through 1985, including a tax increase of $20 billion next year, small enough to avoid tampering with Reagan's 1983 tax cut.

This is $30 billion less than the three-year, $125 billion tax increase proposal in Domenici's plan, and $15 billion less than the $110 billion figure Reagan embraced in unsuccessful negotiations with House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) last week. But it is roughly double the amount of tax increases that Reagan proposed in his original budget last February. It was this budget that the committee unceremoniously rejected earlier yesterday in a vote that had been expected but was nonetheless astounding in its unanimity.

After several days of blocking a Democratic move to force a vote on Reagan's budget, Domenici suddenly reversed himself early in the afternoon, insisting on casting the first vote.

As a hush settled over the normally restive crowd in the room, Domenici cast a firm "no." Even among the president's strongest supporters on the committee, there was not a single "yes," except an accidental one from Sen. Joseph R. Biden Jr. (D-Del.) that was hastily corrected amid laughter from committee members.

The White House immediately sought to play down the vote's significance, saying that it was expected and that meetings to on a compromise had been under way even before the vote.

"I would not count this as a total rejection in any manner," White House deputy press secretary Larry Speakes said. "With some modification, you will see a large portion of the president's budget enacted into law when Congress faces the same choices the president faced."

Meanwhile, Reagan was meeting with his principal budget advisers to lay out guidelines for budget changes, including taxes and spending, although Speakes continued to insist that "there will be no revised Reagan budget."

The final product that went before the Budget Committee represented concessions from both sides, however, and even more changes are considered likely to help the budget through the Senate and, especially, the Democratic-controlled House.

There, O'Neill and others have balked at Social Security cuts, and have insisted that a greater share of the overall deficit reductions come from taxes rather than spending.

On Social Security, the negotiators at yesterday's meeting dropped Domenici's proposal for elimination of the cost-of-living increase scheduled for July, which was a red flag for Democrats.

The savings would be based on recommendations from a special commission now studying steps to guarantee solvency of the Social Security system.

But, in substantially increasing Domenici's proposal for $14 billion in unidentified savings from Social Security in 1984 and 1985 and requiring that $6 billion of the $40 billion total be made next year, they appeared to have made Social Security a major budget issue, which some Republicans had been hoping to avoid.

During last night's session, after Domenici laid out the plan, Democrats repeatedly asked him for assurances that it had Reagan's approval. Near the end of the meeting Domenici left the room and returned to announce that he had spoken with the president.

He quoted Reagan as saying, "I am for it. I will do everything I can to see that is is passed and becomes law."

Democrats failed in two separate efforts to eliminate the 1983 tax cut and to strike out references to Social Security. Both lost in voice votes.

Hollings, whose plan for bigger tax increases failed by a 15-to-7 vote, said he doubted that some of the proposed spending cuts would pass. But he commended Domenici for working out the deal.

"If I were the chairman and I had an intransigent president and a tricky director of OMB and was nonetheless able to negotiate the package , I say that's a pretty good act," Hollings said.