The Republican-controlled Senate, brushing aside last-minute appeals from President Reagan, reversed itself yesterday and refused by one vote to approve a constitutional amendment to require a balanced budget.

The vote was 66 to 34, one short of the two-thirds necessary to pass an amendment to the Constitution.

The Senate approved a similar amendment in 1982 by a vote of 69 to 31, but changes in membership and especially vote switches by Republicans eroded the base of support for the amendment.

Both sides in the fight cited the Gramm-Rudman-Hollings law, passed last year to force a balanced budget within five years, as a factor in the amendment's defeat. "People said, 'We have Gramm-Rudman; we don't need it the amendment anymore,' " said Sen. Paul Simon (D-Ill.), a key supporter of the amendment. Gramm-Rudman-Hollings had "a proper impact," said Sen. Daniel J. Evans (R-Wash.), leader of the fight against the amendment.

Reagan, who has listed the amendment among his top legislative priorities, made a number of phone calls to wavering senators over the last few days, but there was disagreement among the amendment's supporters about the intensity and effectiveness of his effort.

"He did the best he could," said Senate Judiciary Committee Chairman Strom Thurmond (R-S.C.). "I disagree," said Sen. Dennis DeConcini (D-Ariz.).

The proposed amendment would have required a balanced budget each year, except in wartime, unless three-fifths of each house approved a deficit. To win the votes of waverers, especially Democrats, an earlier provision making it more difficult to raise taxes was dropped and the president was required to submit a balanced budget.

But the concessions were not enough. Opponents picked up one vote (Democrat Frank R. Lautenberg of New Jersey) from membership changes since 1982. In addition, Sens. Robert T. Stafford (R-Vt.), Mark O. Hatfield (R-Ore.), Quentin N. Burdick (D-N.D.) and Robert C. Byrd (D-W.Va.), the minority leader, switched from supporting the amendment to opposing it, more than offsetting switches in the other direction from Sens. Wendell H. Ford (D-Ky.) and Claiborne Pell (D-R.I.).

Thurmond and other backers said they will continue efforts to pass the amendment, but Sen. Orrin G. Hatch (R-Utah), a leading backer, said membership changes will be required before it can be passed. Hatch also said some senators, especially those up for reelection this year, were upset at members of the Republican leadership -- presumably Sens. John H. Chafee (R.I.) and John Heinz (Pa.) -- who voted against the amendment.

In debate, proponents contended that an amendment is needed to end what Sen. Howell Heflin (D-Ala.) called the "helter-skelter fiscal irresponsibility" of the White House and Congress over the past 25 years, during which the budget has been balanced only once.

Noting that Gramm-Rudman-Hollings could be repealed or modified at any time, they contended that an amendment is needed to make budget-balancing mandates permanent. But critics said the law should be given a chance to work before revising the Constitution to achieve the same end.

"We're wasting precious time playing Trivial Constitutional Pursuit" when Congress should be focusing on imminent deadlines for compliance with Gramm-Rudman-Hollings, said Senate Democratic Whip Alan Cranston (D-Calif.).

"The Constitution is the embodiment of the most cherished principles of this nation," said Sen. Slade Gorton (R-Wash.). "The Constitution is no place for congressional graffiti."

Looking across the chamber at Hatch, Sen. Howard M. Metzenbaum (D-Ohio) said the amendment was being spearheaded by some of the people who voted in the Budget Committee against a fiscal 1987 budget that would have met initial Gramm-Rudman-Hollings deficit targets. Hatch, a member of the committee, voted against the proposal, which included tax increases and other provisions that he opposed.

In yesterday's voting, 43 Republicans and 23 Democrats voted for the amendment, while 24 Democrats and 10 Republicans voted against it. Virginia senators voted for the amendment; Maryland senators voted against.