Supporters of the balanced-budget constitutional amendment had a falling out yesterday and emerged from an acrimonious Senate debate having altered the measure in a way that most agreed will make final passage more difficult, if not impossible.

The split in conservative ranks produced new language that would write into the Constitution a requirement that both houses of Congress muster a three-fifths vote to raise the national debt ceiling.

The effect was to add a new and volatile element in a legislative struggle that already is extremely close, and Republican leaders admitted their cause had suffered.

"It was a major setback," said the co-manager, Sen. Orrin G. Hatch (R-Utah). "I still think the Senate will pass it. But conservatives who have fought for this for 25 years and who then voted for this new language today may have done themselves in."

Democratic opponents were delighted. It makes today's final vote "unpredictable," said Sen. Alan Cranston (D-Calif.), who earlier had conceded that the amendment would probably pass.

"This makes it less likely," he said. It also will complicate approval in the House.

The original proposed amendment, strongly backed by the Reagan administration, would require Congress to vote balanced budgets unless the country were at war or unless both houses, by three-fifths votes, permitted deficits. Leading supporters such as Hatch had solidly opposed all but one minor change in its language, principally because any major revision would cause trouble in winning passage in the House. They had successfully beaten off a number of changes designed by opponents who hoped to cripple the amendment. But yesterday, Sen. William L. Armstrong (R-Colo.), a fervent supporter, produced the first substantial change with an amendment requiring the three-fifths vote to raise the debt ceiling, something Congress is called on to do almost annually.

That--claimed other conservatives and liberals alike--would make the constitutional amendment unwieldy because raising debt ceilings is difficult now, when only a majority vote is required.

Armstrong contended his change would nail a certain spending limit into the amendment, actually strengthening its bias against big government spending. Hatch countered: "It will strengthen this constitutional amendment to death."

It would, if anything, spur Congress to raise taxes in a crisis because that would be easier than mustering a three-fifths vote to raise debt ceilings, he added.

Hatch's main concern, however, was that it would cause new trouble in the House, where supporters are trying by a discharge petition to rescue the constitutional amendment from the hostile House Judiciary Committee. That petition calls for a closed rule prohibiting amendments, so in order to make Senate and House versions jibe, a conference would be necessary. The House conference members would be controlled by Democrats opposed to the constitutional change.

When Armstrong's amendment came up for a mid-afternoon vote, Democratic opponents of the constitutional change saw their chance to muddy the waters and voted for it in droves, pushing it to a 51-to-45 victory.

A grim-faced Majority Leader Howard H. Baker Jr. (R-Tenn.), claiming the Armstrong change would "impair" chances of final passage, worked furiously to switch votes but fell short and then lost a motion to reconsider, 56 to 40.

When the smoke cleared, jubilant opponents claimed they were near success in blocking the amendment. Cranston said his latest nose count showed 65 members firmly supporting it or leaning toward it, two votes short of the two-thirds necessary to propose a constitutional amendment.

He said 31 are firmly opposed or leaning that way, leaving the final decision up to four senators. His count was taken before the supporters had suffered their debacle on the floor.

In the House, 184 members had signed the discharge petition by mid-afternoon and supporters conceded difficulty in getting the additional 34 names needed to take the amendment away from the Judiciary Committee.

The committee chairman, Rep. Peter W. Rodino Jr. (D-N.J.), a strong opponent, yesterday opened three days of hearings on the amendment but had no plans to report it out unless forced to do so by a near success in the discharge petition drive.

One of his first witnesses was Treasury Secretary Donald T. Regan, who urged him to give the House a chance to vote on the proposed amendment.