The Senate refused yesterday to endorse President Carter's plan to withdraw U.S. ground troops from South Korea in four to five years.

But it balked at a proposal to tie his hands in negotiations with Cuba- an amendment that would have put the Senate on record as opposing any diplomatic recognition unless Cuba compensates Americans for expropriated property and removes all its troops from Africa.

Instead of endorsing Carter's proposed Korea troop pullout, the Senate by a 79 to 15 vote accepted watered-down language worked out by Majority Leader Robert C. Byrd (D-W. Va) merely stating that U.S. policy toward Korea "should continue to be arrived at by the joint decision of the President and Congress."

The Byrd amendment went on to say that if a withdrawal of U.S. ground forces from South Korea is eventually agreed on, it should occur in stages, with regard for the interests of Japan and other nations in the area and through "regular consultation with Congress."

This Senate language is not binding on Carter, but the debate made clear he will face substantial opposition if he attempts to go forward with the troop pullout.

Byrd offered his language after it became clear that a provision of the $1.6 billion State Department authorization bill, which endorsed the President's five-year phaseout plan, would be handily beaten on the floor by a coalition of Republicans, headed by Minority Leader Howard H. Baker Jr. (Tenn.) and Robert P. Griffin (Mich.), and Southern Democrats like James B. Allen (Ala.).

Sen. George McGovern(D-S.D.), floor manager of the bill and sponsor of the original phaseout endorsement, said adoption of the Byrd compromise probably warded off a "worse setback" - a possible floor amendment putting the Senate on record as directly opposing Carter's plan.

The Byrd language, McGovern said, "doesn't say we're for the phaseout or against it. They did refuse to endorse [the phaseout] but they didn't repudiate it" directly, either. Baker and other opponents concurred in this interpretation, saying the final language left the issue for later decision.

The vote on Cuba policy killed an amendment by Bob Dole (R-Kan.) stating the sense of the Senate that, before the Carter administration extends recognition to Cuba and lifts the trade embargo, Cuba provide $1.8 billion in compensation for expropriated American property, withdraw all its troops from Africa, release all political prisoners and renew the 1973 hijacking agreement.

The State Department complained that such specific language would tie the Presidents hands in negotiations. Byrd drafted a much softer compromise amendment declaring that, in negotiations, the rights of citizens "whose persons or property are the subject of negotiations be protected" and that Cuban policies on use of its troops beyond its borders "must be taken into account."

In an administration victory, the Senate adopted the Byrd amendment, 54 to 37, in place of Dole's language. Then it reendorsed the Byrd language, 91 to 1.

In another important action on the bill, McGovern stripped out by unanimous consent a provision - which he himself had inserted in committee - allowing sales to Cuba of U.S. food and medicines but nothing else.

Dick Stone (D-Fla.) had prepared an amendment to strike the provision and said yesterday, "I had 65 votes". Stone said negotiations with Havana are still going on, Cuba has an expanding military role in Africa and has confiscated more U.S. property than any other nation in history, and it was too soon to make even limited concession under those conditions.

Baker, Sam Nunn (D-Ga.), Griffin, Barry Goldwater (R-Ariz.) and others argued that a Korean phaseout plan shouldn't be endorsed without considerable further study of the consequences - particularly whether it would weaken Japanese confidence in U.S. intentions to protect Asian nations against Communist aggression, and whether it would weaken South Korea and invite a North Korean attack.

McGovern, in seeking endorsement of the phaseout, said the President was talking of a "very careful, very orderly withdrawal over a period of four to five years" of the bulk of the 40,000 U.S. forces in South Korea, leaving some 7,000 Air Force personnel to provide immediate air support in case of attack from the North. He said the South Korean army and economy both are very powerful and would be given the latest U.S. weaponry before the troop phaseout - thus signalling Moscow, China and North Korea of U.S. intent to ward off any possible aggression. Further, he said, a 1954 treaty commits the United States to aid South Korea in combatting aggression.

Baker said "I was frankly appalled" when the Foreign Relations Committee adopted the phaseout endorsement, which he said could send "a signal to the world, particularly to the Chinese and Russians, that we don't want to send" - that we wouldn't fight to fend South Korea, thus inviting aggression.

"Leave this question undecided," he pleaded.