Rejecting the fervent pleas of Secretary of State Cyrus R. Vance, the Senate cast a vote last night in favor of lifting economic sanctions against Zimbabwe-Rhodesia.

Because of parliamentary maneuvering, there was no up-or-down vote on the issue, but a 52-to-41 majority of senators voted to table or kill an alternative that would have permitted President Carter to keep the sanctions in effect until at least Dec. 1.

As a result, when the Senate votes, probably today, to pass the pending defense authorization bill, it will include language that would put an early end to the economic boycott against Zimbabwe-Rhodesia.

Administration officials took some encouragement from this result, since their side won 41 votes, enough to sustain a presidential veto later if both houses eventually pass legislation lifting the sanctions.

In testimony yesterday, Vance said a veto was "quite likely" if such legislation passes.

Last night's vote suggested a substantial change in Senate sentiment since the 74-to-19 vote last month in favor of lifting sanctions.

But is still was a blow to President Carter and Vance, who pleaded with bothe the House Foreign Affairs and Senate Foreign Relations committees yesterday to leave sanctions in place, at least for now.

Vance held out the prospect of a softer administration policy toward the new multiracial Salisbury government if Congress would cooperate with the president.

Vance indicated that the United States eventually might recognize the new Salisbury government, led by Bishop Abel Muzorewa, if it can guarantee full rights for all Zimbabwe-Rhodesians and some progress toward internal peace. The multiracial Muzorewa government was formed after April's elections in which, for the first time, blacks and whites were permitted to participate.

Vance vigorously upheld President Carter's conclusion that the election was not "free and fair," citing inequities of the Rhodesian constitution under which the election was held and the fact that important nationalist parties were not allowed to participate.

But he also praised the elections as a sign of progress and added, "It is too early to tell" whether formation of the multiracial government in Zimbabwe-Rhodesia would lead to fuller equality of the races there, truly representative government, "significantly greater opportunity for all citizens" and "new, serious efforts at accommodation with opposition parties, both external and internal. . . ."

A senior assistant to Vance later pointed to this passage as a sign of what the administration would want to see before considering recognition of the Muzorewa government. In his testimony, Vance refused to go beyond this somewhat veiled paragraph, specifically declining to list precise measures the Salisbury government would have to take to satisfy the United States.

"It would be a mistake to list rigid specifics at this point," Vance told the House, noting that the Carter administration wanted to see how the new British government decides to handle Zimbabwe-Rhodesia before acting.

But Vance agreed with Sen. Edmund S. Muskie (D-Maine) when Muskie described the administration's policy as "a new policy toward the internal political developments in Rhodesia."

However, Vance testified, lifting sanctions against Zimbabwe-Rhodesia at this time "would have a serious impact on our national interest" and "would preclude our ability to influence events in Rhodesia."

Specifically, Vance said, a move to lift sanctions now would undermine Britian's position as the legal sovereign power there, aggravate the civil war, perhaps draw Cubans or Soviets into it and undermine U.S. policy in Africa.

Finally, Vance added, lifting sanctions would "represent a retreat from the principles of racial justice which we have strived to achieve in our own country. To have one standard of racial justice at home and another abroad is to deny our common humanity."

Vance returned repeatedly in his testimony to the provisions in the new Zimbabwe-Rhodesian constitution that give the tiny white minority entrenched rights often amounting to a veto over the black majority.

All these arguments appeared to have some effect on the Senate, but not enough to reverse the overwhelming sentiment for an end to sanctions first expressed on the floor last month.

Sen. Frank Church (D-Idaho), chairman of the Foreign Relations Committee, told Vance at morning hearings that the Senate was likely to accept an amendment to the defense authorization bill calling for lifting the sanctions. The bill was on the floor yesterday for debate.

Church asked vance if the administration would accept a compromise extending sanctions for six months, but leaving it up to Congress at the end of that period to decide whether they should be lifted. Vance rejected this idea on behalf of Carter.

Then Sen. Jacob K. Javits (R-N.Y.) surprised both the administration and his colleagues by proposing an amendment to leave sanctions in place until Dec. 1, and longer if Carter certified that this was in the national interest.

Javits' proposal was offered as a substitute to the language the Armed Services Committee had approved in the defense authorization bill calling for an end the sanctions. But the Senate voted, 52 to 41, to kill the parliamentary vehicle that would have allowed for an up-or-down vote on javits' idea.

That leaves the language lifting sanctions in the defense bill. A final vote on that measure is expected today.

Sen. Jesse A. Helms (R-N.C.), a leading supporter of Muzorewa, revealed yesterday that the Carter administration would have an early opportunity to meet the new Zimbabwe-Rhodesian prime minister directly. Murorewa is to be visiting this country early next month, Helms told the Foreign-Relations Committee.