With the United States leading the way, the Organization of American States has voted down an invitation to hold its general assembly this year in Uruguay because of that country's human rights record.

In addition to that unprecedented rebuke in the OAS, President Carter is expected to reinforce the U.S. position on the rights issue in a letter to Uruguayan President Aparicio Mendez. The Uruguayan had written Carter asking support for holding the general assembly in Montevideo.

Uruguay incurred the opposition of a majority of the 25 OAS member nations because it has refused to cooperate with the organization's human rights commission, a body that the Carter administration has supported vigorously as the first line of defense against rights violations in the hemisphere.

Until 1973, when the military effectively took power in Uruguay, the small South American nation was a leader in efforts to extend protection of human rights through the OAS. The military takeover followed insurrection by leftist Tupamaro guerrillas.

Since 1973, Uruguay has been cited increasingly for arbitrary arrests and abuse of prisoners. Amnesty International has stated that Uruguay has more political prisoners per capita than any other Latin American nation.

U.S. Ambassador to the OAS Gale McGee, in voting against acceptance of the Uruguayan invitation Tuesday, said the "one basic concern" of his government "was the integrity of a fundamental institution in the OAS" - the Inter-American Commission on Human Rights.

"We feel that this was an especially awkward moment to be resolving the very generous gesture to host our general assmebly juxtaposed with the host government's stand; on human rights," he added.

The vote came in a closed session of the OAS Councils' general committee. The U.S. delegation released McGee's statement. President Carter's letter, according to U.S. officials, will back up the efforts in the OAS to have Uruguay invite the rights commission to make an on-scene investigation of conditions there.

The seven-member rights commission can act only on invitation from the government under scrutiny.

Voting with the United States to reject Montevideo as the site for the general assembly were Venezuela and Panama. Venezuela earlier broke relations with Uruguay over an alleged human rights violation there.

Ten other nations abstained - Mexico, Costa Rica, Columbia, Honduras, Dominican Republic, Jamaica, Trinidad-Tobago, Barbados, Grenada and Surinam.

Twelve nations voted to accept the Uruguayan offer, one vote short of the majority needed.

In Montevideo, a spokesman said the government regretted that OAS members had not chosen to come see for themselves the conditions in Uruguay. The OAS was criticized two years ago for holding its assembly in Chile, which presented the action as an acceptance of Chile's Rights policies.