Eighty senators joined yesterday in a letter urging Secretary of State Cyrus R. Vance to try to put the issue of Cambodia's treatment of its citizens on the agenda of the U.N. Security Council.

The letter was initiated by Sen. George McGoven (D-SS.D.), who said on Aug. 21 that the reported mass murders of Cambodians by the communist government there justified consideration of an international military force "to knock this regime out of power."

McGovern also revealed yesterday that he had received a letter from the Cambodian regime responding to his August remarks. The letter, mailed from Peking, denounced him for his "wanton democratic Kampuchea," as the Cambodian regime now calls the country.

McGovern's comments were widely reported by some as a reversal of the senator's traditional position in opposition to the use of force in Indochina. McGovern was an early opponent of the Vietnam war.

McGovern said at the time that there was no comprarison between the two situations, and he had long been deeply concerned by reports of genocide in Cambodia.

The letter signed by 80 senators - including representatives of virtually all shades of pinion in the Senate - suggested that the Carter administration has done too little to protest internal violence in Cambodia.

The letter said the administration's decision to try to bring the Cambodian issue before the U.N. Human Rights Commission "seems to be a rather low-key approach in light of the enormity of the crimes being committed in Cambodia."

The letter noted "the practical and political obstacles to action by the Security Council," where Cambodia's only significant ally, China, has a veto.

"Nevertheless, the senators continued, "an initiative of this kind would still be useful as s demonstration of our view that Cambodia has become a uniquely horrible situation warraning a uniquely vigorous response from the world community.

According to unconfirmed reports from refugees and other sources, the Cambodian regime has caused or permitted the murders of thousands - perhaps tens or hundreds of thousands - of the country's citizens.

The Cambodian's letter to McGoven dated Aug. 26, was written in English and stamped and signed by "Department of Press and Information of the Ministry of Foreign Affairs of Democratic Kampuchea." McGovern's office released the text.

"Your hysteric and over-excited tune (sic) of your shameless and wanton attacks against Democratic Kampuchea . . . have eloquently shown your resentment and hatred against our people, our revolutionary army and our government," the letter said, referring to McGovern as "a representative of U.S. imperialism."

Returning McGovern's charge of possible genocide in Cambodia, the letter said it was the United States that had committed genocide in that country during the Indochina war. The letter included this indirect acknowledgment that the new regime had dealt harshly with some Cambodians:

"You have been aware, and for a very good reason, that . . . Kampuchea's people have smashed into pieces all the activities of spying and subversion of the imperialists, the expansionists, annexationists and their followers as well as their attempts at coups de'etat . . ."