The State Department in an unusual public accusation, charged yesterday that Vietnam is forcing large numbers of its citizens to pay for the privilege of fleeing the country as refugees on the high seas.

The statement by spokeswoman Jill Schuker, which originated in the White House, had a notably sharper anti-Vietnam edge than previous declarations, which concentrated on U.S. appeals for international help for stranded "boat people." It was also the first official accusation that Vietnam is profiting from the exodus.

"We deplore strongly the human rights situation in Vietnam, which is forcing tens of thousands of individuals to flee, despite the great dangers of the voyage and the uncertain futures which await them once they do succeed in reaching a place of asylum," Schuker volunteered to reporters at the start of a State Department briefing.

"The practice of forcing people to buy their way out is particularly reprehensible," she added.

The spokeswoman was unable to give any details of the charge that refugees are being forced to pay, but another official summoned to the briefing room declared that "it is obvious that this practice is known at the highest levels" of the Vietnamese government.

Henry B. Cushing, deputy director of the State Department's Office of Refugee and Migration Affairs, said the United States has been aware of the buy-out practice for about six months but has been unable to determine how much of the money goes to Vietnam's national coffers and how much is retained by officials acting outside of prescribed policy. Cushing said that the sum of about $2,000 in gold per person is most often mentioned by well-to-do persons who have paid to get out.

A non-American source with extensive experience in Vietnam confirmed the practice, and estimated that 20 to 30 percent of the money being paid probably stuck to private hands, with the rest going to government coffers as very high fees for paper work or to government officials as graft.

According to Cushing, the latest wave of Vietnamese departures is the result of increasingly harsh measures taken against private enterprisers and other wealth persons, a large proportion of them of Chinese orgin. Vietnam, which is engaged in a growing feud with China, tends to view these people as "ethnically suspect" and a "burden on society," Cushing said.

The practical choice for these people, he added, is going as laborers to "a new economic zone" in the countryside or paying heavily for the chance to get out. Despite payments, he said, the United States believes that these people are legitimate refugees with a rightful claim on the international community for assistance.

There has been longstanding concern about the growing number of people within the Carter administration fleeing Vietnam by boat. Estimates have risen from 2,000 people in March, to 6,000 in July, to 12,000 in October. Officials said yesterday that 10,000 people are believe to have fled by boat in the first two weeks of this month alone.

Malaysia and other countries in the area are increasingly reluctant to accept the rising numbers of refugees, even on a temporary basis. The problem has come to a head in recent days because of the plight of 2,500 persons stranded aboard the freighter Hai Hong off the coast of Malaysia. The Malaysian government has refused to accept this new shipload, partly on grounds that their departure was arranged in Vietnam.

The United States has established a continuing quota of 25,000 Indochinese refugees per year, about half of them to be "boat people" but these places are reported to be all taken for this year.

Republican National Chairman Bill Brock, in a statement issued yesterday, asked the Carter administration to "take whatever action is necessary" to provide help and asylum in the United States, if necessary, for the people aboard the Hai Hong. "There is no excuse whatsoever for any further delay." Brock said.

The strong administration criticism of Vietnam comes on the heels of a Soviet-Vietnamese peace and friendship treaty signed in Moscow, and at a time of division within the government over how fast the United States should move toward normalization fo relations with Hanoi.

The assistant secretary of state for east Asian and Pacific affairs, Richard Holbrooke, who has held several recent talks with Vietnamese officials about normalization, was making a speech on Southeast Asia in Boston when yesterday's statement was issued.

In the address, Holbrooke noted the Soviet-Vietnamese treaty and declared that the United States shares the concern of Asians that it "not become part of an effort to change the strategic and political balance" in the region.