Malaysia announced yesterday that it intends to put the 76,000 Vietnamese refugees encamped there into boats and send them out to sea. Any more who try to reach its shores would be shot "on sight," a top official said.

Against the background of Thailand's current forcible ouster of 80,000 Cambodian refugees, Malaysia's announcement is the most dramatic move of any of the Southeast Asian countries being overwhelmed by the flood of hundreds of thousands of refugees from the Communist nations of Indochina.

Refugee specialists here expressed doubt that the Malaysians would shoot at the refugees' boats. Yet, they said, it is entirely possible that unless some international solution to the refugee problem is reached soon, Kuala Lumpur quickly would move to expel the Vietnamese.

According to Deputy Prime Minister Mahatir Mohamad, the eviction will start immediately. The Vietnamese would be put on boats and towed to interntional waters. Then, he said, Navy patrols would be authorized to "shoot on sight" any boats trying to reach Malaysian shores.

"We will be harsh with them. Being humance has not paid off to us at all. All we are getting is further inflow of Vietnamese refugees and we have every right to expel them," he said.

"If they try sinking their boats, they will not be rescued, they will drown," he said. "Their drowning will be because they sank their own boats."

Malaysia, Thailand and, to a lesser extent, Indonesia have borne the brunt of the refugee flood that has occurred since the Communist victories in Indochina in 1975.

The U.S. State Department said earlier this week that the number of persons who have successfully made it to one of the neighboring countries has soared recently, jumping from an average of 7,500 a month in 1978 to almost 55,000 in May. Most of the flights are made in flimsy boats, and it is estimated that as many as half of those attempting the voyage drown or starve before reaching land.

The United States and Britain have accused Hanoi of deliberately forcing out ethnic Chinese. An Official of the British colony in Hong Kong has accused Vietnam of extorting billions of dollars from those being forced out.

Ghazali Shafie, Malaysia's home minister, joined in the criticism of Vietnam yesterday.

"While telling us we are their freinds, the Vietnamese keep throwing rubbish into our gardens," he said.

Shafie said his country worked out a plan with Thailand and Indonesia for joint naval operations to head off refugees trying to make their way by boat to the three countries.

Foreign Minister Mochtar Kusumaatmadja of Indonesia said in Jakarta yesterday that his country also would close itself to more refugees. He said, however, that it would continue to provide haven for the 31,000 already there.

The Malaysian announcement was received with expressions of grave concern throughout the world.

In Washington, the State Department's press officer, Thomas Reston, said the United States has seeking official confirmation of the news reports from Kuala Lumpur.

"We would find any such action inconsistent with the previous record of the government in Malaysia in terms of humanitarian assistance," he said. "We, of course, would deplore any action which leads to the deaths of refugees."

Informed sources said that while Malaysia's proclaimed policy toward the refugees has usually been harsher than the policy actually put into practice there has been a noticeable increase recently in Malaysian attempts to fend off new arrivals.

One source summed up the dimensions of the problem facing Kuala Lumpur by noting that the number of the refugees reaching there had jumped from about 7,000 a month earlier this year to 15,000 in May.

This, the source said, is more than twice the number that the so-called resettlement countries, the United States, Australia, France and Canada, are prepared to take from Malaysia under the most ideal conditions.

Malaysia's action yesterday added impetus to proposals that an international conference be held on the Asian refugee problem. Japan and Britain have expressed interest in such a meeting and on Wednesday the Asian and Pacific subcommittee of the House Foreign Affairs Committee approved a motion calling for an emergency U.N. General Assembly session.

The informed sources, while not doubting Malaysia's determination to start expelling its Vietnamese if the flow of refugees is not staunched, speculated that the announcement was designed to make considerable psychological impact. As one source put it, the Malaysians probably want to make it clear to would-be refugees that the risks of making a journey have increased greatly.

The source said that there is also a message to Hanoi to do something about the refugee problem or face ever-increasing disapproval from the world community.

In an interview in Bangkok, Nguyen Co Thach, a high official of the Vietnamese Foreign Ministry, denied that his government was organizing the exodus of refugees. He said, however, that some corrupt low-level officials had demanded gold from refugees in exchange for permission to leave.

Thach said Vietnam had tried for two years keep its people from fleeing but was criticized for this by other countries. As a result, in 1977, "We decided to let them leave. Now people are criticizing us again.".

Commenting on Malaysia's announcement, Sir Murray MacLehose, the governor of Hong Kong, said it was "a desperate reaction to a desperate situation," and said the Malaysians were a "humane people" who would not speak in such a way unless they felt under an intolerable burden. CAPTION: Picture, Malaysia has erected a temporary camp for "boat people" on Bidong Island eight miles from the mainland.