The United States has told Vietnam in face-to-face talks at least four times in the past eight weeks that movement toward normalization of relations is impossible under present circumstances, a senior State Department official said yesterday.

Richard Holbrooke, assistant secretary of state for East Asian and Pacific affairs, expressed surprise at Vietnamese statements in Hanoi suggesting that secret talks have been taking place aimed at establishing full diplomatic relations.

Responding to news accounts from Hanoi, Holbrooke and other officials made known in unusual detail the record of Washington-Hanoi contacts since they were resumed at U.S. initiative in June.

The contacts began with a June 6 message from Secretary of State Cyrus R. Vance suggesting that talks between the two nations, which had been dormant since late last year, be resumed. The Vance message, sent to Vietnam's ambassador to the United Nations, Ha Van Lau, through U.S. Ambassador Andrew Young, did not mention normalization of relations but did not rule out such discussion, officials said.

Washington's position on full diplomatic relations was made explicit, according to the U.S. account, in face-to-face meetings June 22, June 29 and July 12 at the United Nations and July 21 during a U.N.-sponsored conference on Indochina refugees at Geneva.

"We reiterated in each of the meetings that present circumstances make it impossible and undesirable to resume progress toward normalization of relations," Holbrooke said.

He added, "We also said that future movement is not impossible, that this remains the ultimate U.S. objective, and that Vietnam's actions will affect it."

In the U.S. view, movement toward diplomatic relations with Hanoi has been in limbo since last November, when it was linked to Washington's concern about the growing refugee flow, stepped-up military action by Vietnamese forces in Cambodia and the Asian consequences of the recent Vietnamese pact with the Soviet Union. This link was made in New York discussions between Deputy Assistant Secretary of State Robert B. Oakley and Tran Quang Co, an official of the Vietnamese Foreign Ministry.

The statements of recent days to members of Congress and accompanying U.S. correspondents by officials in Hanoi seem to reflect a Vietnamese desire to resurrect the drive toward normalization, although Washington officials are puzzled by the chosen methods of expression.

Several visiting U.S. lawmakers have come out of Hanoi calling for faster movement toward full relations. Rep. James H. Scheuer (D-N.Y.), for example, told a press conference in Bangkok yesterday that diplomatic relations shoudl be established "now, today or tonight" in order to "cut down the Soviet Union influence in Indochina" and make Vietnam "less neurotic" about the threat from China. Rep. Lester L. Wolff (D-N.Y.), who led the most recent congressional delegation to Hanoi, told the same news conference that progress has been made but that obstacles remain.

Holbrooke, after receiving reports on the two congressional missions, and that "if continued progress can be made on the refugee issue, the Cambodian issue will be the main sticking point" to full relations. He added that the Soviet role in the region is "also a factor that would influence our actions."

American officials said that, despite hints to the contrary in some public statements, the Vietnamese have not yet displayed real flexibility about their 200,000 troops in Cambodia.

The French news agency, Agence France-Press, which maintains a permanent correspondent in Hanoi, reported from there yesterday that Vietnam is determined to keep its troops on Cambodina soil "for as long as the Hanoi-backed Phnom Penh authorities want." Correspondent Jean Pierre Gallois quoted unnamed Vietnamese officials as also saying that the troops will remain "for as long as the Chinese threat bears directly or indirectly on Cambodia."

At the same time, the French correspondent said that some officials expressed a hope that a partial withdrawal of Vietnamese forces could be carried out at the beginning of next year.

Some Washington officials suggest Hanoi's words and deeds may reflect jockeying within ruling circles and perhaps even a division of opinion in the Vietnamese Politburo. Harassment of U.S. businessmen in early June by Vietnamese security officials at Noi Bay airport after cordial talks with Foreign Ministry officials is cited as evidence of such a division.

Another theory is that highly publicized Hanoi remarks about normalization of relations with Washington are intended to shake up the Chinese enemy and also might be a bargaining ploy to keep the Soviet Union on edge about Vietnam's intentions.

According to Washington officials, there were no direct talks with Hanoi on substantive matters between Oakley's discussions with Vietnamese officials last November and December and Vance's bid for a reopened dialogue in early June.

The Vance initiative reportedly was undertaken at the suggestion of Japan and some Southeast Asian nations concerned about the tide of refugees. Vance, preparing to meet Southeast Asian foreign ministers in Bali in early July, apparently hoped to be able to tell them that the U.S. had been discussing problems directly with Vietnam.

The June 6 message to Ha Van Lau, who had been Vance's negotiating partner during the Paris talks on the Vietnam war a decase ago, suggested a resumption of U.S.-Vietnamese dialogue through a preliminary exchange of views with Lau on major issues, according to the U.S. account.

As high officials prepared to leave Washington for the seven-nation Tokyo summit in late June; preparations were under way for major initiatives on the refugee question. At a White House breakfast June 22, President Carter instructed Vance to inform Vietnam directly of the U.S. plans.

The same day, according to the official account, Oakley flew to New York and asked for an immediate meeting with Vietnamese Ambassador Lau to avoid any misunderstanding of U.S. policy. Oakley is reported to have made it clear that the United States was not prepared to resume movement toward normalization in the current circumstances.

On June 29, in the immediate aftermath of Carter's announcement in Tokyo that the United States would docuble the number of Indochina refugees admitted here, Deputy U.N. Ambassador Donald F. McHenry called on the Vietnamese in New York with another message. Carter and Vance wanted to make it clear that the immigration change in no way condoned Vietnam's treatment of its citizens but was purely a humanitarian gesture, according to the U.S. account.

In early July, Vietnam responded to the June 6 message, saying it was prepared to resume the dialogue as suggested. As a result, Oakley saw Lau in New York on July 12. According to the U.S. account, Oakley said the United States was not prepared to move toward normalization and discussed U.S. concern about the refugees the Cambodian occupation and the Soviet presence in the area. He also discussed Washington's continuing concern about unaccounted for American prisoners of war, officials said.

The last discussion took place in Geneva July 21 between Holbrooke and other U.S. officials and Vietnamese Vice Foreign Minister Phan Hien during the international conference on refugees. Once again, the U.S. position on normalization of relations was made clear, according to Washington's account.

At the same meeting, Hien is reported to have confirmed Vietnam's willingness to accept U.S. consular officials as part of a U.N. delegation to Ho Chi Minh City (Saigon) to process visas of people wishing to come to the United States. That plan has now been cast in doubt by Vietnamese statements of teh past few days, fueling additional uncertainty and the here. CAPTION: Picture 1, RICHARD HOLBROOKE; Picture 2, REP. LESTER L. WOLFF