The United States is ready for new refugee-related contacts with Vietnam but is still far away from resuming negotiations on full diplomatic ties, Secretary of State Cyrus R. Vance told a House subcommittee yesterday.

Vance made his statements in the face of appeals from Chairmans Elizabeth Holtzman (D-N.Y.) and several members of the House Judiciary subcommittee on immigration and refugees for more intensive and more imaginative diplomacy toward Vietnam. Until there is change at the source of the problem in Hanoi, they said, U.S. and other efforts to deal with the refugee flow can be only palliatives.

"Without a dialogue with the pivotal player in the refugee drama, solutions will be all the more difficult to find, misunderstandings will be all the more likely to surface," Holtzman said.

As a sample of the problems, Holtzman cited Vietnam's failure to respond to the U.S. offer almost two weeks ago to send consular officers on temporary duty to Ho Chi Minh City. The U.S. officials, who would be the first on such a mission since the communist takeover in 1975, would process visas for Vietnamese eligible to come to this country to rejoin their families.

Vance expressed puzzlement tinged with frustration at the lack of a response through the established channel of the United Nations high commissioner for refugees. If no answer is received by the end of this week through the United Nations, Vance indicated, he will take up the problem directly with the Vietnamese.

Addressing the broader issue of diplomatic relations, Vance recalled that an early Carter administration drive was frustrated by Vietnam's "unacceptable demand" for an aid program amounting to reparations. When Vietnam dropped this precondition, Vance said, the growing tide of refugees and Hanoi's invasion and occupation of Cambodia raised "very substantial obstacles" to normalization of diplomatic relations.

He expressed no optimism about quick solution of either the refugee problem or the politically tangled Cambodian question.

Vance added, however, "I hope eventually progress can be made in resolving these issues which would permit us to begin talks with respect to normalization. These are very serious problems."

The shelving of the normalization discussions with Hanoi last year coincided with the speedup in normalization of U.S. relations with the People's Republic of China, Hanoi's bitter foe. The China connection is believed by many to have been a factor in Washington's swing away from full relations with Vietnam. In a reference to this, Holtzman asked Vance yesterday to consider Vietnam dealings in the light of the refugee problems and not as "playing the China card" or "global politics."

Ambassador Dick Clark, the States Department's coordinator on refugee affairs, responded to Vietnamese charges that the United States is enticing people to flee on the high seas by announcing a new policy of search and rescue on the part of Seventh Fleet planes and ships.

"Whatever we do to improve the situation could be interpreted that way," Clark conceded to the lawmakers. "But the alternative is so much worse, to let people die at sea."

The State Department said yesterday that preliminary reports "indicate a marked decline" in the number of Vietnamese refugees arriving in Southeast Asia. While about 58,000 arrived in June, about 16,000 arrived in the first half of July and arrival rates in several ports since then "seemed to have dropped even further," a department statement said.