Increasingly bad vibes between the United States and China, dramatically illustrated by the case of defecting tennis star Hu Na, threaten ever-more-serious damage to an international relationship of global importance.

The Chinese retaliatory actions, officially described by the State Department as "an overreaction and inappropriate," are understandable only in the context of an overall relationship that has lost its forward momentum and is in danger of slipping backward at an accelerating pace.

There were signs in both capitals yesterday that key officials understand the danger.

Peking's actions seemed carefully limited to the suspension of exchanges which are mostly symbolic and which do not affect the central business of the two countries.

Washington's reaction was carefully limited as well. The tone of official comment was that of sorrow, not of anger. In other conversations, officials sought to emphasize the major economic, political and strategic interests that the two nations share as "the glue which will hold this relationship together."

At the White House late yesterday, in a ceremony that had been scheduled well before the contentious words and controversial events of this week, President Reagan received the credentials of the new Chinese ambassador, Zhang Wanjin.

The official exchange of remarks, which traditionally are not uttered but are issued for the record, gave no sign of the asylum dispute or the Chinese retaliatory action. Both Zhang and Reagan, according to this written record, expressed their hopes for improved relations and even for expanded exchanges between the two peoples.

For all the efforts at damage limitation, the perils were plain and the threat remained that the cross words and crosspurposes would spill over into additional areas of Sino-American contention.

According to one report from Peking, which could not be confirmed here yesterday, Deng Xiaoping, China's top leader, told House Speaker Thomas P. O'Neill Jr. (D-Mass.) and his delegation there last week that Sino-American relations have deteriorated almost daily since Reagan took office, and that virtually all of the trust and confidence built up between the two countries in the last decade has been lost.

The central problem is that China does not trust Reagan, and there is no sign that Reagan reposes much trust in China. A wariness in the two capitals has replaced the warmth of the trail-blazing Nixon, Ford and Carter years, when the Sino-American relationship was being resumed after the long postwar hiatus.

If there is a belief in good intentions all around, roadblocks can be surmounted and offensive words or deeds can be forgiven or even ignored. But without a background of confidence, even small disagreements can be the cause of suspicion and bitterness.

The current Chinese litany of complaints, reportedly repeated to the visiting lawmakers last week by Foreign Minister Wu Xueqian, includes the Hu Na case, the dispute over Chinese textile imports to the United States, a federal court ruling against China on 1911 Imperial Chinese Railway bonds, resolutions before Congress on conditions for reunification of Taiwan with the Chinese mainland, the slow pace of high-technology sales to Peking, and the high ceiling placed by the administration on U.S. arms sales to Taiwan.

The most important of these is the Taiwan arms issue, which the two sides had hoped to settle in the joint communique of last Aug. 17. Although it averted a Chinese downgrading of relations on a broad front with far more sweep and impact than the steps taken yesterday, the arrangement did not satisfy either side.

The last line of Wednesday's Chinese diplomatic note tied the future development of the relationship to U.S. compliance with the principles of the 1978 communique on the establishment of relations and the joint communique of last Aug. 17. This means, first and foremost, they depend on handling of the Taiwan issues.

The Chinese made public in their diplomatic note that "a Chinese leader," in fact Deng Xiaoping, had "pointed out the gravity of the Hu Na incident" to U.S. Ambassador Arthur Hummel in a meeting marking the completion of the joint communique last Aug. 17.

A series of other representations about the tennis star followed, including a warning to Secretary of State George P. Shultz during his trip to Peking two months ago that cultural and sports exchanges would be affected by the resolution of Hu Na's case.

Why the Chinese chose to allude to Deng's involvement, when his pleas fell on deaf ears here, is a mystery. It is one more sign that much more than tennis was at stake in the case of the 19-year-old athlete.