The Senate yesterday defeated a key amendment characterizing an attack on Taiwan as a threat to U.S. "security interests" as Congress worked on legislation establishing a new basis for American relations with Taiwan.

The House, working on its version of the measure, defeated key amendments also opposed by the administration. Both bodies will resume action on the bills next week.

By 50 to 42, the Senate rejected the amendment by Sen. Charles H. Percy (R-Ill). Percy had said wording approved by the Senate Foreign Relations Committee -- that an attack by China on Taiwan would be "of grave concern" to the United States -- was too weak. That language also is in the House bill.

Administration officials said the Percy language invoked the specter of the mutual defense treaty with Taiwan that the United States plans to abrogate next Jan. 1. They said the amendment would come close to contradicting the terms of Washington's agreement with the People's Republic of China on establishing diplomatic relations. China, they have said, would not tolerate any restoration of the mutual defense agreement.

The administration and its supporters got a scare when a move to table Percy's amendment was defeated 49 to 45. They quickly called on Vice President Mondale and White House congressional liaison chief Frank Moore for some heavy lobbying.

After the Senate declined to kill Percy's amendment by tabling it, Foreign Relations Committee Chairman Frank Church (D-Idaho) said, "The president of the United States is now in the Middle East in an effort to win peace. I think that a vote of this kind while he is in Cairo would send a very unfortunate message....

"If it is the object of the Senate to undermine and place in jeopardy normalization of relations with maniland China, let it be clear to everyone that is what they are doing with this vote."

Church's plea and the intense lobbying of administration officials changed the votes of enough senators to defeat the Percy amendment, 50 to 42, when it came up for a final vote.

President Carter announced Dec. 15 that as of last Jan. 1 the United States would recognize Peking and end diplomatic relations with the Nationalist Chinese government on Taiwan. The measure before Congress would set up a method for the United States to conduct unofficial relations with Taiwan.

Supporters of the administration argued yesterday that Percy's amendment would move the U.S. defense perimeter to a point between two parts of a country that the United States now views as one.

But in an argument over semantics that characterized debate on the Taiwan legislation, Percy said his language "neither pledges the use of force nor rules out the possibility."

In House action, Rep. Ken Kramer (R-Colo.) tried to place an amendment in a section of the bill that would make defensive weapons available to Taiwan and require the president to notify Congress of a threat to the island. Kramer's amendment said an attack on Taiwan would be "dangerous to our security" and the United States would act accordingly. It was defeated 221 to 149 on grounds that it, too, would revive the mutual defense treaty.

House Republicans concentrated on trying to upgrade the status of the unofficial relations with Taiwan, arguing that the new office there should be a "liaison office" rather than an institute.

While the House saw an onslaught of amendments to the measure from Republicans yesterday, none sought to significantly change the key paragraph about an armed attack on Taiwan being of "grave concern" to the United States.

Instead, the Republicans concentrated on trying to upgrade the status of the unofficial relations with Taiwan by arguing that the new office in Taiwan should be a "liaison office" rather than an institute.

They argued that Taiwan still has a government and the Untied States should at least accord it the type of recognition it had given to mainland China since President Nixon visited there in 1972.

But House Foreign Affairs Committee Chairman Clement Zablocki (D-Wis.) said Taiwan already had established its "institute" and "we would have difficulty keeping our agreement" with the People s Republic if a "liaison office" were established on Taiwan.

The liaison office amendment, offered by Rep. Dan Quayle (R-Ind.), lost 181 to 172.

The House also rejected a move by Rep. Robert Lagomarsino (R-Calif.) to call on President Carter to consider withdrawing diplomatic relations from China if it used force against Taiwan. The vote was 197 to 169.

Majority Leader Jim Wright (D- Tex.) argued that the bill the House was working on was "our own creation, not the bill sent by the president. This is Congress asserting itself in foreign affairs."

At first, the White House rejected any language ensuring Taiwan's security, but decided, in the face of congressional pressure and reaction to China's invasion of Vietnam, to accept the "compromise" language in the bill.

Though there is congressional ire over what Rep. Robert Bauman (R- Md.) termed the desertion of an ally and over Carter's failure to consult with Congress before recognizing China, the bills are expected to pass both houses.

The Senate scheduled a final vote on the measure for Tuesday.