The Carter administration, on notice from China that a federal judge's ruling poses "a serious problem" for U.S.-Chinese relations, yesterday began the process of asking the U.S. Circuit Court of Appeals here to overturn the decision.

After filing a formal notice of appeal, Justice Department officials said they will ask today for an expedited hearing by the full nine-judge circuit court rather than follow the normal procedure of having the matter considered by a three-judge panel.

At issue is the ruling issued Wednesday by U.s. dIstrict Court Judge Oliver Gasch that President Carter cannot end the U.S. mutual defense treaty with Taiwan without the approval of either two-thirds of the Senate or a majority of both houses of Congress.

Gasch, ruling on a suit by Sen. Barry Goldwater (R-Ariz) and 24 other conservative members of Congress said Carter had acted improperly when he announced last Dec. 15 that the 1954 treaty with Taiwan would be terminated on Jan. 1, 1980.

China had insisted, as a precondition for establishing full relations with Washington, that the United States break off military and diplomatic relations with Taiwan. In what was considered a pointed reminder of that fact, the Chinese embassy here yesterday issued a statement saying:

"It is a serious matter. We think the United States government will take necessary measures."

In challenging Gasch's decision, administration officials reitereated their contention that the defense treaty carried a provision allowing either country to withdraw on one year's notice and that Carter did not overstep his powers in invoking that step.

At the State Department, where officials were under orders not to comment publicly on the matter, well-informed sources said they did not believe Gasch's ruling will have any short-term effect on relations with Peking.

But, they added, if there are long delays in resolving the case or if the administration ultimately loses its appeal, the consequences would be very serious both for U.S.-Chinese relations and the president's overall ability to conduct foreign policy. The sources said, though, the administration is confident that it will win on appeal.

Within Congress, Gasch's decision and its implications for the struggle between the executive and legislative branches over control of foreign policy drew an initially guarded reaction. Most members seemed reluctant to comment immediately, and those who did tended to divide along party lines.

Senate Majority Leader Robert C. Byrd (D-W. Va.) said: "I find it awfully hard to understand how any appeals court could uphold the ruling." But the Senate minority leader, Howard H. Baker (R-Tenn.), said he agreed with Goldwater's intentions and wanted to see the question of abrogating the treaty brought to Congress for a vote.