The Carter administration moved ahead yesterday with its plan for setting up a private corporation to handle U.S. relations with Taiwan, but there were increasing signs that the move faces rough treatment in Congress.

Even as State Department lawyers were filing papers in the District Building to incorporate the American Institute in Taiwan, the chairman of the House subcmmittee on Asia, Rep. Lester L. Wolff (D-N.Y.), was calling the plan "a charade... duplicity in diplomacy... double talk in diplomacy."

"I don't think this fools anyone," Wolff charged, and asked why it was necessary to "mask" U.S. intentions to continue doing business with Taiwan.

The Taiwan dispute arose from the administration's historic decision last month to reverse long-standing China policy by recognizing Peking as the ligitimate government of China and severing diplomatic relatuons with Taiwan.

The administration wants to continue trade, cultural and other nondiplomatic ties with Taiwan. Tis proposed instrument for that is the institute -- a legally private entity that will be staffed by State Department officials temporarily severed from government service.

Confress will have to authorize the transfer of government functions to the institute and provide funds for its operation. Congressional supporters of the Taipei government have vowed to try and change the proposed legislation to give Taiwan some continued measure of diplomatic recognition.

Wolff, whose subcommittee will be the first in Congress to consider the legislation, threw his support to the Taiwan effort. He said he would favor establishing a liaison office in Taipei similar to the one in Peking that the administration now wants to upgrade to an embassy.

In related moves, Sen. Richard Stone (D-Fla.) called for giving Taiwan's representatives full diplomatic privileges in Washington, and 12 other senators headed by John Danforth (R-Mo.) proposed a resolution stating that if Peking attacks Taiwan, the United States must break relations with China and come to Taiwan's aid.

On Monday, the ceremonial opening session of the Senate was interrupted by efforts or Harry F. Byrd (Ind.-Va.) to introduce a resolution criticizing President Carter for scrappint the 25-year-old mutual defense pact with Taiwan. Senate leaders sidetracked the move, and congressional sources said they probably will succeed in heading off early consideration of Byrd's resolution.

Despite the rumblings, Roger Sullivan, deputy assistant sercretary of state for Far Eastern affairs, said yesterday the administration is confident that Congress will approve the institute legislation and that the Taipei government will agree to have future relations handled on a non-official basis.