As mass anti-American protests continued in the streets, Taiwan opened talks with a high-level U.S. diplomatic team today in a burst of recrimination, saying President Carter's handling of the decision to recognize mainland China was unworthy of a world power.

The talks, designed to smooth the way from full diplomatic relations to "unofficial" contacts, wound up for the first day with no reported progress. The U.S. negotiators are scheduled to return to Washington after a second round of discussions Friday.

Shortly before today's talks were to begin, an estimated 10,000 demonstrators milled outside the Foreign Ministry, some trampling on peanuts spread on the street in a symbolic attack on Carter, and a taxi driver doused himself with gasoline and set himself afire.

"Long live the Republic of China," the driver shouted as flames covered his body.

He was pulled from the burning taxi and later reported in serious condition in a Taipei hospital. The incident and the excited crowd outside the ministry forced postponement of the U.S.-Taiwanese talks and their transfer to the Grand Hotel five miles away, where they were finally held under tight security.

The talks were almost canceled before they began because of the hostile reception accorded U.S. Deputy Secretary of State Warren Christopher and his team on their arrival here yesterday. The Taiwanese foreign minister, Y.S. Tsiang apologized for that incident, but made a stiff protest of the U.S. decision to end relations with the Nationalist Chinese government here in favor of the Communist rulers in Peking.

"We strongly oppose this decision, which we believe is wrong, and which has most seriously impaired the rights and interests of this country," he said in a statement issued as "the basic position" of Taiwan in the talks.

"Although President Carter's decision is so far-reaching, we were advised of it only seven hours before it was made public. This is not the way for a leading world power to treat a longstanding ally and it has aroused indignation among the Chinese people both at home and abroad."

Spokesmen for both sides declined to reveal the substance of the talks. But James Chu-yul Soong, deputy director of the Taiwan Information Office, said the Taiwanese had demanded that the United States continue a "substantial relationship of military cooperation."

Adm. Soong Chang-chih, chief of the Taiwanese general staff, "also asked the American side to continue providing the Republic of China with highly capable aircraft and modern materiel and weapons to enhance Taiwan's defense capabilities," the spokesman added.

The United States has pledged to continue supplying defense weapons to the Nationalist Chinese government on Taiwan in amounts judged necessary by Washington.

The bitter tone of Tsiang's statement and the second straight day of street protests dramatized the resentment felt here over the end of U.S. diplomatic ties. But analysts questioned whether the anger would be translated into long-term refusal to seriously negotiate the switch to unofficial contacts such as those Taiwan has set up with the many countries that already have recognized Peking.

The government of President Chiang Ching-kuo has little maneuvering room in its quest for modern arms and, because of longstanding trade ties, is considered unlikely to envisage a real cutoff in contacts with the United States.

In what was seen as an indication the serious business of working out the new system of relations might come later, U.S. spokesman John Cannon said conversations will continue between Taipei and Washington after the current talks end.

Under the schedule announced by Carter, the United States will sever formal diplomatic ties to the Nationalist Chinese government next Monday, Jan. 1. Ending the relations was one of the conditions set by Peking for the agreement to normalize relations between the United States and mainland China.

The Taiwanese government has based its arguments against Carter's decision on the treaties linking it with the U.S. government. Cannon, relaying to reporters what was said in the talks, said Tsiang stressed this point to the U.S. team, putting particular emphasis on the mutual defense treaty.