The fifth round of Sino-American textile talks ended today without agreement, but the chief U.S. negotiator claimed some progress was made in resolving the bitter trade dispute.

Negotiator Peter Murphy said the week-long talks were held in a "positive atmosphere," with both sides hoping to reach an agreement at the next phase of discussions, for which no date has been set.

His optimism sharply contrasts with the tone of the previous negotiating session, which broke down in acrimony two months ago, setting off a volley of protectionist moves that pushed the United States and China to the brink of trade war.

Anxious to give breathing space to the depressed U.S. textile industry, Washington unilaterally imposed ceilings on 33 categories of Chinese textile imports.

Peking claimed that it deserves special status to offset its overall imbalance in Sino-American trade and retaliated by freezing new purchases of cotton, synthetic fibers and soybeans and threatening to reduce imports of other agricultural products.

The issue has become one of the main irritants in a relationship still badly bruised by differences over U.S. arms sales to Taiwan.

In trade terms alone, it is potentially disruptive because of the big stake that each side has in the other's market. China has become the largest importer of American wheat. The United States, in turn, is one of Peking's main outlets for low-priced textiles.

The textile talks center on three main areas of contention--which categories should be restricted, how high the quotas for these categories should be and what the rules should be for imposing new contols on other textile goods during the life of a new agreement.

China, which is expanding its domestic textile industry as a major foreign exchange earner, hopes to realize an overall increase of 6 percent in textile sales to the United States.

The Reagan administration, however, wants to limit growth rates to less than 2 percent, bringing China in line with the Big Three foreign textile suppliers to the United States: Taiwan, Hong Kong and South Korea.

"The talks have narrowed differences, but the two sides still have a way to go," said a Western diplomat close to the negotiations.

U.S. Secretary of State George Shultz had hoped during his visit to Peking last month to salve the wounds left by the Taiwan arms sales controversy that is blamed for inflaming less complicated issues such as textile quotas.

But the Peking leadership continues to let the problem overshadow the relationship, taking every opportunity to pressure Washington into cutting back arms to the island, which China considers a runaway province.

Premier Zhao Ziyang indicated this week that the Taiwan matter stands in the way of a visit by him to the United States. President Reagan invited Zhao to come this year.