President Carter has lifted import restrictions on color television sets from Japan, while liberalizing current limits on sets from Taiwan and South Korea.

The actions taken Monday drew immediate blasts from the committee to Preserve American Color Televisions, which argued that the domestic industry will shrink under a new wave of imports.

U.S. Special Trade Repersentative Reubin Askew said Carter chose to drop limits on Japanese color TV's because shipments from that country declined significantly.

Under a 1977 agreement, Japan was able to export up to 1.75 million sets here but last year only sent 513,000, the Commerce Department said.

Limits will be increased on TV's imported from Taiwan to 400,000 sets in the next 12 months, and 450,000 the following year, up from 373,000, and on TV's from Korea to 385,000 in the next 12 months and 575,000 during the year after that, up from 204,000.

In May, the U.S. International Trade Commission decided by a vote of 4-0 that the domestic color TV industry would not be injured by a lifting of import restrictions against Japan. But the panel said injury could occur from Taiwanese and Korean color television imports.