A Cabinet-level committee has recommended that President Reagan withdraw special trade privileges enjoyed by four fast-growing Asian nations whose soaring trade surpluses with the United States have created a serious economic challenge to the administration.

Sources said the president is expected to approve the decision of the White House Economic Policy Council, which could be announced as early as Saturday and would take effect next January.

The action reflects the administration's frustration with the so-called "four tigers" of Asia -- Hong Kong, Singapore, South Korea and Taiwan -- whose double-digit economic growth is due in large part to their trade with the United States. The action also reflects the countries' growing stature as industrial powers.

The countries would be removed from the list of 140 less developed countries eligible to import many products into the United States free of duty under the Generalized System of Preferences (GSP), giving them the same status as Japan, West Germany and other industrialized nations.

Together, the four newly industrialized countries (NICs) in Asia received $5.3 billion in benefits under the GSP program by avoiding duty payments, U.S. Trade Representative Clayton K. Yeutter said a year ago. In effect, the duty-free status reduces prices of products from those countries sold in the United States, giving them a significant competitive advantage.

South Korean economists estimated the U.S. action would reduce exports to the United States by $300 million annually. No estimates were available for the other countries.

In the past year the U.S. trade deficit with the four countries grew from $30.7 billion in 1986 to $37.2 billion last year, according to First Boston Corp., and now exceeds the U.S. deficit with Western Europe, which dipped from $32.7 billion in 1986 to around $30 billion last year.

The decision, made Tuesday, is expected to trigger protests in the four countries, all of which are staunch allies of the United States. But it may help the administration in fending off passage of the more protectionist elements in a massive trade bill now before Congress.

The move was led by Treasury Secretary James A. Baker III, administration sources said, after Hong Kong, South Korea and Taiwan resisted pressure to let their currencies increase in value against the dollar as the Japanese yen and currencies of other major trading partners have. Singapore, whose currency has risen in line with the dollar's fall, was included because its trade surplus has also grown, administration sources said.

By preventing their currencies from rising as much as the dollar has fallen, the three NICs have reaped major benefits from the currency changes over the past three years, increasing their exports to the United States and becoming major suppliers of parts to Japanese industries. This Japanese practice of buying less expensive components from the Asian NICs has helped its exports remain competitively priced in the United States.

A declining value for the dollar cuts the cost of U.S. products overseas, increasing their competitiveness, while tending to raise the price of foreign products in the United States.

But removing the preferential treatment given to the four countries is not universally popular in Congress and among American manufacturing and service industries. Sen. Pete Wilson (R-Calif.) said the action is counterproductive "when you yank benefits from countries that have responded favorably to our trade complaints such as Singapore and Hong Kong."

Some congressional trade specialists believe that Hong Kong and Singapore, which maintain no trade barriers, should be allowed to retain their GSP privileges as a sign that countries practicing free trade can get special rewards from the United States.

A coalition of American business also has been lobbying hard against the move, arguing that U.S. firms will suffer along with the NICs. Some the members of the coalition are high-technology manufacturing companies that import low-cost components duty free from the NICs and are concerned that the loss of the GSPs will increase the prices of their products, making them less competitive against Japan and Western European countries. Others are exporters that believe that retaining the GSPs gives the government leverage to force two of the countries -- South Korea and Taiwan -- to ease their import restrictions against U.S. products.

"The GSP Coalition of U.S. Businesses is deeply disappointed that the administration has apparently ignored the negative impact of GSP country graduation on the competitiveness of U.S. manufactures," said Thomas F. St. Maxens and Christine Bliss, heads of the business group.

Administration sources said Baker and Yeutter suggested in preliminary meetings that only South Korea and Taiwan lose GSP privileges, but Secretary of State George P. Shultz said for overall foreign policy reason it was better to treat all the NICs the same. No opposition was registered at the formal EPC meeting Tuesday to removing the privileges of all four NICs, the sources said.