The U.S. merchandise trade deficit increased to $14.4 billion in May, the government reported yesterday, ending a brief improvement that had encouraged hopes the big deficits were receding.

The report caught trade specialists and the financial markets by surprise and sent the dollar reeling in foreign exchange trading. The dollar, quoted at 151.25 yen late Tuesday, dropped to 148.35 yen before climbing back to 148.85. The decline in the dollar, in turn, drove down prices of U.S. government securities, but those issues recovered later. Analysts said the increased deficit indicated the dollar would decline further.{Related story, E2.}

Record imports totaling $34.8 billion were the major cause of the increase in the trade deficit, which forecasters had predicted would settle to about $13 billion from the April level, which was $13.32 billion. The previous import record, $34.7 billion, was recorded in March.

Economists noted that the falling value of the dollar, which helps U.S. overseas sales, increases the price of imported goods and makes the import record look worse than it does when measured by the actual volume of products entering the country.

The May figures contained one bright spot for the nation's competitive stance -- the fourth straight month of increases in U.S. sales overseas and the third straight month when those sales exceeded $20 billion. Further, the export surge was led by a 2.1 percent increase in overseas sales of U.S. manufactured goods, a sector in which the United States, once the world leader, has lagged other countries in recent years.

Finance Committee Chairman Lloyd Bentsen (D-Tex.) called the May figures "bad news" that makes it "imperative that we enact trade legislation this year to give the president the tools needed to establish a consistent, effective trade policy."

He noted that the Reagan administration said in 1986 that the trade performance had turned around, only to see it finish the year with a record deficit near $170 billion, and had predicted a sharp drop this year. "The fact is," Bentsen said, "we haven't even turned the corner."

But William Lilley III, president of the American Business Conference, an organization of medium-sized entrepreneurial companies, said the May figures "portray an economy regaining export momentum" and warned that "protectionist items" in the trade package will cause other countries "to target U.S. exports for retaliation."

Commerce Secretary Malcolm Baldrige, noting that the Reagan administration had warned of "occasional monthly setbacks" against the overall trend of improving trade balances, emphasized that exports in May were 9 percent higher than the average for the first three months of the year.

"The cheaper dollar and productivity gains by American businesses have improved our competitiveness in international markets," Baldrige said.

He blamed the increase in imports on larger quantities of foreign cars and petroleum products coming into the country at higher prices.

"Imports of foreign automobiles have risen sharply despite poor sales, adding to already large inventories," Baldrige said.

The May trade figures showed that the United States imported 6.2 million barrels of oil a day, 600 million more than in April and 200 million more than the January-to-April daily average.

Moreover, this added oil was imported at a higher price -- $18.15 a barrel, 43 cents higher than the April price and $1.34 greater than the January-through-April average.

While oil and manufacturing imports increased, the Commerce Department reported decreases in U.S. sales of foreign-made telecommunications equipment, industrial and electrical machinery, office machines and computers.

Export growth included increases in overseas sales of chemicals, office machines and computers, electrical machines and autos and auto parts. But exports of planes, aircraft parts and power generating equipment decreased.

Once more, the United States ran its largest trade deficit -- $5.1 billion, or more than one-third of the total shortfall -- with Japan. That is slightly higher than April's deficit of $4.9 billion, but generally in line with the figures for most of this year.

Japan has claimed the lower dollar and higher yen makes the trade deficit, as measured in dollars, look larger than it is in yen or in volume terms.

The United States ran deficits of $2.6 billion with Western Europe, up $300 million; $1.6 billion with Taiwan and $1.3 billion with Canada, both about the same as in April; $1 billion with the OPEC nations, up $200 million; $900 million with South Korea, up $250 million; $600 million with Mexico, up $100 million, and $500 million with Hong Kong, up $71 million.