A stubborn U.S. merchandise trade deficit, widely expected to improve this year on the basis of a 40 percent decline in the value of the dollar, has confounded the experts and now is expected to soar to record heights for the fifth straight year.

On the basis of the first eight months of the year, the deficit is running at an annual rate of around $170 billion, even higher than last year's record total of $156 billion.

The Reagan administration has been relying on the lower dollar to improve the trade picture by making American products less expensive in overseas markets, and therefore more competitive, while raising the price of foreign goods in the United States and cutting their sales.

But foreign suppliers, fighting to retain market share, have refused to go along with the U.S. strategy and raise their prices to offset the currency changes. Import prices, for example, increased by just 14.5 percent over the past year, and much of that was due to a sharp 62 percent rise in the price of oil. Commerce Department trade economist David Lund noted that nonoil imports now account for 21 percent of national consumption compared with 12 percent a decade ago.

While the price of foreign goods has not risen enough to discourage Americans from buying them, the increase in their cost has worsened the trade picture by inflating the dollar value of imports.

Similarly, U.S. corporations report that their foreign competitors also are swallowing their foreign currency losses in third-country markets. This refusal of foreign firms to set their prices to reflect currency realignments, combined with what the chief economist of the National Association of Manufacturers, Jerry Jasinowski, decried as a lack of aggressiveness on the part of U.S. companies, has prevented the export surge needed to turn around the trade deficit.

Nonetheless, Jasinowski noted that the decline in the dollar "has allowed us to begin to recapture world markets."

But exports, as measured in the monthly trade figures, are only $19 billion higher in the first eight months of this year than they were in the same period of 1986. Imports, though, increased even more -- by $24.3 billion -- during that period.

Administration officials, including U.S. Trade Representative Clayton K. Yeutter and Acting Commerce Secretary Bruce Smart, cited "the real trade balance" based on an increased volume of exports as measured by data adjusted for price changes and seasonal fluctuations to show that there is an underlying improvement in the trade figures.

"The real trade balance, which affects our growth in production and employment, has improved sharply since the third quarter of 1986 and should continue to make strong gains," said Smart.

Richard Rahn, vice president and chief economist of the U.S. Chamber of Commerce, said these figures, contained in the National Income Account data "show an improvement of 30 percent in real net exports during the past year, and that improvement is likely to continue."

But the stock market and politicians tend to watch the monthly trade figures that show the deficit worsening this year over the 1986 record. Traders noted that Wall Street, with an eye on the flood of foreign investment that has fueled the bull market, has become fixated with the monthly trade figures the way it once was obsessed with the weekly changes in the money supply. The fear is that the deficit will require a lower dollar, which will drive foreign investors away.

And campaigning for the Democratic presidential nomination in Iowa -- which took a double hit from yesterday's decline in agricultural and manufactured exports -- Rep. Richard Gephardt (D-Mo.), blamed the trade imbalance on the Reagan administration. "Until we have an administration that fights for open markets overseas," he said, "American exports of manufactured goods and farm products will continue to fall and the heartland of this country will continue to be left behind."

The administration is trying to soften trade legislation passed by the House and Senate, contending that elements in both bills are highly protectionist and will hurt U.S. industries and farmers by triggering retaliation. The administration is especially opposed to a provision sponsored by Gephardt that would force trading partners to reduce their "excessive surpluses" or face U.S. retaliation.

The failure of the trade deficit to respond to the lower dollar has spurred a cry from some economists for an even greater decline.

But economists have been persistently wrong in predicting when the fall in the dollar over the past 22 months would result in a lower trade deficit.

At first, they said "the J-curve effect" would delay improvement in the deficit for 12 to 18 months. The J-curve describes how the trade deficit worsens before it gets better because the price of imports rises and the value of exports declines for a while after currency values change.

When the improvement failed to arrive on schedule, economists changed the date for the J-cure to start from February 1985, when the dollar began its fall, to the following September, when Treasury Secretary James A. Baker III met with his fellow finance ministers at the Plaza Hotel in New York to officially sanction the currency realignment.

Even that timing was off, and when the expected improvement failed to show this spring, economists extended the life of the J-curve to two years.

Now, however, some economists are acknowledging that what they thought was a J-curve is turning out to be a right-angle L, with a leveling off of the trade deficit instead of the expected improvement from the fall in the dollar.