Commerce Secretary Malcolm Baldrige said yesterday that the economy will not grow as fast this year as the Reagan administration expected, raising the possibility of higher unemployment and larger budget deficits.

Baldrige told a breakfast meeting with reporters that the economy will grow at less than the 3.9 percent rate targeted by the White House earlier this year. He would not disclose the administration's new growth estimate.

"It will be very difficult to make 4 percent this year," Baldrige said. "It will be somewhere between 3.5 percent and 4 percent."

Preston Martin, vice chairman of the Federal Reserve Board, warned late last month that the nation was on the verge of a "growth recession" and that faster money growth and lower interest rates may be needed. "A growth recession must be considered a real threat. In fact, the data currently available suggest that the economy is on the edge between healthy, sustainable growth and a growth recession," Martin said.

In a growth recession, the economy grows at such a slow rate that unemployment may rise and the use of available production capacity may fall.

In April, the Commerce Department reported that the economy grew at a surprisingly sluggish 1.3 percent annual rate in the first quarter. The inflation-adjusted increase in the gross national product was the smallest for a quarter since the end of the recession in late 1982.

Baldrige said yesterday that it is too early to change the administration's growth forecast to reflect the expected lower rate and that the slower growth will not have much effect on the deficit.

A major reason for the slowdown has been the rise in imported goods, which have absorbed consumer and business demand at the expense of domestic production.

The Reagan administration has been trying to get the world's major industrial nations to agree to a new round of trade negotiations in an effort to resolve the import problem. But President Reagan failed last week to win agreement for the talks at the economic summit in Bonn when France refused to go along with the idea. The seven nations involved in the summit were the United States, Britain, France, West German, Japan, Canada and Italy.

Baldrige said yesterday that the refusal of France to go along with setting the date for the talks will lead to increased protectionist pressures in Congress.

Baldrige said he expects the rate of growth to increase in the second quarter to 3.5 percent or more. "I think we'll see a significantly stronger second quarter than we did the first quarter," he said. "For the year, I think we'll have a good year. I think it will be less than 4 percent but not by much. Particularly if we can get the budget deficit reduced, we can come back strongly in the second half."

According to government economists, a one-half-percentage-point drop in real output would result in an increase in the federal budget deficit of about $2 billion in the year that the decline took place. However, if growth continued at the lower rate, the effect of the decline would increase to about $35 billion by the end of the decade.

These calculations assume that other factors, except unemployment, are constant.

A slower rate of growth also could prevent the administration from reaching its goal of an average civilian unemployment rate this year of 7.1 percent. The unemployment rate for the last three months has stayed at 7.3 percent.

The prediction of growth below 4 percent is still higher than forecasts by major private economists of growth between 3 and 3.5 percent this year. In light of the unexpectedly slow growth in the first three months of the year, many economists have begun to lower their estimates for 1985