Federal Reserve Board Chairman Paul A. Volcker yesterday said he expected the economy to grow about 3.5 percent to 4.0 percent this year and repeated earlier warnings that continued large federal budget deficits could harm economic growth.

Volcker, testifying before the Senate Budget Committee, said that he expected to present the Fed's economic forecast to Congress next week, but that his growth estimate would not be much different from that of the administration or the Congressional Budget Office. The administration is forecasting 4 percent growth next year, and the CBO predicts 3.5 percent.

Volcker said it was urgent that Congress and the administration reduce the budget deficit by at least $50 billion in the 1986 budget. Volcker said that the situation would worsen as the country becomes more prosperous because borrowing increases when the economy expands.

Such private borrowing would clash with the needs of the federal government and keep interest rates higher.

"The greatest single action we can take," Volcker said, "is aggressive action on the deficit."

However, in response to a question, Volcker said that the consensus in the financial markets seemed to be that Congress probably wouldn't cut much more than $25 billion from the budget for 1986. Volcker said he didn't know what tax changes, if any, the financial markets were expecting this year.

Volcker said he wasn't worried about the danger of an imminent recession, but about the events that usually would lead to a recession, such as a sharp drop in the investment of foreigners in U.S. assets. Those assets, in part, have financed the federal budget deficit.

"That is the kind of event that could precipitate that recession, with financial market consequences," Volcker said. "So I would worry about that earlier stage, about the events that might grow out of the combination of high deficits and other factors that would precipitate the recession prematurely, rather than worry about the recession itself. We are in jeopardy of that kind of difficulty."

Volcker also said, in contrast to some statements by President Reagan, that the economy cannot grow its way out of the deficit.

Volcker said he could not say when the economy would collapse from the weight of the deficits. "I think the prospects look pretty good" for continued growth in the short run, he said. But he cautioned that in the past, economic prosperity has unraveled very quickly.

The Fed chairman said that the continuing large deficits made it difficult for him and other members of the Fed to conduct monetary policy. He said one policy option was not printing more money. "I don't think we have the option of inflating this problem away," Volcker said. He said such a move would spell defeat.

Volcker said the "inflation scare" was diminishing. "It's greatly improved, greatly muted, yes," he said.

But he said that "I simply do not believe we can keep inflation low until you keep the budget deficits" below $200 billion.

Volcker said that some people may be lulled into believing that the deficits are not harmful because the economy has remained strong since the recession ended 26 months ago. But the large budget deficits have hurt farmers, steelworkers and those in the machine tool industries and those who export or compete with imports, he noted.