President Reagan said yesterday that "interest rates can and should" come down some more, but whether or not that happens "is up to the banks," and not to the Federal Reserve System.

While Congress has been pressuring both the Federal Reserve and, to a lesser extent, the banking industry to take the lead in lowering interest rates, Reagan joined others in his administration in laying the burden squarely on banks. He said he saw "no reason why the banks can't bring those interest rates down another notch or two."

With inflation as low as it is, Reagan told a group of reporters at a White House breakfast, "I believe that interest rates can come down more and should because it leaves the real interest rate higher than is necessary to cope with inflation."

Reagan also said that participants in the Economic Summit scheduled for Williamsburg, Va., at the end of May could decide to consider the question of a new international monetary conference to deal with the question of exchange-rate relationships.

The presidential "jaw-boning" on bank interest rates is indicative of his administration's growing displeasure with the banking industry.

Treasury Secretary Donald T. Regan recently told a congressional committee that banks are maintaining artificially high consumer interest rates to compensate for losses that have been incurred on bad international and domestic loans, a charge that the banks deny.

The president also restated the administration's annoyance with the banking lobby's effort to get Congress to repeal 1982 legislation requiring tax withholding on interest payments and dividends.

"I think they are entirely wrong, Reagan said. "And I think a great lobbying effort has resulted in much distortion."

His remarks came during an upbeat account of economic prospects. He predicted that recovery is under way, and could be upset only by excessively high real interest rates.

Most economic experts, he said, believe the strength of the recovery will exceed the administration's projections, but "we're going to stick with our conservative proposals for recovery and hope we'll be happily surprised."

Reagan rejected the notion that the Fed should attempt to push interest rates any lower. "We have come down from that 21 1/2 percent prime rate to an 11 percent prime," he said. "The discount rate is lower than that. But I think it is up to the banks. And I do not know of what the Fed could do to force that. They cannot give orders."

At another point, Reagan said: "They're very concerned over there at the Fed about the road they're walking, and how narrow it is, or in giving the wrong signal, or a signal that might be taken wrong, by the financial markets--that they might in some way set back this recovery, which they agree is taking place."

In general, his comments about the Federal Reserve seemed to be complimentary. "I must say . . . that the Fed has been cooperating. We've been getting along very well after some violations"--a thought and phrase he did not elaborate on. Earlier in the administration, there had been some sharp exchanges between the Treasury and the Fed over the management of monetary policy.

Reagan also hedged when asked about his considerations in deciding whether to reappoint Paul A. Volcker as Fed chairman when his term expires this summer. "I have to be honest with you and tell you that I have had too much on my plate to even be thinking about that at this time," he said.

The president put in a strong plug for $8.4 billion in authorizations for the U.S. share of larger International Monetary Fund quotas, and for a related emergency lending pool. He pointed out that the transaction involves a non-budgetary "bookkeeping arrangement," and added:

"The IMF is serving a very useful purpose in this time in which the whole world, the international banking situation is walking a tightrope."