President Reagan responded yesterday to the fourth increase in the prime rate in four months by saying "there is no excuse" for high interest rates. White House spokesman Larry Speakes blamed banks and the financial markets for "fear and skepticism" about future inflation, and indirectly faulted the Federal Reserve Board.

Reagan's political strategists have said that no other single issue is potentially more threatening to the president's reelection campaign than rising interest rates.

Yesterday, speaking to a group of farm community officials at the White House, Reagan said nothing directly about yesterday's increase in the prime from 12.5 to 13 percent. Instead, he noted that it had come down "by nearly half" since he took office and said high interest rates generally are caused by "fear of the future."

"I think to get them down they [the markets] just have to finally realize we're serious about keeping inflation under control," Reagan said, repeating an explanation he has offered frequently in the past for high interest rates.

"There is no excuse for the interest rates being at the level they are right now, other than just fear of the future," he said.

Speakes said the prime rate rise "is difficult to explain" in light of low inflation. "We see no reason why the markets and the banks should continue to be skeptical about inflation," he added.

Bankers have come to expect renewed inflation with each economic upswing, Speakes said, but "it simply is not happening in this administration because we've got a new approach and a president who is determined to hold inflation down."

Many economists attribute rising interest rates to increasing private credit demands and competing pressure from federal deficits, which are running $170 billion a year. Speakes said the White House agrees that rising credit demand is forcing interest rates up, but does not think deficits are to blame.

Speakes said that "we mean to reduce the deficit even more in 1985," but that Reagan would not reveal details until after the November election.

In May, when the prime last increased, Speakes faulted the Federal Reserve Board, which controls the money supply. Reagan later said the Fed was "right on target." Yesterday, Speakes indirectly criticized the Fed again, saying "it's hard to accept" that the federal funds rate--which banks charge each other for overnight loans--is fluctuating between 11 and 12 percent when inflation is between 3 and 4 percent.

Reagan, in another day of campaign-style appearances at the White House, told farm representatives that the economy is improving. Also, addressing a group of black administration appointees, he said "do-nothing Democrats" in Congress had stalled his enterprise zones legislation to provide tax breaks to spur inner-city development. On civil rights, he said that, "contrary to a lot of demagoguery," the administration has moved "with vigor and vision."