Vice President Bush said yesterday that both the United States and the Soviet Union face "enormous" economic problems that could affect military spending and hasten an agreement to limit the nuclear arms race.

But Bush also told reporters over lunch at the vice president's mansion that the United States still can better afford to build up its defenses. And he expressed concern that congressional attempts to scale down President Reagan's ambitious rearmament program would "send a signal" to the Soviets "that we're not going to be able to follow through."

Bush, who talked with new Soviet leader Yuri V. Andropov in Moscow last month after attending the funeral of Leonid I. Brezhnev, was questioned about whether the change in leadership in Moscow can be expected to change Soviet-American relations.

"An ingredient that's different looked at over the immediate future is there are some economic problems that haven't always been present on either side," Bush said. "One of the things that's different is the enormous economic overload, not just on the Soviet Union alone, or on the United States alone, but on the bloc countries . . . . "

The vice president said these economic problems constitute a factor "that the Soviet leadership has to take into account just as we have to take enormous economic problems . . . . How that will affect political behavior I don't know."

Bush expressed doubts about the conventional wisdom that the Soviet Union is prepared to devote to defense an unlimited share of its resources, regardless of the costs to its domestic economy.

There is a "new economic dynamic," he said, "and I hope that will be among [the] things that push to a reduction, verifiable and equitable, in nuclear weapons. That's the point I'm making."

He cautioned, however, that strains on the Russian economy are "not enough, necessarily," to produce an early agreement on limiting or reducing nuclear arms. "I'm not saying economics is going to compel agreement," he said. "I'm just saying it's an ingredient there."

Questioned about whether the United States also needs to restrict its defense buildup because of strains on the economy, Bush said, "probably not like the Soviet Union restrains them."

While the Soviets have "methodically" increased the share of their national output going into defense, he contended, "we have not demonstrated that. Indeed, percentage of GNP gross national product for military has gone down" in the United States. "There's a major difference there. Different conditions."

The vice president, who was among the administration officials lobbying congressmen this week for the MX missile, said he was concerned about efforts to trim Reagan's defense expansion.

"I am concerned about this mood," Bush said. "It's not just Democrats, it's other classes as well . . . . I would hate at this juncture to send a signal that we were not going to be able to follow through. Not that every 'T' has to be crossed and every 'I' dotted exactly the way we called for, but that we're not going to be able to follow through on our major commitment to modernization and updating of our forces.

"I hope we can get some success on arms talks," with the Soviets, Bush said. "That's very important to the president. A lot of people don't understand that . . . . "

"I think the American mood is that people want to see a reduction in nuclear arms," he added, noting the success of nuclear freeze proposals in the midterm congressional elections. Some of those ballot propositions, he said, "were worded almost like what we feel. The Wisconsin thing, you know, very interesting.

"We have a job to do to convince people that this president wants a verifiable reduction in arms. No question about that. And we think we know how to get there. And we think the people who want to do it with some quick fix or quick freeze don't know how to get there. Or that it wouldn't be as good a way to get there."

Bush reiterated the administration view that only a military buildup by the United States will produce genuine arms concessions from the Soviets. He referred to the antiballistic missile treaty in 1970, approved only after the Senate voted to develop an ABM system.

Asked if he had seen anything in Moscow that would indicate the Soviets are sending a signal to the United States by relaxing rule in Poland, Bush responded that "we have to be very alert to signals. We set out early on three criteria, and one of those criteria has been met and that's good. My position . . . would be listen, let's keep our ears and eyes open and if something good happens let's be alert to it."

The first condition was release of Solidarity leader Lech Walesa. Bush said he is hopeful that authorities will lift martial rule in Poland on Dec. 13, the first anniversary of the crackdown, meeting a second condition set down by the administration.

Other administration officials said the United States is prepared to make "step-by-step" responses if the Polish leadership takes "substantial actions" to lift controls.

Poland has been seeking a blanket rollback of sanctions that the United States and other members of the North Atlantic Treaty Organization imposed on Poland a year ago. But that is not in the cards, American officials said. Instead, they reported, the United States will respond in equally significant terms to precisely what happens in Poland.

Bush yesterday threw cold water on the idea of an early summit meeting between Reagan and Andropov. He said he believes a summit should take place only if "something was going to come out of it" and not "just to get together to see the color of his eyes, and try and figure out whether he speaks English. Or drinks scotch."