The White House is trying to formulate new ways of dealing with the Soviet Union on economic issues that administration officials hope will convince Moscow to make more spending decisions in favor of butter rather than guns.

A study of this is being done under direction of the White House National Security Council (NSC) staff, according to administration officials.

Although there is dispute within the government as to how vulnerable the Soviets are to economic pressure, some White House officials seem convinced the Soviets are in enough economic trouble to give the United States and its allies new leverage over Kremlin policies.

One senior official claims that "the Soviet economy is in shambles" at a time when the Soviets also have other serious problems in Poland and at home, including an uncertain period of transition in leadership as the era of Leonid Brezhnev comes to a close.

Moscow's oil supply is limited, the official said, and the Kremlin is being forced to sell record amounts of gold to buy grain and and pay off Soviet and East European debts in the West. The official claims the Soviets have cash flow problems.

"They are doing everything but kiting checks," he says. "All of this has been recognized by this administration. So let's address it and see if we can use them to get the Soviet government to make a better trade-off between guns and butter. We want to make them change . . . but you can't do it with military force alone. There has also got to be an economic policy."

The NSC study, officials say, is being directed by Henry Nau, a specialist in international economics on the NSC staff and by staff planning director Norman Bailey. It is meant to produce a presidential directive on general economic policy toward Moscow similar to the recently completed national security Decision Directive 32 that laid out the broad military strategy and priorities for countering the Soviets.

The idea that the United States and its allies have been contributing to Soviet military strength and easing Soviet economic burdens by assorted trade, technology export and credit policies has been a central theme of the Reagan administration.

In his annual military posture report to Congress in February, Defense Secretary Caspar W. Weinberger placed extraordinary emphasis on this point, saying it was "a testimony to our past blindness" that western policies had allowed Moscow to avoid making the choice between its military priorities, overseas adventurism and modernization of its industry and society.

Last month, in outlining the new national security directive of the president, William P. Clark, Reagan's special assistant for national security and head of the NSC staff, added, "We must also force our principal adversary, the Soviet Union, to bear the brunt of its economic shortcomings."

But whether and how that can be done remains a big question. The two most obvious ways of squeezing the Soviets, restrictions on credit and grain, can succeed only if other countries cooperate and, in the case of grain, would require a major reversal of policy by President Reagan.

In contrast to his tough stance against Moscow on a variety of issues, the president continues to be guided by his campaign pledge to farmers to lift the Carter administration embargo on grain sales to the Soviets. That embargo was put in place by Carter in January, 1980, after the Soviet invasion of Afghanistan and was lifted by Reagan in April, 198l.

The continued sale of grain to the Soviets at record levels is constantly used as an example of American inconsistency by West Europeans who have greater economic interests at stake in dealing with Moscow, who resist U.S. pressures to cut back and who believe that trade is a way to induce good behavior by Moscow.

Officials say the NSC study will take another look at U.S. grain export policy. Four countries--United States, Canada, Australia and Argentinia--are the major grain exporters. Whether cooperation among them was ever a real possibility, officials acknowledge that it has been set back by the crisis in the Falkland Islands in which the United States backed Britain rather than Argentina.

In assessing American economic assets and how they could be used to alter Soviet behavior, officials say the NSC study will also focus on eliminating U.S. government guarantees of credit to the Soviets in situations where private banks are unwilling to extend such credit. The arrangements of private banks with Moscow both here and among allies will also be studied.

But here, too, international differences are obvious. At the just-concluded economic summit conference in Versailles, the United States made only limited progress in efforts to curb export credits, interest subsidies and guarantees to the Soviets by West European countries.

Although there is widespread agreement among specialists that the Soviet Union is in serious economic trouble, many warn that the Soviets are unlikely to be brought to their knees by economic pressure. These specialists say the Soviets still have big reserves in gold and cash, that the government can tighten its belt further and that the Soviet population's ability to absorb hardships is well known.