A decision by President Reagan last Monday to seek changes in two dormant treaties with the Soviet Union has opened up one of the most contentious issues between the two superpowers: foreign inspection of rival nuclear weapons tests.

At a meeting in the Cabinet Room of the White House, the president and his National Security Council advisers planned to portray the decision as a step forward in nuclear arms control. Instead, the plan backfired badly, enmeshing the administration in a new domestic and international dispute about its nuclear policy.

"I thought we had moved the process forward an inch," one senior administration official said ruefully at the end of the week. "But the way it comes out is that it looks like everything has gone to hell in a handbasket."

According to other sources, a potentially larger controversy exists behind the administration's latest nuclear decisions than anything that has emerged.

Administration advocates of hard-line bargaining with the Soviet Union are now seeking tougher terms for limiting nuclear weapons tests than any previous administration. They want the United States to insist on the right to send American inspectors to Soviet weapons tests, to drill holes and implant U.S. instruments, to verify Soviet compliance with limits on the size of nuclear explosions.

Other officials are advocating much more limited demands on the Soviet Union. There is no indication which course the president will choose. In either case, new American terms for verification are bound to be reflected as well in the central negotiations on nuclear arms control, the Strategic Arms Reduction Talks now under way in Geneva.

Furthermore, intrusion into Soviet territory to verify nuclear tests means equal Soviet penetration of American test sites. Although the United States is generally free of the Soviet obsession with secrecy, the "I thought we had moved the process forward an inch," one senior administration official said ruefully. "But the way it comes out is that it looks like everything has gone to hell in a handbasket." Joint Chiefs of Staff and nuclear weapons specialists in the Energy Department are reported to be registering adamant opposition to the idea of admitting Soviet observers to American test sites in Nevada.

One Senate specialist said that the Joint Chiefs "turn pasty white at the idea."

None of these specifics was discussed, administration sources said, in the National Security Council last Monday. In that session, Secretary of State George P. Shultz, who carries the responsibility for negotiating on arms control, was the novice on the subject.

He was joining Defense Secretary Caspar W. Weinberger; Gen. John W. Vessey Jr., chairman of the Joint Chiefs of Staff; Vice President Bush; national security adviser William P. Clark, and other senior officials in deciding, as one source described it, "what our declaratory policy should be," rather than the specifics of the policy.

The intention was to portray the administration in a positive way, seeking to improve and salvage two treaties negotiated by the Nixon and Ford administrations.

But what first leaked from the meeting was wholly negative: confirmation that the Reagan administration has no intention of resuming far more ambitious American-Soviet-British negotiations to end all nuclear weapons tests.

Those negotiations were last conducted by the Carter administration in November, 1980.

The formal decision to close off a channel for negotiating what is known as a Comprehensive Test Ban, came as no great surprise; it was publicly foreshadowed by administration officials last year. Even so, confirmation of that position touched off attacks from arms-control advocates.

To try to offset that damage, the White House disclosed the decision to seek changes in the two treaties to limit explosions underground, the only area where American-Soviet testing has been permitted since 1963. Those treaties were negotiated in the 1970s but never ratified. This disclosure, however, only added to the controversy.

The first sidetracked pact, the Threshold Test Ban Treaty, signed in 1974, prohibits underground nuclear tests greater than 150 kilotons--the equivalent of 150,000 tons of TNT. The second, the Peaceful Nuclear Explosions Treaty, signed in 1956, is a companion pact, setting the same level on explosions for civilian purposes, such as changing the course of rivers.

The peaceful-uses treaty contains the only negotiated authority for each nation to send nuclear observers to each others' territory, under tightly controlled conditions. There is no comparable authority, however, to inspect weapons tests.

Both treaties were heavily criticized, largely by liberals, on grounds that the 150-kiloton threshold was too high to be meaningful as an alternative to a halt in testing.

The two treaties were in limbo until this administration, when the Senate Foreign Relations Committee, spurred by Chairman Charles H. Percy (R-Ill.) and other arms control advocates, sought to counter Soviet charges that the United States had defaulted on them.

Administration officials say the two pacts are seriously defective. They charge that the Soviet Union, while ostensibly honoring the ceiling on explosions, has tested nuclear weapons of up to 300 or 400 kilotons, with no recourse in that treaty to produce incontrovertible proof of violations.

Eugene V. Rostow, director of the Arms Control and Disarmament Agency, told the Senate committee May 13, however, that there was more than one barrier to ratification. Rostow testified that he had "run into a profound stone wall" in trying to get action on the two treaties.

"The stone wall," Rostow testified, "is the feeling in many parts of the government that, given the uncertainty of the nuclear situation, the nuclear balance and the need for new weapons and modernization . . . we are going to need testing and perhaps even testing above the 150-kiloton limit for a long time to come."

In addition, Rostow said, there are "grave doubts" about the adequacy of the verification procedure in the weapons treaty. The peaceful-uses treaty contains the only negotiated authority for each nation to send nuclear observers to each others' territory, under tightly controlled conditions.

A senior Pentagon official privately acknowledged last week "concern about a legitimate need for testing in the future over the threshold," but he said, as did Rostow, that the verification inadequacies in the treaties are the main difficulty.

The administration's official position is that the verification weakness is the only problem.

To avoid the charge that the administration was rejecting the treaties, spokesmen were told not to use the term "renegotiation." Officials talked instead about the need to "strengthen the verification provisions," which some officials described as "technical refinements."

One source involved in the process, however, ridiculed these descriptions as circumlocutions.

Rostow, nevertheless, said Friday that he was pleased with the outcome--except for the criticial publicity--because "we have some prospects for improving the verification procedure."

Percy has described himself as "pleased that the administration has decided not to reject" the two treaties, and he expressed hope for an early resolution of the verification requirements. No administration official anticipates that.

If the hard-liners in the Defense Department and other agencies should prevail, the administration would end up asking the Soviet Union not only for advance notice of all weapons tests, but also for the right to choose what tests to inspect.

This would involve, said one advocate of this approach, "our inspectors at their tests" with "our technicians having an opportunity to 'instrument' their test program," combined with authority "to drill holes and to place instruments" for measuring the yield of nuclear explosions.

In such a test observed by Americans, this specialist said, the Soviet warhead being tested "could be shrouded as it is lowered into the test hole," to screen it from the Americans.

When asked how he would expect the Soviet Union to react to such a demand, he said, "It will be a jolt," but added: "This is the only way to verify--and even then there would be some margin for doubt."