U.S. allies in Europe, hoping to entice the Soviet Union toward a more realistic compromise at the Geneva talks on reducing intermediate nuclear forces in Europe, have begun to move away from President Reagan's proposal for a total dismantling of Soviet medium-range missiles

Offered last year, the Reagan "zero option" calls for Moscow to dismantle its 324 SS20 missiles aimed at Western Europe if NATO abandons plans to install 572 Pershing II and cruise missiles in five member countries a year from now.

The Soviets rejected the offer as tantamount to unilateral disarmament on their part, and the negotiations have languished since.

The European allies have endorsed the U.S. position staunchly in the past, but lately some tactical differences with Washington have surfaced about the best manner to prod the Soviets toward a deal that might involve a reduction in the number of Soviet missiles in return for a lesser deployment by NATO.

At a meeting of alliance defense ministers in Brussels last week, the NATO general secretary, Joseph Luns of the Netherlands, remarked that "the United States never said it was the zero option or nothing."

British Defense Secretary John Nott was quoted by reporters as saying, "If the Soviets propose something less, then we'd welcome it and look at it in a constructive spirit."

Secretary of Defense Caspar W. Weinberger, however, disavowed any notion that the United States was prepared to soften its negotiating stance in Geneva.

In Washington, a State Department European affairs official said, "I am not aware of alternative studies to the zero option. Our position is clear, and we are sticking with it."

The tone and nuances of such comments may not reflect a shift in NATO policy, but they do illustrate different perceptions in Washington and Western Europe about how to coax more cooperation from the Soviets at the bargaining table. Washington believes that the Soviets will bend in their response only when they confront a NATO determined to stick by its position and proceed with full deployment if the negotiations fail.

The Europeans, meanwhile, are eager to clutch at some signs of progress in the arms talks to defuse the prospect of violent protests as the deployment date draws near.

For that reason, some NATO officials in Europe have sought to send conciliatory signals to Moscow about the need to initiate new ideas or counterproposals that might break the deadlock.

"I think Weinberger is bluffing," said William Hyland, a specialist on arms control and the Soviet Union who served in three previous administrations. "He must know that the zero option will not work."

Hyland said he suspects, however, that the Reagan administration has become trapped by its own rhetoric and will wait for the Soviets to make the next move even if the Europeans press Washington for a more flexible policy.

"If we act to change our position before the Soviets do, there will be all those remarks that the U.S. blinked, et cetera." Hyland said.

Some Europeans fear that the administration's insistence to strike such a determined pose with the Soviets could divide the alliance badly next year as the quest for progress in the arms talks intensifies.

"The big question is what the West will do if the Soviets produce a reasonable new offer," said a West German diplomat. "We simply cannot afford to sit back and reject it."

The administration's concern about keeping the Europeans in line with its hard-nosed strategy also has affected the Senate debate over whether to approve funds to develop the MX missile in the so-called "Dense Pack" formation in Wyoming.

Reagan and his advisers have argued that if the Senate kills funding for the MX, the negative vote would serve the purposes of the antinuclear movement in Europe and make deployment of the Pershing II and cruise missiles far more difficult.

That contention has stirred some resentment in Congress that the fate of the $26 billion MX program is being linked to the NATO nuclear missile plan, which costs about $1 billion.

"It really looks like a case of the tail wagging the dog," said a Senate Foreign Relations Committee staff member.