The Reagan administration's top arms control adviser yesterday said he doubts that withholding funds from the MX missile would have an "immediate or direct" impact on the arms talks in Geneva, a statement placing him at odds with the administration's main argument for releasing $1.5 billion for the missile this year.

Paul H. Nitze, a central figure in preparations for the U.S.-Soviet talks that began Tuesday in Geneva, said that approval of the missile would send the Kremlin "a strong signal of national resolve" and strengthen the hand of the negotiators.

But he said, "It isn't correct to say you would see an immediate, direct correlation" between the funding and the talks.

Nitze's comments to the House Armed Services Committee were contradicted at the same witness table by Defense Secretary Caspar W. Weinberger, who repeated the warning, saying the Soviet Union "will have won the negotiations before it starts" if funding for the missile is defeated.

The administration has argued that Congress' failure to unfreeze funds to build 21 more MX missiles would weaken the U.S. negotiating position in Geneva because the Soviet Union would have little incentive to trim its nuclear arsenal.

Weinberger stressed that the MX is not a "bargaining chip" to strengthen the U.S. negotiating stance, but he asked, "Why should they seek to reduce their arsenals, if we have signaled we are going to permit them to maintain -- and perhaps even expand -- advantages they currently enjoy?"

The contrasting forecasts passed without comment from committee members who are expected to vote next week on the administration's request for funds that were approved but frozen until another round of voting this month. The committee supported funding of MX production last year and is expected to vote to release the $1.5 billion.

Nitze testified in support of the MX, calling it necessary to modernize the U.S. nuclear force after years of Soviet advances and to signal Soviet leaders that "they should get down to serious bargaining at the negotiating table" in Geneva.

"Congressional support for the MX will send just such a message to Moscow," he said in his prepared text. "It will send a strong signal of national resolve and will greatly strengthen our hand in Geneva."

Under questioning, however, Nitze departed from the administration line on the precise impact on the talks of Congress' MX decision.

"I'm not sure it would make an immediate or dramatic difference," he said.

He described Soviet leaders as "conservative" decision-makers who would realize that eventual funding of the MX was inevitable. Meanwhile, a congressional move to withhold the funds, he said, "would encourage them to increase their opposition and their propaganda."

In a related matter, the State Department announced yesterday that the Soviet Union now has 414 SS20 medium-range mobile nuclear missiles deployed within range of targets in Europe and Asia.

The number is up 36 from the 378 reported deployed when the Soviets walked out of Geneva arms talks in 1983. The report of new deployments is considered important because Belgium and the Netherlands have stated that their acceptance of new American ground-launched cruise missiles take into consideration Soviet continued deployments and the results of arms talks in Geneva.