The Soviets might be able to destroy MX missiles deployed closely together in the proposed Dense Pack formation by sowing the field with nuclear mines timed to explode all at once, a scientific panel has warned the Pentagon.

A big Soviet rocket, the panel said, could carry a load of individual warheads aloft and launch them toward the Dense Pack field. The warheads would manuever after being released to dodge defenses and then bury themselves in the earth around MX missile silos.

Defense Department officials, while conceding that this looms as the biggest single threat to the Dense Pack, said such a Soviet weapon is so many years away that it still would make sense to deploy MX missiles closely together at least into the 1990s.

"I'm optimistic," said one Pentagon executive last week, when asked how Dense Pack looks now that its flaws have been examined extensively by a panel of scientists under the chairmanship of William A. Nierenberg, a physicist who is director of the Scripps Institution of Oceanography.

Nierenberg, in a separate interview, said the mission of his group was not to make a go-or-no-go decision on Dense Pack, but to report what the Soviets might do to defeat it.

Defense Secretary Caspar W. Weinberger is scheduled to tell Congress by Dec. 1 how President Reagan intends to deploy the MX, for years a missile without a home. After rejecting President Carter's proposal to spread MX missiles around the valleys of Nevada and Utah, Weinberger has gone through a series of alternatives, none of them foolproof. Dense Pack at the moment looks to Pentagon leaders like the least rotten apple in the barrel.

Air Force leaders said they are more optimistic than ever that Dense Pack will be approved by Weinberger, partly because the CIA has said it would take several years for the Soviets to perfect the kind of earth-penetrating warheads described by the Nierenberg group.

Other flaws in Dense Pack include the possibility that the Soviets would pin down MX missiles bunched together by exploding one warhead after another over them. Such explosions might spread a curtain of such lethal X-rays over the MX fields that the missiles could not fly through it without being damaged fatally.

One Air Force counter to pin down, officials said, would be to time the launches of MX missiles so they would fly through the gaps between Soviet explosions overhead.

Then there is the same question hanging over Dense Pack that hung over Carter's MX proposal to rotate missiles among concrete garages in a giant shell game: will the Soviets simply build enough warheads to blow up every MX wherever it is based?

The Air Force's answer to that question is that the Soviets would have to fire more than one warhead at each Dense Pack missile silo to destroy the MX lodged there under tons of concrete. And in ganging up on these tightly packed silos the Soviet warheads would knock each other out of action in nuclear fratricide.

To make sure it would take more than one Soviet warhead to destroy one MX in a Dense Pack silo, Air Force specialists have been scouring the country to find earth that would support heavy fortification. The ground in five different states has been adjudged suitable, although the locations have not been decided upon yet, officials said. Sites in Nevada, southwest Texas and Wyoming look especially promising, they said. Each site would be about 15 square miles.

The Pentagon will try to build MX silos that will absorb shock rather than strive for fortifications that withstand pressure. "Like the stuff in a packing box," said one executive in discussing the proposed packaging of MX missiles.

The Pentagon's blueprint for Dense Pack calls for starting with 100 missiles in 100 silos at five locations. Dense Pack would not start out with an anti-ballistic missile defense under the current plan, but one could be added later.

The goal is to start deploying MX somewhere in 1986. Cost estimates are constantly changing as one government office after another tinkers with the Dense Pack deployment scheme to the point there is not a good figure at the moment, officials said.

Weapons leaders at the Pentagon said their idea is to start out modestly with Dense Pack and then see how the Soviets react to it before adding more elaborate and costly features, such as an ABM defense.

One defense executive in the heart of the deliberations, when asked about Soviet counters to Dense Pack, said, "No matter what they do, we can stay ahead of them."

Pentagon officials said the Soviets cannot concentrate on Dense Pack without lowering their defenses against U.S. submarine missiles, cruise missiles and bombers.

Therefore, backers of Dense Pack contended, the Soviets would be more deterred from attacking the United States if MX missiles were pointing at them than they are today, when American Minuteman missiles are vulnerable to a surprise strike.