The United States should consider putting the new MX missile in small submarines rather than on land where they could accelerate the arms race, a Pentagon consultant told the House Armed Services Committee yesterday.

Sidney D. Drell, Stanford physics professor who headed a Pentagon study last year on how to keep landbased ICBMs from becoming vulnerable, warned that the Air Force idea of playing a shell game with MX missiles could create as many problems as it solved.

First of all, Drell told the committee, it would take a land area as big as New Jersey to accommodate some 5,000 missile holes which would be spaced about a mile apart. Under the Air Force concept, about 200 missiles would be moved secretly from one hole to another to keep Soviet gunners guessing.

As Air Force advocates described the system, the Soviets would have to fire missiles at each of the 5,000 missile holes to be sure of destroying the "live" one. This, they contend, would use up Soviet warheads to such an extent that the Kremlin would consider a surprise attack a losing proposition.

But Drell said yesterday that the Soviets might elect to keep building warheads to overcome any number of missile holes that the United States dug. This acceleration of the arms race, he said, could be headed off only if both the United States and Soviet Union agree to limit the total number of warheads deployed.

Currently, the Pentagon is looking beyond the "shell game" scheme and considering MX airplanes which would take off at the first sign of a Soviet attack. Drell contended this concept is flawed, too, because survivability of the MX missiles would depend on getting enough warning. Also, he said the planes could stay aloft only a limited time.

An MX development which makes sense but is being overlooked, Drell maintained, is putting the new missiles on conventionaly powered submarines which would stay within 200 miles of American shores.

Drell told the committee that a fleet of 50 inexpensive coastal submarines, which would be automated like the Skylab space vehicle so a small crew could operate them, would steal quietly along the East and West coasts at about five knots. Each submarine under Drell's concept, which was considered in the 1960s but discarded in favor of bigger submarines, would carry two MX missiles strapped to the outside of the hull.

Keep developing the MX missile, Drell advised the committee, but consider putting the MX in airplanes only as a stopgap measure until the feasibility of a new fleet of missile submarines is fully explored.

Calling the MX submarine SUM, for shallow underwater mobile basing concept, Drell said it would have 500,000 square miles of coastal waters to hide in; the noise from other ships would mask the sound of the sub moving slowly along; communications and navigation would be simple, even to placing markers on the ocean bottom, and the Soviets would have a difficult time destroying a submarine even if they found it.

Drell's submarine plan did not seem to excite many members of the committee yesterday. Rep. Richard H. Ichord (D-Mo.), chairman of the research and development subcommittee, estimated it would take 10 years to bring the new sub into being. He complained the Pentagon has already done too much flitting from one project to another.

Rep. Claude (Buddy) Leach (D-La.) expressed concern that putting MX missiles in coastal submarines would amount to putting too many of the nation's new nuclear eggs in one basket. The U.S. "triad" strategy for nuclear warheads calls for spreading them among submarines, bombers and land missiles.

The committee called the hearing to receive ideas on how to make the current force of 1,054 land-based missiles less vulnerable to Soviet attack. Pentagon leaders have said that Soviet missiles are becoming so accurate that they would be able to destory a large portion of the present land-based force in the 1980s.