The second of the nation's new Trident missile submarines, the USS Michigan, formally joined the fleet today in a commissioning ceremony that dramatized the choices President Reagan confronts as he tries to modernize the nuclear arsenal.

Some weapon specialists, predicting that there will be no politically acceptable way to make a new land-based missile invulnerable, contend that the time has come to put America's nuclear firepower out to sea. The Trident's new D5 missile, now under development, is advertised as being just as destructive as the proposed land-based MX system and less vulnerable to attack.

But, like the MX proposal, the Trident program has drawn protests. Today, about 300 demonstrators outside the gates of the Electric Boat Co., which is building the fleet of Trident submarines for the Navy, declared in songs and placards that the United States and Soviet Union already have deployed too many nuclear weapons and should declare a freeze and then cut back.

"Trident/Typhoon, ships of doom" proclaimed one poster held up as Navy admirals, Sen. Carl Levin (D-Mich.) and other dignitaries filed into the shipyard for the commissioning. The Typhoon is the Soviet equivalent of the Trident submarine.

At one point, demonstrators ripped open a bag of ashes, and police said 11 were eventually arrested.

Adm. Kinnaird R. McKee, sounding the day's theme from the speakers' platform on the 560-foot- long Michigan, said, "Deterrence works and we are at peace."

McKee, who replaced retired Adm. Hyman G. Rickover as head of the Navy's nuclear propulsion program, declared the $1.5 billion submarine "essentially invulnerable." Submarines with nuclear-tipped missiles (the Michigan carries 24) help make nuclear war look suicidal to the Soviets, he said.

The protest that ruffled today's commissioning ceremony points up one problem with Reagan's plans for closing "the window of vulnerability" by deploying new missiles on land in the United States and Europe. Public protests prompted Reagan to look for an alternative to predecessor Jimmy Carter's proposal to deploy MX missiles in the valleys of Nevada and Utah.

The latest deployment scheme, called "Dense Pack," would put MX missiles so close together that incoming Soviet warheads could not knock them all out without destroying each other in the process. An antiballistic missile defense might be added for further protection.

But experts have found flaws in Dense Pack, and the Air Force is expected to brief Defense Secretary Caspar W. Weinberger this week on ways to correct them.

One such flaw, according to some experts, is that the Soviet Union would be tempted to explode one warhead after another over a Dense Pack field. This "pin-down" strike would spread such a lethal curtain of X-rays across the sky that MX missiles could not be fired through it without being destroyed.

Another flaw arms specialists see in Dense Pack is the public fear that would be aroused by explaining that MX missiles would be saved in part because Soviet warheads exploding overhead would destroy each other. Such "fratricide" explosions, while perhaps keeping the MX missiles from being destroyed, would incinerate much of the surrounding land.

Dr. Sidney Drell, a nuclear physicist at Stanford University who has been a frequent consultant to the government on arms control and weaponry, said in an interview that he believes Dense Pack is a loser.

He said if the president insists on deploying a highly accurate, first-strike weapon like the MX, he would be better served by leaving the job to Trident submarines like the Michigan. "As far as accuracy and command-and-control," Drell said, "the Trident with the D5 missile should have it all. It can do everything MX can do, and better."

The Navy has talked about building 20 Trident subs at the rate of one to two each year, although construction could be stepped up. The Trident is replacing the 10 oldest Polaris submarines, which have been retired. But because of the Trident's slow production pace and other delays, the United States faces an under-the-waves missile gap in this decade.

Drell, who philosophically opposes deploying first-strike weapons, said that on technical grounds it would make more sense to forget about the MX and put any new silo-busters in Trident submarines.