The new Reagan team at the Pentagon has decided to move ahead with what amounts to a floating antiballistic-missile system to protect Navy ships against Soviet cruise missiles.

If the plan goes all the way, American skippers would be armed with sophisticated nuclear weapons for sea warfare. At least one advocate acknowledges that this could take the Navy -- and the country -- into dangerous, uncharted waters.

A ship's commander attacked by Soviet cruise missiles could not tell from his radar scope whether they were tipped with nuclear or conventional warheads. If they were nukes, and they hit close, his ship would be sunk.

"When you think the problem through, you see a real dilemma," said former Pentagon research director William J. Perry, who still occupies his old office, but with the title of consultant to Defense Secretary Caspar W. Weinberger.

"You see a Backfire bomber attack coming at you and a number of AS4 cruise missiles coming at you," Perry said. "And you'd really like to use that nuclear-tipped missile to defend yourself, because the nuclear defending missile will win.

"On the other hand," Perry continued, "when the AS4 cruise missile is coming at you, you don't know whether it is nuclear-tipped. We have to be very careful about the release conditions" governing nukes on Navy ships -- "how we would decide when to use them."

If a Navy skipper fired off his new nuclear-tipped missiles in the mistaken belief that he was being attacked with nuclear cruise missiles from Soviet Backfire bombers, Oscar attack submarines or surface ships, the United States would most likely lose out, even if the war was confined to sea, Perry said.

Today, he said, the advantage lies with the offense in a nuclear battle at sea. The Soviets could overwhelm Navy defenses with barrages of cruise missiles and sink much of the fleet, the weapons specialist warned.

He contended that even if the United States managed to sink the Soviet navy while losing its own, the United States would come off second best because it depends more heavily on its superior fleet than does the Soviet Union.

After participating in secret Pentagon debates, which took up much of last year, Perry said he concluded that the gains of arming ships with an advanced nuclear missile outweighed the risks. "It was a close call," he said.

But in a stutter step, former defense secretary Harold Brown left it up to his successors at the Pentagon to determine how a new nuclear defense missile should be handled. Navy officials received approval from Reagan's team at the Pentagon to reequest money for the weapon in the fiscal 1981 supplemental budget request going to Congress late this month.

The money, described as "only a few million," is earmarked to develop, but not produce, a nuclear warhead for the Standard missile already going on Navy ships. But the Standard being deployed today has a conventional warhead, not nuclear. The new one is designated as the SM2N, with "N" standing for nuclear.

Some Pentagon backers of the SM2N contend that it would be fired at drone-like Soviet cruise missiles, not ballistic ones covered in the U.S.-Soviet antiballistic-missile treaty of 1972. But this is not a closed case.