If a nuclear war with the Soviet Union ever started and an American president were trying to stop it, how would he communicate with Soviet leaders whose headquarters and communications systems may have been destroyed by rapidly counter-attacking American missiles?

"There's a real dilemma here that we haven't sorted out," according to retired Air Force Lt. Gen. Brent Scowcroft, the former top national security assistant to President Ford.

The conflict is essentially between an emerging American military strategy that puts heavy emphasis upon knocking out what Reagan White House adviser Richard Pipes calls "the nerve system of the Soviet leadership" and what Scowcroft points out is a presidential need to "commmunicate with the enemy" after an attack has begun.

The dilemma, in Scowcroft's words, is that "the kinds of controlled nuclear options to which we are moving presume communications with the Soviet Union; and yet, from a military point of view, one of the most efficient kinds of attack is against leadership and command and control systems. It's much easier than trying to take out each and every bit of the enemy's offensive forces."

The remarks of Scowcroft, Pipes and many other top military and civilian officials are contained in a just-released record of a symposium on strategic nuclear policies, weapons and command and control held last October at Bedford, Mass. The meeting was sponsored by the U.S. Air Force Electronic Systems Division and the Mitre Corp., an Air Force think tank.

The proceedings contain some of the most candid discussions ever made public on the problems of trying to maintain control over nuclear forces in an actual wartime situation, including detailed explanations of how a president would communicate with submerged U.S. nuclear missile-firing submarines.

The symposium was held just two weeks after President Reagan announced a six-year, $180 billion program to modernize U.S. strategic forces with new missiles and bombers. That program includes an $18 billion investment to try to improve presidential control over those weapons, to help the president communicate with those forces and to give him the information he needs in the midst of the chaos of an atomic war.

The president was praised by participants for being the first chief executive to put substantial funds into this unglamorous sector of the defense budget. Defense officials have long held that an improved ability to survive an attack and maintain communications would contribute to deterring Soviet attack.

At the same time, the U.S. is moving toward a strategy aimed at convincing the Soviets that they could not win a war because this country would retaliate against the Soviet command structure.

"Our present deterrent is geared toward attacking military objectives, command and control, and the nerve system of the Soviet leadership--that whole system that confronts us," Pipes said. "In Soviet terms that is far more menacing and therefore it is a great inducement, first of all not to strike, and secondly to come to the negotiating table, than the previous strategy had been."

Pipes added that "I personally do not believe that the Soviet Union would launch a preemptive nuclear strike against the United States out of the blue, no matter what the balance of power is. I just don't think that is in the cards. The great danger is that, in a world crisis in which there is an escalation of hostility between our two countries and where both begin to alert the forces and so on--where indeed war between us becomes possible--at that particular point they might strike preemptively."

Scowcroft said that even if a nuclear attack were to come in the midst of a crisis, it would still come as a surprise because of the magnitude of such a step, and the president "may have less than 10 minutes in which to make a decision and seek shelter."

Scowcroft, who remains intimately involved in strategic matters as a member of special commissions looking into new American weaponry, said the Defense Department requires confirmation that an attack is under way by more than one means of detecting that missiles have been launched, such as ground-based radar and infrared sensors on satellites.

The president then has to convince himself that missiles are actually en route. "Unless the Soviets are a lot dumber than I think they are, they will take that into account in planning their attack . . . and do their best to make the first indications as ambiguous as possible and thus force the president to take as much time as possible and thus lessen the chances we would respond quickly," he said.

Getting the president the information he needs is a difficult task, Scowcroft said, and he will need to consult advisors who may not be available. If the President stays in Washington and would-be successors such as the vice president are dispersed to secret locations with their advisors, this would deplete the experts available to the president here and raises the question of whether military command centers would know who the president or top civilian authority is at any given time after an attack begins.

When the missiles actually have begun to land, Scowcroft said, the U.S. would be in what he calls "the automatic phase of the war . . . at which time the quick response (retaliatory) systems are discharged against predetermined targets and the battle plan unfolds more or less automatically." In other words, the initial presidential determination that the U.S. was under attack and his decision to respond would trigger a quick and automatic American response of some predetermined size.

The next phase would be trying to manage the conflict or end it, and for that the president would need to communicate with the remaining forces, the American people and the Soviets, Scowcroft said. That's where the new strategy of striking the Soviet leadership, as they might strike ours, bumps into the problem of ending the war.

Rear Adm. Paul D. Tomb, a member of the staff of the military Joint Chiefs of Staff, told the symposium that "unless all elements of our nuclear triad (land- and submarine-based missiles and bombers) receive the message prior to impact of Soviet reentry vehicles (missile warheads) we risk losing large segments of our retaliatory force."

One big question has always been whether messages would get to U.S. missile-firing submarines submerged in distant waters.

But Tomb describes several means to do this. Extremely low frequency (ELF) transmitters based on land, he said, are not very good message carriers because the information moves slowly. But they are "very good bell ringers" to submarine commanders because they send out deep into the ocean a continuous signal that is part of the Joint Alerting Network. If that stops it means that the transmitter may have been destroyed and signals the commander to try another method to receive information.

The next most reliable system would be picking up very low frequency (VLF) transmissions from Navy airplanes or other transmitters using three different kinds of other antennas carried on submarines, including wire devices and special buoys.

If that doesn't work, Tombs said, a submarine could come close to the surface, pop another antenna just above the water and communicate via a Navy satellite that is part of the Submarine Systems Intelligence Exchange System. The commander could also come close to the surface and, with a radio antenna, monitor war messages being broadcast to surface ships on high frequency.

The symposium also was told by Navy officers that the extent of the Reagan administration's plan to put thousands of new cruise missiles on submarines, especially those to be aimed against land rather shipping targets, came as somewhat of a surprise.

One reason, according to a symposium commentator who was not identified in the published account, was that the cruise missiles--small torpedo-shaped flying bombs powered by jet engines--provided an opportunity for the U.S. quickly to add atomic punch "since we are seriously behind the Soviet Union."

"The second reason was that the missiles we will deploy in Europe," a reference to new ground-based cruise and Pershing II missiles, "are vulnerable to surprise attack. Indeed, their presence there, along with pre-positioned material, the nuclear storage and our dependence on runways and other attributes of mobilization, mean that to some degree we are making Europe into a gigantic Pearl Harbor."

The commentator said the U.S. would try to discourage the Soviets from striking at the land-based missiles and thus is adding the sea-based force as "a good way" to convince Moscow that the cruise missile threat cannot be easily removed.