U.S. ships and warplanes in the Mediterranean struck Libyan patrol boats and radars in a second wave of attacks yesterday and Monday while Reagan administration officials warned that any Libyan forces venturing more than 12 miles from shore may be attacked.

The U.S. Sixth Fleet has now destroyed three Libyan ships and damaged a fourth, the officials said, and has conducted two raids against a surface-to-air missile radar site at Surt. There have been no casualties or damage to U.S. forces, and the Libyan toll is unknown, they said.

There has been no Libyan military response since U.S. retaliation began Monday afternoon, but Defense Department spokesman Robert B. Sims said that any Libyan forces in the area are now viewed as hostile. Libya unsuccessfully launched at least six missiles Monday against U.S. planes flying in exercises over the Gulf of Sidra, which Libya claims but which the United States and most other nations consider international waters beyond the 12-mile territorial limit.

The Reagan administration, operating under a secret plan code-named "Prairie Fire," is prepared to conduct more extensive strikes against Libya if U.S. servicemen are killed during the current maneuvers, officials said. The plan was adopted during a March 14 meeting at which senior administration officials spoke of hitting hard at the Libyan military, according to informed sources.

President Reagan continued to draw strong bipartisan support in Congress for his tough military response to Libyan leader Muammar Qaddafi. Secretary of State George P. Shultz, while flying to Athens from Turkey with an unusual fighter escort because of heightened concerns about a Qaddafi retaliation, said other seafaring nations should applaud the U.S. actions.

The Soviet Union, however, condemned the administration as provoking a military confrontation and said the actions of the Sixth Fleet had helped "poison the atmosphere" of reduced tensions resulting from the summit meeting in Geneva last November. Details on Page A23.

The continuing U.S. action yesterday was an extension of the plan adopted by senior American policy makers on March 14, when it was decided to send three aircraft carriers for exercises in and near the Gulf of Sidra, where smaller formations of U.S. forces have exercised before. While the administration has called the maneuvers peaceful and routine, saying that U.S. planes have operated 18 times in the area since 1981, senior officials believed that Qaddafi, a longtime U.S. adversary, was likely to take action that would justify a U.S. response.

An aggressive Libyan response could elicit a U.S. bombing raid on Libyan industrial targets, according to the Prairie Fire plan. Lesser Libyan action that nonetheless resulted in U.S. casualties would elicit raids against Libyan military targets.

Any such attacks would have to receive specific White House approval. The Sixth Fleet commander, Vice Adm. Frank B. Kelso II, has authority to conduct the "proportional" attacks that have been launched so far against the radar site at Surt, which was used to guide Libyan missiles toward U.S. planes, and Libyan boats that were considered threatening to U.S. ships, officials said.

Sims said yesterday that he is familiar "with the words 'Prairie Fire,' " and he confirmed that as a matter of "prudent planning" the United States had considered responses to further actions by Qaddafi. But Sims stressed that senior officials in Washington, rather than Kelso, would decide whether contingency plans would be implemented.

"The situation now, having had hostile activity taken against us, is that obviously we're concerned about Libyan air and naval units that might be outside their territorial waters and impose a threat to us," Sims said. "I would think that almost any that are in that area are going to be judged to be a threat."

Other administration officials said that Defense Secretary Caspar W. Weinberger, while agreeing the Navy should cross what Qaddafi called the "line of death" at the entrance to the Gulf, was not among those hoping to find a way to "plaster" Qaddafi, as some senior administration officials reportedly have described the U.S. actions toward Libya. Weinberger wanted restrained responses to assert U.S. rights, the officials said; the secretary met with Kelso in London on March 17 to make sure the admiral understood his orders.

Officials also said that they believe Qaddafi's likeliest response now is to launch commando raids against U.S. targets in Europe, the Middle East or the United States. Army and Navy officials have sent messages to bases around the world urging them to be alert against terrorist attacks, officials said. Top Navy officials have been given extra security.

At a news conference before leaving Ankara, Turkey, where he stopped as part of 10-day visit to U.S. allies, Shultz was asked if the United States is trying to box in Qaddafi. "Our purpose is not to put Qaddafi in his box, but that is where he belongs," Shultz replied.

Shultz also said that U.S. action benefits everyone who has a stake in free passage through international waters. "Everybody who has that stake should be standing around applauding the United States," he said.

The United States initially filed an international notice that it would conduct its exercise until 6:59 p.m. April 1. But Weinberger said yesterday that the exercise is not likely to last that long.

One Pentagon official said the exercise would have ended as early as today had the Libyans not fired their SA5 missiles.

On Monday, after the first missiles had been fired but before any U.S. response, the commander of the Libyan air force sent a message to the U.S. Navy warning that "we are bound to destroy" the aircraft carriers "unless the aggressive acts are stopped."

But U.S. officials said that U.S. planes in the Gulf of Sidra, which intercepted Libyan planes as much as 35 times a day during past exercises, have seen virtually no evidence of Libyan air force activity since two MiG25 fighters flew toward the fleet before retreating late Monday morning.

Officials said the lack of Libyan air force activity may be partly due to the jamming of Libyan ground radars by U.S. electronic warfare planes. In modern aerial dogfighting, jet fighters are guided toward their targets by radar operators, who see the adversarial fighters as two green dots on a console and advise the friendly pilots on what course to fly to make the intercept.

U.S. jamming has increased the danger for Libyan planes seeking to take on U.S. aircraft. In addition, one official said, "Qaddafi knows that if he sends his planes against us, that we're going to bomb his airfields."

Consequently, Libyan planes generally have limited themselves to flying above their home bases, evidently as a precaution against being bombed while on the ground.

The Libyan navy also has not challenged the U.S. fleet since dawn yesterday, officials said, when a Nanuchka-class patrol boat was attacked by two U.S. A6 attack planes as it steamed past the 12-mile limit. U.S. ships, including a three-vessel group led by the cruiser USS Ticonderoga, have continued their maneuvers since then without encounters, officials said.

Officials with access to battle reports from the Sixth Fleet yesterday provided details on the most recent encounters between the Navy and the Libyan patrol boats. The cruiser USS Yorktown, which is packed with the latest electronic detection equipment, began tracking a Combattante patrol boat Monday night around 6 p.m. EST in the darkened waters of the Gulf, north of the "line of death." It was midnight in the Gulf, which is six hours ahead of Eastern Standard Time.

The Libyan boat, presumed to be armed with antiship missiles with a range of more than 30 miles, suddenly picked up speed as if it were making a firing run. The Yorktown, rather than risk letting the Libyan vessel get within killing range, fired two Harpoon antiship missiles, each armed with a 500-pound warhead.

Both missiles found their target, officials said, so thoroughly destroying the boat and its crew of 27 officers and sailors that U.S. ships found nothing to recover when they searched the area.

In the final engagement reported by the Pentagon, two A6 bombers, one from the USS Saratoga and the other from the USS Coral Sea, spotted a 900-ton Soviet-built Nanuchka missile corvette, which usually carries a crew of 70, steaming west out of Benghazi. As soon as the Nanuchka passed the 12-mile limit, the A6s dived in attack, apparently using a Harpoon missile and Rockeye cluster bombs, which spew shrapnel over a wide area.

After the attack, U.S. forces spotted the crew climbing into lifeboats and a Libyan helicopter searching the water for survivors. The ship was afire and dead in the water. Navy officials said they presumed it sank.

Asked whether the patrol boats attacked by the United States had been threatening American ships, Sims said: "Essentially when you have a ship that is capable of hitting you from 37 or 38 miles, your objective, considering it to be a threat, is not to let it get within that range . . . . You don't wait until you're within the envelope performance capability of its missiles to take care of it."

Sims also clarified yesterday the results of Monday's earlier attacks on two other patrol boats. He said the first Libyan boat attacked was a Combattante speeding east out of Misratah, and that it sank.

But the second ship, a Nanuchka, limped back to Benghazi after its hull had been pierced by shrapnel from Rockeye bombs dropped by A6 bombers. The Nanuchka's skipper positioned his stricken vessel between the attacking warplanes and a freighter outside the port. The A6 bombers broke off their attack for fear of hitting the merchantman.

After the first attack at 3:06 p.m. EST Monday on an SA5 antiaircraft missile site at Surt by two A7s carrying High-Speed Anti-Radiation Missiles (HARM), Navy leaders in the task force in the Gulf believed they had knocked out the radars. But about three hours after that attack, officials said, Navy forces detected radar signals coming from the same site.

The carriers in the gulf carry EA6B Prowler aircraft specially equipped to listen to radar signals. Weinberger said he "would assume" that Soviet technicians helped the Libyans to repair the site at Surt.

At 6:54 p.m. EST Monday, two more A7 bombers from the Saratoga launched a fresh attack against the radars at Surt, knocking them out of action for the second time with HARM missiles, which home in on radar signals. As of last night, the radars had not returned to action, Defense Department officials said.

However, officials said the Libyans had stripped the SA5 site under construction at Benghazi of radars and other equipment and might have stored them near the Surt site in an attempt to keep it operational.

Pentagon officials said four Soviet warships have been operating in the Gulf the last few days without attempting to interfere with the U.S. military operations. A Soviet Don-class submarine tender is tied up in Tripoli where it serves as a command ship and relays information from other warships in the Gulf to Libyan officials. The Soviet command ship, officials said, has been brightly lighted at night, presumably to avoid being attacked by U.S. Navy planes by mistake.