The Reagan administration plans to resume its war of nerves against Libyan leader Muammar Qaddafi by sending the aircraft carriers USS Coral Sea and USS Saratoga back to the Gulf of Sidra area next week, administration officials said yesterday.

In his State of the Union address Tuesday night, President Reagan may spotlight that he is turning up the heat by declaring the United States has no intention of recognizing Qaddafi's claim that the entire gulf falls within Libyan waters.

Such language was put in at least one draft of the president's speech, officials said. But no matter whether Reagan mentions it in his speech, the policy decision to send the carriers closer to Libya has been made, the officials said, setting the stage for some highly visible gunboat diplomacy.

The carriers were pulled away from the gulf region late last week after a week of operations north of the Libyan coast. The decision to return the two carrier battle groups this quickly signifies the conviction of high administration officials that such pressure is making Qaddafi less willing to support terrorism and represents the only practical military option Reagan now has, informed sources said.

They added that if Qaddafi should take some rash action, such as shooting at U.S. ships or planes in the Gulf of Sidra, the two battle groups would give the Reagan administration enormous firepower to retaliate quickly. The battle groups could bomb or shell any of dozens of targets in Libya that the Pentagon has been studying more than a month, they said, while the Soviet warships now in the area would be hopelessly outgunned.

The Coral Sea and Saratoga will be ordered to leave liberty ports in the Mediterranean soon so they can be back near the Gulf of Sidra early next week, officials said.

Qaddafi has threatened to shoot down any planes or sink any ships that cross "the line of death" he has drawn across the top of the elbow-shaped gulf. The United States recognizes Libyan territorial limits as extending only 12 miles from shore. The mouth of the gulf is about 110 miles farther away.

In the week-long exercises just concluded, neither the ships nor aircraft of the two carrier battle groups crossed that line. This may be the case in the second set of exercises off Libya, officials said yesterday, with one declaring that the crossing of the line will not come until late March when even more naval force will be in the region.

"Sooner or later we're going to have to cross it," one official said.

"The line of death" Qaddafi has drawn is at 32 degrees 30 minutes north latitude. Libyan ships were deployed behind that line during the weeklong operations, intelligence officials said, but there were no confrontations with U.S. forces.

Defense Secretary Caspar W. Weinberger pointedly said last week that the United States does not recognize Qaddafi's claim to the whole Gulf of Sidra, adding that U.S. warships will return there to operate in the international part of the waterway.

In contrast to their past splits over taking military action to combat terrorism, Weinberger, who has been a restraining influence in the past, and Secretary of State George P. Shultz, who has been pressing for military action, are allied in sending warships off Libya to keep Qaddafi nervous and give him second thoughts about supporting terrorists.

Weinberger has been saying that the combination of economic and military pressure has already influenced Qaddafi's behavior for the better.

The Joint Chiefs of Staff, officials said, after advising Reagan against bombing dozens of targets they were asked to review, also favor keeping the heat on Qaddafi by sending back the battle groups, Pentagon officials said. The chiefs, they said, could not find targets in Libya that were directly linked to the terrorist attacks on the Rome and Vienna airports on Dec. 27, impelling them to recommend against a retaliatory bombing strike because the "tit-for-tat" relationship was missing.

While contending the display of military might off Libya has political and diplomatic payoff, officials familiar with the administration's game plan for Qaddafi acknowledged there were military risks as well as gains in conducting such intensive gunboat diplomacy. One risk is that Qaddafi will fire some of the Soviet-made SA5 antiaircraft missiles that have become operational at the coastal town of Sirte. SA5s are capable of hitting a plane 100 miles away at high altitude. Libya has a few working SA5 missiles at Sirte, officials said, but only launchers for them at Benghazi. The apparent objective is to cover the entire Gulf of Sidra.

Last week the intelligence assessment was that Qaddafi has control over the firing of the SA5s at Sirte, but there is still argument on this crucial point.