The Reagan administration yesterday ordered two aircraft carrier battle groups in the Mediterranean to begin flight operations north of Libya, Defense Department officials said.

The warplane operations, scheduled to begin from the carriers USS Saratoga and USS Coral Sea last night, were described by one administration official yesterday as "part of the war of nerves" between the Reagan administration and the Libyan leader, Col. Muammar Qaddafi. The Navy planes will be within range of Libyan radar as they fly training exercises but are under orders not to cross into Libyan airspace, the official said.

The carrier operations are the latest in a series of maneuvers by the administration in an attempt to show resolve against Qaddafi, who has been repeatedly accused of supporting international terrorism. Earlier this month, Reagan announced further economic sanctions against Libya.

At the same time, in White House meetings Jan. 6 and 7, Reagan also ordered that more money and manpower be devoted to the development of a CIA covert operation against Qaddafi and that an envoy be sent to Egypt for further discussions about coordinating possible military options, sources said.

Although there have been discussions within the administration recently about ambitious anti-Libyan military options, the sources said that at this point joint action by the United States and its Middle East allies would be undertaken only if Qaddafi attacks a neighboring nation or is found to be responsible for terrorist actions similar to the Dec. 27 attacks on the Rome and Vienna airports.

Some administration officials want to encourage Egypt to be more aggressive in confronting Libya, the sources said. These officials believe Egypt has been too reluctant to request U.S. assistance in any potential anti-Qaddafi moves, said the sources, who spoke on condition that they not be identified.

Anti-Qaddafi feeling runs high in the Egyptian Defense Ministry, where the special envoy was expected to hold his discussions. But other officials in Cairo are reluctant to demonstrate any military alliance with the United States against another Arab nation because of the potential political repercussions in Egypt, the sources added.

A Pentagon team began initial military planning discussions in Egypt late last summer because of administration concerns about possible military and terrorist moves by Qaddafi in the region. The planning began following the hijacking of Trans World Airlines Flight 847 in June in which one American was killed and 39 others held hostage for 17 days.

Sources said that under a plan approved by the president last year, the CIA is working hard to develop a blueprint for undermining Qaddafi, but has been hampered by the absence of a large, well-organized and committed group of opposition forces either inside or outside the country.

One source spoke of the need for some "Qaddafi contras," a reference to the large, U.S.-backed rebel group trying to overthrow the Sandinista regime in Nicaragua. Within administration intelligence circles there is growing skepticism that the CIA's anti-Qaddafi plan will work because of the absence of opposition forces, due in part to Qaddafi's ruthless campaign to kill opponents anywhere in the world. There has been one estimate that it will take up to a year to get any CIA operation off the ground.

At the same time, the CIA wants to identify and cultivate potential successors to Qaddafi who are pro-Western. This problem is worsened by the strong anti-American sentiments prevailing throughout much of Libyan society, according to intelligence estimates.

"There are people in Libya, especially in the military, who don't like Qaddafi," one source said, "but most hate the United States."

Although Qaddafi claims the entire Gulf of Sidra and its airspace as Libyan territory, the United States recognizes territorial waters extending only 12 miles from the Libyan coast. Initially, U.S. planes are expected to begin flying north of the gulf but eventually work their way south within a week, the official indicated. The warships are authorized to sail in the Gulf of Sidra to reassert U.S. rights in the region.

The first indication of the exercises came yesterday when the Pentagon confirmed that the Navy had issued a "Notice of Intent to Conduct Flight Operations" through the International Civil Aeronautical Organization.

That document, which is not classified, says carrier flight operations will be conducted for a week within the Tripoli Flight Information Region, a sector of airspace extending scores of miles from Libya. The notice said all operations would be conducted in international airspace with aircraft operating either under visual flight rules "or within radar surveillance and radio communications of a surface or airborne radar facility."

According to one report, the last time Navy jets conducted operations within the Tripoli flight region was on Jan. 27 and 28, 1985. Qaddafi claimed at the time that the United States was planning to invade his country and cited a similar Notice of Intent.

White House officials denied reports that the current exercise was intended to provoke Qaddafi. In the past month, following the European airport attacks and accusations of Libyan complicity, activity in the Mediterranean has become more intense as Libyan forces were put on alert, the Soviet Union increased its surveillance of the U.S. Sixth Fleet and the United States increased its presence to more than two dozen Navy ships.

Two Libyan Mig 25 fighters flew next to a Navy EA6B electronic jamming plane outside the Gulf of Sidra last week but took no action. The incident was played down as routine by Defense Secretary Caspar W. Weinberger.

On Aug. 19, 1981, Navy fighters shot down two Libyan fighters above the Gulf of Sidra after the Libyans allegedly fired first.