The Carter administration's new "show the flag" orders to warships have provoked a high-level flap in Libya, whose leaders feared the orders were part of an Egyptian invasion plan, U.S. officials said yesterday.

Although the administration has calmed down the Libyan leaders, more flaps loom ahead as the United States uses its warships and planes to challenge other nations' claims to waters and airspace beyond three miles from their shores.

A task force from the U.S. Sixth Fleet is to sail into the Gulf of Sidra off Libya in the next few days for exercises that will place U.S. planes and ships in waters and airspace that Libya claims.

The idea, U.S. officials said yesterday, is not to alarm Libyan leader Muammar Qaddafi by sending Navy ships and carrier planes to within three miles of the Libyan coast.

Instead, they said, the objective is to underline U.S. determination to contest claims to vast bodies of international waters such as the Gulf of Sidra, which Qaddafi asserts is Libyan territory.The gulf is so big that U.S. ships and planes are expected to stay at least 40 miles from the Libyan coast during the upcoming maneuvers.

Although the State Department insists there has been no change in policy because the United States long has recognized only three miles as territorial waters, defense officials acknowledge that there has been a change in the form of an administration decision to send military ships and planes into claimed territorial waters of about 20 countries around the world.

The idea is to challenge such claims in a deliberate but nonhostile manner over the next several months, sources said. The naval exercise off Libya in the gulf and the steaming of the nuclear carrier Nimitz into some claimed territorial waters in the Caribbean were put in this context.

Burma and Uruguay have staked claims to waters 12 and 200 miles from shore, respectively. U.S. Navy ships are expected to challenge those claims soon to strengthen the U.S. case for recognizing only three miles.

In Law of the Sea Conference negotiations, the United States has said it could accept a 12-mile territorial limit as part of an international charter.

Libya's initial alarm about the administration's "show the flag" intentions was triggered by news reports about the policy. Libya's under secretary for foreign affairs. Ali Abdel-Salen al-Turayki, was among the officials who demanded an American explanation of the policy.

In the frantic communications between the State Department and U.S. officials in the Libyan capital of Tripoli, sources said, American representatives warned that Qaddafi feared the policy was part of a secret master plan to help Egypt attack Libya.

Libyan officials were calmed down in the early hours of Aug. 11 and the following day with assurances that no close-in challenges were contemplated and that President Carter was merely re-emphasizing U.S. policy on territorial waters and airspace.

Libya claims as its own the waters south of the 32-degree, 30 minute north parallel of latitude. This claim takes in the Gulf of Sidra, where the Sixth Fleet has conducted exercises in the past and plans to do so again, despite the recent flap.

The Norfolk Ledger-Star obtained and quoted a fleet message sent by Adm. Harry Train, the Atlantic commander, explaining how the administration's challenge to what it considered excessive territorial claims would be implemented.

"In the future there will be planned exercises, transits and overfights by naval and air forces for the purposes of asserting U.S. rights in the face of excessive claims. . .

"In certain instances, we must consider going out of our way to contest a claim. We must clearly avoid any irrational disposition of forces, but must ensure we are seen as unequivocably exercising our rights. . ."