Two U.S. Navy F14 jet fighters yesterday shot down two Soviet-built Libyan Air Force jets after the Libyan planes made what the Pentagon called an "unprovoked attack" on the U.S. jets over international waters some 60 miles off the Libyan coast.

The aerial duel, fought at 20,000 feet in the early daylight hours over the Gulf of Sidra in the Mediterranean Sea, was easily the most serious incident yet in the deteriorating relations between the Reagan administration and the revolutionary regime of Libyan ruler Col. Muammar Qaddafi. The incident produced an American warning to Qaddafi that "any future attack against U.S. forces operating in international waters and airspace will also be resisted with force if necessary."

The clash began and ended quickly; the dogfight lasted only one minute as U.S. pilots shot back under standing orders to fire if fired upon. It was still night in this country and President Reagan, vacationing in California, was not awakened by aides to be told of the incident until some six hours after it was over.

The F14s were patrolling in the skies above a 16-ship U.S. Navy task force that was carrying out a "routine" exercise centered more than 100 miles off-shore.

In 1973, Libya unilaterally declared sovereignty over the waters and airspace of the gulf, extending more than 100 miles into the Mediterranean. The United States has never accepted that claim, recognizing instead the legally accepted international standard of three miles and informally observing a 12-mile limit claimed by Libya for other coastal regions.

In the last seven years, the U.S. fleet has operated four times in the gulf, south of a line between the coastal cities of Benghazi and Misratah that Libya considers the gulf's entrance. The last time was in July, 1979.

The current exercise marks the first time the Reagan administration has sailed into those waters and it may be part of the stepped-up, get-tough-with-Qaddafi campaign by the new administration that included expelling Libyan diplomats from this country last May and efforts to counter -- through covert as well as overt and diplomatic means --Libyan military adventures in Africa.

At a hastily called press conference early yesterday morning, Defense Secretary Caspar W. Weinberger was asked if the exercise was meant as a challenge to Qaddafi. "I wouldn't describe it that way," the secretary answered. "We regard these as international waters. We've had naval and air exercises there before. This one was scheduled for some time and the notification went out in the perfectly normal fashion to airmen and mariners."

Privately, senior officials said the United States did not go into the gulf looking for a fight but also was determined not to be denied legal rights by Qaddafi. "We have long been bugged by Qaddafi's territorial claim," one official said, "and this exercise was meant to test it and show we don't go along."

One senior military officer said the region is always dangerous and U.S. pilots always go into that area "all wired up" to fight if necessary. Another officer pointed out that in 1979, the U.S. embassy was burned down in Tripoli and the Carter administration did nothing about it, implying that the new administration would no longer turn the other cheek.

It was also learned that the commander of the Navy task force was called back to Washington before the exercise by the Joint Chiefs of Staff to make sure that all operational rules, including the rules of engagement in which fire is returned, were understood. High-level sources said the briefing on the exercise extended to the National Security Council, and the Navy officers were told that, given the existing strains in U.S.-Libyan relations, they had better "be on their toes" even more than normally.

These sources insist, however, that the exercise was not meant as a provocation or to draw the Libyans into a fight but rather to establish freedom of navigation and movement in that region.

The exercise began in the pre-morning hours Tuesday Libyan time. Pentagon officials say that in the exercise's first day Libyan jets from several bases made about 40 flights out toward the task force but were routinely intercepted by Navy jets and turned back without incident.

But about 7 a.m. Libyan time yesterday, two of the F14s on patrol picked up on their cockpit radars two more Libyan jets about 30 to 40 miles away and heading toward the fleet. The F14 pilots spotted the SU22 jets visually about five to six miles away. Rather than turn away, however, one Libyan jet fired a Soviet-made Atoll air-to-air missile at the F14s while the other Libyan plane appeared to be moving into position to fire, according to Air Force Lt. Gen. Philip J. Gast, director of operations for the Joint Chiefs of Staff.

Sources say that three of the four Navy fliers in the two F14s saw the missile being fired, although there was no immediate indication yesterday as to whether the Navy had recorded the episode on gun cameras or radar recordings.

The Atoll missile missed and each F14 then fired a heat-seeking Sidewinder AIM9L missile, bringing down both of the Libyan warplanes at 7:20 a.m. local time (1:20 a.m. EDT), the Pentagon said. The attack took place 60 miles from the nearest land, according to Pentagon officials. One parachute was sighted, and a Libyan helicopter and surface ship were seen moving toward the area where the downed pilot was heading. The Pentagon said it assumed the pilots were Libyan but would not comment about other nationalities flying other Libyan planes.

Gast said that when U.S. planes take hostile fire, "the aircraft flight commander has the authority to defend himself. In this case, that's exactly what he did." Weinberger added that the F14 pilots had "carried out their instructions extremely well."

A Libyan government spokesman confirmed that Libyan planes had been fired upon but said eight U.S. jets were involved and did not mention any Libyan craft's having been shot down. The spokesman said the Libyan planes were carrying out "routine reconnaissance and inspection flights" over Libyan territorial waters and that the U.S. attack "endangered world peace." The Libyan news agency also reported one F14 had been shot down, but the Pentagon denied this and said the F14s and their crews had returned safely.

Gast, who was on duty overnight at the Pentagon command center, said word of the clash reached him in six minutes and he immediately notified Weinberger and Gen. David Jones, chairman of the Joint Chiefs. Top White House aides with President Reagan in California and Secretary of State Alexander M. Haig Jr. were also immediately informed.

The Libyans were flying what are basically export versions of the Soviet Air Force SU22 jets, which contain little of the sophisticated electronic equipment now used by Soviet forces. The swing-wing warplanes are more suitable for bombing or reconnaissance missions than dogfighting and are considered no match for the high-performance, 1,400-mile-per-hour F14, which is the most modern fighter in the U.S. fleet. The F14 carries a two-man crew specifically so that one officer can aim and fire the missile while the other flies the plane.

U.S. officials said the basic decision to hold the naval exercise off Libya was made by Reagan at a National Security Council meeting late in July. These officials also said there was considerable discussion before that within the Pentagon about a possible postponement until after the summer. One reason is that there are several hundred additional American dependents in Libya during the school vacation period. There are about 2,500 Americans who work year-round in Libya, mostly for the big U.S. oil companies.

Libya is the third largest supplier of foreign oil to this country and continues to do about $12 billion annually in U.S. business even though the administration regards Qaddafi as a supporter of international terrorism, surrogate of the Soviet Union and advocate of instability throughout northern Africa.

The U.S. put out a notice to mariners and airmen on Aug. 12 and 14 that the exercise would take place in a large region of the south-central Mediterranean on Aug. 18 and 19. The exercise ended at 1 p.m. Washington time yesterday.

The naval maneuvers involved two aircraft carriers and live target practice in which surface-to-air and air-to-air missiles are fired at special target drones. The Navy says it uses the area near Libya because it is out of the main commercial sea and air traffic lanes in the Mediterranean, although statistics show that the region has not been used very frequently by the Navy.

Most of the 16-vessel task force was operating outside the area bounded by latitude 32 degrees-30 minutes north on the map, which roughly conforms to the opening of the Gulf of Sidra where Libya claims its territorial waters start. The carriers, Pentagon officials say, were well to the north, about 100 miles away from the gulf. But two ships, one of them a safety vessel to guard against test missiles going astray, were below the 32-30 line and the dogfight with the U.S. and Libyan planes also took place 30 miles below that line, inside the gulf.

The shooting episode was the first such encounter since March, 1973, when two Libyan air force planes fired on an unarmed U.S. C130 transport about 83 miles off the coast. The U.S. plane escaped unharmed.

Last September, Navy jets chased Libyan warplanes away from an Air Force EC135 electronic intelligence-gathering plane, and the next month there were suspicions that the Libyans had actually fired on the same kind of plane, although actual evidence was never developed.

A statement issued by the Libyan government on Tuesday condemning the U.S. exercise as an "uncalled-for interference and provocation" also contained a warning that Libya "reserves the right to take all measures. . . to safeguard its airspace and territorial waters."

The warning to Libya not to attack again came in a formal protest to the Tripoli government that was conveyed through the Belgian government. The United States and Libya do not have embassies in each other's countries. The United States took all but the final step toward a complete and formal break in diplomatic relations last May when it ousted Libya's representatives in Washington.

Several Republican members of the Senate Armed Services Committee, including Chairman John Tower of Texas, Strom Thurmond of South Carolina, William Cohen of Maine and Warren Rudman of New Hampshire, also condemned the "unprovoked" Libyan attack yesterday and said the U.S. response was "thoroughly justified," as Tower put it.

Sen. Larry Pressler (R-S.D.) said the exercise and the presence of U.S. aircraft over the gulf may have "been at least testing what Qaddafi would do. I just wonder if we gained enough strategic benefit. I don't know."

Sen. J. Bennett Johnston (D-La.) said, "Frankly, I think we sent a fleet in there to provoke an incident and provoke an incident we did. And we, in effect, won. I think that's very proper . . . and if we'd just draw the line with some of the dictators we'll cut down their influence in the world."

Sen. Charles McC. Mathias (R-Md.) and Sen. Carl Levin (D-Mich.), both of whom attended a briefing on the incident, said the action may have just been caused by "the impulsive action of an overzealous pilot in a Libyan plane," as Levin described it.