Brazil permitted four Libyan planes to fly home last week, carrying the arms that grounded them here for two months, but other disputes arose over possible violations of Brazilian territory.

Libya's planes, three Soviet-built Ilyushin jets and one U.S.-made Hercules, first flew from Recife and Manaus in northern Brazil, where they had been detained since April 16, to Rio de Janeiro. There they reloaded the 52 tons of arms that had been stored since Brazilian officials discovered the planes were not carrying medical supplies as the pilots claimed, but arms for Nicaragua.

The planes then left singly, allowing Brazil's embassy in Tripoli to report on arrival prior to the next departure--thus ensuring that the arms would not make their way to Nicaragua. Libya, acknowledging the planes' cargo, had insisted that the planes return carrying the arms.

But after extricating itself from one diplomatic embarrassment, which threatened to cut off Libyan oil supplies and contracts for arms sales, Brazil has shown some bewildering changes in attitude in two new cases that could generate new problems.

Reports surfaced during the Paris air show that British Hercules planes en route to the Falkland Islands had been making up to three landings a day at southern Brazilian airports. Brazil's foreign minister called the reports "pure fantasy," but Britain's Ministry of Defense confirmed the flights. It represented a dramatic turnaround by Brazil, which supplied Argentina with moral support as well as patrol planes during the Falklands conflict.

During the war, a stray British bomber was impounded here shortly after a Cuban plane carrying the Argentine foreign minister, and reportedly packed with Soviet-made electronic warfare equipment, was allowed to pass freely.

An opposition party delegate recently urged clarification of reports that last year other Libyan planes carrying arms to Argentina had passed freely through Recife Airport--where the latest arms cargo was revealed in April.

Brazil's attitude toward shipping has hardened suddenly, and last week the Navy minister gave orders for the sinking of vessels fishing illegally for shrimp inside a 200-mile coastal zone. An American-owned vessel was detained last week.

The Foreign Ministry has denied any shift in policy and said overflights and territorial intrusions are judged on a case-by-case basis.

In the case of the Libyan planes, Brazil alleged violation of an international convention requiring proper identification of air cargo.

Brazil's change of heart toward Britain could involve military sales. Britain has considered the Brazilian-made Tucano trainer plane as a possible replacement for its aging trainers, and the Royal Air Force clearly was pleased by Brazil's help in cutting the logistical costs of supplying the remote Falklands. Planes previously had to be refueled in flight.

Threats by Libya to cancel an order of up to 100 Tucano planes are thought to have influenced Brazil's decision to return the arms to Tripoli.