One year after President Reagan warned Americans to leave Libya and prohibited them from using their passports to travel to the radical North African country, U.S. citizens have been returning in a small but steady stream.

Western diplomats estimate that there are 500 to 600 Americans in the oil-rich country led by Col. Muammar Qaddafi, who regards the United States as his primary enemy. That number is far below the estimated 2,500 here before Reagan's executive order last Dec. 10 caused the exodus of all but a handful of Americans. The order was issued to protect Americans from any hostile act by Libyans or the Libyan government.

In Washington, a State Department spokeswoman said Thursday that the executive order has been extended for another year.

Most of the Americans are believed to be working in the oil industry in the Libyan desert south of the capital. They pass quickly through the Tripoli airport entering or leaving the country under contracts that usually call for them to work 30 days and then have 30 days off.

European officials who replaced the Americans at U.S. oil companies say they have complied with the letter and spirit of Reagan's order and have no Americans working for them.

They say most of the Americans work for ancilliary firms, such as drilling outfits, which do contract work for the major oil companies.

Western diplomats say the Americans usually work in highly technical fields where their skills are hard to replace in the U.S.-dominated industry. However, no Americans appear to be left in management with U.S. companies in Tripoli.

Three American oil companies -- Continental, Marathon and Amerada Hess -- have their own operations in Libya and also participate in the minority foreign share in Oasis, a Libyan government-owned company that still employs some Americans. Occidental's Libyan operations are owned by the government.

The ability of the oil companies, conservative by nature, to avoid rocking the boat and to get on with oil operations was strikingly illustrated at one company. In front of the office of the chief executive is a sign saying "People's Committee," which all organizations in Libya must have in line with Qaddafi's principles of government.

Most of the American managers and technicians have been replaced by Canadians or Europeans who say the shift has had little impact on the Libyan oil industry. The officials, who asked not to be identified, cited some construction delays and administrative problems because of the sudden exodus of Americans.

"Twenty years ago there would have been a far greater impact," a British oil man said. "Then, not so many other nationalities had oil experience."

The biggest impact eventually could be in exploration, another oil man said.

The United States also cut off sales of oil equipment and spare parts, but oil men say this has had little effect because, to prevent American companies from losing business, the ban was applied only to the few products Libya cannot buy elsewhere. They can often be obtained indirectly through Europe.

A ban on U.S. purchases of Libyan oil has applied only to crude, meaning that Libyan oil still reaches the United States after being refined in Europe and the Caribbean.

One European grumbled that, despite the Reagan order, "an American passport is the best one to have at the airport" when going through immigration. "Americans whip through the airport quicker than anybody else," he said, because the Libyan government opposed the U.S. withdrawal and welcomes American personnel.

People calling the State Department to inquire about the travel ban are told the executive order says, "It is illegal to use a U.S. passport to travel to, through or in Libya." Technically Americans coming here do not violate the order, because Libyan officials stamp the visa on a separate sheet of paper and thus the Americans are not using their passports.

Not all the Americans here are in the oil fields. Four are at the Oil Companies' School, which virtually overnight became a Canadian-staffed school with the exodus of 43 American teachers early this year.

Even though there are only 20 American pupils among the 600 students ranging from kindergarten through ninth grade, the school is still predominantly American in outlook.

A new American teacher is struggling to revive the school band and the American recreation director, Skender Brame, who has been in the country 11 years, still maintains a large Little League baseball program, even though many of the parents do not understand the game. One team had an Italian infield and a Chinese outfield.

The school, still known to Libyans as "the American school," has a student body of 54 nationalities. Turks lead the list with 72.

Brame and Barbara Bishr were the only two staff members to stay on after Reagan's order but Bishr was exempted from it because she is married to a Libyan. First grade teacher Honan Mehdi and the music teacher, who declined to be identified, started at the school this fall. All said unemployment in the United States was a key reason for their presence in Libya.

Asked if she felt disloyal, Mehdi said, "No, I'm making a much bigger contribution here. . . . Is President Reagan going to find me another job?"