The government today ordered the expulsion of the 11th Libyan diplomat in two weeks, underlining the extent to which Italy, long protective of its relations with Libya, has begun to crack down on the North African nation.

The expulsion of Mustafa Mohammed Akresh from the Libyan Consulate in the Sicilian capital of Palermo came as the government ordered an investigation of a former secret service chief's account of official Italian support for Libyan leader Col. Muammar Qaddafi a decade ago. He said the purpose was to protect Italian economic interests in Libya.

In announcing the expulsion order served on Akresh today, Foreign Ministry officials said it was not a retaliation for Qaddafi's expulsion yesterday of 36 European officials, including 25 Italians.

Foreign Ministry spokesmen indicated that Akresh was being expelled solely because he had been involved in unspecified activities incompatible with his diplomatic status.

Two weeks ago, in accordance with decisions made at two meetings of European Community foreign ministers following last month's U.S. bombing of Libya, Italy ordered 10 Libyan diplomats to leave. It was left to the Libyan People's Bureau, or embassy, in Rome to pick nine of those to leave, but another Libyan consul was singled out for reasons similar to those stated today.

While the Italian government has been silent on the specific activities that led to the consuls' expulsions, judicial authorities have let it be known that they had to do with suspected support for terrorist activities in Italy.

Ten days ago, Italy arrested a Libyan ex-diplomat in Rome on suspicion that he and one or two others in the Rome People's Bureau last year had plotted the assassination of the ambassadors of the United States, Egypt and Saudi Arabia here. They were suspected of providing a Libyan hit man with a gun and money to do it.

The latest expulsion underlined the deterioration of relations between Italy and Libya, which go back to a colonial relationship in the pre-World War II era.

On April 15, following the U.S. bombing of Tripoli and Benghazi, Libya fired two Scud missiles at the maritime navigation station operated by the U.S. Coast Guard on the Italian island Lampedusa.

Just how cozy relations once were between Tripoli and Rome was underlined yesterday in the newsweekly Panorama in an interview with the former chief of the secret service, Gen. Ambrogio Viviani.

His disclosures have led Defense Minister Giovanni Spadolini to order an investigation of the general, who is still on active duty in the Army.

Viviani was appointed to head the secret service in 1970 by then-prime minister, and now foreign minister, Giulio Andreotti. Orders at the time were to support Qaddafi to "preserve Italian interests in Libya" and "to prevent the ENI [the Italian national oil company] from being forced out of the country," Viviani said in the interview.

"We had to show Qaddafi we were his most trustworthy friends," added Viviani, who remained in charge of the secret service until 1974.

"We supplied him with a great many weapons, organized his secret services and provided him with experts to help modernize the Libyan forces."

The Italian government provided Qaddafi with up to 50 military advisers, Viviani said, and in 1970 it alerted Qaddafi to a coup attempt being organized by his opponents. In 1973, he said, a group of Arab terrorists arrested at Rome's Fiumicino airport with two missiles with which they reportedly planned to shoot down an Israeli El Al jet were quietly released and flown to Libya.

Viviani's career was later to suffer when it was discovered that he had joined the secret Masonic order P2, which Italian authorities said had plans to try to take over the government. But no one has denied his claims made in the interview.

Spadolini ordered the Army chief of staff, Gen. Luigi Poli, to see if Viviani violated the law by revealing state secrets.

The expulsions of Libyan diplomats, new police checks on all other Libyans living and working in Italy, a steady decrease in the number of Italians working in Libya have resulted in increasingly harsh language between Rome and Tripoli.

Prime Minister Bettino Craxi, aides said, still hopes to avoid a full break with Libya both for historic reasons and because it would be economically disruptive here.

Libya gets 30 percent of its imports from Italy and is Italy's 10th largest trading partner. Libya is also Italy's largest source for oil and natural gas. Bilateral trade in 1985 was just under $5 billion, with more than half of that figure accounting for Italian oil and gas purchases.

Although the number of Italian workers in Libya has decreased from 15,000 a year ago to under 3,000, Italian trading, construction and oil companies remain and would suffer from any break in relations.

So, too, would about 50 other Italian companies that are owed more than $600 million by the Libyan government. The declining price of oil caused Libya to defer the payments.

While Italian officials made it clear they would like to avoid the disruption of economic relations with Libya, Qaddafi's recent threats to carry his war with the United States to NATO bases in Western Europe have caused the Craxi government to reassess its relations with Libya and to begin making contingency plans in case a full break cannot be avoided.