The United States is in the unusual position of trying to placate President Habib Bourguiba, leader of the only Arab state that refrained from openly criticizing President Reagan's air raid against Libya.

Along with France and other western allies of this small North African country, the United States is worried that Libyan leader Muammar Qaddafi might choose neighboring Tunisia as a conveniently close soft target to demonstrate his ire.

In the past, U.S. and French naval exercises have sufficed to defuse Qaddafi's friendly and unfriendly moves toward Tunisia, whether they took the form of a momentarily accepted treaty of union in 1974 or last year's spate of letter bombs, mass expulsions of Tunisian workers and other unfriendly acts.

Such demonstrations of force may no longer be effective, according to western and Arab diplomats and Tunisian observers.

Mindful of serious economic problems, a rising tide of Islamic fundamentalism, open jockeying for power and a record of periodic public disturbances, they fear future domestic instability as the 83-year-old Bourguiba's 30-year rule nears an end.

A complicating factor involves changing attitudes toward the United States.

What long amounted to a comforting U.S. guarantee of Tunisia's sovereignty was seriously questioned by what Tunisians took as White House approval for Israel's Oct. 1 raid on the headquarters of the Palestine Liberation Organization here.

That perceived American "betrayal" of Tunisia had not been forgotten when the April 15 air raids against Libya reawakened local misgivings about the United States.

Western diplomats and some Tunisians privately questioned the wisdom of Bourguiba's ill-concealed antipathy toward Qaddafi at a time when many Tunisians sided with Libya as a small Arab neighbor attacked by the world's most powerful nation.

Anti-American, and antigovernment, demonstrations in Tunis and the largely underdeveloped south, where many Tunisians have age-old ties of family and commerce with Libya, bore witness to that sentiment.

So, too, did the banning of four publications critical of the raid and disturbances at the University of Tunis during which l,500 students were briefly detained.

Fueling the tensions there was the death of Othman Ben Mahmoud, an engineering student of Islamic fundamentalist allegiance, in a clash with police. The university's humanities school was closed for 10 days and students are still on strike protesting the continued detention of about 30 colleagues.

Of more long-lasting repercussions, however, may be the recent jailing of Ahmad Mestiri, a leader of the legal opposition that ranges from his Social Democratic Movement to Moslem fundamentalists and Communists.

Mestiri was convicted of unlawful assembly, the technical charge invoked for having led an orderly demonstration protesting the U.S. attack on Libya. Political observers said he was punished on Bourguiba's personal instructions.

Mestiri was released temporarily Tuesday to undergo an operation, but he was expected to be returned to jail.

Opposition sources defended Mestiri's protest as a necessary safety valve to allow an outraged public to blow off steam.

Unless an appeals court reduces the sentence, Mestiri, 61, will be ineligible to run in the November parliamentary elections and Tunisia will be deprived of the leader of the legitimate opposition.

Such considerations worry many Tunisians because of the increasing intrigue at the presidential palace at Carthage, which one observer described as "installments of 'Dynasty.' "

Since last fall Bourguiba reportedly has banished Wassila, his second wife and an influential member of Tunis society, in favor of his niece, Saida Sassi.

A native of the president's home town of Monastir, Sassi is said to be determined to avenge her past humiliation at the hands of the president's spouse by punishing Wassila's proteges.

In trials this month involving mismanagement of public funds, Wassila's son-in-law, banker Tewfik Torjeman, was sentenced to six years in prison and her protege, Mohammed Belhaj, until recently the president of Tunis Air, was jailed for five years and a month.

Habib Achour, 73, the veteran leader of the General Union of Tunisian Workers, was sentenced to an additional year in prison in addition to the sentence he was already serving for alleged financial mishandling of union funds.

Habib Bourguiba Jr., the president's son by an earlier wife, was stripped of his title as presidential adviser last fall, reportedly for opposing his father's plans to divorce Wassila.

In the jockeying for position at the palace, political analysts said Prime Minister Mohammed Mzali is losing ground, although under the constitution he is the president's successor.

Last week Mzali lost the Interior Ministry to rival Zine Abdine Ben Ali, a tough law-and-order enthusiast.

Bourguiba, who reportedly suffers from poor eyesight and hardening of the arteries, presides over Cabinet meetings which, because of his infirmities, are said to be conducted without written agenda or documents.

A daily indication of those in favor is provided by the photograph of guests invited along for Bourguiba's daily constitutional, which is dutifully covered by state-owned television, and his Destourian Socialist Party publication, L'Action.

Despite the government's financial problems, Bourguiba has ordered his ministers to spend millions of dollars cleaning up slums, building dams, highways and canals and creating jobs for the growing number of young unemployed Tunisians.

Yet little meaningful change has occurred since January 1984 when Bourguiba ended riots by committing his personal prestige and rescinding the bread price increases that had sparked the disturbances.

The government's most serious effort to shore itself up involved a major recruitment of security forces whose budget now reportedly outranks that of the armed forces.

For the time being, the security forces seem to have the situation well in hand, but western and Arab diplomats fear an uncertain future for Tunisia after Bourguiba.

"The problem for the United States," a western diplomat said, "is that it has the government with it, but no longer the Tunisian people."