Col. Muammer Qaddafi of Libya, faced with a militant network of exiled opponents and apparently worried that Western countries might join their attempts to overthrow him, is quietly seeking to repair his fractured relations with the United State.

The volatile, 43-year-old Libyan leader has not slowed his frequent public denunciations of American "imperialism," or noticeably reduced his widespread guerrilla ventures. But it appears that cryptic warnings by U.S. Secretary of State Alexander M. Haig Jr. and calls for Qaddafi's overthrow by a host of exiled Libyans and the leaders of Egypt and Sudan are inducing more caution in the "Brother Colonel," as his followers call him.

The Reagan administration has yet to warm to Qaddafi's overtures. There are reports in Washington that the State Department is considering expelling Libyan diplomats because of their government's support of terrorism.

Qaddafi, meanwhile, has sought in recent speeches to deflect Western charges that he is a promoter of terrorism. He has privately urged the U.S. State Department to reopen the U.S. Embassy in Tripoli, which was unofficially closed last May after what U.S. officials described as "months of harassment" and the partial burning of the building by a mob in December 1979.

And most recently, Qaddafi summoned executives of five U.S. oil companies to a Feb. 12 meeting in Libya, which is third among foreign suppliers of U.S. oil. His chief aide, Maj. Abdul Salaam Jalloud, proceeded to lecture the Americans -- who according to U.S. officials represented the Exxon, Mobil Oil, Occidental Petroleum, Marathon and Continentaal Oil companies -- on the need for pressuring the new Republican administration for better relations.

Jalloud subsequently indicated the currently precarious state of those ties when he was quoted by a Kuwaiti newspaper as warning that Libya might be forced to align itself completely with the Soviet Union because of the "strong American pressures Libya is subjects to."

During the flurry of bad publicity for Libya that erupted last summer after the revelation that former president Jimmy Carter's brother Billy had accepted a Libyan loan of $220,000, the Carter administration privately suggested resuming the relations Libya was by then pressing for by designating Belgium as a "protecting power" for U.S. interests in Libya. b

Libya rejected the proposal. "We feel," said Ali A. Houderi, Libya's senior diplomat in Washington, "that a first-class power like the United States States should have its own representation and not have to depend upon a second-class power like Belgium. We want to deal directly with you."

U.S. analysts say they appreciate the Libyan's invitation to sit down and discuss the issue of terrorism, and his refusal to permit a hijacked Pakistani airliner to land in Libya last March. But they say they see no signs of moderation in the ebullient colonel's backing of guerrilla movements -- or in his recent calls for Arab world backing in Syria in a war with Israel in Lebanon.

In fact, the most noticeable signals by the new administration toward Libya until now have been warnings by Haig against Qassafi's support for "world terrorism" and several public criticisms by U.S. officials of Libya's military intervention in Chad last winter. Executives of major oil companies were called for a high-level State Department meeting this week on Libya, but the session was later cancelled.

Qaddafi's apparent sensitivity to the United States comes at a time when a number of exiled Libyan dissidents, including several former high officials in Qaddafi's government and three multi-partisan organizations, are calling for Qaddafi's overthrow. The opposition has managed since last October to distribute anti-regime propaganda inside Libya and claims to have begun sabotage operations against economic and industrial targets, though the sabotage is unconfirmed.

More significantly, some of Qaddafi's opponents say they have asked the United States what it would do, if anything, to help neutralize or overthrow Eastern Bloc assistance to Qaddafi in the event of a coup attempt. oU.S. officials have refused to confirm such contracts.

These claims have undoubtedly helped to spur counterattacks that have been blamed on Qaddafi such as the murder of eight Libyans in London, Rome and Athens last year and attempts on others, including a Libyan student shot and wounded in Colorado in October.

Houderi, a close friend of Qaddafi, denies any Libyan government role in the Colorado and European shootings. And Houderi maintains that it is in the interests of the United States to reopen the embassy in Tripoli. "Your own interests in Libya are much larger than our interests here," he says.

U.S. interests in Libya center around the 12 percent of all U.S. oil imports from there and other trade and contracts worth nearly a billion dollars.

The departure of U.S. charge d'affaires William L. Eagleton Jr. and his staff last year without the formal shutdown of the embassy or the breaking of diplomatic relations left more than 2,000 employes of the American oil companies and other U.S. firms without consular protection.

Houderi says Libya offered compensation for damage to the Tripoli embassy, and would give iron-bound security guarantees if the embassy were reopened. To this, U.S. officials respond that they did present Libya with a detailed damages bill for about $500,000 that has not been paid and add they are wary about the offer of security guarantees.