A West German rocket company that is a thorn in the side of the Bonn government and is currently linked to Col. Muammar Qaddafi of Libya is negotiating for another test launch site far away from the Libyan Sahara, where it now operates.

The company has a reputation for mystery, and Frank Wukasch, its president, would say only that the new site will not be in an Arab or African country. Another source, who generally is well-informed about the activities of OTRAG -- an acronym for Orbital Transport-und Raketen-Aktiengesellschaft -- said it was his understanding that the new site would be somewhere in South America, possibly on an island off the coast of Argentina. Wukasch would neither confirm nor deny that.

Much of the reason for the move, Wukasch said, is to lessen the controversy surrounding the company's presence in Libya and to have a place to go should Qaddafi suddenly decide to expel the firm. In a further effort to erase OTRAG's menacing image, Wukasch also said the firm is negotiating with several respected research institutions and universities for a program of high-altitude, research-gathering rocket shots that would serve as a sort of sidelight to the company's main business: the development of satellite-launching rockets.

The company has been an embarrassment to Bonn, which does not approve of West German firms experimenting with rockets that have potential for military use. OTRAG has always maintained that it is not involved in any kind of military research.

West Germany has tried several means of thwarting the firm. In 1978, it passed a law -- known as "Lex OTRAG" -- that specifically prohibits the export of OTRAG rockets or parts for them. But OTRAG now simply buys and assembles its uncomplicated and relatively cheap launch vehicles outside West Germany.

In 1979, West Germany, with the assistance of France, persuaded Zaire to cancel a long-term contract with OTRAG and expel the firm from the 39,000-mile testing area there over which OTRAG had enjoyed virtual sovereign control. The Soviet Union and a number of African countries had complained about the firm's presence in Zaire. OTRAG found new testing facilities soon afterward, however, at Sebha in Libya, about 500 miles south of Tripoli.

Last year, Bonn sought to revise the tax laws that had encouraged OTRAG to form in the first place under a privilege that permits write-offs for investment in research and development firms. OTRAG's investors, all Germans, according to Wukasch, number more than 1,400.

"It is outside control of and yet still imbedded in the Federal Republic; it would appear the company is both inside and outside Germany," said Bonn government spokesman Lothar Ruehl about OTRAG, which is headquartered in a two-story building in an industrial park outside Munich.

West Germany has made known to Libya its displeasure at OTRAG's shelter there, but Bonn cannot influence Qaddafi in the same way it coaxed President Mobutu Sese Seko of Zaire. A visit to Bonn last month by Libya's second-in-command, Abdul Salaam Jalloud, resulted in a frosty exchange with West German Foreign Minister Hans-Dietrich Genscher and Jalloud's abrupt departure. West German officials said OTRAG was mentioned in the talks, along with other points of difference.

Wukasch said OTRAG settled in Libya largely because Qaddafi "cannot be blackmailed" into expelling the company. He denied previous reports of an arrangement with Qaddafi that promises to give the Libyan leader a 5 percent commission on eventual OTRAG rocket sales -- the same terms that reportedly had been promised Mobutu. "We simply have verbal permission from Libya to conduct testing, that's all," said Wukasch.

Bonn officials find it difficult to believe that Qaddafi, particularly given his reputation for scheming for armaments and striving for prestige in the Arab world, would give OTRAG a site without some sort of deal in return.

OTRAG's stated aim is to develop a low-cost rocket to send satellites into space for such purposes as communications, mineral prospecting, mapping and reconnaissance. It plans to sell to Third World governments that either cannot afford, or would prefer not to use, the launch vehicles offered by NASA's space shuttle or the European Space Agency's Ariane booster. The only other known pioneering commercial venture into space travel is a joint effort by two young U.S. companies, Space Services Inc. of Houston and GCH Inc. of Sunnyvale, Calif.

Reports continue to circulate linking OTRAG with military designs. West Germany's widely circulated illustrated weekly Stern published an article in June tying a Frankfurt businessman, Klaus Dietrich Nickel, and the former chief of staff of the West German armed forces, Gen. Harald Wust, to an attempted billion-dollar deal that would have had OTRAG providing rockets to Saudi Arabia. This month, the Israeli press reported that OTRAG had signed a contract to supply Syria with long-distance carrier rockets for ballistic missiles.

Wukasch, an affable 35-year-old with a degree in aerospace engineering, called such stories "slanderous." He attributed them to efforts by the Israeli and other intelligence services to destroy his company.

Wukasch himself is not actually the power behind the company. That role belongs still to Lutz Kayser, 42, an aerospace expert who has spent more than 10 years developing the OTRAG concept and rocket and retains a majority interest in the company.

"I think Kayser is a blue-eyed dreamer, or was," said Prof. Harry Ruppe, director of the Institute of Space Technology at the Technical University in Munich, who, as a sort of intellectual hobby, has taken a special interest in OTRAG. "I think in the meantime he's become a hard-knuckled businessman, a bit of a wheeler-dealer."

Ruppe is among those who believe the OTRAG rocket will not work as an inexpensive launcher. The Bonn government also rejected Kayser's original concept as feasible for public-funded development before West Germany joined the Ariane effort in 1974. Ruppe, who sat on the official panel that judged the proposal then, said Bonn's doubts about the scientific worth of OTRAG's test flights make West Germany even less willing to tolerate the company as a political nuisance.

Nevertheless, Ruppe would still like to see the company given a chance to come up with a cheap solution for space launches. While agreeing that the rocket could be put to military use by a foreign country, he said it would make a highly inefficient weapon since it uses liquid propellant, which presents storing and transport problems.

OTRAG has reported four test launchings so far and claims all were successful, although one launching in front of Mobutu failed to follow its planned trajectory. Its innovation lies in two basic design principles: off-the-shelf, mass-produced parts including the use of standardized steel pipes that are clustered into propulsion units; and an extremely low-cost fuel containing a mixture of diesel oil and nitric acid.

"It is hard to convince people we are not military," Wukasch said. "Because of Germany's history in the aerospace field, if a private German company starts developing a rocket, people assume it must be for military purposes.

"A year or two from now, when we announce the launching of our first commercial payload into orbit, that will be the beginning of OTRAG's credibility," said Wukasch.