Defense Secretary Caspar W. Weinberger said yesterday that "in recent weeks, the evidence has been mounting that the Soviet Union may be using slave labor" to build its natural gas pipeline from Siberia to Western Europe.

Speaking to a conference sponsored by the Center for Strategic and International Studies of Georgetown University, Weinberger said this "human rights" issue was perhaps "a compelling" reason for Reagan administration opposition to the Soviet pipeline.

He said the evidence about slave labor comes from many sources, including an "important human rights organization" in West Germany and a "number of Vietnamese groups concerned that their political dissidents may have been exported to Siberia to help build the pipeline."

"The evidence is not conclusive," Weinberger said. "I hasten to say that. But the available evidence is profoundly troubling and some have found it very persuasive."

He said the French and West German governments have promised to investigate the reports, and the United States also is "looking into" the matter.

"The pipeline," Weinberger went on, "does add to the dependence of Western Europe, in an undesirable way, on the Soviet Union." More than that, he said, the pipeline "energizes the entire Soviet industrial system" by bringing urgently required energy to the western part of Russia "long before it gets to Western Europe."

Weinberger rejected the European argument that trade with the Soviet Union, rather than confrontation, "is a better road to peace."

He said he wished it were true that "nations that trade together ordinarily don't fight with each other," but said that during the period of detente with the Soviet Union, the Soviets did not substantially reduce their armaments.

"Neither SALT negotiations, nor economic distress among civilians , nor detente, nor anything else has ever slowed the Soviet military buildup," Weinberger said. "With that in mind, it is a little hard to see how trade of this kind, that has such an obvious military advantage in providing this much hard currency earnings , most of which would go into military spending, can do anything but increase the danger to all of us."