Past American efforts to cooperate with the Soviet Union in a policy of detente "reinforced the Soviet prison wall which stretches from the Balkans to the Baltic," and will not be repeated by the Reagan administration, Defense Secretary Caspar W. Weinberger said here today in using some of the toughest official rhetoric since the cold war.

The Reagan administration will have no more of unconditional detente, Weinberger said, but instead will warn Moscow in advance in specific terms what its actions will cost.

Weinberger indicated that this new kind of linkage -- Washington setting penalties on Moscow for acts not yet committed -- will drive President Reagan's policy for deterring Soviet moves abroad.

The defense secretary slashed away at detente shortly after landing here on a mission to thank Italy for supporting the U.S. effort to modernize the NATO theater nuclear forces based in Europe. Italy has agreed to accept nuclear-tipped cruise missiles on its soil.

Standing before a bank of Italian radio and television microphones placed on the tarmac near his airplane, Weinberger said he had recently visited U.S. troops at a West German border outpost. There, the defense secretary continued, he had viewed firsthand the mined, barbed-wire fence that deters would-be escapees from East Germany.

"I saw the Soviet prison wall," Weinberger said, "that great monument to Soviet realism which stretches all the way from the Balkans to the Baltic."

"Other empires in the past," Weinberger continued, "have built walls to keep their enemies out, but the Soviet empire needs walls to keep its own population in."

In contrast to the harsh language delivered at Rome's airport, Weinberger explained his view on how to deal with the Soviets in softer terms earlier aboard his plane. He said the key difference between this and past administrations will be to speak strongly and clearly before the Soviets have taken actions against the interests of the United States and the rest of the Western allies.

"I think it is very important that the Soviets know ahead of time that there is a strong feeling; that we avoid the problems we had in Angola and Yemen and Ethiopia where they went in completely contrary to our interests, contrary to the interest of the free world, and they were unopposed. And they consolidated their position and they're still there."

He hailed the warning to Moscow that NATO defense ministers in Bonn issued in a joint verbal declaration on Wednesday, saying this was an example of setting the penalty before the act was committed.

Specifically the defense ministers said that "the Soviets would gravely undermine the basis for effective arms control negotiations if they were to intervene in the internal affairs of Poland."

Since the Soviets "very much want" to open talks on reducing the theater nuclear forces deployed on both sides of the NATO line, Weinberger continued, the penalty set in advance was a substantial one from Moscow's viewpoint.

Although Weinberger did not disclose who supported the U.S. position behind closed doors, other officials said Britain and West Germany came aboard quite early. British Defense Secretary John Nott confirmed that he had supported the linkage statement.

Weinberger gave Nott credit today, declaring as he gave reporters a glimpse of the pulling and hauling at Bonn: "As the British said, it was inconceivable that we could sit there and discuss theater nuclear forces, whether and when to deploy them and things like that without taking note of what has to be an extremely grave threat to the NATO alliance in the central front."

Weinberger acknowledged under questioning that some defense ministers felt that it should be left to the foreign ministers to make such policy pronouncements, not the defense ministers, whose agenda was limited to discussing nuclear weapons and their role in Europe. This is the argument Weinberger said he made on that point:

"You can't let departmental or bureaucratic boundary lines obscure the fact that defense ministers might very well have been faced with a vastly more difficult task than we have now if some effort were not made now to indicate the seriousness with which a Soviet invasion, directly or indirectly, would have been regarded."

Weinberger conceded that such joint warnings to Moscow are hard to bring about. He elaborated on difficulties that arose in the negotiations at Bonn that led to the defense ministers linking any Soviet incursion in Poland to resumption of arms talks.

What was "most difficult" Weinberger said, was linking arms control talks so directly to the Polish situation.

"There are large blocs in a number of countries that only want to talk disarmament and arms limitations -- that's almost the unilateral disarmament idea.

"They don't want any linkage," Weinberger said of those nations, which he did not identify but which presumably included factions in the Netherlands and Belgium. "They want the arms control talks to go forward no matter what happens." The way he rebutted arguments for arms control talks at any price at Bonn, the defense secretary said, was to "keep pointing out that there's no way in which you can do that effectively while an invasion was going on or the equivalent of an invasion was going on."

Another argument he confronted, Weinberger continued, was that setting penalties in advance might provoke Moscow into taking the very actions the alliance wished to discourage.