Reagan administration officials could try a renounce President Carter's agreement for freeing the hostages in Iran because it was negotiated under duress, but they probably shouldn't, a number of international law experts said yesterday.

Several attorneys and law professors said there are provisions of international law that could be cited to void deals concluded under the threat or use of force, or made in conflict with accepted international norms. But they added that the desire to strike back at Iran could do more harm than good by compounding the international law violations in the crisis, dishonoring Algeria and other Countries who tried to help, and ignoring U.s. Strategic interests in the Persian Gulf region.

The immediate practical effect of stopping the deal would be to keep about $4 billion in Iranian assets in U.S. hands, while forfeiting a chance at a $1 billion pool for settling claims before an international tribunal.

Acting Attorney General Charles B. Renfrew said last night that Justice Department attorneys were filing court papers in civil cases all over the country defending the legitimacy of the arrangement with Iran.

"These are our marching orders and we will continue to march, unless the Attorney General-designate [William French Smith, who takes has yet to be confirmed by the Senate] countermands them," Renfrew said.

Eric Stein, professor of international law at the University of Michigan law school, said yesterday that there is a legal framework that could be used to scuttle the Carter bargain. Article 52 of the Vienna Convention on the Law of Treaties states that a treaty is void "if its conclusions are procured by the threat or use of force in violation of the principles of international law embodied in the United Nations charter," he said.

While the convention has not yet been adopted by the United States, it is considered a part of international law, Stein said.

"This whole thing was blackmail. There's no question about it," he said. "But we have to get over the emotional instinct to strike back. Iran is an important spot and could disintegrate very quickly. We have direct interests in the area."

Don Wallace Jr., a professor at Georgetown University Law Center, said he felt Article 53 of the Vienna Convention might better apply to the negotiations with Iran because it nullifies treaties that conflict with accepted international norms or practice. "Holding human beings hostage is certainly a violation of the norm," Wallace said.

"Legally I wouldn't advise the president to do it," he add. "Even if the Iranians are close to being pirates, trading bodies for money, President Carter's executive agreement is really the same as a treaty. And a key principle of international law is that a country's word is its bond."

Wallace noted, too, that as part of the agreement, the Carter administration agreed to waive U.S. rights at the International Court of Justice. "We won at the World Court and then gave it up. We shouldn't deliver a second body blow to international law by welshing on this deal."

Wallace said that, as chairman of an ad hoc committee on violations of international law for the American Bar Association, he has been studying ways to punish violators. "That's the trouble with international law," he said. "There's no real way to enforce it."

Roger Fisher, a professor at Harvard Law School, said last night that any attempt to repudiate the agreement would be "utterly outrageous and completely ruin our credibility in the Third World. What would the Algerians say?Anyone could say we're no better than they [the Iranians] are."

He noted that he had sent the militants holding the hostages a copy of the agreement that freed the crew of the U.S. Navy ship Pueblo from North Korea in 1968 to show the Iranians that the American repudiation of that Bargain had been cleared in advance with the North Koreans.

"The only question facing us is whether our national security interests in the [Persian Gulf] area are more important than further punishment for Iran," he said. "I believe they are." Thus, he said he disagreed with those who thought the Vienna Convention on the Law of Treaties provided a legal basis for rejecting the bargain. "The only duress that counts is holding a gun to the head of the ambassador who signed the treaty," he said.

Lee Marks, formerly of the State Department legal counsel's office and now chairman of the ABA international law section committee on foreign claims, said the idea of renouncing the agreement "at first blush seems preposterous."

"It would be difficult to make a morally persuasive argument for renouncing the agreement as being made under duress," he said. "It would dishonor the Algerians and the British and the other nations who tried to help. Besides I just don't see what good it could do us."

Marks said that rejecting the agreement would jeopardize a $1 billion pool of money Iran has promised to make available for settling claims of American corporations. He said the court attachments that still are holding $4 billion in Iranian assets are shaky because it is not clear that Iran ever waived its immunity to foreign claims.

Marks, who also represents several American firms who have claims against Iran, said, "I don't know of anyone who would trade the uncertainties of litigation over the attachments for a shot at a $1 billion pool."