With the United States seeking allied support for a possible international trade embargo against Iran, the Carter administration is facing unexpected complications in its relations with Britain stemming from another major international issue -- Rhodesia.

Britain's request for the United States to lift economic sanctions against Rhodesia has placed the administration in an uncomfortable position. If it ends the Rhodesia boycott without a United Nations imprimaur, the United States could damage its own efforts for some form of concerted economic retaliation against Iran in response to the hostage crisis in Tehran.

At the same time, Washington's failure to lift sanctions could undercut British efforts at the crucial stage of the 14-week-old Rhodesia peace talks in London.

Although it is expected to lift the Rhodesia ban soon, the administration is divided deeply on the issue. Some officials argue that if the United States follows the British lead, it would open itself to charges of advancing national interests by sacrificing principles it urged others to uphold.

Other officials have argued that, while the timing is unfortunate, Iran and Rhodesia are entirely separate issues. In this view, the British have a good case against seeking U.N. action on the Rhodesia sanctions since they yesterday resumed control of the breakaway colony.

In a letter to the Security Council president yesterday, Britain formally notified the council that it was lifting punctive measures against Rhodesia since the territory was restored to legality. The boycott was originally imposed by the council -- at Britain's request -- to punish Rhodesia after its white minority unilaterally proclaimed independence in an effort to prevent black majority rule.

Donald McHenry, the U.S. ambassador to the United Nations, has argued that Britain is "on a very wrong course" by not seeking Security Council action on the Rhodesia sanctions.

"We should argue forcefully with the British to save them -- and us -- from themselves," McHenry was quoted as saying in a government memorandum.

McHenry also warned of difficulties that could arise in Rhodesia during the transition to black majority rule and that "real difficulties could be caused by the fact that U.N. sanctions remain on the books."

Another official supporting McHenry's position said yesterday, "You can't seek U.N. actions against Iran without going to the U.N. on the Rhodesia question."

Officials here said that U.S. economic retaliation plans against Iran range from a limited Western action to a full-scale U.N. boycott. They said that "the turn of events" would determine the course of U.S. actions.

Administration sources expressed the hope that the Rhodesia conference in London would produce a final peace settlement by the end of this week. In that case, they said, the question of sactions would become academic.

In informing the Security Council yesterday that "the state of rebellion" in Rhodesia has ended, the British have sought to deflect criticism of their lifting of sanctions.

Apart from the administration's intentions to help the British in pushing a final settlement, President Carter has been under growing congressional pressure to lift Rhodesia sanctions.

The Senate by a 90-0 vote, last week approved a resolution directing Carter to lift sanctions upon the arrival in Salisbury of a British governor, or by Jan 31, unless he decides the move is against national interests.

Embargo has been used frequently by states in times of emergency. Only in this century, however, has it been employed as a collective measure of enforcement or prevention by international organizations.