The Carter administration engaged Iran's revolutionary government in its first diplomatic confrontation yesterday by disclosing - and then publicly rejecting - Iran's demand that Walter Cutler be withdrawn as U.S. ambassador-designate to Tehran.

Iran could accept either Cutler or a "substantial delay in resolving the question of ambassadorial representation," State Department spokesman Hodding Carter told reporters at a briefing. The Iranian rejection of Cutler "is not helpful in helping restore a constructive relationship," he added.

Neither Carter nor Iranian officials here, who seemed taken by surprise by the State Department's announcement, would not say what reasons if any were given by the revolutionary government in withdrawing the preliminary agreement it had already given to Cutler's nomination.

The diplomatic standoff appeared to put a deep chill on relations between Washington and Tehran. They have been steadily deteriorating since the government under the control of Ayatollah ruhollah Khomeini came to power in February.

U.S. officials said it was not clear if the rejection of Cutler, which was made known to Washington late last week, was a sign of general tension between the two countries or if the Iranians were now objecting to Cutler's background, which includes assignments to several authoritarian governments in Southeast Asia and Africa.

Foregin Minister Ibrahim Yazdi, who is reported by U.S. analysts of Iranian affairs to be in increasing political trouble at home, has been sharply attacking the United States in recent weeks and thereatening to break off relations with Washington if U.S. "interference" in Iranian affairs continues.

This criticism began to escalate two weeks ago after the U.S. Senate passed a resolution condemning summary trials and executions in Iran. Tehran retaliated by asking that Cutler's arrival be delayed.

But U.S. sources also report that Khomeini's government gave its orginial agreement to Cutler's nomination only grudgingly hoping that academic figures like Richard Cottam or James Bill, who had shown sympathy for the revolution's original aims, would be considered.

These sources suggest that the Iranians may have had second thoughts about Cutler's particular nomination in the light of Yazdi's charges of U.S. support for efforts to topple his government and the ambassador-desgnate's previous assignments in South Korea, South Vietnam and most recently as ambassador to Zaire.

Zaire's president, Mobutu Sese Seko, has often been depicted as a despotic rule who has amased a hugh personal fortune after being put in power by the Central Itelligence Agency. The Iranian opposition asserted that Shah Mohammad Reza Pahlavi had followed the same career pattern before being chased from his throne in January.

Cutler was named in April to succeed Ambassador William H. Sullivan, who became publicly identified in Iran with the strong U.S. support for the shah through a year of turmoil that was spearheaded by Khomeini, then in exile in France.

"We have n plans withdraw his name or to reassign him," said Hodding Carter, calling Cutler a "distinguished" diplomat and adding that President Carter and Secretary of State Cyrus R. Vance maintained "full confidence" in their ambassador-designate.

Hodding Carter refused to say when or where the Iranian rejection of Cutler had been delivered. Department officials said later that the Iranians conveyed their new position to the embassy in Tehran last week, news agencies reported. The State Department waited until Vance returned over the weekend from a two-week tour of the Middle East and Europe before making the request, and its rejection public.

The effect of the decision is to leave the embassy in the charge of deputy chief of mission Charles Naas, whom Iranian critics have also charged with being one of the shah's strongest supporters before the January collapse. The Iranian Embassy in Washington is also headed by a charge 'd affaires, Ali A. Agah. CAPTION: Picture, WALTER CUTLER . . . called "distinguished" diplomat