The Reagan administration, in its first potential confrontation with Iran, responded in low-key fashion yesterday to reports that Cynthia Dwyer, an American citizen, has been brought before a revolutionary court in Tehran.

State Department spokesman William Dyess insisted that information reaching Washington through the Swiss Embassy in Tehran was that Dwyer, a self-described free-lance writer arrested last May, had undergone "a hearing" on her case rather than "a trial."

Despite news reports from Tehran that Dwyer had been charged with espionage, Dyess said: "We are not aware of any specific charges."

The spokesman drew a distinction between the case of Dwyer, a private citizen, and that of the American diplomatic personnel held as hostage for more than 14 months. Dyess referred to Dwyer as "one of over 1,700 Americans around the world who have been detained" on various charges.

Reminded that Secretary of State Alexander M. Haig Jr. last Wednesday referred to several private Americans remaining in Iranian jails as hostages, Dyess said that Haig "had been speaking in quotation Marks. . . . He did not mean to equate them with the other hostages." The State Department spokesman refused to elaborate on the dwyer case, saying this would be "not helpful."

During the long ordeal of the American diplomatic personnel, the Carter administration several times threatened strong but unspecified action if the Americans were placed on trial. Officials then were particularly reacting to the prospect of show trials, and their likely explosive effect in this country.

In contrast yesterday, department sources, who were not speaking for direct quotation, expressed hope that the court procedure will lead to Dwyer's release as part of a drive by Iranian authorities to resolve issues causing difficulty with the outside world. In this context, officials said, strong comment from Washington now might damage the chances for Dwyer's release.

Last Wednesday Haig said that the future of the remaining captive Americans would be a factor in U.S.-Iranian relations following the end of the "hostage crisis."

The State Department reported unofficial word, through a telephone call to a brother, that Mohi Sobhani, an American of Iranian ancestry, has been released from prison in Iran. There was no word about the third American citizen being held, a man of a Afghan ancestry named Zia Nassry, who is reported to have been charged with spying.