THE IRANIANS had read the letter of the U.S. Navy's rules of engagement in the Persian Gulf correctly, but they plainly violated the spirit when they hit a tanker that, though in Kuwaiti waters, was flying the American flag. To have allowed Iran to continue playing games with the rules would have ruined American credibility. An appropriate response was in order, and the strike on the oil platform filled the bill.

The special requirement was to serve and not undermine the American strategy of working with the Soviets at the United Nations to get a cease-fire or, that failing, an arms embargo on Iran. That dictated a step that clearly related to the protection of shipping, that would draw third-country as well as American public support and that would touch Iran but not drive it altogether beyond U.N. reach.

What Mr. Reagan chose will not satisfy those who believe a single stunning blow -- perhaps a strike at the Silkworm sites -- could bring Iran to reason. This was a relatively modest use of force: no American lives were lost, no Iranian territory was hit, and Iranians on the platform were warned to get off. Mr. Reagan can fairly claim he is neither exposing Americans to excessive risk or endangering others indiscriminately nor, in another context, provoking Iran or allowing this country to be drawn casually into an open-ended war.

Iran may mistake American restraint for weakness. Still, it makes sense now to view Iran's feints and attacks as attempts to influence the United Nations in its continuing deliberations on putting the Security Council's resolution of July 20 into effect. Iran is extreme in ideology but less so in policy. Its attacks on third-country shipping deepen its isolation, but its attempts to elude a U.N. arms embargo belie suggestions that it doesn't care about the possible effects of isolation.

Notwithstanding its touted ''new'' U.N. policy, the Soviet Union could yet end up refusing to support an embargo against a recalcitrant Iran; for now, however, it works with Washington for a cease-fire. And the pressure the United Nations could apply on Tehran by even a leaky embargo stands to count for more than anything likely to be done, and sustained politically, by American arms.

The United Nations is on a track that would end the war on fair terms, allow Iran (through a provision for investigating causes of the war) to broadcast its complaints and usher most of the U.S. fleet out of the Gulf. This remains the right way for the United States to go.