Helicopter-borne Iranian troops boarded an Italian ship 30 miles off the Saudi Arabian coast today and searched it for five hours in what shipping industry sources see as an ominous new phase of the stalemated Iran-Iraq war.

The search of the Italian container ship Merzario Britannia is apparently part of a broader Iranian response to the bombing of its Kharg Island oil depot during the past three weeks.

Today, Iranian President Ali Khameini also claimed that his country's jet fighters bombed oil installations inside Iraq at Ayn Zalah, close to the Turkish and Syrian borders.

Iraq's attacks on Kharg Island are aimed at cutting off the oil revenues Iran needs to continue the five-year-old war. An estimated 90 percent of Iran's petroleum exports is loaded onto tankers through the Kharg facilities.

Baghdad announced on Aug. 15 that the terminal had been destroyed. It repeated the assertion after a follow-up raid on Aug. 30.

"The Iranians said it was kind of superficial damage and the Iraqis said heavy," said a shipping expert at Lloyd's of London. "But it is believed that tankers are still going in and out."

In the past, Iran repeatedly has threatened to retaliate for such attacks by blocking the entrance to the entire gulf at the Strait of Hormuz. The United States and other countries have warned that this could lead to a major escalation of the conflict.

Compared to its threats, the measures taken so far by Iran are not so drastic, but they nonetheless represent unexpected and potentially dangerous new moves.

The Italian freighter seized today was less than a day out of the Saudi port of Damma on what its owners considered a clear route up the middle of the Persian Gulf to Kuwait.

The 22,000-ton Merzario Britannia was the second ship reported seized and searched by Iran in three days, according to Roger Lowes, casualty reporting officer for Lloyd's insurance intelligence service in London. The Kuwaiti freighter Al-Wattyah reportedly was seized on Wednesday evening.

Lowes said the only previous example of such an action was on June 20, when another Kuwaiti ship was detained by Iran for several days.

With their extensive searches, the Iranians appeared less interested in deterring traffic than in directly interdicting arms and other supplies possibly headed for Iraq.

Baghdad, on the other hand, has attempted to halt all shipping to and from Kharg. It lost its own port facilities early in the fighting. Lashing back since 1981, it has attacked numerous oil tankers in the gulf with rockets and bombs, killing dozens of crew members and doing millions of dollars' worth of damage.

The war in the gulf has "taken its toll" on shipping, Lowes said in a telephone interview. "We are aware of 148 vessels either hit, damaged or attacked since May 1981," he said. In addition to these, 89 ships are trapped in the waterways near the Iraq-Iran border at the gulf's northern tip and represent what may be a total loss.

The frustration for both sides in the conflict has been that despite their best efforts they have been unable to shut off each other's revenues or supply lines. That pattern does not appear to have changed with the new assaults and seizures.

Notwithstanding the risks, shippers attracted by the premium prices paid for their services in the gulf continue to ply its increasingly dangerous waters.

Carlo Vesin, a spokesman for the Andrea Merzario company in Milan, which owns the ship seized today, said that 10 Iranian soldiers landed from a helicopter on his company's vessel shortly after noon, gulf time.

The Iranians detained the 20-man crew under Capt. Furio Gerbec and proceeded to search containers carrying television sets, construction materials, food and medical goods bound for Kuwait, Vesin said. Five hours later the ship was freed with everyone aboard.

The Iranian air strike against Ayn Zalah, if confirmed, would be potentially more serious.

Much of Baghdad's long-term strategy is built around the restoration of its oil exports through pipelines to Turkey and Saudi Arabia that can be protected effectively by its superior airpower.

But the Iranian pilots "are much more determined" than the Iraqis, as one European military attache in Baghdad put it.

To hit Ayn Zalah they would have to be. Situated about 155 miles from the Iranian border, it is one of the major oil fields in northern Iraq. Through the same region runs the oil pipeline that at present carries virtually all of Iraq's petroleum exports.

Though difficult for Iran's dwindling Air Force, a successful raid around Ayn Zalah could, for a short time at least, take Iraq's plans for forcing an end to the war and turn them inside out.