DUBAI, UNITED ARAB EMIRATES, SEPT. 5 -- Kuwait today ordered the expulsion of five Iranian diplomats and, in a protest to the United Nations, charged that Iran has fired three Silkworm missiles into Kuwaiti territory since Wednesday, blowing out windows in residential areas and threatening vital oil installations.

Over the Persian Gulf, the air war continued for the eighth day as Iraq reported that its warplanes struck two "large naval targets" near the Iranian coast, bringing to 15 the number of Iranian vessels reportedly hit since Iraq resumed its campaign a week ago against Iranian oil installations and ships.

Tehran asserted in a communique that Iraq has failed to blunt Iranian oil exports, crucial to Iran's war-fighting endurance. Iran's military command claimed it shot down an Iraqi warplane yesterday and said nine persons were killed and 100 wounded in Iraqi air strikes in the northern part of the country.

At the southern end of the gulf, the third outward bound convoy of U.S. Navy-escorted ships cleared the Strait of Hormuz, with two extra supertankers that fell in behind for protection. An Iranian warship shadowed the convoy on the final leg of its journey.

Meanwhile, both Iran and Iraq announced they welcome a coming peace mission by U.N. Secretary General Javier Perez de Cuellar, but diplomatic sources in the region said they saw little hope, from statements out of both capitals, that the international initiative will bring the warring parties to negotiations.

Tehran radio said today that the success of the U.N. mission "is subject to denouncing and indicting the aggressor" and insisted that the Security Council "take practical terms to punish" Iraq for starting the war in September 1980.

The missile attacks on Kuwait, although they produced no casualties, have heightened tensions during a week of unusually high violence in the gulf and pose additional problems for the Reagan administration, which has undertaken a major military effort to protect Kuwait's shipping. A key U.S. objective in the escort operation has been to demonstrate American resolve to insure the security of the moderate Arab states along the gulf whose oil reserves represent a vital resource to the West.

Yet the tactic of firing long-range missiles, like the use of mines in gulf waterways, leaves the United States few ways to offer additional protection short of abandoning its stated position of neutrality and taking military action against Iran.

A U.S. official in the region said tonight that one of the long-range missiles fired before dawn yesterday hit a beach, sending up shards of shrapnel in a powerful explosion that residents of the area reported blew out windows and cracked masonry on buildings.

The impact occurred near Mina Abdullah, 30 miles south of Kuwait city near the emirate's Shuaiba port and major oil refining and storage facilities.

Another missile exploded when it struck the sea early today near Fialaka island, 13 miles off Kuwait's northern coast. A missile fell offshore last Wednesday but was not reported until today.

Pentagon sources in Washington said the missiles are believed to be Silkworms fired from Iranian launchers on the Faw Peninsula, a marshy section of land north of Kuwait seized from Iraq by Iranian forces in February 1986. The Iranian military captured a number of Soviet-made Styx missiles, identical to the Chinese Silkworms, during the Faw campaign.

"Our speculation is that they are Silkworms," a U.S. official in the region said today. "There aren't that many candidates for what this could be."

In a letter of protest to the U.N. secretary general, Kuwaiti Foreign Minister Sabah Ahmad Sabah asserted that "Iran fired a missile at some residential buildings and industrial installations in the south of Kuwait, inflicting damages on them."

Ahmad called on Perez de Cuellar, who will conduct a peace mission to Tehran and Baghdad next week, to "take whatever measures . . . to prevent Iran from continuing its attacks against Kuwait."

Tehran has denied firing missiles at Kuwait but earlier threatened to attack Kuwaiti economic targets in retaliation for Kuwait allowing its ports and highway links to Iraq to be used to transport war materiel. Kuwait also pumps and ships oil on Iraq's behalf to help finance Iraq's war effort in the seven-year-old conflict.

This evening, Kuwait's Foreign Ministry undersecretary summoned the Iranian charge d'affaires, Mohammed Hussein Baktari, and told him that five of his seven-man embassy staff had been declared persona non grata and would have to leave the country within one week.

Diplomatic sources said no reason was given publicly for the expulsion but added that the suddenness of the announcement had to be viewed in light of the general alarm in Kuwait over the missile attacks.

Kuwait's Foreign Ministry also called in ambassadors of the five permanent members of the U.N. Security Council -- the United States, Britain, China, France and the Soviet Union -- to urge that they seek urgent U.N. action.

Kuwaitis have blamed an increasing level of internal sabotage on Iranian-inspired extremists. Residents near the country's large oil installations have in the past year been evacuated several times or have discovered after the fact that explosions and fires near their neighborhoods and schools were sabotage.

The U.S. convoy that left the gulf today appeared to be the largest of the six that have sailed the 550-mile stretch between the Strait of Hormuz and Kuwait since the Pentagon's protective escorts began in late July.

At one point, the 10 U.S warships and energy tankers stretched out along a 12-mile line in the lower gulf. The helicopter carrier USS Guadalcanal sent Sea Stallion mine-hunting helicopters ahead of the convoy, and the transport vessel USS Raleigh provided high-speed patrol boats to counter possible assaults by Iranian speedboats that have attacked merchant vessels with relative impunity for months.

When dawn broke over the convoy of American-flag vessels, shipping sources said, two additional supertankers registered under the Liberian flag -- the Grand and the Docefjord -- had joined the convoy for protection. The Docefjord steamed out of the Saudi port of Ras Tanura and the Grand was reported to be carrying Kuwaiti crude under charter.