The Soviet Union test-fired two ballistic missiles on long-range flights over the Pacific this week, and one hurled its dummy warheads about 500 miles from Hawaii, prompting strong U.S. diplomatic complaints, administration officials said yesterday.

The missiles, launched Tuesday and Wednesday from central Siberia, angered congressional leaders and administration officials, who said the reentry vehicles of one of the intercontinental ballistic missiles fell closer to U.S. territory than in any previous Soviet tests.

U.S. arms control experts puzzled over the unusual tests, which come two weeks after Washington and Moscow made significant progress toward a treaty to eliminate intermediate-range nuclear force (INF) missiles and a summit between President Reagan and Soviet leader Mikhail Gorbachev.

Gorbachev yesterday proposed talks to curb military activity in Europe's northern waters. {Details on Page A16.}

Officials said the Soviets notified Washington last Saturday that they planned a missile test flight over the Hawaiian Islands and into an ocean target area about 350 miles south of the island chain. U.S. officials protested vigorously, Defense Department sources said. Pentagon spokesman Fred Hoffman said this would have been the first time a Soviet ICBM had flown over U.S. territory.

The Soviets also notified the United States of a second target zone about 500 miles northwest of the islands, and warned ships to stay clear of both target areas until next Thursday.

"We were particularly concerned that, based on their announcement, they intended to bracket the Hawaiian Islands from north to south," Hoffman told reporters.

"We protested this," said State Department spokeswoman Phyllis Oakley. "We made known to the Soviets through diplomatic channels our serious concern about missile tests being conducted so close to U.S. soil."

The Soviet news agency Tass announced Tuesday that the missiles would be fired from the missile test site at Tyuratam in central Siberia, Pentagon officials said.

On Tuesday afternoon Hawaiian time, which is six hours behind Eastern Daylight Time, the first test missile was fired, ending in "an apparent failure," according to Hoffman.

The missile is said to have been an advanced version of the SS18, the Soviets' longest-range intercontinental rocket, capable of carrying 10 independently targetable warheads more than 6,800 miles. But the "post-boost element" of the missile appeared to malfunction, Hoffman said, and it fell far short of either target zone.

The second test, conducted Wednesday afternoon, "appears to have been successful," Hoffman said, sending dummy reentry vehicles into the northern target area.

Pentagon officials said the Soviets apparently did not attempt to use the target site that would have taken the missile over the islands. "We do not know whether or not that was a result of our expression of concern," Hoffman said.

He said Moscow announced yesterday that the series of tests was over and that "shipping can move safely through that area."

Pentagon officials said yesterday the Soviet Union has occasionally tested long-range missiles in a triangular region in the northern Pacific, but said the reentry vehicles have never landed this close to U.S. territory. U.S. officials refused to say how close previous tests in the area have been to Hawaii.

Members of Congress expressed outrage yesterday, with Sen. Pete Wilson (R-Calif.) calling the test "a provocation of the worst kind, the most dangerous kind."

"The Soviets were practicing attack on America," charged Sen. Malcolm Wallop (R-Wyo.).

Rep. Patricia F. Saiki (R-Hawaii) complained that the reentry vehicles could have dropped on cities in her state. She said Gorbachev "better understand that Hawaiians are not going to tolerate their state being used as a bull's-eye for Soviet missile tests."

Sen. Robert J. Dole (R-Kan.), a GOP presidential contender, said, "It's hard to square this kind of reckless action with the hearts and flowers we keep hearing from the Kremlin."

Arms control experts said the far-Pacific tests are infrequent, with the Soviets usually confining such tests to an area near the Kamchatka Peninsula on the Soviet Pacific coast.

"This is unusual for the Soviets to be playing these sorts of games," said Michael Krepon, an arms control specialist with the Carnegie Endowment for International Peace.