Prime Minister Robert Hawke told the Reagan administration yesterday that Australia will not allow U.S. aircraft to use Australian bases to monitor an MX missile test, dealing the second blow this week to the Australia-New Zealand-United States alliance (ANZUS).

Secretary of State George P. Shultz, after a meeting here with Hawke, announced that the United States will bow to antinuclear sentiment in Australia and conduct the MX test next summer in the Tasman Sea, between Australia and New Zealand, without Australian help.

Hawke's about-face, which followed press reports in Australia last Friday that his government would provide support facilities, apparently was made without consulting the United States before his arrival here Tuesday.

The Australian action followed the cancellation Tuesday of a planned ANZUS naval exercise after New Zealand denied port access to a U.S. warship because the United States would not say whether the ship carries nuclear arms.

The two incidents have raised fears within the administration of a ripple effect on other allied governments that are also being pressed by domestic antinuclear movements not to cooperate with the United States in activities involving nuclear weaponry.

But administration officials denied that the United States might retaliate by imposing economic sanctions against New Zealand, which exports sizable amounts of dairy products, beef, lamb and wool to this country. Shultz said, "It isn't a question of taking action against New Zealand," and a senior U.S. official who spoke with reporters earlier said, "No, that's absolutely wrong. We've said repeatedly that we are not in the business of sanctions."

The United States is particularly concerned that New Zealand's example could create pressures in Australia and Japan for similar restrictions. A setback in U.S. nuclear collaboration with its South Pacific allies could have an impact in Western Europe, where the governments of Belgium and the Netherlands are being pressed to renege on commitments to deploy U.S.-made, medium-range nuclear missiles.

U.S. officials insisted yesterday that the problem with Australia over the MX test was much less serious than the dispute with New Zealand because the United States has other ways to monitor the splash-down tests and because the Hawke government continues to cooperate on a broad range of other issues involving naval defense of the South Pacific. The officials said that test monitoring now will be done from ships, a more costly operation.

By contrast, Shultz said yesterday that the port-call decision "changed the operational character of New Zealand's participation" in ANZUS in ways that will require a rethinking of the entire alliance.

The MX test warheads will be aimed at an area of international waters in the Tasman Sea wired for sound. The Australian government has said that U.S. merchant ships have been sowing underwater sensors in that sea since early last year. These sensors are somewhat like underwater microphones that transmit the sounds of the warheads hitting the water.

Hawke, who joined Shultz in a brief appearance for reporters after a State Department luncheon, reaffirmed that his Labor government considers ANZUS an "important part of Australia's foreign relations." Hawke said:

"It would be totally mistaken for any conclusion to be drawn that the decision of the United States to monitor the MX test in another way has any impact on the ANZUS relationship. It has none."

While these officials clearly were more angry about the dispute with New Zealand, they insisted that the United States still considers New Zealand "a friend," and they denied that a reexamination of U.S.-New Zealand relations, announced Tuesday, implies a threat to retaliate.

But Shultz underscored Washington's annoyance by saying, "We have great affection for the people of New Zealand. But we also remind them that those who value freedom have to be prepared to defend it."

Sen. William S. Cohen (R-Maine), a member of the Armed Services Committee, introduced a resolution yesterday calling on the United States to join with Australia in a new pact, excluding New Zealand, unless it changes its position. Cohen also advocated a "tougher stance" toward permitting New Zealand's agricultural exports into this country.

However, the senior U.S. official, who briefed reporters at the White House on condition that he not be identified, said: "If New Zealand doesn't want to be an ally, that doesn't mean it's become an enemy. New Zealand is still a friend of the United States. We are not going to take economic retaliation against New Zealand.

" . . . It's been said repeatedly by people in Congress that frequently New Zealand has benefited, gotten special benefits, particularly in the legislative process from the argument that New Zealand is an ally, a loyal and faithful ally, and deserves some special consideration. That's something that might well be affected. But that doesn't mean we're going to go out trying to figure out how to damage the New Zealand economy."

The comments by Shultz and the senior U.S. official suggested a softening of the harsh tone taken by the administration Tuesday. Some U.S. officials said privately that the administration wants to avoid deepening the dispute because that would make it more difficult to patch up some kind of compromise to keep ANZUS intact.

New Zealand Prime Minister David Lange accused the United States yesterday of "bullying tactics" and said: "I regard it as unacceptable that another country should by threat or coercion try to change a policy which has been embraced by the New Zealand people."

Lange reiterated that, despite its differences with Washington on the port-call issue, his government does not intend to withdraw from ANZUS. The prime minister also said he is confident that the United States will not use trade sanctions against New Zealand.

U.S. officials said privately that even if Washington wanted to apply sanctions, it has little leeway because New Zealand gets no preferential treatment in its sales to the United States of lamb, casein, beef and wool. New Zealand, which ranks 47th among America's overseas suppliers, sold $726.7 million worth of goods to this country during the first 11 months of 1984 and purchased $640.5 million worth from the United States.

The greatest potential sanction would be a U.S. dumping of surplus butter and other dairy products, purchased at subsidized prices under the U.S. farm support program, into New Zealand's traditional markets. The United States has taken such actions against the European Economic Community, but it traditionally has informed New Zealand in advance that the move was not aimed against it.

Hawke, in his appearance with Shultz, avoided giving direct answers to questions about the MX test. Instead, he said, the issue did not arise because the administration already had decided to conduct the tests without Australian help.

U.S. officials said privately that they had been made aware Tuesday night that Hawke was facing domestic political difficulties because of the MX issue and wanted to strengthen his hand against forces in the Australian government wanting to take stronger measures against nuclear cooperation. Although Hawke's Labor Party has taken antinuclear positions, he has fought off calls for Australia to deny port facilities to U.S. vessels that are nuclear-powered or that carry nuclear weapons.

U.S. military officials said the Air Force originally had planned to launch aircraft with special instruments from Australian bases to record signals from the 10 MX test warheads on the last leg of their 7,500-mile flight from Vandenberg Air Force Base in California to the Tasman Sea. Now, the officials said, the United States will have to resort to the far more expensive alternative of putting the recording instruments on surface ships and sending them to the area.

The Air Force plans to conduct 20 tests of the MX from Vandenberg, eight from above ground and 12 from modified Minuteman silos. Seven tests already have been conducted. As the tests have become progressively longer with impact areas beyond the range of U.S. air bases, the military has been forced to seek the use of foreign bases for monitoring.