The United States, expressing "grave concern" at New Zealand's denial of port access to a U.S. warship, yesterday canceled participation in a naval exercise of the Australia-New Zealand-United States alliance (ANZUS) and warned that it might retaliate against New Zealand.

The dispute, which White House spokesman Larry Speakes said "goes to the core of our mutual obligations as allies," raised doubts about whether the three nations would be able to continue the 34-year-old pact for mutual defense in the South Pacific.

It also exposed Reagan administration fear that New Zealand's action, prompted by strong antinuclear feeling there, could have a ripple effect among other U.S. allies.

The State Department, charging that antinuclear movements "seek to diminish defense cooperation," warned: "We would hope that our response to New Zealand would signal that the course these movements advocate would not be cost-free in terms of security relationships with the U.S."

U.S. officials refused to identify countries where they fear similar situations could occur. However, the administration is known to feel that New Zealand's example could embolden antinuclear forces in Australia and Japan to create pressure for similar restrictions on U.S. warships.

On Monday, New Zealand's Labor government refused for the second time to permit the destroyer, USS Buchanan, to dock there next month as part of joint maneuvers called Sea Eagle 1-85. New Zealand said it did so because the United States would not say whether the destroyer carries nuclear arms.

The dispute forced Australian Prime Minister Robert Hawke, whose country was to have been host for Sea Eagle, to cancel the maneuvers, and Washington then made its withdrawal announcement. Hawke arrived here yesterday on a scheduled visit, and the ANZUS crisis is certain to dominate his talks scheduled for Thursday with President Reagan.

Australian officials in Canberra said the three governments want the alliance to continue and would begin immediate explorations of how defense interests "can be separately pursued."

Australian Defense Minister Kim Beazley said that would mean replacing the alliance's traditional three-cornered maneuvers with bilateral U.S.-Australian and Australian-New Zealand exercises.

U.S. officials here were reluctant to say yesterday whether the administration wants to continue a partnership with New Zealand that would be circumscribed and indirect.

ANZUS has operated on the premise that the United States would extend its military umbrella to Australia and New Zealand in return for facilities and port access for U.S. warships.

State Department spokesman Bernard Kalb, replying to questions about whether New Zealand might be dropped from ANZUS, said:

"We are considering the implications for our overall cooperation with New Zealand. This consideration will be broad-ranging. But, beyond that, I cannot go into any area of itemization."

New Zealand Prime Minister David Lange, whose Labor Party won power last summer with a program of trying to keep the South Pacific free from nuclear confrontation, said yesterday that cancellation of the exercise was "an utterly expected consequence."

He added, "Certainly it is not the view of the New Zealand government that our policy is hopeful of being changed or backed down upon because of actions like that."