New Zealand and the United States failed yesterday to agree on a compromise proposal from Prime Minister David Lange's government for ending the dispute over nuclear vessels that has disrupted the Australia-New Zealand-United States alliance (ANZUS) since February.

After New Zealand Deputy Prime Minister Geoffrey Palmer presented the plan to Secretary of State George P. Shultz and Defense Secretary Caspar W. Weinberger, U.S. officials said last night that he had failed to convince them that the plan is sufficient to heal the rift and permit U.S. warships to resume calls at New Zealand ports.

The officials said the "parties didn't arrive at a mutually agreeable solution." But they added that both governments hope to find a way out of the impasse and will continue discussions. They did not specify when further talks might take place.

Sources familiar with the plan said it calls for New Zealand to adopt a law that would give Lange authority to determine whether a visiting ship is nuclear-armed or nuclear-powered without asking the United States to provide any clarification.

That would permit Washington to maintain its policy of refusing to confirm or deny whether U.S. warships calling at foreign ports are "nuclear-capable." But, the sources said, Palmer stressed that the Lange government does not regard this formula as a mere rubber-stamp way of permitting port calls despite its commitment to keep New Zealand free of nuclear devices.

As a result, the sources noted, the proposed formula could lead to situations in which New Zealand might decide that an American vessel is in violation of the nuclear prohibition and bar it from entering New Zealand ports. The sources said that would be unacceptable to the United States, which insists that continued naval cooperation with New Zealand requires an unrestricted right to use New Zealand ports.

"From our point of view, this is not a compromise although New Zealand has tried to put a sugar coating on its policy," said one U.S. official who asked not to be identified. "If anything, we regard this proposal as potentially worse than the current situation because its legislative feature would codify their non-nuclear policy into law."

The United States also refuses to make any concessions in its policy of not confirming or denying a ship's nuclear capabilities because of concern that such a move might spur antinuclear forces in Australia, Japan and Western Europe to demand similar treatment.

Lange's government, elected in July 1984 on an antinuclear platform, decided in February to bar port entry to an American destroyer, the USS Buchanan, because the United States refused to say whether it carried nuclear weapons.

The result was to halt full-scale functioning of the 34-year-old ANZUS alliance for naval cooperation in the South Pacific. The United States canceled all maneuvers with New Zealand, suspended the flow of intelligence information to Wellington, criticized the Lange government publicly for failing to fulfill its obligations as an ally, and forced cancellation of the annual ANZUS meeting, which had been scheduled for July.

Lange subsequently has sought to find a formula for coming to terms with Washington without alienating the powerful left wing of his Labor Party, which is committed to the antinuclear policy. However, public opinion polls have shown that New Zealanders, while generally favoring the nuclear ban, also strongly support the continuation of ANZUS and good relations with the United States.