JAKARTA, INDONESIA, JULY 29 -- Secretary of State James A. Baker III, ending three days of talks here with the Association of Southeast Asian Nations (ASEAN), failed today to secure agreement on two divisive issues -- the U.S. policy shift on Cambodia and what to do with the more than 100,000 "boat people" stranded in camps in the region.

At best, Baker appeared to have bought some time from ASEAN countries on the refugee issue. These countries are threatening to turn away Vietnamese asylum seekers if the United States does not agree to the forced return of Vietnamese who are judged economic migrants rather than political refugees. Malaysia has already started pushing off refugee boats, and others are threatening to follow suit.

On Cambodia, Baker did succeed in smoothing some relations ruffled by Washington's abrupt decision two weeks ago to withdraw recognition in the United Nations for the three-party coalition fighting the Vietnamese-installed government in Cambodia and to begin direct negotiations with Vietnam on Cambodia. Baker also said the administration is considering opening a direct dialogue with Cambodian Prime Minister Hun Sen, who in the past has been derided as a "puppet" of Hanoi.

But ASEAN officials said they still disagree with that U.S. policy shift and would continue to support the resistance coalition if a vote is held when the U.N. General Assembly convenes in September. Those officials said the current coalition, whose strongest member by far is the radical communist Khmer Rouge, would probably win enough votes to retain the seat -- an embarrassing loss for the United States. Most European countries as well as Australia and Canada are likely to vote against the coalition.

Indonesian Foreign Minister Ali Alatas said ASEAN would try to head off that confrontation by seeking to persuade the resistance coalition, nominally headed by Cambodia's former ruler, Prince Norodom Sihanouk, and the Phnom Penh regime to agree to an interim "supreme national council" to oversee the country until elections could be held under U.N. auspices. That council, which would include individuals representing all four factions, could hold Cambodia's U.N. seat until the elections.

Baker said the United States would vote to seat the supreme national council at the United Nations, but he was at pains to explain why the council, with Khmer Rouge representatives, would be more acceptable than the current coalition, which includes the Khmer Rouge. In answer to a question, Baker said he saw "a significant difference," since any Khmer Rouge members of the council would first be required to renounce the use of violence and pledge themselves to free elections.

One U.S. official here conceded, however, that the question of the U.N. seat still poses a difficult choice for U.S. policy makers, since any representation that includes the Khmer Rouge is likely to reignite criticism from members of Congress and others who want the Khmer Rouge excluded from any new arrangement. "That is our dilemma," he said.

ASEAN officials said they were not optimistic that such an agreement on a council could be reached in the short time before the U.N. session.

"We continue to differ," Indonesia's Alatas told reporters today in assessing the Cambodia policy dispute. "We understand better the United States' position. . . . We have a different view."

ASEAN nations have opposed the government in Phnom Penh, regarding it as imposed by Vietnam which kept troops in Cambodia from 1979 until last year.

The continuing disagreements between the United States and its ASEAN allies, especially the impasse over the often emotional dispute between Washington and ASEAN over the boat people, turned what is usually a largely ceremonial gathering into a substantive discussion.

The six ASEAN countries -- Thailand, the Philippines, Malaysia, Singapore, Brunei and Indonesia -- are frustrated and angry over Washington's continuing reluctance to agree to forced repatriation of nonpolitical refugees -- believed to be about 80 percent of those currently in camps in the region.

They agreed on the record to consider Baker's proposal to send back a new category of refugee -- "those who do not object" to repatriation -- as opposed to those who actively resist being returned. But ASEAN officials and Western diplomats said the proposal was largely meaningless and a delaying tactic.

Alatas said he recognized "that the United States has moved in its position" by suggesting the new category and by saying the United States was "willing to work toward final resolution by end of 1992."

"We appreciate these moves," he said, "but in fairness and in all candor we in ASEAN think they do not yet go far enough. . . . The flood of boat people . . . is getting to the point that it's almost unbearable politically {and} economically."

Alatas said both sides would continue discussions to try to find a way to break the deadlock.

Baker left this afternoon for Singapore for talks on increased regional economic cooperation with largely the same group of ASEAN foreign ministers and those from ASEAN's Pacific "partners" -- including Japan, Canada, New Zealand, Australia and South Korea.

Those discussions will take place without the region's two largest economically oriented publications, the Far Eastern Economic Review and the Asian Wall Street Journal -- both published by Dow Jones & Co. -- because the Singapore government has barred their reporters from entering. U.S. officials said they had "strongly" protested the ban against the journalists.