The Reagan administration's policy toward the United Nations is under new management highly critical of the world body, and U.N. friends and critics alike are expecting the debate over the future to be more intense this year than ever.

When the United States pulled out of the United Nations Educational, Scientific and Cultural Organization (UNESCO) last year, the administration pledged to use the $47 million it saved to fund similar, better-run programs elsewhere. That money was never spent.

Now both the required U.S. "dues" to the United Nations and voluntary U.S. contributions to U.N. agencies have been slashed in President Reagan's fiscal 1987 budget beyond the requirements of the new Gramm-Rudman-Hollings budget control law.

The cuts, combined with the known views of the new administrative team, have caused speculation among U.N. defenders that the United States is slowly edging out the United Nations' door.

"Polls show that public support is as high as ever for the United Nations," said Edward C. Luck, president of the United Nations Association of the United States, a nonprofit educational and research group, "so rather than posing the question of whether we should get out, the critics are trying to pick a little here and there on the budget and basically disable the organization."

The new team at the Department of State's Bureau of International Organizations admits to a jaundiced view of the United Nations, but denies any hidden agenda.

"Just about any criticism that has been leveled at the United Nations, I am in nodding agreement with," said the bureau's head, Assistant Secretary of State Alan Keyes, in a recent interview. "The system is not meeting our expectations of it."

Keyes was a loyal part of former U.N. Ambassador Jeane J. Kirkpatrick's U.N. team for two years, specializing in economic and social issues, and is considered her protege. He is also the first person in his new job ever to have been posted at the United Nations, and is without doubt the chief architect of U.S. policy toward the United Nations now.

"I have a taste -- a thirst, if you like -- for what is going on behind the scenes there," he said. "You can assume that we will be less willing to support those parts of the U.N. system that are not cooperating in efforts to be more effective."

However, he said, "quitting the United Nations is not in my vocabulary." The departure from UNESCO was "the greatest single sign of engagement with the international system there had been in a long time" from the Reagan administration, he said, because it signalled "a healthy concern for effectiveness."

Other U.N. agencies are showing signs of managerial improvement, Keyes said, but more needs to be done. Budget cuts are like diets, he added: "they won't leave you less healthy unless you make the wrong choices."

The administration wants to slash its voluntary contributions to such agencies as UNICEF, the U.N. Environment Program and the U.N. Development Program by 30 percent, from $266 million this year to $186 million in fiscal 1987. It also wants to "rescind" another $40 million of the fiscal 1986 funding.

At the same time, Reagan's budget would cut U.S. assessed contributions to the United Nations, or the "dues" set by treaty that make up 25 percent of the world body's operating budget, by 16 percent, from $350 million to $293 million.

"There has got to be a tough, tight budget process, and I believe we have made some headway in getting that point across," said Secretary of State George P. Shultz in a recent congressional hearing on U.N. funding.

"There's no doubt that the United Nations has strayed from its original mandate," said Rep. Gus Yatron (D-Pa.), chairman of the House Foreign Affairs subcommittee on human rights and international organizations, at the same hearing. "However, the administration's requests . . . have been unrealistically low."

A $79 million chunk of the dues reduction is being withheld because of a 1985 law, authored by Sen. Nancy Landon Kassebaum (R-Kan.), that made part of the U.S. payment contingent on the adoption of a new voting system at the United Nations to give the United States and other major funders more power.

Such a system is not likely even to be considered any time soon at the United Nations, where voting is dominated by small, impoverished nations, but the Kassebaum amendment remains, in Shultz's words, "a very useful tool, and we don't mind using it."

The "weighted voting" goal is a top priority for Keyes. To help in his new job, Keyes has picked Roger Brooks, a vocal critic of the United Nations who is now the U.N. specialist at the conservative Heritage Foundation think tank, to be director of policy planning.

The appointment is still pending, but it makes people like Luck nervous because of Brooks' close association at Heritage with Charles Lichenstein, former assistant to Kirkpatrick at the United Nations.

An open advocate of U.S. withdrawal from the United Nations, Lichenstein gained fame briefly in September 1983 by telling U.N. members threatening to take the organization out of New York that the U.S. delegation "will be down at dockside waving you a fond farewell as you sail into the sunset."

Brooks denies he feels the same way.

"I am in favor of the United States taking a very serious look at the value of its membership in the United Nations and assessing it agency by agency, conference by conference, organization by organization," Brooks said in a brief interview last month. "I support renewing the foundations of the United Nations to make it more responsive to member states who contribute the most to the organization."

"Pressure for efficiency is not a policy," Luck said. "Reform is fine, but there has to be some kind of vision for what kind of organization you want, where you want it to go and how to get there that's feasible for the 158 other member states. Otherwise you are just denigrating the United States' role."