A whispering campaign designed to single out Ambassador Donald McHenry as the main culprit in the American policy debacle on Israel burst into the open today as New York's Mayor Edward Koch charged McHenry and his predecessor, Andrew Young, with being "Third World-oriented and viciously anti-Israel."

The rumors had begun 10 days ago, immediately after the U.N. Security Council vote cast by McHenry for a resolution deploring Israeli settlement policy in the occupied Arab territories, "including Jerusalem." Then the White House disavowed the vote as a failure in communication for which Seceratary of State Cyrus R. Vance took the blame.

In an interview with the New York Post today, Koch lumped Vance and National Security Adviser Zbigniew Brzezinski with McHenry and Young as being "anti-Israel." At a press briefing later in the day, he added Assistant Secretary of State Harold Saunders, calling the officials a "Gang of Five" against Israel.

Four of the five had no immediate comment on Koch's remarks, and White House Press Secretary Jody Powell said: "We would not agree with that assessment."

Vance rejected Koch's claim as "absolute baloney." He added: "Ed Koch knows damn well . . . that I am not anti-Israel, never have been, never will be."

Although White House officials exempted McHenry from any responsibility for the mixed signals, the rumors persisted:

That McHenry had gone over Vance's head to the president to press for a yes vote; that he had hidden from the White House the true implications of the resolution's text; that he had developed an antipathy toward Israel for its support of South Africa in the days when he negotiated for an agreement on Namibian independence; that he and the Israelis had not hit it off during his swing through the Middle East immediately before the U.N. debate.

All these allegations were angrily denied by McHenry as "totally unfounded and irresponsible" at a pres conference here a week ago. the rumors also were dismissed by his diplomatic colleagues here, who spoke of his obsessive objectivity and punctilious attention to diplomatic norms and channels as a self-professed "child of the State Department."

The rumors originated, according to several sources here and in Washington, with the Israeli diplomatic establishment. The Israelis deny it, but make no bones about their view that, as one said, "his attitude is not one of friendship toward Israel."

The matter seemed to subside late last week, and several leaders of the American Jewish community took pains to point out that an informal decision had been made among them not to go after McHenry, who is black. They expressed fears that a new rift between American blacks and Jews might result, similar to the ugly brawl over the Andrew Young resignation last summer which proved traumatic to both sides.

"The Young experience was not traumatic for the Israelis; for them, Jews are supposed to have problems with blacks," said one voice of moderation within the American Jewish establishment. "But for us it is traumatic. And I am not prepared to tear down our relationships with the black community to make the job of some Israeli ambasssador easier."

One Jewish leader, Nathaniel Saperstein of the National Council of Young Israel, appealed publicly for Jews to refrain from critizing McHenry because "it is clear that he was acting under instructions."

But over the weekend, The New York Times published an editorial suggesting that some "diplomatic guerrillas" in the administration -- unnamed -- had tried to "sabotage" President Carter's Middle East policy. Newsweek magazine ran an item attributed to Israeli sources detailing an alleged episode of ill-feeling between an "arrogant" McHenry and Israeli prime minister Menachem Begin during the U.n. ambassador's visit to Israel.

These, said one member of the American Jewish leadership in New York, "seemedd to be signals that it was kosher to go after McHenry."

At a meeting between the Jewish leadership and Carter campaign manager Robert S. Strauss yesterday, there were repeated questions as to whether McHenry shared the view of U.S. policy that Strauss was defining. wOutside the meeting, protesters denounced the Carter policy, and three were arrested.

"What changed?" asked one participant in the meeting. "All those people were screaming. Suddenly a scapegoat was needed. No one believed that Vance filled the bill. McHenry was the only alternative."

After the Koch interview appeared today in the New York Post, American Jewish Congress executive director Henry Siegman went public with the view that McHenry was anti-Israel.

The ambassador's view of all this, spokeswoman Jill Schuker said, is that he is "saddened and outraged." So far, she said, there has been no reaction from the black leadership, although others close to McHenry fear that this will be forthcoming, and that what one called "the Andrew Young mess" will recur.

Schuker painstakingly denied the rumors point by and insisted that Mayor Koch's allegations are "obviously based on a lack of information or a misunderstannding."

"He had his views and judgement," she said of McHenry's input on Middle East policy. "Operating at his level he is paid to give them to the president. I will not say what that judgement is, but the idea that it is an emotional or visceral judgement is absolutely false and unacceptable."