D.C. Del. Walter E. Fauntroy, in the face of continuing criticism from the local jewish community, withdrew yesterday his controversial invitation to palestinian leader Yasser Arafat to speak in the United States.

Fauntroy said he was withdrawing the invitation, extended last month during a trip to Lebanon, because of an almost total rejection by the Palestine Liberation Organization of a Middle East peace plan proposed by Fauntroy and other leaders of the Southern Christian Leadership Conference.

All along, many of Fauntroy's friends and most of his detractors had doubted that Fauntroy could succeed in getting the PLO to agree to peace provisions it had rejected numerous times before. Some Jews said Fauntroy was being used.

One close acquaintance of Fauntroy said yesterday that the PLO rejection offered a convenient way out of a hopeless situation that could hurt -- but not ruin -- Fauntroy's already slumping political standing.

"They were mouse-trapped," the acquaintance said of Fauntroy and other SCLC leaders, "and they have now found a graceful way to get out."

Fauntroy said he had not been used, and likened the Middle East peace efforts to the early attempts to end segregation in the South. "If we had given up in those first days," he said, "we never would have accomplished anything."

Fauntroy announced the withdrawal of the invitation in an after-business hours speech on the floor of Congress.

"We are disappointed but not deterred from our goal," he said. "we have deferred for the present any plans to invite Mr. Arafat to the United States . . . We are looking for a greater evidence of a willingness to respond positively to our appeal for a comprehensive moratorium."

The SCLC peace plan had two provisions: that both Israel and the PLO, long arch enemies, would recongnize each other's right to exist; and that there would be an immediate end to violence in the Middle East.

Fauntroy had presented the proposal to Arafat during the meeting in Lebanon. The official response was delivered last Friday. In it, PLO agreed to a cease fire only in south Lebanon, Fauntroy said.

Israeli leaders had refused to meet with Fauntroy and the SCLC delegation while they were in Jerusalem. Yesterday, during a speech in which he at one point displayed fragments of an exploded American-made mortar shell found in Lebanon, Fauntroy also criticized Israel.

"We believe," he said, "that the cycle of fear has so affected the leadership of the Israeli government that they be in the dangerous position of being unable to ecognize a sincere momentum toward a comprehensive peace settlement in the region that is to their eventual benefit."

Nevertheless, Fauntroy claimed sereral victories yesterday for his peace mission. He said he had succeeded in "sowing the seeds of peace among parties which are at war," and in "dramatizing the evils of the backwards foreign policy and the need to move behind worn-out formulations and promises which are blocking a solution to the problem."

The United States in deference to Is Israeli sensitivities has pursued a policy of not talking with the PLO, even though it is the major representative of displaced Palestininas, because of the PLO's refusal to recognize Israel's right to exist.

Fauntroy also said that the peace effort had helped carve out a new and broader role for blacks in American foreign policy.

"We were successful in illustrating by our mission that we take seriously the material costs to us as black Americans in any area of the world where the United States has a vital stake," Fauntroy said, "and that we will not be silenced or excluded from participation in those decisions that affect our lives and the well being of this country."

Many black leaders had contended that U.S. support of the Israeli position has aggravated the fuel shortage in this country, which imports much of its oil from Arab states. Blacks have felt the brunt of the energy crisis, they said.

Fauntroy's contacts with the PLO sparked criticism in the local Jewish community, which includes many persons who had played key roles in raising money for Fauntroy's political campaigns. It also threatened to cause what many feared would be the most serious rift between local blacks and Jews since the 1968 riots.

Many black churchmen, community spokesmen, pro-Africa group and Arab-American leaders in the Washington area strongly defended Fauntroy's mission in the face of virtual unanimous opposition from area rabbis. When a polarization appeared to be developing, more moderate black and Jewish leaders and Mayor Marion Barry moved to ease the controversy.

Fauntroy's announcement yesterday appeared to provide a hollow ending to a once highly publicized and personal journey in which he had considered himself, he said, an "annointed ambassador." Less than half a dozen members of congress stayed to hear his 40-minute speech, and only four engaged in a brief dialogue about it.

Fauntroy became involved in the mission in late August, after the resignation under fire of Andrew Young as United States Ambassador to the United Nations.

Young, an old friend of Fauntroy, stepped down after the disclosure that, in violation of U.S. policy, he had met with Zehdi Labib Terzi, the PLO's observer at the United Nations.

Black leaders from throughout the country rallied to Young's defense, Fauntroy, the SCLC chairman, launched an effort with SCLC president Joseph E. Lowery to continue Young's initiative to establish a dialogue with the PLO.

Many observers also saw in the SCLC's effort an attempt to capitalize on the excitement following Young's departure to revitalize a foundering civil rights organization. Some local observers said that Fauntroy was appealing for long-sought national recognition, and also trying to solidify black support in the face of the first possible serious challenge to his congressional seat since 1971.