Secretary of State George P. Shultz, using the U.N. General Assembly as his pulpit, today made a new appeal for Israel and its Arab adversaries to accept President Reagan's challenge for "a fresh start on the road to peace in the Middle East."

But, even before Shultz spoke, the president's Mideast initiative drew a fresh rebuff from Israeli Foreign Minister Yitzhak Shamir, who warned, "Pressing Israel for more territorial withdrawals will not bring peace."

Their starkly contrasting speeches to the assembly underscored anew the long and difficult task facing the administration as it seeks to transform the initiative, announced by Reagan on Sept. 1, into the springboard for negotiations to resolve the Arab-Israeli conflict.

Shultz, asserting that "the task can truly be a labor of love," said the initiative, "with its formula of peace for territory," offers the best hope of satisfying the legitimate aspirations of the Palestinian people, while guaranteeing Israel's right to exist in peace and security.

However, Shamir reiterated the hard-line position of Prime Minister Menachem Begin's government, which has rejected the Reagan initiative's central premise that the Israeli-occupied West Bank and Gaza Strip be granted eventual self-rule "in association with Jordan." Shamir was speaking against the backdrop of worldwide outrage directed at Israel in the wake of the massacre of Palestinian civilians in Beirut, and his speech bristled with scorn and defiance for Israel's enemies in the world body.

The United Nations, he charged, has been transformed by the Arab countries and their allies in the Third World and communist bloc into a forum for anti-Semitism and hostility toward Israel.

Reiterating the Begin government's denial of responsibility for the massacre committed by Lebanese Christian militiamen, the minister said:

"The perpetrators of this crime are well-known. They were not Israeli. Nevertheless, there were manifestations of blind hatred and false accusations leveled at Israel from a number of quarters. Such actions are outrageous and must be condemned."

As has become customary when an Israeli official appears before the United Nations, the delegates of all Arab states, except Egypt, and their allies walked out of the hall when Shamir began speaking. As a result, 89 delegations were absent during his 30-minute speech, although 68 others, including the United States, remained seated.

Shamir did not refer specifically to the Reagan initiative. But he left no doubt that the Begin government, whose long-range aim is to incorporate the occupied territories into Israel, remains unyielding in its opposition to the goal envisioned by the U.S. plan.

Israel, he asserted, remains ready to negotiate an interim, five-year system of limited self-government for the Palestinian inhabitants of the territories and would welcome Jordan and other countries joining the negotiating process. But he stressed:

"We remain convinced that to focus now on what is beyond the horizon is a sure way of inviting failure. . . . There is no need to introduce new concepts and approaches. . . . We are not prepared to reopen the debate on issues, such as the creation of a second Palestinian state, which were rejected at Camp David."

Alluding to the Israeli contention that the Reagan plan is contrary to the Camp David accords, Shamir said: "Israel has cautioned against attempts to reinterpret, renegotiate or bypass them.

"Pressing Israel for more territorial withdrawals will not bring peace. Indeed, there is no peace without security, and further territorial amputations negate security. Those who think that they can weaken Israel while dangling the word 'peace' as a bait are deluding themselves."

Shultz, who spoke after Shamir, gave no sign of being discouraged by the Israeli position or by the failure of Arab governments to embrace the U.S. initiative.

Shultz sought to encourage Arab interest in the initiative through separate meetings today with Boutros Ghali, Egyptian minister of state for foreign affairs; Jordanian Foreign Minister Marwan Kasim and Syrian Foreign Minister Abdel Halim Khaddam. U.S. officials, while refusing to discuss specifics, said later they thought the meetings had gone well, especially with respect to the talks with Khaddam on withdrawal of Syrian forces from Lebanon.

Shultz's special emphasis on the Middle East came in the course of a lengthy discourse touching on virtually all major world issues. At one point, he stirred memories of his predecessor, Alexander M. Haig Jr., who built many of his speeches around what he called "the four pillars" of Reagan's foreign policy

But where Haig's four points had stressed a confrontational approach to East-West issues and a determination to reassert U.S. leadership in the world, Shultz redefined what he called "four ideas" in broader and quieter terms. He listed them as:

"We will start from realism; we will act from strength, both in power and purpose; we will stress the indispensable need to generate consent, build agreements, and negotiate on key issues; we will conduct ourselves in the belief that progress is possible, even though the road to achievement is long and hard."