Secretary of State George P. Shultz told an audience of Israel's American Jewish financial supporters last night that Israel should reciprocate U.S. friendship "by a willingness to listen with an open mind" to the possibilities for peace inherent in President Reagan's Mideast peace initiative.

In his first public speech since assuming office, Shultz told a dinner meeting of the United Jewish Appeal in New York City:

"There will be no peace without Israeli security, but Israel will never be secure without peace."

He added, "When it comes to safeguarding the long-term security of Israel, the friendship and resolve of the United States are second in importance only to Israel's own resolution and strength.

"And, in the final analysis, that friendship and resolve deserve, in return, to be reciprocated by a willingness to listen with an open mind to the views of others."

Shultz' words underscored the administration's determination to reverse the curt refusal of Israeli Prime Minister Menachem Begin's government to consider the peace plan outlined by Reagan Sept. 1.

The U.S. initiative calls for Israel to freeze Jewish settlements in the West Bank and Gaza Strip and for these Israel-occupied territories to gain eventual self-rule "in association with Jordan."

The plan has been rejected by both sides in the Mideast conflict, explicitly by Israel and implicitly by the Arab League at a summit meeting last week.

However, the administration has made clear that it will not accept these rejections as final and intends to pursue its goal of making the Reagan initiative the springboard for a broadened effort at resolving 34 years of Arab-Israeli tensions.

A major element in the administration's strategy aims at winning support within the American Jewish community in hopes that it will help to bring about a reassessment and a more flexible attitude within Israel.

Some of the leading organizations and figures in Jewish communal affairs have said that, while they have questions and doubts about some parts of the Reagan initiative, they believe it deserves careful consideration and should not be rejected out of hand.

It was an effort to further the debate among American Jews that brought Shultz to the Helmsley Palace Hotel for last night's glittering dinner of the UJA, the principal fund-raising organization for Israel in this country. More than 200 of the guests at the dinner have pledged to give $100,000 or more to Israel this year.

The start of Shultz' speech was delayed slightly by a demonstrator attempting to enter the hotel ballroom. The man, who was screaming at the crowd, was identified as a member of Americans for a Safe Israel.

In his speech, which began and ended with about half the room giving the secretary a standing ovation, Shultz sought to allay concerns about the safety of the Jewish state by reminding his audience repeatedly that the president's plan envisons territorial adjustments that will give Israel defensible borders, that it calls for Arab states to recognize Israel's existence and right to live in peace and that the administration has promised not to use U.S. military or economic aid to put pressure on Israel.

The promise that the United States would not impose its views on Israel brought a round of applause, the first of four such interruptions, from the crowd of almost 300.

Shultz noted that Reagan, at the last minute, personally inserted into his nationally televised speech Sept. 1 a passage calling attention to the vulnerability of Israel's borders before it captured the West Bank and Gaza in 1967 and pledging that he Reagan was "not about to ask Israel to live that way again."

"Our vision of the future on the West Bank is one guided by a vision of a secure Israel living with defensible borders and by our abiding belief that it is not in Israel's long-term interests to try to rule over the more than one million Palestinians living in the West Bank and Gaza," Shultz said.

He continued: "The challenge Israel faces now is to combine diplomacy with power to build an enduring political settlement. There is nothing that says the Palestinian self-government in association with Jordan must lead inevitably to a Palestinian state. The president has said we will not support such an outcome."

Asserting that the administration intends to keep pressing its views, he added: "But let me be clear. We have a right to be heard, but we have no intention of using our support for Israel's security as a way of imposing our views . . . . The president has urged consideration of his proposals in the context of negotiations, to be undertaken without preconditions and with no thought of imposed solutions."

"That," Shultz said in reference to the call for a settlements freeze, "is why the United States particularly asked that the parties themselves not preclude possible outcomes by concrete and perhaps irreversible actions undertaken before the process of negotiation is completed . . . .

"We regard," he said to a stony silence, "the continuation of settlement activity as detrimental to the peace process."

He added: "There will be, as I have said, no imposed solutions: any point agreed by Israel and its Arab neighbors will not be opposed by us . . . .

"There is no need now to agree on any principle but one -- that is the need to come together at the bargaining table. To talk. To talk about differences; to talk about aspirations; to talk about peace.

"We ask for nothing more of any of the participants at the beginning of the process. And we have the deepest duty and obligation to ask for no less."

Staff writer Joyce Wadler contributed to this report.