In a move that could have major impact on U.S.-Israeli relations, several highly influential American Jewish organizations are warning Israeli Prime Minister Menachem Begin that he cannot count on their automatic support if his continued rejection of President Reagan's Mideast peace initiative leads to a confrontation with Washington.

Essentially, the message these groups are conveying in public statements and private conversations with Begin and other Israeli officials can be summarized in this way:

Although the commitment of American Jews to Israel's security and welfare remains constant, many are deeply troubled by such aggressive actions of the Begin government as its invasion of Lebanon, initial resistance to impartial investigation of the massacre of Palestinian civilians in Beirut, refusal even to consider the Reagan initiative, and defiance of U.S. objections to additional Jewish settlements in occupied Arab territories.

As a result, if Begin's inflexibility puts him on a collision course with Reagan, he could find the American Jewish community deeply divided and greatly hampered in its ability to muster the backing from Congress and American public opinion that is crucial to Israel's survival.

But there is general tacit agreement about distinct limits on how far American Jews should go in challenging Begin. Almost every prominent figure in Jewish organizational circles says flatly that any attempt by the administration to use U.S. financial aid to pressure Begin would immediately cause the vast majority to rally to Israel's defense.

There also is no evidence to indicate that Begin has been affected by the warnings. But these stirrings are of importance to the Reagan administration, which is working in part on the assumption that neither Begin nor the Israeli public can afford to ignore the concerns of the American Jewish community.

In the most recent example, the national board of the Union of American Hebrew Congregations, which represents approximately 1 million American and Canadian Jews affiliated with Reform Judaism, decided after extensive debate last weekend in Denver to postpone until the UAHC's 1983 convention a vote on a resolution that would explicitly reject the Begin government's long-range goal of absorbing the occupied West Bank and Gaza Strip into Israel and calling instead for a "territorial compromise" similar to that advocated in Reagan's initiative.

UAHC sources said there was a feeling the organization's rank-and-file members should have more time to consider the implications of the resolution, and a strong personal plea was made by Israeli Ambassador Moshe Arens not to do anything now that could contribute to perceptions of a split in the American-Jewish community.

UAHC's president, Rabbi Alexander Schindler, long has been regarded as one of Begin's closest friends in the American Jewish community. While voicing strong support for Israel at the Denver meeting, Schindler cautioned UAHC members not to make that country "our surrogate synagogue" and said American Jews should make greater efforts to "affirm our own identity" apart from Israel.

If the resolution is adopted next year, it would put the UAHC on the same general road taken recently by such organizations as B'nai B'rith International, the country's largest Jewish service organization, and the smaller, but highly influential, American Jewish Committee.

While expressing reservations about the Reagan initiative, both have said it deserves further consideration and should not be rejected out of hand. Last month, AJC President Maynard Wishner, summing up the consensus of his organization's executive council, also focused on the settlements problem by cautioning against "any action which would foreclose flexibility on the issue of territorial compromise . . . . "

While not all of them have stated their views publicly, Jewish community sources say the same attitudes have become clearly dominant within other organizations, including the American Jewish Congress, whose president, Howard Squadron, had been a publicly hawkish defender of Begin; the National Council of Jewish Women, and the Central Conference of American Rabbis, which represents the Reform clergy.

Since there are about 6 million Jews in the United States, there is much room for argument about the degree to which these groups are an accurate mirror of opinion within the Jewish community.

Still, their collective membership is composed largely of well-educated, high-income professional and business people who have been most active and articulate in influencing the American political process and public opinion toward support of Israel.

U.S. officials, looking to an expected meeting here between Begin and Reagan next month, have privately sketched a scenario that calls for Reagan to make clear to Begin that the United States will not back away from its proposal that the West Bank and Gaza, after territorial adjustments to protect Israel's security, be given eventual independence "in association with Jordan." Under this "territory for peace" formula, the Arab nations would be obliged to recognize Israel's existence and right to live in peace.

According to this scenario, if Reagan stands firm, Begin then must choose between risking a direct confrontation with the president or retreating into a more accommodating stance. According to the officials, their hope is that when Begin weighs these options, the realization that a continued hard line could shatter his support in the the American Jewish community might force him into the more moderate course.

However, Julius Berman, current chairman of the Conference of Presidents of Major American Jewish Organizations, said in an interview that he does not believe the administration's script will play that way. He disputed the notion that Begin will be forced to choose between confrontation or accommodation.

"They could simply agree to disagree," he argued, "and if that happens, there really isn't anything that President Reagan could do about it except to try and get tough by using the threat of cuts in military aid or something like that.

"But, if the administration does that," Berman added, "it will be going beyond the limits of what is acceptable to the Jewish community. Whatever split now exists will be healed, and all of the major Jewish organizations, whatever resolutions they may have passed, will unite again behind Israel."

Several other leaders of Jewish organizations expressed a different view. They said that awareness within Congress and the administration of disquiet among American Jews is, by itself, sufficient to hamper severely their ability to influence the government and to counteract the American public's growing sense that Israel is being too aggressive and unreasonable.

One with extensive experience in pro-Israel lobbying in Congress said, "Every time the Begin government announces it's building a new settlement on the West Bank and gets into a slanging match with the State Department, it just doesn't play. On every such go-around, you find that people on Capitol Hill are a little more irritated and a little less willing to listen to you sympathetically."

He asked not to be identified, as did many other Jewish organizational officials who were interviewed. In part, that reflects an ambivalence within the Jewish community about how to make American concern known to the Israelis without undermining U.S. support.

"When we poll our membership, which includes about 50,000 family units, we find a strong unanimity of opinion that Israel should relinquish its claims on the West Bank and seek a reasonable territorial compromise," said Don Feldstein, executive vice president of the American Jewish Committee. "But, among the same people, we also find a fairly strong resistance to the organization and its officers saying that publicly."

Feldstein and others agree that this reluctance stems from fear that public criticism by American Jews only tarnishes Israel's image further. That has led some to stiffen their protective attitude toward Israel.

"Among our members, we've found that many, especially in the smaller, more isolated Jewish communities, have regrouped and hardened their attitudes," said Warren Eisenberg, director of the international council of B'nai B'rith. "They are very resentful of what they feel is a one-sided, unfair portrayal of Israeli actions by the press, and they tend to equate any criticism of Israel with anti-Semitism."

But, Eisenberg added, in a view echoed by several others, "It's clear when you travel around the country that the events of the past few months have caused many Jews to lose their old one-dimensional, idyllic view of Israel. They still love Israel dearly, but they are troubled by things that they think are wrong."

Most of those interviewed agreed that such soul-searching increased sharply after the massacre by Lebanese Christian militamen of Palestinian civilians. But, while the massacre and issues such as the settlements question forced many organizations and individuals to speak out more forcefully, all have admitted that they have had no effect.

One man involved in the effort said: "It's Begin who makes the decisions, and so far he's not giving any ground. He thinks that when he comes to the States and gets a standing ovation from an Israel Bonds dinner, he has everyone on his side without any questions or doubts. This is a man who believes that the American Jewish community will be with him come what may and, when we win one for him, like a foreign aid bill in Congress, he uses that as an example of how he was right all along."

The question, for administration officials and for American Jews uncomfortable with Israel's course, is whether that will continue to be the case.

"At the moment, there's nothing immediate . . . that might force the issue," said Eisenberg of B'nai B'rith. "We have to see what happens in a number of areas--whether Jordan's King Hussein and the Arabs will respond favorably on their side to the Reagan initiative, whether the Israelis move out of Lebanon, what attitudes will be in the new Congress, and, finally, whether Reagan and Begin can reach some kind of understanding when they get together.

"Until all these sequences are played out, we won't really know whether the fears growing in the Jewish community about the future of U.S-Israeli relations are justified or not."