President Reagan addressed two strongly worded messages to Israeli Prime Minister Menachem Begin last night and today urging withdrawal from Lebanon, but the appeals were rejected, administration officials said.

The Israeli rebuffs caused dismay among administration officials and prompted Secretary of State Alexander M. Haig Jr. to cancel plans for a peace-making trip to Israel.

Haig said that he had turned down an invitation from the Israeli government to make the trip because the Israelis had not demonstrated "sufficient flexibility" in dealing with U.S. requests for a cease-fire and withdrawal.

The appeals--which one administration official described as close to demands--were contained in "Dear Menachem" messages from President Reagan to Begin. White House adviser Edwin Meese III described Reagan's messages as "very firm. It was a persuasive argument . . . crafted by the president himself because he feels he knows how best to get the attention of Prime Minister Begin."

One White House official described the Israeli response as "unsatisfactory" and said it accepted the idea of a cease-fire but not withdrawal from Lebanese territory. Another official acknowledged that Begin's response had left the administration without a clear course of action.

Frustration over the Israeli posture became visible today in small ways. Haig said that he no longer knew what the Israeli war aims are in Lebanon. Meese said that the Syrians had "shown restraint" in response to the invasion. Another White House official said the president was "personally disappointed" at Begin's reply.

Administration officials acknowledged also that Arab concern over the failure of the U.S. effort was increasing. Reagan conferred today with Saudi Arabia's foreign minister, Prince Saud al-Faisal, a member of the ruling royal family.

After the meeting, the prince told reporters: "What we are expecting from the United States is a clear sign of its position as regards this unprovoked and premeditated aggression on the part of Israel."

Haig acknowledged today that Soviet President Leonid Brezhnev had sent Reagan a message containing "a frank expression of Soviet concern about the widening conflict in Lebanon." Reagan sent back a message that was "responsive to the tone of the letter," Haig said.

The lack of success of Reagan's initiative brought into the open rarely expressed administration dissatisfaction with Begin's policies. The Israeli response also exposed some of the differences within the administration on how far the United States should go in pressuring Israel to halt its invasion.

The administration's frustration is all the greater because of the restraint it has shown this week in response to the Israeli attack--restraint so great that is has at times seemed to favor Israel. Wednesday, Haig explained the Israeli attack against Syrian missile sites as a reaction to a substantial infusion of new Syrian missiles into the area--reasoning that echoed the Israelis' own statements.

Among administration officials, Haig is considered the most pro-Israel and National Security Adviser William P. Clark the most skeptical of Israeli intentions. But skepticism is becoming increasingly prevalent now in all quarters of the administration.

The policy of restraint toward Israel, nevertheless, carried over into today's North Atlantic Treaty Organization summit meeting, at which U.S. reluctance to adopt a resolution condemning Israel was one of the reasons the summit participants decided not to issue any statement on Lebanon.

Instead, NATO Secretary General Joseph Luns expressed the "concern" that each member nation felt about the situation--a comment similar to those made frequently by U.S. officials throughout the week.

According to administration officials traveling with Reagan, one option under active discussion is a cutoff of U.S. military aid to Israel, which under an agreement between the two nations, is supposed to be reserved for defensive purposes. The difficulty with this approach--used last year after the bombing of Beirut when the administration delayed delivery of F16 jets scheduled to be sent to Israel--is that relatively few weapons are now in the pipeline, an administration official said.

Another option that has been discussed is an Israeli request for U.S. participation in a peace-keeping force patterned after the one used in the negotiated Israel withdrawal from the Sinai. While such a move, which involves placing U.S. soldiers in a potential combat situation, is considered unlikely, the fact that it is under discussion underlines the frustration of the United States in its dealings with its militant allies in the Middle East.

In Washington, however, senior administration officials said the use of sanctions to pressure Israel has been ruled out for the moment as neither desirable nor feasible, although there is strong sentiment for the idea in some parts of the administration, notably in the Pentagon.

For the moment, the administration is left with no option other than to keep pressing for a cease-fire, the officials said. That hope, however, is dampened considerably by the fact that the administration has renounced, at least for now, its toughest leverage: the threat of cutting off Israel's arms supply or imposing other sanctions.

Administration officals professed admiration for the planning and execution of the Israeli operation in a purely military and professional sense, but said there is a feeling that the Israelis deceived them, first in launching the strike and then in concealing their real intentions.

The officials said the administration is also concerned about what is perceived as a growing sense in U.S. public opinion that Israel has overreached in a clearly aggressive and duplicitous manner. The fear is that this will translate into eroded support for Israel.

Thus, while the United States now realizes that pressure on Begin has been ineffective, there still is no consensus on what to do to force Israeli forces back to their own borders. And there is a growing concern that the war will widen unless Israel is persuaded to withdraw.

"We just don't know what to do," a White House official said in Bonn.

One concern of the Reagan administraiton is that moderate Arab states will retaliate against the United States if the U.S. government is unable to bring about an Israeli withdrawal. Already, Arab newspapers are calling for Saudi Arabia to cut off or reduce petroleum supplies to the United States.

The Israeli invasion has cast a long shadow over Reagan's trip to Europe, which ends Friday with a visit to the Berlin Wall.

One of Reagan's themes throughout the trip, in defending British action in the Falklands, has been that nations should not enjoy the fruits of aggression. For this reason, the president opposed any diplomatic efforts that would have allowed Argentine forces to remain in place in the Falklands after a cease-fire.

But Reagan now is faced with Israeli insistence on a similar course of action in at least part of the territory that Israel has captured in Lebanon.

Haig and Clark--and the president, as well--have been preoccupied this week with the invasion of Lebanon during a time when they had expected to focus entirely on the thorny defense and arms control issues taken up today at the NATO summit.

While the administration is generally happy over the results of that summit, the president will return home without a clear idea of what the United States should do to persuade Israel to respect the rule of international law.