The Reagan administration, although angered and dismayed by the Israeli invasion of Lebanon, yesterday sought to avoid public recriminations against Israel while keeping open the option of a tougher response later if the action appears to be endangering U.S. interests in the Middle East.

U.S. officials said privately that Israeli Prime Minister Menachem Begin's decision to strike at Palestine Liberation Organization forces in Lebanon could cause major new strains in U.S.-Israeli relations, both because of its potential for inflaming the Mideast situation and its effects on President Reagan's image at a time when he is touring Europe in an effort to establish his reputation as a leader of the West.

However, the officials continued, the top levels of the administration, while worried that they might have to react strongly unless Israel agrees to a cease-fire, have decided to keep their anger in check until they have a clearer idea of whether the Israelis will agree to a cease-fire or are bent on escalating the conflict to the point where there might be a danger of full-scale war involving Syria and possibly other Arab countries.

Specifically, the sources said, the administration is waiting for the assessment of Reagan's special Mideast envoy, Philip C. Habib, who conferred with Begin yesterday and who is to meet with him again today. According to the sources, Habib's mission is to convince Begin of the need for a cease-fire. His recommendations will be the major factor in determining whether the administration decides to try and rein Begin in by suspending U.S. arms deliveries or applying other sanctions.

According to the sources, the initial indications from Habib, following his first meetings in Jerusalem yesterday, did not leave much room for optimism.

He is understood to have reported that Begin is in a tough, uncompromising mood and unwilling to give any assurances about limiting either the scope of the Israeli military operation or the amount of time that Israeli forces will remain in Lebanon. U.S. officials privately estimated that, even if Israel keeps its promise to pull out of Lebanon, the type of operation being threatened by Israel is likely to take several weeks to complete.

However, the officials stressed, as of last night the administration thought it still had insufficient information to do much more than try to buy time by avoiding statements or actions that might antagonize the Israelis while Habib is still trying to gauge their intentions.

The main U.S. response was a carefully crafted statement, read by State Department spokesman Alan Romberg, that deplored "the spiral of violence" and then went on to declare in even-handed terms: "Israel will have to withdraw its forces from Lebanon, and the Palestinians will have to stop using Lebanon as a launching pad for attacks on Israel."

In Europe, Secretary of State Alexander M. Haig Jr., who is traveling with Reagan, told reporters, "We want the fighting to stop. We want the cease-fire to be reinstituted. We would like to see the central government of Lebanon strengthened and the border area made more secure."

But, in an acknowledgment of Israeli concerns, Haig also added: "We certainly do not misunderstand or misappreciate the vulnerability of the Galilee area to terrorist actions, rockets, artillery shelling of the kind that preceded the Israeli invasion."

In the meantime, a lot of the surface action common to crisis situations was evident here. Vice President Bush summoned senior officials to the White House for a crisis study of the situation, the State Department set up a special Lebanon working group, and the Pentagon dispatched five ships with 1,800 marines aboard to the eastern Mediterranean to help, if required, in a possible evacuation of the more than 4,000 Americans in Lebanon.

However, the officials conceded that, despite all this activity, they essentially are temporizing until Habib gives them a clearer indication of how adamant Begin is and what U.S. response is likely to be most effective in dealing with him.

At a press conference, the Israeli ambassador here, Moshe Arens, said he was unaware of any possible U.S. retaliatory steps such as cutting off arms shipments, canceling Begin's scheduled June 21 visit here or postponing a trip that Defense Secretary Caspar W. Weinberger is slated to make to Israel in August or September. U.S. officials, acknowledging that was the case, added, however, that these options might have to be considered in the near future if Begin proves intransigent in the face of efforts to end the fighting.