The Reagan administration is working on a proposal to convince Israel to withdraw its troops from Lebanon in exchange for a strengthened U.N. peace-keeping force there.

U.S. officials said yesterday that the administration believes a strengthened version of the 7,000-man U.N. unit known as UNIFIL offers the best chance of satisfying Israel's insistence on an effectively policed demilitarized zone as the price for pulling out its forces.

U.S. ideas on the subject have been informally discussed at the United Nations with some of the major contributors to the existing UNIFIL organization and some members of the U.N. Security Council, according to official sources.

The topic is also likely to come up Friday morning in New York when Secretary of State Alexander M. Haig Jr. meets Israeli Prime Minister Menachem Begin in a session announced yesterday.

The administration, described as still seriously concerned about the threat of ongoing Israeli military action in and around Beirut, heard strong protests against Israel yesterday from two of its closest Arab allies, Egypt and Saudi Arabia.

Egyptian Foreign Minister Kamal Hassan Ali, emerging from a 45-minute meeting with President Reagan, warned of "higher risks of a larger conflict" arising from the Israeli invasion.

With Haig looking on, Ali told camera crews and reporters at the White House: "The situation in Lebanon and especially around Beirut is unacceptable. Israeli forces must withdraw from its vicinity. This besieged Arab capital and its government must become free to handle its own destiny."

Ali's statement was much stronger than one he made Monday at the State Department after an initial conference with Haig, and was also stronger than anything the Reagan administration has said publicly in recent days. The statement appeared to be an effort to convince the U.S. government and public as well as an international audience that Egypt is anything but relaxed about the effects in the Middle East of Israel's invasion.

Meanwhile, strong words from Saudi Arabia were transmitted to Washington from the traveling party of Vice President Bush, who met the new Saudi monarch, King Fahd, and his senior advisers.

Bush said before leaving Riyadh for home that Saudi Arabia strongly urged the United States to press Israel for immediate withdrawal of its forces. Foreign Minister Saud al Faisal told reporters the invasion threatens to erode Arab support for the United States "because it creates the view that the United States is at least ineffective in checking the Israeli aggression."

Reagan has also received two strong private messages in recent days from King Hussein of Jordan, according to U.S. sources. Hussein is reported to have expressed very great concern about the Lebanon developments and called on Washington to respond forcefully.

Reagan is to meet with Begin here on Monday; the meeting is being described in tentative language as "scheduled, as of now." The decision to make this meeting less than firm, as a warning signal to Israel, was taken at a National Security Council meeting late Monday, sources said.

The idea of strengthening the existing U.N. peace-keeping force in southern Lebanon, now under intensive discussion, is based on the belief that it would be much easier to build on an existing institution than to negotiate a new one, especially one that includes U.S. military forces.

The current plan is to obtain a temporary renewal of UNIFIL's mandate before it expires Saturday, and then to work with Israel, Lebanon, contributor nations and members of the Security Council to increase its authority and strength later.