The Reagan administration, holding firm to its demand that Israel withdraw immediately from West Beirut, was considering last night whether to send U.S. troops back to the Lebanese capital as part of a new, temporary peace-keeping force involving France, Italy and possibly other countries.

Administration sources, while stressing that no decisions had been made, said the idea was under active consideration because of the need for some force capable of preventing further tragedies such as the massacre Friday of Palestinian civilians and maintaining stability in Beirut until the Lebanese army can assert its authority. Sources said that was the main topic of a 75-minute, Cabinet-level meeting presided over by President Reagan yesterday morning.

The United States joined in unanimous approval by the U.N. Security Council early yesterday of a resolution condemning the "criminal massacre of Palestinian civilians in Beirut." Details on Page A19.

A working-level group of State Department and Defense Department officials was directed to study the options further and report to the White House last night, with the expectation that a decision on what course to follow and the timing of an announcement will be made today.

France and Italy, which participated with the United States in the multinational force supervising evacuation of the Palestine Liberation Organization from Beirut earlier this month, have announced willingness to send troops back in the wake of the killings.

However, the sources said, President Reagan was not ready, as of last night, to acquiesce in reconstituting the force unless he is satisfied that the need cannot be met through other means. These include stationing United Nations observers in Beirut or redeploying there the U.N. International Force in Lebanon (UNIFIL) from southern Lebanon.

In addition, the sources continued, the president insists that any U.S. participation be tied to conditions such as guarantees of cooperation from Israel and the Lebanese government and willingness of other countries to join the force.

In that connection, the sources said, the administration is exploring whether additional countries can be persuaded to send troops.

Reagan's caution about returning all or part of the 800-member U.S. Marine Corps contingent to Lebanon is understood to be rooted in concern about hatred between feuding Moslem and Christian communities there.

It could engulf the force in fighting that would result in U.S. casualties or shooting at Lebanese or Israelis.

The administration fears that such an outcome would stir a hostile reaction from the public and Congress. The possibility of U.S. Marines fighting Israeli forces or any of the Lebanese factions battling each other could create great difficulties for Reagan's hopes of getting Arab-Israeli cooperation on with his new Mideast peace initiative. However, some sources, acknowledging obvious risks in returning Marines to Beirut, said they think Reagan will be forced to conclude that he has no other choice.

They said expanding the number of U.N. observers in Beirut, as called for in a resolution adopted yesterday by the Security Council, or redeploying UNIFIL, whose severely limited powers made it ineffective in southern Lebanon, probably would not calm tensions in the city.

Complicating the situation, the sources added, is confusion on the ground in Beirut.

There were initial indications that Israeli Prime Minister Menachem Begin's government was ordering its forces out of West Beirut in the aftermath of the furor over the killings, but U.S. officials said late yesterday that the pullback may have stalled.

These officials said the United States hoped that last night's Israeli Cabinet meeting would indicate more clearly whether Israel intends to bow to the swelling, worldwide demand that it leave Beirut.

As of last evening, the officials noted, they had no clear sign of how Begin would react to the blistering language used by Reagan Saturday in demanding the Israeli withdrawal and implicitly holding Israel responsible for the massacre.

The sources described Reagan as adamant about withdrawal of Israeli forces that moved into West Beirut last Wednesday after the assassination of Lebanese President-elect Bashir Gemayel. There was no immediate way of telling whether his insistence will put the United States and Israel on what could be the worst collision course in 34 years of frequently stormy relations, they said.

Some sources said they hope that Begin, forced on the defensive by the horrified reaction in and outside Israel to the massacre, will elect to take a conciliatory course and not antagonize Reagan further.

Some sources even speculated that the situation might rebound to the U.S. advantage by forcing the Begin government, which has rejected Reagan's peace initiative, to be more flexible about the plan as a way of demonstrating Israel's desire for peace.

However, the sources also said the administration's immediate focus was on what one called "the hour-by-hour problem of restoring stability to Beirut" and not on what might happen later.

Yesterday's White House meeting initially was scheduled Saturday night as a small session involving only the president's top policy advisers, but Reagan later ordered it expanded to include the full national security decision-making machinery.

Attending with Reagan were Vice President Bush; Secretary of State George P. Shultz; Defense Secretary Caspar W. Weinberger; U.N. Ambassador Jeane J. Kirkpatrick; and CIA deputy director James McMahon.

Also, Adm. James Watkins, acting chairman of the Joint Chiefs of Staff; national security adviser William P. Clark and his deputy Robert McFarlane, and senior White House aides Edwin Meese III, James A. Baker III and Michael K. Deaver.