Pursuing its belief that the Palestinian guerrillas trapped in West Beirut can and should be forced to surrender to Israel's invasion army, the Reagan administration cast two isolated votes of support for Israel's actions in Lebanon against overwhelming majorities in the United Nations yesterday.

In an early morning ballot after 16 hours of discussion and attempts to reach a compromise, the administration broke ranks with its two oldest European allies and 12 other countries on the Security Council by vetoing a resolution calling for an Israeli withdrawal from Beirut's city center.

Late in the afternoon, the United States and Israel were the only two nations to vote against a non-binding resolution in an emergency session of the General Assembly that determined an immediate withdrawal of all Israeli forces from Lebanon and asked the United Nations to consider punitive actions if Israel does not comply. The resolution was approved by 127 other countries.

The Security Council veto came after a day of conflicting signals from a State Department caught up in the abrupt resignation of Alexander M. Haig Jr. It followed attempts by American allies to persuade Washington that it will stand accused of encouraging a bloodbath if its assessment that the Palestinians will lay down their arms rather than fight is wrong.

The 14-to-1 Security Council vote on the resolution, which France introduced and Britain, Japan, Spain and Ireland joined in supporting, also triggered sharp reactions from Arab moderates. Egyptian President Hosni Mubarak told reporters in Cairo that he was "greatly disappointed" by the American veto.

Egypt had strongly supported the French resolution, and Mubarak predicted that American relations with Arab governments would suffer unless a way is found to prevent a massacre of the Palestinians.

Saudi Arabia, in an official statement broadcast by Riyadh Radio, said the kingdom "deeply regrets the rejection of the French resolution, which could have led to positive and effective results in checking the grave deterioration of the situation in Lebanon and restoring security and stability."

The flurry of diplomatic activity in New York, the controversy and mystery around Haig's departure and the proclamation of another cease-fire in Beirut Friday night left the Lebanese crisis still centered on the question that has defined it since Israeli troops arrived at the outskirts of Beirut and cut the guerrillas' only possible escape route, the Damascus-Beirut highway.

The question, in its crudest form, is whether the 5,000 to 6,000 Palestinian guerrillas believed to be in Beirut will put down their guns and agree to the Israeli demand that they go unarmed to live in a country that does not border on Israel, or will be exterminated by the overwhelming Israeli firepower that has reduced much of southern Lebanon and parts of West Beirut to rubble during the past three weeks.

In putting forward and then redrafting its Security Council resolution in an effort to get American acceptance, France was acting on its belief that, in the words of one French official, "the Palestinians will not react like the Argentines did on the Falklands. They will provoke a bloodbath first, and the entire Western world, including the United States, will suffer from the consequences."

The French sought with the resolution to move the crisis out of an all-or-nothing gamble by allowing the Palestinians the face-saving device of moving back into refugee camps around Beirut with small arms, but they made it clear that the Palestinians would have to submit to Lebanese authority and U.N. control in Beirut if the Israelis were ever to pull back farther than the proposed six-mile disengagement zone.

State Department officials feared, however, that the resolution would encourage the Palestinians to think that they could still find a way to bluff their way out of the Israeli trap.

From the beginning of the crisis, Haig reportedly has argued that the kind of Israeli pounding Beirut has suffered this week will cause the Palestinians to crack and surrender, as the Israelis contend.

French officials were convinced after discussions with senior State Department aides early Friday that they could redraft their resolution to win American support or, at a minimum, a U.S. abstention instead of a veto. Foreign Minister Claude Cheysson appeared to be echoing French bitterness that the veto was cast when he described France's "deep regret" over the veto yesterday in Paris.

Reports from Beirut suggested that it was Haig's sudden decision to leave the Reagan administration that encouraged the Palestinians to continue their struggle, rather than the French action.

U.S. delegate Chalres Lichenstein cast the veto in the early-morning hours yesterday after a long night of telling journalists and other delegates that he had received no instructions from Washington. Diplomatic sources said Lichenstein sought a delay in the vote, but France refused, fearing it was a delaying tactic to gather other votes against the resolution.

The United States then did not go ahead with an alternative resolution it had floated in the day's discussions that would have retained many of the French provisions but made disarmament of the Palestinians a major condition for an Israeli pullback.

The council adjourned without scheduling further action.