Secretary of State Alexander M. Haig Jr. succeeded yesterday in apparently forestalling an Israeli veto of European participation in a Sinai peace-keeping force as Israeli Foreign Minister Yitzhak Shamir returned home voicing optimism about a compromise that an Israeli official said is built around a proposed new U.S.-Israeli declaration.

"There are prospects that we will agree on some proposals," Shamir said after a 7 1/2-hour meeting with Haig that had been scheduled to end after lunch but lasted until well after sundown.

"I will bring several ideas to the meeting of our cabinet, and our cabinet will decide," Shamir said.

The Israeli cabinet is scheduled to discuss the Sinai force tomorrow in a session at the hospital room of Prime Minister Menachem Begin, who is recovering from surgery on a broken leg.

Begin has said he would recommend to the cabinet that Israel veto participation in the Sinai force by Britain, France, Italy and the Netherlands because of statements by those nations that he considers contradictory to the Camp David accords.

The European nations have issued a series of statements calling for, among other things, self-determination for the Palestinians, which the Israelis view as a call for creation of a Palestinian state.

But the Israelis emerged from the meeting with Haig sounding new optimism that the impasse can be resolved.

According to an Israeli official, the two sides are considering a compromise in which the United States and Israel would declare jointly that, in effect, participation of the European nations in the multinational Sinai force does not detract from the Camp David accords and does not bring into the process any ideas contained in the European statements.

"Now it depends upon Begin himself," the Israeli official said. He added: "Israel wants European participation in the peace-keeping force. Israel has always wanted this. But the question is: what is the price? "

Haig has maintained that it was important to broaden the Camp David process to include Europeans and that their willingness to join the Sinai force marked the first and best opportunity to do so. He reportedly has argued that an Israeli veto of European participation would be "tragic."

Yesterday, on the threshold of a major diplomatic victory, Haig emerged from his meeting with Shamir offering smiles and optimism but no specifics. He called his discussions "very extensive . . . very constructive and valuable" and "both timely and extremely helpful."

If Israel vetoes participation of the Europeans because of their recent statements, Australia and New Zealand reportedly will withdraw from the force.

This would leave the group only nominally multinational, with the United States carrying the bulk of the burden, along with contingents from Fiji, Colombia and Uruguay.

The United States has agreed to supply enough troops to comprise almost half of the 2,400-person Sinai force.

Before yesterday's meeting, the Israeli cabinet had been expected to veto European participation. Shamir declared before his arrival here that the Europeans' conditions were "unacceptable."

But after yesterday's lengthy meeting, Israeli officials spoke with what they conceded was new optimism. "They are coming out with something," one Israeli official said. "It's not lost. That much is certain."

This official also spoke with new encouragement about the possibility that European nations could be included in the peace-keeping force provided for in the Camp David accords.

"It may help," he said. "At least, they won't be involved in any anti-Camp David activity."

Under the accords, Israel is scheduled to return to Egypt in April the remaining Sinai territory it has occupied. Israeli Chief of Staff Lt. Gen. Rafael Eitan warned Thursday that Israel would not withdraw from the Sinai if a multinational peace-keeping force was not on the ground there by April, as the accords provide.

Haig and Shamir said yesterday that they also discussed U.S. and Israeli proposals for strategic cooperation, the situation in Lebanon and the overall peace process in the Mideast.

The United States and Israel recently exchanged widely varying drafts outlining a new agreement for strategic cooperation.

The Israeli draft called for sweeping U.S. commitments to preposition equipment including tanks in the region and to station other equipment there that would allow U.S. planes and ships to operate from Israeli airfields and ports.

The U.S. draft was far more limited, providing for prepositioning some medical supplies and limited amounts of military materiel.

These matters were discussed yesterday, according to an Israeli source, but no accord was reached, as the talks focused mainly on the Sinai force.

Shamir said there was no linkage between the discussions on the Sinai force and the strategic cooperation agreement.

The semi-official Egyptian newspaper Al Ahram reported today that Israel will drop its objections to European participation in the peace-keeping force in return for more U.S. aid and "privileges," United Press International reported.

The newspaper said Israeli Defense Minister Ariel Sharon would press for the exchange in a coming visit to Washington for talks on U.S.-Israeli strategic cooperation.

Al Ahram said, "Israel will exert diplomatic efforts in order to get more privileges during the strategic cooperation talks so that the price for acceptance of Europe's participation in the force would be getting privileges and bigger aid from the United States in return."