President Reagan and House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) yesterday agreed to a compromise congressional resolution that would allow U.S. Marines to remain in Lebanon at their current strength for 18 months while sidestepping a constitutional confrontation on the War Powers Resolution.

Congressional leaders of both parties predicted prompt passage of the measure despite objections from Senate Democrats who want a shorter time limit than 18 months, which would push the question of ultimate withdrawal beyond the 1984 elections.

"This resolution is flying," said House Foreign Affairs Committee Chairman Clement J. Zablocki (D-Wis.), who scheduled a hearing on the proposal for this morning. Secretary of State George P. Shultz is scheduled to testify.

In language crucial to congressional Democrats, the resolution declares that "U.S. armed forces participating in the multinational force are now in hostilities requiring authorization of their continued presence under the War Powers Resolution."

Reagan, in a two-minute statement issued on the South Lawn of the White House before he departed on a political trip to South Carolina, said he had "substantial reservations" about the declaration but would sign it if passed in its present form.

This statement, worked out in advance with congressional leaders, reserved the president's right to challenge the War Powers Resolution of 1973, which he maintains violates executive prerogatives.

O'Neill called the compromise agreement an "acknowledgment" of the validity of the 1973 resolution despite Reagan's reservations. From the administration's point of view, the compromise avoids a confrontation on the War Powers Resolution at little cost and promises to give Reagan the bipartisan backing he seeks for his policies in Lebanon.

Reflecting an opinion widely held at the White House, Rep. Samuel Gejdenson (D-Conn.) said the administration wanted to settle the dispute with Congress because "Ronald Reagan wants some coverage on Lebanon."

One key provision of the compromise agreement would permit Reagan to order "such protective measures as may be necessary to ensure the safety of the multinational force in Lebanon."

This provision, which sources said was discussed in some detail in negotiations leading to the final agreement, would allow continued offshore artillery and tactical air support for the 1,200 Marines engaged in Lebanon.

Administration officials said they believed this provision also would cover the sending of forward observers to positions held by the Lebanese army as long as the activity was related to protection of the Marines.

Administration officials insisted, however, that the broadly worded provision was not an attempt to widen the scope of Marine activities in Lebanon, which are limited by the resolution to continued participation in the peacekeeping role of the multinational force.

"You can't have language that is so specific it becomes involved in day-to-day order-giving," said one White House official. "But the legislative history of the resolution should make it clear that we're not expanding the role or mission of U.S. forces in Lebanon."

The resolution would limit the Marines to their current role and strength--approximately 1,200 troops on the ground in Beirut and 600 support personnel offshore--by making reference to agreements drawn up with Lebanon on Sept. 25, 1982, when the Marines became part of the multinational force.

O'Neill's agreement yesterday gave a stamp of approval to a compromise that had been worked out Monday by White House negotiators, Zablocki, Senate Majority Leader Howard H. Baker Jr. (R-Tenn.), Senate Foreign Relations Committee Chairman Charles H. Percy (R-Ill.) and other congressional leaders.

Senate Minority Leader Robert C. Byrd (D-W.Va.) continued to oppose the compromise agreement, saying he would not support it unless the administration could offer "a better justification" for the 18-month time limit.

Sen. Edward M. Kennedy (D-Mass.) also expressed reservations, saying that he regards the 18 months "as a blank check for far too long a period."

Though the White House would welcome the backing of as many Senate Democrats as possible, administration strategists made a deliberate decision late last week to virtually ignore Byrd and concentrate instead on winning O'Neill's support.

Reagan is assured of backing in the Republican-controlled Senate, but the White House needed O'Neill's and Zablocki's support to gain approval of the compromise resolution in the Democratic-run House.

On the House side, most Democrats--including some liberals who were wary of the agreement last week--seemed inclined to go along with their leaders and vote for the agreement.

Fifteen Democrats on the Foreign Affairs Committee met yesterday afternoon. According to Rep. Peter H. Kostmayer (D-Pa.), the group took an informal vote on the agreement. Three Democrats opposed the 18-month extension--calling for a six-month authorization instead--and the rest approved.

Such liberals as Tom Lantos (D-Calif.), Stephen J. Solarz (D-N.Y.) and Michael D. Barnes (D-Md.) said they would vote for the agreement. These three and other Democrats said they were reluctant, but felt they had no choice other than to give Reagan the extended authority.

"Eighteen months is a bad solution," said House Majority Whip Thomas S. Foley (D-Wash.).

"But any other time period is even worse. A year brings you into this again in the middle of a presidential campaign. Six months, or anything shorter, tells the other people fighting in Lebanon that they only have to bide their time and we'll be gone," Foley said.

One key influence on the House Democrats is their generally strong support for Israel. The Israelis are worried about upheaval in Lebanon and prefer to have American troops on the scene.

After being stalemated on the issue for the past 10 days, the congressional leaders now are eager to pass the resolution and put Lebanon behind them as quickly as possible.

Baker, who called the agreement "a major milestone, an achievement of a true bipartisan approach," said he hopes to complete Senate action this week. Final House action is expected next week.

On Air Force One en route to South Carolina, a senior official emphasized that Reagan believes he will be asserting "the inherent constitutional powers of the commander in chief" once he signs the resolution. Asked who had won the fight over the resolution, the official replied: "The country won."