White House and congressional negotiators reached tentative agreement yesterday on a compromise that would allow U.S. Marines to remain in Lebanon but limit their role and declare that they are subject to the War Powers Resolution, according to administration and Capitol Hill sources.

These sources emphasized that the agreement remains subject to approval today by House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), who is scheduled to meet with other House leaders to discuss the compromise.

If O'Neill fails to approve, they said, the delicately worded compromise is likely to collapse.

The agreement calls for Congress to declare that President Reagan should have invoked the War Powers Resolution of 1973 and declared that the Marines in Lebanon were in danger of "imminent involvement in hostilities" after two Marines were killed Aug. 29.

But it would go on to give congressional approval for the Marines to remain in Lebanon as part of a multinational force that also includes British, French and Italian forces. The White House disagrees with the congressional interpretation of the War Powers Resolution, which would have required Reagan to withdraw the Marines within 90 days unless Congress approved an extension. Like other presidents before him, Reagan says he believes that the resolution invades executive prerogatives.

The tentative compromise in effect sidesteps this constitutional issue. If Congress passes the measure, Reagan will sign it, but only after saying he disagrees with the congressional interpretation of what the War Powers Resolution requires.

This would leave the constitutional resolution of the issue to another day while giving Reagan the bipartisan support he is seeking for keeping the troops in Lebanon.

"Nobody wants to provoke a constitutional confrontation at a time of diplomatic and military crisis," said one administration official.

Other provisions of the agreement would seek to limit the scope of U.S. involvement in Lebanon and the duration of the commitment there.

One provision defines the mission of the Marines in terms of their participation in the multinational force and according to agreements signed with the government of Lebanon on Sept. 25, 1982. This would largely restrict the Marines to a peace-keeping role around Beirut.

But this provision contains a potentially controversial phrase that, in addition to the peace-keeping role, would allow U.S. involvement for "such measures as are necessary to protect the Marines."

This phrase would allow continued use of naval artillery and tactical air support, which the administration recently authorized in Lebanon and which has increased tension between the United States and Syria.

The compromise calls for limiting the size of the Marine force to approximately its present level of 1,200 men plus the offshore naval and air support. It would limit the duration of the U.S. involvement to 18 months, which would push the decision on withdrawal beyond the 1984 elections.

The administration is eager for speedy congressional action on the proposal, in the belief that a bipartisan declaration of support will be helpful to continuing U.S. diplomatic efforts to obtain both a cease-fire and a new power-sharing agreement in Lebanon.

But administration officials also are worried that a premature attempt to force a settlement will collapse and have a negative effect on administration diplomatic initiatives.

When it became evident late yesterday that O'Neill had not signed off on the agreement, the White House canceled a meeting today between Reagan and the congressional leadership.

O'Neill was unvailable for consultation yesterday afternoon, sources said, because he was golfing. A broadcast report of the reason for O'Neill's unvailability disturbed Republican negotiators, who were concerned that it would embarrass O'Neill and make him less likely to agree to the compromise.

Involved in the negotiations yesterday were representatives of House Foreign Affairs Committee Chairman Clement J. Zablocki (D-Wis.), Senate Majority Leader Howard H. Baker Jr. (R-Tenn.), White House chief of staff James A. Baker III, legislative liaison Kenneth M. Duberstein and presidential assistant Richard G. Darman.

Other congressional leaders were kept informed on the continuing negotiations.

The White House is hopeful that Senate Minority Leader Robert C. Byrd (D-W.Va.), who has argued that the War Powers Resolution should have been invoked on Aug. 29, when the first two Marines were killed in Lebanon, will support the compromise because it recognizes the validity of the law.

The administration's viewpoint is that Congress has been basically supportive on policy issues despite its sharply differing position on the War Powers Resolution.

Sen. Baker said yesterday that the war powers dispute is "about the legalisms" and that he didn't see "any real division on the fundamental question" of keeping the Marines in Lebanon.

But one administration official acknowledged that doubts are growing even in Republican ranks about the wisdom of a long-term U.S. military commitment in war-torn Lebanon.

"That dispute is muted now but it's certain to grow if there are more casualties," said one Republican congressman who asked not to be identified. "For this reason it's important to put this war powers debate behind us and express a semblance of unity behind the president."

The congressman said it was important for Reagan to "clearly define U.S. policy objectives" once the compromise resolution is passed.