The tug-of-war between the administration and Congress over extending the deployment of U.S. Marines in Lebanon continued yesterday as the two sides tried again to reach agreement, and failed again.

After 90 minutes of futile negotiations with White House officials, Senate Democratic leaders strode onto the Senate floor and introduced a resolution demanding what President Reagan has refused to give: a presidential acknowledgment that the U.S. troops have been engaged in "hostilities" since Aug. 29, when two Marines were killed by shellfire. Since then, two other Marines have died.

If "hostilities" exist, deployment of the 1,200-man Marine contingent would be subject to limitations in the War Powers Resolution, which says that U.S. forces under fire overseas must come home within 60 to 90 days unless Congress authorizes a longer deployment or declares war.

The senators said that White House chief of staff James A. Baker III again rejected the Democrats' demand and asked for a broad congressional authorization of continued deployment with no conditions or deadlines.

In the House, meanwhile, Democratic leaders said their proposal, which would authorize continuing the deployment for 18 months with no conditions, was in abeyance. Junior House Democrats complained that the leadership proposal would free Reagan of any congressional control.

This battle over constitutional prerogatives is the first such standoff since the War Powers Resolution was enacted at the close of the Vietnam war. The law applies to any U.S. troops--except those stationed at U.S. bases--deployed overseas for more than 60 days.

The Marine force, which has been on the job in Lebanon for 11 months, is the first U.S. contingent to meet the 60-day test since American soldiers left South Vietnam.

Two sections of the resolution arguably could apply to the deployment of Marines in Lebanon. One says that U.S. troops engaged in "hostilities" must come home within 90 days unless Congress authorizes a longer deployment. The other section lets the president make decisions indefinitely with no need for congressional approval.

The current dispute comes down to an argument over which section of the law applies to the situation in Lebanon.

The Democratic senators' resolution, approved 29 to 0 in the Democratic caucus after yesterday's negotiations ended in stalemate, would state as a matter of law that the stricter provision--requiring explicit congresssional authorization for further deployment--is in effect.

The resolution seems to be more of a negotiating tactic than a serious legislative endeavor. It is a joint resolution, requiring passage by both houses of Congress and the president's signature to become law. If President Reagan vetoed it, a two-thirds vote of each house would be needed to override.

When Congress returned Monday from its five-week summer recess, there was talk in both houses of resolving the war powers question within a week or two. That seems unlikely now, because neither Reagan nor congressional Democrats seem inclined to budge, at least for the time being.

Legalistic disputes often are resolved by a compromise couched in language fuzzy enough to let each side claim victory. Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) proposed such an out yesterday; Democrats replied that Reagan must provide a "clear and specific" statement.

"It is the president's responsibility to tell the Congress specifically that the War Powers Resolution is in effect," Minority Leader Robert C. Byrd (D-W.Va.) said last night. "He must tell us specifically how long the Marines will be there, and for what mission."