The House Foreign Affairs Committee voted yesterday to authorize deployment of the U.S. Marine force in Lebanon for another 18 months, as House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) scrambled to reassert control over dissident Democrats unhappy with the authorization.

In an atmosphere of political and parliamentary confusion, with House members moving in many different directions, O'Neill set a floor vote for next week on the agreement he and congressional Republicans struck with President Reagan earlier this week on Lebanon.

That deal, which the committee approved yesterday, 30 to 6, calls for Congress to approve the 18-month authorization in a bill declaring that the deployment is covered by the War Powers Resolution, which gives Congress some authority over such deployments.

Reagan has agreed to sign such a bill while expressing disagreement with the application of the War Powers law.

About 80 Democrats decided yesterday that they will try to defeat O'Neill's agreement on the floor and substitute a bill that would bring the Marines home within 60 days unless Reagan, in a report to Congress, could persuade members that a longer deployment is needed.

Other Democrats said they would work for a deployment longer than 60 days but not as long as 18 months.

A bipartisan group of fewer than 10 members of the House Armed Services Committee was to leave last night for Lebanon. Led by Rep. Samuel S. Stratton (D-N.Y.), who has called for immediate withdrawal of the Marines, the group planned to return to participate in debate on the O'Neill-Reagan agreement.

Positions seemed slightly less fragmented in the Senate, where Majority Leader Howard H. Baker Jr. (R-Tenn.) appears to have most of his 10-vote Republican majority in harness to support the 18-month authorization.

But Senate Democrats and some Republicans continued to express doubts about that proposal.

O'Neill was fast and firm yesterday morning in responding to Democrats on the House Appropriations Committee, who revolted against his agreement Wednesday by passing an amendment that would cut off all funds for the Lebanon deployment as of Dec. 1.

The speaker sidetracked the amendment by shuffling it off to another committee, and chided Appropriations for "meddling."

O'Neill criticized Democrats who oppose his agreement, saying, "It's easy to run when you get dozens of phone calls to get the troops out of there."

Many House members have found that their constituents favor immediate withdrawal of the Marines. But O'Neill recognized he would have to deal with critics in his own party in order to retain his authority.

"We happen to be the party leaders here, and I hope to lead my party," he said.

Accordingly, his aides said, O'Neill is willing to let dissidents in his party have a chance to shorten or otherwise change the 18-month authorization when it comes to the House floor.

The potential tactical problem is that the White House may back out of the deal if the time period is shortened.

Foreign Affairs Committee Chairman Clement J. Zablocki (D-Wis.), who helped work out the agreement with the White House, said a presidential veto is "inevitable" if Congress does not permit a deployment for 18 months.

Several representatives said the House is nowhere near a consensus on the question as members are pulled one way by their party leadership, another by constituents and another by fear of endangering the Marines if the squabble continues too long.

"It's fluid as hell," Rep. Les Aspin (D-Wis.) said. "It's really a bad situation to legislate in, with guys changing their minds overnight."

Several members said they were disturbed by Wednesday's congressional testimony from Secretary of State George P. Shultz. Shultz sought to win support for the 18-month agreement, but his testimony apparently had the opposite effect.

The secretary indicated that Reagan has reserved the right to increase the size, mission or duration of the Marine deployment without congressional authorization.

"That can't make anybody feel comfortable," Rep. David R. Obey (D-Wis.) said.

The proposal that seems to have the greatest, albeit reluctant, support in the House is the 18-month authorization agreed to by Reagan, O'Neill and congressional Republicans.

"Nobody wants 18 months, but no one, no one, has a better solution," Rep. Daniel A. Mica (D-Fla.) said.

This position prevailed in the Foreign Affairs Committee yesterday as one member after another took pains to explain unhappiness with the proposal.

Persuaded, though, that they had no alternative, the members rejected two amendments to change the 18-month authorization and sent the measure to the House floor.

The emerging subplot to the dilemma over Lebanon involves efforts by O'Neill and Majority Leader James C. Wright Jr. (D-Tex.) to quell a mutiny among fellow Democrats.

After the dissidents passed their amendment in the Appropriations Committee Wednesday evening, Wright stormed onto the House floor to criticize party colleagues for their "peremptory unilateral action."

When the Democratic leadership team met yesterday morning, Obey, a leader of the dissidents, and O'Neill got into a shouting match about the Marine deployment.

Obey said he related, in what he later called "strong terms," his visit with a constituent, James Clark, whose son, Lance Cpl. Randy Clark, was killed by artillery fire two weeks ago while on duty in Lebanon.

Obey said Clark's father told him he was not surprised that Reagan would commit troops to Lebanon, although it was a mistake.

"But I expect you guys to do something about it," the senior Clark said, according to Obey.

Obey, a liberal Democrat who generally agrees with O'Neill, has argued that the Marines should be brought home immediately.

Reflecting the House's multi-faceted approach to the Lebanon situation, that view is shared by some of the most conservative House Republicans.

"Lebanon is a quagmire," said Rep. Toby Roth (R-Wis.), a strong conservative. "I don't think the American people want . . . our Marines to get killed in somebody else's war."

Rep. Bob Edgar (D-Pa.), another dissident, said he sympathizes with O'Neill's difficulty in providing leadership on such a confused issue.

"They want to take hold, but it's like trying to get a grip on Jell-O," Edgar said.