The House of Representatives yesterday rejected the argument that Lebanon may become another Vietnam and adopted a resolution authorizing President Reagan to keep U.S. Marines there as long as 18 more months.

After a full day of sharp debate, the House voted 270 to 161 to back its leaders and accept a bipartisan agreement they worked out two weeks ago with the White House.

The 18-month authorization will come up for final action today in the Senate, where approval seems likely but not certain.

The resolution authorizes deployment until March, 1985, if the president considers it necessary. But it also asserts that the deployment is controlled by the War Powers Resolution, a 1973 law giving Congress some say over the dispatch of U.S. troops to hostile areas in foreign countries. Reagan resisted such an assertion of congressional prerogative, but has agreed to sign the resolution anyway.

The 1,600-man Marine force was sent to Lebanon one year ago today as part of the multinational peace- keeping force. Five Marines have been killed--one by a mine, four by shellfire--and 38 have been wounded. The administration says it is not clear who is firing shells at the U.S. troops.

The Marine casualties, and the confused situation in Lebanon, prompted numerous House members from both parties to argue yesterday that approval of an extended deployment could be the first step in a gradual escalation similar to the early years of the war in Vietnam.

"Just as in Vietnam, we are asking young men to fight and possibly die in a war our government is not committed to win," shouted Rep. Sam Gibbons (D-Fla.), in a refrain heard throughout the day. "If we are there to fight, we are far too few. If we are there to die, we are far too many."

But House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), who addressed a hushed House chamber in firm but quiet tones at the end of the debate, said the analogy was not accurate.

"I believe," O'Neill said, "that this resolution clearly limits the scope and role of the United States forces in Lebanon so that the danger of a Vietnam-type escalation is avoided.

"I believe the president when he says he has no plans to change the peace-keeping role of our Marines," O'Neill said. "I believe he is expending every possible effort to see that differences in Lebanon are settled through negotiations, not through force of arms."

Though O'Neill received a standing ovation from colleagues in both parties when he finished his five-minute talk, he was still unable to hold a majority of his fellow Democrats in support of the agreement on the subsequent vote.

Democrats voted 134 to 130 against the 18-month authorization despite the resolute support it received from their party's leadership. The margin of victory for the Reagan-O'Neill agreement was provided by Republicans, who voted 140 to 27 for the resolution.

The president last night thanked the House for "its strong bipartisan vote . . . supporting our policies in Lebanon and the continued presence of the U.S. peace-keeping force . . . . A spirit of cooperation between members of the two parties and between the executive and legislative branches of our government has been the traditional hallmark of a successful foreign policy.

"Today, we continue the process of restoring that bipartisan spirit. Now, we look to the Senate for a similar demonstration of responsible leadership."

That final vote supporting the 18-month authorization came after the House defeated, 272 to 158, a counterproposal sponsored by Reps. Clarence D. Long (D-Md.) and David R. Obey (D-Wis.) that would have cut off all funding for the troops in Lebanon on Nov. 30 unless Reagan, in a report to Congress, persuaded the legislators to approve longer deployment.

The resolution that was passed allows the president to keep the Marines in Lebanon in their present role as part of the four-nation peace-keeping force. It prohibits him from expanding the size or mission of the Marine unit without prior congressional approval and requires that he report to Congress every two months on the Lebanon situation.

More than 100 of the 434 House members--an uncharacteristically large audience--were on the House floor for most of the seven-hour debate yesterday.

So many members wanted to take part that several were limited to speeches of only 30 seconds. The oratory covered the whole of American history and included quotations from poets ranging from Kahlil Gibran to Alfred, Lord Tennyson.

Rep. Toby Roth (R-Wis.) recited Tennyson's "Charge of the Light Brigade" in full during the debate, then told his colleagues that "this time, it is ours to reason why." Urging that the Marines be withdrawn immediately, Roth told the House: "Do not ride into that valley of death in Lebanon."

Despite the fairly wide margin of victory, the 18-month authorization was supported only reluctantly in many cases. A fairly common view was expressed by Rep. Ed Jones (D-Tenn.), who voted for the measure.

"I find it very difficult to commit our troops to 18 more months in a difficult situation overseas," he said. "On the other hand, I find it very difficult to vote to shirk our nation's responsibilities in the world."

The issued seemed less difficult for those who opposed the 18-month authorization. They blasted away zealously, arguing that the Marines are in a dangerous situation for no clear purpose.

"Nobody knows what the mission is," said Rep. Elliott H. Levitas (D-Ga.).

"Are we going to join in shooting at the Syrians or the Druze? What is our mission, other than being shot at?" he asked.

"Let's look at the political question," said Long. "How do you go back and tell your people that you voted to keep the Marines as sitting-duck targets in Lebanon for 18 solid months?"

Both the House resolution and the closely related one pending in the Senate today declare that the Marines' deployment in Lebanon is covered by a section of the War Powers Resolution that says U.S. troops cannot remain in a hostile situation in foreign territory more than 90 days without express congressional authorization.

Reagan has agreed to sign the resolution, although he will reserve the right to disagree about application of the war powers law. Several members said yesterday that this concession is a signal victory for Congress.

"No president ever before has recognized the validity of the War Powers act," noted Rep. Clement J. Zablocki (D-Wis.), chairman of the Foreign Affairs Committee and one of the chief House negotiators who struck the deal with Reagan. The specific section of the act that covers the Lebanon situation has not been applicable to any other deployment since the law was passed in 1973.

In his speech, O'Neill explained how the agreement with Reagan came about.

The speaker said that the president had initiated contact--a point O'Neill emphasized--after four Marines were killed by shellfire in Lebanon in two incidents about a month ago. When the speaker met with Reagan and his aides, O'Neill said, "they asked for and wanted an open-ended commitment."

"I said, 'Mr. President, no way are you or anyone else ever going to have another Tonkin Gulf' " resolution. But O'Neill said it was he who first proposed authorizing an 18-month extension of the Marines' deployment.

Noting that many of his fellow Democrats consider the proposal a sellout, O'Neill said he expects that the Marines will be brought home sooner because of "diplomatic efforts currently under way."

The speaker has been criticized on the issue not only by Democrats in the House but also by his fellow party members in the Senate.

Democrats in that chamber seem likely to vote today almost unanimously against the 18-month authorization. They have largely coalesced behind an alternative that would bring the Marines home in 60 days unless Reagan, in a report to Congress, persuades members to vote for a longer period.

But the Republicans have a 55-to-45 majority in the Senate, and if they stick together the 18-month proposal should prevail there as well. Several Republican senators have expressed concern about the 18-month plan, but only one, William V. Roth Jr. (Del.), has indicated he will vote against it.