The Senate Foreign Relations Committee gave further proof yesterday of how Congress is split by the presence of U.S. Marines in Lebanon, reversing itself twice in 10 minutes before adopting by a single vote a resolution authorizing the president to keep the troops there up to 18 more months.

The resolution was first approved, 9 to 7, along party lines, then amended to allow only six more months of deployment as Sen. Charles McC. Mathias Jr. (R-Md.) switched to vote with the Democrats.

It was amended back to 18 months and shipped off to the Senate floor after Majority Leader Howard H. Baker Jr. (R-Tenn.) took Mathias aside in the committee room and persuaded him to switch back.

President Reagan has agreed to an 18-month provision. Mathias had been lectured on party loyalty by Vice President Bush before the committee met, and first voted with his fellow Republicans for the 18-month limitation, which passed, 9 to 7.

But when Sen. Claiborne Pell (D-R.I.) offered a Democratic proposal for six months instead, Mathias also voted for that.

The Democrats then pulled out a proxy from Sen. Paul E. Tsongas (D-Mass.), who was absent. With Mathias and the proxy, they prevailed, 9 to 8.

Had that vote stuck, the Republican resolution on Lebanon would have failed its first test in the Republican-controlled Senate.

There was silence in the committee room. Baker stared balefully at Mathias, then took him aside.

After several minutes of whispered conversation, Mathias returned to the committee table and announced he would change his vote. The Republican resolution prevailed, 9 to 8, and is likely to come up on the floor Wednesday.

Mathias explained afterward that he switched his vote because he feared "the whole agreement was coming unraveled real fast" after the six-month alternative was passed.

Mathias said Baker told him, "This thing is turning to ashes." Mathias said he changed his vote because "I thought it was important to at least get this to the Senate floor."

The "bipartisan agreement" on Lebanon that Reagan announced Tuesday calls for Congress to adopt a resolution authorizing the Marines to remain in the multinational peace-keeping force in Lebanon through March, 1985.

This would be done under the War Powers Resolution of 1973, which gives Congress partial control over foreign deployments of U.S. military forces. Reagan is expected to sign the resolution although he disagrees on the application of the War Powers provisions.

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) has endorsed the agreement, as has the House Foreign Affairs Committee. But many House Democrats are in revolt over it, and some Republican representatives have expressed reservations as well.

Among other things, members of both parties have expressed concern about testimony Wednesday by Secretary of State George P. Shultz. He came to Congress to sell the agreement, but evidently had the opposite effect.

Shultz indicated that the Reagan administration does not feel strictly bound by its agreement with Congress. He suggested that, despite limits in the agreement on the scope and duration of the Marine deployment, Reagan would be free to expand the force or its mission and keep troops in Lebanon longer than 18 months without congressional authorization.

"The secretary of state told us that our authorization is irrelevant," Sen. Alan Cranston (D-Calif.) complained during the committee meeting yesterday. "We can vote for 18 months or six months or whatever, but it doesn't mean anything. The question is moot in light of the Shultz testimony."

Sen. Charles H. Percy (R-Ill.) said he, too, was concerned about what Shultz said. "I do not believe that the president has authority to do anything more than is stated in our resolution," he said. Percy said he will take up the question with Shultz and the president.

The senators' debate before yesterday's committee votes revealed a broad range of views about what should be done with the Marine force. The 1,600-man unit was sent to Lebanon almost a year ago--the anniversary will be next Thursday--as part of a four-nation peace-keeping force.

Some members of Congress criticized the deployment from the beginning. The criticism grew stronger two weeks ago after four Marines stationed near Beirut airport in the midst of a civil war were killed by shellfire.

Cranston and Sen. Christopher J. Dodd (D-Conn.) said Congress should simply rest on a provision of the War Powers Resolution that says U.S. troops engaged in hostilities overseas must come home within 90 days unless Congress authorizes a longer deployment.

In voting for an extended deployment, Dodd said, "We're sailing off without any clear sense of what the policy is . . . . In a month or so, we're going to regret this."

Republicans argued that the 18-month provision will permit Reagan to pursue U.S. objectives in Lebanon.

But even among Republicans there was no strong statement of support for the agreement.

"We all agree it's a quagmire," Sen. Nancy Landon Kassebaum (R-Kan) said. "We all agree 18 months is a long time. But it's not a blank check."