President Reagan, seeking to shore up support in Congress for a resolution to continue the deployment of U.S. Marines in Lebanon, promised yesterday that he will not expand the size or mission of the Marine force without congressional approval.

In a letter to leaders in both houses, Reagan offered a pledge that his secretary of state, George P. Shultz, refused to make in testimony last week. "It would be my intention," he wrote, "to seek congressional authorization . . . if circumstances require any substantial expansion in the number or role" of the U.S. troops in the multinational peace-keeping force in Beirut.

Reagan's letter came as both houses moved toward votes this week on a resolution authorizing him to keep Marines on station for up to 18 more months and as notes of uncertainty continued to be heard all over Capitol Hill.

The 18-month plan has the support of the majority leadership in both houses--Republican in the Senate and Democratic in the House--but many members in both parties dislike some or all of the proposal, and most predictions of the outcome still are hedged.

Sen. Arlen Specter (R-Pa.) said yesterday that he would prefer to support his fellow Republican in the White House, but "the president's letter left me with more doubts than I had before" about whether the country could be drawn into a war.

The same fear was voiced dramatically across the aisle by Sen. John C. Stennis (D-Miss.), the Senate's senior member, who for years was chairman of the Armed Services Committee.

In an anguished speech that drew considerable attention, Stennis, who was unstinting in his support of the Vietnam war, said he could not vote for the 18-month resolution because it reminded him of the 1964 Gulf of Tonkin Resolution.

During Vietnam, Stennis recalled in sad tones, "the wisest and best men in the Senate would exclaim day after day . . . , 'How in the world did we let outselves get into this war?'

"I cannot vote for this resolution in its present form," Stennis went on. "I tried. I tried to rise to the occasion for the good of the country, but I could not go for it."

Reflecting the uncertainty about the resolution, the House Rules Committee voted last night to bring the question to the House floor today under terms permitting votes on an 18-month authorization, a two-month authorization or any period in between.

As the debate has continued, shorthand has developed for the complex arguments on both sides of the question. They are the "Gulf of Tonkin" side and the "cut-and-run" side.

The "Gulf of Tonkin" camp is made up of members who say they fear that Lebanon could turn into a new Vietnam--particularly if Congress gives the president an 18-month authorization to keep U.S. troops there.

The countervailing "cut-and-run" argument--as articulated yesterday by Senate Foreign Relations Committee Chairman Charles H. Percy (R-Ill.)--holds that "America cannot cut and run any time someone starts firing at us. We cannot just say that we're tired, that we no longer care about our responsibilities in the world."

In the House, where Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) helped negotiate the 18-month proposal, the Democratic leadership was in the curious position of defending a Republican president against attacks from fellow Democrats.

O'Neill and Majority Leader James C. Wright Jr. (D-Tex.) praised the president's letter yesterday. "I think it's an effective antidote to the fear that the president might have tried to ignore the terms of the 18-month resolution," Wright said. "I think it will reassure some members."

"I didn't find it reassuring," complained Rep. David R. Obey (D-Wis.). "It doesn't say what really will happen."

Reagan's letter was an attempt to recapture political ground lost after Shultz angered members in both houses last week with his testimony supporting the 18-month resolution. Under pointed questioning, Shultz essentially said Reagan would reserve the right to expand the size, mission or time of Marine deployment regardless of what Congress said in its resolution.

When the members reacted angrily, Senate Majority Leader Howard H. Baker Jr. asked Reagan to send a letter of "clarification." That letter came up with some fanfare yesterday; TV networks lined up cameras to record the letter's arrival--in a chauffeured limousine--on Capitol Hill.

Although it expressed an "intention" to get congressional approval before increasing the size or role of the U.S. force, the letter was somewhat more vague about the time. Reagan promised only to "work together with the Congress" if he decides after 18 months that Marine deployment should continue.

In Senate debate yesterday, Specter asked Percy whether U.S. troops in Lebanon were "at war" last week when American forces shelled enemy outposts that apparently were firing at the Marines.

"They weren't shooting BBs," Percy replied. "They were using cannon. They were using big guns. And that was a war."

If so, Specter asked, "aren't we ignoring our constitutional responsibility to be the branch of government that declares war?" Percy said the dictionary definition of "war" might differ from his personal view.