Achievement of a Lebanon cease-fire appeared yesterday to be paving the way for congressional approval of a compromise resolution that would authorize President Reagan to keep U.S. Marines in the Beirut area for an additional 18 months.

As the Senate began debating the measure, some members who had reservations about the idea--among them Sens. Charles McC. Mathias Jr. (R-Md.) and Barry Goldwater (R-Ariz.)--said they now believe that the resolution will pass when it comes up for a vote by the full Senate, expected Wednesday.

However, a new cloud was thrown over the compromise because of congressional concern about recent remarks by Secretary of State George P. Shultz.

Senate Minority Leader Robert C. Byrd (D-W.Va.) and House Majority Leader Thomas P. (Tip) O'Neill Jr. (D-Mass.) said they interpreted Shultz's comments to mean that the administration reserves the right to expand the size, mission or time of the Marine deployment without congressional approval.

Byrd said Shultz was indicating that if Congress passes the resolution Reagan will sign it and then promptly toss it aside. O'Neill, saying "I would expect the president . . . to keep his word," said he will ask Reagan for his personal assurance that he intends to abide by the compromise agreement.

Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) also said he intends to ask Shultz, who is at the U.N. General Assembly in New York, to explain his remarks.

In the meantime, a partial clarification came from Robert H. Pelletreau, a deputy assistant secretary of state, who testified yesterday before the House subcommittee on European and Middle East affairs.

When Chairman Lee H. Hamilton (D-Ind.) asked about the situation, Pelletreau replied, "I assure you that these questions come from a mistaken impression." He added that Reagan signified his position in June when he signed a Lebanon emergency-aid bill and acknowledged his intention to comply with a provision in it requiring congressional authorization for "any future substantial expansion in the number or role of U.S. forces in Lebanon."

"There is no current plan to increase either the number or role of the Marines," Pelletreau said. Under questioning from Hamilton, Pelletreau also said the administration agreed that such steps as moving the Marines to Beirut's Chouf region or adding another battalion to the Marine contingent in Beirut would constitute a "substantial" change in the context of Reagan's June statement.

Both Pelletreau and Shultz, speaking at a news conference in New York, said that despite the cease-fire the 1,600 Marines can be expected to remain in Lebanon for some time. Shultz stressed that he doesn't envision a role for the Marines in overseeing the cease-fire and noting that trained U.N. observers are already in Lebanon, added: "It seems logical to us to make use of them."

With the cease-fire in place, congressional sources said it seems almost certain that the compromise resolution will pass the Republican-controlled Senate. In the House, which tentatively is expected to begin debating the measure on Wednesday, a sizable group of the majority Democrats are opposed, but O'Neill said he still supports the 18-month authorization.

Goldwater, who had called for the Marines' withdrawal, announced that he would vote for the resolution because he believes that "it is dangerous for Congress to intervene in military command decisions."