The House Appropriations Committee voted yesterday to cut off all funding for U.S. Marines in Lebanon on Dec. 1 unless President Reagan certifies to Congress by then that the unit is governed by the War Powers Resolution, which prohibits such foreign deployments without specific congressional approval.

The 20-to-16 vote brought to the surface a mutiny by some House Democrats against their leader, Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), and the tentative agreement he and Republican congressional leaders have worked out with the White House.

If Congress approves the agreement, Reagan would be authorized to keep Marines in Lebanon for 18 more months without formally invoking the War Powers Resolution.

Rep. David R. Obey (D-Wis.), a leader of the protesters, said 78 House Democrats have signed a letter demanding the right to amend the agreement and shorten the 18-month period. The White House has implied strongly that it will back out of the deal if the time period is shortened.

Meanwhile, Senate Democrats challenged not only the 18-month authorization but also an "escape clause" in the document that would permit Reagan to take any "protective measures as may be necessary to ensure the safety of the Multinational Force." The Marines were sent to Lebanon last year as part of a four-nation peace-keeping unit.

Senate Minority Leader Robert C. Byrd (D-W.Va.) read the clause and said, "That's a hole that you can run Amtrak through."

Still, leaders of both chambers continued to move toward final votes next week on the agreement, which includes a congressional assertion that the War Powers Resolution is in effect regarding Lebanon. Reagan has agreed to sign legislation containing this assertion but has reserved the right to disagree with it.

Despite the mutiny, there were indications yesterday that the 18-month agreement can gain approval in both houses. In the Senate, where Republicans have a 55-to-45 majority, no serious revolt against the president seemed evident.

Many House Democrats said yesterday that they will support the deal, if only because they see no reasonable alternative. Further, several liberal Democrats who might normally oppose Reagan on foreign policy are supporting the Marine deployment because Israel wants the Marines on the scene and many House Democrats are strong supporters of Israel.

The measure approved yesterday by the Appropriations Committee demands that Reagan acknowledge application of the law. It reflects a general sense among some Democrats that the agreement is unsatisfactory and a specific feeling that the War Powers Resolution requires a presidential declaration of Congress' authority in the Lebanon matter.

Secretary of State George P. Shultz toured Capitol Hill yesterday to promote the agreement but suggested that the administration may not feel bound strictly by its terms.

Shultz indicated that Reagan might still have the right to expand the force or its mission without prior congressional authorization.

Testifying before House and Senate committees, Shultz diplomatically evaded questions on this point for most of the day.

Late in the afternoon, though, when Sen. Paul S. Sarbanes (D-Md.) asked whether the White House "is reserving its authority to increase the numbers and role and time period," Shultz replied that "the president has no intention of turning over to Congress his constitutional prerogatives as commander-in-chief."

"In the end, as a matter of principle, the president . . . can't let go," Shultz said.

Shultz said the escape clause in the agreement, authorizing Reagan to use any "protective measures," should not be a cause for concern. It is "nothing exceptional," he said. "It's strictly a matter of self-defense."

At a White House luncheon yesterday with editors and broadcasters, Reagan said that members of Congress who wish to invoke the War Powers Resolution with no extension of time for the Marine deployment are "very shortsighted."

He also said a cease-fire in Lebanon is still possible. Asked about the ultimate result of the conflict, Reagan said, "We obviously can't guarantee victory" to the U.S.-supported Lebanese government.

The Appropriations Committee vote on a possible funding cutoff was attached to a continuing resolution, an essential funding bill that must be passed by Oct. 1 to permit the government to operate after the start of the new fiscal year.

Obey, one of the amendment's co-sponsors, said the measure was offered partly as a symbolic protest against the agreement and partly to help force the House leadership to permit floor debate on a shorter time limit for the Lebanon deployment.

O'Neill had planned to bring the Lebanon resolution to the House floor under rules that would limit debate and amendments. The leadership has particularly tried to avoid amendments shortening the 18-month period, on grounds that they would jeopardize the deal with Reagan.

Shultz declined to predict that Reagan would reject the agreement if the time limit were changed. But, he said, "If you change the time, you change something fundamental."

Testifying with Shultz were Marine Corps Commandant Gen. Paul X. Kelley, who corrected reports that the Marines have 1,200 men ashore in the Beirut area.

On a daily basis, he said, the Marine force totals about 1,600 men. Another 2,000 Marines are stationed on ships or at nearby bases so the 1,600-man unit can be reinforced quickly, if necessary, he said.