Negotiations between the Reagan administration and Congress on continued deployment of U.S. Marines in Lebanon remained deadlocked yesterday as both sides backed away from a proposal by the House Democratic leadership to let the troops stay there through the spring of 1985.

Senate Democrats and some junior House Democrats balked at the resolution put forward by House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) to authorize President Reagan to continue deployment for 18 months.

They objected on grounds that it is a blank check with too few limits and no White House acknowledgment that the Marines are involved in hostilities of the sort requiring a report to Congress under the War Powers Resolution.

The White House was also dug in on the war powers issue, which involves not just deployment in Lebanon but also a basic question of prerogative: to what extent must the president consult Congress before deploying troops as commander-in-chief?

By evening, O'Neill was suggesting that he might withdraw what he called "our generous proposal."

"We extended the hand of friendship and support to the president, and he refused to pick it up," the speaker complained, suggesting that his next step might be to rewrite his proposal and "give them a shorter period."

Negotiations are to continue today. While the Democrats do not want to write the president a blank check, neither do they want to obstruct U.S. policy in Lebanon and risk being blamed for collapse of the President Amin Gemayel's government, which the Marines are supporting as part of a multinational peace-keeping force.

Some Democrats do favor a resolution authorizing deployment for fewer than 18 months.

Among the critics is Senate Minority Leader Robert C. Byrd (D-W.Va.), who said, "It is not the responsibility of Congress to just . . . pull out a time period and put that in a resolution."

Byrd called a meeting of the Senate Democratic Caucus, which issued a paper demanding that Reagan "recognize that operative provisions" of the War Powers Resolution have been triggered because the Marines are facing hostile action. The Democrats demanded that Reagan tell Congress how long he wants to keep Marines in Lebanon and define their mission.

Meanwhile, aides to Byrd and Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) met with White House officials Richard G. Darman and Fred F. Fielding to discuss a "working paper" that is to be a basis for eventual agreement on the contretemps.

Members of Congress said sentiment to bring the Marines home now is not strong in either chamber, even though opinion polls show that a majority of Americans would support that position. Instead, the argument between the president and Congress concerns terms under which Congress would authorize continued deployment.

The White House is resisting any effort to invoke the section of the War Powers Resolution that sets time limits on foreign deployment of U.S. forces. Members of both parties in Congress have insisted that Reagan agree that a limit took effect as of Aug. 29, when the first two of four Marines were killed near Beirut.

White House negotiators have also opposed any congressional effort to impose conditions on further deployment, such as limits on numbers of troops or a specified date by which they must return.

The proposal put together Tuesday by O'Neill and House Foreign Affairs Committee Chairman Clement J. Zablocki (D-Wis.) was designed as a compromise. It would permit Reagan to keep the troops on site for 18 months, plus an additional 60 days if acutely necessary, and would set no other conditions. It requires, though, that the White House acknowledge application of the War Powers Resolution to the situation.

Zablocki said yesterday that the 18-month authorization was chosen deliberately so the issue would not come resurface until after the 1984 elections. "Nobody wants to have to get in a squabble about this again in the middle of an election," he said.

Congressional leaders of both parties met yesterday morning with Secretary of State George P. Shultz and White House Chief of Staff James A. Baker III, and Zablocki emerged from the session saying, "We are not that far apart." Later in the day, though, a "final" negotiating session between Zablocki and James Baker was called off as the two side retrenched.

O'Neill had planned to bring his compromise resolution before the House next Tuesday, under rules that would permit debate but no amendments.

Some junior Democrats complained vigorously when they learned of this plan, and those complaints, coupled with pleas from House Democrats, apparently led O'Neill to think twice about his 18-month proposal.

On the Senate side, Thomas F. Eagleton (D-Mo.) was delegated by Byrd to prepare a Senate Democratic version of an authorizing resolution. The caucus tentatively approved the resolution yesterday, but Byrd said it would be reconsidered in light of the White House response to yesterday's developments.

At the State Department, spokesman Alan Romberg denied that Lawrence S. Eagleburger, undersecretary of state for political affairs, will travel soon to Israel to seek help in Lebanon.

Romberg said Eagleburger has long-standing intentions to visit Israel. Eagleburger met here yesterday with Israeli Ambassador Meir Rosenne.