In a setback for the Reagan administration's hopes that an agreement for withdrawal of foreign troops from Lebanon could be reached by the end of the year, Israeli Foreign Minister Yitzhak Shamir said today the negotiations could drag on for months.

Shamir, who was speaking to a visiting U.S. congressional delegation, blamed Syria for the prospect of protracted negotiations although there appear to be a number of other problems in getting the negotiations started. His assessment was one of the most pessimistic an Israeli official has offered in recent weeks.

According to Israeli officials, Shamir told members of the House subcommittee on the Middle East that a recent demand from Damascus that Israeli forces be the first to withdraw from Lebanon was the major obstacle to an agreement.

"Syria's position is not helpful," an official said. "The whole thing is up to them. They can speed up the negotiations or slow them down."

At one point, Reagan administration officials hoped that a withdrawal of Israeli, Syrian and Palestinian forces from Lebanon could be arranged by the end of the year. But that prospect is rapidly fading. U.S. envoy Morris Draper visited Jerusalem earlier this week before going on to Beirut, but there was no indication here of progress toward the start of the withdrawal negotiations.

Israel is demanding that the remaining Palestine Liberation Organization guerrillas be the first to leave Lebanon, after which it has proposed the simultaneous withdrawal of Israeli and Syrian troops. But in addition to this question of timing, Draper's mission has not yet reached agreement even on the mechanics of the talks.

The Israelis are asking that the negotiations be conducted by a high-level committee of both political and military representatives from the two countries in the capital cities of Beirut and Jerusalem. The Lebanese have countered by proposing that the talks be held among lower-level officials in border cities such as Metulla in Israel and Rosh ha Niqra in Lebanon, sites of the 1949 cease-fire negotiations between the two countries.

Israeli officials, maintaining that a full peace treaty with Lebanon remains their goal, say the appearance of cease-fire negotiations between basically belligerent nations is what they are trying to avoid.

"That is not what we are looking for," an official said. "We are beyond the point of cease-fire negotiations with Lebanon. We are looking for the talks to be conducted by a political committee that eventually could lead to a peace treaty."

According to officials, Shamir also complained to the congressional delegation about Egyptian attitudes toward Israel, accusing Egypt of violating the Camp David accords' provisions on normalization of relations with Israel.