oreign Minister Yitzhak Shamir, who is to travel to Washington on Friday for possibly critical meetings on the troop-withdrawal negotiations with Lebanon, reportedly said today that Israel must compromise in the talks if it is to have any hope of winning a Syrian troop withdrawal.

Israeli radio reported tonight that Shamir told a closed session of the Knesset (parliament) Foreign Affairs and Defense Committee that Israeli and Lebanese negotiators are exploring the possibility of Israeli, Lebanese and U.S. troops patrolling a "security zone" in southern Lebanon after a troop withdrawal.

Shamir's remarks were the most concrete indication yet that Israel is looking for a way to drop its demand that five Israeli-manned "observation posts" be established in the security zone as part of a troop-withdrawal accord. It was also a suggestion that American troops, now confined to the Beirut area as part of the multinational force, could become involved in patrolling the Israeli-Lebanese frontier.

Lebanese officials last week told reporters that Israel had already indicated its willingness to drop the demand for observation posts and accept instead the creation of joint Lebanese-Israeli inspection teams to monitor the security arrangements.

But Israel publicly rejected this proposal, arguing that such "inspection teams" would be insufficient to prevent a return of Palestinian guerrillas to southern Lebanon. The negotiations now appear to center on the difference between this Lebanese proposal and Israel's desire for large, active and daily joint patrols roaming the southern Lebanese countryside.

Shamir's reference to Syria was also an acknowledgment of the difficulty of Israel's negotiating position until now in demanding the right to maintain a residual military force in southern Lebanon while expecting Lebanon to gain agreement for a complete withdrawal of Syrian and Palestinian troops from other parts of the country.

In announcing the foreign minister's trip to Washington, an Israeli official sought to dampen expectations that Shamir's meetings with Reagan administration officials will lead to a breakthrough in the talks. He said Shamir, who is to meet with Secretary of State George P. Shultz, special Middle East envoy Philip C. Habib and possibly President Reagan, is making the trip "to coordinate the positions between the two countries" in the Lebanon negotiations.

"Don't expect to have an agreement on the table" after Shamir's meetings in Washington, the official said.

Nevertheless, the trip comes at a critical point in the negotiations, just a week after Israel rejected Lebanon's latest compromise proposals as conveyed to them by Habib. Underscoring the importance of the trip, officials said that Shamir would be accompanied by David Kimche, the director-general of the Israeli Foreign Ministry and head of Israel's delegation in the negotiations with Lebanon.

In Khaldah, Lebanon, Associated Press reported, Kimche told Tuesday's session of the talks that Israel had cut back its initial insistence on full commercial ties with Lebanon and would settle for "a very limited trade exchange. . .to symbolize the hoped-for new relationship" between the countries." Lebanon has indicated that a broad accord would invite retaliation by other Arab states.

The reported mention by Shamir of the need for compromise in the negotiations coincided with Israeli press reports today that he would carry to Washington "substitute proposals" for those rejected last week.

Habib spent much of last week here discussing with Israeli officials the latest Lebanese suggestions for a troop-withdrawal agreement. Israel rejected these, but officials here were at pains to suggest they were anxious to pursue other possible compromise proposals and were not wedded to all of the details of their current negotiating position.

According to the independent newspaper Haaretz, Israel now seeks the right to send frequent patrols of Israeli soldiers into southern Lebanon in conjunction with Lebanese forces commanded by Saad Haddad, a renegade Lebanese Army officer and longtime ally of Israel.

The security-zone arrangements are one of the two major sticking points in the negotiations that Shamir is expected to discuss in Washington. The other is Israel's insistence that Lebanon agree to "normal relations" with Israel, including the free flow of goods and people across the border.

The Israelis have long criticized the U.S. position as too often siding with Lebanon on these main issues. Shamir's trip "to coordinate the positions" in the talks appeared to indicate an Israeli interest in closing the gaps separating it from the United States.