Israeli and Lebanese representatives remained deadlocked today over the wording of an agenda after their fifth effort since Dec. 28 to begin negotiations for the withdrawal of Israel's soldiers from Lebanon.

Both sides, however, said that their positions were flexible. A spokesman for U.S. envoy Morris Draper, who is mediating in the talks, emerged after the closed meetings to express a measure of optimism.

"It was a very constructive meeting today, and the differences are narrowing," the spokesman, Christopher Ross, said following the day's discussions at the dilapidated beach hotel in this war-ravaged, former coastal resort strip on the outskirts of Beirut.

Israeli Foreign Ministry spokesman Avi Pazner was not certain of progress. "I don't know if differences are narrowing, but the mood is certainly good," he said.

The principal stumbling block to beginning the talks has been Israel's insistence that one of the topics be the "normalization of relations" between Lebanon and Israel.

The Lebanese have resisted strongly including the term on the agenda, saying that if they even hint that they might agree to that now, Saudi Arabia would wage a crippling economic boycott against Lebanon, and Syrian and Palestine Liberation Organization forces would dig into their positions in northern and eastern Lebanon.

Ghassan Tueni, the Lebanese coordinator of the talks, insisted in an interview yesterday that Lebanon's acceptance of normal relations with Israel now would be tantamount to acceptance of occupation by Israel, Syria and the PLO and destabilization of the country.

He said Lebanon would be willing to negotiate normal relations with Israel along with other Arab nations later in the context of President Reagan's Middle East peace plan. But he said it is out of the question for now.

"The moment we announce the word normalization . . . the Syrians would say, 'Aha, you have sold out. We have to save Lebanon from the Israelis,' " Tueni said. If Syria stays, then so do the PLO and the Israelis, he said.

"Our respect for Israeli intelligence is too great to think that they don't know that," he added.

An Israeli official circulating among reporters in the press room who declined to be identified appeared to indicate some softening of Israel's position on this matter.

"We insist on a proper agenda," he said. "This is not just a technical matter. This is the basis of the whole discussion. We are prepared to negotiate on the wording of it, but we shall insist on the substance of what we call normalization."

Israeli officials, apparently unhappy about acquiring an image of intransigence in the negotiations, seemed to take pleasure in informing reporters that the Lebanese were the ones who today rejected proposals that Draper had laid on the table last week to get the talks started. Details of Draper's proposals were not disclosed.

Asked for confirmation of that, a Lebanese official replied, "This is just a media game--nothing else."

Draper, apparently aware that his proposals would be rejected, came to the talks today with new ones. The Israeli and Lebanese negotiators are to take them back to their governments, and they are to be discussed Thursday, when the sixth meeting will be held in the town of Qiryat Shemona in Israel near the Lebanese border.