U.S., Israeli and Lebanese negotiators reported for the first time today that they have begun to discuss substantive issues in the negotiations for the withdrawal of Israeli armed forces from Lebanon.

"I can say from having attended the sessions that the discussions are very serious and sensitive," U.S. spokesman Christopher Ross told reporters at the conclusion of 5 1/2 hours of talks at a hotel in Beirut's southern suburbs.

The negotiators, who read the same statement to reporters in English, Arabic and Hebrew, said a subcommittee had been formed to deal with negotiations on the issue of declaring the "termination of the state of war" between the two countries and that the full session had also begun discussions on the 27- to 30-mile security zone that Israel has insisted be established in southern Lebanon.

Although he refused to discuss the content of the discussions, Israeli Foreign Ministry spokesman Avi Pazner said that "there was good work being done" by the subcommittee and the full session.

The upbeat assessments came amid reports of stepped-up diplomatic activity for progress in the talks, which were stalled for three weeks as negotiators haggled over how the agenda should be written and in what order topics should be discussed.

That logjam was broken after President Reagan dispatched Middle East special envoy Philip C. Habib to Jerusalem with what has been described as a toughly worded letter to Israeli Prime Minister Menachem Begin.

Habib remained in Jerusalem today while his colleague Morris Draper mediated the talks here. Meanwhile, Saudi Crown Prince Abdullah was in Damascus yesterday reportedly acting as a mediator between Syria and the United States in an attempt to accelerate Syrian troop withdrawal.

Negotiators here said they expect more subcommittees to be formed when they meet again Thursday in Qiryat Shemona, in northern Israel, and that they will discuss then the possibility of holding more frequent meetings.

The issue of terminating the state of war between Lebanon and Israel has been regarded by Western diplomats as the easiest to resolve. Indeed, the only dispute between Lebanon and Israel is whether any state of war has existed between the two countries since they signed an armistice agreement in 1949.

Lebanese officials note that they stayed out of all Arab-Israeli wars that followed that agreement. Israeli officials say such a declaration is needed now because Lebanon refused to meet with Israel on the eve of the 1967 war and that it later signed the Cairo agreement, which permitted armed Palestinians to operate in Lebanon.

The issues of a security zone and Israel's desire for open borders allowing the flow of trade and people and its desire to reach an agreement for more normal relations between the two countries are regarded as much more difficult to resolve. Lebanese officials, fearing isolation in the Arab world, have said they are willing to discuss such issues but only when other Arab nations agree to do so.

Lebanon's Prime Minister Shafiq Wazzan is fond of using the image of a wedding party when talking of normalizing relations with Israel. "We'll go when there are lots of other guests," he says.