Israel today released three of the four Lebanese Shiite Moslem militia leaders it had arrested last week, as part of a compromise arrangement to allow resumption of negotiations between the two countries on security arrangements in southern Lebanon.

The negotiations, which were suspended Saturday by Lebanon to protest the arrests, are scheduled to resume Thursday at the U.N. headquarters in Naqura in southern Lebanon.

The three members of the Shiite militia Amal were released in southern Lebanon this morning. The fourth, Mahmoud Fakih, described by Israelis as an Amal leader active in planning attacks on Israeli forces in southern Lebanon, remained in Israeli custody, but there were strong indications tonight that he also will be released Thursday or soon thereafter in connection with the resumption of the talks.

Lebanese Prime Minister Rashid Karami, announcing the agreement to resume talks, said in Beirut that his country had interrupted them "to affirm that we cannot accept anything such as pressure and blackmail."

The four men were arrested in Sidon last Thursday, the same day that the Israeli-Lebanese negotiations opened. A second session scheduled for Monday was canceled after the Lebanese government, responding to the demands of Amal leader Nabih Berri, said it would not participate in the talks until the prisoners were released.

The compromise that led to the announcement of a resumption of the negotiations was largely brokered by U.N. Middle East specialist Jean-Claude Aime, but was assisted by U.S. Assistant Secretary of State Richard W. Murphy, who has also been traveling in the region, according to Israeli officials.

A senior Israeli Defense Ministry official said tonight that the episode involving the arrests and the breakoff of the talks after only one meeting had deepened skepticism here that it will ever be possible to reach agreement on security arrangements in southern Lebanon with the Lebanese government.

Israel has insisted on such an agreement as a prerequisite for withdrawing its forces from Lebanon, which Israel invaded in June 1982. Israel now occupies southern Lebanon from the Israeli border to the Awwali River, about 30 miles north of the border.

"The lesson is the Lebanese weakness, the continuing divisions in the Lebanese camp," the senior Israeli official said. "The lesson is that the Shiites themselves are not able to make up their minds.

"Even with the Syrians, they the Lebanese may not be able to do anything. We still believe that Syria is the key to Lebanon, but maybe even the Syrians can't influence them. Then things are really bad."

Nevertheless, the official said Israel had no choice but to attempt to negotiate with Lebanon because key portions of the security arrangements it is seeking in southern Lebanon require Lebanese consent. These include an expansion of the territory currently patrolled by U.N. soldiers and the continued deployment in southern Lebanon of the Israeli-supported South Lebanon Army.

Israeli officials have blamed the suspension of the talks on internal Lebanese political factors, chiefly pressure on the Beirut government by the Shiite community, which is unenthusiastic about the negotiations. Israel has denied assertions that it provoked the Lebanese action by making the arrests, insisting it must act against those thought to be responsible for attacks on its soldiers regardless of the diplomatic consequences.

The senior Defense Ministry official said Israel initially accepted a compromise proposal by Lebanon Monday night. This arrangement reportedly involved the release of two of the Amal leaders before the talks resumed, and the release of the other two, including Fakih, on the day the negotiating teams returned to Naqura.

The official said that for two days there was no response from Beirut to the Israeli acceptance of the proposal, but that this morning Defense Minister Yitzhak Rabin conferred with Prime Minister Shimon Peres and then ordered the Army to release the three prisoners.

A few hours later, he said, Israel received word that the Lebanese negotiating team would return to Naqura on Thursday.

During the dispute over the prisoners, Israel proposed a cease-fire arrangement with Amal, but there were no reports today that a cease-fire was part of the agreement.

Special correspondent Nora Boustany reported from Beirut:

Amal leader Berri, a Cabinet minister and the driving force behind the suspension, is the man who stands to gain the most out of the dispute, in the view of many observers here.

His mainstream Shiite movement has come under mounting criticism from Moslem zealots such as the Iranian-inspired Hezballah party and many Shiite and Sunni clerics as being too conciliatory. The militant Society of Moslem Ulema last week protested any negotiations with Israel and demanded an unconditional withdrawal of its troops from Lebanon.

Describing the talks as a "sacrilege," opponents of Berri, vying for influence in the increasingly restive Shiite camp, threatened to take action against Lebanese negotiators.

Berri's prompt move to suspend the talks, with resumption made conditional on the release of the Amal prisoners, and his call for a supporting general strike in southern Lebanon on Monday, largely silenced his critics.

Israel's reference to Fakih as an "underground" activist involved in "many terrorist actions," also has enhanced Amal's status among Moslem militants as being in the lead of resistance operations.

Meanwhile, Lebanon's Cabinet today named Maj. Gen. Mahmoud Tay Abu Dargham, a Druze, as the Lebanese Army's new chief of staff, replacing Maj. Gen. Nadim Hakim, also a Druze, who died in a helicopter crash Aug. 23.

The Cabinet also approved a plan for Lebanon's Army to take over security for the coastal highway linking Beirut with the Awwali River in southern Lebanon. The road has been closed for months by fighting between rival Druze and Christian militias, who reportedly have agreed to the Army takeover.