Gun battles broke out today between Israeli troops and Moslem militiamen in southern Lebanon in the most intense fighting there since Israeli forces withdrew last June, security sources said. But Israel continued its massive sweep in response to the capture of two soldiers, despite threats that they would be killed.

Shiite Moslem militiamen holding the Israeli soldiers, who were seized in an ambush in southern Lebanon on Monday, threatened to kill one of them at 9 p.m. tonight unless all Israeli troops were withdrawn from Lebanon by then.

After the deadline, an anonymous caller told news services in Beirut that one of the Israeli soldiers had been killed, but another caller who said he was speaking for the captors denied that either had been killed, special correspondent Nora Boustany reported from Beirut.

Lt. Gen. Ori Orr, chief of the Israeli Army's Northern Command, said the guerrillas' ultimatum would have no effect on the Army's search for the captured soldiers.

Timur Goksel, spokesman for the U.N. peace-keeping force in southern Lebanon, warned tonight that the situation there had reached a dangerous point that he said could escalate into full-scale battle between pro-Israeli forces and Moslem militiamen.

Israeli military officials said four guerrillas were killed today by a joint patrol of Israeli troops and the Israeli-allied South Lebanon Army, after the guerrillas attacked the patrol near Haris. U.N. officials said they were told by Haris residents that the Israeli patrol had "responded with tank fire" when it was ambushed.

Shiite Amal militia officials said in Beirut that four of their fighters were killed and eight seriously injured in the clash at Haris.

Haris is about four miles northwest of Kounine, site of the Moslem guerrilla ambush Monday in which the two Israelis were abducted and two SLA militiamen were killed while patrolling in the Israeli-declared "security zone."

Security sources in southern Lebanon estimated that three battalions of Israeli troops, supported by tanks, armored personnel carriers and helicopter gunships, were involved in the sweep through 15 to 17 villages, Israel's largest military operation in southern Lebanon since it withdrew most of its troops last June.

Officials of the U.N. peace-keeping force said from their headquarters in Naqura, Lebanon, that Israeli helicopter gunships and at least 1,200 troops were operating against villages north of the Israeli security zone, inside the area controlled by the U.N. peace-keepers.

The U.N. sources said the Israeli forces were in control of a 6-by-9-mile area inside the U.N. zone, searching villages, including Kharabet Salem, Aita Zott, Haris and Touline. All of the males in the villages were reported to have been rounded up and interrogated by the Israeli troops.

Later today, the Israeli forces were reported to have extended their operation farther north, to Qabrikha and Mazrat Froun.

U.N. peace-keeping forces were deployed in each of the occupied villages to observe the Israeli search operation, the sources said.

In Tibnine, the hometown of Shiite Amal militia leader Nabih Berri, three miles northeast of Haris, Israeli troops rounded up residents and interrogated them for the second time in three days and detained 10, sources in the area told Boustany. Tibnine also is headquarters for the Irish contingent of the U.N. force.

Security sources in southern Lebanon said U.N. troops prevented a clash between an Israeli Army unit and forces of Amal, the dominant Shiite militia in the region. The sources said 200 Amal gunmen entered an area the Israeli Army was searching and demanded that they leave. The Israelis refused, and the U.N. forces removed the Amal militiamen while the search continued, sources said.

Meanwhile, Israeli government sources said that intensive diplomatic efforts had begun in an attempt to negotiate the release of the two captured soldiers, both of them students in special religious schools that combine part-time military service and religious study.

The captives were identified as Yossi Fink, 20, whose family immigrated here from England 14 years ago, and Rahamim Alsheikh, 20. Both were said to be students at the religious school in the Karnei Shomron settlement in the occupied West Bank.

Israeli officials would not discuss the diplomatic efforts. A Foreign Ministry spokesman said, "As long as the military operation is still under way, no useful purpose would be served by discussing this matter."

However, other Israeli government sources said intense diplomatic activity, mostly through U.S. channels, was under way in an effort to contact the Islamic Resistance Front, an umbrella group of radical Shiite Moslem guerrilla forces that includes Hezbollah.

Spokesmen for the front said in Beirut yesterday that they are holding the soldiers and that they "have been moved to a safe place."

A Shiite Moslem faction said today that in retaliation for Israel's incursion into southern Lebanon, it killed Dr. Elie Hallak, 52, a prominent Lebanese Jew who had been kidnaped in Moslem west Beirut last Feb. 21.

The faction, called the Organization for the Oppressed on Earth, sent an announcement of his death and a picture of him to the Beirut daily newspaper An Nahar. It said it would not hand over his body unless Israel withdraws "from all occupied territory" and releases Arab prisoners it holds.

Hallak, a physician and vice president of the Supreme Jewish Community Council of Lebanon, would be the fourth Lebanese Jew killed by Moslem captors since Dec. 25.

Meanwhile, three kidnaped Spanish Embassy officials were freed today and were handed over to Spanish Ambassador Pedro Manuel Aristegui by Shiite Amal militia leader Nabih Berri.

Relatives of two Shiite Lebanese men imprisoned in Spain for attempting to kill a Libyan diplomat there in 1984 had kidnaped the embassy officials on Jan. 17.