Israel's highest court today ordered the government to show why the deportation of three Hebron area Palestinian leaders earlier this month should not be declared illegal because no effort was made to grant the deportees the right of appeal.

In another setback apparently prompted by public pressure, the government said tonight that two Arab families forcibly moved from their West Bank homes to an abandoned refugee camp in the Judean desert near Jericho will be allowed to return to their homes.

A military spokesman said the decision was made on humantarian grounds and that the families would be moved out of the derelict refugee camps by Wednesday morning. In each family, a son had been accused of throwing rocks or a gasoline bomb at military vehicles. The government said their banishment was intended to impress upon West Bank residents that they will be held responsible for the actions of their children.

The Israeli court decision, while far from ensuring the return of the exiled Arab leaders, represents a setback for the government of Prime Minister Menachem Begin, which in November tried unsuccessfully to deport Nablus Mayor Bassam Shaka for allegedly inflammatory statements. Shaka's expulsion order was sent by the high court to a military appeals board, and the government dropped the matter.

The acting chief justice of the High Court of Justice, Haim Cohen, sharply rebuked the government today for its handling of the expulsion of Hebron Mayor Fahd Kawasme. Halhoul Mayor Mohammed Milhem and Sheik Rajab Tamini, Hebron's religious leader.

The three were deported to Lebanon after Palestinian gunmen opened fire with automatic weapons and threw grenades at a group of settlers in Hebron, killing six and wounding 16 others. The government accused the Arab leaders of making anti-Israeli public declarations that created an atmosphere for the attack, the worst on Jews in the 13-year occupation of the West Bank.

The court, hearing an appeal by lawyers for the Hebrew area leaders, gave the government 45 days to produce evidence showing why the deportations should not be declared illegal. The court does not have the power to reverse the expulsions, which were made under the 1945 British Mandatory Emergency Security Act, but it can order the case to be reviewed by the military appeals board. The mandatory act is still in effect in the occupied territories.

During today's hearing, Judge Cohen challenged State's Attorney Gabriel Bach several times, saying that the government had produced an exhaustive account of the events leading up to the expulsions but had not explained why due process was denied Kawasme, Milhem and Tamimi.

"The murder may have been a traumatic experience, but this does not justify the procedure you followed," Cohen said. Bach conceded that the law must be respected and that "lessons will be learned and any shortcomings examined." But he said the expulsions were made hastily because of immediate security needs.

The three judges issued their order solely on the basis of the government statement without even hearing arguments by the appellants' lawyers.

Affadavits in support of the three leaders were submitted by Labor Party Secretary General Haim Bar-Lev, a former Army chief of staff; retired Army Gen. Mattiyahu Peled, who is now a peace activist; and Victor Shemtov, former health minister and now secretary general of the leftist Mapam Party.

Shemtov said he knew Kawasme and found him to be one of the most reasonable West Bank leaders. Kawasme declared openly that he recognized Israel's right to exist alongside an independent Palestinian state, Shemtov said.

Meanwhile, the president of the Jerusalem district court today upheld a three-month detention order against the radical leader of the Jewish Defense League, Rabbi Meir Kahane, and one of his followers, Baruch Green, who have been accused of conspiring to organize reprisal attacks against Hebron Arabs in the wake of the May 2 attack.

In deciding to allow two West Bank families to move back to their homes from derelict refugee camps, the government apparently was reacting to unfavorable publicity about conditions in the two camps. At the Ein Sultan camp, for example, Yaakub Shumali yesterday told how he and his family had to stay up at night to drive away wild dogs, scorpions and snakes.

"The first night was the worst," said Shumali, 60, a teacher of Arabic literature at the Swedish school near Bethlehem. His son, Tarik, 17, was accused of throwing stones at a military patrol near their village of Beit Sabur.

"We didn't even have a match, and the dogs were frightening. I can tell you, I was in the Jordanian Army, but I was never frightened like this," Shumali said.

There is no electricity in the Ein Sultan camp, a warren of dilapidated shanties. Shumali, his wife, Georgette, and two daughters said they groped around in the dark Thursday as Israeli soldiers dumped their belongings out of a truck.

The huts selected for the Shumalis in their banishment reeked with the excrement of dogs, goats and humans. Their windows were gaping holes with no cover and the roofs had fallen down.

About three miles away, in the abandoned Akbat Jabar refugee camp, the family of Ahmed Mohammed Kabbi, 27, who was accused of throwing a molotov cocktail at an Israeli Army vehicle, was living in similar conditions.

Shumali, a former major in the Jordanian Army who studied at Fort Slocum, N.J., in 1960 and 1963, appealed to the United Nations, the International Red Cross and Israeli Defense Minister Ezer Weizman to intervene on his behalf. Hanna Atrash, mayor of Beit Sahur, appealed to Pope John Paul II for the Shumalis, who are Roman Catholics.

"These are inhuman conditions," Shumali said yesterday, "I am not a refugee. If it is proved my son is guilty of throwing a stone, then my son will have the penalty. Why should my family be punished?"