JERUSALEM, DEC. 26 -- Three Palestinian guerrillas, apparently seeking to capitalize on the unrest in the Israeli-occupied West Bank and Gaza Strip, entered Israel through its border with Jordan but were intercepted and captured after a gun battle, the Israeli Army reported today.

Meanwhile, police here used tear gas to break up a demonstration tonight by hundreds of Israeli members of the Peace Now movement who defied police orders and tried to march on Prime Minister Yitzhak Shamir's official residence to protest the Army's crackdown in the occupied lands and call for his resignation.

The territories remained quiet for the fourth day and military officials said they believed the disturbances in which 21 Palestinians were killed were finally over. More military trials of the estimated 1,000 or more alleged rioters presently in prison are expected to take place Sunday.

The infiltration, which occurred last night, was the first from Jordan in a decade and appeared designed to repeat the pattern set by last month's raid on an Army camp in northern Israel, an incident that Palestinian youths said inspired them to confront Israeli troops in the territories in recent weeks.

That attack was conducted by a lone commando who flew into Israeli territory from Lebanon on a motorized hang glider and killed six Israeli soldiers before he himself was shot dead. It was the single most effective assault ever carried out against a military target here and a half dozen officers and lower-ranking soldiers have been disciplined over what senior Army commanders conceded was a dramatic failure of security measures.

But this time the soldiers involved reacted with textbook efficiency, according to the Army. A military spokesman said three men sent by the Baghdad-based Palestine Liberation Front crossed the Jordan River under a light rain and cut through a barbed wire border fence near Kibbutz Maoz Hayyim just east of the town of Bet Shean about two hours after sunset yesterday.

The Palestinians, who were between 20 and 25 years old, were on a terrorist mission, according to the Army. They wore military fatigues under civilian clothing and were armed with Kalashnikov assault rifles, hand grenades and metal spikes that they apparently planned to use to halt and ambush Israeli vehicles. Bet Shean is on the main highway leading from Jerusalem to the Sea of Tiberias and the Upper Galilee region.

Their entry was detected by an Army patrol that, guided by an Arab Bedouin tracker, followed the muddy footprints of the infiltrators. Meanwhile Bet Shean and agricultural settlements in the area were put on alert.

The soldiers came upon the Palestinians hiding in a wheat field about 400 yards from the border and opened fire. One of the Palestinians was wounded before he and his companions surrendered.

The military commander of the West Bank and central Israel, Maj. Gen. Amram Mitzna, told Israeli radio, "The most important thing that I can learn from this incident is that our system that was not challenged for the last 10 years here in this area acted like it should and each of the soldiers, each of the parts of this system, did what they had to do {in} this very quiet part of the border in Israel."

Mitzna added that, following two weeks of Palestinian riots, "Maybe some people thought that Israel is weak, Israel cannot defend itself, and I hope that they learned the lesson . . . . They tried something very big and it failed."

He said the men might also have been attempting a major attack to mark the Jan. 1 anniversary of the founding of Fatah, the mainstream movement of Yasser Arafat's Palestine Liberation Organization.

The Palestine Liberation Front, headed by Mohammed Abbas, claimed responsibility for the infiltration, which it said caused Israeli casualties and was designed to express solidarity with Palestinians in the occupied lands. The PLF was responsible for the October 1985 hijacking of the Achille Lauro cruise ship.

The Israeli peace demonstrators had been given police permission to hold a rally in a main downtown square but denied a permit to protest in front of Shamir's residence. But during a candle-lit procession, leaders asked the protesters to drop their signs and candles and walk to the residence individually to express their objections to Israeli policy. Witnesses said about half of the crowd of 1,000 to 2,000 did so.

When they turned a street corner near the residence, police fired tear gas and the crowd quickly dispersed without further incident, a witness said. It was the first major Israeli protest against government policy since the disturbances began in the occupied territories nearly three weeks ago.