The Rev. Daniel Berrigan, his brother Philip and about 40 other demonstrators joined hands yesterday in Lafayette Park for a protest of Reagan administration policies and quietly spoke the names of those they said they brought with them in spirit.

"All the political prisoners in the world . . . " said Philip Berrigan. "All the Plowshares prisoners," said Daniel Berrigan, speaking of those serving long prison terms for antinuclear activities.

"Helen and Sylvia and Jeanette, and all those displaced by illness and lack of concern," said one man. "Billy Neal Moore, a friend of mine on death row in Georgia," said John Dear, a Jesuit priest.

The protest was part of a three-day gathering of Jesuits and their coworkers in Washington to discuss U.S. intervention in Central America and the arms race. It brought out the Berrigan brothers, antiwar protesters of the 1960s who gained attention as members of the Catonsville 9 -- the group that burned draft records in a small Maryland town, setting off a slew of draft board raids.

"I've lost count of the number of times I've been arrested, maybe 60 or 70," Philip Berrigan told a reporter.

"But a thousand times is never enough as long as humanity is in peril."

Berrigan said the group had gone to the White House to remind people about nuclear war, nuclear accidents as emphasized by the Soviet nuclear reactor disaster at Chernobyl, and that "40 wars are raging around the planet, a lot of them fought with American weapons."

About 14 in the group crossed Pennsylvania Avenue to carry their protest to the sidewalk in front of the White House, where they planned to get arrested for "civil disobedience" as they prayed in a small circle. But they apparently had not counted on a twist in the federal regulations: A group must hold a sign or placard stationary on the sidewalk within a certain area to be in violation of the law.

After a discussion, they decided to pray and sing but not to thwart the law.

But from the start, the Berrigans -- with Daniel facing trial and Philip sentenced to seven years for burglarizing a nuclear plant -- had decided not to expose themselves to arrest.

"We're letting them do this one," Daniel Berrigan said as he got into a waiting car. "We're taking what is called a rain check."