A federal judge here said yesterday that it was unconstitutional for the Library of Congress to ban patrons who have protested the library's new early closing hours and likened its practice of sending "banning letters" to tactics used against South African leader Nelson Mandela.

"Banning is not one of the remedies present in the Constitution," said U.S. District Judge Harold H. Greene. "Perhaps in the Soviet Union or South Africa people are banned, but not in the United States."

Greene's comments came as he issued a 10-day restraining order that blocks the library from banning five persons who filed suit after they were given letters notifying them that they could not enter the library until Oct. 1.

The library said the five, members of the "Books Not Bombs" group that has protested the library's new hours, had disrupted patrons and that their activities had cost "thousands of dollars in staff time."

Russell Mokhiber, the protest leader and one of those banned from the library, later entered the building shortly before the 5:30 p.m. closing time but left within minutes.

No arrests were reported.

On March 10, the library, which had been open from 9 a.m. to 9:30 p.m. each day, began closing on Sundays and at 5:30 p.m. the rest of the week except Wednesdays as a way to meet budget cuts mandated by the Gramm-Rudman-Hollings budget balancing law.

Assistant U.S. Attorney Michael Martinez told Greene that the library was willing to rescind the banning letters if the five promised not to continue their protests. He said that the library had the authority to control the activities of those using the library and that banning the five was preferable to arresting them each time they protested because of the cost of overtime pay for library staff.

Greene quipped that he had never heard the government argue it was too expensive to arrest people.

He said disruptive patrons could be ejected or arrested but that "so far as I can see," the banning letters are "utterly unconstitutional."

The government's attempt to extract promises from the five in exchange for lifting the banning orders was the same tactic the South African government has at- tempted to use with Mandela, Greene said.

Martinez told Greene that there is no "constitutional right . . . to demonstrate in the Library of Congress," but Nina Kraut, attorney for the five, argued that the banning letters "amount to a prior restraint on speech . . . and to censorship."

Mokhiber, 31, a lawyer who works for Ralph Nader's Corporate Accountability Research Group, was arrested for unlawful entry on March 13 and participated in protests the next day. He was handed the banning letter when he tried to enter the library on March 17.

Mokhiber said in a affidavit filed in court that the ban had made it "extremely difficult" for him to complete research on a book he is writing about corporate crime.

The four others involved in the case are J. William Hirzy, senior scientist for the Environmental Protection Agency and a vice president of Local 2050, National Federation of Federal Employees, Bee Wardlaw, David West and Steven Blume.