A federal judge, upholding a half-million-dollar civil verdict against First Virginia Banks Inc., ruled that the company's refusal to rehire a black maintenance man after he complained of racial harassment was serious enough to justify sizable punitive damages.

U.S. District Judge Albert V. Bryan Jr. found that the "award, although exceptional, is not so high as to indicate that the verdict was improperly motivated or the result of a failure to follow the court's instructions."

The order, received this week by both sides, bolsters a six-member jury's finding in July that First Virginia retaliated against janitor Robert L. Holland by forcing him out of his job after he complained that supervisors called him names such as Chicken Little and Watermelon Man.

Holland, 46, a Woodbridge resident and former Army sergeant, worked in First Virginia's maintenance department for 10 months in 1988 and 1989.

Victor M. Glasberg, Holland's attorney, said First Virginia's request that the award be overturned or reduced demonstrated that company executives have maintained "the same uncompromising position they took since the beginning when they offered to settle for $1,000."

Thomas P. Jennings, attorney for First Virginia, said yesterday the company had not decided whether to appeal the judge's rulings. Jennings would not comment further.

Attorneys for First Virginia argued that the jury's award of $521,000 was unwarranted because company supervisors testifed that Holland resigned and never notified them he wanted his job back. They also claimed that the size of the damages suggested the jury acted out of "passion, bias or prejudice."

Jurors in the case said in interviews that they calculated the award by estimating how much it would cost to send First Virginia executives to an equal employment opportunity training program. "They should have had at that company some program to deal with discrimination issues," said juror Kathleen C. Key.

Jurors added that a letter Holland wrote to his superiors made it clear that Holland did not know he had lost his job and wanted to be reinstated. Juror Alan K. Spear said, "The whole resignation thing sounded like a hoax -- something put up by the bank."

Several Northern Virginia residents said they complained to First Virginia after learning about Holland's suit. Laura C.G. Shaw, writing for a group of Arlington teachers, told the company the name-calling and retaliation were "deplorable and inexcusable," and that "it is not tolerated in the education community."

Shaw said in an interview that Justin C. O'Donnell, the president of First Virginia Bank, responded, telling her Holland's treatment "was shameful and the bank was embarrassed."

O'Donnell could not be reached yesterday. Company representatives have said the name-calling was part of the maintenance department's informal atmosphere and was stopped after Holland complained.

Judge Bryan also awarded Glasberg $87,388 in legal fees, almost double the $45,000 Glasberg said he would have accepted from First Virginia after the three-day trial in July.