The Occupational Safety and Health Administration, in an unusual move, said yesterday that it is investigating possible wrongdoing in OSHA's New York-area office, which failed for four years to correct serious health problems at a Brooklyn thermometer factory where more than 100 workers may have been exposed to dangerous levels of mercury.

"This case really bothers me," said acting OSHA Director Patrick Tyson. "How the hell could something like this happen? How could we have gone this long without the system catching it? . . . . It is clear to me that we made some mistakes."

The agency, frequently criticized by organized labor for failing to vigorously enforce safety laws, yesterday cited the company, Pymm Thermometer Co. Inc., for 16 alleged violations.

The citations included charges of "willful" exposure of workers and proposed $30,100 in penalties. Company officials, who have 15 days to appeal the citations, were not available for comment.

The agency also said it has temporarily removed the two top officials of the area OSHA office, area director Anthony Merisola and supervisor Michael Paris, pending an extensive review of the New York operations by OSHA officials from Washington. Merisola and Paris were temporarily transferred to another OSHA office in New York, officials said.

The investigation has already shown that a second company handling mercury has been allowed to continue violating OSHA standards, according to Tyson, who said the second firm would be cited by OSHA soon.

The agency's internal investigation has been broadened to "look at all the open case files" to see if the office was lax in requiring correction of hazards, he said.

Tyson said that a medical screening by Pymm of its employes in 1985 revealed 26 with symptoms of high mercury exposure, and several cases of apparent mercury poisoning. Mercury causes severe kidney and nervous system damage.

But New York City health officials found that more than half the approximately 100 employes, and several of their children, showed evidence of potential mercury-related problems, according to New York Attorney General Robert Abrams, who last year sued Pymm over the alleged violations.

Yesterday's actions were prompted in part by strong criticism leveled at OSHA by Abrams, who cited the Pymm case as "a dramatic example of OSHA's derelict behavior."

Pymm was cited in 1981 for serious violations that OSHA said resulted in overexposure to mercury. But Tyson said that the New York office granted three one-year extensions to the company, failing to follow procedures to require the company to show progress in correcting some serious violations.

"This is not standard operating procedure . . . . This is an area of great concern to us here for procedures potentially being violated," Tyson said.

When criticism by Abrams and city officials generated media attention last year, OSHA inspectors from Washington were sent to Pymm and found that the company was using a basement -- that had never been inspected by OSHA -- to recycle mercury from broken thermometers, Tyson said. "The conditions there were pretty rough," he said.

Leo Carey, OSHA's deputy director of field operations, said efforts would be made to notify workers of potential health problems. But OSHA sources said that Pymm, located in a low-income area with a high employe turnover, would have trouble locating former workers to encourage them to seek medical treatment.