Joseph B. Kennedy is irritated. His round face is turning rose-pink and he is spitting out his words.

Some claim that indignation is Kennedy's normal state.

Since he became an administrative law judge 11 years ago and began hearing cases under the federal coal mine laws, Kennedy has raised hell with nearly everyone who has come before him -- and a few who have not.

He has accused the Labor Department of lax safety enforcement, imposed large fines on coal companies, chastised the United Mine Workers for not doing more to protect miners, criticized his fellow judges and, most recently, lectured his bosses at the Federal Mine Safety and Health Review Commission.

The Justice Department and Rosemary Collyer, former chairman of the commission, are the subjects now stuck in Kennedy's craw.

But first, some history.

There are 1,121 administrative law judges, commonly called ALJs, assigned to 29 federal agencies. All of them are lawyers and some are former judges. ALJs conduct hearings, issue decisions, levy fines and interpret federal laws.

Kennedy became an ALJ at the age of 51. He graduated with honors from George Washington University Law School in 1949, then, for the next 24 years, he criss-crossed from private practice to government jobs with the Army, the Federal Trade Commission and the Armed Services Board of Contract Appeals. In 1973, he became an ALJ and was assigned to the Interior Department, which at that time was responsible for administering federal mine safety laws.

One of his first cases involved small-mine operators who contended that regulations designed to control coal dust could not be met. Kennedy surprised all sides by visiting several small mines to see conditions first-hand, trips that convinced him the companies could meet the standards.

In another early case, Kennedy ruled that the Interior Department was responsible for protecting not only coal miners, but the public as well. The department reacted by taking the case out of his hands.

Kennedy again butted heads with Interior officials in 1976 when he fined a coal company for 41 safety violations. His action outraged Kent Frizell, then undersecretary of the interior, because the agency had promised the coal company that it would not take action until a federal district court had ruled on several questions in a related case.

Frizell stayed Kennedy's decision and placed an official reprimand in his file -- a move that caused such a public flap that then-Attorney General Edward H. Levi investigated it. Levi eventually removed the reprimand and Congress passed legislation to prohibit agencies from rating ALJs under the civil service system's evaluation program.

Kennedy's current problems stem from his unbridled criticism of the Reagan administration's policy of "cooperation, rather than confrontation" between safety inspectors from the Labor Department's Mine Safety and Health Administration and coal operators.

"It's easy not to see the real implications of 'cooperative enforcement' if you are the type of judge who only looks at the trees," Kennedy said. "But if you can see the forest, in other words, see a pattern in the cases that you hear, then you know that death rates have been zooming as coal production has increased."

Fatalities in coal mines have increased from 32 for the first six months of 1983 to 57 for the same period this year. But the fatality figures have gone up and down since Reagan took office, and MSHA officials contend that it is too early to tell whether the new enforcement policy is to blame.

But Kennedy said he is convinced. "Inspectors are so torn between their sworn duty to enforce the law and the administration's policy of 'cooperative enforcement' that it is well nigh impossible for them to reconcile their findings of violation with their attempts to trivialize gravity and culpability," he wrote in one decision.

Kennedy stepped up his attack recently by accusing MSHA inspectors in another case of "ignoring the conditions of wanton, if not criminal, endangerment." Labor Department attorneys asked the review commission to strike Kennedy's comments, arguing in part that Kennedy could not prove his charges. The commission voted to remove the criticism.

Shortly after that, Kennedy received a call from a miner in Kentucky who claimed he had information about federal inspectors accepting payoffs and kickbacks.

Kennedy notified the Justice Department, which, he said, asked him to introduce one of its investigators to the informant. But the investigator backed out a few hours before Kennedy left for Kentucky.

Kennedy briefed Justice officials when he returned and submitted a $479 travel claim to his boss, Collyer. She refused to pay, saying Kennedy had overstepped his authority and misled her by claiming that Justice had asked him to go. Justice officials insist they never asked Kennedy to investigate on his own.

Kennedy's critics claim that he is arrogant and egotistical. "He has become a crusader for coal miners," complained one coal company attorney. "He is simply trying to stick it to the Reagan administration," said a Labor Department official. Added an MSHA official: "He thinks he knows more than anyone else about mining and what's best for miners. He acts like a god."

But Kennedy scoffs at the criticism. "I've taken an oath to uphold the law and see that it is faithfully executed," he said. "If MSHA doesn't want to do it and the commission doesn't want to do it, that doesn't relieve me of my commitment and responsibility.