ANNAPOLIS -- Maryland's second-highest court has invalidated rules used by the state's largest provider of malpractice insurance to deny coverage to doctors, an action the insurer says could force it to raise malpractice premiums.

The Court of Special Appeals this week upheld a Baltimore judge's ruling that the Medical Mutual Liability Insurance Society of Maryland broke the law when it refused coverage for Dr. Michael O. Magan.

If the ruling is allowed to stand, Medical Mutual may lose its reinsurance or be forced to pay more for it, said J. John Spinella, the insurer's president.

Insurers contract with reinsurance exchanges such as Lloyd's to insure themselves against large claims.

Spinella said no decision had been made yet on whether to appeal the case to the state's highest court.

Magan, an obstetrician-gynecologist, had to stop delivering babies for nearly a year after Medical Mutual denied him insurance in 1985. The firm turned Magan down because it had paid $964,000 on two malpractice claims made against him, with another claim pending at the time.

When Magan protested, Medical Mutual offered to insure only his gynecological practice, but at double the customary premium. The doctor went to court, and in May 1986 Circuit Judge Thomas Ward found that the company had violated the insurance code by refusing to cover Magan.

Ward ordered Medical Mutual to provide Magan obstetrical insurance coverage at its standard rate.

But the judge also declared that the company, created by the General Assembly in 1975 to deal with a malpractice insurance crisis, was required by law to cover all licensed physicians in the state, regardless of their risk of being sued.

The special appeals court sidestepped that issue, hotly contested by lawyers for Medical Mutual and state Insurance Commissioner Edward J. Muhl.

In an opinion written by Judge William W. Wenner, the court found that the guidelines used by the firm in denying coverage were based on a "vague" and "subjective" review of a doctor's claims history.

"The guidelines, particularly as they speak to 'claims history,' cannot be objectively, uniformly and fairly applied, as we believe the statute requires," Wenner wrote.

Medical Mutual insures about 5,600 physicians, nearly 90 percent of the doctors practicing in Maryland.

The firm refuses insurance coverage to fewer than 150 a year, according to Spinella.

However, Spinella said that nearly all of those turned down were refused on the basis of their claims history. He contended that there is no numerical formula that can be applied to past malpractice claims to decide whether a physician represents too great a risk of being sued.