The Maryland Court of Appeals, in a ruling that could limit future increases in medical costs for thousands of Maryland families, held yesterday that the state's insurance regulatory agency can bar Blue Shield from boosting reimbursements to doctors until it justifies them on "sound actuarial principles."

The ruling ratifies, at least in part, a controversial effort by former Maryland Insurance Commissioner Edward J. Birrane Jr. to contain spiraling medical fees by ordering contracts between doctors and Blue Shield to be subject to review by the state Insurance Division.

In the past, rate increase requests by Blue Shield, the nationwide nonprofit health service group through which subscribers pay doctors for medical services, have been approved routinely with little examination of the causes of the increases, according to state officials.

At stake are millions of dollars each year in increased fees paid out by Blue Shield's 800,000 Maryland subscribers in reimbursements to the more than 10,000 doctors who participate in the Blue Shield program, according to attorneys in the case.

The ruling affects subscribers throughout the state who are enrolled in either Blue Shield of Maryland Inc. or the Maryland component of Medical Service of the District of Columbia, a Blue Shield organization.

The state's highest court, in a unanimous opinion, reversed a Baltimore city judge who ruled in 1982 that Birrane exceeded his authority in requiring the review of rate increase requests.

The court agreed with Birrane's contention that Maryland law empowers him to examine the formula for reimbursing doctors before approving changes--including fee increases--in Blue Shield-doctor contracts.

Birrane expressed satisfaction with the ruling yesterday, saying he and former Assistant Attorney General Michael L. Cohen, who had worked with him on the case, "had been mighty lonely, hacking out a whole new theory of law. . . trying to apply it to the ever-increasing fees paid out to the doctors." Birrane and Cohen are now in private practice together.

Charles J. Steele of Washington, an attorney representing Blue Shield, said yesterday he had not yet read the 32-page opinion from the Court of Appeals and could not say if Blue Shield will appeal the case to the Supreme Court.

Birrane said in an interview that the case began in 1981 when he started examining rate increase requests by Blue Shield. He said that for years Blue Shield payments to doctors had been routinely "ratcheted upwards" by a formula based on reimbursing doctors for "usual, customary and reasonable" fees, known as UCRs. Under the formula, Blue Shield would pay a doctor in the program for 100 percent of his "usual" fee, so long as it did not exceed the "customary" fee, the amount that 90 percent of the doctors in the area charged.

"We saw a fallacy in this," Birrane said. By raising their "usual" fees each year, he said, the doctors automatically raised the so-called "customary" fees in a never-ending spiral with little or no cost control exerted by Blue Shield.

The Court of Appeals yesterday said the Insurance Division can now freeze Blue Shield's "customary" fee schedules, and Blue Shield cannot obtain further increases without a complete review of the data supporting the increase request.