Maryland's insurance commissioner, asserting that he wants to halt a "previously unchecked spiral of medical and physician fees in the state," imposed several unprecedented restrictions yesterday on the fees doctors are paid by the state's largest health insurer.

One major change, due to take effect July 1, places a new and lower ceiling on the fees doctors may receive for their services from the insurer, Blue Shield of Maryland. A second change would limit the annual increases Blue Shield may seek for its physician payments in coming years. The restrictions also will effect a second insurer, Medical Service of the District of Columbia, which covers some doctors in the Washington metropolitan area.

A spokesman for the Maryland Medical Society said the changes appear to affect only a small number of doctors because most doctors' fees fall below the ceiling. But the spokesman added that in July some doctors with higher fees may decide to drop out of the insurance plans . . . a move he said could ultimately hurt consumers as they would bear more of their own medical costs.

But Insurance Commissioner Edward J. Birrane predicted yesterday that his actions will benefit consumers either directly or through a "ripple effect."

Birrane said the new restrictions were spurred by his belief that rising fees paid by Blue Shield to physicians were increasing the rates consumers paid for medical insurance. During hearings held to study the matter last fall, a Federal Trade Commission economist testified that Maryland Blue Shield prices were 12 percent higher than the national average.

Birrane said Maryland was the first state to "actively attempt to halt the previously unchecked spiral of medical and physician fees" through the insurance industry. As insurance commissioner, Birrane has the authority to regulate Blue Shield, the insurer, and thus, indirectly, doctors' fees.

However, a spokesman for Blue Shield said that the company's average increase in payments to physicians in Maryland grew at a much slower rate than the Consumer Price Index between 1976 and 1980. "Birrane is of the opinion our actions cause physician fees to increase at higher rates," said Nicholas Greaves, Blue Shield's vice president of public affairs. "We do not agree. We feel it has moderated fee increases."

Birrane also announced yesterday he was "freezing" at current levels the maximum fees Blue Shield pays to physicians. But Greaves contended that move means little because Blue Shield would not be seeking an increase until July 1 even without the "freeze."

Greaves said Blue Shield has not yet determined whether it will accept the new restrictions or fight them in court. But Byron Roberts, a spokesman for the insurance commissioner, said, "They have no recourse. They can go along with it or fold up their tents. It is in the domain of the insurance commissioner to do this."