Maryland Insurance Commissioner Edward J. Muhl said yesterday that "hysteria" had caused insurance companies to discontinue coverage of day-care centers and that Maryland officials had blocked an attempt by one California firm to cancel 242 day-care policies.

Muhl also said that state officials were studying a wide variety of options, including the creation of a mutual liability insurance fund in Maryland similar to the one created in 1974 by the General Assembly to help physicians hit by an insurance malpractice crisis.

Muhl testified yesterday before the House Select Committee on Children, Youth and Families, which is holding hearings on the nationwide insurance crisis in the day-care industry. Muhl, who appeared on behalf of the National Association of Insurance Commissioners, critized insurance companies for wholesale cancellation of day-care policies, saying the insurance industry was overreacting to news accounts of sexual abuse and court suits involving some day-care centers.

Muhl said a California insurance carrier canceled policies at 242 Maryland day-care centers before they were to expire. He said he recently ordered that firm to revoke the midterm cancellations, although he said the company does not have to renew the policies.

Muhl said he was sympathetic to the plight of the insurance industry, which had its worst year ever in 1984. He said the Association of Insurance Commissioners expects the industry to sell $67 billion less in insurance this year than last year.

Muhl said he has signed 20 notices of insolvency for Maryland insurance firms this year. Those firms have gone out of business or must stop writing policies because of financial problems.

Insurance industry executives testified yesterday that the industry was in a slump and urged Congress to allow the industry to voluntarily work out the problem. But Rep. George Miller (D-Calif.), chairman of the select committee, said Congress would be forced to get involved if the insurance industry failed to take immediate action to prevent the closing of thousands of day-care centers.

Insurance industry representatives have said the financial slump was caused by excessively low rates that companies charged during the late 1970s, when they were trying to increase business. They tried to make up the losses through investments, but were hurt by falling interest rates, according to Marvis A. Walter, senior vice president of Insurance Services Offices Inc., which compiles statistics and rate information for the industry.

Because of the financial problems, insurance firms began dropping high-risk industries, and they place the blame on the high awards given in liability cases.

"The potential for multimillion-dollar judgments exists" even though no such award has yet been paid in a child-abuse case, said Frank Neuhauser Jr., vice president of AIG Risk Management Inc., an insurance group based in New York. "Many of us believe we are living under a tort system that is completely out of control."

But J. Robert Hunter, former head of the Federal Insurance Administration during the Ford and Carter administrations, said that the insurance industry is using the courts "as a scapegoat."

"It's a self-inflicted problem, and to take it out on day-care centers is wrong," said Hunter.