The Maryland State Insurance Division has fined Anne Arundel Community College's insurance company $10,000 and ordered it to pay any back salaries that may be awarded in a pending sex-discrimination suit filed by teachers against the college.

School officials, however, hold little hope for a quick settlement in the 5-year-old case. The order and fine are expected to be appealed by the insurance company and were greeted only with "cautious optimism" by college President Thomas E. Florestano.

"It's going to be a long drawn-out process," Florestano said, referring to the expected appeal and continuing negotiations.

Florestano had said earlier this month that the insurance company's refusal to pay retroactive salaries was impeding an out-of-court settlement with the teachers.

A class-action suit was filed in Baltimore federal court against the 9,000-student college on behalf of 89 female teachers in 1978, alleging that the school discriminated against them in pay, benefits and promotions--claims that school officials deny.

Sporadic negotiations have borne no out-of-court settlement, and another hearing before U.S. District Judge Frank A. Kaufman is scheduled for next Wednesday.

The college, located in Arnold, filed a complaint against the Chicago-based Continental Casualty Co. this year after the company said it would not pay for any back-pay settlement that came out of negotiations or a court decision. Under the school's $2 million indemnity policy, the company says it ordinarily would reimburse the college for monetary awards, except in discrimination cases.

Assistant Insurance Commissioner Thomas Raimondi, in a ruling made public earlier this week, supported the college's argument that retroactive pay, which could amount to as much as $500,000, is included in the college's policy covering wrongful professional acts such as sex discrimination.

Raimondi also ordered that Continental Casualty be further investigated for alleged unfair trade practices and fined it $10,000 for "unreasonably delaying payment."

Meanwhile, insurance company attorney Stuart Ross said he will pursue a suit filed two months ago by Continental requesting Kaufman to rule on the question of back pay.

Ross said Raimondi did not have authority to decide whether the company was liable for the claim. The lawsuit is against the college and separate from the planned appeal of Raimondi's decision.