A federal judge last week ordered one of the country's largest mortgage insurance firms to pay $680,000 to two former Northern Virginia employes who say the company breached its contract to pay them commissions for a lucrative business deal.

The ruling by James C. Cacheris, U.S. District Court judge in Alexandria, is another in a series of blows for the financially troubled TMIC Mortgage Insurance Co., formerly known as Ticor Mortgage Insurance Co.

The company, based in California, recently was placed into conservatorship by California regulators after it was overwhelmed by potential losses from defaulted mortgages it insured at Equity Programs Investment Corp., the Falls Church real estate syndicator.

Cacheris' decision came after a two-week trial last fall over acrimonious charges between Ticor and the ex-employes in its Falls Church office, Ralph M. O'Hair and Jamie B. McLaughlin, who have since started their own mortgage brokering firm.

At issue was a highly unusual brokerage program O'Hair and McLaughlin helped originate in 1983 at Ticor. Under the program, Ticor would earn fees by arranging for the sale of mortgages from savings and loans to the Federal National Mortgage Association and the Federal Home Loan Mortgage Corp.

Ticor also set up an incentive compensation program, under which O'Hair and McLaughlin would be paid special bonuses for any fees beyond $1.4 million Ticor earned from this and other mortgage marketing programs.

The program was wildly successful, far beyond the goals established by the company, with Ticor realizing total fee income on the order of $13 million from mortgage marketing.

Under terms of their contract, McLaughlin and O'Hair said they ought to have been paid $1.6 million as a result of this performance, but they were paid only about $945,000.

After they made a formal demand for the rest of the money in 1984, they were fired, and Ticor filed suit against the two and another employe. The company charged them with conspiring to intentionally underestimate the fee income the firm would earn in 1983. The employes filed a countersuit, accusing Ticor of breach of contract and fraud.

While the third employe subsequently reached his own settlement with Ticor, Cacheris' ruling last week substantially upholds the substance of McLaughlin and O'Hair's version of events. While Cacheris dismissed the fraud claims, in a written opinion he said he found O'Hair and McLaughlin to be "much more credible witnesses" than Ticor Mortgage's chief witness, former president Gary Bradford.

O'Hair said last week that he was pleased by the ruling, although he expressed dismay over the huge legal fight he said Ticor forced on him.

"They tried to overwhelm us financially, and I spent a year of my life spending an awful lot of my money" on legal expenses, he said.

TMIC officials declined comment.