One of three firms competing for a contract to operate Washington's first legal lottery is under investigation in Ohio for failing to pay $50,000 in prize money to 13 persons who apparently held winning lottery tickets, state officials there said yesterday.

Glendinning Companies Inc. of Westport, Conn., is being investigated by the Ohio attorney general's office and the state lottery commission over the firm's handling of "Horse Race," a state-sanctioned instant lottery that Glendinning has operated since August.

Colleen K. Nissl, assistant attorney general, said yesterday that the investigation began after several people called the lottery commission earlier this year to complain that the Westport firm would not honor apparent winning tickets from games in November and December.

The firm maintained, Nissl said, that the numbers on the 13 tickets duplicated those on lottery tickets already claimed by other prizewinners. But, the state's contract with any company operating a lottery requires the company to "pay the prize value on any ticket that was misprinted or produced in error," Nissl said. "What you see is what you get," she said.

Ohio officials have been negotiating with Glendinning for several months to resolve the problem but have not yet reached any agreement. "We do not know yet if there is going to be any legal action," Nissl said, adding "we can go to court to enforce the contract."

Meanwhile, the firm is continuing to operate the "Horse Race" game under a contract that expires later this month, according to Ross Sprague, deputy director of the Ohio Lottery Commission.

Glendinning officials did not return several telephone calls from a reporter yesterday.

The D.C. Lottery and Charitable Games Control Board is expected to announce next week which of three companies will be awarded the contract to operate a city-sanctioned instant lottery scheduled to begin this fall.

The contractor will be authorized to print and distribute 10 million tickets that would sell for $1 each and probably pay between $2 and $25,000 to holders of winning tickets. The contract would be for about one year, and the city is expected to receive 30 percent of the revenues.

The other two companies vying for the contract are Raven Systems and Research Inc. of Washington and Scientific Games Development Corp. of Atlanta.

During a closed meeting Thursday, Glendinning received strong endorsement from several board members who felt the other two firms were either too inexperienced or too controversial to be awarded the franchise, according to one board official.

The official said board members were apprehensive about Raven, a local computer systems management firm because it has never run a lottery. "The feeling is if we come up with a lottery company that doesn't work, the public is going to look at us as incompetent idiots," the official said.

Raven is bidding for the contract in partnership with Webcraft Inc. of New Jersey, which has held lottery contracts in several states and has been awarded a contract to operate Ohio's "Horse Race" lottery after the current contract with Glendinning expires.

The D.C. gambling board official said board members were reluctant to award the contract to Scientific because that firm spent $90,000 in 1980 to support efforts to pass the referendum that legalized gambling here.

Glendinning is not the only firm competing for the contract here that has had troubles with state lottery officials elsewhere.

The Arizona attorney general's office investigated the purchase of Scientific Games by Bally Manufacturing Corp. of Chicago, which operates a casino in Atlantic City and is the largest manufacturer of slot machines in the nation. The investigation found no improprieties, however.

Three months ago, Arizona officials canceled a contract that Webcraft held to operate that state's instant lottery after allegations that the numbers on tickets printed by Webcraft could be detected without removing the covering label.

The allegations were made by Scientific, which subsequently was awarded the Arizona contract. Mott said the problem has since been resolved, and accused Scientific of making the allegation to gain a competitive edge. Scientific officials deny the allegation.