A Prince George County Circuit Court appeals panel set aside yesterday a judge's ruling that had ordered the county to rehire 37 jail guards who had been fired for their participation in a strike in 1980. The rehiring would have cost the county $1.5 million in back pay.

The three-judge appeals panel sent the case back to Judge James M. Rea, who decided the case last December, and told him to reconsider the grounds for ordering the reinstatements.

The guards were fired by former county executive Lawrence J. Hogan, who contended that the August 1980 walkout at the County Detention Center in Upper Marlboro was illegal. Last December, however, Rea ruled that the strike was legal and ordered the county to reinstate the guards with back pay, benefits and seniority. Rea agreed to delay the effective date of his order until the case was appealed.

County attorneys argued, and the appeals panel agreed yesterday, that Rea had exceeded his authority when he decided that the guards' strike was legal. The county argued that Rea, who had handled judicial cases that arose at the time of the 1980 strike, improperly based his reinstatment order on knowledge of those earlier cases as well as records before the County Personnel Board. That board had upheld the firings after the strike.

The appeals panel said yesterday that Rea should not have relied on facts from those earlier cases but should have considered only the records of the personnel board's rulings.

County Attorney Robert B. Ostrom told the appeals judges that 10 days before the 1980 strike, the Prince George's Public Employees Relations Board decided that a strike by the guards at that time would be illegal. After the strike, the personnel board, relying on the employe relations board decision, upheld the guards' firings, with a few exceptions.

Ostrom argued in court that Rea had no right to declare the strike legal and overturn the ruling of the employe relations board.

Yesterday's decision was reached by Circuit Court Judges William H. McCullough, James H. Taylor and and Richard J. Clark, sitting as an appeals panel.

Several weeks ago, the county offered to return about 15 of the fired guards to their former positions, with a small portion of their back pay, but the offer was turned down, according to John Wesley White, chief administrative officer for the county.

Leonard R. Goldstein, a lawyer representing 29 of the fired guards charged that the county was trying to harass the guards into accepting less than their full back pay.

White said that the county needs at least 16 additional guards at the troubled jail, but he added that the county can ill afford the estimated $1.5 million in back pay, or the "sweeping right to strike" that would result from Rea's order.

White said county officials are still willing to negotiate with the guards.