After a protracted fight and several public hearings, St. Mary's County officials have won an appeal to use the top floor of the county library to house overflow prisoners until the chronically crowded county jail is replaced by a new detention center in about two years.

The town's appeals board agreed to issue a temporary occupancy permit to the county that would allow a maximum of 26 work-release or weekend prisoners and others convicted of nonviolent offenses to stay in dormitory facilities above the library.

The county sought approval of Leonardtown officials because the public library is in an old state police barrack and National Guard Armory just inside the town limits.

When the library proposal was first made in March, it outraged many in the small Southern Maryland community who contended that a jail annex was not a proper use for a library in a residential neighborhood. Library staff members and town residents told town officials that they were worried about security.

County officials argued throughout the debate that emergency relief was desperately needed for the jail. It routinely holds 75 prisoners, twice the number it was built to house in 1942.

The jail is described by southern Maryland law enforcement officials as one of the oldest and least adequate facilities in the region. Prisoners were occasionally sent 35 miles to the north to Charles County's detention center.

Leonardtown planning commissioners rejected the jail annex proposal twice after town residents attending a public hearing voted informally against the plan. County officials then took their case to Leonardtown's Zoning Appeals Board.

"Fear was the driving force behind the citizens' objections" to the library proposal, said County Administrator Edward V. Cox. "The rezoning was not discussed on the basis of fact. No questions were asked about how we planned to handle security."

In its case before the board, the county argued that the town's planning commissioners acted illegally when they used an informal vote by residents as the basis for their decision, according to town secretary Marilyn Enwright and official minutes of the appeals board hearing.

The county also cited existing zoning law that says the county may use a public building even in a neighborhood zoned for single-family houses, provided the need can be demonstrated, Enwright said. The present jail, in downtown Leonardtown about three miles from the library, is also in an area zoned for detached houses.

The town appeals board ruled that before the library facility can be used, the county must provide separate entrances for the prisoners and library patrons. Guards are to be stationed on each floor during library hours. And the alarm system for the jail annex must be wired into the main jail.

The county must vacate the library by July 31, 1987, or face immediate eviction, according to the agreement.

The St. Mary's jail has been the subject of several critical grand jury reports in recent years.

In two upstairs cells, 14 prisoners share sleeping quarters meant for six. There is one shower and one toilet.

In another room, mattresses for 16 other inmates are rolled up and stashed under bunks during the day.

The jail does not have air conditioning, and temperatures on the second floor have gone above 110 degrees on some days, a condition that one grand jury report called "inhuman."