What do you do when you have a statewide prison system seething with 11,500 inmates--150 percent of capacity, as the statisticians like to say--and a lot of them are sitting around doing nothing?

"You build on what's there and prioritize," says Maryland's new public safety and prisons chief Frank A. Hall. "Once you've got your priorities, you set out doggedly and try to do them."

Hall, 43, a hard-smoking alumnus of the Massachusetts and New York state prison administrations, takes a pragmatic approach and offers no fancy solutions or quick cures for Maryland's troubled correctional system.

Its problems "are not going to be solved in the next couple of years," Hall said. But he adds that he has high hopes for relieving prison overcrowding with the construction program currently on the drawing boards and by sorting out prisoners more rationally to assure that violence-prone inmates and other troublemakers stay in more secure confinement.

After security and crowding, Hall said he will turn to new rehabilitation programs. "There are a lot of people in the system who have nothing to do," he noted. But once additional cell space is built and beds are taken out of jammed-up gymnasiums, work sites and other "program areas," new vocational rehabilitation can begin, he said.

The Washington-born Hall, who is to be formally installed today as Maryland's new secretary of public safety and correctional services, refused in a recent interview at his new office here to label himself a conservative or liberal prison administrator.

"I take a balanced approach, and I'm not an ideologue," he said.

Appointed by Gov. Harry Hughes to succeed Thomas W. Schmidt, Hall inherits a sprawling 8,000-employe bureaucracy responsible not only for prisons but a dozen other agencies ranging from the state police and civil defense to the Parole Commission and something called the Sundry Claims Board.

But the prison system, with its 11,500 inmates shoehorned into eight facilities around the state, is the most sensitive and most difficult to bridle.

Hughes' first public safety and prisons boss, Gordon Kamka, a controversial reformer who preached getting more convicts out of the stone-and-steel environment of penitentiaries and into community-based facilities, left under a storm of protest in early 1981 amid allegations of lax prisoner work-release programs, disciplinary upheavals among inmates and general mismanagement.

He was replaced by Schmidt, the state's budget secretary. Though not a penologist, Schmidt had a reputation as a skilled nuts-and-bolts manager and won quick praise for tightening discipline and limiting abuses, such as walk-offs and narcotics transactions, in the work-release programs.

Hall, a prison administrator by training who previously headed the Massachusetts prison system and later directed the youth division of the New York state system, begins his $60,200-a-year job in Maryland today with no illusions.

One of his first priorities--relieving overcrowding--is already being addressed with bricks and mortar. A temporary facility with 450 beds should open in December at the state prison complex in Hagerstown with another 750 beds there some time after that. The state legislature also appropriated $75 million for two new facilities in Somerset County on the Eastern Shore that will house 1,500 inmates. But those facilities will not open for at least three years.

Meanwhile, more convicts are pouring into the system every day, exceeding the outflow of inmates attaining parole or completing sentences.

Hall says he has a "hunch" that the prison system will get still more inmates under the state's new uniform sentencing guidelines for judges as well as the trend toward mandatory sentences and parole limitations triggered by public outcries against increased violent crime.

Are there other remedies to overcrowding besides building more prisons? Yes, says Hall. One is an increased use of probation, the placement of selected defendants under close state supervision in lieu of jail time. Another is increased use of restitution, requiring convicted persons to repay victims of thefts and other crimes involving monetary loss, rather than sending them to prison. But both alternatives, Hall acknowledges, require the participation of sentencing judges, which cannot always be counted on.

Hall is opposed to another, more controversial alternative--a law, like one currently in force in Michigan, that would automatically trigger the release of selected short-term prisoners every time the official capacity of the prison system is exceeded.

"It's a last-resort mechanism," he said. " . . . It shakes the confidence of the public; it shakes the confidence of law enforcement officials. It's like a desperation effort."

Under the Michigan law, prisoners convicted of nonviolent crimes with fewer than 90 days to serve are released each time the state-wide system exceeds its capacity. In the last two years, the law has been invoked four times with 600 to 800 inmates winning early release each time.

Hall said he also hopes to "refine" the prisoner classification system, the process, already overhauled to some extent by Schmidt. It assigns inmates to maximum, intermediate and minimum security facilities to help them re-enter the outside world upon parole, but at the same time assures their confinement until then.