By John Katzenbach

Ballantine. 438 pp. $24.95

John Katzenbach's ninth novel, "The Madman's Tale," is a big, ambitious exercise in horror, doubly horrific in that it takes place not only in a mental institution but largely within the mind of an inmate there, one Francis Petrel. When we meet Francis, he is 41 years old, long since discharged from Western State Hospital in Massachusetts and living a quiet life, trying to maintain his fragile hold on sanity. Then a chance encounter with a former fellow inmate moves him to relate the events surrounding a series of murders at the hospital 20 years earlier, in 1979. For the rest of the novel we move back and forth between Francis as he writes his story -- off his medication, hallucinating, increasingly unhinged -- and his narrative of the long-forgotten murders.

Francis is a schizophrenic who has been guided by voices since childhood but seemed harmless until, during a family argument, he threatened his parents and sisters with a kitchen knife. The outburst landed him at Western State, where he is forced to adjust to a frightening and dangerous new world. Katzenbach was a reporter before he took up fiction, and he has clearly spent some time observing mental institutions; his richly detailed description of life there is entirely persuasive.

Ironically, the first friend Francis makes is almost certainly sane. Called Peter the Fireman, he is a Boston firefighter who burned down a church (to punish a priest who had abused his nephew and other youths) and was sent to Western State for evaluation. Francis also becomes friendly with an inmate who thinks he's Napoleon and another who fancies herself Cleopatra. We meet the hospital's medical director, an Indian named Dr. Gulptilil, called Dr. Gulp-a-pill by the inmates; the surly social worker Mr. Evans, called Mr. Evil; and a pretty nurse-trainee called Short Blonde for her hair. The plot kicks in when Francis and Peter the Fireman find Short Blonde, who had been on night duty, raped and murdered, with four of her fingers sliced off.

A Boston prosecutor named Lucy Jones hastens to the hospital, because she thinks the murder is linked to three others that involved blond women who had fingers sliced off. We learn that Lucy, whose beauty is marred by an ugly scar on her face, was raped and disfigured when she was a law student by a man who wore a ski mask and was never captured. On the theory that an investigation in a madhouse demands unorthodox methods, Lucy recruits Francis and Peter the Fireman to assist her, over the objections of Dr. Gulp-a-pill. Other murders follow, and it seems that someone highly intelligent is able to move through the hospital and kill at will. There are hints that he might be the man who raped Lucy years earlier. The killer is glimpsed at night, seeming to glow in the dark, and becomes known as the Angel. Or was he only a hallucination?

Katzenbach, having plunged us into an all-too-real world of madness, keeps us guessing over the identity of the killer. Everyone is a suspect. Dr. Gulp-a-pill and Mr. Evil are suspicious characters, Peter the Fireman is no Boy Scout, Napoleon might be more dangerous than he seems, and scores of men are wandering about who have violent pasts and are classified as everything from mentally retarded to psychopathic. Nor can the reader ever forget that Francis himself has been called an "unreliable narrator." Could his true role in the events be less innocent than he tells us? When plot twists seem bizarre or melodramatic -- such as Lucy's scar, for which she refuses plastic surgery -- we remind ourselves that this is, after all, a madman's tale.

I am no expert on horror fiction -- I am cursed with a literal mind, don't believe in ghosts and have never finished a Stephen King novel -- but "The Madman's Tale" suggested two points of comparison to me. One being the black-and-white movies of the 1940s that confronted us with crumbling mansions, bursts of lightning, locked rooms, distant screams, creepy servants and mad scientists. Katzenbach's madhouse achieves that same claustrophobic sense of dread. Like those movies, he sometimes serves up dialogue that is both stilted and ominous, such as this exchange when the social worker objects to Lucy's plan to use Francis and Peter the Fireman in her investigation:

" 'I'm not certain that these two can assist you all that well,' Mister Evans said, shaking his head.

" 'Perhaps they can, perhaps they cannot,' Lucy Jones said. 'That remains to be seen. But one thing is certain, Mister Evans.'

" 'And what is that?' he asked.

" 'At the moment, they are the only two people I don't suspect.' "

(Meanwhile, up in the balcony, we spill our popcorn and shout, "No, Lucy, don't trust those two crazies!")

Katzenbach, of course, is not a 1940s screenwriter but a skilled and sophisticated contemporary novelist who knows exactly what buttons he's punching and why. The other work his novel brought to mind was Dennis Lehane's recent "Shutter Island," another first-rate horror story that is set in a mental institution and plays with our minds -- although Katzenbach offers a more optimistic ending than Lehane did. If "The Madman's Tale" is sometimes over the top, why wouldn't it be? It's a tour de force, superior storytelling designed to scare your pants off and likely to succeed.