NEMESIS By Rosamond Smith William Abrahams/Dutton. 276 pp. $18.95

THREE YEARS ago, the pseudonymous novelist Rosamond Smith made her debut with a book titled Lives of the Twins. It was about a young woman who falls in love with her psychotherapist. Soon after setting up house with him, she discovers that he has a twin brother, also a psychotherapist, and that the two brothers are not on speaking terms. Under an assumed name, she begins consulting the brother. Naturally, she goes to bed with him, too. All this, happening rather quickly, sets the stage for a taut psychothriller with an above-average intelligence quotient.

Even before Lives of the Twins was published, word got out that the amazingly prolific Joyce Carol Oates was its true author. ("Rosamond Smith" is, in fact, a kind of feminine twin of the name of Oates's husband, Raymond Smith.) Unlike Doris Lessing, who published two novels as "Jane Somers" to make a point about the difficulties faced by unknown novelists, Oates had no polemical intent in mind. On the contrary, she seems to have invented Rosamond Smith as an experiment in the manner of the brilliant Portuguese poet Fernando Pessoa. Pessoa invented a quartet of poets, gave them names, imagined their lives, and then went ahead and created their oeuvres.

Oates hasn't told us much about the "life" of Rosamond Smith, but with Nemesis, Smith's third and latest novel, several things about her become abundantly clear. A specialist in credible nightmares, she regards humanity with a cool asperity and is especially sharp on social and professional relationships. Disaster and violence bring out the poet in her. From the scene of a car crash the injured are "borne off amid a jubilation of sirens." A woman runs away from an assailant, but "in the radiance of terror she was unable to scream."

Above all, Smith is interested in the imaginative possibilities and limitations of the Gothic mode, on the one hand, and the classic murder mystery on the other. She has a particular fondness for the great Gothic theme of the double and for the recurring detective motif of the assumed identity. But this is hardly a surprise, since a "twinning" of the self and a made-up name would seem to be the ground conditions for Rosamond Smith's very existence as an author. ON THE BASIS of Nemesis one can also say that Smith is a close reader of newspapers and a shrewd observer of local controversies. At Princeton, the university where Joyce Carol Oates holds a titled professorship, a prominent male professor was accused last year of sexually assaulting a graduate student, also male, whom he met at a party for incoming graduate students. The circumstances are much the same in Nemesis, and the first part of the book is an arresting fiction from that highly controversial front, the place where academic politics and sexual politics overlap.

On one level, Nemesis is a contemporary specimen of an old standby, the campus homicide novel. Who killed the vicious, pompous, universally detested Prof. Christenson, the much-honored mediocrity who terrorized his colleagues and students at the Forest Park Conservatory of Music? Virtually everybody had a sufficient motive, but the most likely suspect is a first-year graduate student named Brendan Bauer. Just a few months earlier, Bauer was raped by Christenson -- and could only look on helplessly as Christenson was dealt with leniently while his own name was blackened. Was it Bauer who injected the cyanide into the dozen chocolate truffles that Christenson received in the mail that fatal December morning? And if not, who framed Brendan Bauer?

The detective in Nemesis is an exceedingly minor character, with little if any effect on the action or the outcome, and the book's ingenuity has less to do with the solution to a puzzle and more to do with the development of a plot, in the literary sense. Nemesis is in three parts, each corresponding to a specific generic archetype. Part one, the story of an academic scandal, turns into the murder mystery of part two, and then both are transmuted into the Gothic melodrama of part three. Watching the metamorphosis is part of the fun.

Part one introduces us to the perfect Gothic heroine: the talented, appealing, unmarried, sexually repressed Maggie Blackburn. Maggie is the professor who throws the party where Christenson and Bauer meet; she is also the first person to whom Bauer reports the rape. An odd couple, Maggie and Brendan are linked by parallel psychological disorders: her susceptibility to fits of amnesia, his stuttering.

In part two, the murder mystery takes over. Smith cleverly lifts her murder method from Anthony Berkeley's classic whodunit, The Poisoned Chocolates Case (1929). She tells us that poison, whether injected into chocolate truffles or otherwise administered, is the "weapon of choice" for those who can't bear violence -- and for those who are wise to the ways of the genre, that seems to be a pretty broad clue to the culprit's identity. But then another professor is killed, and this time the method is quite different: Nicky Reickmann's throat has been slashed from ear to ear. Is there perhaps a second murderer?

The level of hysteria, suitably high in part three, culminates in lurid revelations: incest, madness, violent scenes from childhood. It turns out that the irreproachable X and his twin sister are living together as man and wife, Y is in grave danger, and Z has disappeared. Rarely have the Gothic and the homicidal gone so well together. It is pleasant to consider what Smith will do for an encore.

David Lehman is series editor for the annual "Best American Poetry" anthology.