A FATAL INVERSION By Barbara Vine Bantam. 272 pp. $14.95 TALKING TO STRANGE MEN By Ruth Rendell Pantheon. 280 pp. $16.95

RUTH RENDELL, along with P.D. James, is Britain's principal recent gift to crime readers. Rendell writes three different sorts of crime story. The best-known are puzzles confronting Chief Inspector Wexford in the country town of Kingsmarkham, orthodox detective stories with a sharp edge of social comment and a fine ear for the nuances of contemporary speech. In the second sort of Ruth Rendell stories some flaw in personality, often with a sexual basis, leads to what may seem on the surface almost purposeless violence. The Wexford books are excellent detective stories, but the best of the studies in abnormal psychology (A Demon in My View, A Judgment in Stone, The Killing Doll) are powerful novels.

The third and most recent face of Ruth Rendell is Barbara Vine. Why use another name when the identity of Barbara Vine is no secret? To emphasize that these are not straight detective stories nor psychological studies, but something different again: densely-plotted leisurely mysteries with an almost Victorian flavor about their intricate development. The last decade has seen the full flowering of Rendell's talent, and the current offerings are Rendell brands two and three, a psychological thriller and the second Barbara Vine mystery.

A Dark-Adapted Eye, the first Barbara Vine, told us in the opening chapter of a woman being hanged for murder, and A Fatal Inversion begins with a couple burying a pet dog in the garden of their big old country house and finding as they dig in a cemetery for animals "the bones, the fan play of metatarsals, of a very small foot." The bodies of a young woman and a tiny child have been buried here, and the woman has been shot. The mystery in the novel, not solved until almost the end of it, is who they are and how they died.

This is a book that calls for a reader's full attention. The crucial events in the story take place in the summer of 1976, the longest and hottest for many years in Britain, and the narrative shifts constantly between past and present, showing how the shadow of the awful thing that happened at rambling old Wyvis Hall, in the heart of Suffolk country, has affected the four people who survived it.

In one aspect the story is about the destruction of innocence. Adam Verne-Smith, a footloose 19-year-old, is left Wyvis Hall by his Uncle Hilbert. He takes his friend Rufus down to look at the place and its 20 acres before selling it, but once there they are seduced by the lake, the warmth, the silence, the well-maintained garden, the carefully netted fruit cage with its vermilion strawberries. It is like the Garden of Eden, Adam feels, a paradise that can be entered only on the condition of committing "some frightful sin or crime that must result in expulsion from it."

THE FIRST CRIME is trivial, hardly one at all. Neither Adam nor Rufus, a medical student, has any money. Why not sell some of Uncle Hilbert's things, his Waterford glasses, little tables and so on, to maintain themselves? They all belong to Adam, even though he is not supposed to sell them before the estate has been valued. And why not start some sort of commune, getting rid of a few more things, and stay in this lotus land forever? They are joined by a young Indian with a Jewish girl friend, and Rufus picks up a girl named Zosie one night in his old banger. Adam, who has a taste for anagrammatic jokes, calls the old banger Goblander and Wyvis Hall Ecalpemos, someplace inverted. This tiny commune, like larger ones, has its sexual entanglements.

The telling of the story is less straightforward than such a summary makes it seem. Contrasted with the lost paradise of Ecalpemos is the present in which Rufus is a smooth successful medical consultant, Adam an anxiety-ridden man with a boring wife and a much-loved daughter, the young Indian submerged in the menacing urban emptiness of a London ghetto where the least offensive graffito says "Go Home To Pakistan." They agreed never to meet again after the terrible thing happened that drove them out of paradise, but when the bones are discovered and they learn that the police are treating the woman's death as murder, alarm bells ring . . .

The revelations when they come are not entirely unexpected, although the neatest ironical trick is turned in the last pages. But the greatest compliment one can pay Ruth Rendell as Barbara Vine is that she doesn't depend on tricks. She has the gift of devising an ingenious plot, and of maintaining interest by driving along the narrative with the zest and cunning of Wilkie Collins, marks of an extremely skillful crime writer who is also a very good novelist.

A writer who like Rendell refuses to accept the usual limits of the crime story is bound to miss the bull's eye sometimes, and Talking to Strange Men is no more than an outer, even though the main theme is handled with typical assurance. John Creevey, manager of a garden center selling all kinds of flowers and plants, is a lonely awkward man whose interests are confined to his plants and his wife Jennifer. When, after two years of marriage, she leaves him for a former lover, he carries on with life mechanically but thinks only of how he can induce her to return.

Blended with this theme is another in which teen-age boys attending two local private schools have evolved an espionage game played with total seriousness. Moscow Centre is opposed to London Centre, and they use all the apparatus of espionage, drops for code messages, safe (i.e. empty) houses for meetings and debriefing sessions, agents (some of whom defect), moles, etc. Creevey happens to see code messages being left in a pillar beneath an overpass, decodes some of them, concludes that they are being exchanged by criminals, and tries to use them to get his wife back.

The idea is ingenious, but even Rendell's skill cannot make the mock spying activities seem plausible, so that the story's would-be tragic finale seems merely confected. It may be that at the moment Ruth Rendell is writing too much -- an average of two books a year plus short stories is one that nobody except Simenon has maintained for long without some loss of quality. But a single misfire does not detract from the total achievement. The two Barbara Vine books (A Dark-Adapted Eye shouldn't be missed), the best of the Wexfords and of the psychological books make up a body of work that says as much about the habits, manners, talk and behavior of people in contemporary Britain, as those of Kingsley Amis, Iris Murdoch, or any other novelist in current practice.

Julian Symons' many books include "Bloody Murder," a history of crime fiction, and the forthcoming "Makers of the New: The Revolution in Literature, 1912-1939."