Secrets -- layer upon layer of suppressed knowledge -- interlace the fabric of the Longley family. Writing under the pseudonym of Barbara Vine, British mystery novelist Ruth Rendell has concocted a chilling tale of the secret loves and hates, the hidden malice and unyielding resentments that led Vera Hillyard to be hanged for murder in 1950. Left teasingly open-ended is the central secret that precipitated the murder. As with Frank Stockton's famous story -- "The Lady or the Tiger?" -- readers are left to draw their own conclusions.

In "A Dark-Adapted Eye," Rendell, whose two dozen detective novels have earned her best-sellerdom, has turned away from that genre's neatly plotted crimes and tidy solutions to explore the more devious territory of the human psyche. She has spun her haunting psychodrama against a background of prewar England's social climate and of Victorian mores that placed a higher premium on appearances than on truth. So subtly has she told her story, moving backward and forward in time over a 35-year period, and changing voice as one character after another contributes memories and insights, it would be unfair to detail the plot. Let us, instead, meet the principal characters.

We begin with the narrator, Faith Severn, who was Vera Hillyard's niece. She is propelled into reconstructing the events that led to the murder when a journalist proposes to write a book about the case. In 1939, 10-year-old Faith was sent from wartime London to stay in a country cottage with her father's two sisters, Vera and Eden. Her early memories are of the suffocatingly close affection between her aunts and of their seemingly idyllic life style of home-baked high teas and evenings spent at needlework. Faith is intimidated by priggish, snobbish, endlessly critical Vera, who tyrannizes everyone except the much younger sister to whom she is slavishly devoted. It is only much later that she comes to see Vera as one who endured as much tyranny as she imposed.

Beautiful, golden-haired Eden is vain, selfish and amoral. While ostensibly reciprocating Vera's love, she subtly undercuts her at every opportunity. When she leaves the village to enlist in the Women's Royal Naval Service, she tells lie after lie about her activities and her whereabouts. Vera, who is privy to the truth, protects her sister by hiding her knowledge.

Then there is Francis, the son born to Vera and Gerald Hillyard, the military officer she met and married in India. Francis is consumed with hatred for the mother who sent him to boarding school in England when he was 7 and left him there when she returned from India to become a surrogate mother to the newly orphaned Eden. Whenever he is home on holiday, teen-age Francis subjects Vera to cruel pranks; his malicious trickery is often stealthily abetted by Eden.

A frequent visitor to the country cottage is Chad Hamner, a journalist introduced to the family by Eden. Originally thought to be Eden's beau, he is later suspected of fathering Vera's illegitimate son. Chad has his own secret, one that is accidentally discovered by Faith, who has kept it to herself for she, too, is a Longley to whom harboring secrets is a way of life.

More an observer than a participant is Helen Chatteriss, Vera and Eden's kindly and generous older half-sister, who took Vera under her wing and arranged her marriage to Gerald.

Then there is Tony Pearmain, scion of a wealthy merchant family, who is ignorant of Eden's past when he marries her and whose desire for children is frustrated when Eden miscarries.

Crucial to the story is Jamie, born 10 months after Vera's soldier husband has left England for the front. The object of a tragic tug of war between the sisters, Jamie was 6 at the time of the murder. He professes to remember nothing of his life before then.

Finally, we meet Josie Cambus, Vera's neighbor and friend, who was an eyewitness to the murder but whose testimony fails to save Vera from the hangman's noose.

Probing the decades-old memories of the survivors, Faith uncovers a web of false fac,ades and shattered loyalties that leaves no one untouched. "Murder reaches out through a family," Faith concludes, "stamping transfers of the Mark of Cain on a dozen foreheads, and though these grow pale in proportion to the distance of the kinship, they are there and they burn into the brain."

As Barbara Vine, Ruth Rendell has burst brilliantly out of the mystery writer class. The characters in "A Dark-Adapted Eye" are not puppets manipulated for the sake of a tricky plot. They are the plot, and they will linger in your memory long after you have closed the book with the contented sigh that is the ultimate tribute to a first-rate novel.

The reviewer is a New York writer and the former editor of "Dialogue."