A FAMILY MADNESS. A novel by Thomas Keneally. Simon and Schuster. 336 pp. $17.95.

WHAT HAPPENS when an exceedingly ordinary man falls obsessively in love with a woman whose family bears the weight of a convoluted, violent history that he cannot possibly understand?

Nothing good, naturally. This latest novel by the prolific Australian writer Thomas Keneally traces the course of an improbable involvement between Terry Delaney, a part-time security guard and professional Australian rugby player, and Danielle Kabbel, the daughter of a crazed Belorussian emigr,e who spent the formative years of his childhood in a family of World War II Nazi collaborators.

Throughout his career, Keneally has been drawn to exotic (for an Australian writer) cultures and subjects. His last novel, Confederates, dealt with the American South during the Civil War and was widely praised in this country for its uncommon outsider's insight into what would seem to be a quintessentially American subject.

Keneally is best-known in the United States for a non-fiction book, Schindler's List. It recounts the extraordinary story of Oskar Schindposition as overseer of an enamelware factory in Cracow to save Jewish workers from the gas chambers during the Nazi occupation of Poland.

Schindler's List is a true story that, as conventional book blurbs assert, "reads like a novel." A Family Madness, in contrast, is a work of fiction that reads more like the semi- journalism known as "faction."

The novel alternates between Australia in the 1980s and a narrative history of the Kabbel family's previous existence in Belorussia and post-war deportee camps. The Belorussian segments are presented in the form of a history written by Danielle's father, Rudi, and of a diary by her late grandfather, Stanislaw, who served as chief of police in a small, Nazi- occupied town near Minsk.

This choppy, artificially intrusive structure -- which also includes excerpts from Delaney's rugby match diary -- detracts from the believability of both the plot and its characters.

Not that there is anything inherently implausible in the early portions of the Kabbel family narrative. There were, of course, many fanatical Belorussian (and Ukranian, Latvian, Lithuanian and Estonian) nationalists who cheerfully collaborated with the Nazis. Some of them remained enmeshed in nationalist fantasies long after the war, when they had emigrated to countries like the United States or Australia.

It is also plausible for a character like Rudi Kabbel to go into the private security business in his new land (his father, after all, was a police chief) and ultimately to adopt a paranoid survivalist creed.

What is implausible, though, is the idea that all of Kabbel's Australian-born children would, instead of engaging in the generational conflict that more commonly characterizes immigrant families, become captives of their father's most destructive fantasies.

Terry Delaney, the most engaging and developed character in the novel, is a contented husband before he meets Danielle. When he makes love to his wife, he has a sense of "remaking and honoring the earth."

His greatest love, though, is rugby. He is a third-grade rugby player (in American terms, the equivalent of a minor- league baseball player). In this regard, he is a familiar type -- an athlete with enough talent to play professionally but not quite enough to make it as a player in the big leagues.

"A sport could be to people like Delaney not merely a sect but a cosmology, a perfect model of an imperfect world . . . Delaney had all the mental attributes necessary to the professional athlete. The mystery of why his talent was not greater was one he accepted together with the other mysteries."

DELANEYis also a guilt-ridden Irish-Australian Catholic, poised uneasily on the cusp of the Second Vatican Council even though he confesses to a priest who hawks situation ethics from the pulpit. (His theological predicaments seem somewhat anachronistic in American terms; I assume from Keneally's description that the combination of ethnicity and Roman Catholicism retains an insularity in Australia that it has lost in the United States.)

When Delaney first encounters Danielle (his security firm has gone out of business and her father hires him), he finds her reading the novels of Graham Greene. Would a young woman who takes courses in the English-language novel be interested in a relatively uneducated man? What makes her reply "certainly" when Delaney, with few preliminaries, asks if she will become his "lover" in an exchange that seems out of place even though Keneally takes pains to tell us it also seems "strange" to Delaney?

A man like Delaney and a woman like Danielle could fall in love (anyone can fall in love with anyone, after all), but Keneally simply does not lay the psychological groundwork to explain their passion.

In familiar fashion, a magnetic father like Rudi Kabbel could lead an entire family to destruction -- but Keneally does not make his personal charisma believable.

When Kabbel first meets Delaney, the father delivers an impassioned lecture on the extermination of the Belorussian bison by the Germans and Russians. Paternal diatribes of this ilk are more apt to drive away an ardent suitor than to fuel obsessive love. Moreover, such lectures are quite likely to persuade one's children that the whole continent of Australia is too small to accommodate both generations.

Although he is a writer of great skill and versatility, Keneally fails this time to make us enter into the world he has created. He has spun a tale that seems implausible rather than improbable.