EARLY IN VICTOR CANNING'S convoluted new thriller, the villain says to the hero, "'Pray God, Seyton, that life always sends you complicated situations -- they are so much easier to deal with than the simple ones.'" This typically witty Canning epigram rings true for all his characters. When faced with the most complex and dangerous situations -- this time involving spies, doubles, mysterious religious organizations, neo-Fascism and "Birdcage" (Canning's ruthless British intelligence operation) -- they handle themselves with aplomb. But when faced with simple crises of love, sex and loyalty, they fall to pieces.

Entirely nonviolent on its elegant surface, The Satan Sampler has a powerful undercurrent of menace that builds toward a genuinely unexpected climax. The reader is dazzled by the intricate architecture of the plot, by engagingly oddball characters, and by a rather disturbing exploration of the lure of right-wing politics in England's current economic malaise. With le Carre's Smiley's People and William Buckley's Who's on First, the publication of this splendid new Canning novel makes 1980 look like a good year indeed for espionage fiction. MISTAKEN VIRTUES. By Joanna Trollope. Dutton. 284 pp. $9.95

THE BEST THAT can be said of Caroline Harding, the unlikely heroine of Joanna Trollope's second historical romance, is that she is "the least offensively plain" of Parson Harding's children. Her nose alone, according to the equally unglamorous man who is forced to marry her, is "enough to make the world pity him."

Despite its unattractive protagonists and title, Mistaken Virtues is a thoroughly attractive novel. Set in 18th-century England and India, the story of Caroline's travails has bracing, cynical dialogue and a biting humor that are delightfully at odds with the sentimentality (such as the gushy ending) required by the genre. The character sketches alone make the novel worth reading:

"It was not that he was a priggish man, but rather that he felt that sin, like the classic definition of tragedy, only achieved greatness, and therefore any kind of distinction and interest, when commited by the great. Those people, obsessed by their lust for gold and pleasure, seemed to him to fall as short of greatness as was humanly possible."

Joanna Trollope writes superb historical romance, and she could probably do something more ambitious if she chose. Anyone who can get as much suppressed tension into a dinner conversation as she can has major potential. THE TREASURE OF SAINTE FOY. By MacDonald Harris. Atheneum. 242 pp. $11.95

MacDONALD HARRIS is a minimalist. In The Treasure of Sainte Foy, a suspense novel about a church treasure heist, his clipped syntax and spare diction are not just a means of keeping his story moving, but a way of pinpointing the suggestiveness of tiny details. The obligatory sex scene early on in the book, for example, is too terse to be erotic, but is strangely provocative nonetheless: "It is an awkward moment turning back the corners of the bed. It is too domestic, and too explicit. The white bed linen, when revealed, is almost more suggestive than a naked body." All the formula scenes (including the stakeout, the heist, the shootout) seem fresh and nonformulaic.

Harris, author of six other novels (including The Balloonist, a National Book Award nominee for 1977) does allow himself great expansiveness in detailing church architecture and the history of his rural French setting -- sometimes to elegant effect, sometimes to the point of pedantry. Indeed, he occasionally seems more interested in art objects than in people. His characters are as detached from each other as he seems from them. This is a rather cold book, but it is certainly an original and enticing one. NIAGARA. By Robert Lewis Taylor. Putnam's. 500 pp. $13.95

THIS LONG, SPRAWLING historical novel about Niagara Falls in the early 19th century is a curious mixture of liveliness and wordiness. All of Niagara's many characters -- the young newspaperman who is the narrator, the river rats, the stuntmen, the con artists, the European aristocrats, the Southern colonels -- are aggressively extroverted, and all of them talk too much. There are so many bubbly bits of Americana, so many sustained passages of folksy dialect, so many tall tales and anecdotes within anecdotes, that the reader wishes after awhile that Robert Lewis Taylor weren't quite so taken with his delightful subject.

But just when all the high spirits threaten to cancel each other out, Taylor (Pulitzer Prize-winning author of The Travels of Jamie McPheeters) invariably comes through with a character portrait of irresistible vividness: "He was polite as a half-drunken man could be, but his role at the table was feeding a wretched small dog, which climbed to his lap to make things easier. The fare was spaghetti disguised by curry (most sea-fare then), and we found this dreary preservative at every meal thereafter. The captain was not sober enough to prevent the spaghetti from slipping from both sides of his month, and it was on these strands that the dog fed." With so many rambunctious characters appearing and vanishing, the Falls themselves make the most lasting impression. THE NIGHTWALKER. By Thomas Tessler. Atheneum. 192 pp. $8.95

ALTHOUGH EMBELLISHED with bisexuals, punk-rockers and a berserk Vietnam War veteran, this werewolf tale makes little attempt to get away from its formula. Much of the novel, including the fortune-teller scenes and the telling of the story from the monster protagonist's point of view, reads like a slightly updated version of The Wolf-Man. Nevertheless, The Nightwalker has an uncluttered narrative drive that keeps the reader turning its appropriately few pages. Thomas Tessier is a crude, unpleasant writer, but his animal brutality is in tune with his theme. The brevity and straightforwardness of The Nightwalker are in admirable contrast to the wordiness of so many horror best sellers.