WAITING FOR THE END OF THE WORLD. By Madison Smartt Bell. Ticknor & Fields. 322 pp. $16.95.

FROM TIME TO TIME you can hear the clanging toll of literary complaint, the ink-stained hands wringing and wringing in the night. All the really "big" subjects are foreign. Lucky Milan Kundera: he's got Soviet bloc repression. Lucky Gabriel Garc,ia M,arquez: he's got the phantasmagoria of Latin America. Lucky N.K. Narayan: he's got the strangeness of South India. Lucky V.S. Naipaul: he's an exile, a floating sensibility that is nowhere and everywhere.

American novelists often find their successes in "smaller," more personal subjects and some of the dominant voices in contemporary fiction -- Raymond Carver, Ann Beattie among them -- employ a brittle, minimal tone to voice their ironies and worlds. Their countless imitators are so excruciatingly minimal as to be quite nearly invisible.

Madison Smartt Bell, a young novelist born in Tennessee, educated at Princeton and steeped in the culture of contemporary New York (or better, Brooklyn), suffers no anxiety of subject. He himself was born into the atomic age, and his second novel Waiting for the End of the World keeps the ultimate bomb poised and ticking at its center. That is Bell's boldest stroke: to confront the prospect of atomic terrorism in a realistic, convincing way, to trace its roots in a slice of desperate and pathological New York, to harness the natural anxieties of the reader. The subject is as "big" as they come.

Bell's first novel, The Washington Square Ensemble was set mainly among the seemier strata of Greenwich Village, and now he has crossed the teetering Williamsburg Bridge to a similar Brooklyn. His Brooklyn is not the increasingly prosperous frontiers of Park Slope and Brooklyn Heights, but the crumbling margins, the neighborhoods that are far too desolate and old, even for slummers and other pretenders. Bell, who lives in the Williamsburg section of Brooklyn, has obviously kept his eyes open.

His protagonist is Larkin, a deeply confused and deluded refugee from the middle-class who lives from day to day, nightmare to nightmare. He is a failed pianist, a drunk, an epileptic, a character at once nasty and sympathetic. As he stumbles along the streets, past burned out warehouses and exhausted tenemants, he is witness to a myriad of inexplicable events. And because of his ratty condition, he is hardly noticed. He watches as a beautiful girl, a girl he has seen before in the streets, is terrorized by a brute. No one notices the witness:

"Larkin faded closer to the wall. The man's glance rotated through him without reaction. Larkin folded his arms, gripped his elbows hard. As he had long suspected, he had become invisible."

EVENTS GROW MORE inexplcable, more terrifying as the story develops. Larkin hooks up with some equally disturbed partners: David Hutton, a Vietnam veteran, Charles Mercer, an ex-cocaine runner, Ruben Carrera, a fatherless child and victim of a lunatic mother and Simon Rohnstock, a sour child of privilege, pathetic revolutionary and the ringleader of this mad "cell." After executing a series of "warm-up" murders, their scheme is to plant an atomic bomb made of stolen plutonium in the catacombs under Times Square. The novel moves toward the success or failure of that mission with terrific power; the plausibility of the plan gives the novel its engine.

The less successful aspects of Waiting for the End of the World are in the plan of the novel itself. After a dramatic prologue that describes the theft of the plutonium, Bell lingers over Larkin, his strangeness, his world, his invisibility. Here is some of the strongest writing in the book. The portraitof Larkin's desolation is powerful and deeply felt. But as he moves on, Bell can be too self-conscious, his structures and intentions too naked. A long chapter on the "cell" is written in the fashion of interwoven sketches. The intention, it seems, is to create a dossier, a compendium of motives that will somehow explain the cell's apocalyptic mission. But the form Bell chooses here feels too deliberate and "written" and the motives themselves too "created."

Bell also relies too much on the New York Post. Stangely, and perhaps sadly, that tabloid -- "HEADLESS MAN IN TOPLESS BAR" -- has been mined endlessly in recent years. The shock appeal is long gone. Bell devotes a whole chapter to "The State of the Earth According to the New York Post" and the effect is, once more, too self-conscious, too much of an interruption and "idea," not enough a true part of the novel.

Bell struggles with his voice at times, relying too much on a stilted irony. The prose is sometimes clogged with similes, redundancies and the occasional mistake: "In the passenger seat was Billy Morris, older and heavier than Henderson and far less trim." Or "Rita Jenrette, ex-wife of a discredited US senator . . ." (Mr. Jenrette was a congressman.) There are passages here of real brilliance; the descriptions of Brooklyn and Larkin are full of fire. But the lapses hurt. In general, Bell's best, most controlled writing so far, has appeared in short stories such as "The Naked Lady," which was published in Best American Short Stories 1984.

But ambition can sometimes carry a work, and ambition certainly makes Waiting for the End of the World well worth reading. Madison Smartt Bell provides promise: promise of his own talent and promise that young American writers are not all retreating from "big" subjects.