"Money," says Lyle Stuart, his nose wrinkling slightly in distaste above they luxuriant, black moustache and beard. "I don't care that much about money.

"I turned down a $70,000 bid for serial rights to 'Jackie Oh!' from The National Enquirer to take a $30,000 bid from the Ladies' Home Journal. I just didn't think it was a National Enquirer kind of book."

Publisher Stuart is the man who brought you such goodies as "The Sensuous Woman," "The Last Chance Diet," "Naked Came the Stranger" and "The Anarchist Cookbook." His small firm in Secaucus, N.J., has been associated in the public mind with controversial books that make big money, and his is currently struggling with another kind of controversy. The Justice Department is investigating ways to stop the publication of "Dirty Work," a book about the CIA that "blows the whistle on 850 agents," according to advance publicity.

Stuart and former CIA agent Philip Agee, who wrote the foreword to "Dirty Work," may face espionage charges and/or a civil suit to stop publication of the book. If it brings Stuart back into a courtroom, he will not come as a stranger. He got the money that allowed him to start as a publisher in a lawsuit, and he has been in and out of court frequently since then, fighting charges that range from obscenity to libel against King Farouk.

The $22,000 stake on which Lyle Stuart Inc. was founded in 1956 was won in a libel suit against Walter Winchell, the culmination of a lenghty feud between the columnist and Stuart, who was at the time the editor of Mad magazine and of his won monthly expose magazine, The Independent. Angered at an attack by Winchell on singer Josephine Baker, Stuart wrote a biography called "The Secret Life of Walter Winchell," and the war was on.

Stuart's version:

It escalated when someone entered a pornography complaint against one of the companion-publications of Mad magazine, and the police came to the publishing office. Because publisher William Gaines was worried about being dragged off by the police, Stuart agreed to go instead. The case was quickly dismissed by a magistrate, but that was not the end of it.

According to the publisher, Winchell got a garbled version of the story and announced on his radio program: "Attention, all booksellers: Anyone selling the fifth of Lyle Stuart will be subject to the same arrest."

"When I heard that," says Stuart, "I smelled money."

The $23,000 settlement, as Stuart remembers it, came quickly after Winchell was heard in court calling Stuart a "son of a bitch." Part of it went to finance the first book he published, a book on health called "The Pulse Test."

"I published that book because I believed in it," Stuart says, "and nobody else would publish it. I didn't want to be a book publisher. I still don't." The round face broadens into an innocent smile - the victim of circumstance accepting his fate. "It just happened."

Since 1956, his basic policy has been the same; he published books that other people won't publish. Now that he has a reputation for it, the manuscripts come to him almost automatically, and so does the controversy that surrounds many of his books: "That just happens, too. I certainly don't look for it. I don't have to."

Among the other things that just happen to Lyle Stuart, whether he cares about money or not: He claims he has won a total of $166,505 on his last 10 trips to Las Vegas ("It's not the amount," he insists, "but the fact that I came out ahead on every one of those 10 trips"). He tells the story in "Casino Gambling for the Winner," which he wrote in three very intensive days and which is the book he really wants to talk about: "All you have to do," says the promoter, is tell people that the book exists."

That is a characteristic of many Lyle Stuart books, from "The Anarchist Cookbook" to "The Sensous Woman." They don't usually get good reviews (Stuart doesn't care about reviewers and sends out review copies only reluctantly), but somehow people hear about them and eventually want to buy them.

"I published 'The Anarchist Cookbook' against the wishes of everyone in my office," Stuart recalls. "I liked it, but nobody else did - and of course no other publisher would touch it. You know, it tells you how to make Molotov cocktails and blow up police stations.

"I went out on the road to promote that book, because the author was only 21 and nervous, and then I went into court to defend it. One judge in Denver announced in court that the book would be given a fair trial even though he had received a bomb threat a few days earlier."

Is there any kind of material he considers unfit to publish? "Let me put it this way," he answers. "Long ago, in partnership with Norman Mailer, I published a book called 'Jiu Jitsu Complete.' Some of the blows in that manuscript were lethal, and I took them out.

Other than books with lethal blows, Stuart says, "I try to publish books that interest me personally - I don't think I ever did a book with the idea, that controversy would help it sell it. Until a few years ago, I turned down anything that I thought was commercial - anything that other publishers might be willing to do."

With "Jackie Oh!" (described in his catalogue as "the first truly initimate biography of Jacqueline Kennedy Onassis"), Stuart has clearly overcome his aversion to possible commercial success. Somehow, he has fallen into similar successes with a long string of earlier titles, beginning in the '50s with the sexual liberation books of Albert Ellis and continuing through "The Sensuous Woman," "The Sensuous Man," "Captive City" (about the Mafia in Chicago), and "The Last Chance Diet."

His promotional masterpiece was perpetrated in the late '60s with a book that had one of the most perfectly resonant titles of all time: "Naked Came the Stranger, by Penelope Ashe - a novel about the wildly varied amorous adventures of a suburban housewife.

Author Ashe, blonde and svelte, proved almost as popular as the book's contents, and she made the rounds of the talk shows with a line of chatter that was both glib and sensational: You gotta tangle your guts in those typewriter keys."

When it was revealed that Penelope was a hoax (the book was written by a group of staffers from the Long Island paper Newsday, each coauthor being assigned a chapter), the news boosted sales even more. Ultimately, 25 people earned between $7,000 and $8,000 each in royalties on the book.

What Stuart is trying to promote right now is a contract to publish the memoirs of Field Castro, who he says is a long-standing personal friend.

"I read sections of it 10 years ago in Hanava - he had it translated into English - all about the time he spent in prison and how much Dostoevski meant to him. I thought it was terrific.

"Castro is very reluctant to publish - very self-effacing, you don't see his picture anywhere in Havana. He says, 'Who care what I like to eat, who I'm living with?

"But I'm very persistent. I'd say have a 50-50 a chance,"