Gay Talese is unhappy.

He has just spent nine years to earn $4 million in the writing of his much-bruited and critically bruised book on sex in America, entitled "Thy Neighbor's Wife."

It is a "great book," he will tell you, riding in a taxi to a radio talk show; a book that "is written exquisitely, in my humble opinion," he will say in a limousine, or while he punches the elevator button, over and over, heading up to his hotel suite. He is an intense, delicately gestured man who turns away from his listener to sit in three-quarter profile while a thunderstorm gathers outside.

He says: "I was always well liked. I'm likable. But now it's changing.

People who don't know me are angry with me. This venom, this meanness . . ." p

He keeps going back to it, a tongue to a sore tooth, and he has just that look on his face as he talks about the critics, a puzzled, wide-eyed flinch: "This mean-spirited response to me has not been pleasant. I can take it professionally, it won't hurt the book, the book is going to find an audience. But (he touches his left eyelid with his little finger) I've caused such unhappiness in the reviewers. The book has made them more unhappy than they were before they read it. I've made them unhappy and vengeful, inspiring in them a desire to destroy, a desire to destroy me, I've done that, or the book has done it, or the both of us have done it."

He adjusts the position of a drinking glass, moves it one inch across a table.

"I'm getting like Miss Lonelyhearts in the novel, from all the letters he got, depressed, I'm getting a sense of human nature from these critics."

And on and on . . . "ugly" . . . "destroy" . . . "a depth of anger . . . "

"He had first noticed a massage parlor in his neighborhood one night while returning home from P. J. Clarke's tavern with his wife. Flickering from a third-story window on Lexington Avenue, near Bloomingdale's was a red neon sign that read Live Nude Models, and he was amazed that such an establishment could operate so openly.

"The next day at noon, alone, he returned to the building, climbed three flights of steps . . . "

That is Talese in 1971, as described by Talese in the book, beginning his nine-year journey through what he believed to be a "sexual revolution" in America. Along the way, he imperiled his marriage by telling reporters he was experiencing the sexual revolution firsthand, from patronizing and managing massage parlors to joining group sex parties at a California retreat called Sandstone.

He puzzled and titillated the literary world. He compiled a roomful of notes, tapes and journals he paid girls to keep in massage parlors. He acquired the knack of asking extraordinary questions of ordinary people. Yesterday, for example, he could ask a Washington man in a wheelchair, whom he had known for five minutes: "Do you go to prostitues? Massage parlors? Do you believe you're a sexual being? Do you find that sometimes, your partners are giving you -- for lack of a better term -- a mercy f--?"

It was clear soon after he started the book that it would be a long haul. He insisted that any major character be identified by real name. Thousands of people would talk to him, but it took years to find the ones who would trust him with their names, and sign releases. When he finally got to writing the first of it, in 1975, it came slowly, as little as a paragraph a day.

The public waited. Having read earlier best-selling works such as "The Kingdom and the Power" (1969), and "Honor Thy Father" (1971), they would take the sex book on faith.

Talese, now 48, had been a founder of what Tom Wolfe called the "new journalism." But he had eschewed its pyrotechnics for a gray, smooth interior-monologue sort of prose he'd polised at The Times. He was an arbiter of reality, standing, as an early book title had it, between "Fame and Obscurity." He made legends of men who built the Verrazzano Narrows Bridge, he showed us Frank Sinatra with a head cold. Always, he'd straddled contradictions: a year-round resident of a summer resort; an Italian kid in an Irish town; a New Jersey kid at the University of Alabama, smart with bad marks; a chronicler of the rules who couldn't play by them.

He had grown up in Ocean City, N.J., most of it empty and boarded up for nine months of the year. He was the son of an Italian tailor; a poor student, an indifferent socializer, never an athlete. What he liked, he says, was knowing about people, how they lived, the most ordinary facts of their lives. He began his career at 15 by writing a story about a pet cemetery, hoping his style would echo the "urbanity, the distance" of sports columnist Red Smith, whom he read in the Philadelphia Dispatch.

His father had to rely on a friend to get Talese into the University of Alabama, where he finished in the bottom half of his journalism class in 1953. aFinally, after an Army stint, in his early 20s he became a copyboy at The Times. His first story was about the men who pushed customers along the Atlantic City boardwalk in rolling chairs, and he probed their lives, the whole history of boardwalk chairs, the most ordinary of facts that became his personal property, somehow, by the mere noticing of them.

In Esquire in the mid-'60s, he went on to chronicle the despairs of a Joe DiMaggio at loose ends; Frank Sinatra during a lull in his career; Alden Whitman, who wrote obituaries at The Times; Floyd Patterson; Joe Louis . . . they all became fascinating in their ordinariness.

The cool marble of Talese's prose was flecked with the most extraordinary detail. He once found a chauffeur in New York so rich that he had his own chauffeur. He discovered scenes like Joe Bonnano, the aging Mafia don, soothing his daughter-in-law, advising her to have patience and courage. At the end of the new book, he described himself on a beach at a New Jersey nudist camp. Some boats, possible containing family acquantances from Ocean City, have gathered.

Many of them "held silvery telescopes and dark binoculars, and they sat rigidly on their decks and swayed in the water and squinted in the sun. They were unabashed voyeurs looking at him; and Talese looked back."

Seeing; being seen. By the time he began this book, Talese had become a celebrity at literary gatherings, dinner parties with close friends David Halberstam or Michael Arlen; or that station of fame's cross on Second Avenue known as Elaine's. He was established, respectable. In 1959 he had married Nan Ahearn, an editor at Random House (now at Simon and Schuster) and fathered two daughters.

Urbanity and distance: Sometimes, writing on deadline for magazines, he would pin the typewritten pages to a red felt wall behind his desk, then walk to the other side of the room and with a pair of his father's opera glasses (later, after a theft, he would use David Halberstam's binoculard) he would read his own words, the watcher watching the watcher watch . . .

In 1969, when "The Kingdom and the Power" came out, there was a bit of critical grumbling that The Times didn't deserve Talese's monomaniccal attention, but the book was a best seller. And in 1971, when he rendered the Mafia as a collection of routine, sometimes boring and usually ordinary people in "Honor Thy Father," some readers complained that he was giving gangsters moral sanction.

And now the sex book: "I'm really just writing about other people," he says in the hotel suite. "It's not condemning, it's saying this is the way people are. These are ordinary people, they are your neighbors."

The book virtually ignores the growing prominence of homosexuals, the women's movement, and the whole weird world of fetishes that have now been reduced to code words in sex magazine classified ads -- S & M, B & D, leather, water sports . . .

"I wanted to write about middle-class couples, the average man."

Hugh Hefner? Screw magazine publisher Al Goldstein? Massage parlor owners?

"Their customers are middle-class, middle-aged men," he says. "That's who I was interested in."

A current line in New York has it: Talese set out to write a book on the '70s, and wrote one about the '50s.

And, after 548 pages of determined, almost dogmatic mundanity, even the author's own excursions -- so much celebrated in New York and Esquire magazines -- have become harmless. Witness Talese in a massage parlor:

"'Oil or powder?" she asked, approaching the table. He looked uncertainly around the room.

"'Are there showers in here?' he asked after a pause.

"'No,' she said.

"'Then I'll take powder.'"

The criticism has lacked focus. In The New York Review of Books, Alexander Cockburn tries to write a spoof review analyzing Talese's sex data in terms of Marx's theory of surplus value. In The New Republic, Barbara Harrison states that the book is not only "boring, it is grim." Then she goes for another three pages in a dudgeon higher than that which boredom usually provokes.

"I want to understand this," Talese says. "I want to understand (the critics') anger and write about it. I want to interview them all, and see what's in there. I'm going to write about this. It's a new experience for a guy who's gone to a lot of places. I want to start with one Mr. Yardley (book critic for The Washington Star) and go on to Mr. Mordecai Richler at New York magazine, then Barbara Harrison . . . "

After all, isn't he the man who tamed Frank Sinatra and Joe DiMaggio, who brought us the great gray New York Times on a platter and turned the Mafia into the folks down the street? Couldn't he do the same for bunch of book reviewers?

He admits that he may have provoked them, that it might have been another book if it hand't been this one. "I try to look for God in the devil. Maybe one of the things I want to explore is anger, the anger in these people. Maybe what scares them is that I'm saying there's nothing to be scared of. I don't frighten very easily. If someone came into the house with a knife, I'd be cool. I'd engage him in conversation."

And here his little mouth curves upward in a perfect puckish U, and his eyes are bright and huge. "I might ask him questions about himself and make him feel better about himself without his knife. Maybe he'd give up his knife."

And the burglar would become just another troubled light-night visitor, ordinary, out-of-place.

There is nothing to be scared of -- not even, apparently, Gay Talese himself, a lion of Elaine's, possessor of a Manhattan town house, 40 suits, $4 million from his book and movie rights, and wife and children who have stayed with him Through It All.

He is impeccable -- sprinting around his bedroom, tossing custom-fitted shirts and Brioni suits (from Giorgio's of Rodeo Drive in Beverly Hills) into his luggage, calling for a bellman twice, blinking, shooting his jaw . . . i

"I'll be going back to Europe with my father for the next book," he says. His next book is about himself, his family, his father's immigration from Italy. The burglar, this time, will disarm himself . . . the Catholic boyhood in Ocean City, all those summer houses standing gray and empty through the winter, the pet cemetery . . . and somewhere, there just might be a knife.