RICHARD M. NIXON'S memoirs, a president's portrayal of a controversy-ridden era, have started to stir new waves of social, political and publishing-industry ferment, even before the book has gone on sale.

An incipient protest campaign, including attempts to organize a book boycott, has been mounted by a scattering of Nixon detractors and is gaining widespread publicity. Some book sellers, moreover, have begun to express doubts that the $19.95 memoirs will catch on with the book-buying public. They say that the price may be too high, the book may reveal too little and readers may be wearying of Watergate.

"I just don't think people are going to spend $20 to find out not $3 worth of information." Barry Denenberg, general merchandise manager for the Brentano's book store chain, said in an interview last week. "If Nixon came out with seven incredible things, his book would sell."

Not all book dealers share Deneberg's view. Some large book distributors and book store chains are stocking up for a substantial sales rush, contending that "RN: The Memoirs of Richard M. Nixon" will have widespread market among Nixon supporters, history buffs, researchers, libraries and others.

Among memoirs protesters, the most outspoken appears to be a small committee formed in the Washington area with the slogan, "Don't Buy Books by Crooks." It calls itself The Committee to Boycott Nixon's Memoirs. One small Connecticut book store has refused to sell the 1,184-page book because of its owners' distaste for the former president. A lawsuit has been filed in New York and legislation proposed in Congress to stop political and other wrongdoers from profiting from royalties from their books.

All such developments have been greeted with a mixture of dismay and disagreement by Harold Roth, the president of Grosset & Dunlap, which is publishing the hard-cover version of Nixon's memoirs. "When you talk about boycotting reading the book, now you're beginning to talk about censorship and abridgement of freedom of speech," Roth said in an interview. "Certain aspects of the controversy are deplorable."

Noting that he himself was "not a political supporter of Mr. Nixon," Roth said tht Grosset & Dunlap had given considerable thought to a number of issues before it decided to publish the president's memoirs. But now, he said, he is optimistic that the book will sell. The first printing was 225,000 copies and, Roth added, "We're out of stock - we're getting orders beyond that." He described the memoirs, due out in a week or so, as "an important part of history" and "a darn good book."

THE DEBATE over Nixon's autobiographical work is taking place as some book dealers say they have begun to detect signs that the public is losing interest in the stream of books that has followed the Watergate era. "There is a declining interest in Watergate books," remrked Arthur Coons, president of Walden Books, the nation's largest book store chain. Many book dealers say, for example, that H. R. Haldeman's recently published book, "The Ends of Power," has sold fewer copies than they initially had expected, although sales, they add, have been substantial.

On Nixon's memoirs, the publishing world appears divided. Coons, despite his belief that Watergate books are losing their appeal, said that the 465-store Walden chain has ordered the memoirs in "substantial quantity" and expects the book to sell reasonably rapidly. Dick Fontain, senior vice president and merchandising director for the 309-store B. Dalton Bookseller chain, said Dalton had ordered about 12,000 copies of the Nixon book, but added, "It certainly begins to look in retrospect like that was an overly optimistic order." He said 8,000 copies probably would have been enough. Deneberg said the 30-store Bretano's chain ordered only 628 copies of the memoirs, less than a third of the number it would stock if it expected strong sales.

The nation's two largest wholesale book distributors, Ingram Book Co. and Baker & Taylor, also seem divided over Nixon's memoirs, with Ingram apparently dubious about the book's prospects and Baker & Taylor predicting substantial sales. Baker & Taylor distributes a significant number of its books to libraries.

The first book store apparently to boycott Nixon's memoirs has been The Remarkable Book Shop in Westport, Conn. "We don't feel that we want to work to promote Nixon's profit," said Sidney B. Kramer, one of its owners. The Connecticut Kramer is not related to the founder of Sidney Kramer Books in Washington, but the owners of the Washington firm also are dubious about the Nixon book. David Tenny, vice president of the Kramer chain in Washington, explained, "I'm getting a total of 25 copies (of Nixon's memoirs) for all four of our retail stores. I think it's going to be a bomb."

The court and congressional efforts to prevent wrongdoers from profiting from book royalties have not centered on Nixon's memoirs, though they have Watergate overtones. One such legislative proposal has already cleared the House, but it would apparently not apply to Watergate figures. Debra Jenkins, a legal secretary for the New York firm of Berney & Cousins, is suing to prevent Haldeman from receiving book royalties, contending that they should go, instead, to the U.S. Treasury. "To me, Watergate is one of the most despicable, demoralizing times in American history," she said.