CBS Inc. retreated from the troubled paperback book business today by selling substantially all of the assets of its Fawcett Books Group to Random House Inc. for an undisclosed sum.

CBS also wants to sell its Popular Library imprint and reportedly has been holding discussions with Richard Gallan Books.

CBS said that the discontinued paperback book business will decrease 1981 earnings by 72 cents a share.

According to an industry publication, BP Reports, book sales at Fawcett increased by only 1.7 percent to about $61 million from 1978 through 1980.

CBS does not disclose performance figures of its publishing companies and would not confirm those statistics. Random House is a privately owned company, and an executive there declined to discuss figures.

The assets that Random House is buying are Fawcett's six imprints, including Crest, Gold Medal and Premier, which will become part of Random House's Ballantine Books. Fawcett's list of authors includes James A. Michener, Alistair MacLean, Helen MacInnes, John Updike, John D. MacDonald and Victoria Holt.

Paula Gottschalk, vice president for corporate information at CBS, said "there are thousands of contracts involved, some assignable, some not. Those authors who don't chose to go, CBS will have to honor their contracts."

Fawcett has 177 employes who are not included in the sale. Both CBS and Random House said they will attempt to find places in their organizations or elsewhere for the displaced employes.

In June, CBS settled a 3-year-old antitrust suit filed by the Justice Department by agreeing to sell Popular Library in exchange for being able to keep Fawcett.

Industry officials wondered whether Random House will have antitrust problems of its own by acquiring Fawcett, reportedly the sixth-largest publisher of paperbacks.

Random House President Robert L. Bernstein said his company's attorneys do not anticipate any antitrust problems. "We think the industry has substantially changed," said Bernstein. "The business is much worse."