With only a handful of companies making up the mass market paperback book industry, the agents, editors and executives are at once extremely competitive, gossipy and very closely knit--in short, just like your average family.

So when CBS Inc. announced last month it was selling Fawcett Books to Random House, the family members came together to mourn the passing over three-martini lunches and long, gloomy phone conversations. CBS also made it known that it was looking to sell its other famous paperback line, Popular Library.

One characteristic of the industry is that its members are tight-lipped about profits and sales. No matter that executives regularly jump from one publishing house to another, discretion is maintained. And because most paperback houses either are privately held or owned by large conglomerates, their performance figures don't have to be disclosed.

But industry statistics do show that back in 1976 when CBS acquired Fawcett, the publisher ranked No. 3 in the industry in sales, behind Bantam and Dell and ahead of Pocketbooks, Ballantine and Avon. Now, according to industry statistics, the ranking is: Bantam, Pocketbook, Harlequin, New American Library, Dell and Fawcett.

The news that CBS was cutting Fawcett loose caused concern in the industry over the plight of the nearly 200 Fawcett employes, whose jobs were not guaranteed either by CBS or by Random House. According to the terse press release, Random House was acquiring "substantially all" of Fawcett's assets--the six book lines, the contracts of its famous authors, and the backlisted titles--but not the personnel.

"What terrifies book agents and just about everyone else is the shrinking of the business," said a former Fawcett executive who requested anonymity. "To have a company as outstanding as Fawcett essentially disappear is very shocking to everyone."

"I think it's a bellwether, a portent of things to come," remarked Knox Burger, an agent and former Fawcett editor.

Many on the editorial side of publishing blamed Fawcett's woes on CBS managers educated at business schools like Harvard. "Fawcett and Popular were ruined by Harvard MBAs," said Patrick O'Connor, editorial director of Pinnacle Books and a Fawcett alumnus. "Book publishing is a unique business, it can't be done with case studies."

Jack Romanos, vice president and publisher of Bantam Books who worked in marketing and sales at Fawcett from 1963 to 1973, said, "The classic marketing techniques don't apply to the business. In the years CBS owned Fawcett the business changed, but CBS didn't realize it."

The CBS announcement quoted CBS/Publishing Group President Peter A. Derow, who moved to CBS in 1981 from Newsweek, where he was chairman and president of the Washington Post subsidiary, as saying, "Mass market paperback book publishing does not fit into our long-range plan."

Derow declined to explain what had gone wrong since CBS paid $50 million for Fawcett Publications Inc., including its line of magazines such as Women's Day. (Fawcett Publications, founded as a magazine publishing firm by Wilford H. Fawcett in 1919, remained in his family for 58 years before its sale to CBS.) Publishers Weekly quoted industry sources as estimating that Random House agreed to pay $10 million for Fawcett Books and to assume $20 million in debt.

But while CBS demurred from post mortems, there was no lack of explanations for the failure by others in the industry. They blamed an oppressive management structure and overpayments for books that turned out to be flops, and said CBS failed to anticipate trends.

For these and other reasons, Fawcett under CBS was particularly hard hit by the malaise that has affected most pocketbook publishers in recent years, critics said. As paperback prices have climbed, buyers have been more selective. Paperback books are passed along to other readers the way hardbacks are. Book unit sales--as opposed to dollar sales--have remained flat in the industry since 1974, according to a Bantam executive.

Another paperback executive noted that sales in supermarkets have stagnated because "everything else there costs so much."

In a dull market, Fawcett became a loss leader. An industry newsletter, BP Reports, said Fawcett's annual sales have remained flat since 1978 at about $60 million. CBS, like others, bought into the paperback business during the boom times of the mid-'70s when profit margins were high. But recently, when those margins began to decrease and the books for which they paid huge advances failed to sell, CBS grew disenchanted, according to a number of trade industry executives.

One former Fawcett executive confirms this but also says CBS brought a lot of executive baggage with it. "Yes, there were losses, but some of it had to do with overhead that comes with a large corporation like CBS, where many people don't contribute to the bottom line, just comment on it."

One reason CBS is blamed for Fawcett's troubles is that Fawcett has such a strong stable of best-selling authors under contract, such as John D. McDonald, James Michener, Isaac Bashevis Singer, Victoria Holt, Helen McInnes and Isaac Asimov.

"With Mary Stewart and Michener, they had Middle America," says O'Connor of Pinnacle.

In recent years, all paperback publishers have overpaid for titles from time to time, but Fawcett did so more often than others, industry executives say. Recent works that didn't sell well enough to justify their price include Linda Goodman's "Love Signs" for $2.25 million, William Kinsolving's "Born With the Century" for $1 million, and "Rage Against Heaven" by Fred Mustard Steward for a reported $750,000.

Among the trends that Fawcett muffed was the rise of the romantic novel, a phenomenon that has made Harlequin Books of Toronto into an overnight success by producing $1.75 books "like bars of soap," in the words of one executive.

In 1979, when Fawcett weighed in with its Coventry line of heart-throbbing tales of earlier centuries, they were no challenge to the contemporary romantics being pumped out by Harlequin and Simon & Schuster's Silhouette books.

Another trend missed by Fawcett but exploited by the aggressive and succesfull Bantam Books was the rise in sales to the young reader as readership by their elders dropped off. Bantam, which in 1981 enjoyed its second most profitable year in the face of industry problems, attributes part of its success to serving the growing numbers of child readers.

Only one hurdle stands in the way of the sale by CBS of Fawcett--the Justice Department. A year ago, CBS settled an antitrust suit and was able to keep Fawcett by agreeing to sell Popular Library.

Now CBS wants to sell both, and attorneys for Random House and CBS have asked the government for a quick reading on whether the sale presents antitrust problems.

Random House, which is owned by Newhouse Newspapers, publishes Ballantine Books. CBS and Random House in their presentation to the Justice Department probably have argued that both Ballantine and Fawcett have suffered from the consequences of high advances for paperbacks and deteriorating retail sales. Ballantine and Fawcett together stand a better chance of surviving than either would alone, the reasoning goes.