Xerox Corp., the multibillion dollar office automation and insurance company, is negotiating to sell its R. R. Bowker Co. publishing subsidiary, according to sources including an executive whose company sought at one point to buy it.

Sources indicate that the likeliest buyer of the publishing group would be International Thomson Organization Ltd., a Toronto-based multinational conglomerate with interests in publishing, travel and oil.

R. R. Bowker Co. is best-known for publishing major reference books and magazines, including Books In Print, the listing of all available book titles, and Publishers Weekly, the trade journal of the book publishing industry. Among its other reference books and magazines are Library Journal, Literary Market Place and American Men & Women of Science.

International Thomson and Xerox executives refused to comment on the possible acquisition.

Analysts said that the sale of Bowker would not come as a surprise since Xerox has trimmed back its non-essential activities.

"The company's future direction is not in the publishing area, so I wouldn't be surprised by this sale," said Eugene G. Glazer an analyst with Dean Witter.

"Xerox is a company known for getting rid of something that doesn't fit into its corporate structure," said Phil Cavalier of Pershing & Co. "And Xerox's balance sheet could use a little strengthening."

An announcement naming the purchaser of the wholly owned subsidiary is expected imminently, said Bowker employes who asked not to be named. The future sale of Bowker was described yesterday by one employe as "very sudden and very hushed."

Xerox acquired the New York-based company, which now has about 300 employes, in 1967. The firm was founded in 1872 by Frederick Leypoldt and Richard Rogers Bowker. Bowker and Leypoldt later joined Melvil Dewey, who invented the Dewey Decimal System, in setting up the American Library Association in 1876.

Bowker also produces data bases, and is planning to publish a software encyclopedia in June. The company already publishes software catalogues for Apple and IBM and a hardware catalogue for CPM.

Cavalier said that about four percent or $400 million of Xerox's total $11.6 billion revenue comes from publishing activities, which include Bowker.

International Thomson is the top holding company in the publishing empire built by the late Lord Thomson of Fleet. The company earned $109 million on revenue of $2.3 billion in 1983.