Reston Publishing Co., a Prentice-Hall Inc. subsidiary that made a big splash in the personal computer book market, will lay off nearly half its work force Jan. 2.

The Reston company appears to be a victim both of a decline in the demand for computer books and internal reorganizations at Prentice-Hall, which is about to be acquired by Gulf & Western Industries Inc.

Prentice-Hall spokesman Robert Shaw said the layoffs were the result of cost-cutting at the Reston subsidiary and a reduction in some of the subsidiary's development and publishing activities. Shaw declined to say which of Reston's activities would be "reassigned" to other Prentice-Hall divisions.

Shaw said about 60 of Reston's 130 employes would be laid off. The layoffs will affect "clerical- to management"-level workers, he said.

A year ago, the 12-year-old publisher was riding high on the sale of books targeted toward personal computer users. Its profits had increased steadily, and last year's sales were estimated at $30 million. The parent company's total sales were $450 million.

This year, however, the computer book boom fizzled. Books published by Reston were returned at an unexpectedly high rate.

Early this year, two of Reston's key executives -- Larry Benincasa and Nikki Hardin -- left to join a California-based software publisher, Ashton-Tate. Both said they were lured by significant stock options that gave them a stake in Ashton-Tate's success as well as the chance to do something new.

Several weeks ago, the subsidiary's president, David Ungerer, resigned, and Lloyd Rich, formerly director of development and corporate planning for the parent company, took over as Reston's acting president.

In addition to computer books, Reston publishes books about personal and business finance, consumer topics and agriculture.