The chairman of the House Judiciary Committee yesterday accused the Justice Department of withholding documents to frustrate his panel's probe of alleged improprieties in the department's dealings with Inslaw Inc., a Washington-based computer software company.

The committee is considering whether to subpoena the documents or to attempt in some other way to force the department to produce the documents.

The case involves Inslaw, which wrote a computer program that allows the Justice Department to keep track of a large number of court cases. Inslaw and its top executive, William A. Hamilton, have accused the department of conspiring to drive Inslaw out of business so that friends of high-ranking Reagan administration officials could get control of the program and market it profitably. Hamilton's testimony yesterday reasserted those claims.

A federal bankruptcy judge concluded that the Justice Department "stole" Inslaw's proprietary software and did, in fact, try to drive the firm out of business. Those findings were upheld on appeal by a U.S. district judge, but the legal battle continues.

The House committee's complaints mirror those of others who have sought to get to the bottom of the Inslaw affair. An earlier Senate probe found the Justice Department uncooperative, and the judge in the bankruptcy case, George Francis Bason Jr., said numerous department witnesses seemed to suffer from amnesia.

"The department has thwarted attempts by Congress to learn the complete truth concerning the Inslaw case," said House Judiciary Committee Chairman Jack Brooks (D-Tex.). "Justice has repeatedly denied both the House and Senate investigating committees access to critical documents that may prove the department's innocence or guilt."

At issue are internal memorandums and other documents used by the Justice Department in the bankruptcy case and in defending against a lawsuit filed by Inslaw.

Attorney General Richard L. Thornburgh wrote Brooks, refusing to release "litigation strategy" files relating to Inslaw. According to Brooks, Thornburgh argued that these are "sensitive documents related to the ongoing litigation" and thus are "privileged" and "shielded" from congressional inquiry.

This assertion of privilege was challenged by Steven R. Ross, general counsel to the clerk of the House. Citing legal precedents from the Teapot Dome, Watergate and Iran-Contra scandals, Ross told the panel that it has the legal authority to force the Justice Department to produce the papers, and that the only question for the panel was how it wanted to go about it.

Ross noted that the panel could issue a subpoena or use its contempt powers. He also noted that as the authorizing committee for the department, it could threaten the department's funding, a tactic Brooks said had been effective in the past.