The Justice Department, responding to a request from a three-man federal appeals court in New York, said yesterday it was "willing to discuss a possible settlement" of the decade-old antitrust case against the International Business Machines Corp.

Justice said it has been studying specific proposals for relief to those who may have been affected by IBM's alleged antitrust violations, and that those recommendations could "provide a basis for discussion of a possible consent decree in the IBM case."

U.S. Court of Appeals Judge William Mulligan in New York yesterday asked IBM attorney Thomas Barr to seek a meeting with Attorney General Benjamin Civiletti to discuss a possible settlement of the case. The litigation has been in the trial phase since May 1975, and Mulligan called it "a Frankenstein monster."

Mulligan's suggestion came during oral arguments on IBM's motion for a stay of the trial while the appellate court considered another IBM motion calling on Judge David Edelstein to remove himself from the case because of his alleged "bias and prejudice."

The court of appeals turned down the motion for a stay, but agreed to hear arguments Oct. 16 on the IBM petition to remove Edelstein. Later in the day, Edelstein reconvened the trial, which had been delayed for several weeks while first Edelstein, then the appeals court considered IBM's motions.

Government trial attorney Robert Staal responded to Judge Mulligan's suggestion that the two sides attempt to settle by saying the trial was probably only about half over, and that "This is widely acknowledged to be one of the most important antitrust cases in legal history."

But at least one member of the three-man appellate court, Judge Ellsworth Van Graafeiland, was not impressed, claiming "that argument leaves me cold -- after four and a half years."

The government filed suit during the closing days of the Johnson administration, charging IBM with monopolizing the general purpose computer market and calling for the computer giant to be broken up. For its part , IBM has argued that the exponential growth of the computer industry is proof that it did not unfairly dominate the business.

Judge Mulligan told counsel for both sides during yesterday's hearing, "I wish you would spend some time -- both of you -- talking with the attorney general. The possibility that this case may last another 4 1/2 years is frightening."

IBM counsel Barr said the trial should have taken no more than a year. IBM has long charged the government with delaying the case by changing attorneys on the case several times, each time slowing down progress and forcing reevaluation of tactics.

It was Judge Edelstein's ruling on June 29, ordering IBM to comply with a government subpoena, that finally provoked IBM's call for Edelstein's removal. IBM said it would require 62,000 man years to retrieve some 5 billion documents needed to comply with the subpoena.