Faced with one of the most complex decisions in antitrust history, a federal judge today ordered a two-month delay in the formal close of the government's antitrust suit against International Business Machines Corp. in hopes that the two sides would move to settle or narrow the case.

The decision by U.S. District Judge David Edelstein, made after an unusual personal plea for additional time from Assistant Attorney General William Baxter, came as an IBM lawyer, who joined in the request for delay, said the next few months represent the final opportunity to reach a settlement of the 12-year-old suit.

Thomas Barr, the IBM attorney, said this "is the last clear chance to dispose of any part of the case other than litigation."

The decision means that the government will have until Dec. 1 to file their final charges, or findings of fact, in the case, with response and other arguments running through the middle of March 1982. If no settlement has been reached at that point, it appears Edelstein will then be in a position to begin writing a decision.

The case was brought in Jan. 1969 during the closing days of the Johnson administration. The government alleges that IBM pricing and marketing practices resulted in the monopolization of the general computer business. Trial closed last year after more than five years of court presentation.

Baxter's appearance during the half-hour session here followed a request for the meeting by Edelstein, who has been encouraging the two sides to either narrow or settle the case. Baxter said he had just begun the "monumental effort of getting on top of the case."

The antitrust chief said he would review the case with the trial staff and read any IBM summary documents the company's lawyers submitted. "Until I do that, I cannot responsibly make any decision with respect to the case," Baxter said, noting that he had been spending much of his time on matters involving the government's antitrust suit against American Telephone & Telegraph Co. But Baxter said he "still entertains" hopes that the case could be either settled or narrowed.

Reading from notes of a telephone conference with Baxter, Edelstein said Baxter had called divestiture or breaking up IBM "Draconian," a phrase that Baxter said after the hearing he could have used. Baxter also used the phrase "total cloning" in referring to the split of IBM that the government has been seeking in the suit.

Edelstein said it is "inconceivable to me that some agreement cannot be reached on some points and issues." But Edelstein, responding to editorial criticism that he had not done enough to bring the case to a close or settlement, said it is not his job to "bludgeon, harass or twist arms" in encouraging settlement talks.