The war of words over the decision by the Reagan administration to drop its antitrust suit against International Business Machines Corp. intensified today as the Justice Department and IBM told a federal judge that he had no business continuing to oversee the case.

In addition, an attorney for IBM charged U.S. District Court Judge David N. Edelstein, who had heard the case, with a historic bias against the company and said continuing proceedings against the company are costing millions of dollars.

IBM's lead counsel, Thomas D. Barr, appearing during a three-hour hearing before Edelstein, said that the session itself--which was called to hear arguments on whether Edelstein has the power to evaluate the dropping of the case--was bringing "shame to the judicial process."

"Your honor has no role to play," Barr said. "Your honor has no function, no purpose."

Although the assistant attorney general for antitrust, William F. Baxter, dropped the 13-year-old litigation more than four months ago, Edelstein called the hearing to consider whether he has the authority to oversee a lengthy proceeding on the merits of the government action under the so-called Tunney Act. Edelstein has criticized both Baxter and the dismissal decision.

Enactment of the statute was prompted by secretly negotiated or politically motivated settlements of antitrust suits from 1956 through the early 1970s. It grants federal judges the authority to evaluate judicial consent judgments. The legal debate today focused on whether that statute applies to dismissals such as the Jan. 8 action taken to close the IBM suit.

Edelstein opened the acrimonious session by saying that his only concern in taking these actions is protection of the "integrity of the judicial process" and "the public interest." Despite "ad hominum attacks, I will not be deterred," he said.

Edelstein went on to deny a motion filed by IBM late yesterday that asked the judge to remove himself from the case. In court papers, Barr accused Edelstein of having "personal bias and prejudice against IBM" and of being "engaged in a campaign to smear" Baxter. Two years ago, the company had brought a similar motion concerning Edelstein to a federal appeals court where the request was denied.

IBM said that Edelstein's refusal to lift an order requiring them to keep billions of pages of documents is costing IBM $100,000 a week and that the government's costs are similar.

The hearing was called by Edelstein after he received friend-of-the-court briefs from judicial activist Philip M. Stern and from a legal group associated with Ralph Nader, Public Citizen Litigation Group. Another hearing has been set for June 21 to consider allegations raised by Edelstein and IBM competitors that Baxter has historic ties to the giant computer firm.

On the other hand, Paul McGrath, the assistant attorney general in charge of the civil division, who is handling the matter since Baxter removed himself, and IBM's Barr argued that the Tunney Act's oversight mechanisms were built around a court's monitoring of consent agreements, not around cases that are closed.

"The court, I respectfully urge your honor, is not the people," McGrath said. "It is this court's function only to apply the law. There is nothing that authorizes the court to proceed."

Edelstein did not say if or when he would issue a ruling on the issues raised at the hearing.