The Justice Department yesterday filed a civil antitrust suit against International Business Machines Corp., forcing it to agree to sell a division of the Rolm Corp. to win approval for its acquisition.

To settle the lawsuit, IBM agreed to divest itself of Rolm's Mil-Spec Computer Division, clearing the last hurdle blocking its purchase of Rolm.

The acquisition of the company, one of the country's leading manufacturers of office-telephone switching equipment, is expected to give IBM a major presence in the $3-billion-a-year PBX (private branch exchange) office-switching market, turning the computer giant into a leading telecommunications competitor.

Assistant Attorney General for Antitrust J. Paul McGrath said the Justice Department was challenging the deal as originally announced in September because "it may substantially lessen competition" in the market for "mil-spec" commercial-based computers, a highly specialized brand of computers manufactured to meet rigorous military specifications, including the ability to withstand harsh environmental conditions.

According to the Justice Department, Rolm sold approximately $75 million worth of such computers last year, accounting for about 50 percent of the total market. The department also noted that IBM had indicated to the U.S. military that it intends to enter this market and begin taking orders in 1985.

Most industry analysts and an IBM spokesman in Armonk, N.Y., said today the forced divestiture of Rolm's Mil-Spec Computer Division would not substantially affect the economics of the deal. IBM spokesman Peter Kuhn said the division accounts for only about 15 percent of Rolm's total business, while the rest of the company is involved in the office-switching equipment that was of most interest to IBM.

"This in no way diminishes the value of the acquisition to IBM," Kuhn said.

The Mil-Spec Division "is of great sentimental value to Rolm and not much else; they IBM could care less," added Howard Anderson, managing director of The Yankee Group, a Boston-based telecommunications consulting firm.

Kuhn said Rolm shareholders are scheduled to meet this morning in Santa Clara to approve the sale to IBM, after which IBM will complete its acquisition of the firm "almost immediately." Rolm will retain its current management, with Rolm's president, M. Kenneth Oshman, reporting to Paul J. Rizzo, vice chairman of the IBM board, Kuhn said. IBM will have six months to sell Mil-Spec.