The already ugly takeover battle between American Express Co. and McGraw-Hill Inc. turned uglier here today.

American Express, which wants to buy the big publishing company, filed suit charging that in their attempt to block a merger, the company and its chairman Harold W. McGraw Jr. have, among other things, made false statements, broken securities laws and libeled American Express.

McGraw responded in a statement that the publishing company has complied with all securities laws and called the American Express suit a "diversionary and spurious tactic designed to distract attention from the real issues," which he said are "protecting the independence and integrity of this publishing company."

Last week, in a surprise announcement, American Express said it would pay $34 share, or about $830 million at told, for the 24.5 million shares of the information conglomerate that publishes Business Week magazine and runs Standard & Poor's financial rating service, among other things.

American Express had hoped the merger would be a friendly one. The $34 offer was substantially larger than the $26 level McGraw stock closed at the day before the takeover proposal was announced, on Jan. 9.

But McGraw quickly dashed any hopes for a friendly merger and angrily attacked American Express and some of its officers, including President Roger H. Morley, who served on McGraw-Hill's board until he resigned late last week.

Monday, McGraw-Hill's board unanimously voted to fight the takeover attempt.

In a letter to American Express' directors informing them of the publishing company's decision to fight, McGraw said the travel card and banking concern lacks the "integrity, corporate morality, and sensitivity to professional responsibility essential to the McGraw-Hill publishing, broadcasting and credit rating services relied upon by so many people."

Today, in a suit filed in U.S. District Court for the southern district of New York, American Express said McGraw's allegations were false, that American Express' good name had been damaged, and that McGraw and the company have been making statements designed to convince McGraw-Hill shareholders to reject the American Express proposal without making proper filings with the Securities and Exchange Commission.

American Express asked for compensatory damages and also asked the court to require McGraw-Hill to correct its previous statements.

But despite the ugly turn of the battle, American Express remains intent on acquiring McGraw-Hill, officials said. The company, which faces increased competition in both its traveler's check and credit card operations, wants to broaden its domestic base.

American Express has been rebuffed in at least two earlier attempts at friendly takeovers: at Walt Disney Productions and Book-of-the-Month-Club.

Analysts expect that American Express that might wear down some of the resistance at McGraw-Hill.

Chairman McGraw himself, however, has taken the American Express offer as something of a personal affront. He claims that when he turned down overtures from American Express chairman James D. Robinson III last June, Robinson assured him American Express would make no move to acquire McGraw-Hill. American Express officials said Robinson gave McGraw no such assurances.