American Express Co. substantially increased its offer for McGraw Hill Inc. from $34 to $40 a share, but at the same time told the publishing company's directors that if they persist in fighting the takeover attempt, American Express will withdraw.

Today's development was the latest, and perhaps most masterful, tactiv in the bitter three-week merger battle between the credit card giant and the big publisher.

American Express threw the ball back to McGraw-Hill, in effect daring it to face shareholders suits if it neither recommends acceptance of the proposals or at least agrees "not to oppose it by propaganda, lobbying, litigation or otherwise."

At the same time, American Express gave itself a graceful exit from the takeover bid.

McGraw-Hill stock closed at $26 a share on Jan. 8, the day before American Express made pulbic its takeover offer of $34 a share. It closed last Friday at $31.75, and did not trade today because of the offer.

Yesterday American Express Chairman James D. Robinson III formally withdrew the offer and said American Express would make a new one, of $40 a share, if McGraw-Hill directors agreed to stop fighting.

That would make the total deal worth more than $1 billion, one of the largest cash takeovers in history.

But in a letter to each of MeGraw-Hill's directors, Robinson said "no offer will be made without your board's recommendations of, or agreement notto oppose, such an offer."

Robinson admonished that "you should be aware that if your board fails to act as indicated, your stockholders will be precluded from having an opportunity to decide on an American Express Co. offer."

Harold L. McGraw Jr., chairman of the publishing company that produces Business Week magazine among other things, has fought a takeover attempt in the cours, government agencies and the press with the unaminous backing of the board of directors.

But one major shareholder, McGraw's cousin, Donald C. McGraw Jr., has said that the company should put the American Express offer to its share-holders, Donald McGraw holds about 622,000 of the 24.5 million shares of McGraw-Hill stock outstanding. Other major shareholders apparently are pressuring the board to permit American Express to make the offer.

McGraw-Hill said that the board will consider American Express' latest offer at its next meeting, scheduled for Wednesday. But the company insisted that the directors already have set froth their position "as to the legality and appropriateness of any combination of McGraw-Hill and American Express."

Harold McGraw and the board have said that the independence and integrity of the company's publications and of its financial rating servicie, Standard & Poor's, would be compromised if the American Express takeover succeeds.

In his letter today, Robinson assured the directors that McGraw-Hill would be operated as an "independent subsidiary" and that American Express, in advance, is willing "to work out with you suitable arrangements to insure" the independence of the magazines and Standard & Poor's.

He cited the example of the Economist, a London magazine owned by a large conglomerate but operated by a special set of trustees.

American Express officials say they had hoped the takeover would be a friendly one, but Harold McGraw and his fellow board members dashed that hope. American Express finally filed last week in federal court, charging that McGraw-Hill, in its fight to prevent a takeover offer from being made, had libeled the company. McGraw-Hill, for its part, has asked federal and state courts and the FCC to block the takeover attempt.