The first rupture in McGraw-Hill Inc.'s united stand against a takeover bid by American Express Co. surfaced today.

Donald C. McGraw Jr., a member of McGraw-Hill's board and a major shareholder, cabled his cousin, chair man Harold L. McGraw Jr., telling him to "change" his opposition to the American Express proposal.

Donald McGraw and the rest of the McGraw-Hill board last week unanimously rejected a $34-a-share bid by American Express for the 24.5 million outstanding shares of the large company that, among other things, publishes Business Week magazine, operates Standard & Poor's financial rating service, and runs four television stations.

Wall Street analysts have said that American Express can be expected to sweeten its $34 offer, which is worth about $880 million. In his telegram, Donald McGraw urged his cousin to seek such a sweeter deal.

Donald McGraw advised Harold McGraw to stop fighting the proposed takeover and instead "negotiate with the top management of American Express and such other bidders as may offer to acquire the stock of McGraw-Hill at a higher price."

On Jan. 8, the day before American Express made its proposal public McGraw-Hill stock closed at $26 a share.

Donald McGraw owns or controls 622,000 shares of McGraw-Hill stock. In his telegram he citied his "fiduciary responsibility" as president of the Donald C. McGraw Foundation Inc. and as a trustee for other trusts that control company shares.

Harold McGraw controls about 804,000 shares.

Chairman McGraw has led the company in a complicated and acrimonious fight against the takeover attempt, one that has been waged in the courts, government agencies and the media.

The Federal Trade Commission confirmed today that its staff has begun preliminary inquiry into the proposed takeover. But FTC sources said such inquiries are routine in large takeover attempts and does not necessarily mean that the agency will try to block the takeover.

McGraw-Hill had sales of $659 million in 1977. Amercian Express had revenues of $3.4 billion from its travelers check, credit card, insurance and international banking operations.

McGraw-Hill had no comment on Donald McGraw's telegram to his cousin beyond stating that it "will be brought to the attention of the McGraw-Hill board at its next meeting." The company said the next regularly scheduled meeting is Jan. 31.

A spokesman for American Express said that the company is "glad that a major stockholder in McGraw-Hill has seen fit to consider our proposal" and said that "we believe that all stockholders in McGraw-Hill deserve the option of considering our proposal."

Although Donald McGraw voted with the company's board last week to reject a takeover, he was the only major family shareholdr who did not sign the report with the Securities and Exchange Commission last week supporting the company's opposition to the American Express proposal.

McGraw-Hill has said that the fundamental issue in the takeover attempt is the independence of the publishing company and has argued that American Express and some of its officials -- including chairman James D. Rbinson III and Roger H. Morley -- are morally unfit. Morley had been a member of McGraw-Hill's board during the time American Express was considering that attempt to acquire the publisher.

American Express, in a filing with the SEC, acknowledged that Morley was privy to some confidential Mcgraw-Hill data, but said the decision to try to acquire McGraw-Hill was in no way made on the basis of that information.

American Express has filed suit against McGraw-Hill charging, among other things, that the company and chairman McGraw have libeled American Express.

McGraw-Hill, for its part, has asked both federal and state courts and the Federal Communications Commission to block the takeover attempt.