American Express began to lay the legal groundwork yesterday to buy out McGraw-Hill, as one of the most dramatic and bitterly contested recent takeover attempts moved from Manhattan's corporate suites to state courtrooms and hearing rooms in Washington's regulatory agencies.

In a filing of papers an inch and a half thick, American Express formally notfied the Securities and Exchange Commisson of its intention to acquire McGraw-Hill's 25.6 million outstanding shares at $34 a share for an estimated $880 million. The credit cardcompany also announced plans to file soon with several other federal and state regulators.

The offering price remains the same as first proposed by American Express a week ago. Noticeably absent this time, though, is American Express' willingness to consider paying a combination of cash and stock for the publishing company in an arrangement that would have provided certain tax advantages for the McGraw family, which controls about 20 percent of the company.

This option was dropped because of the family's resistance to the takeover bid, according to an informed American Express source. But American Express still has reserved the right to modify, amend or revoke its proposal before the final offering is made.

McGraw-Hill's directors unanimously rejected the initial bid Monday and vowed to block the merger attempt every step of the way. Such opposition, together with American Express' determination to take its offer directly to McGraw-Hill's shareholders, threatend to ensure that the fight will be lengthy.

McGraw-Hill publishes Business Week and 60 other magazines. It also owns the Standard & Poor's rating service and four television stations. American Express issues credit cards and travelers checks and also owns the Firemen's Fund insurance company, a travel agency and an international bank.

Either the Justice Department or the Federal Trade Commission must rule on the antitrust implications of a merger. Officials at both agencies said yesterday no decison has been reached on which agency will handle the investigation. The case well could set a number of precedents governing the takeover of communications companies by commercial firms.

In addition, American Express must receive prior approval from the Federal Communications Commission for the transfer of licenses for McGraw-hill's four television station. This process alone normally takes 90 days in uncontested cases, and a contested case can take much longer.

To make sure the FCC will have its say, McGraw-Hill yesterday filed a brief petition with the commission asking for a statement on the policy in this situation. American Express, in turn, said it will ask the commission to speed its required review and grant early approval to the merger which would be subject to later review.

McGraw-Hill also filed suit in New York state court to force American Express to appear there under the state's own antitrust code. A spokesman for the publishing company promised more suits to come.

Mean while, one of McGraw-Hill's shareholders entered the legal fray on the side of American Express. Harry Lewis filed a class action against McGraw-Hill, its chairman and directors for refusing to negotiate with American Express "to secure the highest price available for its stock."

The suit asks the New York state courts to enjoin McGraw-Hill's directors from continuing to oppose the merger and using corporate funds to thwart the deal. it also seeks compensatory damages.