American Broadcasting Cos. is offering to buy Macmillan Inc. for securities valued at about $337 million, the publishing firm said today, apparently scuttling a Mattel Inc. bid Macmillan had tentatively accepted Aug. 30.

Macmillan said ABC proposed merger discussions and it had agreed to talk.ABC still is studying Macmillan as a takeover prospect but meanwhile has offered securities valued at $25 each for all Macmillan shares.

Mattel, the Hawthrone, Calif., toymaker that has diversified into publishing, had offered cash and securities valued at $24.50 a share for each Macmillan share in a bid worth about $329 million.

Macmillan and Mattel said two weeks ago that they had agreed in principle to merger terms but that the deal still was subject to final approval by directors, shareholders and regulatory authorities.

In a statement after Macmillan disclosed it was talking with ABC, Mattel said, "We regret that Macmillan's board has reached the announced decision and that the anticipated association between Macmillan and Mattel will not take place." Mattel refused to comment on what caused the breakdown in its merger talks.

Mattel, which also owns Ringling Bros., Barnum & Bailey Circus, purchased Western Publishing Co. in June. It said today it still believed Macmillan's "management and prospects are compatible with our own."

ABC'S effort to expand its publishing group, which last year contributed about 4 percent of total pre tax operating earnings to ABC, comes as the parent of a rival broadcasting network is seeking to get out of the publishing field.

RCA Corp., parent of NBC, said last week it is putting its Random House subsidiary up for sale because the unit doesn't fit into RCA'S growth plans. No bids for Random House have been announced.

ABC offered to issue a preferred stock for 55 percent of Macmillan's shares and subordinated debentures for the remaining shares. Both ABC issues would carry a value of $25 for each security.

Macmillan's subsidiaries include the Berlitz language schools, Katharine Gibbs Schools, the Brentano's bookselling chain, and Gump's, the San Francisco specialty store.

Meanwhile, Colgate-Palmolive Co. announced today that it was negotiating to sell its loss-ridden Helene Rubinstein cosmetics subsidiary to L'Oreal S. A., a giant French beauty products concern. No details were given.