American Express Co. plans to add the giant mutual fund company Investors Diversified Services Inc. to its growing financial services empire, which already includes a huge brokerage firm, a major insurance company and a large international bank.

American Express, long known for its travel cards and travelers checks, announced that it agreed in principle to acquire all the assets and liabilities of Alleghany Corp. for American Express stock valued at about $1.05 billion.

Alleghany's major holding is Investors Diversified Services, a Minneapolis-based company that manages 14 mutual funds, sells insurance and annuities and manages pension funds. IDS has about $15 billion in funds under its management.

Sanford Weill, president of American Express, said the acquisition of IDS, which has a sales force of 4,500, will give American Express "the opportunity to broaden the base of our financial services products."

The acquisition of Alleghany's assets still is subject to a formal agreement and approval of the boards of directors of both companies, shareholders of Alleghany as well as regulators.

But Fred M. Kirby, chairman and chief executive of Alleghany, has agreed to vote his stock in Alleghany for the sale, provided the Alleghany board backs the definitive agreement. Kirby and his family control about 43 percent of the 8.4 million shares of outstanding Alleghany stock.

Alleghany will remain in existence as an investment company (one that owns stock in other companies) after the transaction and will become the single biggest shareholder in American Express. Alleghany also owns MSL Industries, which makes fabricated steel products.

Weill said IDS and its core of salesmen serve mainly households with yearly incomes between $30,000 and $65,000. Many of those households presumably have American Express cards, but Weill said IDS will give American Express more direct contact with the 35 million families in that market. "All of them have insurance . . . . All of them are looking for savings products . They are much more long-term oriented in their investment philosophy" than more sophisticated investors.

American Express chairman James Robinson III, the architect of American Express's plan to become a widely diversified supplier of financial services, said the company is moving to the "next tier of the marketplace."

Many families who deal with IDS do not deal with brokerage firms.

Weill was chairman of Shearson Loeb Rhoads, then the second biggest securities firm in the country, when American Express acquired it two years ago. Today Shearson/American Express is the fourth biggest Wall Street firm.

Earlier this year American Express, which has owned a foreign bank for years, bought the non-U.S. business of financier Edmond Safra, paying him $500 million for his Luxembourg-based Trade Development Bank Holding S.A.

American Express is one of a number of major companies--including Sears, Roebuck & Co., Prudential Insurance Co. of America and Merrill Lynch & Co.--that is building a financial services empire designed to be able to serve nearly all the needs of its customers.

Besides its traditional travel business, the international bank and the brokerage firm, American Express owns the big Fireman's Fund Insurance companies and a joint cable television venture with Warner Communications.

Last year American Express had revenues of about $8 billion and profits of $581 million. Alleghany, with revenues of $1.34 billion, had 1982 profits of $57.9 million.