American Express Co. announced today that it is backing out of one of the largest takeovers in the history of the finance business--its proposed $1 billion purchase of Investors Diversified Services Inc. and IDS' parent, Alleghany Corp.

The unraveling of the deal came after top American Express officials carefully studied IDS' resources with an eye to its near-term profitability and the stability of its 4,500 member sales force. American Express concluded IDS had only limited technological and managerial potential to effectively cross-market American Express financial products, industry sources said.

Publicly, however, American Express officials provided no specific reasons for the decision. In a terse statement, American Express, the giant financial and travel services concern, said it "is not willing to proceed" with the deal as announced five weeks ago, but said it would be willing to continue talks with Alleghany officials "only on the basis of revised terms and conditions."

Financial analysts, several of whom had been openly critical of the deal and had downgraded their ratings of American Express stock, left little hope that the deal could be salvaged. "Realistically, it's a dead deal," said A. Michael Frinquelli, a vice president at Salomon Brothers Inc. The analyst said he had restored a buy rating to the company's stock after lowering his American Express rating from buy to hold recently in the aftermath of the merger announcement.

Another American Express watcher, Michael Lewis, a vice president at The E. F. Hutton Group Inc., said that if the deal does fall through "it's a hidden blessing" for the company. American Express "really doesn't need IDS to maintain a very attractive growth rate," Lewis said.

Alleghany Chairman F. M. Kirby said his company is "surprised and disappointed" at the decision, noting that American Express did "indicate willingness to proceed at a reduced price." There were no public indications from American Express that its problems with the transaction stemmed from the purchase price, although James D. Robinson III, the American Express chairman, called IDS "a fine company" and said the announcement "does not foreclose the possibility of future discussions on a different basis."

Just two weeks ago, American Express officials confidently said that they foresaw no major stumbling blocks that could block the deal and enthusiastically talked of the loyalty of the 1.3 million IDS customers.

Minneapolis-based IDS, a leading direct marketer of mutual funds and insurance primarily in the Midwest, was considered the key to the deal, an acquisition that would quickly move American Express into a large middle market whose investment needs are not generally served by the company's Shearson/American Express brokerage subsidiary.

But observers said that American Express had second thoughts because of the high price American Express was paying for a company that reported operating earnings of but $65 million last year, and because of the fact that American Express officials thought they would have to substantially upgrade the IDS operation. "If the back office or the front office could not handle new products, that would take away from any synergistic benefits," Frinquelli said.

The initial purchase price for the entire Alleghany firm, which includes MSL Industries, a steel products concern, was set at 15.25 million shares of American Express common stock, a package worth more than $1.01 billion at the time of the announcement. Since that time, American Express implemented a 3-for-2 stock split and the purchase was adjusted to 22.875 million shares.

According to Frinquelli, in order for American Express to maintain its projected 1983 and 1984 profits after its takeover of Alleghany, its new holding would have to make a profit of about $90 million on its own, about double the analyst's projections.

American Express stock, after the announcement, reopened trading off 1 1/8 from its previous trading price. By the close of today's trading the stock was off 1/8, closing at 44 7/8. "That's as it should be," Frinquelli said. "This has to be viewed as positive for American Express. It's a tribute to management that they would cut their losses short."