American Express Co., the giant financial services company, and the brokerage firm Shearson Loeb Rhoades Inc. announced yesterday that they were discussing a possible combination of the two firms.

The companies began talks of a joint venture -- reportedly one that would try to combine the American Express entertainment card and a money market mutual fund run by Shearson. However, the companies said, those talks were expanded into a discussion of whether the two companies might merge totally.

Both Shearson and American Express said they would announce the results of those talks before stock trading begins at 10 a.m. today.

There was speculation in Wall Street last week that American Express might be willing to pay as much as $1 billion to buy a presence in the brokerage industry. That would be about $80 a share for Shearson stock, which closed at $42.125 a share last Thursday. Officials at neither company would comment on the talks or the price that might be involved were a merger to take place.

Neither American Express nor Shearson stock traded Monday. American Express closed at $43.125 last Thursday, the last trading day before yesterday.

The revelation that the nation's second-largest brokerage house and the giant international financial services company were discussing a marriage follows by days a successful merger between the Bache Group Inc., the eighth-largest broker, and Prudential Insurance Co., the nation's largest insurer.

As the brokerage industry begins to explore new types of financial products to sell other than the traditional stocks, bonds and commodities, and as other financial services companies -- such as insurers -- begin to explore new ways to market their products, talks such as those between Shearson and American Express will become more common. Other large brokerage firms have reportedly talked to American Express.

Perrin Long, of Lipper Analytical Services, said that some sort of joint venture between Shearson and American Express makes sense.

Shearson, with nearly 4,000 brokers scattered around the country, provides a ready sales force for the insurance products marketed by American Express' Firemen's Fund Insurance Co. Firemen's Fund sells through insurance brokers as well as through mailings to American Express card holders, Long said.

Long said that Shearson -- as well as several other big brokerage houses like Dean Witter Reynolds, Paine Webber Inc. and E. F. Hutton & Co. -- are working on special money market mutual funds similar to the cash management accounts run by the nation's biggest broker, Merrill Lynch & Co.

American Express last year had revenues of $5.5 billion. Shearson, with capital of $246 million, is the second-largest broker, although only about a third the size of Merrill Lynch, according to the Securities Industry Association's 1980 handbook.

Prudential said yesterday that 70 percent of Bache shareholders had agreed to sell their stock at $32 a share. The Prudential offer for Bache expired at midnight Friday. Since Prudential controls that much Bache stock, a shareholder vote to merge with Prudential is ensured.

The Prudential offer effectively squelched an attempt by three Canadian brothers -- Samuel, Hyman and William Belzberg -- to acquire control of the brokerage firm, an attempt that was fought by Bache management