Bankers Trust New York Corp., the nation's eighth largest banking company, revealed last night that "active discussions" are in progress which could lead the sale of most of its retail banking operations in metropolitan New York to a Canadian bank.

Alfred Brittain III, the chairman of Bankers Trust Co. and its parent holding company, said Bank of Montreal is expected to acquire 89 of the 104 branches in the New York area as well as Bankers Trust's installment loans.

In addition, the Canadian bank has expressed an interest in the credit card business of Bankers Trust. "But these discussions are at a preliminary stage and it is too early to have a judgement about the outcome," according to a Bankers Trust statement.

With assets of $26 billion, Bank of Montreal is a major retail-oriented consumer banking firm with 1,200 branches across Canada. In terms of deposits, it is the 34th largest bank in the world.

Although the Canadian institution has had a representative office in New York for 120 years, the contemplated agreement would mark a major expansion in the U.S. market and make Bank of Montreal one of the largest foreign-owned bank businesses in this country.

Terms of the proposed sale, which would require regulatory approval, were not disclosed in last night's surprise announcement. But Bankers Trust said assets to be purchased by Bank of Montreal total about $1 billion and account for 4 percent of the $25.4 billion in assets of Bankers Trust New York Corp.

The credit card business being discussed would have assets of $400 million and involves the New York firm's Visa card operation.

Not included in the transaction would be commercial accounts serviced by corporate banking divisions of Bankers Trust, as the New York company moves to concentrate its resources on corporate and international banking, which have been accounting for a growing share of overall revenues and profits.

Last year, for example, the international share of revenues increased to 49 percent in 1976 while international profits accounted for 79 percent compared with 58 percent of overall earnings in 1976.

Thomas Parisi, a Bankers Trust spokesman, said the remaining 15 branches will be used to service commercial accounts.

Asked about the profitability of the branches being sold, Parisi said, "Most aspects have been profitable in recent years." Later, though, he added, "The profitability of consumer banking has been under pressure."

Parisi estimated the deal would take another six months to work out final details and get the necessary regulatory approval.