Madison National Bank will inaugurate this week the Washington area's first system for home banking by computer.

About a hundred customers have signed up for the service, which offers the use of a Commodore 64 personal computer and related equipment in exchange for maintaining a $1,000 minimum balance, non-interest-bearing checking account plus a $15 monthly fee.

Besides allowing customers to pay selected bills, transfer funds and obtain account information, "Home Teller" will soon incorporate another major feature: electronic brokerage service. Madison, following the example of a growing number of banks, plans to open a discount brokerage on Nov. 1 in its Georgetown office. Customers participating in the "Home Teller" program will have the ability to transmit, buy and sell orders to the brokers for immediate execution.

Three years in the planning, Madison's system links consumers directly to the bank's mainframe, thus avoiding charges by intermediaries. Unlike other systems in operation elsewhere, it will concentrate exclusively on financial functions. However, Glenn Schickler, vice president for marketing, said the bank had received feelers from outside companies proposing to add services like shopping and travel. Customers will, of course, be able to buy and use software for other pursuits such as games, taxes and stock market quotations.

Madison has the capacity to increase the number of customers using the service to 2,500 over 18 months. For people who already own a Commodore 64 personal computer, the monthly charge is reduced to $10, but the minimum balance requirement still remains. In time, Schickler said, it is possible the service may be offered for a fee only in conjunction with interest-bearing accounts as well. Schickler added that Madison is also working with Atari to make the two systems compatible.

At the outset, customers can pay up to seven bills per month electronically from among a pre-approved group; the number will be doubled later. Nearly 100 department stores, credit card companies, banks, mortgage lenders and other businesses have indicated their willingness to participate, the bank said.

Schickler said that the bank would update the equipment as necessary, but it is legally prohibited from selling used computers to customers.

Madison has hired Federated Securities of Pittsburgh to help it develop a discount brokerage operation. It will be managed and staffed locally. Limited at first to bank customers, the service may later be offered to non-customers.

Those with home computers can place an order electronically after the transaction is authorized. The customer need not have the purchase amount in his or her checking account at the time. Transfers can be made electronically from other accounts and margin accounts are accepted.

A half dozen or more financial institutions in the Washington area have opened discount brokerages in the past year in what has become an accelerating trend. After the Bank of America pioneered by acquiring Charles Schwab & Co., the Federal Reserve last month authorized all bank holding companies to offer discount brokerage services. More than 600 banks nationwide have subsequently begun offering the service.

A recent survey by the Bank Administration Institute and Arthur Andersen & Co., the accounting firm, predicted that three-quarters of all banks will be offering some type of brokerage services by the end of the decade. The most popular arrangement will be through a franchise with an existing broker or another bank, but a sizable number will get into the business by acquiring existing firms, the survey said. The remainder are expected to establish operations from scratch, as Madison is doing. By 1990, banks are expected to capture 21 percent of the brokerage business.