The Securities and Exchange Commission yesterday proposed to limit the amount of its customers' money a brokerage firm can lend to an investor to buy securities on margin. The limit was adopted to protect customers' accounts when firms fail.

A single individual could not engage in margin transactions exceeding one-fourth of a brokerage firm's net liquid assets unless an exemption is granted by the New York Stock Exchange or the National Association of Securities Dealers. Examiners from those two markets must assess the quality of the collateral for the margin account before allowing additional activity.

The SEC proposed to exclude from customer borrowings for reserve requirement calculations the balances of relatives of the firm's principals. The net effect would be to require an increase in the amount in the reserve accounts when funds are loaned to relatives.

The action was triggered by several spectacular broker-dealer collapses. In 1983, Bell & Beckwith, a Toledo, Ohio, broker, failed when inadequate security was discovered in margin accounts handled by the firm's managing partner. The accounts, which were in his wife's name, were supposedly backed by Japanese securities worth $488 million. In reality, they were worth only $5,000. Debts to the firm's customers totaled $21 million more than its assets.

Earlier this year, Volume Investors in New York folded when it was unable to meet margin calls. The default was caused by the failure of three customers to come up with $26 million to cover losses on gold options. The scandal tied up $13 million in customer accounts. In July, the Commodity Futures Trading Commission, which has jurisdiction, proposed to increase capital requirements for such firms and to set margin guidelines for certain risky options transactions.

The SEC first suggested limits on borrowing from customer accounts in February 1984, and again in March 1985, but the industry rejected them as too severe. With the modifications announced yesterday, the proposal has won the support of NASD, the self regulatory organization for over-the-counter trading. The Securities Industry Association gave its lukewarm endorsement. However, the New York Stock Exchange responded that it would be very reluctant to make exemptions unless uniform industry standards were instituted.

The Big Board also objected that the costs of such a program seemed to outweigh the benefits, especially for smaller and medium sized firms. The SEC estimated the cost per program at $30,000 and said the initial costs for the 350 to 400 firms that do margin lending would be about $10 million.