A potentially serious problem has emerged for Donald T. Regan, the Wall Street stockbroker named by President-elect Ronald Reagan last week to be secretary of the treasury.

It is the leading role played by his brokerage house in promoting what the Internal Revenue Service has branded an illegal tax avoidance scheme that costs the government $3 billion to $4 billion a year.

For more than a decade, Regan's firm, Merrill Lynch, Pierce, Fenner and Smith, has helped wealthy investors cut their federal tax bills through an investment known as a "commodity tax straddle."

The IRS has ruled -- and its contending in various court and administrative actions, some involving customers of Merrill Lynch -- that "tax straddles" are illegal and cheat the government out of billions by creating phony business losses that are used to lower taxable income.

If confirmed as treasury secretary, Regan would have perhaps the major voice in setting Reagan administration tax policy and would oversee the IRS, which is part of the Treasury Department.

Merrill Lynch pioneered the use of "tax straddles" to avoid taxes, arranges more of the transactions than any other Wall Street company and has become the central target of the federal effort to halt such tax duductions.

IRS agents now are auditing the returns of thousands of customers of Merrill Lynch and are seeking a court ruling that could force the firm's customers to pay millions in back taxes.

Last month, Merrill Lynch tried to avoid an unfavorable court decision by paying off two of its customers to drop a lawsuit against the IRS. The brokerage firm apparently feared the customers would lose their cases, giving the IRS a precedent-setting victory that would wipe out the lucrative straddle business.

To avoid such a ruling, Merrill Lynch offered to give the customers twice as much as the government claimed they owed it. A tax court judge in New York refused to let the customers drop the case and criticized Merrill Lynch for trying to delay a decision. Further hearings are expected this week.

Three congressmen yesterday urged the Senate Finance Committee to make tax avoidance schemes an issue at Regan's confirmation hearings. The three, Reps. Charles Vanik (D-Ohio), William M. Brodhead (D-Mich.) and Benjamin Rosenthal (D-N.Y.), are sponsors of legislation that would outlaw tax deductions for artificial commodity trading losses.

Merill Lynch spokesmen yesterday refused to respond to questions about the company's tax straddle business. They said Regan would not comment until his confirmation hearings, likely probably sometime next month.

Regan has been president of Merrill Lynch since 1971. During the tenure Merrill Lynch has made millions of dollars in commissions on its straddle business.

For more than 10 years, a Merrill Lynch vice president named Thomas P. O'Hare has headed a special department devoted primarily to helping customers reduce their tax bills. O'Hare's section was known internally as the "Tax Straddle Department" until 1976 when the legality of the tactic was first challenged. The name was then changed to "Financial Service."

Catering only to taxpayers in the 50 percent tax bracket and up, the Merrill Lynch department arranged to cut customers' taxes in return for a fee of about 10 percent of the taxes saved.

The wealthy customers reduced their taxes by making a carefully arranged series of investments in commodity futures.

The most commonly used technique was an esoteric transaction that is known as a "butterfly spread" because a diagram of the deal looks like a butterfly's wings. Because the investments were usually made in the silver futures market, they also are often also called "silver butterflies."

Use of silver futures to create tax losses was considered so tricky that most Merrill Lynch account executives were forbidden by company policy from handling the investments themselves and were ordered to refer all customers directly to O'Hare's office in New York.

In effect, a customer simultaneously agrees to buy and sell a given amount of some commodity at a specified price on a future date. By the end of the year, the price has changed and the customer is faced with losing money on one half his agreement, making money on the other. He takes his loss, and uses it to reduce his taxable income. Then early in the New Year he also takes his profit. He breaks even but has a tax break.

Under Regan, Merrill Lynch has become the most aggressive Wall Street firm in marketing its investment services to the public, but the company has carefully kept a low profile in selling its tax-reducing plans, knowing they were open to challenge.

The "silver butterflies" are never advertised. The company's legal department warned several years ago that "Merrill Lynch policy prohibits the sending of form letters regarding the use of tax straddles."

In a memo that was obtained by The Washington Post, Merrill Lynch attorney Robert E. Aherne explained why: "The most compelling reason . . . for the present Merrill Lynch policy is that many tax straddles to obtain beneficial tax treatment is subject to attack by the Internal Revenue Service. Merrill Lynch, by the use of a form letter, might bring the attention of the IRS to this area. The IRS might then step to end the tax benefits."

Even inside Merrill Lynch, the methods of selling of tax deductions were considered too sensitive to be written down. Instead, Merrill Lynch used a 12-minute tape recording to teach brokers how to spot potential customers.

The tape recording begins with a warning that the tape is never to be removed from the Merrill Lynch office and is never to be played for members of the press. The recorded message is that only customers in the 50 percent tax bracket who want to shelter at least $20,000 from taxes are considered eligible for the program.

The tape was first played in public last month at a U.S. Tax Court hearing in a lawsuit filed against the IRS by two couples who were Merrill Lynch customers. Herbert J. and Ruth D. Jacobson and Harry Lee and Patricia Ann Smith.

The IRS in 1977 refused to allow deductions the Smiths and Jacobsons claimed for silver trading losses arranged by Merrill Lynch and ordered them to pay a total of $58,000 in back taxes.

Government tax lawyers claim the two couples -- and thousands of other Merrill Lynch customers -- are not entitled to dudct their losses on their silver trading because the transactions were made only for tax purposes, with no potential for profit. Under established tax law, if there is no profit involved in a business transaction, no loss can be claimed for tax purposes.

The silver tax straddle involves simultaneously buying and selling contracts for future delivery of silver.

If the price of silver goes up, the investor will make money on the contract to buy silver (because the metal will be worth more than he paid for it) but will lose an equal amount on the contract to sell. If the price goes down, the investors makes money on the contract to sell and loses on the one to buy.

Either way, the customer gets a tax deduction on the loss. Usually that is claimed in the current year. The profitable contract is held until the following year. Even then the customer can avoid taxes on the profit, by setting up a second straddle to defer the income for another year.

The "butterfly" version of the transaction calls for buying and selling four futures contracts in a preplanned sequence that IRS auditors claim virtually assures the desired tax results regardless of what happens to the price of the commodity.

The IRS in 1977 issued a ruling that butterfly spreads are illegal. Since then, Merrill Lynch and other companies have continued to offer variations of that forbiden device while the IRS ruling is being challenged. v