In an attempt to end what critics have called "one of the most flagrant abuses" of federal tax laws, Internal Revenue Service attorneys will go to tax court today in a complicated case that could prove embarrassing to their boss, Treasury Secretary Donald T. Regan.

The case is the government's final effort to eradicate "the silver butterfly," an exotic tax avoidance scheme that allows billions of dollars to flutter through a loophole in the income tax code.

The silver butterfly is a peculiar form of commodity trading that the IRS contends is used only to create artificial tax write-offs. The technique has been aggressively promoted by Merrill Lynch Pierce Fenner & Smith, the nation's biggest stockbroker.

Treasury Secretary Regan was chairman of Merrill Lynch before being named to the Reagan administration. Today's trial involves two Merrill Lynch customers and commodity transactions that occurred when Regan was running the company.

Testifying before the Senate Budget Committee last Wednesday, Regan said he has scrupulously avoided any interference with the IRS lawsuit since he became secretary of the treasury and overseer of the IRS. Regan acknowledged his former job at Merrill Lynch could require him to recuse himself from discussions of tax law changes involving commodity trading.

Regan said the Reagan administration has not yet decided what position to take on legislation introduced in Congress that would go even farther than the IRS lawsuit in seeking to curb use of commodity trading to avoid income taxes.

Sponsors of the two bills -- Sen. Daniel Patrick Moynihan (D-N.Y.) and Reps. Benjamin S. Rosenthal (D-N.Y.) and William M. Brodhead (D-Mich.) -- have labeled commodity tax schemes one of the most flagrant abuses and biggest loopholes in the federal income tax law. They say commodity tax gimmicks cost the treasury at least $3 billion a year.

The trial, scheduled to begin this afternoon in U.S. Tax Court in New York, involves two Merrill Lynch customers who were ordered to pay $57,000 in taxes after IRS auditors refused to let them deduct losses claimed for silver trading transactions arranged by Merrill Lynch.

The IRS contends the losses are phony, deliberately created by a commodity trading technique known as the butterfly spread or a tax straddle. The version promoted by Merrill Lynch and other brokers has been called a silver butterfly because the commodity traded was silver, but similar transactions have been carried out in soybeans and other commodities.

Last November, Merrill Lynch tried to prevent the case from going to trial by offering to pay the customers $114,000 -- twice as much as the government claims they owe in taxes -- if they would drop the case. IRS attorneys accused Merrill Lynch of trying to avoid a court decision that would be damaging to the company and its customers and persuaded Tax Court Judge Arthur Nims to order a trial on the issue.

Several of Regan's former colleagues at Merrill Lynch have been subpoenaed as witnesses, including vice president Thomas P. O'Hare, head of the department that arranges tax straddles. O'Hare is said to be the most skillful manipulator of silver butterflies and the man who personally handled the complicated transactions.

The trial also is expected to make public for the first time evidence gathered in 1977 by a federal grand jury that investigated possible tax fraud charges related to Merrill Lynch's transactions in commodities and other tax-avoidance schemes. The grand jury investigation did not result in any charges or other action and did not involve the two taxpayers in the current case.

This case was started by two California couples, Henry and Patricia Smith and Herbert and Ruth Jacobson, who made substantial profits on a real estate venture. Through the San Diego Merrill Lynch office, the couples arranged a series of silver trades that enabled them to avoid paying taxes on their real estate profits.

When the Smith and Jacobson tax returns were audited, the IRS refused to allow the silver tax deductions. The two couples challenged that decision and sued the IRS in Tax Court.

The silver transactions were made on the Commodity Exchange Inc. of New York in silver futures contracts. A futures contract is an agreement to buy or sell a commodity at some future date for a specified price.

A speculator who buys a silver futures contract makes money if the price of silver goes up because he get metal that is worth more than he agreed to pay for it. But if the price goes down, the speculator loses money on the contract because the silver is worth less than the buyer agreed to pay for it.

The silver butterfly straddle involves buying and selling silver futures simultaneously in a carefully planned pattern that, when diagrammed, looks something like a butterfly.

If the price of silver goes up, the investor makes money on the contract to buy, but loses an equal amount on the contract to sell. If the price goes down, the investor makes money on the contract to sell and loses on the buy side.

The gains and losses work out about the same regardless of what happens to the price of silver. The trick is to arrange the losses so they occur in one year and delay the profits until the next year.

Then the losses can be claimed as a tax deduction in the first year and used to offset other income. Taxes due on the profits are deferred a year, and can be stalled indefinitely if the taxpayer arranges another straddle the following year.

The gimmick can also reduce the tax rate an individual pays because of differences in the tax rates on various kinds of income.

In the top tax bracket, the rate is 70 percent on "unearned income" from investments. A taxpayer with $100,000 of investment income could arrange silver futures losses to wipe out that entire amount. The following year, the $100,000 profit would be taxed at only 28 percent, because futures contracts held for more than six months are considered long-term capital gains and taxed at the lower rate.

The IRS 1977 ruled that butterfly spreads are illegal because the transactions are not intended to make a profit. Federal courts have repeatedly held that business deductions can be claimed only for profit-motivated deals.

Expert witnesses hired by the IRS have calculated that the odds of making even a small profit on a silver butterfly are less than the odds of winning at roulette, where the payoff is 35 to 1.

Attorneys for the Merrill Lynch customers are expected to argue that the silver transactions were made in hopes of earning a profit and not just to create tax write-offs.