The billionaire Hunt brothers of Texas defied a subpoena issued by a House subcommittee and refused to show up yesterday to testify about their role in last month's collapse of the silver market.

The subcommittee immediately voted 6 to 0 to recommend that the Hunts be cited for contempt of Congress, and the panel's irate chairman, Rep. Benjamin Rosenthal (D.N.Y.), threatened them with jail.

A lawyer representing Nelson Bunker Hunt and W. Herbert Hunt argued that the brothers ought to be able to ignore the House subpoena since they have agreed to testify before a Senate Agriculture subcommittee on Friday.

Rosenthal accused the Hunts of "forum shopping" and said they arranged to testify in the Senate in hopes of finding a friendlier reception there.

Congress is investigating whether the Hunts used their money and their influence with banks, brokers and federal officials to manipulate the silver market. Even if they were not rigging the market, the Hunts' speculative spree threatened to bankrupt stockbrokers and to set off a widespread financial crisis, the investigators believe.

(While the Rosenthal subcommittee was taking action against the Hunts, another panel of the House Government Operations Committee recommended, 8 to 0, that Energy Secretary Charles Duncan be cited for contempt for refusing to turn over documents on President Carter's decision to impose an oil import fee.

The Hunts' refusal to cooperate with the House inquiry is a "direct challenge to the integrity of this institution," declared Rosenthal, lecturing the Hunts' lawyer on the independence of the two houses of congress. l

Rosenthal said he would ask the House Government Operations Committee to vote as soon as possible to recommend that the House cite the Hunts for contempt.

In recent years the House has supported every subcommittee recommendation to issue contempt-of-Congress citations, penalizing two or three persons per term for refusing to testify.Among those cited in recent years for contempt of Congress were Watergate burglar G. Gordon Liddy and Koreagate figure Honcho Kim.

After a contempt citation is issued, the case goes to the Justice Department for prosecution as a federal misdemeanor, carrying a fine of from $100 to $1,000 and a sentence of from one month to one year in jail.

"I don't think a $1,000 fine would have much effect on them," Rosenthal said of the brothers, who reportedly lost more than $1 billion when silver prices fell from $50 an ounce to $10 in two months.

"A month in jail is not too small" a penalty to teach the Hunts they cannot ignore a subpoena from Congress, he commented after an hour-long session.

Federal Reserve Board Chairman Paul Volcker and Comptroller of the Currency John Heimann are to testify today before Rosenthal's panel.

Volcker and Heimann are also among the witnesses on the agenda for hearings Thursday and Friday by a Senate subcommittee chaired by Donald Stewart (D-Ala.)

Both panels are investigating a rollercoaster rise and fall of silver prices in the past year, which has been blamed on heavy buying by the Hunts.

The Hunts made millions of dollars when silver prices rose and used their proftis to buy more silver and make other investments. When prices started to fall, the profits vanished and the Hunts found themselves heavily in debt for silver worth a fraction of what they contracted to pay for it.

The Hunts lost so much money when prices collapsed that they have been forced to mortgage family oil and natural gas wells worth $3.2 billion so they can borrow about half that much to pay their debts.

The loan sought by the Hunts is so big that the banks consulted first with Federal Reserve Board Chairman Volcker, who for months has opposed making loans for speculative purposes.

Volcker approved the loan, apparently fearing that if the Hunts could not pay, banks and brokerage houses might go broke and the entire U.S. economy would be shaken.

Hunt attorney Roger Goldberg of Dallas told Rosenthal's panel yesterday his clients are so busy straightening out their financial affairs that they have not had time to prepare their congressional testimony.

Goldberg complained that the Hunts are targets of "multiple investigations by a variety of government agencies. As a basic matter of fairness, these investigations ought to be coordinated.

"We do not feel our clients should be caught in a competition for their testimony," he said. If House members have questions for the Hunts, he suggested, they ought to ask someone in the Senate to raise the issues for them.

"That's not the way things work around here," responded Rosenthal, calling the Hunts' argument for not appearing "illogical and impertinent."