The U.S. Tax Court ruled yesterday that Florida attorney Ronald Anselmo took an improper tax deduction for gems donated to the Smithsonian Institution.

Anselmo had valued the stones at $80,680, more than five times what he paid for them, but the court put their value at $16,800. As a result, Anselmo will have to pay about $30,000 in federal taxes.

The stones were part of a large donation of aquamarines, tourmalines, amethysts and topaz given by Anselmo, two of his law partners and a retired judge to the Smithsonian in December, 1977. The gifts cost about $70,000 and the donors took $350,000 in tax deductions over the next two years, a savings of nearly $100,000 in taxes above their $70,000 investment in the gems. Anselmo's attorneys claimed that the gems, which were bought in Brazil, should be appraised at the value they would sell for to retail customers, prices often several times higher than what retail stores pay.

But the tax court ruled that the standard for the relatively low-quality gems donated by Anselmo was the price that retailers would pay wholesalers.

The donations have been the focus of articles in The Washington Post, the subject of congressional hearings and the impetus for policy changes at the Smithsonian, which until recently had been willing to accept such donations without scrutinizing the reasonableness of appraisals.