President Reagan's collection of miniature bronze saddles that visitors to the Oval Office find so fascinating really belongs to Walter Annenberg, former ambassador to Great Britain and art collector extraordinaire.

They almost belonged to Reagan until somebody told Annenberg he'd have to pay a gift tax if he gave them to the president.

"I said to hell with that," said Annenberg, one of the world's wealthiest men, "so instead they're on loan."

Portraying a history of western saddles starting with a Spanish war saddle, circa 1514, and ending with a stock saddle, dated 1910, the collection is estimated by Annenberg to be worth $50,000.

He bought it at an art auction in Sun Valley, Idaho, a few years ago where he and his wife, Lee, owned a home. Sculpted by Paul Rossi, former director of the Thomas Gilcrease Institute of American History and Art in Tulsa, the collection consists of 12 saddles, each no larger than eight inches high. In 1981 when the Annenbergs sold their Sun Valley home, Walter Annenberg decided the collection would look great at the White House.

"I knew the president was interested in that sort of thing. And, of course, the president was wild about them," said Annenberg, who decided there were just too many government "technicalities" to overcome to give them to Reagan.

The solution finally came from White House counsel Fred Fielding, who advised Annenberg to lend them to the White House while Reagan is in office, even though as an old friend Annenberg could have given the president a gift valued at more than $35 as long as the president reported it on his annual disclosure form.

"Ambassador Annenberg and the president have a pre-existing relationship," Fielding said. "Obviously, the ambassador wasn't trying to pitch anything."

If giving saddles to someone who is supposed to sit taller in his than everybody else is out ("The president, you know, has to be purer than the driven snow," said Annenberg), nothing could prevent Annenberg from giving the collection to Reagan later when he's a private citizen. Except, of course, that pesky gift tax which Annenberg, as donor, would still have to pay.

Meanwhile, Annenberg says he is well aware of the acquisitive habits of the White House curator's office and of White House committees trying to build a collection of art and antiques. Neither is he worried that the White House might try to hang on to his saddle collection.

"I have an official receipt showing that the saddles are a loan," he said. "They are not a gift."