These are frustrating days for the Smithsonian Institution's National Portrait Gallery.

The Washington museum, which is fighting for the right to pay Boston's Athenaeum $5 million for Gilbert Stuart's portraits of George and Martha Washington, has been another precious group of presidential portraits, done by the same artist, slip out of its hands.

To add to its chagrin, the gallerey has lost this rare matched set of portraits of this nation's first five presidents to as sister institution-the National Gallery to Art.

Two of theses portraits, of George Washington and John Adams, were gifts to the National Gallery. The other three-of Thomas Jefferson, James Madison and James Monroe-were purchased through its Ailsa Mellon Bruce Fund. All five Stuarts come from Thomas Jefferson Coolidge IV of Boston, Mass., whose family has owned them since 1853.

No price was disclosed, but art world sources familiar with the transactionm say that the National Gallery paid Coolidge $1 million, which is half of the appraised value of the five-portraits set.

Boston Mayor Kevin White has compared the Portrait Gallery's purchase of the Attenaeum portraits of George and Martha Washington to "the Louvre selling the Mona Lisa."

Because a group of Boston citizens in 1831 raised $700 toward the purchase of these paintings, which Stuart's widow sold to the Athenaeum, a private Boston library, the courts will have to rule on the $5-million sale the Portrait Gallery has proposed.

But Coolidge is a private citizen, and he can send his pictures anywhere he wants.

Unlike the unfinished Athenaeum paintings, the five Coolidge portraits-the only such surviving set-were not done from life. Stuart (1755-1828), who lived before photograph, made a living painting replicas. George Washington posed thrice for the artist. One of the resulting pictures, the so-called Vaughan portrait, already is in the National Gallery's permanent collection.

With its new acquisitions, the National Gallery, in fact, owns 39 Stuart portraits including four likenesses of Washington. The National Portrait Gallery does not yet own one.

Its director, Marvin Sadik, first contacted the Coolidge family two years ago, as soon as he heard that the set might be available. As soon as he decided to go after the pictures he informed his gallery's commissioners, who act as his trustees.

Three of those commissioners also are on the board of the National Gallery-Chief Justice Warren Burger, Smithsonian Secretary S. Dillon Ripley, and J. Carter Brown, the National Gallery's director.

"The Coolidge offer," Brown said yesterday, "came out of the blue."

"As a Portrait Gallery commissioner," he continued, "of course I knew of the negotiations for the Athenaeum pictures. The Coolidge set fits totally with our collection. With George and Martha at the Portrait Gallery, and the presidential portraits here, it seemed the best of both worlds for both institutions."

The five Coolidge pictures already are here. They will go on view April 27 in the Gallery's West Building. But pending the result of Mayor White's litigation, the Athenaeum portraits will remain in Boston where they have been displayed, on loan, in the Boston Museum of Fine Arts since 1876.

Sen. Edward M. Kennedy (D-Mass.), who views the proposed sales as "a tragedy for the artistic heritage of Massachusetts," said yesterday he was "delighted" with the National Gallery's acquisition. "but this development is all the more reason why the Athenaeum paintings should remain in Boston," Kennedy said.

Stuart painted the five presidential portraits as a favor for his friend, Col. George Gibbs of Newport, R.I., between 1810 and 1825. They were purchased from the Gibbs estate by Thomas Jefferson Coolidge, a grandson of the third president.

His great grandson, Thomas Jefferson Coolidge IV, said yesterday he decided to part with them "for reasons of security. They were hanging in my mother's house, in Manchester, by the sea," he said. They're painted on wood. We were worried about the effect the cold winters and hot summers therw would have on their condition."

Coolidge's lawyer, George Abrams of Boston, who negotiated for the family with the National Gallery, said, "A number of institutions, among themthe Portrait Gallery and the White House, where they were on loan during the Kennedy administration, showed interest in the portraits. There was no conflict of interest on the part of Carter Brown. There was no competition between the National Gallery and the Portrait Gallery. Jeff Coolidge decided where he wanted them to go.

"The fact that Mayor White is suing to stop the Athenaeum sale is merely a coincidence: Carter Brown, Marvin Sadik, Teddy Kennedy, and I were classmates together at Harward, class of 1954." CAPTION:

Pictures 1 through 5, Matched portraits by Gilbert Stuart of this nation's first five presidents, left to right: top, George Washington, John Adams and Thomas Jefferson; right, James Madison and James Monroe.