Art collections, like fine gardens are best developed slowly. The best collectors show us more than what they've bought. Through pruning and refining, and by the careful matching of one work to another, they make of varied objects an orchestrated whole. Buying art is easy, all it takes is money. Collecting art requires more, a unifying point of view, an eye for subtlety and harmony. Collecting art takes time.

There is something rushed and awkward about "American Folk Painting: Selections From the Collection of Mr. and Mrs. William E. Wiltshire III," now at the Philips. It includes some first-rate pictures, but is not a first-rate show. Its quality is uneven, it is overloaded with dark and formal portraits, it seems something of a grabbag. It was put together in five years by a Virginia businessman whose impatience shows.

William E. Wiltshire III. an insurance executive from Richmond, has assembled - and dispersed - other collections in the past.

He first bought Civil War memorabilia. That collection has been sold. His second collection, of antique American furniture, was auctioned in New York earlier this year. He began to built his third collection, of American folk pottery, aboyt 10 years ago.

He did not only buy the pottery called "redware"; he studied and promoted it. While some collectors hoard their works of art at home, where they can see them daily, Wiltshire often lends his to museums. As late as 1974. Wiltshire was still buying redware pots at auction. In 1975 he lent his finest pieces to the Abby Aldrich Rockefeller Folk Art Collection in Williamsburg. The show was called "Folk Pottery of the Shenandoah Valley." His next step was to publish a book of the same name. Then his works were sold.

The Wiltshire pottery collection was auctioned off a month ago at Sotheby Parke Bernet. Bowls by anonymous potters often are sold without references, but in the antique market - which today is booming - references do help.By the time they got to Sotheby's, many Wiltshire pots has accrued good credentials. They had been shown in a museum, they'd been mentioned in a book, and their new credentials were cited in the auction catalogue when Wiltshire's pottery was sold.

One piece in the sale, a mid-19th-century pottery lion by John Bell, fetched $18,000, much more than the estimates and an auction record for an American ceramic.

Wiltshire began buying the folk paintings now on view at the Phillips just five years before he sent them out on tour.

His show opened last year in Richmond. This year it will go to art museums in Pittsburgh, Columbus, Ohio, Denver and San Diego. Next year it will visit Seattle, Fort Worth and New York. All of Wiltshire's paintings, whether fine or trivial, Will have an impressive exhibition history when, at the end of 1979, they are finally sent home.

Wiltshire's lavish catalogue treats all of his pictures, even six-inch sketches, as important works of art. Each work in the show is reproduced in color. Color plates are costly; Wiltshire paid the bill.

Wiltshire was asked if this, his fourth collection, as the other three have been, eventually will be sold. "No," he said. "Not in the next four or five years. After that, who knows?" Wiltshire says his taste lately has changed. His furniture and redware were auctioned, he explains, because they do not fit the comtemporary design of the new house he is building. "But I'm buying paintings, still."

"Most dealers and collectors of folk painting suspect that the great days when the best examples of this art form could be found in quantity are over," the catalogue informs us, and the catalogue is right. "The market, once so plentiful, has largely dried up," wrote John Walker, then director of the National Gallery of Art, in 1969. Wiltshire had not then begun his collection. His competitors - among them Col. Edgar William and Bernice Chrysler Garbisch, who began to form their 2,600-item collection during World War II - already had skimmed the cream from the available supply.

Considering the market, and the narrowness of his options, Wiltshire did not purchase badly. Some strong and charming, are in his exhibition. What is missing is a context that might let these pictures sing.

Though Isaac W. Nuttman's bushel-basket "Still Life" of 1865, with its piles of grapes and peaches, is a large, important picture, its importance is not stressed, for it is the only still life included in the show.

John Rasmussen's 1880 farm scene is as colorful and detailed as any Grandma Moses. This picture, says the catalogue, is the only landscape of its sort by that Pennsylvania Almshouse Painter. Were other works by Rasmussen on display beside it, its uniqueness might be seen. Sheldon Peck's double portrait of 1845, Thomas Chambers' exuberent Hudson River landscape and Winthrop Chandler's 1773 portrait of the stern-faced and bewigged Rev. Ebenezer Gay are other worthy pictures on display.

Though once they were dismissed as primitive and crude, American folk paintings today are much admired for their historical associations, their innocence, the boldness of their patterns, their humor and their charm. All of Wiltshire's pictures are American antiques, but the weaker ones also are primitive and crude.

Look, for instance, at that somber, thick-necked child painted, without charm, by Jonathan Budington of Connecticut. Jacob Maentel's 1815 watercolor is a work of no distinction. Too many Wiltshire portraits, the hand-made snapshots of their era, are dark and bland and stiff.

The itinerant limners who produced them often were self-taught amateurs, and their lack of training shows. Unlike their academic colleagues, they did not understand perspective, or the play of light and shade. Though capable of handling draperies and lace, many of these artists has trouble painting hands. There is a child in a green dress here, by an anonymous artist of the 1830s, whose hands resembled blobs. Those of the lady portrayed in profile by the Borden Limner look more like lobster claws. "American Folk Painting: Selections From the Collection of Mr. and Mrs. William E. Wiltshire III" closes at the Philips Collection, 1600 21st St. NW, April 9.