"It's the hornworms, they get on the tomatoes and they're the most awful thing, they're huge," says Jennifer Russell, assistant director of the Whitney Museum of American Art.

"Oh no, we wouldn't spray them. We pick them off. Not me. I make Tom pick them off."

She is talking about her partner, Thomas N. Armstrong III, the museum's director.

Armstrong celebrated his 10th anniversary at the post on Saturday. He has been in the museum business 17 years.

He wanted to be a farmer.

This helps to explain why there is a vegetable garden on the roof terrace of the museum at 945 Madison Ave., in the roaring heart of Manhattan.

Starting today, the crop will be sold on the street on Tuesday evenings. Invitations have been mailed to noted vegetable fanciers from Woody Allen to Robert Oldenburg, director of the Museum of Modern Art, which has no vegetable garden.

Already several baskets of tomatoes and one pattypan squash are waiting in Armstrong's office along with the All-American Vegetable Stand T-shirts and the eager little signs that say "$1.00 each" and "2 for $1."

Armstrong makes no apologies for these round numbers. They are the very reason he quit studying agriculture at Cornell in his sophomore year.

"I realized that the others in the class had been raised on farms and would go back to them," he says, gazing out his fifth-floor window at the 16 tubs of vegetables. "I didn't know a thing about the business of farming. I get a kick out of selling tomatoes, but I'd hate to price one out."

He estimates they have cost him around $15 apiece.

But such tomatoes. They look loved. The plants are seven feet tall except for one that exuberates out of control at nine feet.

He also has one pepper plant, two eggplants ("Did you know eggplants are born purple? I thought they would be white or something"), two squash and a cucumber "crawling along there." The cucumber is rather unhappy because it has been moved three times.

This is definitely an art museum garden. The tubs are rimmed with lantana to mask their raw wood edges, and a life-size Robert Laurent bronze nude lounges inscrutably among the leaves.

Armstrong, 52, was raised in Summit, N.J., the son of a metallurgical engineer. For six boyhood summers he worked on a dairy farm, pitched hay, grew flowers, won prizes at the Sussex County Fair. After his disillusionment with agriculture he graduated in 1954 as a bachelor of fine arts, to the wary relief of his parents. They had been nervous ever since he turned down Princeton to go to Cornell. He was still a long way from metallurgy.

"I studied for a master's at NYU but never graduated because I couldn't pass the German exam. So I went into business for eight years."

Later, helping to run the Abby Rockefeller folk art collection at Williamsburg, Armstrong took up daffodils and won some prizes with those, too, proving that museum directing has more to it than meets the eye. He is celebrating his anniversary with a kite-flying contest at Fisher's Island, where his family has a summer place.

Meanwhile, the farm has attracted considerable attention, notably from population activist Stewart R. Mott, who lives nearby at 800 Park Ave. Addressing Armstrong at the "Whitney Agronomy Institute," Mott claimed in a letter that the pigeons who dine on his rooftop tomatoes and eggplants have been seeding and fertilizing the Whitney garden, thus entitling him to half the proceeds from Armstrong's roadside stand.

Armstrong rejects the claim utterly. "Talk to Joe Mahon," he says. "He's the one who tells us what to do."

Mahon, a Whitney building engineer who is in reality an operatic baritone, raised the plants from seeds in flats at his home in Spring Lake, N.J.

"These are not ordinary seeds," Mahon says. "You can't buy them on the market. They aren't hybrids. They come from the Seed Savers Exchange, been handed down by farm families for generations."

The exchange, in Princeton, Mo., boasts 3,000 members nationwide who grow and perpetuate vegetables of good family. One variety is the Brandywine tomato, which traces its lineage to West Virginia in the 1870s and turns out two-pound fruits. Another is the Sabre, 1890 vintage, 2 1/2 pounds of tomato, pink on the outside, blood-red inside -- the ancestor of the beefsteak tomato. And the Andrew Rehart, named for a visionary farmer.

"These seeds are the real stuff. You take Howling Mob white corn. When the corn blight hit in '70, Howling Mob wasn't affected because it's genetically different from these hybrids. Hybrids don't breed true."

There used to be thousands of tomato varieties, Mahon says, just as there were 7,000 different kinds of apples once, but the mass market killed them off in its search for produce with longer shelf life.

"The size of these things you wouldn't believe," he says. "And the taste! You have just never had vegetables like these. Let me tell you about my purple broccoli . . ."