Stuart Downs is a happy man.

At 30, he's already director of a private, nonprofit presidential museum, the only one in the country dedicated to James Madison.

The work is rewarding, the pay is steady and the location -- in sight of the sweeping Blue Ridge Mountains about 95 miles southwest of Washington -- is stunning.

But one thing irks him.

"Sometimes I'd like to take out an ad in The New York Times and say, 'Somebody please adopt this president,'" he says, staring out the window at a car passing by the museum on Rte. 15 -- James Madison Highway.

What frustrates the soft-spoken Fairfax County native is that, unlike George Washington or Thomas Fefferson, whose Virginia homes are tourist mecas, Madison has gotten little attention here on his home turf.

Part of the blame may be Madison's, whome John Kennedy once called one of the country's most underrated founding fathers.

"His letters were formal and intellectual, while Jefferson's were more charming," agrees Fredricka Teute, a consulting editor on Madison's papers at the University of Virginia. "Also, he was short and not too attractive." a

Even the James Madison Museum, a small storefront operation in an old, red brick Chevrolet dealership, shows signs that the father of the Constitution isn't big box office. There are no Madison T-shirts, mugs, paperweights or key chains for sale here.

In fact, there is no gift shop. The only souvenirs are a few books ["Dolley and the 'great little Madison'" is the best seller], pencils [red, white and blue only], and bottles of White Rose perfume, touted as "Dolley Madison's favorite."

A lone receptionist collects the $1 admisssion fee to the six exhibits inside, where a dozen Madison relics are displayed under glass.

Business is slow, and Downs thinks he knows the reason why: Montpelier.

"A lot of people look at us and think we don't have a shot at the big house," he says, meaning Madison's historic home at Montpelier Station a few miles west of this rural county seat.

There is no connection between the museum and Montpelier because Montpelier is private -- very private.

A graceful 2,000-acre estate with a house built by Madison's father and enlarged by the president and his wife, it was purchased in 1901 by William duPont of Wilmington, Del., and remains one of the few major presidential homes in the nation that is in private hands.

DuPont's daughter, Marion duPont Scott, former wife of Western movie star Randolph Scott, still lives there, as she has for decades, in splendid seclusion.

Cattle and thoroughbred horses graxe on the rolling pasture, out of sight of the highway. Queries from schoolchildren about the property, where Madison lived until his death at 86 in 1836, are forwarded to the museum for answering. l

The public is allowed on the grounds, but only to visit the small gravesite where James and Dolley Madison are buried and to attend annual horse races on the estate's own track. Few outsiders in this century have glimpsed much more of the farm or the inside of Madison's mansion.

"It's a private residence," says V. R. Shackleford Jr., an Orange attorney who represents the owner. "People certainly don't knock on the door [it's a half mile from the highway] and ask to come in."

Visitors with business at Montpelier enter from the road through unmarked brick posts near a neatly restored railroad station. The driveway winds for 100 yards through woods, then opens suddenly onto the grounds and a dramatic view of Madison's distant house on a hill.

The mansion, recently painted a soft peach color, appears much as Madison left it. Boxwoods growing in the formal garden were planted when Madison lived there. Two wings of the house were enlarged by William duPont (there were sheep in the basement when he brought it), but otherwise the architectural features have survived.

The view across the farm to the Blue Ridge, 20 miles away, is the same one Madison saw when he stood on his four-pillar portico.

"This is my fantasy," says Downs, stopping his car well short of the mansion on a recent visit. "People could park down by the road and take buses up to the house . . . ."

Not surprisingly, it has crossed the minds of Downs and the board of the James Madison Memorial Foundation, operators of the museum, that Montpelier would be the ideal monument to the fourth president of the United States. But whether and when the estate might be opened to the American people are questions that go politely unasked in public in this bastion of landed Virginia gentry.

Even approaching the elderly Mrs. Scott on the subject would be delicate, says a National Park Service official in Washington, and would have to be done by "someone on her social level."

Those who speak for the owner say they have no idea what lies in store for Montpelier. Chester Hazard, the estate's business manager and an employe since 1928, refers such questions to Shackleford.

The lawyer says only that the property eventually will pass to five nieces and nephews of Mrs. Scott, who is childless. In the meantime, Shackleford adds, "I'm not in a position to say what it's future will be. It's not a national monument."

It is, however, a National Historic Landmark, having been so declared in 1970 by the National Park Service. But landmark status carries little legal weight, according to preservation experts.

"It mostly limits what the government could do." says Larry Goldschmidt of the National Trust for Historic Preservation, in Washington -- "like build an interstate through it."

Neither the Park Service nor the National Trust are in a position to buy the property, even if it were for sale, according to officials. Tax records on file at the Orange County courthouse -- where Madison's will leaving Montpelier to Dolley is displayed -- put the estate's market value at $3.8 million. Both agencies have stopped acquiring historic sites, mainly for budget reasons

Back in Orange, the museum presents Montpelier in a 15-minute, narrated slide show, with the recorded voice pointing out the estate's considerable charms and its historical significance.

But what about the future?

"I frankly don't know what will happen," says attorney Shackleford. "I'm sure a lot of people share your curiosity."