It is like a curtain falling.

J. Carter Brown is speaking about children and change, about a son who had cancer as an infant and a daughter he is just beginning to know and a long and painful divorce that ended in a nasty skirmish over custody. First he turns his gaze toward the catalogues and papers precisely stacked on his desk. Then he looks down at his lap. Then he turns to his visitor, but as their eyes meet his gaze rises and his lids lower and he is speaking, as he so often does, with his eyes completely closed.

Carter Brown is not a man given to impassioned confession or self-revelation or even eye-to-eye contact. He is a man given to a meticulous and chilly charm, professional accomplishment and the habit of shrouding his soul from the observing world when he speaks of the things that matter most intimately to him. In that sense he has always been something of a public mystery, and especially so in January with his unexpected announcement that he would retire as director of the National Gallery of Art by the end of the year.

Why would he leave? After 23 years as director and eight years before that at the Gallery, his love of his job was obvious and unstinting. Now 57, he has never appeared to grudge the time and travel his position demanded, or to feel torn between his professional and his private lives. Despite two marriages and two children, he seemed somehow not to have a private life. More than a father or a husband, he was the National Gallery.

Brown himself has offered a number of explanations for his planned departure. He had begun his career young and so was free to finish it young as well. He wanted to spend more time with his children, he said in his letter to the gallery board. He wanted to devote more time to his family's philanthropies. He wanted to have more time to himself.

By now he realizes a lot of people just do not believe him.

"Ahh, the rumors," he says, sitting in his pristine office in the gallery's East Building. "I really should collect them all. They're so delicious, and they're not even rumors when you hear people talk about them -- they're truth." The tales abroad in the city and certain artistic circles of the country include battles with his board, battles with NGA benefactor Paul Mellon, and perhaps the most persistent, that Brown's health is failing.

He goes down the list. Not true. Not true. Not true. To dispel what may be the most persistent rumor, that he is terribly sick, he refers to annual checkups and the fact that "I have given blood to my son" for the 14-year-old's recent series of operations to repair a curvature of the spine. This is an oblique but clear denial to whispered suggestions that Brown, a man who tends toward the gaunt at the best of times, has AIDS.

Being Carter Brown, he takes no visible offense at addressing such questions. He is, after all, master of a certain type of wealthy WASP charm -- part effusiveness, part reserve, part noblesse oblige, part everyman. "It's the mind-set," he says of the rumors. "Everyone's always looking under the bed. No one can take anything at face value." He is smiling with a deliberate ease that suggests he will lightly mock the mind-set but never do anything so impolitic as decry it.

"Two years from now when I'm suddenly building log canoes in Tahiti, people are going to say, 'That's what he was up to!' But as far as I know, I have no hidden agenda."

Yet Brown's description of the moment when he decided to change his life is far from compelling.

"I did think of it one morning, lying in bed. I was thinking of various options."

Various options. Lying in bed one morning, and like that, a life is changed.

But nevertheless, he insists, that is how it happened. And having come up with one particular option, he did some research and discovered he was in fact eligible for retirement. (Although one might assume a pension would be of little interest to a descendant of the family after whom Brown University is named and the son of the man who was once called the richest baby in the world, Carter Brown says of a pension, "It makes a difference." And his brother avers, "Our father was embezzled twice and he died a relatively poor man. By today's standards, we are very far from wealthy.")

The disbelief that greeted Brown's retirement was fueled by more than a prurient Washington cynicism. When his predecessor at the gallery, John Walker, told the New Yorker's Calvin Tomkins in 1989, "I worked very closely with Carter for eight years, and I didn't know him any better at the end than I did at the begining," both colleagues and observers knew just what he meant. When asked if he sought anyone's advice before deciding to retire, Brown says no, and when asked who his friends are, whom he relies upon for support, he mentions only his brother and sister.

Brown has about him a polish that inevitably evokes curiosity about what resides below the exterior, and over the years -- and particularly now -- there have been many keys proposed to explain his personality.

"I guess I come from a New England tradition in which you're somewhat reserved," he says now. "That just seems a decorous way to live."

Now, albeit in a voice that dips almost into a whisper, he offers somewhat more about his life outside the gallery than he is usually disposed to offer: His "charming" son, who at 14 is undergoing a series of operations to remedy the damage left by treatment for a devastating form of childhood cancer. His daughter, at 8 "so beguiling -- she is going to be a menace." His second marriage, to their beautiful and wealthy mother, Pamela Braga Drexel Brown, which crumbled, slowly and painfully, and only last December reached its end in a final tussle over custody.

"I have a wonderful relationship with them," he says of John Carter Brown IV, who is called Jay, and Elissa Lucinda Rionda Brown. "Though it's confusing to them about loyalties. What kids cannot understand is why their parents aren't sticking together, especially when there is a person who is as confused -- and the person is their father."

He ends the sentence with a punctuating silence. He chooses to go no further.

Brown did not have children until he was 43 years old. When his son was 2 he was found to have neuroblastoma, a cancer that doctors told his parents had an 80 percent chance of killing him. The treatment itself was traumatic, and because of it the growth spurt of adolescence has left Jay with a curvature of the spine. He has already had two operations to insert steel rods in his spine and fuse several vertebrae. "He's been the most fabulous sport. He's so bright and so charming. He's very athletic, but now he can't do much. He's doing a little lacrosse."

Brown describes the nights he and his wife spent at Jay's hospital bedside more than a decade ago as "a rough patch." This is, to put it mildly, a decorous understatement. He goes on. "It was the first child for me. My only son. In the old days we all had lots of spares -- now we pin so much to one child." He does not seem to hear the icy strangeness at the heart of this sentiment. He goes on. Since his son's cancer and the steel rods and the spinal fusion, there is the temptation to coddle the child, but he resists the urge. He says this, and now he is going on into a description of a study that showed baby gerbils who are sheltered from danger and choice grow up to be timid and incompetent adult gerbils.

Brown tells the gerbil tale with gusto, his voice infused with the sort of enthusiasm and delight in acquired knowledge that fills the tapes he records to guide visitors through gallery exhibits. This is my metier, the tone of voice says -- sharing an intriguing insight that reflects on one's life but is not of it.

The father and son share many interests, and over dinner they discuss architecture and sailing. His daughter, still a child, seems to be more of an enigma to him. Like her mother, she loves to ride, a hobby for which her father apparently has no natural taste. Last summer, however, he took her on a two-week trip to France and was pleasantly surprised by her endurance and curiosity. "I didn't even take the nanny this time. She's old enough that I could care for her myself."

Brown and his wife come from a formal, moneyed world of nannies and boarding schools, and despite his desire to spend more time with his children there are plans for Jay Brown to leave home in September for boarding school. "But there are vacations and weekend visits," Brown says.

The legal conclusion to his marriage and the fight over the children came on Dec. 19 with an agreement to share custody. One month later, Brown announced that he would leave his job by the end of this year. That progression, Brown's brother Nicholas believes, is telling.

"Things have come easily to him in some departments, but I think his private life has been arduous for him," says Nicholas Brown, Executive Director of the National Aquarium in Baltimore. "I think the important thing to recognize is that he's reached a new plateau in his personal life -- with two unhappy marriages behind him, but very, very happy with his children. I think frankly the fact that his divorce got settled -- he had the joint custody and he just felt it was propitious."

The road to divorce was a long one. Word of the separation came in 1988 and was almost as surprising to the outside world as the announcement of his retirement. In the public divorce papers Pamela Brown said the separation began around August 1987, although the Browns continued to live together in their Georgetown house until February 1991. Asked whether the divorce was particularly bitter, Brown says, "What divorce isn't? We're both adults." He pauses. "There seemed to be some animosity there I didn't really understand. Why should someone be angry?"

But the public divorce papers suggest that anger flowed in both directions. Carter Brown's request for joint custody severely criticized his wife's skill as a mother, and accused her of attempting to harm his relationship with his children "by saying derogatory things to them" about him.

Pamela Brown declined to be interviewed for this story, and Carter Brown dismisses the corrosive rhetoric of the petition, saying, "Those documents are all part of the litigation process, and that's all settled now. Everything is worked out and we have resolved the custody." In fairness, such charges are not uncommon in custody battles.

Whatever anger was, or remains, within him, he reveals none of it when he describes his wife's eventual departure from his house. He describes this event as if telling a peppy anecdote about a tricky yet successfully concluded gallery deal. One day his deputy director, art historian John Wilmerding, came into his office and announced he was leaving the gallery for a professorship at Princeton and a position at the Metropolitan Museum of Art. "I said, 'John, before you leave this room, promise you will give your house to my wife.' " Apparently, John promised. The Georgetown house ("a lovely historic house," says the gallery director) is just down the block from his home and, notes Brown, "They saved paying a broker's fee."

It is certainly possible to read the legal papers relating to the divorce and infer emotional connections: on the one hand, a grueling battle over the love and loyalty of one's children, and on the other, a man's decision to remake his life. But despite the emphasis on his children in his letter of resignation, Brown says the divorce had nothing to do with his decision to retire. "That was all way back last fall." The custody agreement does "give me a clear responsibility in parenting," he says, and this is why he has stressed it.

And does he look on that as a new or different responsibility now?

"If you have a functional marriage and a wife who isn't working and who can be the parent full time, I do think that is a different situation than being a single parent 50 percent of the time."

Brown now says that his job did in fact wear him down. "I was living a life in which I felt a rope had been attached to every limb and was being pulled by horses running in different directions."

But there remains the question of just how radically he has changed his life. As he pointed out at the time of his announcement, Brown sits on 14 different boards of trustees and admits, "I am not the kind of person who lolls around." He has assiduously repeated that he has removed himself from the process of selecting a new director, and says that when the trustees asked him to maintain an office in the building he declined.

But the ties that bind a man to an institution he has helped shape are tight. Asked how he imagines the museum will change over the next five or 10 years, he speaks not of the future but of the past, about the East Building, which he shepherded into the world, and how its architect, I.M. Pei, created a building that could be very flexible. He expects to see that flexibility used even more in the future, he says.

And perhaps, perhaps he will still have an emeritus role to play at the gallery. "I could have used a me in the past few years when there were negotiations in Russia or Mexico," he says.

If it is hard for others to imagine him out of this office, with its Rothko and briefcase holding a file marked "Expedite," perhaps in some way it is hard for him as well. If that is so, however, Carter Brown is unlikely to say.