UP UNTIL NOW, it has not been easy to feel particularly sorry for Billy Carter. He has been, by all accounts, a goof-off, a publicity hound, an embarrassment to his family, and particularly to his brother, who happens to be president of the United States. But until the Libyan connection, Billy Carter had not done anything really dreadful or really politically damaging to the president. Since a lot of us have had, at one time or another, a relative we'd rather forget about, it has been simple to go along with the Carter family attitude that Billy is, well, you know, Billy.

In fact, it now appears that we haven't really known Billy at all. We've been looking at Billy by way of Jimmy Carter, cringing with him at Billy's antics, and then looking the other way. But now, in an interview with Sally Quinn of The Washington Post, Billy Carter's wife, Sybil, has said, in an anguished voice, look at Billy. Look at what merely being the president's brother has done to Billy. And if we look at Billy the way Sybil looks at him, and see what being the president's brother has cost him, it is not a very pretty picture.

Take the matter of the peanut warehouse. When he became president, Jimmy Carter put the peanut warehouse in a blind trust.This was a rather routine thing to do, which no one paid much attention to at the time. But according to Sybil Carter, the peanut warehouse had been run by Billy Carter while Jimmy Carter was governor of Georgia. Billy's share of the warehouse was put into trust, too, and when other people started running the show at the warehouse, Billy did what a great many normal, red-blooded Americans would do. He quit. If you look at the thing the way Sybil Carter looks at it, Billy Carter got totally ripped off. Having lost his way of making a living, he decided to pick up on the country fair and speech offers that had started coming in. The way Sybil Carter, mother of six tells it, it was work.

About this point, it is time to say, okay, but what about the Libyans, and the answer, based on the evidence that has surfaced so far, has to be, what about the Libyans? So far, the Libyan connection seems to be a series of very dumb moves by Billy Carter who still doesn't seem to be able to grasp that there is no way he can escape being the president's brother, at least part of the time.

But Sybil Carter raises real and troubling questions about whether he has to be the president's brother all the time. Have we, in fact, not gone overboard on presidential relatives, stripping them of their privacy, treating them as curiosities, when most of the time there is no compelling reason to do so and no particular public interest to be served?

Take the case of Amy Carter. a.k.a First Child and First Daughter. Amy Carter's schooldays became more famous than Tom Brown's. At first, there was a point to the story. Amy going to a public school in Washington was a measure of the Carters putting their daughter where their mouths were when it came to public education.

But it didn't stop there. We read about Amy's first day in school and about her last assembly. We read about how she adjusted to her school and about her showing a plant at the Arboretum. We read about her school's alumni day and its extended day program. We got stories about Amy's teacher, who gave her a dog. We read about the president attending her violin recitals, and about her going to classes for gifted children, and about where she was going skiing over Christmas, and about her birthday parties and what she wore. We read about her tummy viruses and her tours of department stores and about her braces and what she wanted for Christmas, and we read about her pet herimt crab. What we didn't read was, who cares?

This is not to put down Amy Carter or Billy Carter or other members of the Carter family. It is to say, however, that we, the press and the public, have taken the First Family and tried to turn it into royalty, following their travels and entertainments at schools and so forth the way the British follow their royals, forgetting, it seems, that British royals are trained from birth to be royals, that they are paid handsomely to do so and that, furthermore, the concept of a royal family was something we rejected a couple of hundred years ago.

It is difficult to say who is to blame for this. The media would not be covering the relatives of candidates if readers and viewers weren't interested in them, and ever since the Kennedy White House we've known that the public cares about every button and bonnet on the First Ladies. We have followed Kennedy family bouts with alcoholism, drugs, their marriages, their childbirths, their graduations, their jobs, their vacations.

The Reagans are getting their share of it, too: their son can't be a ballet dancer without thinly veiled questions being raised about his sexuality. We have come to view a politician's entire family, not just his wife, as public figures, ignoring the fact that most relatives of politicians have no more training or obligation to be public figures than do our own relatives.

Sybil Carter, in the Washington Post interview, talked of mobs of people coming to the peanut warehouse, of how her family had to move from a home in Plains to one 25 miles away, on a secluded road, to regain its privacy. "People would walk in the house without knocking on the door," she said. "We'd step out in the yard and people would be taking our pictures . . . My children could not ride their bikes in the yard, they could not play in their own yard."

Billy Carter made money by being the president's brother. The way Sybil Carter looks at it, it was a way to support the family. Billy Carter lost his job when his brother became president and the way that was handled may tell us something about his brother. But Billy Carter also lost something else: He lost his privacy, and for no particularly good reason other than general curiosity, he became someone to be watched, covered, gawked at, photographed and followed. To Sybil Carter this clearly has had something to do with destroying a man.

And that may tell us something about ourselves.