ANN ANDERSON likes to say she's the only one of President and Rosalynn Carter's staff to take him seriously about going home at a decent hour husband, the one who wrote the suspense story, "The President's Mistress," took Carter even more seriously. Anderson quit as a White House speech-writer - putting words in someone else's mouth - to stay home full time. "I'd rather write my own novels than anyone else's speeches," he says.

If you could see their home, you'd understand why they prefer it to hanging around the White House.

It all might have been put together on a Hollywood sound stage as the perfect set for a Jimmy Carter-type small town, complete with tomato vines in the backyard and bicycles (unchained) in the front. And, of course, as is appropriate, Ann and Patrick Anderson and their two small children are notably movie-star handsome. The adults are just barely 40, but look about 15 years younger. He has full, curly, dark brown hair; she has close-cropped, neat dark hair. Both have lithe, fencer figures. They speak with the soft accents of Carter country.

The neat, yellow-brick house in Waterford, Va., perches on a rise somewhat primly, as a house that's been around since 1815 has every right to do. The stylish front porch is suitable for rocking and passing the time of day. Inside are all the amenities of home: fireplaces, handsome paintings, antique furniture and two children, one of each sex, of course. The house, among others, is open during the annual Waterford fair this weekend.

There are shady trees, and brilliant marigolds and impatiens in the garden. Across the street from the Andersons' house is a stable. It and the house are under a protective easement to protect the area's ambience.

On weekends, Ann Anderson is right out there in the backyard raising tomatoes - with much success. Daughter Laura, 9, and her mother ride on their ponies, stabled across the way.

Mondays through Fridays is different. Then neither Ann nor Patrick Anderson have what you'd call your average small-town job. They get up early. He fixes breakfast - and very well, too. Then she leaves at 8 a.m. to drive to the White House and her job as Rosalynn Carter's deputy press secretary. Laura heads for school, and 2 1/2-year-old Michael is picked up by his morning baby sitter.

And Patrick Anderson settles down at his IBM electric typewriter, with a fire warming in the fireplace, to write another chapter in his new suspense novel. "I'm right in the middle now," he said. "And it's doing well, so I'm not even tempted to stop to go to town for lunch. I write from 8 a.m. to 2 p.m. five days a week 1 long ago came to understand that what I do is write. And that's the best way to go at it. I find I write best early and taper off later when I get tired."

Anderson's most recent book. "The President's Mistress," about a presidential intimate who is murdered, often said to be aroman a clef, sold briskly, with the paperback rights going high. Answering a question about how he explained to his wife about the sex scenes in the novel, he looked wise and said "Never explain." And she looked complacent and said, "Never ask." And then he added, "I tell her I learned it all going to lunch with friends at the White House."

It was the $250,000 paperback sale that gave Anderson the financial foundation to go off on the Carter campaign trail.

Anderson told how the couple became involved with the Carters. "We went to a wine and cheese party at the next-door neighbors' and met Peter Bourne and Mary King (early Carter supporters). Ann is from Atlanta (he's from Texas), so we were vaguely aware about Carter's governorship. But we learned a lot more about him then."

Later, The New York Times called Anderson to ask him to do a story about the presidential campaign, and he suggested a profile on Carter. He followed him in Florida and spent a day with him in Plains. The resulting story was credited by Rosalyn Carter as being the first big boost for her husband's candidacy.

Later, after one Carter speechwriter had quit in a huff, Anderson became the chief Carter writer, subsidized, as he said, by the paperback sale (minus the agent's 10 per cent and the hardback publisher's 50 per cent, and so on).

"When Carter had to make his women's rights speech, I knew I needed help I could count on. So I called Ann, and she picked up Michael (then about a year old) and moved to Atlanta for two months. I was headquartered in Americus, but I'd come up to Atlanta or she'd come down for th weekends. Michael went to a day care center every day while Ann wrote the first draft Laura stayed behind with friends."

Ann Anderson added, "A great many of my friends thought I was crazy. But it was a once-in-a-lifetime thing. We are both glad we did it."

After the campaign, it was first said he'd continue as the speechwriter, but he decided to go back to his novels. "They are going well, so I thought I should keep at them. But I wouldn't have missed the experience of the campaign."

So Ann Anderson became the member of the family with the White House job. She had known Mary Hoyt, Mrs. Carter's press secretary, when both worked at the Peace Corps in public relations.

The Andersons had met on The Nashville Tennessean when she was fashion editor and he was the police reporter (good experience for the suspense novelist if not the presidential candidate speechwriter). After they were married, they came to Washington in 1961, when one of his Tennessean editors brought him along to the Justice Department to work for Bobby Kennedy. He left later to write his first book. "The President's Men," a non-fiction account of how the aides work, and later, "The Approach to Kings" and "Actions and Passions," both fiction.

Anderson doesn't seem a bit put out that his wife goes off to the White House every day. "My work has frequently incovenienced her, so I am willing to put up with some for her. And, after all, no White House job is ever forever. It's good to do it while it's there."

"Mary Hoyt is good about letting me leave more or less on time every night, except, of course, when there's a state dinner." Ann Anderson says. "Usually then its 1 a.m. 1 first thought I would stay in town with friends on those nights. But I always drive home. I like being here at breakfast with the family." The drive between Waterford and the White House is better than an hour each way.

She admits that some housekeeping has had to slide with her new job, but nobody worries about the last speck of dust. A cleaning woman comes once a week. Anderson says his wife is a good cook, but she protests she doesn't do so much anymore, except on the weekends when they invite friends, some of them from the campaign, to have a Sunday dinner in the country. The Carters haven't done much private visiting, Ann Anderson notes, though they have managed a couple of church suppers.

"They do have some people come in often for dinner," she said. "But most nights, he goes back to work after dinner. Both of them really like their jobs you know. I can understand why she does. She has always had the talent to do things, and now she has the place to do them. And it's pretty neat for a woman to have all the mundane house-keeping chores all taken care of, as she does. She's going to be even busier this fall, especially on her mental health program."

This is Ann Anderson's first full-time job since they bought the Waterford house in 1969. Until then they had been living in a house they owned on Capitol Hill. They had driven to Waterford, just beyond Leesburg, many Sundays, for the pleasure of the country. One day they saw and bought the house, priced then at a mdoest $45,000 - it's worth at least twice that today.

Originally, according to the local story, what is now the living room, was a school room. It has a separate entrance onto the porch. The room is unusually large, perhaps attesting to that early use. A smaller sitting room with a fireplace and a television set (it will go into the cabinet when Michael learns not to turn it off in the middle of the news) is on the other side. At the back, in what is probably a later addition replacing an earlier el, is a good-sized dining room and a splendid kitchen with a cooking counter. Off the side is a covered patio that Ann Anderson wanted and he didn't and now they both do. The shed tin roof was copied from country roofs in the vicinity.

All the walls have Victorian-style fauna and flora patterns, including big roses in the immense bathroom, which serves the children sometimes as a rowdy room. There are three bedrooms in the front wing, and then down steps and up more steps to the master suite over the back el. The Andersons have a hall of closets, a bath and a bedroom big enough for a chaise lounge.

You can see why the Andersons would be so fervent in their support for the Waterford Foundation. Before the Carters, Ann Anderson was chairman of the committee that successfully put together, with the help of Tom Richards, a consultant on land planning, an elaborate program of scenic easements to preserve not only the facades of the fine old Waterford homes, but the open spaces, the hills and fields and valleys which are just as, if not more, important to the peace and prosperity of the town.

The Waterford Foundation owns 10 buildings and 40 acres of open space, but thanks to Ann Anderson much of the town is protected by the easements. Essentially, scenic easements mean that the owner guarantees to maintain certain historic sites, and, if the property is sold, it is sold with a driveway easement. The concept is of vital importance to historic areas that want to maintain their charm.

Waterford must be doing something right. Despite the lack of public transportation to the town, houses have doubled in price. "There are nice who work in town. Others work nearby. We enjoy the people. It's pleasant for me, since I can work anywhere," Patrick Anderson said.

So that's the way it is with the Andersons: the suspense novelist with a head full of murderous plots and counterplots and fearsome fugues; the hand in the daily goings and comings of the world's emperors and kings; Laura the horseback rider and Michael the clown prince.

So what are they going to do, eight years hence, when it's all over but the books of reminiscenes? "I think I'd like to a gardener," says Ann Anderson. "I'm apprenticing in the Jacqueline Kennedy garden."

The 34th annual Waterford homes tour and crafts exhibit ends today in the 18th-centure Quaker village in Loudoun County. Tickets are $3, which includes parking and entrance to all homes and exhibits. Hours are 10 a.m. to 5 p.m. CAPTION:

Picture 1, The neat, yellow-brick home of Ann and Patrick Anderson and their children, Laura and Michael, in Waterford, Va., perches primly on a rise.; Picture 2, Inside are all the amenities appropriate for a house that's been around since 1815: fireplaces, handsome paintings and antique furniture. Photos by Tom Allen - The Washington Post Picture 3, The Andersons' front porch., by Tom Allen - The Washington Post