When Andrew Pickens Miller was a 9-year-old boy in Fairfax County, the old Byrd Democratic organization ousted his father from the General Assembly by smearing his mother as a Communist.

Helen Hill Miller's most subversive activities had been writing a biography of George Mason, working for the U.S. Department of Agriclture and holding a Ph.D. in political science from the University of Chicago. But in the closed Virginia society of 1941, where erudition and nonconformity were suspect, that was enough.

"You don't ever forget things like that," Democrat Andrew Miller, now 45, said the other day in a rare lull in his race for the U.S. Senate from Virginia. "No matter how old you are at the time, you never forget."

There are those who say Miller's vivid memory of those family wounds have shaped much of his life, and are shaping today his exceedingly close race against Republican John W. Warner.

Miller's Oxford-educated father, Francis, had been independent of the Byrd organization in the Virginia legislature and was an internationalist known to travel abroad for such groups as the World Student Christian Federation.

When the votes were counted in 1941, the elder Miller lost by some 180 votes in a campaign described by the conservative Richmond newspapers as "a preposterous and dishonest campaign of innuendo" directed against "one of the most brilliant and useful members of the General Assembly."

Francis Pickens Miller went on to other glories, including honors with the Office of Strategic Services in World War II and a lectureship at the Yale Divinity School. and to other defeats, including two bitter campaigns in Virginia - one for the governorship in 1949 and one three years later against Harry F. Byrd Sr. for the U.S. Senate.

Those who know his son well say Andrew P. Miller, a Princeton-educated Phi Beta Kappa and sevev-year Virginia attorney-general, was affected profoundly by the attacks on his parents and determined early in life to avenge them.

He has done so, they point out, by painstakingly building a political career over nearly two decades in a state traditionally suspicious of the alien and the new, and by avoiding the sort of high ideological profile against which his father's enemies trained their guns.

Driven by a combination of personal ambition and Presbyterian zeal, he remains, they say, an intensely private man in the public arena: highly sensitive to criticism and, above everything else, cautious in his every move.

it is that same sense of caution that some Democrats say has given Miller a bland, if not fuzzy image - an image that denies him the charisma and warmth that Warner and his other opponents over the years have projected.

Miller's caution. however, is not totally a product of personality. It is also the fruit of a recent Virvinia Democratic Party history of bitter factionalism and regular defeat, against which Miller's plodding moderation has emerged as what Miller says is the party's last, best hope.

Upset in last year's gubernatorial primary by the party's strident, populist gadfly, Henry Howell, Miller emerged this year as a subtly defferent candidate - less defensive, less certaiin, more fatalistic.

Last summer, a week after one of his campaign planes slid off a runway and caught fire (Miller was unhurt) his Republican opponent, Richard Obenshain, was killed in a plane crash near Richmond.

"That taught me to rely more than ever on my Presbyterian sense of predestination," Miller said a few weeks later. "There only so much we ourselves can do."

Andy Miller, however, had not previously left much to chance. When he left the University of Virginia law school in 1960, he moved to the small town of Abingdom in Southwest Virginia - one of the few regions in the state where the Byrd organization that had pilliried his father had minimal power.

Throughtout the 1960s he allied himself with the young moderates in the Virginia Democratic Party who carried the state for Lyndon Johnson in 1964 and two years later seized the reins of the party from the aging Byrd stalwarts who had held them nearly 40 years.

Miller was one of the so-called "Spong pygmies" who elected William B. Spong Jr. to the U.S. Senate that year in an underfinanced campaign against incumbent 70-year-old Sen. A. Willis Robertson, who fielded the biggest spending exhibition the state had then seen.

Three years later, Miller ran and won an impressive race for attorney general, only to see the Democrats succumb to factional bickering and lose the governorship to the Republicans for the first time since Reconstruction.

For seven years he walked a political tightrope as the Democratic lawyer to two Republican governors - a career that endeared him little to the partisans of either party.

Even his critics found him "smart as hell," but they also called him evasive, equivocal and almost paranold in his caution.

Miller's caution seems intertwined with an almost obsessive belief in the power of reason and data. His speeches offer more scholarship than persuasion, and he litters them with legal terms like "pursuant to" and "with respect to" that may command respect but generate little enthusiasm.

Last week at a joint appearance in Newport News with Warner, a fatigued Miller was asked his stand on natural gas deregulation and launched into an exhaustive examination of the bill that was clearly more than the audience wanted to hear.

The more restless the audience grew, the more Miller piled on the facts until titters could be heard even among his supporters.

When Warner stood up to answer, he drew laughter by asking, "Well Andy, are you for it or against it?" Miller, visibly angered, refused to hand over the microphone and launched still another speech, expanding his answer further.

Miller's supporters say his passion for detail reflects the kind of exhaustive analysis that alone can make possible solution of the problems of a complex world.

An while critics deride his "bureaucratic" expansion of the attorney general's office from a staff of 21 attorneys to 84, Miller points out that the expansion was accompanied by a major shift in concept of the office from that of a caretaker legal referral service to "one capable of handling the state's legal business inhouse and doing so in a very competent way."

He also points out that private atopinions issued by his office was overturned by the Virginia Supreme Court.

He also points out that private attorneys, many of them cronies of state political figures, profited hugely from legal business farmed out by the state before he was elected, sometimes with the taint of conflict of interest.

And the expansion of the office, he says, has "not cost the taxpayers one dime." In just one public interest suit filed by him, Miller says, the state was able to reverse an Environmental Protection Agency decision that would have forced the state to spent $12 million on buses - more money than the amount appropriated for state legal services while he was in office.

Miller has campaiged on such issues throughout the years, initially refusing over and over again to either attack or needle Warner's sometimes erratic public statements.

When in late October he began to raise questions about Warner's integrity, he did so with documented statements, attaching less importance to the statements themselves than to the pattern they represented.

And if he made one speech a day about integrity, to keep the issue alive, he spent the rest of the day on the issues.

Many of those who know Miller say he shrinks from any sort of attack that might hint at the sort of thing used against his parents so long ago. But, ironically, his speeches on integrity are inevitably his most powerful, calling forth neither acid nor invective but a kind of moral indignation.

"In recent years Virginia has not had the kind of representation she has had in her best years in the past," Miller says. "She has not made the kind of vigorous contribution to national affairs. I believe we can do so again."