Both men seeking to succeed Sen. William I. Scott (R.Va.) yesterday criticized the senator's decision to leave Washington days before the Senate adjourned.

But the sharpest words over Scott came not from Democrat Andrew P. Miller nor Republican John W. Warner, the two senate candidates. They came from State Sen. L. Douglas Wilder, a Richmond Democrat, who called Scott "a buffoon" in remarks at a black businessmen's breadfast that Miller attended in Richmond.

Warner, who has attempted to put distance between himself and Scott, said later in a Roanoke television appearance that he was "disappointed" over Scott's decision to leave the Senate Wednesday, four days before the 95th Congress adjourned.

Warner told public television interviewer Forest Landon, "Bill Scott is not an issue in this campaign." But Warner also said he believes Scott's conservative voting record "has been consistent with the thinking of the vast majority of Virginians."

Scott, who left Washington for a vacation with his wife after complaining he was ill Tuesday, came under attack at a meeting of Democratic state legislators here. State Sen. Adelard L. Brault of Fairfax, who began his legislative career defeating Scott, called the retiring senator "thats disaster that has been representing us."

Attacks on Scott and on Warner's integrity have become twin themes of Miller's campaign. Yesyerday, to the delight of Democratic legislature who have endorsed him, Miller indicated he intends to pursue those points. The legislators approved a resolution charging that Warner "has attempted to maling Andrew P. Miller without regard to the facts" and accusing Warner of inexperience and "ignorance." Virginia's senate seat "should not go to the highest bidder," the resolution said.

Afterwards Miller appeared to wide applause and once again stressed what he calls the integrity issue - the apparently conflicting statements b Warner about whether he sought the support of labor, why he was appointed Navy Secretary, and who he really supported in the 1970 Senate race.

"Truth has no high road or low road," Miller said later. "It spends for itself." Miller had begun to stress what he says are Warner's inconsistencies and said he will continue to do so despite Warner's charge that it is the "low road," and despite statements by Warner aides that it will only hurt Miller.

"They're just trying to contain their losses with respect to weakness," Miller said yesterday. "It's not just something that's effective. I feel very strongly about it."