Four Virginia Democrats considered to be likely contenders for Harry F. Byrd Jr.'s U.S. Senate seat have agreed to a weekend summit meeting where they will try to agree who should be the party's nominee.

Former Virginia attorney general Andrew P. Miller of Alexandria, state Senate Majority Leader Hunter B. Andrews of Hampton, state Democratic Chairman Owen Pickett of Virginia Beach and Alexandria lawyer William Thomas, a former state party chairman, will meet in Fredericksburg Saturday to discuss the race, according to politicians.

Fairfax Commonwealth's Attorney Robert F. Horan Jr. said yesterday he, too, is actively exploring a run for the Senate but said that he has not been invited to the meeting. "I guess I'm not one of the heavies," quipped Horan, a fourth-term county prosecutor who narrowly lost a race for the House of Representatives in 1972.

The Fredericksburg summit first was intended as a meeting between Miller and Andrews but was later expanded to include the other two, according to one political aide. Thomas, an influential lobbyist with close ties to Gov.-elect Charles S. Robb, has only recently been added to the list of Democratic hopefuls although several weeks ago he advocated that the party look for a "fresh face" to run next fall.

Democrats have been slower than Republicans to narrow the field of senatorial candidates since Byrd announced Nov. 30 he would retire at the end of his term. By late last month, Rep. Paul S. Trible of Newport News had emerged as the GOP front-runner after two potentially strong candidates, Rep. Stan Parris of Woodbridge and former Fairfax state delegate Wyatt B. Durrette, took themselves out of the race.

The Democrats, triumphant after last fall's statewide elections, have been more cautious, waiting, some say, for Robb to indicate his preference. As Trible's campaign has appeared to be gaining momentum, some Democrats have sensed a greater need to get their own party moving.

"Potential candidates are talking to each other to try to sort it out," said Pickett, a state legislator, and, like Thomas, a Robb ally. "There is some thinking that it would be desirable in some way to decide on who the candidate will be without having to go through the rigors of the convention."

It is unlikely, however, that one candidate will emerge from the meeting in Fredericksburg this weekend. "I fully expect there will be a contested race for the Democratic nomination ," said one political aide yesterday. "But they're all gentlemen. They agreed to sit down and talk it over and so they will."

Neither Miller nor Andrews would comment this week on their political plans although, according to one politician, Miller already has commissioned a poll on his chances in a race against Trible. Miller has said he will make an announcement at the end of the month, after the Robb inauguration and a meeting of the Democratic Central Committee in Richmond Jan. 23.

Of the four, only Miller and Andrews have had experience at statewide races. Miller lost the Democratic gubernatorial nomination to Henry E. Howell in 1977 and the state's other U.S. Senate seat to Republican John W. Warner the following year. Andrews sought the Democratic Senate nomination then, but lost to Miller.

Two Arlington Democrats yesterday withdrew as candidates from another Democratic race, saying they will not seek the party's nomination to oppose Republican Rep. Frank Wolf. County Board Member John Milliken and Del. Warren Stambaugh took themselves out of the race, citing their desires to remain in their current elected positions.

Milliken, 36, a former aide to former representative Joseph Fisher, who held the seat, said: "I could not reconcile on the one hand making an effort to regain a Democratic majority on the County Board while at the same time threatening that by running for Congress."

The exit of Stambaugh and Milliken leaves former delegate Ira Lechner the only Democrat so far to declare for Wolf's seat. Other Democrats from the district, which includes Arlington, Loudoun, the northern half of Fairfax County as well as Fairfax City and Falls Church, are keeping their options open. State Sen. Charles Waddell (D-Loudoun), Fairfax County Supervisor Martha Pennino and former state delegate Raymond Vickery Jr. are also weighing the race, as is Democratic activist Peter Halpin, 28, Robb's former campaign secretary.