Stung and demoralized by the latest in a long series of defeats, Virginia Democrats are grouping for a way to rebuild their party without the help of its embattled leader.

Almost all Democrats officeholders and party officials blame the party's decade of defeats - including the apparent loss of senate nominee Andrew P. Miller on Nov. 7 - on a weak and underfinanced party organization.

That has focused much of the criticism on state party chairman Joseph T. Fitzpatrick of Norfolk and a central committee, dominated by moderate-liberals, who have controlled the state party since 1972.

Already under way are two steps to revitalize the party - both taken by men considered potential candidates for governor in 1981 and both without consulting Fitzpatrick, who is considered a member of the party's liberal wing.

"Had we had a better, in depth organization, we would have won the Senate race," said state Sen. Hunter B. Andrews of Hampton, who is attempting to direct a committee of state legislators rebuilding the aprty. Another effort to evaluate Miller's apparent loss to Republican John W. Warner is being directed by Lt. Gov. Charles S. Robb of McLean, another potential nominee for governor.

"We had a very fine candidate, but he was beaten, apparently beaten, by a gentleman who didn't come into the race until August and who won because he had that Republican organization behind him that turned out the vote," Andrews said in an interview.

Miller supporters are still nourishing flickering hopes that Monday's official count or a possible recount will reverge Warner's thin, unofficial margin of 5,190 votes out of 1.22 million cast. However, as the day of the official canvass approached, a deep-seated pessimism among Democrats was increasingly apparent in a chorus of private grumbling against party chairman Fitzpatrick and the central committee.

Despite the complaints of poor party organization, there appears to be little likelihood of changes in Democratic leadership before 1980. Fitzpatrick said in an interview that he fully intends to serve out his second four-year term. But he already has announced that he will not seek reelection at the state convention in the summer of 1980.

"I don't believe the obituary can be written of the Democratic Party of Virginia yet," he said the other day, adding that his critics are dead wrong if they "think that campaigns can be won in this office."

The generally conservative Democratic state legislators, who hold overwhelming majorities in both houses of the assembly have not played an influential role in party affairs since what is often characterized as the "liveral takeover" of the party by supporters of presidential nominee George M. McGovern six years ago.

The decline of the legislator influence and their tepid support of more liveral statewide candidates has been accompanied by outright defections by many Democratic supporters, including business contributions, to the Republicans.

"What we've got to do now," said one Democratic public official in Richmond recently, "is get the horses back into the party."

Democrats who hoped that the moderate-conservative Miller would succeed in wooing conservatives voters back to the party say they were dismayed by his failure to reverse recent GOP gains, especially in most of the state's populous suburbs.

Their gloom is deepened by the perception that Warner, a novice in elective politics chosen in August to replace the late Richard D. Obenshain, proved to be an inept candidate.

This perception fortifies their conclusion that the state GOP organization, first under the chairmanship of Obenshain and now under chairman George N. McMath, has outclassed the Democrats in the critical identification and courtship of voter blocs.

The Democratic disarray has put pressure on Robb, the only Democratic state officeholder and the presumptive favorite for the gubernatorial nomination in 1981, to fill the organizational void. Not all Democrats say he is doing it.

In Febraury, Robb formed a committee to improve party organization and named as its chairman former Sen. William B. Spong, now dean of the William and Mary Law School. Spong was widely admired in the party as a public official, but many Democrats regard him as a poor politician. His inattention to politics, they believe, contributed to his defeat by Republican William L. Scott in 1972.

When Spong and Robb mailed out a questionaire this week asking party officials for detailed comments on almost every aspect of the Miller campaign, some party figures dismissed it as futile wheel spining. "The heavy-weights are going to dump this right into file 13," one said.

Robb has not made any public statements about the party's future since the election. He has recently returned from a national lieutenant governor's conference in Austin, Tex., and was on a hunting trip in Maryland yesterday and not available for comment.

Fitpatrick is obviously chafed by Robb's initiatives in party affairs. "I think Chuck needs to emerge on a plane above party politics," the chairman said in an interview. "It's hard to make any friends in party politics.""

The Miller campaign questionaire, Fitzpatrick said, "seems to be directed at a critique of the campaign operations.Since Chuck was the designated Miller campaign manager, I guess he would have to regard any criticism as being directed at himself.

Fitzpatrick said he has tried "to keep both liberals and conservatives in the party by following a middle course." He is especially proud of the fact that the party has finally paid off a $150,000 debt accumulated before he became chairman.

However, his party is beset with its own dire predictions of the consequences of a Miller defeat. Miller himself said that a Democratic loss in the Senate race probably would lead to Republican gains in next year's General Assembly elections and "make it very difficult for Chuck Robb" in 1981.

"We all said we had to win this one if we are going to remain a viable political party," Andrews said yesterday. "It's elementary. A political party has to win to stay alive in politics . . . Now we have to rebuild this party from the bottom up."