Former Virginia attorney general Andrew P. Miller won the Democratic nomination for the U.S. Senate last night after his seven opponents conceded failure in their efforts to block his selection.

Miller, 45, who was denied his party's nomination for governor in a bitter primary last year, had come within 37 votes of winning the Senate nomination when his opponents gave up the race at the party's two-day convention here.

The outcome will pit Miller this fall against Republican Richard D. Obenshain, 42, a conservative Richmond lawyer, whom Miller beat handily in 1969 when he launched his political career by becomong the state's chief legal officer.

While Miller's acceptance speech was optimistic, most party leaders agree that Miller will face a much more difficult race against Obenshain this November than he did nine years ago. Virginia Democrats have not won a Senate seat or governor's race in the last 10 years - a record unequaled by Democrats in any other state in the country.

In accepting the party's nomination, Miller signaled the opening of an acerbic campaign with scornfully sarcastic comments aimed at the GOP nominee and his party.

Labeling Obenshain an extremist" given to "irresponsible rhetoric," Miller ridiculed the Republican's description of himself as a "Jeffersonian Democrat with a small d."

"The only thing that Thomas Jefferson and Dick Obenshain have in common is that they both live in the 18th century," Miller said. "Mr. Jefferson belonged there. But Mr. Obenshain is just out of step with the millions of Virginians who are living their lives today."

Miller's sharp attack on Obenshain and Sen. William L. Scott, the Republican incumbent whose retirement is opening the Virginia Senate seat, was the last in a series of similar assaults made by other candidates and the speakers who nominated them thoughout the day.

Former state delegate Ira M. Lachner of Arlington, who nominated state Sen. Clive L. DuVal II of McLean, told the delegates, "We have an opportunity this fall to replace a certified dummy with a senator Virginia can be proud of."

DuVal was trailing Miller by 784 votes when the race ended. Both he and Miller made a bid in their speeches to the convention to embrace moderate Republicans, especially followers of former governor Lindwood Holton. DuVal said that when the Republican convention last week rejected Holton and nominated Obenshain, "the old elephant turned into a dinasaur."

Miller is known to be among those who as state officials in Richmond regarded Holton as a weak and often intemperate administrator; but he said last night: "Lindwood Holton was a good governor and he is a good man. I invite the homeless "Holton Republicans" to join us. There's plenty of room in the mainstream."

Obenshain was state Republican chairman when Scott upset Democratic former senator William B. Spong in 1972. Miller said that he will make that a major campaign point of attack. "He (Obenshain) is the man who gave us Bill Scott, and we're never going to let him forget it," he said to the cheering delegates.

Six hours after the balloting began in William and Mary Hall here, Miller accepted the nomination under the eyes of his wife and three children, and of his parents, retired Col. and Mrs. Francis Pickens Miller. The elder Miller, 83, ran unsuccessfully for the Democratic nomination to the Senate in 1952 against the late Senator Harry F. Byrd Sr., founder of the state's dominant, conservative Byrd organization.

The younger Miller's victory became inevitable last night after it became clear that his two closest rivals, DuVal and born-again Christian G. Conoly Phillips, were not able to increase their delegate totals.

As the third rool call was being tabulated, DuVal, Phillips and three other trailing candidates met in a search for an alternative candidate to Miller. "It was clear that our supporters could not be directed ecept for the possible exception of Conaly Phillips'," DuVal said in an interview. Phillips concurred. "I could have directed my votes," he said. "But the others could not."

The unusual joint withdrawal came, campaign sources said, after evangelical talk show host Pat Robertson, a close adviser of Phillips, and Paul Goldman, an unofficial adviser to candidate Hunter Andrews, agreed that no one candidate should be credited with giving Miller the nomination.

The decision to make the vote unanimous was made in a mobile van parked a front of the hall. While votes for the third ballot were being tabulated, five of the seven candidates who trailed Miller all night - DuVal, Conoly Phillips, Rufus Phillips, Carrington Williams and Andrews - gathered on the steps of the hall to, as DuVal put it, "decide on our options."

DuVal said it was quickly decided that if they - one or two of them - withdrew Miller would go over the top on the next ballot, or perhaps after two more. The next option was for the also-rans to unite behind one of their number, but again, DuVal said, "we felt our supporters couldn't be directed."

The remaining option, which was hashed out as the five men moved from the steps to the trailer of a DuVal supporter, was to go to the platform together and call for a unanimous nomination of Miller, even though he had fallen short of the needed majority on the third ballot.

The losing candidates who were not involved in the decision were Flora Crater and Frederick Babson. They were left out "because we couldn't find them," DuVal said.

Crater and Babson joined the others at the platform, and made the decision unanimous, although Crater expressed irritation that she had not been consulted. She said the next time party leaders should remember to "include the women" in their decision-making.

It was 7:57 p.m., according to Miller's wristwatch, when Andrews, selected by alphabetical order, made the motion that gave the victory to Miller.

Phillips, a Norfolk city council member, told the convention, "The church community is rising as it has never risen before in the area of politics, and this fall the people of this community will all be Democratic votes."

Phillips, an automobile dealer and a moderate-conservative in the party political spectrum, said his nomination would revive the flow of contributions to Democrats from business and provide "money, workers and votes" from the churches across the state.

Both Miller and Conoly Phillips supporters staged rallies here before the nominating session began. About 700 miller delegates massed in the College of William and Mary football stadium and then marched on the convention hall. Phillips mustered 400 supporters at a prayer breakfast. they sang hymns as they waited to enter the meeting room and after breakfast heard Robertson preach and Phillips and his wife Betsy speak.

Before the nominating speeches began, the convention was addressed briefly by Rep. Herbert E. Harris of Northern Virginia's 8th District and by former lieutenant governor Henry E. Howell, who was soundly defeated last year as the party's candidate for governor.

Howell, as expected, endorsed DuVal, a long time ally in his populist causes if not an initator of Howell's wheeling style. Even with the support of Howell, however, the soft-spoken DuVal failed to attract delegates in the numbers that he and some of his opponents expected.

Only 432 delegates were committed to vote for him on the first ballot after the April 15 city and country delegate elections and after that he failed to pick up uncommitted votes at the pace he predicted.

The contrast between the Democratic convention yesterday and the Republican meeting a week ago in the Richmond Coliseum demonstrated anew how the balance of power in statewide politics has shifted in recent years.

As recently as decade ago Republican nominations to statewide office were hardly contested and were handed out to nominees chosen at conventions attended by a small core of party faithful. Last week the Republicans staged the largest state politic convention ever held in the United State. A total of 7,800 delegates and alternates, attracted by four vigorous campaigns, named Obenshain by a narrow margin over former Navy secretary John Warner.

About 4,500 people - 4,000 of them delegates and alternates - attended the Democratic convention yesterday. Virginia Democrats traditionally have nominated candidates in party primaries, but turned to a convention for this race to save money for their candidates and reduce the intraparty strife that has accompanied recent primaries.

Although Miller came out of the April 15 delegate elections with a commanding pluraltiy in the field of eight, his opponents united against him in a fight over convention rules. They denied it was a stop-Miller movement, but they finally succeeded the week before the convention in forcing Miller to accept rules that made it more difficult for hm to win the nomination.