Democrat Charles S. Robb was elected governor of Virginia yesterday, defeating Republican J. Marshall Coleman and leading his party to a sweep of the state's three top offices. Robb overcame a determined effort by President Reagan on Coleman's behalf, ending 12 years of GOP rule in the state.

Robb, the conservative son-in-law of the late President Lyndon B. Johnson, spearheaded a remarkably strong coalition of traditional Democrats and conservative independents in winning nine of the state's 10 congressional districts. With all but 2 percent of the vote counted, Robb was beating Coleman by nearly 54 to 46 percent to become the first Democrat to win the statehouse since 1965.

Robb's stunning victory was matched by his two Democratic running mates. Richard J. Davis overwhelmed Republican Nathan H. Miller in the lieutenant governor's race by 56 to 44 percent, and Democrat Gerald L. Baliles won a narrow victory over favored GOP opponent Wyatt B. Durrette Jr. in the state attorney general's race.

Robb went before his cheering supporters at the John Marshall Hotel in Richmond at 12:30 a.m. today, praising the support he got from teachers, blacks, union members and conservatives. "This is our new beginning," Robb told the jubilant crowd, which included his wife, Lynda Bird, their three children, his mother-in-law, Lady Bird Johnson, sister-in-law Luci Johnson, and his parents, James and Frances Robb.

"How sweet it is," he cooed, reporting that Coleman had called to concede. He praised his opponent as "one whale of a campaigner and competitor."

Coleman went before television cameras 10 minutes earlier to concede. He predicted to a dejected crowd of campaign workers that the Republican Party would return to power in four years, retaking the governor's mansion in 1985. "I think that all of us in this room are committed to the cause, the cause that will continue despite this setback we have suffered tonight."

With his wife Niki at his side, Coleman first attempted to joke away his defeat. He said he had delayed his statement, believing that the early returns had transposed his vote totals with those of Robb.

Earlier Coleman's campaign workers began drifting away from the Hyatt House headquarters in suburban Richmond, many of them crying as the television reports confirmed their fears. "Two things beat Coleman," said one of the depressed volunteers, "the weather and Nathan Miller."

State Sen. Herbert Bateman, who had been the choice of party leaders for the spot won by Miller at a GOP convention, said Coleman had lost the battle of the media. "It's not that Virginians had changed their minds about what they believe in," he said, noting that the GOP did well in legislative races, picking up at least four seats. "I don't think the Republican Party has embarrassed itself." But the clear feeling in the banquet room was that the Republicans were not used to losing.

The Democratic triumph was a major rebuff for President Reagan, who put his political prestige on the line by making a personal appearance on Coleman's behalf and appearing in a Coleman television ad. The result was also a major blow to Republican Gov. John N. Dalton, who campaigned vigorously for the Republican nominee.

Republicans had counted on Reagan's appearance in Richmond a week ago to turn the tide for Coleman. But Robb's supporters said the presidential visit had galvanized blacks and other Democrats fearful of Reagan's budget-cutting programs.

"I'm glad he Reagan came," said former Fairfax Democratic chairman Emilie Miller. "We ought to send him a thank-you note."

Republican leaders, in the unaccustomed role of graceful losers, took heart in the substantial gains they won in the 100-member Virginia House of Delegates. They said returns showed the GOP capturing eight new seats, bringing their total to 33.

Robb rolled up margins not only in traditional Democratic strongholds but also in areas that Republicans have dominated in recent years. In the Northern Virginia suburbs, which gave Reagan a majority of landslide proportions last year, Robb took 54 percent of the vote.

Robb captured by a narrow margin the Richmond area's ultra-conservative 3rd District, thanks to a heavy turnout of black voters in the city. He also broke even with Coleman in traditionally Republican Virginia Beach.

Glenn Croshaw, a Virginia Beach lawyer who chaired the joint Democratic campaign, raised his glass at Robb headquarters and toasted "Virginia Beach -- that great Democratic city by the sea."

Coleman, the state attorney general, won only the 7th Congressional District, his home base in the Shenandoah Valley. Robb even took the 6th District, the heart of the state's Republican vote.

Final returns in Fairfax County, which in recent elections gave large majorities to Republicans, showed Robb beating Coleman 52 to 48. The totals for the 161 precincts that make up the state's most populous county showed Robb 91,438, Coleman 84,939. The Democratic tide in Fairfax also appeared to have bumped two incumbent Republican legislators. Durrette, who lives in Vienna, however, thumped Baliles in Fairfax County 58 to 42 percent, piling up a 28,000-vote margin.

Robb's strength in Northern Virginia carried beyond the normal Democratic inner suburbs. While he carried Arlington County and Alexandria, he also beat Coleman by a small margin in Prince William County. Coleman carried Loudoun County.

Davis, 60, a former Portsmouth mayor, was attempting to take advantage of conflict-of-interest allegations against Miller in making integrity a theme in his first campaign for statewide office.

Durrette, 43, a former member of the state legislature from Fairfax County, was alone among candidates for the top offices in attracting solid conservative support.

Both gubernatorial nominees voted early yesterday, Coleman in The Fan section of Richmond, and Robb at Langley High School in McLean. Then both men set out for a final round of handshaking, including one last stop at what has become a politicians' favorite, Metro stations in Northern Virginia.

Yesterday's voting climaxed an eight-month-long campaign that grew increasingly bitter, personal and mud-stained as it progressed. Robb and Coleman, dubbed "Tweedledum and Tweedledee" by critics who could discern little or no difference between them on important issues, seemed intent on establishing themselves as cautious conservatives.

Absent of real philosophical differences, Coleman and Robb each hired a nationally known political consultant, and the campaign turned into a battle of style over substance. Before the last radio and television ads were played last night, the rival campaigns had spent a record $4.5 million, with Coleman outspending Robb by about $380,000.

Nearly half of it financed media campaigns that were roundly denounced by the public and for which both candidates privately apologized. They contended the content of the harsher commercials was somehow beyond their control.

By fall, each candidate was devoting as much time attacking the other as promoting himself. Coleman portrayed Robb as a closet liberal, a son of President Lyndon Johnson's Great Society who, if elected, would emulate the big-spending policies of his famous father-in-law. Over and over, Coleman hammered the point that Robb, despite his own professed conservatism, belonged to the same political party as Walter Mondale, Edward M. Kennedy, Hubert H. Humphrey and other liberals.

Part of Robb's strategy was to make the brash, youthful Coleman the issue. Robb lashed out early, with a series of controversial radio ads painting Coleman as soft on drug pushers and flip-flopping on tax issues, and mocking his uniform sentencing proposal for convicted felons as "uniformly misleading."

Robb's attack succeeded in putting Coleman on the defensive. The Republican was especially enraged by Robb's drug ad, which led Coleman to respond with an ad defending his record and asking whimsically "what Robb's been smoking." That phrase, perhaps the most memorable of the campaign, hurt Coleman even further.

Coleman sought to make the election a referendum on the policies of the Reagan administration, and at the start, he had help from Robb, who was unwilling to challenge the president on any of his proposed budget cuts. As complaints arose from the poor and elderly about the cutbacks, Robb timidly withdrew his support of the Republican plan.

Although Coleman ominously warned about the infusion of out-of-state money in the campaign, it was Coleman, not Robb, who went to Texas for two fund-raisers, a move that Robb strategists said helped defuse the Texas connection.

By late September, all the published polls showed Coleman trailing by 5 to 11 points and every new event seemed to go Robb's way. Reagan had to postpone a long-awaited personal appearance at a Coleman fund-raiser, sending wife Nancy instead. Miller, Coleman's running mate, was subject to conflict of interest charges. So was state highway commissioner William Wrench, a member of Coleman's campaign finance committee. Wrench was forced to resign his post after it was revealed he had proposed rerouting the Springfield Bypass in Fairfax County near some of his land holdings.

President Reagan did make it to Virginia, speaking at a rally in Richmond a week ago last night. He told a cheering partisan crowd it "does not good to clean up Washington if we don't eelect the right kind of officials at the state level."