The hair's-breadth margin in Virginia's U.S. Senate election demonstrated anew the Republican hold on the state's moderate-conservative voter, election analysts agreed yesterday, but it also showed that a Democrat under certain circumstances can loosen that grip.

Republican John W. Warner and Democrat Andrew P. Miller emerged from the closest general election in Virginia's short era of two-party competition separated by about 0.3 percent of the 1.22 million votes cast. Warner had a tiny lead in unofficial returns, but that could be changed by an official canvass.

Larry Sabato, who has analyzed state election results for the University of Virginia's Institute of Government for 10 years, said the performance by Warner, win or lose, is another demonstration "that the symbols of conservatism have been transfered from the old Byrd Democrats to the Republican Party."

However, he said, Miller's performance - after a decade of Democratic defeats in races for governor and senator - shows that a Democrat still can make inroads on the conservative vote.

"The composite Virginia voter perceives himself to be a moderate-conservative," Sabato said, "and he perceived Miller to be on about the same spot on the ideological spectrum.

"For the first time in 10 years., there was a split in the conservative coalition that has elected Republicans in Virginia. There were confusng signals going out from influential conservatives and you saw it reflected in a split of the conservative suburban vote between the candidates."

Sabato said the split in the suburbs across the state caused him to overestimate the size of the Warner margin even though he had accurately forecast the turnout, which was larger than most campaign officials expected.

"Virginia is unlike the rest of the nation when it comes to the effect of a large turnout," he said. "In a heavy Virginia turnout, the additional vote comes from the generally conservative suburbs and should help the Republican candidate. We saw that happen last year in the governor's care" when Republican John N. Dalton beat populist Democrat Henry E. Howell with almost 56 percent of 1.25 million votes." Nationally, large voter turnouts tend to favor democrats, Sabato said.

The suburban split in the Senate race yesterday produced comfortable margins for Miller in Northern Virginia and in Albemarle County, which surrounds Charlottesville, and large Warner margins in Virginia Beach and the counties around Richmond. While Sabato pins these results on "confusing signals" from conservative leaders, some Republicans have another explanation.

"As far as I'm concerned, Northern Virginia was the whole story in this election," said one GOP official analyzing the closeness the statewide vote. "We were ahead by six points in Northern Virginia going into the last week and then the candidate (Warner) had a memory lapse about those Nixon contributious and we dropped like a rock."

Warner had claimed in interviews that he did not contribute to former President Richard Nixon's reelection campaign in 1972 when Warner was undersecretary of the Navy. However, it was disclosed that he made contributions of $5,000 and that his former father-in-law Paul Mellon, and other Mellon family members were major Nixon contributors.

Whatever, the reason, the Washington suburbs, which have delivered at least a standoff for recent Republican candidates, went for Miller by about 10,000 votes.

Raymond Colley, Democratic chairman in Northern Virginia's 8th Congressional District, said he believes that vigorous House races won by incumbent Democrats Herbert E. Harris II in the 8th and Joseph L. Fisher in the 10th District, helped Miller in the Washington suburbs.

"It was a piggback effect," he said. "Workers who were doing something for Joe and herb were doing the same for Andy. There were joint phone banks, mailouts and leaflet distribution."

From the Miller campaign, the analysis of top aide Timothy W. Finchem was that his candidate had succeeded "in moving the image of the party back to where it should be." Finchem said this meant appealing to moderate-conservatives without alienating black voters, labor and liberal Democrats.

If the unofficial Warner lead holds up Finchem said, the Miller loss would have to be blamed primarily on a larger than expected Warner margin in the Richmond area, where the city's daily newspaper editorialized heavily for the Republican and against the Democrat.

"We lost that media market by 30,000 when it should have 20,000," Finchem said. Finchem also said the Miller margin of 4,250 in the 9th District of Southern Virginia, where the Democratic candidate once practiced law, was a disappointment, as was the heavy Warner margin in Virginia Beach and among white voters in Norfolk.

Finchem theorized that Warner's service as secretary of the Navy helped him among both active and retired Navy personnel living in Virginia Beach.

Despite fears that Miller would not inspire a heavy turnout of black voters. Sabato estimated that the black vote probably came close to the 9 percent of the total vote that turned out for Howell in the 1977 race for governor.

Sabato and a CBS election analysis put Miller's share of the black vote at 85 to 90 percent. If the official canvass finally does give the election to Miller by a small margin, black Miller voters will have a claim to a decisive role in his victory.

Howell, the former lieutenant governor who beat Miller in last year's Democratic primary for governor, generally praised the Miller campaign, in which he was given no formal role.

"To get more than 600,000 votes, he obviously did more things right than he did wrong," Howell said yesterday.

Howell also said that Miller's decision to exclude President Carter from his campaign probably was right in view of Carter's standing in Virginia polls. "Virginia is a very, very conservative state," he said. "The president is doing a tremendous job, but he doesn't seem appreciated here."