Virginia Republicans are increasingly frustrated and at odds over strategies to attack state Sen. L. Douglas Wilder, the Democratic candidate for lieutenant governor and first black to be nominated by a major party for statewide office.

Some of the GOP leaders are particularly upset that charges against Wilder largely have been led by former governor Mills E. Godwin and former senator Harry F. Byrd, once prominent segregationists whose backgrounds have muddied their message against Wilder.

Knowledgeable Republicans said today former GOP governor John N. Dalton, who is traveling in Europe, specifically rejected a request last week from the campaign of Wyatt B. Durrette, the GOP candidate for governor, that Dalton take over as point man for party attacks on Wilder to cast him as a liberal.

"The reaction to Godwin and Byrd has had a chilling effect," said one Republican familiar with Dalton's decision. "Rightly or wrongly, the campaigns are now perceived as a Mills Godwin . . . production."

Wilder, a state senator from Richmond, began the race as a distinct underdog but according to recent polls has moved out in front of his opponent, Republican state Sen. John H. Chichester of Stafford County.

"The Republicans for years have shouted, 'liberal,' 'liberal,' 'liberal,' and now they've got one, they don't know what to do with him," said Larry J. Sabato, a University of Virginia political scientist and longtime student of Virginia elections.

Sabato, who last year called Wilder a "100-to-1 shot," said Wilder could score a historic victory because of his exceptional campaign, largely favorable media coverage and Republican missteps by emphasizing Godwin and Byrd.

The Republican campaigns are troubled "because of race; they didn't know how to handle it ," Sabato said. "Wyatt Durrette doesn't have a racist bone in his body. The same can't be said for Byrd or Godwin."

Godwin, whose racially tinged remarks in the 1981 gubernatorial campaign were considered costly to the GOP nominee that year, has twice criticized Wilder this year, and each time he has found himself mired in controversy.

Several weeks ago, he complained that Wilder, among other issues, once proposed repeal in 1970 of the state song, "Carry Me Back to Old Virginia." The song has references to "massa" and "darkey" and is considered offensive to blacks.

Last week, Godwin charged that the Democrats "were trying to hoodwink" the voters by keeping Wilder's liberal record a secret and vowed to step up his criticism of Wilder.

At the same time, Byrd remarked that any lieutenant governor is "a heartbeat away" from becoming governor, a remark that was considered by some to be racially insensitive.

Sabato said the Republicans have frittered away chances to criticizeWilder for his late payment of some taxes, slum properties in Richmond and a 1978 reprimand before the Virginia Supreme Court over his handling of a minor 1966 traffic case.

"That's enough to sink a battleship. A white candidate would have been finished months ago," said Sabato.

He noted that the Republican National Committee ran a newspaper ad that accused Wilder of being soft on spouse abuse. The ad failed to mention Chichester and was considered ineffective by many.

"They've got nuclear bombs and they're using BBs," Sabato said.

Some Republicans, including Durrette and Chichester's campaign manager, Dennis Peterson, have complained that the news media brands any attack on Wilder as racist.

"I think if any of us breathed and Sen. Wilder's name was in the breath, you in the press would interpret it as racist," Durrette told reporters recently.

Earlier this year, Durrette called Wilder "one of the most liberal" candidates ever to seek the state's number two office.

In a heavily publicized response, Wilder challenged Durrette by comparing their records in the General Assembly and said Durrette's comments could be interpreted as "racist," which Durrette strongly denied. Durrette has largely left such comments now to Godwin and Byrd.

Chichester has tried repeatedly to raise questions about Wilder's voting record, only to have Wilder respond by pointing out that Durrette, or other prominent Republicans had at one time voted the same way.

Wilder has repeatedly accused Chichester of "mudslinging" and "distortion" but has not made any recent charges of racism against his opponent. "We have not mentioned it," said Paul Goldman, Wilder's campaign manager.

In addition to Dalton, other more moderate leaders have taken low profiles in the campaign. Former governor Linwood Holton of McLean, who has never been comfortable with the Godwin wing of the party, declined to discuss the campaign this week.

Former representative M. Caldwell Butler of Roanoke, who gained fame during Watergate by supporting President Nixon's impeachment, also has stayed away from the campaign.

Butler said today that he hasn't "had the time" to campaign and that he has not been asked. Butler said he is supporting the GOP ticket.

Some Republicans also said GOP Sen. John W. Warner also has shied away from suggestions that the Republican delegation to Congress attack the Democratic ticket.

"I suspect those persons know better than to ask Warner because they know the answer would be 'no,' " said Andy Walhquist, Warner's administrative assistant. "It's not his style to attack anybody in the opposition."

On Tuesday night, Wilder himself managed to keep the "liberal" issue alive.

Bounding up to a podium in a joint appearance with Chichester before the Richmond Jaycees last night, Wilder was all set to reject Chichester's criticism of him as a liberal.

"I'm a human," Wilder began. "I am not a conservative. I am not a moderate. I am a liberal."

Guffaws rippled through the restaurant as Wilder caught his mistake. "I am not a liberal," he said with embarrassed laughter. "Talk about putting your foot in it!"