Virginia Republicans, never before faced with a significant party contest for a U.S. Senate nomination, began gathering here last night to decide a four-way Senate race whose outcome few party leaders are willing to predict.

Former GOP national cochairman Richard D. Obenshain is the acknowledged front runner as the nominating convention opens today. But former governor Linwood Holton and former Navy secretary John Warner are both thought to be within striking distance of the nomination, which will be decided Saturday afternoon.

Only the fourth candidate, state Sen. Nathan H. Miller of Rockingham County, is considered out of the running, but he continues to say that a deadlocked convention will turn to him as a compromise candidate.

The four candidates are seeking the Senate seat being vacated by the retirement of Republican William L. Scott. Six years ago, Scott had the GOP nomination for the asking because Democratic incumbent William B. Spong was considered unbeatable. Scott ran a media-oriented campaign painting Spong as a liberal and won a decisive upset victory.

Virginia Democrats will choose their Senate nominee at a Williamsburg convention June 10 from a field of eight candidates. They are former attorney general Andrew P. Miller, state Sen. Clive L. DuVal II of McLean, Norfolk City Council member G. Conoly Phillips, former Fairfax County supervisor Rufus Phillips, former state Del. Carrington Williams of Fairfax, state Sen. Hunter B. Andrews of Hampton, former Fairfax supervisor Frederick Babson and Falls Church feminist Flora Crater. Miller has a commanding lead in committed delegates to the Democratic convention, but apparently lacks a majority at this point.

Much of the uncertainty about the outcome of the Republican convention stems from the unusually large number of delegates and alternates - about 8,500 - expected to attend as a result of liberal delegate allocations and vigorous preconvention campaigns.

Obenshain, state party chairman from 1972 to 1974, has the image of a master of conventions and the loyalty of many influential city and county chairmen. However, a majority of the delegates now descending on this city have never attended a Republic convention and are thought unlikely to be as responsive as past convention delegates to the influence of party regulars.

At last year's GOP state convention, Obenshain and other members of the Republican establishment failed to put across their choices for attorney general and lieutenant governor, a failure that some say may have signaled a new attitude of independence among party rank and file.

When the insurgent candidates won at the GOP convention last year, an influential party official pointed to Warner and said, "He's the happiest guy here. He knows now that he has a shot at the Senate over Obenshain."

Although moderate Holton and conservative Obenshain have in the past been on opposite sides of party fights, many campaign officials here yesterday said that they now share the loyalties of party regulars, who view Warner as the insurgent outsider.

Hours of meetings between party leaders and representatives of the four campaigns have apparently resolved all potential disputes over convention rules and delegate credentials except for a dispute over whether voting will be secret ballot.

Warner's convention manager, William Tucker, has been asking for a secret ballot in the belief that many delegates who want to vote for Warner may be reluctant to do so publicly under the eyes of party regulars from their city or county who are Obenshain loyalists.

As of last night, the Holton managers had refused to join Warner in seeking a secret ballot. "You don't worry about a secret ballot unless you are an outsider. Holton manager George Cook of Alexandria said at a press conference.

However, Warner may get the secret ballots he wants because party chairman George N. McMath said yesterday he has urged city and county chairmen to use them to speed roll call tallies.

Party leaders are predicting that it will take from two to five ballots to decide the nominations. McMath hopes to complete each ballot within an hour. The balloting is expected to begin about 1:30 p.m. tomorrow.