Andrew P. Miller, front-runner for the Democratic nomination to the U.S. senate in Virginia, suffered a tactital setback yesterday as he and the seven other candidates for the nomination agreed to state convention rules that improve the chances of his opponents.

The compromise worked out in a meeting with state party chairman Joseph T. Fitzpatrick will permit all candidates to remain in the race through four ballots and will require the winner to obtain a majority of all delegates authorized to vote whether all vote or not.

The candidates also agreed to require voting by secret ballot. Some of Miller's challengers wanted a secret vote because they fear some supporters may be reluctant to oppose Miller under the eyes of local party leaders loyal to him.

Miller expects to receive about 1,200 votes on the first ballot of the nominating convention Saturday in Williamsburg, about 200 short of the majority of 1,399 that he needs to become the nominee.

Eversince the April 15 delegate elections, Miller has resisted efforts by his seven opponents to overturn temporary convention rules that they thought would help him. They objected to a rule requiring the low candidate to drop off after each ballot and to another permitting the nominee to be elected by a majority of those voting, rather than a majority of the entire convention.

Miller finally offered last week to negotiate with his opponents, who apparently would commanded a majority of votes in any rules fight on the convention floor.

The result of yesterday's negotiates will require the low candidate on the fifth and successive ballots to drop out. Miller's opponents said in interviews they feel this will allow plenty of time for their supporters to consolidate behind Miller's strongest challengers.

Miller told reporters after the rules agreement was reached that he does not view the results as a setback for him. "It is a victory for party unity and party unity is what I have been talking about throughout this campaign," he said.

Miller is a former attorney general who is viewed as a moderate-conservative in his party. His closest rival in terms of delegates commited to him on the first ballot is state Sen. Clive L. DuVal 11 of McLaen, a moderate-liberal.

The other candidates, listed in the order of their commited delegates, are Norfolk City Council member G. Conoly Phillips, former Fairfax County Supervisor Rufus Phillips, former State. Del. Carrington Williams of Fairfax, state Sen. Hunter B. Andrews of Hampton, former Fairfax Supervisor Frederick Babson and Fa lls Church feminist Flora Crater.