When Virginia state Sen. Nathan H. Miller was accused of conflict of interest this fall, the Senate's Democratic leadership was loud in calling for an investigation of Miller, the Republican nominee for lieutenant governor.

A month later, Miller stands rejected by the state's voters, but the promised legislative probe of the charges againist him is still disorganized and confused.

Members of a bipartisan Senate subcommittee who met this week to establish procedures for the investigation of charges that Miller used his elected office to benefit clients of his Shenandoah Valley law firm found themselves stymied.

The problem is that Virginia, a state that long has prided itself on clean government, never has investigated one of its legislators for alleged wrongdoing so far as Senate leaders can remember. There are no written rules for going about it.

"The committee has a serious responsibility," said Sen. Joseph V. Gartlan (D-Fairfax), "and it's going to necessitate careful development of procedures so that when people are involved and the public is involved, they will think we have acted fairly."

The rules the subcommittee sets for conducting its investigation are likely to establish the scope of the probe and, legislators say, could influence its outcome. For example, subcommittee members must decide whether they are restricted to investigating only formal charges lodged by senators against other senators, or whether they may look into newspaper reports and complaints from private citizens.

With conflict of interest looming as one of the next legislative session's hottest issues, the five members from the Senate Privileges and Elections Committee huddled behind closed doors to ask legal advice on how to conduct the investigation. Under Virginia law, governmental officials are allowed to hold private meetings to discuss legal matters with their lawyers.

Sen. Howard P. Anderson (D-Halifax), white-haired, courtly chairman of the group, said the panel's unanimous decision to close the meeting did not indicate that the group will bar the public and press from hearing any testimony against Miller or others.

Still undecided is whether the subcommittee will investigate unresolved conflict allegations against some Democratic legislative leaders. Those charges are likely to make the Democrats move very cautiously in the investigations, some legislators say.

Republicans complained bitterly last month when a Rules Committee hearing on the Miller conflict charges was chaired by Sen. Willard J. Moody (D-Portsmouth), who twice has been accused in legislative proceedings of using his elected office to help his private legal clients. Without acknowledging that Moody presented the Democratic leadership with a problem, Senate Majority Leader Hunter B. Andrews effectively limited Moody's role in the dispute by shifting major responsibility for the Miller probe away from Moody's Rules Committee to the five-member P&E subcommittee.

Republicans in the Senate say they will insist that whatever committee examines Miller the probe must be evenhanded. "If they go after Senator Natie Miller in any vindictive sense, the fur's going to fly," said Sen. Ray L. Garland (R-Roanoke.) "Then we may have to start swapping allegations."

Regardless of the outcome of the investigation, Democratic and Republican Senate leaders say they intend to tighten the state's ethics laws.

"I think it is almost a certainty that the Senate will tighten its rules," says Sen. Wiley F. Mitchell (R-Alexandria). "There is a great deal of frustration on the part of many people who are concerned that no matter how hard they try to abide by the rules of the Senate, those rules are so ambiguous that they might get caught in a trap not of their own device."