ONCE UPON A TIME, the president decided who should be nominated to be federal judges, and the Senate confirmed or rejected his choices, just as the Constitution says it should be done. But that was before the full flowering of a custom that came to be called senatorial "courtesy," a you-scratch-my-back-I'll-scratch-yours arrangement that effectively conferred upon individual senators the power to thwart nominations to judgeships in their home state. Sen. X, that is to say, would simply indicate objection, on whatever grounds, and his fellows would respect it. Inevitably, senatorial "courtesy" was transformed, as a practical matter, into senatorial nomination, a system quite the reverse of the one established by the Constitution. Since Sen. X was going to have the last word, in any case, it became simpler to let him do the nominating, with the president, acting like a clerk, accepting the senator's choice as his own.

When Jimmy Carter came to town, he set out to regain some of the power his predecessors had yielded. He argued that better judges would be chosen if citizen nominating commissions sent the president a public list of qualified judicial candidates, if he made his choice from that list, and if the Senate then did only its constitutional advisory duty by ratifying or rejecting it.

President Carter has succeeded in putting such commissions to work in selecting all of the 35 new appellate judges and about half of the 117 new district court judges. He appoints the members of the appellate commissions; the senators involved appoint the district court commissions. That, at any rate, is the theory. We have, in the immediate vicinity, two examples of how it works in practice.

In Maryland, Sen. Paul Sarbanes has chosen not to participate in the commission system. He plans to pick the state's two new judges himself. While he has made some changes in the old system under which he would have picked the judges in secret, he has changed little that is fundamental. His office is encouraging the public to give him advice about the qualities a judge should have and to recommend specific candidates. But he - alone and without making public the names of those he is considering - will decide who gets the judgeships.

Let us stipulate that Sen. Sarbanes will make a conscientious effort to select good judges. He may well come up with nominees who are as good as those who might have been picked under the Carter system. But by sticking with the old senatorial nominating system, he is only reinforcing the idea that judgeships are used primarily to repay political debts.

The reason Sen. Sarbanes has advanced for refusing to join the Carter plan is that the Constitution gives him the responsibility of advising the president about who should be on the federal bench. But that is just the point: He is not advising the president, which the commission system would permit him to do.He is telling the president what the decision should be.

In Virginia, on the other hand, Sen. Harry Byrd Jr. decided to go along with the Carter plan and is now suffering from one of its frailties. His commission produced a list composed exclusively of white males to fill the state's four new judgeships. The administration, which wants to increase the number of female and black judges, finds it hard to believe that Virginia has only white males who are qualified to be judges.

Whether the absence of women and blacks from the commission's list is a gross error of omission or a commentary on the state of Virginia is unclear. But one of the prime sources of the new federal judges - the state court system - is remarkably barren in that state. No woman and only two blacks have served in this century as judges of Virginia's courts of record. Few, if any, other states are in that situation. North Carolina, for example, had a female supreme court justice two decades ago.

The commission system, in this instance, has created a problem that might not otherwise have arisen. If the administration has found female or black judicial candidates in Virginia, it must now persuade Sen. Byrd to ask his commission to reconsider its list. That's embarrassing to him and the commission. Under the previous system, in which Sen. Byrd himself would have been doing the nominating, the administration could have negotiated with him before any names became public.

The Carter plan isn't perfect; problems will arise under it. The real test will be the quality of the judges it produces. But it is having a substantial impact in the states where senators have accepted it - and at least some effect in others, like Maryland, where its existence is forcing the senators who reject it to defend what they are doing and to make at least some improvement on their past practices.