Donald Trump said in a Wall Street Journal interview that U.S. District Judge Gonzalo Curiel — who is hearing a case against Trump University — had an “inherent conflict of interest” because the judge is “of Mexican heritage” and Trump is urging “building a wall” to keep out illegal immigrants from Mexico (and other places).

But the legal system does not view this as a “conflict of interest” that requires a judge to recuse himself, or even counsels in favor of recusal. First, a judge’s race, ethnicity, sex and the like aren’t grounds for recusal, even if the case directly involves questions that relate to one of those factors. Neither black nor white judges, for instance, have to recuse themselves from cases that involve alleged race discrimination by whites against blacks (or blacks against whites). Hispanic judges don’t have to recuse themselves from immigration cases that involve Hispanics or from cases that involve constitutional law, administrative law or immigration law questions that especially bear on Hispanic immigrants.

And that’s so even when the cases directly involve issues that relate to race, ethnicity and the like. It is therefore likewise so when the cases are on neutral contract or tort questions, but the parties (or their owners or managers) have spoken out on controversial identity-related issues. You can’t get a Hispanic judge recused just by speaking out on immigration in a way that may disproportionately displease Hispanics — just as you can’t get a white male judge recused just by criticizing white males or calling for reparations for slavery or race- and sex-based affirmative action.

Now might judges dislike — or like — certain litigants because of the litigants’ politics and how those politics bear on various groups to which the judges belong? You bet, because before anything else, judges are people. And there are indeed some rules that are aimed at preventing such judicial bias, for instance when the parties to the case are the judge’s relatives or when the judge owns stock in one of the parties. Some of these rules apply in some cases because of the possible “appearance of impropriety” that certain relationships between judges and litigants can create.

But most such possibilities for judicial bias, and most things that may appear to create a risk of bias, do not justify recusal, for several (partly related) reasons:

1. Especially when it comes to political and identity issues, most judges would have some possible bias. Liberal judges may be biased against conservative litigants, but conservative ones may be biased in favor of them. Pro-immigration-from-Mexico judges (who may be disproportionately, though by no means solely, Mexican American) may be biased against Trump, but anti-immigration-from-Mexico judges (who would likely be disproportionately, though by no means solely, non-Mexican American) may be biased in favor. To the extent black judges are more likely to sympathize with black plaintiffs suing white defendants under anti-discrimination laws, white judges are more likely to sympathize with the defendants (again, reflecting the simple reality that some such racial sympathies do exist in some measure).

2. Judges, no less than people in other jobs, are supposed to be treated equally without regard to race, sex, ethnicity, religion and the like. Concluding that Hispanic judges can’t sit in Trump cases because of their ethnicity or that Catholic judges can’t sit in abortion cases because of their religion or that female judges can’t sit in abortion cases because of their sex would wrongly deny those judges an opportunity to have a say in those cases. And that’s especially wrong since, as #1 points out, judges from groups that form a majority on the bench can’t be equally recused in those cases because then there’d be too few judges eligible to hear the cases.

3. If all it takes to get a judge thrown off the case is to say things that the judge may dislike, litigants will find it easy to manipulate their cases just by saying things. Trump, of course, has his own political reasons for saying what he’s saying about immigration — but if Curiel has to recuse himself in this case, then the next litigant who doesn’t like Curiel’s rulings would likewise be able to get a different judge simply by echoing Trump’s statements. Don’t like your Jewish judge? Say something anti-Semitic. Don’t like your rich judge? Talk about how evil the 1 percent are. The legal system can’t tolerate such opportunities for manipulation.

Litigants have to deal with the judges they’re dealt, even when the judge’s race, ethnicity, sex, religion, sexual orientation, political affiliation or socioeconomic status may indeed have some tendency to predispose the judge against the litigant (or in favor of the litigant’s opponents). We demand that judges be impartial despite those factors — but, again, judges being human, it’s inevitable that some partiality will remain in some situations. It’s an imperfect system, as all human systems are. But it’s a much better system than one in which Donald Trump’s cases can only be heard by judges who have no cause to dislike Donald Trump’s opinions.