Supreme Court Justice Stephen G. Breyer is an engaging, intellectually serious advocate of judicial modesty and compromise. Some years ago, he put forward an admirable concept he called “active liberty.” It asked judges to recognize “the principle of participatory self-government” as the heart of the Constitution’s purposes.

If I believed that today’s judicial conservatives shared Breyer’s inclination toward compromise and restraint, I might agree with his warnings last week against the movement to enlarge the Supreme Court.

Unfortunately, most right-wing judges are not who Breyer wants them to be, and the court on which he serves is not as apolitical as he wishes it were.

The Supreme Court faces a legitimacy crisis not because progressives are complaining but because of what they are complaining about: a reckless, right-wing, anti-democratic court majority, and a conservative court-packing campaign marked by the disgraceful Republican blockade against President Barack Obama’s nomination of Merrick Garland in 2016 and the unseemly rush to confirm Justice Amy Coney Barrett just before President Donald Trump’s defeat last November.

So I respectfully dissent from the skepticism Breyer expressed about court enlargement in a lecture at Harvard Law School last week precisely because I share his underlying principles.

For the same reason, I applaud President Biden for creating a commission on Friday to examine reforms to the courts, including the possibility of adding Supreme Court justices. Even if the commission doesn’t endorse enlargement, it underscores the nature of the crisis we confront.

Breyer’s Harvard lecture offered a thoughtful historical argument for why protecting the court’s legitimacy is vital to protecting liberty.

“If the public sees judges as politicians in robes,” he said, “its confidence in the courts and in the rule of law itself can only diminish, diminishing the court’s power, including its power to act as a check on other branches.”

What Breyer declined to point out is that conservative justices are the ones who have turned themselves into party bosses through decisions such as Shelby County v. Holder, which gutted the Voting Rights Act, and Citizens United v. Federal Election Commission, which opened the floodgates to big money in politics.

“The rule of law depends on trust, a trust that the court is guided by legal principle, not politics,” Breyer said.

Right. But then he added: “Structural alteration motivated by the perception of political influence can only feed that latter perception, further eroding that trust.”

“What I’m trying to do,” he said at another point, “is to make those whose initial instincts may favor important structural change or other similar institutional changes, such as forms of court-packing, to think long and hard before they embody those changes in law.”

Alas, the good justice has the causation arrow pointing in the wrong direction.

Senate Minority Leader Mitch McConnell (R-Ky.) and Trump were the court-packers. There would be far less talk of court enlargement if McConnell and Trump had not abused their power. Nor would enlargement be on the table if conservative justices had not substituted their own political preferences for Congress’s decisions, notably on voting rights and campaign finance reform, 5-to-4 rulings on which Breyer, rightly, joined the dissenters.

Breyer also seemed unhappy that in reporting on a jurist’s decision, media outlets are more inclined than in the past to mention “the name or the political party of the president who nominated the judge to office.” But there’s a reason for this. Particularly in cases affecting participation in elections, a judge’s partisan provenance is hugely predictive of how he or she will decide them.

In fact, supporters of court enlargement have already thought “long and hard” before we got here. We’re not the radicals. We’re not the judicial activists. We’re not the ones trying to overthrow decades of precedent.

In my ideal world, we would not have to worry about a thoughtful justice such as Breyer spending the rest of his days on the court. But that world no longer exists.

This is why many liberals are calling on the 82-year-old justice to resign. They want a Democratic president, backed by a Democratic Senate, to install his replacement. After Garland, only a fool (something Breyer certainly is not) would believe that a Republican Senate would give a Democratic president’s appointee, no matter how moderate or qualified, a hearing.

The irony for Breyer is that he must quit to give his own principles a fighting chance in the future. I’d be sad to see him go, but not nearly as sad as I would be if Justices Elena Kagan and Sonia Sotomayor were the only liberals left on the court.

In his speech, Breyer declared that “the Constitution itself seeks to establish a workable democracy and to protect basic human rights.” That’s a bracing vision, and it’s what advocates of court enlargement are trying to protect.

Read more: