Chandeliers dangled above the reception, and oil paintings of past chief justices lined the walls. The ceremonial room at the Supreme Court spoke of the institution’s history, and appropriately so. Justice Neil M. Gorsuch had just taken his formal oath, making him the 101st associate justice of the nation’s highest court.

The reception concluded, and a smaller group continued on to a second celebration. Gorsuch, 49, swept through the crowd. A beaming Justice Anthony M. Kennedy watched his former clerk whose career was shaped in his own chambers, according to George Washington University Law School professor Jonathan Turley. “There was an almost parental pride that you could see in their relationship,” Turley recalled. “[Kennedy] clearly had a deep respect and affection for him. He saw Gorsuch as part of his legacy.”

Since Kennedy was confirmed in 1988, he has adopted a relatively predictable approach: consistently voting in favor of individual rights (outside of criminal justice and capital punishment) but otherwise voting with the right. Kennedy, according to Turley, bridged the bench, not because of his votes but his voice. “He had impeccable credentials as a jurist,” Turley said. “He showed that you could have a conservative philosophy and still protect individual rights.”

That’s why, some say, the retiring justice is irreplaceable.

The weight of the impending shift in the court will depend on whom President Trump selects as a replacement. Many of Kennedy’s historic opinions succeeded by a vote of 5 to 4, held together by only one: his own. That legacy could be undone by a nominee with an even slightly more conservative philosophy.

Experts say that Kennedy’s retirement will be the most consequential shift the court has seen in decades. But a few have come close.

Robert Bork, the justice that never was

“The last time the court had the potential to shift this dramatically was with Robert Bork’s seat,” said Jeffrey Rosen, professor at George Washington University Law School and chief executive of the National Constitution Center.

Justice Lewis Powell retired in 1987. Because he had been the swing vote on the bench, President Ronald Reagan’s nominee had the potential to tip the court’s jurisprudence right or left. Reagan selected conservative Robert Bork, a legal academic and judge who adhered to originalism, the belief that the Constitution’s meaning should only be interpreted based on the intent of its authors.

“Historically, it was the most controversial appointment, until Merrick Garland,” Rosen said, referring to the nominee of President Barack Obama who was denied a Senate hearing.

Bork’s nomination was brought to a vote in the Senate, where Sen. Edward Kennedy (D-Mass.) aggressively opposed him and his “extremist view of the Constitution and the role of the Supreme Court”:

“Bork’s America is a land in which women would be forced into back-alley abortions, blacks would sit at segregated lunch counters, rogue police could break down citizens’ doors in midnight raids, schoolchildren could not be taught about evolution, writers and artists would be censored at the whim of government, and the doors of the federal courts would be shut on the fingers of millions of citizens for whom the judiciary is often the only protector of the individual rights that are the heart of our democracy.”

Bork was not confirmed.

Next, Reagan appointed Judge Douglas Ginsburg, who withdrew his nomination amid allegations that he smoked marijuana while teaching at Harvard. Anthony M. Kennedy, a moderate California Reagan Republican, was the president’s third choice. He was unanimously confirmed.

The controversial confirmation of Clarence Thomas

To many, there has never been a more dramatic transition than Clarence Thomas replacing Justice Thurgood Marshall in 1991. Marshall, the first black justice and a champion of civil rights, was a liberal who believed in limiting the Constitution. After nearly 30 years, he left the bench, prompted by dismay in the direction the court was drifting, The Washington Post has previously reported.

The shift following his departure was pronounced.

Thomas, nominated by George H.W. Bush, was the antithesis of Marshall: a conservative and staunch originalist.

Decades before #MeToo, his confirmation hearings concluded with accusations from Anita Hill, a former employee who said Thomas sexually harassed her in the workplace. Her salacious testimony positioned the nominee, a former head of the Equal Employment Opportunity Commission, at the center of a national scandal. In a scathing statement, Thomas denied the allegations and denounced the commission for committing a “high-tech lynching.”

“We’re in perilous times, but any time a justice retires there will be a great deal at stake in his or her replacement,” said Michael Dorf, who clerked for Kennedy in the 1991-1992 term and is now a professor at Cornell Law.

The vacancy left by Marshall’s retirement swung the court’s center further from the liberal majority. However, there was no question about which direction the court would go. Justices Kennedy, Sandra Day O’Connor and Byron White trod on, maintaining the middle, Turley said.

A conservative moves the court to the right

O’Connor, the first female Supreme Court justice, was replaced by Samuel Alito in 2006. O’Connor’s resignation is a better parallel to Kennedy’s, according to some attorneys and legal academics, among them David Lat, founder of the legal website Above the Law.

Initially O’Connor was reliably right-leaning, only falling into the role of swing justice later near the end of her tenure. When she was replaced by Samuel Alito, a down-the-ticket conservative, “it shifted the court,” Lat said. “You saw O’Connor’s precedents getting tossed out in everything but name as a result.”

This transition may be even more significant.

When Marshall stepped down, he left a court that spread across the political spectrum. The addition of Ruth Bader Ginsburg and Stephen G. Breyer offered further balance.

This court, however, has very little center, according to Dorf.

“It has a conservative wing and a liberal-ish wing. There are differences within each, but they’re pretty far apart,” he said. He expects to see an incremental shift in jurisprudence on noteworthy issues, like abortion and race-based affirmative action. “A more stable five-justice majority could effectively kill Roe v. Wade by 1,000 paper cuts.”

With Marshall replaced by Thomas and O’Connor by Alito, the court slid slightly to the right. But if a conservative nominee is confirmed to replace Kennedy, the closest the court will have to a centered swing voter is Chief Justice John G. Roberts Jr., experts said. Nominated by President George W. Bush, Roberts is more conservative than Kennedy and as chief justice has remained reliably to the right, though in recent years he has been called wobbly

“I could see this shift falling in between the O’Connor-Alito and Marshall-Thomas shifts,” Lat said. “We forget Justice Kennedy is fundamentally a pretty conservative justice. People think about the LGBT decision, but on free speech and business interest he was generally conservative, but with a lower case ‘l’ libertarian into individual rights. The person chosen to replace him may hold the same philosophy.”

Gorsuch’s appointment assuaged concerns from the left, yet spoke to the continuity and tradition of the court. According to Lat, the front-runners for the nomination are Brett Kavanaugh and Raymond Kethledge, both conservative former Kennedy clerks.

“It would be a nice selling point when Trump can tell the public this jurist is being replaced by someone he mentored and respects,” Lat said.

Having Gorsuch on the bench may have reassured Kennedy he would not leave the court imperiled, but his timing is precarious. Like O’Connor, Kennedy may soon see his legacy undone.

Read more: