When I heard Paul Manafort had been sentenced to a mere 47 months in prison for eight criminal charges involving extensive tax and bank fraud, I thought back to my first client, a man I represented in 2010 through a Stanford Law School clinic that challenged life sentences imposed for minor offenses. My client had previously been sentenced to life in prison under California’s “three strikes” law for stealing a pair of pants from a Sears. He had stolen the pants, he explained, so he could try to return them and use the money to buy a carseat for his soon-to-be newborn baby, whom he would need to drive home from the hospital. His other two “strikes” were for two unarmed burglaries of unoccupied homes he had committed on the same day when he was 18 years old.

After his trial, the judge declined his trial lawyer’s request for leniency, despite a history of poverty, childhood abuse and drug addiction, and sentenced him to 25 years to life in prison. (After 14 years, he was released because of the Stanford clinic’s advocacy.) Manafort’s sentence reminded me not only of that client, but of the many clients I have represented as a public defender who were given extremely harsh sentences, despite having grown up in challenging conditions that could not have been more different from Manafort’s privileged background.

Yet it was Manafort who got sympathy. U.S. District Judge T.S. Ellis III, who sentenced Manafort to dramatically less than the federal guidelines prescribed — those call for 19 to 24 years — explained that Manafort had lived an “otherwise blameless life,” despite his multiple serious crimes, and earned “the admiration of many.” Ellis afforded Manafort the benefit of the doubt again and again, even speculating that Manafort’s fraudulent failure to disclose a loan on an application may have been the oversight of a “very busy man.”

Manafort was convicted of crimes committed over almost a decade of his life: evading taxes, concealing a foreign bank account and defrauding banks of millions of dollars. Even after being charged in federal court and released on bail, he tried to tamper with witnesses. After convictions on eight charges, he still failed to apologize (as Ellis noted). When he wasn’t committing crimes, he was using his power and connections to protect corrupt and violent world leaders.

Still, the 78-year-old white judge, looking at the 69-year-old white defendant, saw someone whose numerous crimes were a digression from what he deemed an essentially well-behaved life. I suspect this is because Manafort does not look like most people sentenced in the criminal justice system — disproportionately it is people of color, whose lives are more often characterized by trauma, poverty and inequality rather than wealth and privilege. Sometimes those circumstances lead them to be wrongly accused and convicted, as police disproportionately target members of impoverished communities and people of color. Sometimes that trauma and poverty causes such desperation that they commit crimes. But almost always, it means that, if they are caught, they are denied the very compassion Manafort begged for and ultimately received.

Compare Manafort’s “otherwise blameless” life with the lives of my clients. I’ve known people sentenced to more time than Manafort — sometimes for crimes such as gun possession or burglaries — even though, as children, they were sexually and physically abused. (Indeed, sometimes they had prior offenses, but Manafort, too, committed serial criminal acts.) I’ve had clients whose parents, often themselves victims of tragedy, introduced their children to drugs. Clients who had, at one point, watched as a loved one was shot and killed in front of them. Clients who have themselves been shot and live with bullets in their bodies.

Our government does not do enough to protect people like them from harm or provide them with opportunities to obtain education or jobs that might put them on a steady path. Yet, as soon as they make a misstep, as soon as they come into contact with the criminal justice system, suddenly there are ample resources available to incarcerate them.

Compare that with Manafort and most other white-collar defendants. Their access to resources — wealth, education, family support — help them rack up accomplishments and live comfortable lives. When they abuse their power and take advantage of others, the fruits of their early advantages become a mitigating factor in their punishment. If anything, isn’t there less of an excuse for a professionally successful person to break the law than for someone who has faced so much adversity?

If Manafort deserves leniency simply because there was a time in his life when he was not committing crimes, then why does so much blame fall on my clients, for whom that observation also holds true — and who have suffered from poverty, abuse, poor education, job discrimination and police violence?

Let me be clear. I am not advocating harsher sentences for white-collar defendants, or for anyone. And I absolutely do not believe judges should issue decisions to conform with public opinion. But my clients deserve compassion and discretion in sentencing, too, and they often do not receive it.

I hope the result of Manafort’s sentencing is not a collective angry demand for tougher sentences for white-collar criminals. Longer sentences fuel mass incarceration and have little, if any, relationship to public safety. Instead, let’s challenge across the board overly punitive sentencing guidelines and mandatory minimums. Let’s speak up when people who don’t look like Manafort, and had none of his advantages, receive harsh penalties — just as this judge spoke up for Manafort. Until we do that, none of us can claim to be blameless when it comes to the perpetuation of an unfair criminal justice system.