James Esseks is director of the ACLU Lesbian Gay Bisexual Transgender & HIV Project. He is counsel in Fulton v. City of Philadelphia, which is before the Supreme Court, and was counsel in Obergefell v. Hodges.

Supreme Court Justices Clarence Thomas and Samuel Alito said in a statement this week that they see Obergefell v. Hodges, the 2015 Supreme Court decision that brought marriage equality to the whole nation, as something the court needs to “fix.” LGBTQ advocates immediately raised the alarm and asked what it meant for our families and future.

Obergefell is indeed vulnerable, but overturning that ruling directly is not the only way for the court to undermine LGBTQ equality. More likely, it will dilute our rights by creating a constitutional license to discriminate based on religious beliefs. In fact, the court is set to hear oral arguments in a case that could significantly scale back LGBTQ rights on Nov. 4, the day after the election.

In Fulton v. Philadelphia, the court could create a precedent that allows discrimination against LGBTQ families. More specifically, private agencies that receive taxpayer funding to provide government services — such as foster care agencies, food banks and homeless shelters — could be given a constitutional right to deny services not just to people who are LGBTQ, but even to religious groups, such as Jews, Muslims or Mormons.

The Fulton case involves the foster-care system. The city of Philadelphia, like many governments, hires private agencies to evaluate prospective foster parents according to the state’s screening criteria. One of those agencies, Catholic Social Services, insisted on following its own religious screening criteria because it objects to approving same-sex couples as foster parents. When the city didn’t renew its contract, CSS sued the city, arguing it has a constitutional right to be paid taxpayer dollars to perform this service and to change the contract terms to fit its religious beliefs.

CSS’s objections, no doubt, stem from sincere religious beliefs. Nevertheless, LGBTQ parents are just as qualified as any other parents to provide foster care, and our families shouldn’t be treated differently by government programs.

A ruling in CSS’s favor could go far beyond impacting LGBTQ people; it could create a constitutional right to override the needs of those accessing government services, as well as the government’s own judgment about how to provide those services. Such a ruling could negate our collective commitment to nondiscrimination based on race, sex and religion whenever a government contractor identifies a religious basis for its refusal to provide services. That would be a breathtaking change in the country’s discrimination laws, yet in a ruling at an earlier stage of the case, three justices (Thomas, Alito and Neil M. Gorsuch) indicated they would likely side with CSS.

The people who would be hurt most directly by such a ruling include the children who will remain in Philadelphia’s group homes rather than with foster families, just because of CSS’s insistence on turning away qualified LGBTQ foster parents.

Freedom of religion is important, and it’s one of our nation’s fundamental values. That’s why it’s protected by the First Amendment to the Constitution. But that freedom doesn’t give any of us the right to impose our beliefs on others, or to discriminate.

Fulton isn’t the only religious exemptions case that could come before the court in the next few years. Pending cases would address whether a flower shop has a constitutional right to turn away customers because they’re gay. Or whether a hospital has a right to deny care to patients because they are transgender.

Let’s be clear: LGBTQ people — particularly transgender people and LGBTQ people of color — continue to face alarming rates of discrimination. Many of us still go through our days wondering if we’ll be turned away from stores, harassed on public transportation or denied access to health care. Civil rights laws protecting LGBTQ people are critical for us to live our lives and participate fully in our communities.

These aren’t the first cases in which courts were asked to create a license to discriminate in the name of religion. Thankfully, courts have repeatedly refused to endorse such a right. Multiple courts have said religious schools can’t pay women less for doing the same job as men even when it’s based on a sincere religious view that men are heads of household. And the Supreme Court said decades ago it is “frivolous” to say the Constitution’s protection for religion gives a business the right to discriminate against customers based on race.

In more recent years, attempts to create carve-outs in our nondiscrimination laws have focused on LGBTQ people. But there’s no reason the rules should be any different: Even sincerely held religious beliefs that necessitate discrimination should not be enough to override the American commitment to nondiscrimination.

The outcome of these cases can do serious harm to LGBTQ people, even if the court does not directly overturn Obergefell. That’s truly something to fear from the Supreme Court.

Read more: