Immigration enforcement agents cannot be sued over their actions, a three-judge panel of the Fourth Circuit Court of Appeals ruled Friday, because they are not dealing in criminal law.

Eight residents of Northern Virginia sued Immigration and Customs Enforcement agents who detained them while searching for two men who had previously lived in the area. Seven of the eight men were subsequently put into deportation proceedings.

The Legal Aid Justice Center argued that the agents went rogue, violating the men’s constitutional protections in a way that stripped the agents of immunity from a lawsuit. A federal judge in Alexandria had let the case move forward.

The Richmond-based Fourth Circuit disagreed, saying the only avenue to contest such interactions is immigration court.

“Immigration enforcement is by its nature addressed toward noncitizens, which raises a host of considerations and concerns that are simply absent in the majority of traditional law enforcement contexts,” the judges wrote.

One of the men is a U.S. citizen, but the government argued, and the court agreed, that he also is barred from suing ICE for damages.

“It really does create an ‘ICE agent exception’ to the Fourth Amendment to the Constitution,” Legal Aid Justice Center director Simon Sandoval-Moshenberg said. “Not even U.S. citizens, if this ruling is allowed to stand, can recover on civil rights lawsuits if ICE agents kick down their doors.”

The avenues to address such alleged violations in immigration court are “extremely limited” for noncitizens and nonexistent for citizens, he said.

The group is considering asking for a review by the full Fourth Circuit of the decision made by judges Paul V. Niemeyer, A. Marvin Quattlebaum Jr. and Allyson K. Duncan.

The panel based their ruling on a 2017 Supreme Court decision finding strict limits on what kinds of lawsuits can be brought against officials. In that case, the court blocked constitutional claims from men detained by federal agents after the Sept. 11, 2001, terrorist attacks.

The two incidents at issue occurred in February 2017. According to court records, ICE agents stopped men living in South Arlington and Annandale, Va., identified themselves as police, and asked them about a fugitive named Santo Del Cid. After learning nothing, they demanded to go into the men’s homes and arrested several for lacking proper identification.

Del Cid had no known connection to the South Arlington home; the residents said he had been kicked out of the Annandale apartment several years earlier.

Under President Trump, ICE began increasing such “collateral arrests” of people who are in the United States illegally but are not facing criminal charges. The raids and arrests have sparked a backlash in Northern Virginia, home to hundreds of thousands of immigrants.

The Legal Aid Justice Center initially challenged the Trump administration directive, but the Supreme Court subsequently ruled that officials could not be sued for carrying out an administration policy. Ultimately, the plaintiffs argued the agents still violated the Constitution by failing to get consent for the searches, even under the Trump policy.

But the appeals court held that the group “specifically targeted the Trump Administration’s immigration enforcement policy with the purpose of altering it.”