The cautious federal bureaucracy responsible for policing Wall Street faces several options as it tries to salvage a high-profile case against Citigroup, and all of them amount to high-stakes gambles.

A federal judge put the Securities and Exchange Commission on the spot Monday by rejecting the $285 million settlement the agency had proposed to resolve charges that Citigroup misled investors about an investment product tied to subprime mortgages.

The judge complained that the $285 million was mere pocket change for a firm of Citigroup’s size, and he protested that the deal would leave the truth of the government’s allegations in question by allowing the bank to neither admit nor deny wrongdoing.

The ruling by U.S. District Judge Jed S. Rakoff threatens more than just the SEC’s case against Citigroup. It challenges the SEC’s decades-old practice of settling most of its cases instead of taking them to trial.

If defendants were forced to admit wrongdoing, many would refuse to settle because the admissions could be used against them in private lawsuits or other law enforcement actions, lawyers say.

Rakoff challenged the SEC to put up and prove its case.

The agency could take Citigroup to trial. That would risk a potentially embarrassing defeat that would expose the SEC to criticism that it was incompetent or overreaching.

In a legal brief, Citigroup telegraphed that it has defenses — “a number of substantial factual and legal issues that would need to be litigated in the absence of a settlement.”

The SEC essentially accused Citigroup of making $160 million by unloading deteriorating assets on unsuspecting buyers. One of the SEC’s main allegations is that Citigroup failed to adequately inform investors that even as it was selling them the investment, it was betting that the investment would decline in value.

Whether investors were given sufficient information would be a point of contention.

Citigroup says its marketing materials included this disclosure: The firm “may be expected to have interests that are adverse to those of the Noteholders.”

If the SEC won the trial, Rakoff’s ruling would stand unchallenged, threatening future settlements and leaving the state of the law unclear. From his bench in Manhattan, Rakoff presides over many Wall Street cases.

The agency could try to renegotiate its deal with Citigroup. In a previous case involving Bank of America, Rakoff rejected the $33 million settlement the SEC initially proposed but later accepted a revised $150 million deal, even though he said it was deeply flawed.

Whether or not it was intended as a hint, a footnote in Rakoff’s ruling Monday highlighted what could be part of a winning settlement. The judge said the Citigroup pact compared unfavorably to the deal Goldman Sachs struck in a similar case. Unlike Citigroup, Goldman made what Rakoff called an “express admission” — “Goldman acknowledges that the marketing materials for the . . . transaction contained incomplete information.”

Rakoff’s ruling Monday focused largely on the SEC’s request that he enter an injunction against Citigroup; it isn’t clear what would happen if the two sides dropped that piece of the settlement.

However, one prominent Wall Street lawyer said renegotiating the Citigroup deal appears to be a non-starter. The lawyer spoke on the condition of anonymity because he represents companies involved in enforcement actions. To satisfy Rakoff this time, the lawyer said, it appears that Citigroup would have to admit wrongdoing — an untenable step.

That leaves a third course: appealing the judge’s ruling.

Some lawyers interviewed for this story said it was not clear whether the ruling could be appealed.

If it could, and if the appeals court sided with Rakoff, the SEC would be in deeper trouble. Other judges in the same circuit — which encompasses the center of the nation’s financial industry — would be bound by the same standard.

“Then they’re stuck with Judge Rakoff’s ruling,” potentially taking away the SEC’s ability to rely on settlements, said former SEC commissioner Roel Campos.

The SEC says it lacks the resources to try every case and would pursue fewer investigations if it were tied up in court.

If the SEC couldn’t file a conventional appeal, it could ask an appeals court to issue an extraordinary order known as a “writ of mandamus” commanding Rakoff to accept the settlement on the grounds that the judge was biased or his opinion was beyond the pale. Attacking the judge so directly would be an extreme step, lawyers said.

The situation “presents a lot of difficulties,” said Bruce A. Hiler, a securities lawyer and former SEC enforcement official.

Although the SEC’s approach to enforcement may seem bizarre to the public, Campos said, the alternative for companies and the SEC “is very bad.” For practical reasons, Campos said, the SEC “seems at times not to punish severely enough the wrongdoers.” The SEC “doesn’t want to put companies out of business unless they’re a thoroughly corrupt enterprise,” and it “doesn’t want to expose companies to being automatically liable in lawsuits.”

“The question is, Should that system be defended?” he said.

One of Rakoff’s complaints is that Citigroup’s penalty would pale beside the more than $700 million that the SEC says investors lost.

The settlement he rejected called for Citigroup to give up $160 million of profits from the transaction and pay $30 million of interest on that, plus a penalty of $95 million.

In a statement Monday, SEC enforcement director Robert Khuzami said the judge had overlooked limits on the amount of money the SEC can recover. “The SEC does not currently have statutory authority to recover investor losses,” he said.

However, in an earlier court filing, the SEC said that, under the law, it could have imposed a penalty twice as large as the $95 million. In the filing, the SEC said it determines penalties based on a number of factors, including the “extent of injury to innocent parties” and the “degree to which the penalty will recompense or further harm the injured shareholders.”

SEC Chairman Mary L. Schapiro echoed Khuzami’s point in a letter this week to one of the SEC’s Senate overseers. Legal limits often prevent the SEC from imposing penalties that match investors’ losses, she wrote.

In many frauds, Schapiro added, “the maximum penalty available to the Commission may not adequately reflect the seriousness of the violation or the impact on victims of the fraud.”

Schapiro proposed changing the law to allow larger fines.