A federal judge accused the Securities and Exchange Commission on Monday of failing to do its job in one of the biggest cases to emerge from the mortgage meltdown, rejecting the agency’s plan to settle a lawsuit against Citigroup.

In a setback to the agency chiefly responsible for policing Wall Street, U.S. District Judge Jed S. Rakoff said that if the charges against Citigroup are true, the SEC settlement is too weak to hold the bank accountable. He ordered the two sides to prepare for trial.

The SEC had accused Citigroup of misleading investors about an investment tied to the deteriorating housing market.

The sharply worded order adds to a growing body of criticism from Rakoff and others that the SEC can be a toothless enforcer. In past cases, the judge has accused the SEC of letting executives off the hook, accepting a paltry fine and leaving the truth in doubt by allowing companies to settle without admitting or denying wrongdoing. Other critics have said that the agency is too close to the industry it oversees.

In a recent report on a different case involving Citigroup, the SEC’s inspector general revealed that the firm had hired a former boss of the top SEC enforcement official to participate in its legal defense. The company ultimately succeeded in defusing potential fraud charges against two executives.

Monday’s order comes at a time when street protesters are complaining that the government has not taken Wall Street to task for its part in the financial crisis.

Rakoff, who previously faulted the SEC’s handling of a case against Bank of America, criticized the SEC’s decision to charge Citigroup with negligence instead of knowing or intentional fraud. The Manhattan judge said that allegations leveled by the SEC seemed to reflect the more serious offense.

Rakoff also slammed the SEC for following its standard practice of allowing defendants to settle charges without admitting or denying wrongdoing.

Some “apologists” may favor “suppressing or obscuring the truth,” Rakoff wrote. “But the S.E.C., of all agencies, has a duty . . . to see that the truth emerges; and if it fails to do so, this Court must not, in the name of deference or convenience, grant judicial enforcement to the agency’s contrivances.”

The court “concludes, regretfully, that the proposed Consent Judgment is neither fair, nor reasonable, nor adequate, nor in the public interest,” he wrote.

Citigroup had agreed to pay $285 million to settle allegations that it misled investors while marketing a complex investment linked to subprime loans in early 2007. The bank was actually wagering against its customers, betting that the investment would lose money, the SEC said.

Citigroup made $160 million in profit on the deal, and investors lost more than $700 million, the SEC alleged.

In a statement, SEC enforcement director Robert Khuzami said that the agency will review the court’s ruling “and take those steps that best serve the interests of investors.” Khuzami defended the deal that the agency had negotiated with Citigroup.

“These are not ‘mere’ allegations, but the reasoned conclusions of the federal agency responsible for the enforcement of the securities laws after a thorough and careful investigation of the facts,” he said.

Summarizing the SEC complaint, Rakoff wrote that the transaction allegedly allowed Citigroup to “dump some dubious assets on misinformed investors” at a time when the bank saw that the market for mortgage-backed securities was beginning to weaken.

If the SEC’s allegations are true, Rakoff wrote, “this is a very good deal for Citigroup.”

If the charges are untrue, he said, “it is a mild and modest cost of doing business.”

“It is harder to discern from the limited information before the Court what the S.E.C. is getting from this settlement other than a quick headline,” Rakoff wrote.

By the SEC’s own account, Rakoff wrote, Citigroup is a repeat offender, but the penalty the SEC has negotiated — $95 million of the proposed settlement — “is pocket change to any entity as large as Citigroup.”

Citigroup reported a third-quarter profit of $3.8 billion.

Rakoff’s calls for tougher settlements have made him a folk hero to many critics of Wall Street. But it is unclear whether a higher court would uphold his arguments if the SEC appeals, or whether many other federal judges will follow his lead.

If the judge’s ruling became the new standard, it could fundamentally alter the way the SEC does business and make it harder for Wall Street firms to make cases go away.

The standard language used in settlements — neither admits nor denies wrongdoing — is one of the carrots that induces defendants to settle.

The SEC has said that without settlements it would be forced to battle cases in court, tying up limited resources. As a result, the SEC said, it would be able to pursue fewer cases.

Federal courts across the country have “repeatedly approved for good reason” the language that Rakoff rejected, Khuzami said. Blocking such settlements “would divert resources away from the investigation of other frauds and the recovery of losses suffered by other investors,” he said.

The SEC has argued that defendants would refuse to settle many cases if they were forced to admit wrongdoing. Such an admission could be used against them in private lawsuits.

Rakoff said that was part of the problem: Investors defrauded by Citigroup would not only recoup less than than their total losses; they would have to establish their own proof in any private lawsuits against the firm without any help from the SEC.

Rakoff has said that he is required to show some deference to the SEC’s judgment. But in Monday’s ruling, he said that the agency was asking too much of the court. Rakoff said that imposing settlements such as Citigroup’s $285 million deal can be unfair to the defendant and be susceptible to “abuse” by the SEC.

“An application of judicial power that does not rest on facts is worse than mindless, it is inherently dangerous,” he wrote. It “serves no lawful or moral purpose and is simply an engine of oppression.”

Citigroup said that it disagreed with the judge’s ruling.

“We believe the proposed settlement is a fair and reasonable resolution,” said spokeswoman Danielle Romero-Apsilos. “In the event the case is tried,” she said, “we would present substantial factual and legal defenses to the charges.”

Donald C. Langevoort, a law professor at Georgetown University, predicted that Rakoff would be overturned if the SEC appeals. He said that the SEC does not have the resources to try most of its cases.

William R. McLucas, a former SEC enforcement director now in private legal practice, said that many companies would fight the SEC in court rather than admit wrongdoing. But individuals charged by the SEC might admit to charges they believe are wrong because they lack the resources to defend themselves, McLucas said.

Former SEC chairman Harvey L. Pitt, however, predicted that regulated financial firms would generally settle, because losing a fraud trial could put them out of business.

“The consequences of losing at trial are so catastrophic for a regulated firm that they really have to settle at some point,” Pitt said.

In an earlier case against Citigroup, the SEC accused the bank of understating its exposure to risky subprime loans. A different judge accepted a $75 million settlement in that case, even as she opined that the penalty was too small to serve as a deterrent.

The SEC’s inspector general investigated that earlier case and recently reported that the bank hired a former boss of SEC enforcement director Khuzami to represent it in the matter. After the lawyer and Khuzami spoke, the SEC abandoned its pursuit of a fraud charge against one executive and agreed to reduce the fraud charge that another executive had already accepted.

The inspector general exonerated the enforcement director, finding that the settlements were “part of a negotiation process that involved several members of the Enforcement staff.”

Michael Smallberg of the Project on Government Oversight offered a different perspective. The appearance of such a “chummy relationship,” he said, “is one of the things that’s causing the public to doubt the SEC’s commitment to holding these firms and their executives accountable.”