Correction: An earlier version of the article incorrectly said that the New York Department of Financial Services had barred Deloitte from advising state-chartered banks for a year. Only one division of the company, Deloitte Financial Advisory, has been barred, and the order affects only new assignments for up to a year. This version has been corrected.

It was 6:30 p.m. on a frigid February day in 2002 when Eugene Ludwig got a phone call that would catapult his fledgling consulting firm into a Washington powerhouse.

Promontory Financial Group today trades on its well-honed expertise of the financial markets and regulatory system — a skill that has only compounded its prestige even as it’s been drubbed in the press as a shadow regulator. Back then, it was little more than a one-man show.

On the line, a woman with a thick Irish accent asked Ludwig to hold for the chairman and chief executive of Allied Irish Bank.

“I thought it was a practical joke,” Ludwig recalls. “It must have been 11:30 p.m. or midnight over there. Who works at that time?”

It was no joke. One of Ireland’s most-esteemed banks had learned that a foreign-exchange dealer at its Baltimore subsidiary had concealed hundreds of millions in currency losses. The revelation rattled senior management and raised serious doubts about the bank’s internal controls.

“It was the first big fraud case of any consequence in Ireland,” said Lochlann Quinn, who was Allied Irish’s chairman and the man on the line a decade ago. “There was just amazement and confusion that something like this could occur.”

As Ludwig listened intently to Quinn explain that the bank needed him to get to the bottom of its mess, he wondered why Allied Irish had picked him. Promontory had been in business only a few months, operating out of a back office at Covington & Burling, where Ludwig once worked as a banking attorney.

“We didn’t even know Eugene, but he had an excellent reputation,” Quinn said. “We figured that he had the kind of credibility we needed.”

Ludwig had run the Office of the Comptroller of the Currency, the regulator charged with oversight of the nation’s banks. A classmate of Bill Clinton’s at Oxford and a fellow Yale Law School graduate, the president put him to work.

A picture of the two is prominently placed on the wall in Ludwig’s office on 17th Street NW. A few inches over is a photo with Fed chief Ben Bernanke. Ludwig’s OCC tenure was largely defined by his push for fair lending and his controversial support of banks moving into the securities market.

After leaving office, he parlayed his reputation into a $3.15 million job at Bankers Trust New York, where he served as vice chairman, through the landmark merger with Deutsche Bank.

To address the Irish crisis, he called on a former Bankers Trust colleague, Duncan Hennes, an expert in structured finance. They dissected the trading book, uncovered governance weaknesses and documented the $750 million in losses. Ludwig flew to Dublin to present the report to the board and dozens of reporters in a news conference that Quinn recalls lasting four hours.

“There was a significant emotional and psychological relief,” Quinn said. “That year our stock traded higher than before the incident.”

Word spread. Within weeks, Promontory’s trickle of business was a bubbling brook. “The character and the direction of the business were pretty much defined by that” job, Ludwig said.

Trading in financial messes

Eleven years later, Promontory is called on by the nation’s biggest banks and financial firms the world over as the ultimate fix-it shop. Founded just before Sarbanes-Oxley, which handed more work to lawyers, consultants and auditors, its timing was impeccable.

The dramatic change in the landscape since the financial crisis unraveled global markets has at once propelled Promontory’s business and threatened the reputation it’s built since rehabilitating that of the Irish bank. It’s been thrust into a firestorm over financial consultants, calling into question the influence it yields in the regulatory world, in which billions of dollars are at stake for its clients.

A glaring example emerged last fall, when ProPublica reported that Promontory and other consulting firms were paid nearly $2 billion by banks to examine shoddy mortgage files. The banks were supposed to pay out millions of dollars to borrowers for flawed foreclosure practices. Despite the consultants’ substantial payday, not a single dime of relief reached the homeowners.

The revelation set lawmakers on edge.

“You’re supposed to represent the public, but you’re representing this private interest,” said Sen. Sherrod Brown (D-Ohio), at a Senate Banking hearing in April. “Why should the public think that’s a good arrangement?”

Now federal and state regulators are creating new codes of conduct for consultants, rules that could disrupt the multi-billion-dollar industry.

To its critics, Promontory represents the outsourcing of oversight, a firm brimming with former regulators who can hold sway with old colleagues or stand in for them altogether. It is a place where Washington insiders have used connections to carve out a niche practice, for which they charge a premium.

“What these banks are paying for is not advice, it’s influence,” said Yves Smith, creator of the liberal financial blog Naked Capitalism. “They are paying for somebody to credibly advance the most aggressive positions.”

To its clients, Promontory is the ultimate cleanup crew with a keen knowledge of how to navigate the latticework of rules that govern the financial sector.

As one attorney put it in the trade publication American Banker: “If the regulators are saying jump on your right foot for 10 miles, we’ll tell you 20 miles. And once you’ve done it, we’ll tell the regulators that you ‘get it,’ and you will pay us well for repairing your regulatory relationship.”

Ludwig, 67, is a big draw for clients.

“Gene is very knowledgable, very professional,” said Bob Wilmers, chief executive of M&T Bank, which recently hired Promontory. “The guy is good and has gotten good people. And in the consulting world people count.”

With 383 employees from Toronto to Tokyo, Promontory has worked on almost 1,600 projects. The firm has advised the European Union and the Indonesian government on bank supervision. It cleaned up money-laundering lapses at Riggs Bank to prepare the ailing institution for a sale to PNC Financial Services Group. The Vatican is its client.

Although garnering the most attention, enforcement actions make up less than 10 percent of Promontory’s work. The firm offers other business solutions and has a thriving cyber security unit.

“Gene is respected at the highest levels, so he’s able to go into the C-suite and lay out the options very clearly. He gives it to them straight,” said H. Rodgin Cohen, a partner at Sullivan & Cromwell, who has teamed with Promontory on projects.

Finding the right people

As Ludwig sees it, Promontory is built on the premise of helping financial firms create stronger, safer institutions by identifying weaknesses and offering ways to correct them. The model is the core of most consulting work, but one Ludwig said was not widely used in the service of financial compliance when he started Promontory.

Rather than return to his roots as a lawyer, he leveraged his experience at Bankers Trust and expertise as the head of the OCC to set up shop.

“Translating what the government really wants people to do and helping them put together a plan was a natural thing to do,” he said. “It was not just a legal matter. It involved economics, management and specialty experience. . . . That wasn’t something any group was doing as a regular diet.”

Finding the right people to build out the firm was a challenge, however. Ludwig turned to Hennes from Bankers Trust and Alfred Moses, an old partner at Covington & Burling, to get started.

The relationships he cultivated and the reputation he earned as comptroller proved most useful in turning Promontory into a major player and put it toe to toe with other marquee consultancies like Deloitte and PricewaterhouseCoopers.

Within a year of launching the firm, Ludwig enlisted Alan Blinder, a former vice chairman of the Federal Reserve; Mark Jacobsen, a former chief of staff at the Federal Deposit Insurance Corp.; and Susan Krause Bell, a former senior deputy comptroller of the currency.

Ludwig says he is proud of amassing a staff with deep roots in Washington because their understanding of the rules and how they are meant to be applied is invaluable for the industry.

“We are here to implement what the government wants done,” he said. “People who come here have high integrity and want to help institutions do the right thing in the aide of financial stability.”

Consulting firms across the industry reach into the ranks of regulatory agencies to build their staffs, said Gilbert Schwartz, a banking lawyer.

“You get people who understand what the agencies are looking for,” he said. “They give you the best advice they can on what the regulators are going to say and how it should be implemented. That’s just the way the industry works.”

As it stands, Promontory’s staff includes alumni of seven federal financial agencies, five central banks, three state regulators and the Treasury. Nearly two-thirds of the firm’s U.S. managing directors once worked at a regulatory bureau.

Executives at Promontory have met with members of Treasury and the Federal Reserve at least 10 times in the past two years, but the firm doesn’t consider itself a lobbyist. It took up issues like the Volcker rule that limits risky trading, according to the Sunlight Foundation, a nonprofit that tracks government.

In Washington, the lines between lobbying and advocacy are easily blurred. In 2009, it was a registered lobbyist because of its work for General Motors.

Ludwig said executives at Promontory are rarely up on Capitol Hill. And if they are, it’s often because they are reporting findings in a review or have been asked to share views on the impact of regulation.

“I can’t say, or would not say, there’s never been an occasion in which one of our people didn’t advocate for something,” Ludwig said. “But I will say this: We are adamant that nobody here take a position that they don’t believe in.”

Hiring the regulatory elite has its complications. Promontory was ensnared in a controversy in January over an executive trade. Within days of the firm hiring Julie Williams, the former chief counsel at the OCC, the bank regulator replaced Williams with a managing director at Promontory, Amy Friend.

Ludwig defends the move, citing ethics rules that restrict interaction between former regulators and their old agencies.

In many ways, Promontory was the natural destination for the likes of Mary Schapiro, 58, who stepped down last year as chairman of the Securities and Exchange Commission. There are simply not many places where someone with 30 years of experience regulating Wall Street (earlier as the head of both the Commodity Future Trading Commission and the Financial Industry Regulatory Authority) could land.

To avoid even a hint of impropriety, she says, she’s resigned “to not ever represent a client at any federal regulatory agency. I just didn’t want to do that.”

She focuses on advising overseas regulators on market structure.

Schapiro is at home among her peers. “The talent pool is deep. The people are knowledgable, and there was a comfort level in a place where lots of people had worked as regulators,” she said.

The collaborative environment and intellectual rigors of policy implementation — and not just advocacy work — attract talent, according to staffers.

Ludwig, who is gregarious and known to put in long hours with enthusiasm, engenders loyalty among his workers. As a former analyst in New York put it: “He’s the kind of person that remembers the names of even the junior guys.”

Also legend among his workers is the holiday party he hosted, welcoming the entire staff into his $11.5 million estate on Foxhall Road for an “incredible” affair.

The pay is not too shabby, either. Salaries for senior executives hover around seven figures, people familiar with the firm’s compensation structure say. Before heading to the OCC, Amy Friend was paid $1.2 million at Promontory, according to financial disclosures obtained by Bloomberg News.

To maintain that kind of pay scale, Promontory reportedly charges its clients in the ballpark of $1,500 an hour.

The firm, which is privately held, would not discuss its financials.

Ludwig, however, said, “I tell clients all the time: If you think we are charging too much and we are not adding value, I don’t want to do that. You can’t build a business, not a business as long as we’ve been going and growing, if you’re not providing value.”

Crisis makes for engaged clients.

“When you’re looking at open-heart surgery, you don’t ask about the cost,” said Cohen, of Sullivan & Cromwell. “These are very sophisticated clients that are looking for quality.”

A tarnished reputation

Not all of its clients have fared well. In 2008, after the CFTC flagged serious risk-management failures at MF Global, the brokerage firm hired Promontory to clean up its internal controls.

Promontory issued a report in 2011 declaring MF Global was on a solid path under Jon Corzine’s leadership, despite a series of analyst reports to the contrary. Senior management at MF Global “had set a tone at the top that supports a best-practice, enterprise-wide risk-management framework,” the report said.

Within months, in a great whodunnit, $1.6 billion in customer money went missing from its accounts. The brokerage went bankrupt. Today Corzine faces civil charges for his role in its collapse.

Promontory has been criticized for overlooking glaring problems at the firm. Ludwig calls that “unfair.” He said the firm was hired to determine whether MF Global had lived up to the CFTC consent order, not to identify weaknesses throughout the organization.

“We did it thoroughly and correctly,” he said. “It is something that you worry about because people will zing you if anything goes wrong in some other part of the institution.”

The consent order, however, did call on Promontory to make recommendations to “ensure the effectiveness of MF Global’s . . . risk management, supervision and/or compliance programs.”

Last year, Promontory’s work for Standard Chartered Bank drew attention. Asked to conduct an anti-money-laundering review, it determined that the British bank had funneled about $14 million of illicit money to Iran. The number fell well short of the assessment by New York State’s Department of Financial Services, which accused the bank of processing at least $250 billion in illicit transactions.

Ludwig says the transactions Promontory uncovered were closer to the findings of Treasury’s Office of Foreign Asset Control, which identified about $24 million in illegal transactions to Iran. Promontory was specifically looking for violations of OFAC standards, whereas regulators in New York had broader jurisdiction over the bank’s records.

Perhaps no assignment has dogged Promontory as much as the independent foreclosure review. After regulators at the OCC and Fed penalized 13 mortgage servicers for abuses, Bank of America, Wells Fargo and PNC hired Promontory to identify troubled mortgages.

Although Promontory says it worked at the behest of the regulators, it was paid a total of $927.5 million by the three banks, according to documents provided by the Senate Banking Committee. The firm said it performed several million hours of labor combing through 250,000 mortgage files over 17 months.

But a year into the review not a single borrower had received any relief payments, while the seven consultants racked up a $2 billion tab.

Consumer groups and lawmakers raised doubts about the independence of consultants handpicked by banks accused of wrongdoing. As pressure mounted, regulators scuttled the review and negotiated a new $9.3 billion agreement with most of the banks. The work of Promontory and others was used to determine final compensation.

The deal was not enough to satisfy members of Congress who demanded an explanation. A General Accounting Office report laid most of the blame at the feet of regulators, who were accused of failing to provide clear criteria.

Still, at the Senate Banking hearing, Brown said there were inherent conflicts of interest in the regulatory consulting model. Most of the firms involved in the review had worked for the banks that hired them in the past, he noted.

Konrad Alt, a managing director at Promontory, said the firm did not allow the banks to run the review, an accusation lobbed by consumer groups. Mortgage servicers assembled the files, he said, but final determinations were made by the consultants.

“We stand behind our work. We did it objectively. We did it carefully. We did it to the highest professional standards. In that sense, the process worked,” Alt said. “The perception that it didn’t work comes about because it took so long.”

Alt added, “It was never realistic to believe that a project of this magnitude could be done quickly.” Retrieving documents and waiting for regulators to develop guidance for the project only contributed to delays, he said.

In the wake of the review debacle, the OCC has asked Congress for more authority to rein in consultants.

Brown’s office said he is working to grant regulators more supervisory powers over consultants, but it requires legislative action.

Accounting firms, like PricewaterhouseCoopers and Ernst & Young, with consulting practices adhere to professional standards set by the American Institute of Certified Public Accounts. Pure consulting shops, however, operate with no formal regulatory oversight.

Efforts are underway at the OCC to develop more stringent guidelines for independent consultants used in enforcement actions. The guidance will focus on the consultants’ qualifications and monitoring their work.

“Our objective is to ensure that banks operate safely and soundly and in compliance with the law. And if the retention of a consultant helps them to do that, then it’s something we’re all in favor of,” said Daniel Stipano, deputy chief counsel at the OCC. “We just want to make sure they have the expertise and resources.”

He said the OCC is taking a look at the standards for consultants being implemented by New York state’s top financial regulator, Benjamin Lawsky.

The head of the state’s Department of Financial Services issued new guidelines in June, after slapping Deloitte with a $10 million fine and barring one unit of its firm, Deloitte Financial Advisory, for a year from advising banks chartered in New York. The penalties stemmed from the consulting giant’s work on the Standard Charter case, in which it was accused of helping the bank flout U.S. sanctions.

“I would not call it an isolated incident, but because we have other matters pending I don’t want to comment in any detail,” Lawsky said. “Often the results of the work consultants do are unsatisfactory and in many cases it’s because there are misaligned incentives.”

He added, “Regulators need to be far more active in managing consultants and make sure they know that the consultant works for the regulator.”

Going forward, Lawsky wants consultants to disclose financial ties that could compromise their independence. Consultants will also have to inform the department of any area of work in which the bank has asked them to make changes, and submit drafts of their reports. The idea, he said, is to “get under the hood” of the relationships between the consultants and the banks.

“Consumers and taxpayers deserve clear guidance from regulators to prevent conflicts of interest,” Brown said in an e-mail. “We know that independent consultants like Promontory provide oversight services and advisory services that sometimes border on advocacy.”

Ludwig said Promontory has “complied with every legal requirement” and “intends to comply with the rules that the agencies adopt.”