Eric Holder (AP Photo)

U.S. Attorney General Eric Holder Jr.’s departure – or more precisely his replacement – is an issue with huge implications for the housing sector, and a former White House official bluntly explained why.

Jim Parrott, a former housing advisor in the Obama White House, cast much of the blame for today’s tight mortgage lending on the Justice Department’s aggressive enforcement actions in the wake of the housing bust, including the record multi-billion settlements reached with big banks in recent years.

It’s not so much the size or the number of the settlements, but rather the open-ended nature of the Justice Department’s agenda, and the sense that lenders are “just not sure what closure looks and feels like,” Parrott said at the Bipartisan Policy Center’s housing summit in Washington this month.

“On the law enforcement side, there’s still a clamoring for retribution,” said Parrott, a former senior advisor with the National Economic Council. “There hasn’t been enough of a connection between that clamoring and the clamor for access to credit.”

The comments come at a time when lenders are imposing higher than usual credit scores and other tough standards on people applying for government-backed mortgages. The lenders say they’re exceeding the government’s own criteria in a bid to insulate themselves from more financial penalties and lawsuits. And several analyses suggest that millions of potential home buyers are getting shut out as a result.

The financial penalties came after regulators forced lenders to buy back billions of dollars in government-backed mortgages after the housing bust or pay for losses on some of the loans that went bad. But many of the high-profile lawsuits – including the record $17 billion settlement with Bank of America last month – were led by the Holder’s Justice Department, which is pushing to hold lenders accountable for misdeeds that contributed to the housing crisis. (And some critics say Justice hasn't gone far enough.)

Parrott said that the regulators that oversee Fannie Mae, Freddie Mac and the Federal Housing Administration “increasingly get the nexus between uncertainty and access,” and they're responding to it. (More on that later.) But law enforcement officials at the Justice Department, not so much. In political circles, versus regulatory ones, there’s much less consensus on what the right path should look like, he added.

Uncertainty about when this particular phase of enforcement actions will end makes it tough for lenders to quantify their legal risks, and as a result, their executives and boards don’t want to take a chance on borrowers who have less than pristine credit, Parrott said. “I’m worried that will be a factor that will keep depressing access to credit,” said Parrott, who now advises financial institutions on housing finance issues and serves as a part-time senior fellow at the Urban Institute.

So the guessing game begins as to who Holder’s replacement will be.

“Our view is that the odds favor the President picking someone who will pick up where Holder left off in pursuing the big banks as the President has never given an indication that he thought the Justice Department had gone too far,” Jaret Seiberg, an analyst with Guggenheim Securities, wrote in a note to analysts Thursday. “As a result, we are dubious that this change will result in less big bank litigation or help open the credit spigot.”

However, the White House is alarmed enough about the tough lending environment that it’s been talking to key players in the industry about what it will take for lenders to ease up, most recently on Sept. 16, when the chief executive of the Mortgage Bankers Association, David H. Stevens, brought executives from some of the nation’s largest banks to the White House for a meeting with top housing officials.

Earlier this month, Housing and Urban Development Secretary Julian Castro said that boosting homeownership will rank as a top priority during his tenure. Castro, who joined HUD in June, acknowledged that tough lending requirements are an obstacle. “The pendulum has swung too far in the other direction,” Castro said at the Bipartisan Policy Center conference.

The Federal Housing Administration, which is part of HUD, is working to working to compile a single set of guidelines to clear up lender confusion about when the agency can take action against a lender. Meanwhile, the regulator that oversees Fannie and Freddie loosened some of its policies on mortgage buybacks. Now, lenders cannot be forced to repurchase a mortgage if the loan survives three years with no more than two 30-day delinquencies and if the 36th monthly payment is on time.