Jamie Dimon, chief executive of JPMorgan Chase, said he’s had it with the Federal Housing Administration and may even stop doing business with the agency.

About six months ago, JPMorgan agreed to pay the government $614 million to settle a lawsuit alleging that it improperly approved thousands of mortgages that were backed by federal agencies, most notably the FHA – a popular source of mortgages for cash-strapped, first-time home buyers.

On Tuesday, Dimon said the bank lost a “tremendous sum of money on FHA,” and he went on to blast the government for taking the bank to court over what should have been nothing more than a “commercial dispute” between the agency and the bank.

“So the real question is, should we be in the FHA business at all?” Dimon said while discussing his company’s earnings on a call with analysts. “We’re still struggling with that.”

The comments highlight an ongoing point of tension between the banking industry and the government at a critical point in the housing market’s recovery. Both sides agree that a full recovery can’t take place until credit is more widely available to potential home buyers, especially the all-important first-time buyers. But they’re struggling to figure out how to make that happen.

The government often points out that lenders demand higher credit scores than required by FHA and other federal agencies that back mortgages. But banks say they’re lending to the least risky borrowers in order to avoid defaults that could lead to the type of lawsuit lodged against JPMorgan and others. Most recently, SunTrust agreed to pay millions of dollars to settle allegations that it failed to comply with FHA standards.

While FHA requires a minimum credit score of 580 for most borrowers,  lenders now typically impose a minimum score of 640 on their FHA lending, Jim Parrott, a former housing adviser in the White House, wrote in an analysis for the Urban Institute. This means that the 13 million people with credit scores between 580 and 640 are unlikely to find lenders willing to make them a loan, Parrott wrote.

Here’s a look at the sharp drop off in FHA loans for the lower credit score borrowers:

The FHA does not make loans; rather, it insures lenders against losses if the loans go bad.

But there’s a rub. If the agency determines that the loans went bad because the lender made mistakes or failed to meet the agency’s quality standards, then FHA can force the lender to pay back or “indemnify” the agency. Moreover, if the Department of Justice finds that the mistakes are a breach of the law, it can sue the banks for big dollar settlements as it did with JPMorgan and the others.

The problem is that the rules that govern when and why the FHA or the Department of Justice take action are not necessarily clear. “If lenders don’t find the rules clear and refuse to lend because of it, then policy makers who care about access to credit are faced with a problem,” Parrott wrote in his Urban Institute analysis.

Dimon said the bank is "thoroughly, thoroughly confused" about how it was treated by the government. (The bank acknowledged some bad conduct in its handling of FHA loans dating back to 2002 as part of the settlement.) Until the government comes up with "some kind of safe harbors or something, we're going to be very, very cautious in that line of business," Dimon said.

JPMorgan, the country’s second largest mortgage lender, did not disclose how much business it currently does with FHA, but Dimon said its FHA lending is “way down.” Overall, JPMorgan reported that mortgage originations had fallen 66 percent during the second quarter compared to the same period last year.

The FHA is working to compile a single set of guidelines that will make the rules more clear. The agency declined to comment on Dimon’s comments Tuesday. But in the recent past, FHA Commissioner Carol Galante has said the agency wants to “make it easier to do business with us, reduce compliance costs and ultimately increase access to credit.”

Jaret Seiberg, an analyst with Guggenheim Securities, said he sympathizes with Dimon.  "Credit can't be widely available if lenders have to worry that each loan could result in litigation that wipes out years of profits," Seiberg said. "It's not a sustainable system right now."