DALY CITY, CA - JULY 12: Customers use ATMs at a Wells Fargo Bank branch office on July 12, 2012 in Daly City, California. The Justice Department announced Thursday that Wells Fargo Bank, the largest residential home mortgage originator in the United States, will pay nearly $175 million to settle accusations of discrimination against qualified African-American and Hispanic borrowers between 2004 and 2009. The alleged discrimination is in violation of fair-lending laws. (Photo by Justin Sullivan/Getty Images) (Justin Sullivan/GETTY IMAGES)

In one of the largest fair-lending payouts in history, Wells Fargo agreed on Thursday to spend at least $175 million to settle federal accusations that it steered black and Latino borrowers into high-cost loans and charged them excessive fees.

The settlement with the nation’s largest home mortgage lender is rooted in a lawsuit filed four years ago by Baltimore over fair-lending violations. It culminated Thursday in what federal officials called “systemic discrimination” spanning 36 states and involving more than 34,000 minority customers over five years.

“This is a case about real people — African American and Latino — who suffered real harm as a result of Wells Fargo’s discriminatory lending practices,” said Thomas E. Perez, assistant attorney general for civil rights.

The Justice Department identified about 4,500 black and Hispanic Wells Fargo homeowners in the Washington and Baltimore regions who were targeted for unfavorable loans or charged what Perez dubbed a “racial surtax.” As part of the settlement, Wells Fargo is directing $50 million to the Washington region, Baltimore and six other metropolitan areas to help residents afford down payments for new homes.

However, the bank has denied any wrongdoing. In the settlement, Wells Fargo said it treated all of its customers fairly but that it wanted to avoid protracted litigation and provide “important and meaningful” assistance to borrowers. The payout represents 4 percent of the bank’s $4.2 billion profit during the most recent quarter.

“Our commitment to our customers and to turning the housing market around is stronger than ever,” Mike Heid, president of Wells Fargo Home Mortgage, said in a statement Thursday.

The settlement carried particular significance in Baltimore. The city was among the first to accuse the bank of discriminatory lending, spawning similar lawsuits across the country. The case prompted Justice and the Office of the Comptroller of the Currency to launch their own investigations, which they later combined into the complaint that was settled Thursday.

In a separate move, Baltimore withdrew its lawsuit Thursday after Wells Fargo agreed to provide an additional $3 million in homeowner assistance to residents. It also committed to making $125 million in prime loans to low- and moderate-income borrowers in the city over the next five years.

“This in a way brings to a close a certain chapter,” said John Relman, who led the city’s case against the bank. “This is the way you start to make things better on the ground.”

Justice’s investigation centered on Wells Fargo mortgages made between 2004 and 2009 by independent brokers. It found that highly qualified black borrowers were four times as likely to receive a subprime loan as similarly qualified white applicants. Hispanic borrowers were three times as likely to get a subprime loan. Those mortgages carried higher interest rates and unfavorable terms that often resulted in borrowers falling behind on payments or losing their homes altogether.

Of the 34,000 minorities that Justice said were subjected to discrimination, 4,000 were improperly steered into subprime loans. It estimates they will receive an average of $15,000 each under the settlement.

The problem, Justice found, was that brokers could earn more money making subprime loans than by originating prime mortgages. The company capped compensation at 4.5 percent of a prime loan, while the cap for subprime mortgages was 5 percent. That translated into a $1,500 difference for an average loan of $300,000 and created an incentive for brokers to originate as many subprime loans as possible.

Wells Fargo tried to address the issue by creating a filter intended to identify prime customers. But according to Justice, brokers easily bypassed the process by encouraging borrowers to skip the down payment or withhold certain documents — steps disqualifying them for a prime loan. Often borrowers were not told that they were eligible for a better interest rate.

Federal investigators said senior Wells Fargo officials knew about those practices but did little to stop them.

In addition, even when black and Hispanic customers got prime loans, they paid higher fees than white borrowers, Justice alleged. The average African American taking out a $300,000 prime loan was charged $2,064 more in broker fees than a similarly qualified white customer. Latino borrowers paid an average of $1,251 more.

“The department’s action makes clear that we will hold financial institutions accountable, including some of the nation’s largest, for lending discrimination,” Deputy Attorney General James M. Cole said.

Wells Fargo said Thursday that it will no longer work with independent mortgage brokers, which originate about 5 percent of its loan volume.

As part of the settlement, it is also reviewing subprime mortgages made by in-house loan officers to determine whether any black or Latino borrowers may have qualified for prime mortgages instead. The compensation for those customers would be added to the $175 million settlement.

Justice estimated that identifying and distributing the payments to consumers would take about a year. But it also acknowledged that the long-term consequences to minority communities could take far longer to repair.

The Washington Post reported this week that a growing chorus of consumer advocates, economists and civil rights leaders are raising concerns that the fallout from the boom in subprime lending has endangered the financial futures of black Americans by lowering their credit scores.

“The impacts of lending discrimination and the harm to a person’s credit can be far reaching — inhibiting a range of opportunities that affect a person’s ability to find housing, good employment or access higher education,” said Perez, the assistant attorney general.

The Wells Fargo agreement comes less than two months after Justice negotiated a $21 million settlement with SunTrust Mortgage over similar allegations. Last year, it reached a $335 million agreement with Countrywide Financial, now owned by Bank of America, in what is the largest fair-lending settlement in the department’s history.