As San Francisco has become the most expensive city in the country, its African American population has shrunk. (Justin Sullivan/Getty Images)

SAN FRANCISCO — The Willie B. Kennedy Apartments are exactly what the neighbors have been wanting: new affordable housing in a market with little of it, homes for seniors in a city flush with young tech, real investment in a historically black part of town that has long been losing its black population.

But residents of San Francisco's Western Addition neighborhood won't be prioritized when the wait list opens today for the 98 studios and one-bedroom apartments for low-income seniors here. Last year San Francisco passed a law that would reserve 40 percent of units in new affordable housing projects for people from the communities surrounding them. It gave hope to residents of this rapidly changing neighborhood that they'd have the first chance to live in the new building, financed in part with federal money. 

This week, however, the Department of Housing and Urban Development said the plan violates fair-housing requirements, a defeat for an idea that officials in San Francisco hoped could be used as a bulwark against gentrification, here and in other increasingly unaffordable cities. The decision pits a civil-rights law written decades ago to protect minorities from discrimination against a city trying to stem their displacement. And it heightens debate over how far cities can go in trying to maintain racial diversity.

In a lottery that could attract thousands of applicants, the neighborhood preference law would have set aside 40 percent of units at the Willie B. Kennedy Apartments for people — regardless of race — currently residing in the district around it, significantly boosting their odds. The effect, in one of the remaining San Francisco neighborhoods with a sizable black population, is that black tenants would have been more likely to receive some of the units. Long-time residents most threatened by the area's rising costs would have more likely benefited from the city's effort to create affordable housing here.

That law would have taken a tool that has been wielded in the past by white communities to keep out minorities to help them remain in their neighborhoods instead. In all-white suburbs, residential preferences historically meant that public housing stayed all-white too. Such policies operated as a subtler form of discrimination, when zoning laws and restrictive covenants barring blacks became illegal. 

In San Francisco, officials hoped they could craft a version of the same idea that would address gentrification forces that were unknown when the Fair Housing Act was passed in 1968.

"We knew what we were getting into with this legislation, and we knew we had a high bar that we had to clear," says Jeff Buckley, the mayor's housing policy adviser. "But we felt it was very important for us to push against established norms out there because of the very unique issues in San Francisco — issues that are becoming less unique as time goes on."

In San Francisco, the African American share of the population has fallen from nearly 14 percent in 1970 to less than 6 percent today. "The black community is dying in this town," says Amos C. Brown, the president of the San Francisco NAACP, which pushed for the neighborhood preferences. "The patient is bleeding and we must do something to provide emergency care."

Housing costs in the city have soared, threatening its diversity, and neighborhoods like the Western Addition are now dotted with high-end developments. According to city data, the number of households in the district making 50-100 percent of the region's median income shrank by 13 percent from 2009 to 2014. Over the same time, the number earning more than twice the median income grew by 35 percent.

This same pattern — with urban neighborhoods growing whiter, more affluent and more expensive — is increasingly apparent in other coastal cities like Los Angeles, New York and Washington. And San Francisco, Buckley says, wanted to create a legally defensible policy that other cities might apply in places like Los Angeles' Boyle Heights neighborhood, or Washington's Anacostia.

In its letter rejecting the idea for the Willie B. Kennedy Apartments, HUD said the neighborhood preferences "could limit equal access to housing and perpetuate segregation." HUD said through a spokesman that the agency struggles with circumstances such as those raised in San Francisco "when we’re asked to consider well-intended policies that may have unintended harmful consequences." But the spokesman said no one at the agency could recall when HUD has ever approved a neighborhood preference.

A similar policy in New York City is also currently facing a lawsuit.

In San Francisco, the law's backers were taken aback at HUD's suggestion that they might "perpetuate segregation" while trying to help people who have long been victims of it.

"What’s happening now has already created segregation," says London Breed, president of the San Francisco board of supervisors, who grew up in the neighborhood around the new project. "What's happening with the lottery system now has created a form of segregation."

Thursday morning, she and several other community leaders gathered in front of the Willie B. Kennedy Apartments to denounce HUD's decision. On Friday, as applications open for the property's lottery without the neighborhood preference, the units here will be equally available not just to lower-income seniors across the city, but from far outside San Francisco as well.

Given the small remaining black population in San Francisco, as well as the extensive competition from beyond the city, it's likely blacks could occupy just a handful of the 98 units here despite the demographics of the neighborhood, and despite the history that blacks were heavily displaced from the Western Addition by waves of urban renewal — often with official promises that they could return one day.

If the city's severe housing pressures remain unchecked, former San Francisco mayor and now-U.S. Sen. Dianne Feinstein (D) wrote HUD on Friday, the trend could "destroy the fabric and culture" of neighborhoods such as the Western Addition.

Bobby Sisk, who works for the nearby Bethel A.M.E. Church that is a partner on the project, says the conflict here challenges what "fair housing" really means. "There’s a difference between a legal definition and a moral definition," he says. "A legal document does not express what is actually happening to folks. It has no feelings, it has no blood, it has no tissue."

With its strict emphasis on applying equal access to individual properties, HUD can fail to account for the ways the rest of the housing market remains deeply unequal, says Leland Saito, an associate professor of sociology at the University of Southern California.

Housing raises the same thorny questions that have surrounded affirmative action in education: Is equal access by itself enough to right historic discrimination? What if equal access perpetuates unequal outcomes? To create truly fair housing, Saito argues, cities will have to try exactly the kind of proactive policies San Francisco had in mind, which implicitly acknowledge that maybe some groups should get advantages.

"The fair housing law is incredibly important," Saito says. "But what’s at issue here is 'how can we apply the fair housing law in ways that take into account how race works in our society?'"

Ideas like San Francisco's can make other fair-housing advocates nervous, because just as the city wanted to create a model for other communities fighting to preserve diversity, it could open the door anew to older forms of discrimination.

"Neighborhood preferences in the age of a color-blind Supreme Court are dangerous," says Myron Orfield, a legal scholar the University of Minnesota. "Why on earth wouldn’t a white suburb put a residency requirement on their subsidized housing? The Supreme Court — until Merrick Garland gets in there — would have a hard time saying 'if one’s okay, why isn't the other one?'"

Inside the Willie B. Kennedy Apartments, the floors are polished, the appliances are plugged in and the blinds are drawn, awaiting the new tenants and the political wrangling over the lottery. Don Falk, head of the Tenderloin Neighborhood Development Corporation that developed the building, walked through the finished units for the first time Thursday morning. Each has floor-to-ceiling windows, a spacious kitchen and honest-to-god closest space.

"People who win the lottery," Falk says, standing in an airy second-floor unit, "really do win the lottery."