Like many metaphors, "the other side of the tracks" was originally a literal epithet. Blacks were often historically restricted to neighborhoods separated from whites by railroads, turning the tracks into iron barriers of race and class.
Look at racial maps of many American cities, and stark boundaries between neighboring black and white communities frequently denote an impassable railroad or highway, or a historically uncrossable avenue. Infrastructure has long played this role: reinforcing unspoken divides, walling off communities, containing their expansion, physically isolating them from schools or parks or neighbors nearby.
Research, in fact, suggests that American cities that were subdivided by railroads in the 19th century into physically discrete neighborhoods became much more segregated decades later following the Great Migration of blacks out of the rural South.
You can see echoes of that pattern in Pittsburgh today on this map drawn using the racial dot map created by Dustin Cable at the University of Virginia's Weldon Cooper Center for Public Service. Railroads there have historically amplified neighborhood divisions created by the city's hilly topography:
Similar railroad divides persist in Hartford, Conn.:
Segregation, in effect, has been built into the physical environment of many American cities, making it that much harder to undo.
A century after many of those railroads were built, the pattern was repeated in a modern form: through the construction of even more imposing highways (many of which both destroyed and separated minority neighborhoods).
Here is Shreveport, La.:
School attendance zones were divided by it, too. Today, the city looks like this:
St. Louis, too, was historically divided by a four-lane boulevard, known as the Delmar Divide, whose significance lingers in the city today:
In Buffalo, New York, that divide runs along Main Street:
In Detroit, the infamous 8 Mile Road is also a municipal dividing line between Detroit and its northern suburbs:
In Washington, D.C., the Anacostia River has long served to isolate and separate black communities on the east side of town.
But another element of the environment — created by government — became over the years a de facto buffer between white and non-white communities: Rock Creek Park in Northwest D.C.
Milwaukee, one of the most segregated cities in the U.S., is carved up by all of these forces: by highways, local roads and railroads, by parks and rivers, often multiple geographic barriers reinforcing each other in the same part of town. As the city's Hispanic population has grown, it has been similarly contained south of downtown, in a patch of the city hemmed in on all four sides by railroads tracks and buttressed by highways and water.
Here, lastly, are just some of Milwaukee's many divides: