America’s great symbols of capitalism, Wall Street and the New York Stock Exchange, continued to rally this summer as Americans suffered through the overlapping crises of the coronavirus pandemic, unemployment, devastating police violence against Black citizens, sagging retail sales and precarious health care. President Trump even warned in the final presidential debate: “They say the stock market will rule if I’m elected. If he’s elected, the stock market will crash.”

But Democratic nominee Joe Biden countered, “Where I come from, in Scranton and Claymont, the people don’t live off of the stock market.” After pointing out how much billionaires had earned during Trump’s presidency, Biden wondered: “What happens to the ordinary people out there? What happens to them?”

Biden is right to ask this question because a flourishing stock market amid suffering and inequality is nothing new.

The values and outlook of Wall Street bankers, insurance agents, traders and the pro-business press have long been at odds with the welfare of the average citizen. The stock market is often viewed as a shorthand way to assess the health of the overall economy, as Trump is using it during the campaign. Unfortunately, historically it is more accurate to see a rising market as the inverse of the plight of many Americans. In particular, the prosperity of Wall Street has come at the expense of Black Americans and communities of color.

At no time, perhaps, was this more true than in the decades before the Civil War, when largely due to the interests of Wall Street and the cotton trade, New York became the most pro-South and proslavery city north of the Mason-Dixon Line. In fact, similar to plantations along the Mississippi River, much of the city’s wealth depended on the cultivation of cotton grown and picked by Black people.

The rise of Manhattan and Wall Street, like the nation itself, was closely intertwined with the wealth generated by cotton bales harvested by the labor of enslaved people. Brokers directed shipments of thousands of cotton bales from New Orleans and other Southern markets to the textile factories of New England and the United Kingdom. Banks lent credit to White plantation owners and financed land purchases and the buying of Black bodies to work that land. Firms like New York Life insured Southern enslavers against the death or absconding of the people they enslaved.

Wealthy White New Yorkers in the 1800s knew precisely where their wealth came from, which is why they worked so hard to please the slaveholding South. This included support for the controversial Fugitive Slave Clause of the Constitution, which promised that free cities would return runaways who had escaped bondage to find a precarious liberty in the North.

In fact, many on Wall Street actively worked to help enslavers round up Black children, women and men in the shadow of the stock exchange building. Abetted by police officers, judges, marshals, slave catchers and other officials, members of what became known as the New York Kidnapping Club arrested accused runaways and eagerly transported them into Southern slavery, whether they were really runaways or had been born in freedom.

At the same time, city officials allowed the use of New York harbor to outfit ships intended for the transatlantic slave trade, which had been rendered illegal by Congress in 1808. Why? Because Wall Street, city officials and even judges all knew its interests lay in the continuation of the valuable cotton trade with the South.

Equally complicit was the White press. Gerard Hallock’s Journal of Commerce (the Wall Street Journal of its day) could always be counted on to defend the values and best interests of Wall Street despite the human costs, particularly on the 15,000 or so Black residents who called New York home. When the Black press printed stories like that of 7-year-old Henry Scott, seized as a runaway while sitting at his school desk, Hallock ignored such injustices and actually defended the Fugitive Slave Clause and the subsequent Fugitive Slave Law of 1850.

And so, Black New Yorkers were terrorized by police officers, judges and slave hunters in the decades before the Civil War. Unsurprisingly, White New Yorkers voted overwhelmingly against Abraham Lincoln for president and widely defeated a proposal in the same year to grant Black men the right to vote.

Activists called attention to this greed and immorality that endangered the safety and welfare of the city’s most vulnerable citizens. Led by protester and provocateur David Ruggles, other African American activists like Thomas van Renssalaer, William Johnson, James McCune Smith and Philip A. Bell joined forces in the Vigilance Committee to sound an alarm whenever a suspected fugitive had been arrested or whenever an enslaver plowed into the wooden docks along the Battery. Ruggles even started his own magazine, the Mirror of Liberty, to call attention to the machinations of the Kidnapping Club.

But, despite such valiant efforts, the interests of Wall Street prevailed, and the cotton trade grew to massive proportions by 1861. As the national compromise over slavery crumbled, and as the Civil War seemed inevitable with each passing day, Wall Street magnates, led by Moses Taylor, A.T. Stewart and other rich New Yorkers, worked to preserve the Union by appeasing the White South, thus protecting the cotton trade upon which Manhattan’s wealth so desperately relied. They held rallies, organized meetings, traveled to the nation’s capital to meet with congressmen, all in the hopes of avoiding the breakup of the Union by bowing to the demands of Southern enslavers.

When they failed to prevent secession or war, these wealthy New Yorkers barely hid their disdain. Led by the pro-South and virulently racist mayor, Fernando Wood, the city even considered its own secession from a Union that had voted for Lincoln, the “Black Republican.”

When the war erupted in the spring of 1861, White New Yorkers, including many on Wall Street, pledged their support for the Union cause, sweeping under the rug of patriotic fervor their previous support for the South and slavery. Yet, the impulses of the recent past would not be surrendered so easily, and Manhattan soon became the place of publication for one of the nation’s most racist newspapers of the 19th century. The New York Weekly Caucasian almost single-handedly publicized the phrase “White supremacy” and spread its nefarious ideology to subscribers across the country throughout the Civil War and in the years afterward.

Americans, especially citizens of color, today may look at their bank accounts and wonder why their happiness and security are so out of step with the impulses of Wall Street. Ruggles, the antebellum hero, understood the answer: Wall Street protected its own, regardless of the human lives that had to be sacrificed. Those lessons ring true today, reminding us that the cultlike worship of the free market and “shareholder capitalism” are symbols not of the health of the economy, but instead manifestations of the chasm between the wealthiest and the poorest Americans.