It is not known exactly where in the White House President Trump was Sunday morning when he tweeted that four female congresswomen of color should “go back” to the “totally broken and crime infested places from which they came.” And it’s unclear if he “gazed out at the nation he leads,” as the New York Times’s Peter Baker opined, from the portico or the Oval Office or the residence. Maybe it was a part of the mansion he reportedly described to his golf buddies as “a real dump” early in his tenure. Maybe not.

But one thing is for sure, the building that houses the 45th president, his wife and youngest son was constructed by enslaved people. They were descendants of men, women and children taken against their will from Africa, a continent to which Trump now wishes some members of Congress would return.

This summer, the nation is marking the 400th anniversary of the first enslaved Africans in Virginia. The arrival of “20 And odd Negroes” in Jamestown in 1619 marked the beginning of a brutal subjugation that left millions in chains.

Construction of 1600 Pennsylvania Avenue began 1792, according to the White House Historical Association’s website, after land for the District of Columbia was ceded by slave states Maryland and Virginia.

Planners initially intended to import European workers for the job, but “recruitment was dismal,” the WHHA said. So they turned to the enslaved, who provided “the bulk of the labor” needed for the White House, the U.S. Capitol and other government buildings.

The enslaved laborers started right at the source of the construction materials — a quarry in Stafford County, Va. — where they were trained to cut stone.

At least five of these men are known, listed in wage rolls as Tom, Peter, Ben, Harry and Daniel. The first four were owned by White House architect James Hoban. Daniel was owned by Hoban’s assistant.

But, “no one will ever know how many slaves helped to build the United States Capitol Building — or the White House,” wrote William C. Allen, a historian for the Architect of the Capitol, in a 2005 report.

If this all sounds familiar, you may be remembering that in 2016 former first lady Michelle Obama recognized that she, a woman descended from slaves, lived in a house built by them.

This prompted a pre-disgraced Bill O’Reilly to claim without evidence that “slaves that worked there were well fed and had decent lodgings provided by the government.” The WHHA told The Washington Post in 2016 it knew of no documents proving this.

In the 2016 kerfuffle, some also claimed the enslaved men who built the White House were paid, citing an Independent Journal article that linked to a 2009 PolitiFact story as evidence. But PolitiFact explains there’s only record of one enslaved man receiving compensation for his labor, and he worked on the U.S. Capitol. More importantly, wage rolls were kept to provide compensation to the slave owners for the rental of their property.

Though the White House was ordered constructed at the beginning of President George Washington’s administration, he never lived in it. (It wasn’t finished until 1800, after he had died.) So he took his slaves with him to temporary capitals in New York City and Philadelphia.

That got complicated in the latter city, since Pennsylvania law decreed that any enslaved person brought from another state for more than six months was automatically free. So, as historian Erica Armstrong Dunbar explained in the Times, Washington made sure to take his enslaved staff over state lines every six months so that none would be accidentally freed.

Once the roof on the White House was completed, enslaved people were not sent back to Africa, but down to the basement. That’s where slaves owned by presidents lived with their families for 50 years, according to the White House historians. Seven presidents brought slaves to the White House — Thomas Jefferson, James Madison, James Monroe, Andrew Jackson, John Tyler, James K. Polk and Zachary Taylor.

Read more Retropolis: