Clarence Lusane, author of "The Black History of the White House," isn't one of those people.
The chair of Howard University's Political Science Department, Lusane has done extensive research on the enslaved people who built the structure and later lived among 10 of the United States' first 12 presidents.
He called the first lady's comment a "pivotal moment" in U.S. history.
"I'm glad that she mentioned the role of enslaved Americans at the White House, because she presented a larger audience with a history that most people are not being taught in our schools," Lusane, also a professor emeritus at American University, told The Washington Post. "I certainly wasn't taught that not only were many of our presidents slave owners, but that the most renowned building in our nation was, in part, built by slave labor."
Unlike at the U.S. Capitol Building and the site of the Liberty Bell in Philadelphia, "there's nothing at the White House that acknowledges its slave history, and perhaps a million each year visit the site," Lusane added.
While the history of slavery at the White House isn't widely known, historians say there's no debate about the accuracy of the first lady's comments.
Even Fox News host Bill O'Reilly partially agreed with Obama, acknowledging on "The O'Reilly Factor" Tuesday that her statement about slave labor at the White House was "essentially correct," according to Media Matters. But O'Reilly disagreed with the first lady's framing, telling his viewers that enslaved peoples at the site were "were well-fed and had decent lodgings provided by the government, which stopped hiring slave labor in 1802."
He also noted that there were white laborers "working" on the site as well.
O'Reilly also failed to cite and historical records to bolster his claims about the humane treatment of people whose very existence was by definition inhumane.
O'Reilly's comments provoked a furious backlash on Twitter.
The White House Historical Association’s website says that when planners struggled to recruit European labor, they “turned to African Americans — enslaved and free — to provide the bulk of labor that built the White House, the United States Capitol, and other early government buildings.”
Construction on the president's home, the site notes, began in 1792.
The precise number of enslaved people forced to work during the multiyear construction is unknown, but Lusane told The Post that his research shows enslaved workers were extensively involved in the effort to develop Washington at the end of the 18th century.
"We know quite a bit, including the names of a number of the people who were enslaved. Some of them were skilled laborers, such as those who worked in carpentry or masonry," he told The Post. "We have the payment records from the people who owned them."
The White House Historical Association said slaves were trained at the government’s quarry in Aquia, Va., to cut the stone that was later laid by Scottish masons to create the "walls of the president's house." The construction force included white laborers from Maryland and Virginia and immigrants from Ireland and Scotland, the association added.
The construction process forced enslaved people to endure backbreaking labor, Lusane said, such as cutting down trees, dredging swamps, removing dirt and rocks and bringing materials to the site from distant rock quarries.
"There would have been a sizable number of enslaved people involved," Lusane added. "They were building the city as a whole. It took 10 years, and you can be pretty sure that given the work — and the possibility of injuries, diseases, and accidents — that people died."
In 2005, PolitiFact noted, a congressional task force issued a report, entitled "History of Slave Laborers in the Construction of the United States Capitol," that found "plenty of evidence of slave involvement in the Capitol's construction."
"Perhaps the most compelling evidence were records of payments from the commissioners for the District of Columbia — the three men appointed by George Washington to oversee the construction of the capitol and the rest of the city of Washington — to slave owners for the rental of slaves to work on the capitol," PolitiFact reported. "The records reflect 385 payments between 1795 and 1801 for 'Negro hire,' a euphemism for the yearly rental of slaves."
The task force concluded that nobody will ever know the precise number of slaves used in the construction process, but it found that the brutal labor closely resembled the kind used in the construction of the White House. From PolitiFact:
"Slaves were likely involved in all aspects of construction, including carpentry, masonry, carting, rafting, plastering, glazing and painting, the task force reported. And slaves appear to have shouldered alone the grueling work of sawing logs and stones."Slave crews also toiled at the marble and sandstone quarries that provided the stone to face the structure — lonely, grueling work with bleak living conditions in rural Virginia and elsewhere. 'Keep the yearly hirelings at work from sunrise to sunset — particularly the Negroes,' the commissioners wrote to quarry operator William O'Neale in 1794."
In an interview with C-Span in 2011, Lusane detailed the intimate and coercive relationship many of the nation's early leaders had with the people they kept in bondage. In his book, he tells the story of Oney Judge, a woman enslaved by President Washington who managed to escape. Washington attempted to organize her kidnapping, Lusane said, but Judge — who was living as a free woman in New Hampshire at the time — eluded her former master and would eventually learn to read and live into her 80s.
Her decision to escape the grip of the nation's first president, she would later tell interviewers, was inspired by the American Revolution, Lusane said.
Hundreds of years later, Lusane told The Post, stories like Judge's do not feature prominently in American history because of our insistence on presenting presidents in a positive light.
And yet, he noted, their physical legacy remains.
"As it turns out, slaves became extremely skilled because they did the work around the plantations, around the farms. All of the large structures that were built up and down the East Coast — from libraries to universities to city halls to mansions — were built by slave labor. That meant people had carpentry skills, masonry skills, even some architectural and design skills."
The first lady brought some of that legacy alive Monday night, correcting a historical record in a way that historians like Lusane would like to see memorialized at the White House so that the nation can finally move forward.
"What struck me was that her remarks were unique in terms of the perspective of a woman of color," he said. "It's hard to imagine someone who was not a woman of color giving that particular speech. She did it in a way that was perfectly toned, and she talks about the country addressing issues of difference without exacerbating those issues.
"It's a history lesson that is so valuable," he added.