Housing and Urban Development Secretary Ben Carson decried political correctness during a congressional hearing Tuesday instead of apologizing for controversial comments he made recently about transgender people.

Asked by Rep. Jennifer Wexton (D-Va.) on whether he felt the need to apologize for his remarks about “big, hairy men” trying to infiltrate women’s homeless shelters during a meeting with HUD staff in San Francisco last month, Carson declined.

“No,” he said, his hands clasped at the table where he had sat for nearly three hours during the House Financial Services hearing about affordable housing. “I think this whole concept of political correctness -- you can say this, you can’t say that, you can’t repeat what someone said -- is total foolishness, and it’s going to destroy our nation, and we need to be more mature than that.”

Carson explained he was simply relaying the concerns of a women’s group from Alaska, whose name he could not recall, during an internal HUD meeting. He clarified he was not referring to transgender women as “big, hairy men.”

“I didn’t describe transgender women that way,” Carson said. “I was relating a story that a women’s group told me about big, hairy men -- who are not transgender women by the way -- coming into their facility and having to be accepted because of the rules that were in place.”

The Washington Post first reported in September that Carson’s remarks about transgender people had visibly upset many of the roughly 50 HUD staffers who attended the San Francisco meeting, and prompted at least one woman to walk out in protest.

During that meeting, Carson also lamented that society no longer seemed to know the difference between men and women. Several staffers present told the Post they had interpreted Carson’s remarks as an attack on transgender women. Carson told the group single-sex shelters should have the discretion to turn away transgender people.

In May, HUD introduced a proposal that would allow federally funded shelters to deny people admission on religious grounds or force transgender women to share bathrooms and sleeping quarters with men.

Carson has a history of making dismissive comments about transgender people. While running for president, he referred to transgender people as “abnormal" and said they should not be in the military.

In the days following an internal and public uproar over Carson’s comments in San Francisco, Carson defended himself in an agency-wide email. HUD staffers said they were disappointed that he did not express any remorse.

Transgender advocates have said Carson invoked an insulting trope in his reference to men pretending to be women to gain access to battered women’s shelters.