Ben Carson, conservative? Yeah, but the acclaimed doctor and author tried to downplay politics at a fundraiser for his Carson Scholars Fund Sunday.

“I don’t think education is liberal or conservative,” he told us. “This is a time when people need to drop the labels and recognize that unless we begin to change the way we do things, we’re going to fall behind. Forget about labels — focus on getting the job done.”

But it was probably inevitable that the word “conservative” would come up repeatedly at a party with former GOP president candidate Herman Cain, Bush consigliere C. Boyden Gray and conservative commentator Armstrong Williams.

“This is what conservative diversity looks like!” laughed Williams as the men posed for pictures in the backyard of Gray’s Georgetown manse.

Cain — charming, dapper, relaxed — happily worked the crowd and then explained why Republicans, preoccupied with fiscal issues, need to reclaim schooling as a ideological issue. “Conservatives have been too silent about the things they believe in to improve education in this country,” he said.“Some of us are going to get loud about vouchers and charter schools.”

While the afternoon was officially a fundraiser  to establish a D.C. chapter of Carson’s scholarship program, it was also a pep rally for Carson himself. The Johns Hopkins neurosurgeon, a featured speaker at CPAC last month, withdrew as commencement speaker at the university last week after protests about his recent remarks about gay marriage. Those comments, he told the crowd, were overblown and mischaracterized by the media: “If you’re black and conservative, you must be a hater.” Carson added that his recent appearances have been greeted by huge, enthusiastic audiences who think the flap is “ridiculous.”

Aside from that brief aside, the bestselling author and his wife, Candy, steered clear of politics and kept the focus on his students. The couple started a reading project and fund almost 20 years ago, giving away 5,600 scholarships to academic stars as young as nine years old. Their goal: Making kids cool for being smart.

“You give a fourth, fifth, six grader a thousand bucks,” explained Carson. “They’re the big man on campus.”