It was a widely viewed segment, back in November 2012: David Silverman, president of American Atheists, debated Fox News host Bill O’Reilly about Jesus Christ, Christmas trees, federal holidays, fascism, religion and philosophy. You know, an installment in O’Reilly’s yearly war on the “War on Christmas.”

“It is a fact that Christianity is not a religion; it is a philosophy,” said O’Reilly at one point. Silverman: “So you are going to actually tell me on live television that Christianity is not a religion?”

It got a lot more heated, with the two bashing each other over the winter solstice and whether O’Reilly would approve of the governor of New York state placing a “Winter Solstice sign” on his door. “I have got no problem with it. Okay?” O’Reilly insisted.

Silverman: “Yeah you would. I think you would.”

(Irate) O’Reilly: “Don’t tell me what I think; when I said I didn’t have a problem, I don’t have a problem.”

Looking back at that moment, Silverman thinks he did some damage: “I totally kicked his a– in that interview and he hasn’t had me on since then,” he says. Though Silverman has appeared on other Fox News programs over the past two years, he still insists, “They’re afraid of me. In the past several times I’ve been on Fox News I have hammered them and driven it home and made them look bad.”

The alleged banishment, says Silverman, “strokes my ego like crazy but that’s not what this is about.”

Is this just an activist looking to gain some altitude by taking a pot shot or two at the Undisputed King of Cable News?

Perhaps, but Silverman’s point gained some traction in the 2014 “War on Christmas” series. Earlier this month, O’Reilly ran a segment criticizing Silverman’s group for sponsoring billboards around the country urging children to “Skip Church” on Christmas. To chew over the matter, O’Reilly turned to psychotherapist Karen Ruskin, who proceeded to rip American Atheists over the campaign. “It’s horrifically insulting, it’s really disrespectful but not unlike the bully who tries to push other people down in order to make themselves feel better. That’s what’s happening here,” said Ruskin.

It was an agreefest, nothing even close to the fabulous cable television that O’Reilly and Silverman produced two years before. The O’Reilly-Ruskin segment did include a short statement from American Atheists spokeswoman Danielle Muscato, but why not just have Silverman on the air to defend his billboards? “What we’re doing is obviously newsworthy for us to talk about, but our position is so strong that they’re afraid to bring us on to explain ourselves,” says Silverman.