Rep. Paul Ryan (R-Wis.) said in an interview Tuesday that he spoke to Congressional Black Caucus member Barbara Lee (D-Calif.) and that Lee knows that he "doesn't have a racist bone in my body."

The phone call came after some controversy over Ryan's remark that some in the inner city don't have the desire to work because it's not a part of the culture. Lee was among Ryan's chief critics, calling the remark a "thinly veiled racial attack.”

Ryan said in an interview set to air on the "O'Reilly Factor" on Fox News Tuesday night that he spoke to Lee and she knows him better than that.

“She does not believe that I have these views," the potential 2016 presidential candidate said, according to advance remarks from the interview. "She knows me well and she knows that I don’t have a racist bone in my body.”

Ryan added that Lee didn't apologize to him but said he's not mad at her.

"I’m a big boy," Ryan said. "I understand that if you challenge the status quo, that if you get into these issues, sometimes you will be misinterpreted. But also I really believe that we have to have a real conversation about how to fix these things without throwing these charges at people.”