Attorney General Eric Holder on Thursday brushed aside questions about when he might step down, suggesting he will do so when he's too tired to continue or when he's accomplished his goals.

Holder was pressed repeatedly at a hearing by Senate Appropriations Committee Ranking Member Richard Shelby (R-Ala.), who suggested Holder might be too much of a distraction right now. Holder left open the possibility but said he's quite capable of leading the Justice Department amidst controversy.

"The tipping point might be fatigue; you get to a point where you just get tired," Holder said. "But beyond that ... there are certain goals that I set for myself and this department that I set in 2009. When I get to the point where I’ve accomplished all the goals I’ve set, I will sit down and talk with the president about a transition to a new attorney general."

Shelby pressed Holder on whether he can continue as attorney general given his unpopularity and the distraction he has created.

Holder said the department is running just fine with him at the helm.

"In spite of the recent controversies that you’ve mentioned, the department is fully engaged in the work of protecting the American people," Holder said, "and I am fully engaged in that regard."

Recent polls have shown Holder is pretty unpopular among the Americans who are familiar with him. A new NBC News/Wall Street Journal poll shows 10 percent of Americans view him favorably, while 32 percent view him unfavorably.

Holder is currently dealing with controversies over the Justice Department's monitoring of journalists at the Associated Press and Fox News, and late Wednesday news broke that the Obama administration used a secret court order to collect information on all phone calls made on the Verizon network.

Update 2:09 p.m.: White House senior advisor Valerie Jarrett said in an interview with HuffPost Live on Thursday that Holder "will be in his position for quite a while":