After a challenging week, President Obama left Washington on Friday to try to change the subject from the problems facing his administration to one facing the country: jobs.

In remarks at Ellicott Dredges, a company that which makes dredging equipment, Obama said lawmakers in Washington should spend every single day focused on the economy and helping people get back to work.  "Our focus cannot drift," he said, just after the House Ways and Means Committee held its first hearing on the scandal at the IRS that forced Obama to ask for the acting commissioner's resignation this week.

The White House found itself focused on everything but jobs in the last week, defending itself against continuing revelations about the attacks in Benghazi, Libya, and the controversy over the Justice Department's subpoena of Associated Press phone records.

But Obama was far away from all that Friday, visiting an elementary school and a community center in Baltimore on the second stop in his “Middle Class Jobs & Opportunity Tours,” an attempt to focus attention on his strategy to improve the economy.

He also planned to announce that he has signed a memorandum to modernize the federal infrastructure-permitting process to speed up projects

Last week, the president visited Austin, Tex., on the first stop of his jobs tour. There, Obama said he saw three areas the country needed to focus on.

"Number one, we've got to make America a magnet for good jobs," he said. "Number two, we've got to help people earn the skills they need to do those jobs. Number three, we've got to make sure people’s hard work is rewarded so that they can make a decent living doing those jobs."

The elementary school that Obama visited  provides comprehensive early childhood education, which the president emphasized in this State of the Union address.

While at the school, Obama quizzed students on simple arithmetic. When a student couldn’t come up with the answer to “2 minus 1,” Obama said in a sympathetic tone that “subtraction is tougher than addition.”

Finally, Obama will stop at a community center that helps provide work skills training for parents to help them earn a living.

The Associated Press contributed to this report.