It’s been a rough few weeks for the federal government.

First, the lavish spending of the General Services Administration on a wild Las Vegas retreat came to light. Then came the Secret Service scandal where a number of agents advancing the President’s trip to Colombia were caught with prostitutes.

WASHINGTON, DC - APRIL 18: Members of Bankrupting America dress as a clown and a mindreader as they hand out "resumes" to make fun of a hearing on General Services Administration (GSA) today in front of Dirksen Senate Office Building April 18, 2012 on Capitol Hill in Washington, DC. (Photo by Alex Wong/Getty Images)

Presumptive Republican presidential nominee Mitt Romney is starting to ramp up his rhetoric on the subject, suggesting that President Obama needs to take more forceful action.

“I’d clean house,” Romney told conservative radio talk show host Laura Ingraham on Wednesday morning. “The right thing to do is to remove people who have violated the public trust and have put their play time and their personal interests ahead of the interests of the nation.”

(Romney was more measured on the Secret Service during another radio interview in Ohio today; “We are a nation, after all, under law and the president has confidence in the head of the Secret Service, as do I,” Romney said.)

Romney’s comments come less than 24 hours after White House press secretary Jay Carney addressed the twin scandals. Said Carney:

“The President believes that everyone who serves the American people by working for this government needs to hold themselves to the highest standards of public service. And there’s no point in comparing the singular incidents of one agency to another, but that principle is one he made clear during the campaign that he would bring to the office. It is a principle that he clearly set forth early on his presidency both in the words that he spoke and the actions that he took, and it is a principle, as I think was made clear in the wake of the GSA incident, that he believes should be enforced.”

White House officials note that the GSA Administrator resigned, her two top deputies were fired and four other officials were put on leave in the immediate aftermath of the Inspector General’s report detailing the agency’s wrongdoing — moves that leave little room for Romney to criticize.

Obviously, both scandals are too recent — and the full scope of each remains too unclear — to draw concrete conclusions about what they might mean (or not mean) to President Obama’s political prospects this fall.

But it is worth noting that one of the central pillars of Obama’s election in 2008 was competency.

After eight years of President Bush — and the debacle over FEMA director Michael Brown in the wake of Hurricane Katrina — Obama sold the idea of his Administration as a meritocracy from top to bottom.

His embrace of the “Team of Rivals” idea for his Cabinet — most notably in his selection of Hillary Rodham Clinton as Secretary of State — affirmed that idea. As we wrote during the early days of Obama’s presidency:

“Obama seems far more focused on ensuring that his nominees have impeccable credentials and a readiness for the job rather than that they fit into a specific ideological box or share a particular vision on the issue (or issues) they will oversee in his Administration.”

“The American public, long weary of the perceived mismanagement and incompetence of Bush (Katrina, war in Iraq, the economy), is reacting well to Obama’s approach to the transition thus far.”

Since that time, there’s been a relative paucity of polling that allows us to track with any certainty whether there has been any significant erosion in the public’s trust in Obama as a competent manager.

In a national poll released on Tuesday, CNN did ask whether people trusted Obama or Romney more to “manage the government effectively”; 46 percent chose Obama while 38 percent opted for Romney.

Romney will, undoubtedly, spend most of his time between now and November making the case that he is the ultimate turnaround artist (Winter Olympics, Bain etc) and that the last four years have made clear that President Obama lacks the experience to make the economy work again.

Competence is at the core of that argument. If Romney can use the GSA and Secret Service scandals to erode confidence in Obama’s ability to competently manage the affairs of the federal government — the most basic task of any president — his challenge of unseating the incumbent becomes considerably easier.