President Barack Obama, accompanied by Health and Human Services Secretary Kathleen Sebelius, on Feb. 10, 2012. (Pablo Martinez Monsivais/AP)

The diversity of President Obama’s second-term Cabinet, which has been taking shape in recent weeks, has been getting plenty of scrutiny. So much so, in fact, that it was the subject of one of just seven questions asked of the president during Monday’s news conference. “I’m very proud that in the first four years, we had as diverse, if not a more diverse White House and Cabinet than any in history, and I intend to continue that,” the president said. He urged critics not to “rush to judgment” until the full slate of new nominations came together.

The defense came in response to multiple critiques of the way the new Cabinet is taking shape. Some say it’s too insular, with usual suspects rather than a team of rivals, and thus is positioned to offer the president little more than a “yes-man” response. Others have said it’s too white, as the top ranks fill with white advisers — and as figures like Labor Secretary Hilda Solis (the first Latin American female Cabinet member), Lisa Jackson (the African-American head of the Environmental Protection Agency) and Energy Secretary Steven Chu (the Asian-American scientist who has won the Nobel Prize) either resign or are expected to do so.

And of course some say it’s too male, as pundit after pundit has pointed out, with the top four positions (secretary of defense, secretary of state, secretary of the Treasury and attorney general) all now held or expected to be held by men.

Much of the criticism seems directed at the discrepancy between the president’s reliance on women to get re-elected and the lack of top women being named to his team’s senior ranks. As a result, the focus seems to be more on how female voters will view the nominations — the so-called “optics” of the Cabinet — rather than the impact on other women who already advise the president or remain on staff.

And yet that’s precisely what should be of more concern. I don’t think many American voters are going to decide to vote or not vote Democratic based on whether the president puts a certain number of women into top slots in his Cabinet. Rather, they will be watching to see what the president does on issues that matter to women — health care, equal pay, the economy, education and the like. Having more women in the Cabinet should, of course, help him to focus on those issues, and would offer greater diversity of input into ensuring those issues don’t get short shrift. But that’s what matters to voters more than optics.

Where optics do matter is with the message this sends to current women on the president’s team. I don’t know exactly how the latest picks have affected the feelings of women already in the White House. But at least one reporter says she understands it’s not good. Sunday on “Meet the Press,” Andrea Mitchell said that after a recent story of hers addressed the diversity problems, she “did not get one complaint” from the administration. She said she “talked to several people inside the White House — women — and they said ‘No, we didn’t have any problem about what you wrote about this week.’ The women are not happy.”

I personally think a president who heavily weighed naming Susan Rice as his secretary of state is not trying to keep women out of top posts in his Cabinet. If the final second-term team does appear to be too male when all is said and done, we can and should judge him then. But as he decides the makeup of his team, the president’s concerns should be just as much about how the nominations look to people on the inside as they do to those on the outside.

If talented female staffers sense any kind of closed doors at the top, or begin to think they’re not getting the seats at the table or the attention of the room when they should, the problem becomes not just one of optics and a lack of diverse input but a morale problem as well. This is not a brand new issue for this president — the early days of his presidency were marked by friction with female staffers that was featured in a book about the Obama White House. Let’s hope he learned from that experience and won’t let it happen again.

Jena McGregor is a columnist for the Washington Post’s On Leadership section.

All the president’s men: Obama’s Cabinet diversity issues