The White House is catching plenty of flak for its nomination of Chuck Hagel as defense secretary. Not just from Republicans seeking to fight the nomination, but from other corners decrying the lack of diversity at the top of President Obama’s Cabinet.

The Washington Post ran a story Monday that noted the discrepancy between a president who made women’s issues core to his reelection campaign and yet named new top advisers — John F. Kerry as secretary of state, Chuck Hagel at Defense and John Brennan at the top of the CIA — who were all white men. On Tuesday, the New York Times ran its own story featuring a cringe-inducing photo from the Oval Office that looks more like the makings of a corporate softball team than the president’s top group of advisers.

Meanwhile, Bloomberg View columnist Margaret Carlson offered the ultimate critique with an op-ed titled “Obama to Romney: Send me your Binders Full of Women,” writing that “at the rate he is going, Obama is going to have a Cabinet that looks more like the Augusta National Golf Club than America.” Ouch.

The lack of women at the very top of the still-forming new Cabinet is a cause for concern—not just because of the missing diverse inputs they’d bring to the president’s decision-making, but, as I’ve written before, because of the particular contributions certain women could make. However, the debate over diversity in the president’s Cabinet shouldn't just be about how many women there are, but how many new voices are present on a team that has long been criticized as too insular. In naming new Cabinet members, the president needs to balance selecting not just women vs. men, but people he trusts and knows well vs. those who could add fresh perspectives and alternative outside views.

Consider the names he’s already chosen and those who are seen as front-runners for the spots that remain open. Kerry, the senior senator from Massachusetts, may not have served in the president’s Cabinet thus far, but he was a leading surrogate for the president during the campaign and stood in as Mitt Romney in debate preparation.

Hagel, a former senator from Nebraska, has spent recent years as a professor at Georgetown, but he has already been co-chairman of Obama’s Intelligence Advisory Board. Brennan, meanwhile, is currently the White House counterterrorism adviser and has worked in the administration since the 2008 campaign.

Others being considered for top jobs are also insiders. Jack Lew, the president’s current chief of staff, is being nominated to lead the Treasury Department. Deputy National Security Adviser Denis McDonough and former Biden Chief of Staff Ron Klain are both rumored to be possible contenders to then fill Lew’s job. And top names for the attorney general’s job, meanwhile, if and when Eric H. Holder Jr. steps down, include current Homeland Security Secretary Janet Napolitano and current Deputy Attorney General James Cole.

I’m not saying any of these picks would be bad ones. And it’s understandable that the president would want to choose people with whom he’s comfortable, who know the inner workings of an agency or who have been witness to the ups and downs of the top jobs by previously serving in deputy roles. In many cases, it’s simply good leadership to promote people from within and form a team that’s familiar enough with each other that they can get to work quickly.

That said, especially when it comes to forming an advisory team, leaders also need to make sure they’re looking far enough afield in order to get the most diverse insights. As with most things leaders do, selecting a team is all about striking the right balance—between men and women, between various ethnic groups, and between experienced old hands and unconventional voices.

Diversity is not a one-dimensional issue. Those critiquing the president — especially given his traditionally close inner circle — should be just as concerned about the backgrounds of the people on his team as they are about the makeup of their gender.

Jena McGregor is a columnist for the Washington Post’s On Leadership section.