When I wrote my article on the subtle, sexist whispering campaign against Janet Yellen, my example of a rare not-so-subtle sexist comment came from Richard Fisher, president of the Dallas Federal Reserve. Asked about Yellen on CNBC, he worried that the pick would be "driven by gender," even as he allowed that she was "extremely capable."

Fisher's camp feels there's context that I didn't capture in Fisher's quote. So here's the full exchange, from CNBC's Squawk Box on May 20, 2013:

MR. SORKIN: We had a big conversation about whether Ben Bernanke will remain the chairman, whether Janet Yellen would be the next one. Do you want to offer us a little insight as to how you think people are thinking about this?
MR. FISHER: I personally would like to see Ben stay. I think he's extraordinary. He's progressed enormously on the job. He was given a pile of X, and he's sorted his way through it. I admire him immensely. And the question is, can someone else do this job better, particularly given what has been learned as a practitioner in the job, as we've done something that's never been done before?
So that would be my preference. Whether it'll happen or not, time will tell. It's a presidential decision, and we'll see if it's driven by gender or other considerations and so on. Janet's extremely capable. There are other capable people. I can tell you one thing for certain. It won't be me.

People get understandably defensive when you use the word "sexist." But here's the simple fact: There's no male candidate Andrew Ross Sorkin could've named who would've elicited fears from Fisher that the pick was being "driven by gender." Not Don Kohn. Not Alan Blinder. Not Roger Ferguson. It would've been either a laugh line or a controversy if Sorkin had asked about Tim Geithner's chances and Fisher had brought up his gender.

When a woman is up for the job -- no matter how qualified -- the lurking worry is that the pick will be "driven by gender." When a man is up for it, gender never enters into the conversation. That's how privilege works in practice: Gender is invisible when it comes to male appointees but a constant presence when it comes to female appointees.

That gets the situation backwards.

There's never been a female Federal Reserve chief. There's never been a female Treasury Secretary. There's never even been a female president of the powerful New York Federal Reserve. This isn't an accident, and it's not because women can't manage those positions. It's because gender really has played a driving role in appointment processes for a long time. It's just done so on behalf of men.

Decades ago, that role was explicit. Women weren't even considered. Today, it's subtler: It's in the way the characteristics we associate with leadership are stereotypically male. It's in the way boys and girls are pushed towards different subjects (think about the famous, and oft-replicated, experiments that show women score lower on math tests when they have to identify their gender before they begin). It's in the way that professional networks function, with men going to play golf or basketball on the weekend, and not even thinking to invite their female subordinates. It's in the way that Janet Yellen isn't seen as having "gravitas" despite her long experience at the Fed.

Because no one involved hews to gross and outdated notions of female inferiority, the process is considered free of sexism -- and woe to anyone who suggests otherwise. But it's nevertheless a process that's heavily influenced by gender, that ends with men in a hugely disproportionate number of top positions, and that manages to turn the gender card around on any women who manages to make it through to the end. Neat, huh?

Those last few paragraphs are easy enough to nod along with. The problem, as always, is when this critique goes from the general to the specific. It's fine to say that there's lingering, institutional sexism is society. But it feels ridiculous and offensive to say that the lingering, institutional sexism is why Alan Krueger ends up chairing the Council of Economic Advisers rather than Rebecca Blank.

And, in a way, it is ridiculous and offensive: Krueger is immensely qualified, and the decision was based on many, many different factors. And no one wants Krueger to pay for the legacy of generations of overt sexism. But it repeats itself again and again, with many different candidates across many different positions. If you don't think gender is playing a role at some point in the process, than you have to admit to believing some genuinely sexist ideas about men being more innately fit for leadership positions.

And then, when a female candidate does threaten to break into that top echelon, the whispers begin that the pick is really driven by gender, that more qualified men are being passed over, that it's all just about political correctness. But the truth is typically closer to the opposite. A woman (or an African American, etc) can only get to the point where she can hold a top position after clearing a much higher bar than the male candidates.

If you take Yellen as an example, she has more direct Fed experience than Ben Bernanke did, or than Alan Greenspan did, or than Paul Volcker did. In order to be competitive for the job, she needed to be much more qualified for it. And even so, she still has to face concerns that her pick would be "driven by gender," or that she lacks "gravitas."