On March 13, Conor Lamb delighted most Democrats with his apparent upset win in the special election in Pennsylvania’s 18th Congressional District. But one of Lamb’s campaign promises had raised eyebrows: The candidate openly endorsed replacing Nancy Pelosi as leader of the House Democrats, perhaps because she has long been unpopular. Other Democratic candidates for Congress also have been distancing themselves from Pelosi, suggesting that her future as Democratic leader may be in danger.

Some Pelosi defenders have suggested that sexism lies behind her unpopularity. As Peter Beinart of the Atlantic writes, “Nancy Pelosi does her job about as well as anyone could. But because she’s a woman, she may not be doing it well enough.”

While gender bias may be why some voters judge Pelosi harshly, closer analysis suggests a more straightforward explanation for why she is unpopular: She was the speaker of the House.

House speakers become unpopular over time

The chart below shows the net difference between Gallup favorability and unfavorability ratings for each of the past three House speakers during their service in that role. As you can see, all three lost popularity along a relatively similar trajectory. By the end of two years in the job, speakers tend to be disliked more than liked, regardless of party or gender. (In fact, Pelosi’s net approval ratings were positive for a longer period than they were for both of her male successors.)

Why? That’s not entirely clear. Maybe it’s because they become the symbol of their party — and so the other party relentlessly attacks them in ads. Maybe it is because Congress itself is disliked, because speakers must make unpopular decisions, or because in today’s politically polarized environment, simply being speaker of the House is enough to make people dislike you.

Pelosi, unlike her recent predecessors and successors, has remained House minority leader since her party lost the majority in 2010. Her national approval ratings have stayed low, which certainly could be because of gender bias. But since we can’t compare her with other recent speakers-turned-minority-leaders — others have resigned, lost reelection or left leadership after their party lost its majority status — it’s difficult to know for certain.

National popularity is not as important as you might imagine for leaders’ job security

National opinion may be largely irrelevant to whether congressional Democrats choose to keep Pelosi as leader. Democrats in the House, especially younger ones, have reasons other than gender bias to be unhappy with her.

For instance, Pelosi has made some public relations missteps in recent months, such as initially defending then-Rep. John Conyers Jr. (D-Mich.) against allegations of sexual harassment. In addition, young ambitious lawmakers have been frustrated that Pelosi and her other long-serving colleagues in leadership have closed off avenues for upward mobility. These junior lawmakers may also have weaker personal or professional ties to the minority leader than their more senior colleagues.

Perhaps most important, House Democrats have suffered disappointing elections in recent years. Historically speaking, revolts against incumbent congressional leaders are more likely after election losses; members of Congress hold their leaders accountable for defeat.

This is especially so when the party loses an unexpectedly large number of seats, as happened to Democrats in the House in 2010 and 2016. And indeed, immediately after those two elections, another Democrat challenged Pelosi for the party leader spot. Though Pelosi survived both revolts, some congressional Democrats — including the House’s possibly newest member, Lamb — maintain that the party needs a new face at the top if it ever wants to win control of the House.

Pelosi may well remain her party’s leader after 2018. She has been a prodigious fundraiser and an agile legislative strategist. Her opponents in the House haven’t yet found a viable and willing challenger who could win the votes of a majority of the caucus. If Democrats retake the House in 2018, they may reward Pelosi with the speakership again.

But Pelosi’s boosters should not dismiss her critics as being guided by gender bias or unfair polls. What the public thinks of Pelosi may worry some political observers. But, in the end, the only opinions that matter are those of her party colleagues — and whether they believe someone else can do a better job in her place.

Matthew Green is a professor of politics at Catholic University and co-author (with Douglas Harris) of the forthcoming book “Choosing the Leader: Leadership Elections in the U.S. House of Representatives” (Yale University Press). His most recent book is “Underdog Politics: The Minority Party in the U.S. House of Representatives” (Yale University Press, 2015).