House Speaker Paul D. Ryan spent Wednesday afternoon touring a homeless shelter with embattled Rep. Rod Blum in Waterloo, Iowa. The pair explained how the House GOP agenda could help alleviate poverty in Blum’s district.

At the same time, seven of Blum’s colleagues in the House Freedom Caucus were holding a private meeting here in Washington in the apartment of North Carolina Rep. Mark Meadows (R), who led the effort to dump Ryan’s predecessor as speaker.

Freedom Caucus Chairman Jim Jordan (R-Ohio) called the secretive meeting with little notice, causing some attendees to rush back from their districts. Several lawmakers weren’t even aware of the session and reporters scurried across the city scouting potential locations.

The meeting’s overt topic was defending vulnerable Republican colleagues ahead of Tuesday’s election. But there was no avoiding the elephant in the room — the debate over whether the group should try to unseat Ryan as speaker.

“Of course his name came up,” Rep. Scott Perry (R-Penn.) said after the meeting. “He’s in the news every day — he’s the speaker.”

A reluctant leader of his fractious conference, Ryan has waged an aggressive charm campaign to capture the support of the 40-something hardline conservatives who form the Freedom Caucus, a perpetual thorn in the side of the House GOP leadership since the tea party wave sent most of them to Capitol Hill in 2010.

The Wisconsin Republican has personally  campaigned for Freedom Caucusers in their districts and listened to them here in Washington, fielding their late-night texts and chatting about the budget over beers in his office. He added a second weekly meeting on the last day each legislative week where members can vent their frustrations and hash out differences before heading home for the weekend.

All that has not been enough, however, for the hardliners, who want to see the House GOP take an uncompromising stance on everything from spending cuts to immigration, with no room for negotiation.

Several Freedom Caucus members have publicly floated plans to try to deny Ryan the speakership in their party’s leadership election scheduled for the week of Nov. 14. They accuse Ryan of breaking his promise to restore “regular order” to the spending process and blame him for not doing more to avoid a budget negotiation likely to end in a massive spending bill negotiated with Democrats at the end of this year.

“Ryan called last year’s budget deal a crap sandwich,” said Rep. Dave Brat (R-Va.), who unseated former Majority Leader Eric Cantor in a 2014 primary. “This year it is going to be a crap sandwich with extra portions and Obama at the table before he leaves office.”

It is difficult to estimate just how many Freedom Caucusers actually support Ryan’s ouster. The guarded group keeps their official membership rolls a secret, leaving it up to members to disclose their affiliation. At least five of them, including Blum, have publicly supported reelecting Ryan as speaker.

Many of these hardliners hail from deep-red districts where GOP nominee Donald Trump is a hero and voters cheer chaos in Washington. They often say their supporters don’t fear deep budget cuts or the specter of a government shutdown. That dynamic gives them license to adopt a take-no-prisoners form of politics antithetical to the political finesse typically required to pass legislation.

A little over 12 months after taking the job, that leaves Ryan in much the same position as Boehner — well-liked but at the hardliners’ mercy. Next year, their clout is only likely to grow if, as expected, the House GOP loses seats on Tuesday — those losses won’t come in the hardliners’ districts. 

Ryan insists he is running again for speaker, despite growing whispers that he will choose to bow out rather than wrestle to win the job again, only to fight endlessly with a large minority of his members. 

“I am going to seek staying on as Speaker,” Ryan told WTAQ’s Jerry Bader, while stumping for GOP candidates in Wisconsin. “This is the typical chatter you have every two years,” he added of recent suggestions on Capitol Hill that he’ll bow out of GOP leadership. “They call it ‘palace intrigue’ in the Hill rags.”

The bomb-throwing approach is wearing on the nerves of Ryan allies tired of being blamed for a dysfunctional Congress.

“It is frustrating,” said Rep. Adam Kinzinger (R-Ill.). “We have to quit governing from crisis to crisis.”

Kinzinger, who is close to Ryan, said that not all Freedom Caucusers are unreasonable but some won’t be satisfied with any outcome the group hasn’t endorsed.

“You can’t on the one hand tout how you love the Constitution and on the other hand basically believe that in a divided government — or even a government with all one-party rule — that you aren’t going to have to compromise,” Kinzinger said. “If you love the Constitution, you have to be open to compromise. They’re not mutually exclusive.”

Most conservatives say Ryan has done a lot of things right. He has an open door policy — during the day members can drop in to his office in the Capitol, at night they can find him in his personal office where he still sleeps during the week. Members text him, they call him and they have personal relationships with his chief of staff, David Hoppe.

Earlier this year Ryan invited members to his office for an informal “budget and beers” gathering where they chatted the budget process ahead. He’s also personally donated to several Freedom Caucus members and made campaign appearances for them.

Since 2014, Ryan’s leadership PAC has donated just under $100,000 to Freedom Caucusers, according to a Republican source familiar with the spending. Those donations, the source said, have gone directly to over a quarter of the hardliners, including Reps. Alex Mooney (R-WestVa.), Barry Loudermilk (R-Ga.), Mark Sanford (R-S.C.), Warren Davidson (R-Ohio) and Mick Mulvaney (R-S.C.), one of the lawmaker’s at Wednesday’s meeting.

Ryan also headlined a fundraising event for Jordan’s February birthday in Washington.

But the hardliners want their friendship with Ryan, which they admit is a real thing, to translate into actual policy influence.

“Everyone likes Paul,” Perry said after the meeting. “We have a bigger mission here. It’s not personality based”

That mission includes changing the rules of the House to allow groups like theirs to wield more power. They want to better access to leadership slots and committee gavels, and they want the chance to force votes on conservative legislation. They also want Ryan to join them in the fight.

Many see the upcoming leadership election and spending negotiation as a litmus test for Ryan. They worry the year-end budget negotiation will end with a massive omnibus spending bill filled with a litany of concessions to Democrats — they would rather vote on several smaller spending bills.

“I think everyone has honeymoon phase,” Meadows said in September before talk of Ryan’s ouster took off. “You try to give people a little bit of latitude until the results show something different.”

Some Freedom Caucusers, like Rep. Trent Franks (R-Ariz.,  said that even if they disagree with Ryan, it’s misguided to think that someone else would better be able to solve the House’s problems. Franks said he warned members of that back when they started the charge to strip Boehner of his gavel.

“I said to them, ‘If we do this you understand we’ll be here again,’” Franks said. “It’s not always the personality that’s the problem here. It is the systemic issue that plagues the person in charge no matter who they are.”

Franks compared the debate over Ryan to the disagreement within the party over endorsing Trump. He believes Republicans can debate behind closed doors, but publicly they should unite to protect their majority power.

“It is vital that when we’re choosing in a binary choice — between someone who is not perfect and someone that’s’ a disaster — that we make the right choice,” Franks said. “I think that cedes power to the Democrats.”

Democrats are happy to take advantage of the GOP divide.

Over the course of the two years that Republicans held comfortable congressional majorities , Democrats were able to negotiate $80 billion in spending increases. They used last year’s year-end spending bill to maintain tax credits for wind and solar energy that Republicans abhor and reauthorize a land and water conservation fund. Democrats also blocked nearly two dozen politically-motivated initiatives including measures to weaken campaign finance laws and ban the U.S. from accepting refugees from Syria.

“At some point it [obstructionist tactics are] going to force the conference to go to Democrats and say ‘What do you need so we can get some things through?’” Kinzinger said.