Rep. Paul Ryan met Wednesday with a group of hard-right lawmakers who hold the power to block his ascension to the speaker’s chair.

The meeting with the House Freedom Caucus, held just off the House floor, broke up late in the afternoon with an ambiguous conclusion, setting up a high-stakes decision for a group that played a key role in easing current speaker John A. Boehner into retirement and blocking Majority Leader Kevin McCarthy’s bid to succeed him.

Caucus members leaving the meeting said they plan to meet later in the evening to further discuss whether to abandon their endorsement of Rep. Daniel Webster of Florida in order to back Ryan, who said Tuesday that a Freedom Caucus endorsement is a prerequisite for him to agree to serve as speaker.

“We need to decide whether Paul Ryan is the kind of guy who is going to change this place,” Rep. Raul Labrador (R-Idaho) said after emerging from the meeting.

Ryan (R-Wis.) said little upon departing, calling the gathering an “exchange of ideas on how to make Congress work better.”

Several Freedom Caucus members described the meeting in positive terms, while declining to say whether it had moved them any closer to backing Ryan’s speaker bid.

Rep. Mo Brooks (r-Ala.) said Ryan agreed that legislation should be moved only with the support of the majority of the majority and that the discussion also calmed the nerves of immigration hawks like him, who are concerned about some of Ryan’s past policy positions. But he made clear, Ryan faces a big challenge in getting the group’s support.

“Paul Ryan probably made some progress… but to get 80 percent of Freedom Caucus to switch from Daniel Webster?,” he said. “It’s going to be difficult for Paul Ryan to shift that in two, three days.”

Under the caucus’s rules, 80 percent of its roughly 40 members must support a candidate to grant an endorsement.

Earlier in the day, Boehner told members that Republicans will vote internally to nominate a speaker next Wednesday, with a floor vote to follow Thursday.

This scheduling decision was made after Ryan said Tuesday night he would run for speaker, but only if the conference unites behind his candidacy.

“This is not a job I’ve ever wanted, I’ve ever sought,” Ryan said. “I came to the conclusion that this is a very dire moment, not just for Congress, not just for the Republican Party, but for our country.”

The biggest question facing Ryan is whether the Freedom Caucus will back his bid and agree to the conditions he has set for entering the race.

Ryan’s declaration that he would serve as House speaker if and only if he receives formal backing from major House GOP factions — including the Freedom Caucus — effectively gives the group veto power over his ascent.

Other party groups, such as the conservative Republican Study Committee and the moderate Tuesday Group are considered to be much less likely to balk at Ryan’s demands.

Those demands include not only the endorsements but also giving Ryan time with his young family and changing House rules to end the ability of a simple majority to remove a sitting speaker.

“This is a lot to digest,” said Rep. David Schweikert (R-Ariz.). “We’re not ready to make a decision.”

Several caucus members said any change to the process for removing a speaker would be hard to swallow, if not a complete nonstarter, and that there focus is on allowing rank-and-file members to have more of a say in the legislative agenda.

“A lot of people like Paul,” said Rep. Justin Amash (R-Mich.), a Freedom Caucus member who said he would not rule out support for Ryan. “We’ve said from the outset, it’s not about the person, but the process. If we can have a speaker who will respect the process, open up the floor, and allow us to participate, that’s someone we could get behind.”

It is unclear when the group will make a decision on whether to rescind its standing endorsement of Webster and back Ryan, but it’s possible it could come late Wednesday night after Freedom Caucus members meet again following a series of floor votes.

In an interview, Webster said he would remain in the speaker race, and that no one who endorsed him had told him of a change of heart.

“Maybe they’ll be polite and won’t tell me,” he said. “I haven’t talked to Paul Ryan, either, but he’s very busy.”

Still, Webster was one of the few conservatives who could envision a way for the Freedom Caucus to switch its support and meet Ryan’s terms. “They could say that they endorse him, but that members may not personally vote for him,” suggested Webster. “That said, I overthink things sometimes.”

Boehner demurred when asked Wednesday about the Freedom Caucus’s role.

“I think Paul Ryan would make a great speaker, but this decision is up to the members,” he said, adding that “we all know Paul Ryan, right? He is a very good member. He works hard. He is very bright. And he has a good relationships, I think, with all the wings of the party. So, that’s why I think he will be doing fine.”

Ryan met early Wednesday afternoon with the Republican Study Committee, which counts over two thirds of the Republican conference as members. At the meeting, a committee spokeswoman said, members were surveyed by secret ballot on whether to endorse Ryan.

The tally, which will not be released publicly, will be referred to the RSC’s steering committee and it will decide Thursday whether to grant an endorsement.

Ryan is set to meet Thursday with the moderate Tuesday Group, which is expected to embrace his speaker bid.

David Weigel and Kelsey Snell contributed to this story