JANESVILLE, WIS. — House Speaker Paul D. Ryan is something of a political darling here in his hometown. There was near universal agreement among voters here Tuesday that Ryan seems like a stand-up guy, most people just call him Paul, and they are proud to see him on the national stage.

But some Republicans in his home state and across the country still feel uncomfortable that he never seemed to fully embrace GOP presidential nominee Donald Trump during the campaign.

“It hurts the party that he doesn’t support the Republican candidate,” said Zach Molitor, 20, who cast a ballot for his hometown congressman at the Janesville public library and said he still keeps at home a framed note he received in sixth grade from Ryan. “I’m disappointed.”

After months of criticism over his soft support for Trump, Ryan spent the last several days trying to shake the idea he wasn’t in the GOP nominee’s corner while making the case to voters that supporting Republican candidates up and down the ballot is the best way to enact a conservative agenda for the country.

That’s the case he made Monday night at a rally 50 miles northeast of his hometown where an excited group of Republicans crowded into a strip mall warehouse, cheering the mentions of secure borders and attacking ISIS, while booing when speakers brought up Hillary Clinton and Justice Ruth Bader Ginsburg.

But the loudest applause line came when Ryan held himself up as an example of GOP unity.

“I voted Trump, Pence, [Sen. Ron] Johnson, some guy named Ryan and every other Republican on the ticket,” Ryan said to booming applause during an event that was part of his recent mad dash across Wisconsin in an attempt to shore up votes for the embattled Johnson. “Republicans, we need to come home. We need to unify and we need to go vote.”

Ryan’s plea for unity was as meaningful for the party in the waning days of the election as it is for his own political self-preservation. Regardless of who wins the presidential election, he will return to Washington next week as the leader of a House GOP in crisis. Several members of the approximately 40-member House Freedom Caucus have openly discussed removing Ryan as speaker, in part, because they are upset about his early waffling over whether to support Trump and then his timid endorsement of the nominee until recent days.

He likely has enough votes from members of his party to be chosen as the GOP nominee for speaker during leadership elections next week. But he will also have to survive a vote on the House floor in January where he will need 218 votes to keep his gavel.

Republicans currently hold 240 seats in the House, but the GOP is widely expected to lose at least 10 seats on Tuesday. If the GOP majority slips to around 230 it would take less than one third of the Freedom Caucus Members to end his one-year run as leader of the House.

If Trump wins, Ryan will likely have to quickly repair his relationship with a nominee who has called him out for not giving him enough support. This relationship could be helped by Republicans’ strong showing in Wisconsin — Trump was leading in the state early Wednesday morning — following the speaker’s recent campaign efforts there, including a rally with vice presidential nominee Mike Pence.

Trump had a clear advantage in the battleground states early Wednesday morning and House Republicans kept control of the House with what appears to be a stronger margin than expected.

“Speaker Ryan called Donald Trump earlier this evening, and the two had a very good conversation,” Ryan spokesman Zack Roday said. “The speaker congratulated Trump on his big night and also spoke with his good friend Governor Mike Pence.”

Ryan easily won reelection and told the crowd Tuesday night that he was eager to return to watching the results pour in from across the country.

“I’ve just been sitting there watching the polls,” Ryan said. “By some accounts this could be a really good night for America.”

Ryan’s allies have pointed to all he’s done to help the GOP maintain control of the House this election as evidence he is worthy of remaining speaker. In the past year, he has participated in 196 events in 31 states in support of vulnerable Republicans. He also transferred $40 million to the House Republican campaign arm and helped candidates raise $20 million by doing special guest appearances at campaign events.

But despite his fundraising work and calling for solidarity on the trail in recent days, Ryan’s less than enthusiastic embrace of Trump is still causing some heartburn for many Republicans.

“I’m glad he came around,” said Bob Young, a former mayor of Augusta, Ga. and a former regional director at the Department of Housing and Urban Development in the George W. Bush administration who traveled to Wisconsin to help campaign for Republicans. “I wish he would have come around a lot earlier but you’ll take what you can get.”

As they left the Monday night rally, most voters expressed confidence the party would come together after the elections and said the discord was all par for the course in a difficult political year.

Some borrowed a phrase Ryan himself likes to use when Republicans have differences: They said it was a family conversation.

“We’re family and we have internal bickering sometimes,” said Joseph Galvan, who traveled to Wisconsin from Elgin, Ill.

Young and others said that if Trump loses to Clinton they expect GOP leaders like Ryan to get to work putting the party back together.

“At some point if we’re going to be a player four years from now on a national stage,” Young said. “There’s no question that we have to come together as a party.”

Sean Sullivan contributed to this report