Rep. Paul D. Ryan (R-Wis.) has had a delicate relationship with Janesville, Wis., ever since he was a 28-year-old Ayn Rand enthusiast first running for Congress two decades ago from the old, Democratic-leaning union town.

By the time Ryan revealed Wednesday that he was relinquishing the House speaker’s gavel and his congressional career, the amicable rapport he had long nurtured with both Republicans and Democrats in his home town had become frayed.

While many local politics watchers believed he would have retained his seat if he tried, “he was going to have the toughest race of his life,” said Tim Cullen, a retired Democratic state senator who worked with Ryan over the years.

For his first seven terms, this youngest son in the fifth generation of a prominent Janesville family was popular — if not ideologically in sync with — his hometown constituents. Many viewed him as decent and hard-working, even if they disagreed with his fiscal conservatism.

The rapport began to ebb in 2012, when Republican presidential candidate Mitt Romney chose him as his running mate. And it has deteriorated further since Ryan reluctantly became speaker 2½ years ago.

Resentments have built in town that Ryan, who always prided himself on a direct touch with his constituents, has not had an in-person town hall meeting in his district since weeks before he became speaker. Though he is home often on weekends, he has been a less frequent presence at the grocery store or gas station, trailed as he now is by a security detail.

National groups have placed critical billboards along Milton Avenue, Janesville’s commercial corridor leading north from downtown. Late last month, dozens of Wisconsin high school students marched into town to urge Ryan to adopt stricter gun laws. They were joined by Randy Bryce, a well-funded former ironworker and union organizer seeking the Democratic nomination for Congress in what would have been a challenge to Ryan.

“Our goal’s been to repeal and replace Paul Ryan with a working person,” Bryce said in an interview Wednesday after the speaker’s announcement, borrowing the language Republicans have used for their unrealized goal of dismantling the Affordable Care Act. “Everybody’s happy that we got the repeal done. Now we’re starting on the replacement.”

Since Wisconsin’s 1st Congressional District was reconfigured in 2002, the inclusion of part of heavily Republican Waukesha County has given GOP candidates an advantage. Many locals had expected Ryan to prevail over either Bryce or Cathy Myers, a Janesville school board member running against Bryce in the Democratic primary.

But with Ryan third in succession to the presidency and often aligned with President Trump’s polarizing agenda, Cullen said, “the disconnect between a lot of [Janesville] people’s views and what Paul D. Ryan’s views are . . . became more distinct.”

In Ryan’s sudden absence, four months before the Wisconsin primaries, it’s unclear who will emerge as the Republican front-runner. Among the names that have been floated are people not yet in the race: Robin Vos, speaker of the Wisconsin Assembly, and Bryan Steil, a manufacturing executive who has been encouraged to run for office in the past and is friendly with Ryan.

While Bryce has brought in $4.7 million from 113,000 donors — many of them outside Wisconsin — Republicans suggested that he might suffer without Ryan as a foil. Myers has unabashedly attacked Bryce as a flawed candidate, but Republicans expect him to come out of the summer as the nominee. “Randy Bryce is an idiot, but they’re stuck with him,” said a Wisconsin GOP strategist, Brandon Scholz.

For much of his career, Ryan got support from Democratic voters, with a reputation not as conservative as his views. Rep. Mark Pocan, a Wisconsin Democrat, remembers the surprise expressed by a friend from Ryan’s district in 2010 when Republicans won control of the U.S. House and Wisconsin’s governorship and legislature.

“He said, ‘Everyone I voted for got beat, except for Ryan,’ ” Pocan recalled. “He voted for Democrats — but he voted for Ryan, too. That was how it used to be.”

That year, Ryan emerged nationally, if not locally, as a symbol for his party, as Democrats tried to run against his “road map for America’s Future,” which included budget-balancing cuts to Medicare and Social Security. Only those listening to conservative radio heard him creating distance with Democrats, portraying himself as an escapee from a Wisconsin liberal madhouse.

“This stuff came from these German intellectuals to Madison,” Ryan said in an interview with Glenn Beck. “Look, I grew up in the orbit of Madison. I know who these people are. I know what they think.”

Ryan’s relationship with his constituents changed during the 2012 campaign due to his presence on the GOP presidential ticket, and then changed again in 2016 when he would not fully support Donald Trump. In an August 2016 speech condemning white nationalists and their support for Trump, Democratic presidential candidate Hillary Clinton singled out Ryan as the sort of mainstream Republican who Trump was betraying.

“Even in the last election, if you drove around Kenosha, you’d see Hillary signs and Ryan signs in people’s yards,” said Erin Decker, chair of the Republican Party of Kenosha County, which is in Ryan’s district.

During his Capitol Hill announcement of his decision not to run for reelection, the speaker portrayed his retirement from Congress as a homecoming, allowing him to avoid being “a weekend dad” for his three teenage children.

Yet the end of his congressional tenure will mark the first time in Ryan’s adulthood that he may be in Janesville anything approaching full-time. After graduating from Craig High School, on the city’s more affluent east side, he left for Miami University of Ohio and then moved to Washington, where he held internships and congressional staff positions. He was in his 20s and single when he decided to move home and run for an open congressional seat. An aggressive campaigner from the start, he knocked out several more veteran Republicans even before the primary.

People’s attitudes toward his decision to leave Congress “will all depend on where your politics were when you woke up this morning,” Cullen said.

Even as partisanship has escalated, Janesville has accrued advantages as the speaker’s home town. It is a small industrial city that has been struggling to rebuild its economy since the nation’s oldest operating General Motors assembly plant closed there in the midst of the Great Recession.

“Part of his legacy really is he’s put us on the map nationally,” said Dan Cunningham, vice president of government relations for Forward Janesville, the equivalent of a U.S. Chamber of Commerce for the town of 65,000. “Paul has opened many doors for us.”

Last month, Forward Janesville brought about 50 business executives on an annual trip to Washington. There was a private dinner with the speaker at Bobby Van’s restaurant, and, at the White House, “they were opening doors wide,” Cunningham said, including a meeting with White House press secretary Sarah Huckabee Sanders.

Before this year’s trip, with rumors flying that Ryan might give up his seat, Cunningham told Forward Janesville members: “Guys, this is the year you are going to come. There is all this speculation. Come now.”

Next year, Cunningham knows, the trip to Washington will not include a visit to the House speaker’s balcony.