Back when Unite the Right rally organizer Jason Kessler was just starting to make an infamous name for himself in Charlottesville, he had a young friend and sidekick named Isaac Smith.

Together they called for preserving the city’s Robert E. Lee monument, formed a group devoted to “defending Western Civilization” and flaunted provocative alt-right symbols at raucous political demonstrations.

But Smith pulled away as Kessler made plans for the Aug. 12, 2017, rally, alarmed by extremists that Kessler had lined up to speak — including self-proclaimed white nationalist Richard Spencer, a leader of the alt-right movement seeking a whites-only state.

“He’s affiliated himself with people who are — to put it mildly — ideologically distasteful,” Smith told Charlottesville’s Daily Progress in a June 2017 interview. “I want nothing to do with that.”

As the first anniversary of the rally looms Sunday, some civil rights activists laud Smith as an encouraging footnote, an example of someone who turned away after flirting with extremism.

But Smith’s dalliance with Kessler continues to dog him. In recent weeks, he was blasted as a white supremacist, as part of an attack on a Republican congressional candidate who happened to cross paths with him at a public event — a claim that went viral when it was hitched to an oddball flap over Bigfoot erotica.

The political storm raises a question: Can someone who wants to walk away from extremism be allowed to do so?

“People should be permitted to leave the alt-right,” said Smith, 22, who insists he never was part of the movement, just on its fringes. “I want the people who actually joined the movement to be comfortable leaving it, and they’re not.”

Smith was a restaurant cook in late 2016 when he first teamed up with Kessler, a provocative blogger upset with then-Vice Mayor Wes Bellamy’s call to remove a monument to Lee from a downtown park.

Kessler unearthed anti-gay, anti-white tweets that Bellamy had posted several years earlier, and Smith — who is gay — stepped up to say he was offended. They pushed, unsuccessfully, for Bellamy’s removal.

From there, the two launched the group Unity and Security for America, which pushed the notion that foreign entanglements and immigration — legal and illegal — threatened the existence of “Western Civilization.” They also hooked up with Corey A. Stewart, who at the time was running for governor on a promise to preserve the state’s “Confederate heritage” and is now Virginia’s Republican Senate nominee.

Both appeared at Stewart’s side with placards of Pepe the frog and gladiator-style shields, symbols adopted by the far-right movement. Smith sometimes introduced Stewart at events. In an interview with The Washington Post at the time, Smith said he enjoyed needling liberal activists, although he rejected the alt-right label.

“The term I might use is ‘dissident right.’ It’s the part of the right wing that has fun,” Smith, who says he is not working on Stewart’s Senate bid, said at the time. “It’s not like I’m sending a picture of a member of the Ku Klux Klan holding a noose. It’s a smiling frog. Why does this upset you so much?”

Smith said he got second thoughts about Kessler after the blogger and Spencer led a torch-lit rally at what was then called Lee Park in May 2017, an event Smith said he did not attend. As Kessler began lining up Spencer and other overt white nationalists to speak at the August rally in Charlottesville, Smith broke with him.

“The final straw with Jason Kessler was his open and deliberate embrace of white identity politics, which I found unacceptable and refused to go along with,” Smith told The Post recently.

He moved out of state to try to make a new start as a welder, but he soon returned to Charlottesville, where he helped organize events aimed at community healing. Among other things, he recruited Daryl Davis — a blues artist known nationally for trying to befriend Ku Klux Klansmen and change their racist views — as a speaker.

“He’s not a white alt-right or anything else. He was never into that,” Davis, who is black, said of Smith. “I stand behind Isaac Smith 100 percent.”

Smith also worked as a waiter and retail clerk. He was active in conservative Republican politics but left provocative rhetoric and symbols behind. These days, he volunteers that his father is Jewish, although Smith was not raised in that faith.

“I’ve learned from experience that politics is not about us versus them,” Smith said. “And as soon as it is, escalation is bound to follow.”

Life stayed fairly normal until late June, when the Democratic Party of Virginia accused GOP congressional candidate Denver Riggleman of campaigning with “avowed white supremacist Isaac Smith.”

The party based the claim on news footage that showed Smith attended a public event for Riggleman, who is seeking an open congressional seat in a district that includes Charlottesville. Smith and Riggleman campaign manager Joe Chelak both said he has no role in Riggleman’s campaign.

The Democrats later removed the claim about Smith from its website, a decision party spokesman Jake Rubenstein said came after an emotional appeal from Smith’s grandfather. Rubenstein said he was aware that Smith had denounced Kessler ahead of the rally but said it was still fair to describe Smith as a white supremacist who fanned the flames ahead of the event, “whether he backed off at the last minute or not.”

The party’s claim went viral when Riggleman’s Democratic opponent, Leslie Cockburn, hitched it to a strange story about “Bigfoot erotica,” a reference to Facebook posts Riggleman says he made in jest — before running for office — about the mating habits of Sasquatch.

“My opponent Denver Riggleman . . . was caught on camera campaigning with a white supremacist,” Cockburn tweeted July 29. “Now he has been exposed as a devotee of Bigfoot erotica. This is not what we need on Capitol Hill.”

In recent weeks, Smith has moved out of state again. He declined to say where he is living but said he is training to become an emergency medical technician.

He said he will steer clear of Charlottesville on Sunday but did not rule out appearing elsewhere with Republicans who “want to take a stand against the alt-right.”

“Oftentimes people will judge you by the company you keep,” Davis said. “But unfortunately, he’s being judged by the company he’s dismissed.”