Cardinal George Pell left Sydney two years ago to great fanfare. The longtime archbishop of Sydney and lauded “defender of the faith down under” had just been appointed chief of the Vatican’s finances — sometimes described as the second most powerful position in Rome. It is the highest an Australian has ever ascended in the church.

But Pell couldn’t leave behind the child sex abuse scandal that had rocked his hometown or the hushed accusations that he had a part in it. Those accusations have grown louder and much more public since last month, when an Australian commission began hearings into a notorious pedophilia ring just down the road from where Pell once served as parish priest.

Amid fervent coverage in the Australian news media, an accusation that his handling of the issue was “sociopathic” and testimony that Pell ignored warnings about his abusive colleagues and attempted to bribe one victim to stop him from speaking out, Pell has now hired lawyers to fight off attacks from the decades-old scandal he can’t quite shake.

The issue came to a head this Sunday, when Peter Saunders, a British survivor of sexual abuse by a priest who had been specially appointed by Pope Francis to serve on the Vatican’s Commission for the Protection of Minors, said that Pell should be removed from his position.

“He has a catalogue of denigrating people, of acting with callousness, cold-heartedness, almost sociopathic, I would go as far as to say, this lack of care,” Saunders told the Australian television program “60 Minutes.” “… I think it’s critical that he is moved aside — that he is sent back to Australia and that the pope takes the strongest action against him.”

Pell has vehemently denied Saunders’s allegations, and Vatican spokesman the Rev. Federico Lombardi came to his defense Monday, saying that Saunders didn’t speak on behalf of the Vatican commission and that Pell’s own remarks on the matter “must be considered reliable.”

But Saunders isn’t the only person accusing the prominent cleric. Ten thousand miles from Rome, in Pell’s hometown of Ballarat, abuse victims and fellow priests have given testimony under oath that Pell knew about the dozens of children who were mistreated there.

Pell was ordained in Rome in 1966 and moved to Oxford for his PhD. But when he returned to Australia in 1971, he moved back to his home state of Victoria, and two years after that he was back in Ballarat. There he served as a parish priest and vicar. Just down the road, St. Alipius Primary School had become the site of one of Australia’s most notorious pedophilia rings.

The sexual abuse there has been blamed for “hidden trauma and landscape of death” in the town, in the words of advocate Peter Blenkiron, who spoke to the Australian Broadcasting Corp.

One former St. Alipius student who testified before the commission offered a photo of his fourth-grade class, lawyer for the commission Gail Furness told the ABC. Of the 33 boys pictured, 12 had committed suicide.

One of the worst abusers was Gerald Ridsdale, a now-defrocked priest convicted of sexual offenses against more than 50 children. Ridsdale lived with Pell in Ballarat for two years.

Ridsdale, who told his family he had “hundreds” of victims, according to the Australian newspaper the Age, was moved to nine different parishes over the course of his career — something the church is often accused of doing when faced with allegations of abuse.

Pell said he knew nothing about his colleague’s treatment of children until the 1990s, when Ridsdale was arrested and charged with dozens of counts of sexual assault over the course of two decades.

When Ridsdale appeared in a Melbourne court in 1993, Pell stood alongside him, dressed in a priest’s garb.

“To me it’s absolutely outrageous, and it demonstrates once again the callousness, the cold-heartedness and the contempt that George Pell appears to display for this whole issue and particularly for the victims of these dreadful crimes,” Saunders, the Vatican victims advocate, told “60 Minutes.”

Pell has since said the decision to accompany Ridsdale may have been offensive, according to “60 Minutes.”

In 1994, Ridsdale pleaded guilty to 46 counts of sexual abuse and was sentenced to 15 years in prison. He was sentenced to an additional four years in 2006 after pleading guilty to 35 more charges, and eight more years in 2013, after additional offenses came to light during the Australian inquiry into the Church’s handling of abuse allegations. All told, he is convicted of abusing 54 children. The youngest was 4 years old.

Meanwhile, Pell’s star continued to rise. He was named archbishop of Melbourne in 1996 and then Archbishop of Sydney in 2001. A 2003 article on his appointment as cardinal called his ascent to the upper echelons of the Church “inexorable,” noting Pell’s moral fervor and reputation as a “staunch and public upholder of Catholic purity.”

But Australia’s current investigation into the Church’s abuse scandal has raised questions about Pell’s attitude toward abusers and their victims.

Last August, when Pell appeared before the commission for a broader inquiry on the Church’s handling of abuse cases, he said that the Church should not be held legally responsible for the actions of its priests, comparing it to a hypothetical trucking company that hired a driver who molested women.

“It would not be appropriate, because it’s contrary to the policy, for the ownership, leadership of that company to be held responsible,” Pell said, according to the Religion News Service. “Similarly with the church and the head of any other organization. It is, I think, not appropriate for legal culpability to be foisted on the authority figure.” The remarks drew anger from victims’ advocates, who accused the church of failing to protect children.

The recent Ballarat hearings have only further embroiled him in the scandal.

Ridsdale, the convicted abuser, testified in May that his behavior was “no secret” in his various parishes, according to the Melbourne Herald Sun. A series of letters and interviews published by the commission showed that the Church hierarchy knew about his abuse as early as the 1970s but did nothing about it beyond moving him to a new parish and sending him to counseling. Other priests gave testimony that Pell was probably part of high-level meetings about Ridsdale’s offenses by 1977, an allegation Pell has denied.

One of the most biting accusations at the recent hearings came from David Ridsdale, Gerald Ridsdale’s nephew, who said he had been one of his uncle’s victims. He testified that he told Pell, an old family friend, about the abuse in 1993, but Pell — then a bishop in Melbourne — had seemed unfazed.

“His first reaction was, ‘Oh, right,'” David Ridsdale said, according to coverage by the Australian Broadcasting Corp. Then, according to Ridsdale, Pell asked, “I want to know what it will take to keep you quiet.”

“Some days I don’t know who I’m angrier at: Gerald for being a sick monster, or George for the way he reacted and dealt with the issue,” Ridsdale added.

Pell denied the allegations in a statement released after the initial round of hearings and has offered to give evidence before the commission, according to the ABC.

“I am committed to complete cooperation with the Royal Commission,” he said in his statement. “I have the deepest sympathy for the victims of abuse, their families and the community of Ballarat for what they have suffered. Once again, I will answer allegations and criticisms of my behavior openly and honestly.”