St. John’s Episcopal Church for decades has been the cheery-yellow backdrop for smiling presidents going to worship on holidays or before their swearings-in. In the past few days, its exterior — across Lafayette Square from the White House — has become the backdrop for arson, fiery protests and bible-carrying politicians and pastors.

The rewriting of St. John’s as a symbol continued Wednesday. A group of clergy — including the region’s Episcopal bishop, Mariann Budde — planned a large afternoon prayer vigil outside the church. They had to move it a block away, however, because of an expanded security perimeter created by police agencies trying to control another day of mass protests over the death of George Floyd in Minneapolis police custody.

“A church is not just a building,” St. John’s Rector Robert Fisher told those who gathered near crowds of protesters at 16th and I Streets NW in the baking sun. “The damage to the building from fire is easily fixed. What’s harder is healing.”

Attendees silently knelt in front of a row of military police who were blocking access to St. John’s, a 204-year-old structure with a steeple and high white columns where a small basement fire was set during the protests Sunday night.

They carried a banner that read, “We love, act, and pray in solidarity with those who love, act, and pray in justice.” One man encouraged the police to kneel with them, saying through a black mask in a hoarse voice, “It only takes one.”

The crowd broke its silence to echo that, chanting, “It only takes one.”

Those present spoke about their anger at President Trump, who drew worldwide attention to the church by walking in front of its boarded-up facade Monday and holding a Bible aloft. Police used rubber bullets and tear gas to clear peaceful protesters from the area before the president strolled over — crossing a line, some said.

“I think people wanted to respond. We felt compelled to be in solidarity and to say to the president: ‘We are more than a prop,’ ” said the Rev. Keith Byrd Sr. of Zion Baptist Church in Northwest Washington, who returned to the church Wednesday after praying and protesting there with about 25 black Baptist pastors Tuesday.

“He has merged some things and totally perverted some things with his photo op — that is untenable to God-fearing people,” Byrd said. “We had to respond. This is not political, not about blue, red; it’s about decency, respect and who we are as a nation. That’s why St. John’s has become this symbol.”

The Rev. Daryl Paul Lobban, minister for justice and advocacy for the Episcopal Diocese of Washington, was also at the vigil. “Churches have always been springboards for social action and social justice,” he said. “There’s a connection between a church building and uniting for justice. And now, after the visit from President Trump, there is energy to try and reclaim that space back.”

St. John’s Episcopal Church held its first service on Oct. 27, 1816. It’s often called the “church of the presidents.” According to the church website, every president since James Madison has attended at least one service there.

Before his swearing-in ceremony, President-elect Franklin D. Roosevelt attended a service at St. John’s, setting a precedent that has been followed by several presidents since, including Trump.

Located on the north side of Lafayette Square, St. John’s has naturally been in the vicinity of all kinds of protests over the decades. Its leadership turned overtly activist in the early 1960s, said Richard Grimmett, who chairs the property committee at St. John’s and wrote a history of the church.

When the Rev. Martin Luther King Jr.’s March on Washington was being organized, the pastor was on summer vacation, Grimmett said. A younger assistant pastor wanted to get St. John’s involved and called the head pastor to come back. They organized a service during the march and emphasized that “the church is open to everyone now and in the future.”

Some older members, Grimmett said, had advised against becoming connected to the protests, for fear “some might do some damage to the church physically.”

Since then St. John’s has been active, though not in a very public way, Grimmett said.

But in recent days, as protests over police brutality have spread through downtown Washington, with Lafayette Square as the focal point, St. John’s has been front and center.

The diocese has had multiple members of its clergy out with water and snacks, showing support, Lobban said.

Sunday night, when some protests turned violent, a small fire broke out in the nursery of the basement of one of the church’s buildings. Authorities believe the blaze was started through a window, Budde said.

While the bishop was distressed about the fire, she told The Washington Post she does not want it to detract from the protests and the work of addressing systemic racism. Her searing condemnation of Trump’s visit to the church the next day has exploded worldwide.

The challenges of uniting protesters was on display at Wednesday’s vigil, with some in the crowd of protesters confronting clergy — including those of color — who suddenly were in the media spotlight. The vigil concluded amid fractiousness as some protesters yelled at each other for confronting or refusing to confront the police.

“Help us to tear down walls of misunderstanding,” one black pastor said.

One of the protesters who had criticized the clergy apologized, saying they had been upset to see attention diverted from a fellow protester who was facing off with police.

“We’ve all just got to work together,” said the Rev. George C. Gilbert Jr. of Holy Trinity United Baptist Church in Northeast Washington, “and we apologize if you feel in any way we took anything away from your movement. Because it’s all our movement.”

CORRECTION: Earlier versions of this article were not clear about the precedent set by president-elect Franklin D. Roosevelt when he attended a service at St. John’s Episcopal Church on the morning of his swearing-in. Some, but not all, of Roosevelt’s successors also worshiped at St. John’s on their inauguration days.