Brexit has revived fears that Northern Ireland will return to violence. After three decades of “The Troubles,” deadly warfare in which almost 3,500 people died, violence mostly ended after the 1998 Good Friday Agreement was signed. That peace deal relied in part on European Union membership, which enabled free trade and free movement between Northern Ireland and the Republic of Ireland. That satisfied both British unionists, who wanted Northern Ireland to remain part of the U.K., and Irish republicans, who wanted to join the Republic of Ireland.

Many feared that Brexit threatened that truce. When the U.K. decided to leave the European Union, observers feared that introducing a “hard” border between Northern Ireland, still part of the U.K., and the Republic of Ireland, still part of the E.U., might undermine the peace and encourage militant Irish republicans to attack the border.

The compromise that Boris Johnson negotiated involved what’s called “the Irish backstop,” a complex deal that keeps Northern Ireland within the U.K. customs union while it operates under E.U. rules. The goal was to enable goods and people to continue to pass freely between the two parts of Ireland.

But even presuming that the next U.K. parliament enacts Johnson’s Brexit deal, as expected, post-Brexit violence in Northern Ireland still looms. In October, a spokesperson for the so-called New IRA republican paramilitary group told Channel 4 that any border infrastructure would be a “legitimate target for [an] attack.” Shortly afterwards, the political party Saoradh, thought to have links to the New IRA, slammed the deal: “The British border remained as an obstacle to the Irish people having real democracy and national self-determination.”

Radical republicans aren’t the only threat, however. Northern Ireland’s unionists (who want to keep Northern Ireland in the U.K.) and loyalists (politically extreme unionists) have been angered by British concessions to the Republic of Ireland and Europe. Loyalists are hinting that they too might take up arms again. Here’s what’s happening.

Northern Unionists don’t like the deal that Boris struck

Boris Johnson’s Brexit proposal was designed to make it less likely that militant republicans would mobilize around the border. However, the compromise imposed trade barriers between the island of Great Britain and Northern Ireland, angering Northern Irish loyalists. Influential loyalist commentator Jamie Bryson, the editor of the “Unionist Voice,” summed loyalist sentiments up: “Republicans have been rewarded yet again for threatening peace with the constant stream of Brexit concessions.”

In an earlier comment, Bryson, who is believed to have significant insight into loyalist paramilitary organisation, said that loyalists were closely watching the outcome of Brexit. In the past month, loyalists have held several meetings throughout Northern Ireland, complaining of “serious discontent around the proposed Betrayal Act designed to create an economic United Ireland.”

Some feel threatened by what is happening in Northern Ireland

Northern Ireland unionism and loyalism are both politically linked to Protestantism, which used to control Northern Irish politics. Since the peace process in the 1990s, a new Catholic middle class emerged in Northern Ireland. After the Good Friday agreement, Catholics became more prominent in politics, threatening Protestants and unionists’ once-dominant position. This led, as political scientist James McAuley outlines, to a “distrust of unionist leadership, a strong belief that loyalists had throughout the conflict been used, both ideologically and logistically, by established unionist politicians.”

The 2013 Peace Monitoring Report gives some examples of these changing social and economic realities. For instance, in that year, 13 out of the 15 areas with the worst academic and school results were unionist. Someone coming from a poor nationalist background had a roughly one in five chance of going to university, while the chances for a poor unionist were only one in ten. Protestants were badly hit by the economic recession and the disappearance of industries that they used to dominate. In 2013, Protestant youth unemployment stood at 24 percent, while Catholic unemployment was 17 percent.

In the same year, loyalist protests over the perceived removal of British identity emerged in the form of so-called “flag protests,” sparked by Belfast’s city council decision to fly the Union Jack over city hall only on designated days. This led to weeks of sometimes violent street demonstrations.

This explains why loyalists are so unhappy

The combination of external challenge and internal threat explains the resurgence of loyalists’ worries. Union between Great Britain and Northern Ireland is a core element of unionist and loyalist identity. When the loyalist paramilitary Ulster Volunteer Force declared its war to be over in 2007, it stated that “the constitutional question has now been firmly settled” and “the Union remains safe.”

Now, that is not so clear. Some think that Johnson’s deal establishes a border through the Irish sea between the U.K. and Northern Ireland, calling the constitutional status of the union into question. Loyalists furthermore feel threatened by public discussions over Irish unity following Sinn Féin requests for a referendum on a United Ireland within five years, even though the Good Friday Agreement says that only Britain can hold one.

The anger articulated at recent loyalist meetings to oppose the Brexit deal resembles the loyalist frustration over the removal of the Union Jack in Dec. 2012. For loyalists, both events threaten the relationship of Northern Ireland with the U.K. Should Brexit take place, expect street protests. For now, loyalists are focusing on supporting Democratic Unionist Party (DUP) candidates in the U.K.’s Dec. 12 parliamentary elections.

Loyalists might turn to violence

Despite these increasing tensions and divisions, neither loyalists nor republicans seem likely to turn to violence in the near future. But longer term, loyalists rather than republicans may be the ones who start the fighting. That may have been what happened in the past. In his seminal treatise on Northern Ireland, scholar Brendan O’Leary insists that the violence that launched the Troubles was started by loyalists who worried that they were losing status and advantage in Northern Ireland.

Research by political scientists Jessie Blackbourn and Kacper Rekawek found that working-class loyalists saw the peace process as a defeat for their cause. Brexit is making them angrier still. In 1966, the Ulster Volunteer Force killed its first Catholic civilian. There is a small – but real -- risk that some Northern Irish loyalists may start the violence again.

Dieter Reinisch is a historian at Institute for Advanced Study, Central European University, in Budapest, and an adjunct professor in international relations at Webster University in Vienna.