When Philando Castile, a black man legally carrying a concealed weapon, was shot dead by police during a traffic stop in Minnesota this month for no apparent reason other than that he was armed, it might have seemed odd that the National Rifle Association failed to rally behind the case. The Second Amendment protects everyone’s right to bear arms, not just white people’s, right?

By the light of the law, the answer is easy: The Constitution prohibits racial discrimination in all rights, including the right to bear arms. By the light of history, however, the answer is far more complicated. From America’s earliest days, the right to bear arms has been profoundly shaped by race. Indeed, for much of our history, the right’s protections extended almost exclusively to whites.

The founding generation that adopted the Second Amendment also enacted racially discriminatory gun laws. Fearing slave revolts, early American lawmakers prohibited slaves — and often free blacks, too — from possessing weapons of any kind. Even in states where blacks were allowed to have guns, such as Virginia, they had to first obtain the permission of local officials. And while the “well regulated Militia” mentioned in the Constitution largely fell out of military use, such groups continued to be employed to capture runaway slaves.

After the Civil War, the question of guns and race changed: Many blacks from the South had obtained firearms when they fled to join colored units of the Union Army. When the war ended, the Army allowed them to keep their guns as compensation for unpaid wages. As many of those black soldiers returned to their home towns, those guns were seen by white racists as a threat to the enforcement of white supremacy. Armed blacks could fight back.

So Southern states passed the Black Codes, which among other things barred the freedmen from possessing firearms. Racists formed groups like the Ku Klux Klan, riding at night to terrorize blacks and take away their guns. Congress, still controlled by the North, reacted by proposing the 14th Amendment to make the Bill of Rights, which previously limited only the federal government, a limit on the states, too. It was the greatest expansion of constitutional rights in American history — and, as historians have shown, it was prompted in part by the desire to protect the right of freedmen to have guns for self-defense.

Discrimination continued despite the 14th Amendment, and it affected the scope of the right to bear arms. In the early 20th century, an influx of immigrants from Italy, Greece, Hungary, and elsewhere in eastern and southern Europe — who, in the biases of the era, were seen as inclined toward committing crime and carrying hidden weapons — led states and cities to enact laws to restrict concealed carry. These laws, which were supported by the NRA, gave broad discretion to local authorities to decide who had sufficiently good reason to carry guns in public. In a society marred by racism, minorities were rarely deemed worthy of exercising that right. Even the Rev. Martin Luther King Jr. was turned down when he applied for a concealed-carry permit in 1956 after his house was bombed.

The civil rights movement was primarily about access to schools and jobs, but it also came to encompass guns and the Second Amendment. Despite King’s rejection of violence and its symbols, others saw guns as potent signifiers of empowerment and equal citizenship. Malcolm X and the Black Panthers took up arms and articulated a novel view of the Second Amendment: Not only did it guarantee the right to have a gun at home, it also protected the right to have a gun in public, where the threats (at least to blacks from police) were usually found. These activists also interpreted the Second Amendment as providing a right to take up arms against a tyrannical government — which, in their case, meant racist police officers. In standing up against police violence, Malcolm X and the Black Panthers were ancestors of Black Lives Matter. Lawmakers in states such as California responded by passing new gun regulations intended to disarm black radicals.

Armed civil rights activists were also early forerunners of the modern gun rights movement. The Panthers’ view of the Second Amendment eventually caught on among white activists, too. In the 1980s, the NRA began to argue that the amendment protected a right to carry guns in public — and launched a remarkably successful 30-year campaign to overturn the very concealed-carry laws the group had endorsed years earlier.

Today, gun rights activists unwittingly echo Malcolm X when they say the Second Amendment entitles people to own weapons in case they need to revolt against a tyrannical government. And when open-carry advocates go to a protest with rifles slung across their backs, they are mimicking the Panthers, who in 1967 showed up at the California statehouse to protest gun-control proposals with long guns in their hands. Oh, the irony: The modern gun rights movement — mostly white, rural conservatives — grew out of ideas first promoted by black, urban, left-leaning radicals.

Gun politics remain highly racialized. Racial minorities are currently among the biggest supporters of gun control and whites the biggest opponents. In recognition of the nation’s changing demographics, the NRA is making a major push to diversify: Its new spokesman, Colion Noir, is an engaging African American millennial. Yet the NRA’s annual convention remains largely a sea of white folks. And as the NRA’s reluctance to make a statement in support of Philando Castile suggests — many believe the group would have immediately backed a white concealed-carrier in such circumstances — there is still a long way to go.

All Americans can claim to be part of “the people” whose right to keep and bear arms is guaranteed by the Second Amendment. Yet because of the interaction of guns and race, that right has not been equally enjoyed by racial minorities. As long as the unfortunate strands of our racist past continue to shape our attitudes — and those of police officers like the one who shot Castile — racial minorities will continue to be the Second Amendment’s second-class citizens.