This article has been updated

In 2015, The Fact Checker awarded President Barack Obama two Pinocchios for a series of inconsistent and vague statements that he made about the United States and the rate of mass shootings compared with other countries. We noted that quantitative measures of cross-comparative crime statistics, especially where the crime is not consistently defined (e.g., “mass shooting”), usually end up being apples-to-oranges comparisons.

In defending the president’s remarks, the White House pointed to research by University of Alabama criminal justice professor Adam Lankford. In a paper that had not yet been published, Lankford conducted a statistical analysis of the total number of public mass shooters per country from 1966 to 2012 in 171 countries and controlled for the national population size. He said his data showed the United States had significantly more mass shooters, with 90 between 1966 and 2012, compared with 202 in the rest of the world.

Now there's a study that claims to undermine Lankford's research, generating headlines in right-leaning media. The Fact Checker raised questions at the time about Lankford's findings, so we decided it was worth revisiting the issue.

The Facts

The FBI does not officially define “mass shooting” and does not use the term in Uniform Crime Report records. In the 1980s, the FBI established a definition for “mass murder” as “four or more victims slain, in one event, in one location,” and the offender is not included in the victim count if the shooter committed suicide or was killed in a justifiable homicide, according to a Congressional Research Service report detailing the definitions.

But such definitions are in dispute. The 1998 shooting at Thurston High School in Springfield, Ore., which had two fatalities and 25 people wounded, would not qualify under the FBI definition but still might seem like a mass shooting to others. In recent years, federal authorities have tended to speak of “active shooters” when three or more people are shot, not just killed.

Using the four-person-killed definition, Lankford found that the United States had far more mass shooters (90 shooters in the 46 years, or 31 percent of the total) than the other countries, which averaged 1.7 public mass shooter per country. His research excluded gang-related shootings, drive-by shootings, hostage-taking incidents, robberies and acts of genocide or terrorism.

At the time, he declined to provide The Fact Checker with the underlying data on the worldwide shootings he had collected. Once he published the study, in 2016, he still declined to release the data. He would only say it was based on a report from the New York City police department and an FBI report, as well as a survey of news reports overseas.

In his study, Lankford only specifies the U.S. number of shooters (90) in comparison to the only other countries that he said had shooters in the double digits: the Philippines (18), Russia (15), Yemen (11), and France (10).

Lankford’s research has been widely cited but has been disputed by advocates of gun rights. John R. Lott Jr., president of the Crime Prevention Research Center, recently published a new study that claims to upend Lankford’s findings. Lott primarily relied on data obtained from the University of Maryland’s Global Terrorism Database (GTD) and then supplemented it with research of foreign news accounts.

Lott said that he found 1,491 mass public shootings worldwide in a shorter time period — between 1998 and 2012 — and that fewer than 3 percent were in the United States. Compared to Lankford’s 31 percent of all shooters, Lott calculated that the United States produced fewer than 1.43 percent of the mass public shooters. (He counted a total of 43 shooters in the United States between 1998 and 2012.) Overseas, "We find at least fifteen times more mass public shooters than Lankford in less than a third the number of years," he wrote.

Lott has a controversial reputation. Jaclyn Schildkraut, associate professor at the State University of New York at Oswego and co-author of “Mass Shootings: Media, Myths, and Realities,” said her review of Lott’s report suggests he drastically inflated the number of international shootings and minimized the context of the U.S. mass shootings picture.” She said his count of 43 incidents in the United States was too low; her count for the same period, with a different definition for mass shooting, is 188. Lott said he used the same definition as indicated by Lankford in his report.

Unlike Lankford, Lott has released all of his data in nearly 500 pages of appendixes so people can reach their own conclusions. Lankford declined a request to release his research or to discuss in any way his findings in comparison to Lott’s report. (Update, April 1: Lankford released his data.) However, in an email exchange with The Fact Checker, Lankford acknowledged the Mumbai attack in 2008 that killed 168 people was not included in his overseas tally.

When The Fact Checker spotted the Mumbai attack on Lott’s list, we asked Lott to remove terrorism cases from the totals for the four countries listed by Lankford — the Philippines, Russia, Yemen and France. Our hope was to provide as much an apples-to-apples comparison as possible.

Without terrorism cases, Lott’s count of shooters fell dramatically. In the Philippines, the number of shooters fell from 120 to 11, in Russia from 65 to 21 and in Yemen from 65 to 3. Only France did not have a significant decline, going from 5 to 4. This is for the 1998-2012 period, and with the exception of Russia, the number of shooters is lower than Lankford’s calculations for 1966-2012.

Drilling down into some of the individual incidents that were left on Lott's list, we found several more that could be called into question: five in Russia and three in the Philippines. So we went back to Lott with our concerns. Below is each case, with Lott's responses. The exchange helps shed light on how much these numbers can turn on definitions and judgment calls.

March 18, 2000: "At a bus station in Stavropol, police patrol decided to check a bag belonging to the brothers Murat and Umar Salpagarov and Umar opened fire, killing two police officers and two women." Our objection: This does not seem like a public mass shooting/active shooting if the shooting occurred because of the police search.

Lott's response: The murderer "ran out of the police department's office and started shooting at people inside the bus station's building,” the news report said.

Feb. 5, 2002: "Two Russian paratroopers who deserted their base killed nine people before being shot to death by police overnight in Tatarstan.“ Our objection: Should the violence of deserters be categorized in the same category as public mass shooters?

Lott's response: The disgruntled paratroopers killed four civilians and five police officers; their disillusionment with the military does not appear relevant to disqualifying the incident as a mass public shooting

Sept. 6, 2004: "Three criminals shot and killed four people in the Snezhinka cafe-bar in Surgut, Russia." Our objection: Was this a robbery, a hate crime or more akin to gang violence?

Lott's response: Three unidentified criminals armed with automatic weapons without warning started firing wildly on people in the Snezhinka cafe-bar. A news report says this was possibly a drunken quarrel but police do not exclude the "response" of the Russian population to the hostage-taking in Beslan, since the visitors of this cafe are mainly from the Caucasus. “If a drunk person uses a machine gun to randomly shoot up people (and the [Global Terrorism Database] says that this is neither terrorism nor insurgency or guerrilla related), it would appear to fit the type of case that you are still willing to include,” he said. “If you think that it is revenge for the Beslan school massacre where 385 people were shot to death and something over 500 more were wounded, I leave it to you whether you want to count it as terrorism.”

Sept. 17, 2005: "Three Russian deserters have been captured after a shooting spree that left three army troops and three police officers dead.” Our objection: Should the violence of deserters be categorized in the same category as public mass shooters?

Lott's response: “This is more akin to a case of workplace violence. This shooting wasn’t directly related to a military conflict, and the shooters were no longer soldiers.”

Sept. 30, 2007: "An argument [in Dagastan] between creditors and debtors reportedly deteriorated into a shooting spree," and nine people, including a local police officer, were killed. Our objection: This does not seem like a public mass shooting/active shooting; it sounds as though it has more in common with a robbery or other form of felony mass murder.

Lott's response: “Workplace attacks can arise for many reasons. In the U.S., cases include people feeling slighted by others or not getting promotions or getting fired or even disputes over money. This 'shooting spree' didn’t just involve the several people initially involved in the discussion.”

Aug. 25, 2001: "Abu Sayyaf Group (ASG) members fired on a mini-bus, killing six" in the Philippines. This seems to be militia or guerrilla attack.

Lott's response: This is designated by the GTD as terrorism, not an insurgency, guerrilla or similar attack.

Jan. 5, 2005: "Suspected members of a renegade pro-government militia attacked a group of farmers." This appears to be militia group behavior.

Lott's response: This is designated by the GTD as “other crime,” so not an insurgency, guerrilla, or similar attack, nor is it a terrorist attack. Again, we are relying on the GTD for this classification

Nov. 23, 2009: "Andal Ampatuan, Jr. and his clan (a 'leading Muslim political clan') attacked Esmael Mangudadatu's family members and supporters, 57 people killed ... The Philippines classifies this as a terrorist attack.” This appears to be terrorist group behavior.

Lott's response: “We have provided all the info that we had, including that the Philippines classified that as a terrorist attack, but our rule has been to rely on the University of Maryland GTD as the determining factor for whether it is a terrorist attack. Countries can have political reasons for classifying attacks in different ways, and rather than get into a debate on these issues we relied on the GTD. We spent a lot of time talking to the GTD, and they seemed to have good reasons for how they generally classified cases.”

Nevertheless, Lott was willing to adjust his calculations of the number of shooters again after these eight cases were removed. With the exception of Russia, the yearly rate turns out to be quite similar for both researchers. Recall that Lott estimated that there were 43 shooters in the 1998-2012 time period in the United States.

Lankford count of number of shooters (1966 to 2012):

France: 10 (0.212 yearly rate)

Philippines: 18 (0.322)

Russia: 15 (0.319)

Yemen: 11 (0.234)

Total: 54 (1.149)

Lott count of number of shooters (1998 to 2012), minus terrorism and eight other cases

France: 4 (0.2667 yearly rate)

Philippines: 5 (0.333)

Russia: 9 (0.60)

Yemen: 3 (0.20)

Total: 21 (1.40)

The Bottom Line

We appreciate Lott's willingness to recalculate his results based on our questions, as well as his decision to make public the voluminous data he has collected as he tried to replicate Lankford's findings. Lankford has chosen to keep his data set private, which makes it difficult to understand how the two researchers come to such different conclusions. Indeed, Lott was unaware that Lankford did not include the Mumbai attacks in his data set until The Fact Checker learned that from Lankford. (That's because Lankford refused to respond to his queries.) Lott's results changed when he removed cases that might not fit the public's definition of a mass shooting, just as they would change if a different definition of mass shooting is used.

A consistent definition of mass shooting — and public databases that display the relevant data — thus are necessary for the two sides of the gun debate to even begin to have a discussion on this issue. Lankford should reveal his dataset so it can receive the same scrutiny as Lott's work.

Update, April 1: Lankford published a response to Lott in which he also released his data set. He makes the case that when consistent definitions are used, Lott’s own data confirms Lankford’s findings: The United States has a disproportionate number of public mass shooters — six times — who attack alone. “By definition, firearms are needed for people to commit mass shootings, so in countries where it is easier for dangerous or disturbed individuals to legally purchase firearms—like the United States—there is an increased likelihood of an attack,” Lankford writes.

Send us facts to check by filling out this form

Sign up for The Fact Checker weekly newsletter