The Washington Post

Good judgment in forecasting international affairs (and an invitation for season 3)

This is a guest post by Michael Horowitz, an associate professor at the University of Pennsylvania and currently a Council of Foreign Relations fellow at the Pentagon. —Erik Voeten

The Economist’s The World in 2014 issue just hit newsstands, focusing international attention on the geopolitical outcomes we can expect to see over the next 12-14 months. The issue features an article by University of Pennsylvania psychologist Phil Tetlock and journalist Dan Gardner on the Good Judgment Project, a research study funded by the Intelligence Advanced Research Projects Activity (IARPA, the U.S. government’s analog to DARPA), which makes such geopolitical predictions every day.

Since 2011, IARPA has posed about 100-150 questions each year to research teams participating in its ACE forecasting tournament on topics such as the Syrian civil war, the stability of the Eurozone and Sino-Japanese relations. Each research team was required to gather individual predictions from hundreds or thousands of forecasters online and to generate daily collective forecasts that assign realistic probabilities to possible outcomes.

The Good Judgment Project emerged as the clear winner and the Good Judgment Project forecasters have demonstrated the ability to generate increasingly accurate forecasts that have exceeded even some of the most optimistic estimates at the beginning of the tournament. The accompanying graphic shows the predictions from three GJP forecasting methods on a recent question concerning whether the first round of chemical weapons inspections in Syria would be completed before Dec. 1 –- an incredibly important and complex topic. Each prediction is a probability forecast that can range from 0 (a forecast of a 0% chance that the event will occur) to 1 (a forecast that the event is certain to occur).


Graph by Michael Horowitz

In this case, the question resolved as a “yes” because the Organization for the Prohibition of Chemical Weapons (OPCW), who was conducting the inspections, confirmed that they had completed the first round of inspection before Dec. 1. Since the question resolved as a “yes”, the closer each forecasting method’s predictions were to 1, the better they did. As the graph shows, after some initial uncertainty about whether inspections would occur, our forecasters generally converged on the correct answer well before the outcome.

How does the Good Judgment Project achieve such strikingly accurate results? The Project uses modern social-science methods ranging from harnessing the wisdom of crowds to prediction markets to putting together teams of forecasters. The GJP research team attributes its success to a blend of getting the right people (i.e., the “right” individual forecasters) on the bus, offering basic tutorials on inferential traps to avoid and best practices to embrace, concentrating the most talented forecasters into super teams, and constantly fine-tuning the aggregation algorithms it uses to combine individual forecasts into a collective prediction on each forecasting question. The Project’s best forecasters are typically talented and highly motivated amateurs, rather than subject matter experts.

Here’s your chance to “get on the GJP bus.” If you enjoy world politics and appreciate a good challenge, consider joining the Good Judgment Project, which has openings right now for Season 3 forecasters. The Project will give you the opportunity to receive training, to get regular feedback on your forecasting accuracy, and to test your forecasting skills against those of some of the most accurate forecasters around. Interested? To find out more and to register, go to www.goodjudgmentproject.com.

The Freddie Gray case

Please provide a valid email address.

You’re all set!

Campaign 2016 Email Updates

Please provide a valid email address.

You’re all set!

Get Zika news by email

Please provide a valid email address.

You’re all set!
Comments
Show Comments
The New Hampshire primary is today. Get caught up on the race.
What to expect in the New Hampshire primary
The Post's Philip Bump says ...
Since he proclaimed that he'd win New Hampshire last summer, Bernie Sanders has seen a swing of about 50 points in his direction. Impressive. But not as impressive as the guy on the other side of the political aisle. Donald Trump has led the Republican field in New Hampshire for almost 200 days, and has held a lead in 51 straight live-caller polls -- every poll stretching back to last July.
The feminist appeal may not be working for Clinton
In New Hampshire, Sen. Bernie Sanders is beating Clinton among women by eight percentage points, according to a new CNN-WMUR survey. This represents a big shift from the results last week in the Iowa caucuses, where Clinton won women by 11 points.
New Hampshire polling averages
Donald Trump holds a commanding lead in the next state to vote, but Marco Rubio has recently seen a jump in his support, according to polls.
New Hampshire polling averages
A victory in New Hampshire revitalized Hillary Clinton's demoralized campaign in 2008. But this time, she's trailing Bernie Sanders, from neighboring Vermont. She left the state Sunday to go to Flint, Mich., where a cost-saving decision led to poisonous levels of lead in the water of the poor, heavily black, rust-belt city. 
56% 41%
Upcoming debates
Feb. 11: Democratic debate

on PBS, in Wisconsin

Feb 13: GOP debate

on CBS News, in South Carolina

Feb. 25: GOP debate

on CNN, in Houston, Texas

Campaign 2016
State of the race

To keep reading, please enter your email address.

You’ll also receive from The Washington Post:
  • A free 6-week digital subscription
  • Our daily newsletter in your inbox

Please enter a valid email address

I have read and agree to the Terms of Service and Privacy Policy.

Please indicate agreement.

Thank you.

Check your inbox. We’ve sent an email explaining how to set up an account and activate your free digital subscription.