By David Brown
Washington Post Staff Writer
Friday, December 12, 2008
Many of the world's poorest countries have for decades routinely exaggerated the number of children being immunized against disease, apparently driven by political pressure and, more recently, financial incentives.
That is the finding of a huge analysis that has provoked heated discussion even before its publication in the Lancet, a European medical journal.
Since 1986, progress in childhood immunization in the developing world has been about half that officially reported by governments in the developing world. Not only are year-to-year improvements overstated, but the total percentage of children immunized is far lower than publicly acknowledged, the study found.
The two-decade trend masks extreme variations, with some countries overstating their gains four- and fivefold, and a few countries understating them.
The analysis -- which compares official immunization coverage with what was found in house-to-house surveys -- casts a shadow on the emerging strategy of "pay for performance" in global health assistance.
Specifically, it suggests that the pioneering Global Alliance for Vaccines and Immunizations (GAVI) may have paid out twice as much in performance rewards as it should have: $290 million instead of $150 million.
Of 51 countries that have received reward payments since 1999, six overestimated their immunization gains by a factor of four, 10 overestimated them by a factor of two, and 23 by less than two. Eight underestimated their progress.
GAVI has suspended reward payments to all countries, pending further review.
"By early next year, we will modify, drastically change, or possibly put in place a new system of incentive performance," Julian Lob-Levyt, the executive director, said yesterday from London after a two-day meeting with GAVI's partners to discuss the findings.
He noted that there was no suggestion any money has been diverted for personal gain.
GAVI was started with a $750 million "seed grant" from the Bill and Melinda Gates Foundation. Ironically, the new analysis, done by a group of researchers at the University of Washington led by Christopher J.L. Murray, was also funded by the Gates Foundation.
The study is an example of how health statistics can vary depending on their source -- and possibly also with the interests of the people generating them.
"Is there intent? We can't say," Murray said. "All we can say is that there is over-reporting, and the over-reporting occurs in the presence of financial incentives."
Murray and his colleagues examined coverage with diphtheria, tetanus and pertussis (DTP) vaccine. Children should get three doses of it before their first birthday.
The researchers compared three sources of data: the official account of children vaccinated reported by governments; the "best estimate" made by the World Health Organization and UNICEF, which adjust the official counts if they find them implausible; and surveys, usually underwritten by independent agencies, that scientifically sample a population. The last are considered the most accurate, although they sometimes underestimate coverage.
The researchers amassed all surveys done over 20 years in 193 countries. They estimated that DTP coverage was 59 percent in 1986, 65 percent in 1990, 70 percent in 2000 and 74 percent in 2006. Official reports put DTP coverage at 90 percent in 2006, and WHO and UNICEF put it at 79 percent that year.
In the 1980s, long before incentive payments began, UNICEF mounted an intense political campaign to boost immunization. In 1990, it declared the goal of 80 percent global coverage met. But it is now clear, Murray said, "they weren't even close."
Among the many things GAVI does is offer an award of $20 per additional child immunized each year over the number immunized in a baseline year.
The researchers found that in some countries (such as Bangladesh, Indonesia, Niger and Mali), the baseline coverage was recorded as being lower than in previous years. They also found that in many (such as Niger, Mali, Chad and Ethiopia) the difference between official and survey estimates of coverage grew after the baseline year.
Both of these strategies -- depressing the baseline and exaggerating improvements -- would tend to inflate reward payments.
Overall, DTP coverage since 1999 increased by about five percentage points, according to the surveys, slightly more than half the nine percentage-point increase reported by governments.
GAVI performs "data quality audits" that test the validity of official counts by following the data trail in four health districts per country. But that is not sufficient to detect over-reporting, Murray and his colleagues concluded.
Lob-Levyt said GAVI has been an "ambitious, quite high-risk endeavor" from its start, and he said "we very much welcome" Murray's data.
Other experts said they doubted the findings would mortally wound either GAVI or performance-based aid.
GAVI is "a global health program that learns. It experiments, looks for feedback, and changes," said Ruth Levine, a health economist at the Center for Global Development, a think tank in Washington. "This represents an opportunity to do that again."
The challenge now, she said, is "to adapt the measurement of progress in ways that reduce the risk of gaming."