Co-Director, Center for Refugee and Disaster Response , Johns Hopkins University Bloomberg School of Public Health
Thursday, October 19, 2006 12:00 PM
Gilbert Burnham , lead author of the controversial Johns Hopkins University study that finds vastly higher civilian casualty toll than other research, joined washingtonpost.com World Opinion Roundup columnist Jefferson Morley online Thursday, Oct. 19, at noon ET to discuss the study's conclusion that more than 600,000 Iraqis may have died since March of 2003.
World Opinion Roundup: Is Iraq's Civilian Death Toll 'Horrible' -- Or Worse? , ( Oct. 19, 2006 )
The transcript follows.
West Orange, N.J.: Steven Moore, writing in the WSJ, claims your study uses too few clusters and did not include demographic questions to determine whether your respondents were typical of the population. Any comment? How does your study discriminate deaths of insurgents (terrorists), from violent deaths of civilians, from the natural incidence of death? Do you subscribe to claims that huge numbers of Iraqis died due to causes related to malnutrition during the pre-2002 sanctions? How about during the Iraq-Iran war? How many people would one expect to die in 4 years in a population of 27 million under "normal" conditions?
Gilbert Burnham: Thanks for this question. A cluster survey has two components--the number of people (or households) and the number of sampling points. The number of people is the key determinant in whether the results are valid. The number of sampling points determines the error rate or confidence interval. There are strict formulae for the sample size, but the number of clusters is usually chosen according to the precision desired. Our study has adequate numbers to support our final decision, and the number of clusters is adequate to support the level of percision (different from accuracy) that we were aiming for. We felt that more clusters would have resulted in more risk for interviewers, and a bigger number was not necessary for the level of percision we desired. Much thought and consultation went into the number of clusters with some of the best brains in sampling.
Children in Iraq clearly have had poor nutrition for some years, and this without doubt has contributed to increased deaths.
As far as number dying. You can use our numbers(5.5/1000/yr) or the CIA or the US Census Bureau numbers and multiply this out to see what the normal death rates are
Arlington, Va.: Hi,
There have been many articles in reputable MSM like the Wall Street Journal raising serious questions about the methodology used. How do you respond? Thanks!
Gilbert Burnham: The methods we use are standard methods, used all over the world, and heavily supported by the US Government. We stand by these, and there has been little criticism from of the science. Mr Moore has done many surveys for Mr Bremer, but judging by his comments, he is not a statistician.
Minneapolis, Minn.: I have a couple of skeptical questions, not politically motivated but in the interest of good assessment of your work.
First, if I understand correctly, you had a very low refusal rate, which is surprising. How did you get to the households you surveyed? Specifically, did you choose them genuinely and fully randomly, or did you have to go through some set of community leaders?
Second, what is your explanation for the fact that there appears to be such a discrepancy between your numbers and the number of death certificates issued that might reflect your numbers?
Gilbert Burnham: Our concern is for populations caught up in war. To improve protection of civilians we need to have data about who is dying and why. It is hard to collect these data, which is why there are so few reports. Our intent is to help develop better methods to do this. The study was originally intended to be completed by July or August of 2006, but a number of delays happened and we finished it when we did. If we can discuss these findings to discover better ways of protecting innocents from death in wars, then we will feel that the risks of all were worthwhile.
San Francisco, Calif.: Mr. Burnham, have you read the recent Wall Street Journal editorial addressing your study and if so, do you have any rebuttal for their assertions? Why did you only use 47 cluster points? (a U.N. study used almost 50 times that many clusters, and over 10 times as many interviews, and came up with much lower numbers). Why no demographic data on the interviewees? Additionally, how do you rebut the assertion by Iraqi Body Count (not steadfast supporters of this war) that your study implies over 1,000 violent deaths/day in the first half of 2006, with only 1/10 being detected by the media, government, and citizenry? Thank you for addressing my question.
Gilbert Burnham: In a previous answer I indicated that the number of clusters will narrow the confidence interval, but the sample size is the prime determinant of the validity of the sample. We spent a long time deciding on the number of clusters with demographers and statisticians, and for the scope of our study, we feel 47 were adequate. Reviewing the data from these 1849 households suggests that 47 clusters were adequate.
Data collected at specific sites such as clinics and morgues never equal the total events happening--that is why every country has a census process--to go to the people and find out what really happened. When wars come, the process of collecting information breaks down. We tried to find this information from health authorities in the individual governorates during the survey, and it was usually either not there or hopelessly out of date. Even in systems where it makes it to the top, it is often seriously manipulated for political reasons--which we saw in the Soviet Union and in China, and is likely to be happening in Iraq. So the most accurate answers come from going to the people--a basic principle of sampling and census activities.
Bridgewater, Mass.: Thank you for taking the time to discuss your study here.
Since your figures are differences between the death rates before and after the beginning of the war, did you compare your "before" rates to any kind of known, or at least, official, values? (I read somewhere on the web yours would have implied a lower rate than Israel's, and for years we'd been reading about the effect of sanctions on the Iraqi health system, especially for children.)
Secondly, I've also read that if your rates were reasonable, people would be fleeing the country in droves. That assumes that individuals have an idea of the overall situation, which seems unlikely with as little media coverage as there must be (or even electricity to communicate it). Did your researchers gain any idea of what their respondents thought was happening throughout the rest of the country?
Gilbert Burnham: Two good questions. We compared households with themselves--known as a cohort study. So we were most interested in finding out about the morality experience pre-invasion for that household. But it turned out that the mortality rate for the houses in our study was the same as what the CIA uses for Iraq and the estimates of the US census bureau.
As far as the overall mortality rates--know as the Crude Mortality Rate or all-cause mortality rate, this is a function of many things--number of children, diseases people have---but it is heavily dominated by the number of old people in a society. So countries with many old people have higher death rates. Sweden has a death rate of 11/1000/yr compared with 5.5 for Iraq and 8 for the USA. Hungary is another country with a high mortality rate, and it has 40% of its population over age 55. Go to Google and type in 'population pyramid' and the US census bureau site will give you a lot of interesting easy to understand graphics for different countries. Try it--no advanced demographic degrees required!
Munich, Germany: After reading about casualty figures in Iraq at around the 40 to 50 thousand mark, I was shocked to read about 600 thousand civilian casualties. But then, when I read how the number was deduced by a poll and then extrapolated, I wasn't so sure anymore that I could trust this figure.
Theoretically, the 600,000 figure might actually be too low, because the questionnaires couldn't be carried out in the most dangerous areas, but still, a theoretical value leaves much to be desired.
Do you have any idea how the 40,000 to 50,000 thousand figure was reached? Do you know how casualty figures have been obtained from past conflicts like Vietnam?
Gilbert Burnham: All estimates, whether what TV people are watching or what foods they eat are from samples which give rates. In the case of mortality studies the rate is the number of deaths per 1000 people per year. In the case of the Iraq study we could compare this with the same households before the invasion. This rate is then applied to the estimated population in the areas being studied, and from that the number of deaths estimated--a very standard approach used in every country. Now around that estimate there will be some degree of uncertainty if this is exactly the right number. That uncertainty is known as the confidence interval. In our study we were 95% confident that the total number of excess deaths was between about 400,000 and 900,000. A big range you say, but our goal was to say has there been a large increase in deaths following the invasion, and the statistic tells us yes there has been. If we wanted to have a smaller confidence interval, then we would have taken a larger sample size and sampled from more clusters. However this would have meant much more risk to interviewers. If we had a more narrow confidence interval, would that have answered our research question better? probably not. One last point that is hard for many people to understand. The number of people or households interviewed and the number of clusters used does NOT depend on the population of the country. At a certain point, taking more samples from more clusters does not increase the validity of the answer--and we calculated those levels before the survey.
Bangor, Maine: To Bridgeport, people are leaving the country in droves. I believe the number is roughly 2 million over the past 18 months. Passport offices have been ordered to stay open seven days a week to accommodate the crowds.
Gilbert Burnham: Good point. As the population changes--in this case from people fleeing the high violence in Iraq, some assumptions in the estimations will change. We were looking at mortality from 1 Jan 2002, so we used the 2004 UNDP population estimates--but things are changing fast, and a new survey would have to make some changes in the estimated population by areas. It is hard to say how this shift would affect estimates of violence, although reducing the population size evenly across all areas would certainly reduce the totals.
Washington, D.C.: Dear Dr. Burnham and Mr. Morley: I wish I could say I was surprised by the reactions to the Hopkins study, but I'm not (I'm still disappointed, though). Of course, I'd be willing to bet a month's worth of my salary that the Hopkins number of 655,000 excess Iraqi deaths is closer to reality than President Bush's 30,000, but I'd be interested to know from Dr. Burnham what he thinks the lowest possible "true" number is, say the bottom of the 99% confidence interval.
Gilbert Burnham: I am going down the list to address some other questions. I am not fixed on the 654,000 number--it well could be above or below--but statistically we are 95% confident that the true number is in that range. My greater concern is to move away from the number and think of the lives that this represents--something that the President and I are in 200% agreement on. I want to take it farther to say we need to find better ways to protect the innocents caught up in deadly warfare, and this starts--from our public health perspective--with data. Now not all the people killed were innocent, we are quite sure of that. But statistical methods have not advanced to the point where motives before death can be easily determined--so we must depend on what people tell us.
Boston, Mass.: Since the vast majority of the world has no understanding of statistics do you ever consider modifying your sampling methods? Perhaps if you went with a much larger sample size (like 10x) and wiped out a lot of the uncertainty it would be more influential. I know the goal is not to convince people, just lay out the data, I was just wondering if it was on the teams' minds.
Gilbert Burnham: This consequence of study has been a massive exercise in building statistical awareness and awareness in sampling. As we develop new methods, they get more complicated, and harder even for experience field people to understand--yet their capacity to provide more valid results also improves. What's the solution? perhaps statistics 101 is offered in 1st grade? I don't know, but almost all health data you read comes from samples--and many of the methods are truly complex!
Brooklyn, N.Y.: I liked your methodology of sampling. That's probably not the real issue, which I'm suspecting may be reporting biases. I'm still puzzled on the huge (10 to 1) discrepancy in deaths reported between sources. Given that you were surveying people living in the same streets of the same neighborhoods, to what extent did different respondents inform you of the same person's death. What steps did you take to remove double-counting, and how many of these double reports were there?
Gilbert Burnham: Keeping bias out of sampling is a huge challenge, and we spend much of our time before a survey thinking about this. People living close together are more likely to have common experiences, so a cluster survey may have built in bias in that manner. To get away from that we increase the sample size. In this survey at the end we calculated this clustering effect, which turned out to be 1.6--meaning that people living close together were 1.6 times more likely to share experience. Fortunately, we had anticipated this, and the sample size was more than adequate to compensate for clustering. Double counting of deaths was a risk we were concerned with. We went through each record by hand to look for this, and did not find any double counting in this survey. The survey team were experience in community surveys, so they knew to avoid this potential trap. We were lucky to have Iraqi doctors well educated in community surveys to do the work.
Portland, Ore.: Thank you for taking my question, which is in two parts.
First, could you briefly identify potential sources of sampling and nonsampling error in your study?
Second, it has been implied by sources associated with the administration that the timing of this study was politically timed. But didn't the editorial board of the Lancet decide when to run this study? How long did this study undergo peer review by the Lancet until the decision was taken to publish it?
Gilbert Burnham: There are a variety of potential sample pitfalls to be wary of. Since you seem to have some knowledge in this area, I would refer you to the discussion in the Lancet article for our discussion of these. We have tried to prevent these from occurring, and I think we were successful.
This was not politically timed. We wanted it out much earlier for exactly this confusion risk. We started working on it in November 2005, and hoped to have it out in July or August, but many delays happened. The Lancet had a very detailed peer review, and then we worked closely with an editor to get the paper in its final form. This followed a standard track through the Lancet process--something over which no author has control--although we wish some times!
Baltimore, Md.: Is your group continuing to sample the population for future analysis of the ongoing death rate?
Gilbert Burnham: Of course after any research the scientists always say "what did we learn from this about doing another study?" we learned a lot from the 2004 study, and we know things we would like to expand next time. However, at this point I would like for the security to improve before there was another study. Not just from the danger side, but so we could spend more time at each house to learn more of how households coped with conflict. These data would help the humanitarian agencies know better how to address the needs of people in conflict.
San Francisco, Calif.: "Data collected at specific sites such as clinics and morgues never equal the total events happening." Yes, but a discrepancy of an order of magnitude? Is this common? Also, you say the number of clusters improves the accuracy, which I assume means narrows the margin of error. What are the high and lows estimates from your study?
Gilbert Burnham: This order of magniture can be huge, depending on the the state of systems and the level of conflict going on. In many places the health system may stop completely trying to tabulate numbers. In others it appears that they manage to capture only about 20% of events which are measured through surveys.
Surprise, Ariz.: For Americans to really put the number of deaths of Iraqi civilians in perspective, don't you think we should show Americans that 30,000 Iraqi deaths with their population of 26,000,000 would be equivalent to 345,000 American deaths with our population of 300,000,000. That would be a hundred times more Americans than were killed in 9/11.
Gilbert Burnham: We could look at this in many ways. probably 10% of Vietnamese died in that war, 1.4% of Americans died in the American Civil War, Perhaps 10% of some Eastern European countries died in WWII. However, I don't like these comparisons because we are talking human lives here, and there should not be a threshold where something suddenly becomes intolerable. All deaths, American soldiers (and I am a Vietnam era Army vet) and all civilians, as well as combatants are tragic events that we must try to prevent wherever this might be possible.
Washington, D.C.: Dr. Burnham,
As a demographer, I can attest to the fact that the use of cross-sectional survey sampling to estimate death rates is a pretty standard demographic method. Assuming that the survey is representative of the mortality experience experience of the population at a moment in time, you should get an unbiased estimate. I think that your survey is probably as good as one could hope given the terrible circumstances in Iraq. However, what I am curious about is how you determined the baseline mortality from the earlier period (2002). Did you calculate these yourself or did you use a third party estimate, such as the U.N. or U.S. Census Bureau? Personally, I think that the accuracy of the baseline is probably the greatest weakness in your study.
Gilbert Burnham: Good question. This was a cohort study so we used the same household experiences from 1 Jan 2002- March 2003 as the comparison period. It also came out that this was almost exactly the US Census Bureau estimate as well--which of course made us feel good.
Jefferson Morley: Dr. Burnham
First of all my apologies for coming the discussion late. Second, thanks very much for time to talk to us.
I think you have defended your methodology quite admirably. I'd like to take the discussion beyond the numbers into political and human realities.
Take the very most conservative estimate of civilian deaths in Iraq derived from your study. As a public health official what adjectives would you use to describe that situation for the Iraqi people?
And: if the U.S. government (including the official who criticize your findings) wanted solid numbers on the civilian death toll in Iraq, what would you suggest they do to assist researchers?
Gilbert Burnham: Welcome Jeff,
Our Center in Baltimore is one that is focused on public health in disasters--and we look at many types of disasters--being heavily involved in estimating deaths in Aceh after the tsunami and also from famine in North Korea. Of all the disasters--war is the worst--any way you sample, any way you count. Life is obviously terrible for the Iraqi people--something that the President and I will agree on. How can we use data to find ways to improve the changes of these people--so they don't have to flee their homes and their country? This is the critical question. We may not have that answer now, but I hope we have made a step toward showing we can get those data--so now no excuses for thinking about the big protection issues!
Saint Brieuc, France: Hello, and thanks for taking my question.
Is the pre-war death rate really dependable, and stable?
Also, is it really possible to use an estimated death rate to say what would have happened? How can we be sure how many of these deaths would have happened without the invasion? (say, a collapse of the health service, an insurrection etc).
Gilbert Burnham: Last question I think. Probably not too bad, although we did not depend on someone else's estimate. Clearly Iraq will need a careful census, and maybe even a pre-census to get at these issues. But now is not a time when there is the stability needed for a thorough job.
Austin, Tex.: If you use the pre-war death rate of 9 that the UN used, versus 5.5, would the total number of deaths attributable to War decline to 100K?
Gilbert Burnham: No, as we used the death experience in the households we sampled as our comparison
Oslo, Norway: What exactly is this study supposed to prove? That Iraq was better off with Saddam Hussein in power?
Gilbert Burnham: I want to use this message to say thanks to everyone for great questions, and I am sorry that I did not get to everyone's question in the time I had. This survey was supposed to prove if the data are there to support a large increase (doubling) of the prewar mortality rate. That study question was answered. We are public health people not political scientists or policy makers. Our place on this planet is to generate numbers and data and information for others to use in making the right decisions. Somehow We have found ourselves in the middle of a political storm when our intents were to help other to think seriously about what happens to innocents--and even the perhaps not innocents--who get caught up in conflict. We have a huge job to convince people that we do NOT have political motives in this. We are disaster people. However I am not so sure we are getting our non-political message across as well as I had hoped.
Thanks for this question and to all of you for your great questions.
Good bye from Baltimore
Jefferson Morley: Our hour is up. Thanks to Dr. Burnham for sharing his thoughts and insights into his most important study. And thanks to everybody for asking such penetrating questions. Given the obviously high level of interest, I will be writing about this issue again. Look for World Opinion Roundup every Tuesday, Wednesday and Thursday at washingtonpost.com. If you don't see if on the home page, just put "Morley" into the site search engine and you will get the latest posting. Take care.
Editor's Note: washingtonpost.com moderators retain editorial control over Live Online discussions and choose the most relevant questions for guests and hosts; guests and hosts can decline to answer questions. washingtonpost.com is not responsible for any content posted by third parties.