2 November, 2004: ... but one hundred thousand deaths is an easily-abused statistic

[ Home page | Web log ]

Most of my half-dozen readers will by now have heard about the research conducted by Les Roberts, Riyadh Lafta, Richard Garfield, Jamal Khudhairi and Gilbert Burnham, published in the current issue of The Lancet, estimating the number of civilians killed during the war against Iraq (so far). If you have not already done so, please read the paper; it is interesting, and will tell you much more about it than any number of `about the research' pieces in the newspapers or (god forbid) on people's web logs.

... Now you have done so, you will be familiar with the central conclusion of the piece, expressed by its authors as a quiet understatement:

In this case, the lack of precision does not hinder the clear identication of the major public-health problem in Iraq-- violence.

Very briefly, the way the study was conducted was to conduct surveys of clusters of households randomly assigned to bits of Iraq weighted by population. Each survey asked householders to record deaths of members of the household in a period prior to the invasion of Iraq and in a similar period after the invasion. From these data they estimated the death rate before and after the invasion, and from this formed an estimate of the total number of excess deaths from the invasion.

The authors acknowledge that, under current conditions, Iraq is not an easy country in which to conduct such a survey:

During September, 2004, many roads were not under the control of the Government of Iraq or coalition forces. Local police checkpoints were perceived by team members as target identification screens for rebel groups. To lessen risks to investigators,we sought to minimise travel distances and the number of Governorates to visit,while still sampling from all regions of the country.

-- and the survey certainly isn't as accurate as the ideal which could be expected in a country at peace. The headline figure -- of a conservative estimate of 98,000 deaths due to the war -- is the center of a 95% confidence interval which stretches from 8,000 to 194,000 deaths. (Note that this figure does not include information from Fallujah, an area which has suffered more violence than most and in which it was very difficult to do the survey. Including the figured from Fallujah increases the estimate by about 200,000 but broadens the confidence interval further.)

A number of commentators have argued that this wide uncertainty makes the study worthless. This is nonsense, but it is important to understand what it actually means.

Specifically, under the assumptions of the survey, the authors estimate that there is a 95% chance that the two limiting values they obtain enclose the true number of deaths. Loosely, you can turn this around and regard this as a 95% chance that the true value lies in the interval (and hence a 5% chance that the true number lies outside that interval). Further (if we assume a symmetric distribution, which is plausible), there is a 50% chance that the total number of deaths exceeded 98,000, and a 2.5% chance that it was less than 8,000.

To put this into English, it means `we're not sure exactly how many, but a hell of a lot of people almost certainly died'. Make up your own mind, but in my view the low estimates of the number of dead in Iraq should now be regarded with deep suspicion.

There have been several criticisms of this study, most of them rather silly. Tim Lambert and Daniel Davies on Crooked Timber discuss and rebut a variety of complaints. A few more criticisms should, perhaps, be added:

Another source of criticism of this research has been 10 Downing Street. Last Friday, the Official Spokesman complained that,

Firstly, the survey appeared to be based on an extrapolation technique rather than a detailed body count. Our worries centred on the fact that the technique in question appeared to treat Iraq as if every area was one and the same. In terms of the level of conflict, that was definitely not the case. Secondly, the survey appeared to assume that bombing had taken place throughout Iraq. Again, that was not true. It had been focussed primarily on areas such as Fallujah. Consequently, we did not believe that extrapolation was an appropriate technique to use.

This is -- I've searched in vain for a politer way to put it -- crap.

Almost all national-level statistics are based on extrapolation (or, more accurately, sampling). Are we supposed to believe that (say) the Census or (for another example) Government research showing public support for ID cards are worthless because they did not count each individual living person in the country?

The implication that no casualty figures could be accurate unless they are derived from a `detailed body count' is also absurd, especially given that Coalition forces have refused to conduct any such research; in any case, a `body count' would severely underestimate the total number killed -- partly because many bodies will not be recovered (for instance, those killed when bombed buildings collapse), and partly because it's now impossible accurately to count the bodies of those who have already died and been buried.

Further, the survey did not treat `Iraq as if every area was one and the same', as even a cursory inspection of the paper will tell the reader. Similarly, the survey did not `assume that bombing had taken place throughout Iraq'; instead, samples were taken at numerous locations in order to account for the geographical distribution of damage (many of the sampled areas were unbombed, as you would expect). Specifically, as I have remarked, the headline number excludes Fallujah, because of the high concentration of bombing and difficulties of conducting the survey there.

And yesterday, the Official Spokesman was at it again, repeating the same false statements and adding some new stuff to confuse the lobby journalists:

Asked to explain further the Government's previous concerns and doubts about the methodology applied in the ``Lancet'' article about the number of Iraqi deaths, the PMOS replied because it relied on the extrapolation technique assumed, that Iraq was uniform in terms of intensity of conflict. It wasn't. The article also assumed that bombing was general throughout Iraq, which was not the case. The Iraqi Department of Health had issued figures that showed over a 6 month period there were about 3,000 deaths, which was a long way short of the figures quoted in the ``Lancet''. The Iraqi DOH measured those figures by the number of people who came into hospitals throughout Iraq, and it was very difficult to rely on any such figures quoted in the ``Lancet'' with any certainty.

So: the Iraqi Department of Health counted the number of people who came into hospitals and then died. And they got a different figure from a survey which attempted to estimate the total number of people who died, whether in or out of hospital. Quelle fucking surprise. It's -- at the risk of pointing out the obvious -- a war zone! People who get hit by a LASER-guided bomb don't have time to go to hospital before they die! Of course the fucking numbers differ.

What's going on here, then? I am not sure, and indeed am having trouble untangling my cynicism about others' competence from my cynicism about others' honesty. But my best guess: part of this is misdirection -- the idea, I think, being to try to convince journalists who may be too lazy to check the story properly that It's Not Quite As Bad As They Think; as for the rest, well, the charitable explanation is that the Downing Street press office doesn't know any statistics and doesn't know anyone who does. The more cynical explanation is that this is simple and shameful dishonesty, a feeble attempt to discredit a study which suggests that the war against Iraq has killed the very people it was supposed to save by the tens of thousands.

This is one of the most disappointing pieces of government information I've seen since -- not coincidentally -- that crap they put out about `weapons of mass destruction'. A sad business, all in all.

Moving on, many supporters of the war have been embarrassed by this study, apparently because they do not understand what a war is. For the avoidance of doubt, the fact that one hundred thousand people may have been killed as a result of the late unpleasantness does not necessarily mean that the war was a mistake or `wrong', any more than the fact that a third to one half that number were killed in a single day in Dresden in February 1945 made the war against the Nazis a mistake or `wrong'. (Of course, the civilians killed in Dresden were killed deliberately, whereas -- charitably -- most of those killed in Iraq have been killed recklessly.)

It may tell us more about the conduct of the war (is it really a good idea to supress an insurgency by dropping five hundred pound bombs on urban areas, even if they are LASER-guided?); but in my opinion it is more telling that many apologists for the present war -- usually the first people to compare Saddam to Hitler and Iraq in 2003-4 to Germany in 1939-45 -- have confused these two issues. In wars, people die, often by mistake and often in large numbers. The case the war apologists have now to make is not about the exact numbers of deaths -- later, we will better know how many died, but for the moment 98,000 is the best estimate available -- but whether the deaths were necessary.

And, frankly, it doesn't look very good so far, does it?

Lastly, as a brief comment on the conduct of the war, I will draw your attention to this film taken from an F16 flying over Fallujah in April. (Don't pay too much attention to the commentary on that page, but the video is linked from there; sadly it's in the proprietary -- and rubbish -- Microsoft `Media Player' format, but `xine' will play it.)

Channel 4 News covered this footage, which shows a pilot guiding a bomb towards a building, presumably under instructions from a forward air controller on the ground. While the bomb is falling, a crowd of people appears in the street adjacent to the building and the pilot asks whether he should direct the bomb onto them. The person on the ground tells him to, and he does.

There is some controversy about whether the people bombed -- from a rough count there are thirty to forty people on the video -- were combatants or not. You can read about that in the sites I've linked to. That's not what I want to draw your attention to, though. Here are some frame grabs from the video (lousy quality, sorry). First, a general view from before the crowd emerges, to give a sense of scale:

Street scene

Note cross-hair, showing the original target of the bomb, designated by a LASER shining from the 'plane onto the ground. The pointing of the LASER can be changed as the bomb is falling, allowing the target of the bomb to be altered. The street in the center of the picture is quite wide -- about 40 meters, judging from the size of the people in the second picture, and it is surrounded by two- and three-storey buildings.

Crowd, about to be bombed

Here is the crowd of people. By this stage the pilot of the 'plane is steering the bomb towards them. For those who can't play the video, the dialogue between the pilot and ground controller ran,

Pilot: I got numerous individuals on the road. Do you want me to take those out?

Controller: Take 'em out.

(As an aside, remember that the controller on the ground may have a much better view of proceedings than the pilot -- these people may have been running towards his position shooting at him, for instance, and we would not be able to tell that from the video.)

Note the counter at the lower right of the picture, reading `000:11' -- that is the time before the bomb hits the ground.


And here is the cloud of debris about a second after the explosion.

Bearing in mind the width of the street and the nature of the surrounding area, consider this:

Most everything will be severely damaged, injured, destroyed, or killed within 20 meters of a 500-pound bomb blast....

Safe distances for unprotected troops are approximately 1,000 meters for 2000-pound bombs and 500 meters for 500-pound ones. Even protected troops are not entirely safe within 240 meters of a 2,000-pound bomb or 220 meters of a 500-pound bomb.

During the war proper -- before the `Mission Accomplished' stunt -- the Americans dropped about 18,000 guided bombs. (This ignores a further 10,000 or so `dumb' bombs.) As a brief estimation exercise, if all of those were dropped in urban areas with a population density of 5,000/km² (probably an underestimate), then for a uniform distribution of targets we'd expect around 100,000 casualties from bombs with a 20-meter lethal radius. That's certainly an overestimate (even the US Air Force no longer bombs at random) but the order of magnitude is... suggestive.

Copyright (c) 2004 Chris Lightfoot; available under a Creative Commons License.