I’m reminded of the recent case of Reinhart and Rogoff who introduced major errors in data processing and analysis into an influential paper on macroeconomic policy. The errors were undiscovered for years.
In both these cases, I think the problem is not so much the errors — mistakes will happen, and it’s understandable that researchers will be less likely to catch their errors if they go in the direction that support their views — but rather that there are many fragile links in the chain that connects data to policy recommendations. This is one reason that many people are starting to recommend that the “paper trail” of the statistical analysis be more transparent, so that researchers (including me!) simply aren’t able to make mistakes such as accidentally removing a minus sign in a computer file or losing a column of data in an Excel spreadsheet.
The other problem is that people who are caught out in their mistakes often go on and on about how the mistakes don’t really alter their conclusions. For example, here’s Richard Tol the effects of his missing minus signs and new data (in addition to adding the overlooked estimates, he added five more recent estimates that had appeared after the 2009 paper had been written):
Although the numbers have changed, the conclusions have not. The difference between the new and old results is not statistically significant. There is no qualitative change either.
But then he also says:
The assessment of the impacts of profound climate change has been revised: We are now less pessimistic than we used to be.
The original impact curve projects an impact of −15 (−7 to −33) percent of income for a 5°C warming, whereas the corrected and updated curve has −6 (−3 to −21) percent. This is relevant because the benefits of climate policy are correspondingly revised downwards.
I’m a little bit worried because this last sentence contains so many minus signs, but I’ll assume they’ve all been checked carefully.
In any case, I think Tol needs to get his story straight: in one place he said the qualitative conclusions did not change; in another place he points to some changes and say they are relevant for policy.
Okay, so all this got me wondering: What exactly was being claimed here? Here are the two graphs from Tol’s revised paper:
Figure 1 shows the estimates based on the corrected data (with the minus 1’s restored) and the overlooked previous studies. Indeed, the curves aren’t very different, but . . . what is that big positive point at 1.0 degrees? This point seems to be (a) singlehandedly keeping the estimate above zero, and (b) driving a big curve in that fitted quadratic line. This point is credited to this 2002 paper by Tol. The list also includes a negative estimate of the impact of a 2.5 degree warming, also from Tol in 1995. Lots of research got done between 1995 and 2002, I assume — indeed, the 2002 paper reports “a new set of estimates” and “two methodological improvements,” so I’m surprised that the 1995 estimate was just kept in as is.
Here’s the point. In his 2002 paper, Tol found a positive economic effect of a small increase in temperature. But, assuming he still stands by his 1995 paper, he finds a big drop in economic performance as temperature rises still more: the estimated effect (compared to no warming) is plus 2.3 percent of gross domestic product for a 1 degree increase, and then drops to minus 1.9 percent for a 2.5 degree increase. So there’s some major nonlinearity in his model. This could well be correct, but I think it would make sense for him to explore what in his model is driving this result.
Okay, now here’s Tol’s Figure 2 which shows the estimates including the new studies:
That point at (1, 2.5) remains, and now there’s a new outlier at (3.2, minus 11.5). I don’t know what’s the story with these, but of course either of them could be correct; the whole point of this sort of meta-analysis is a recognition that different studies are using completely different methods with completely different sets of assumptions.
That quadratic curve
But I want to go back to one point, which is Tol’s remark that the revised estimate based on the new data “is relevant because the benefits of climate policy are correspondingly revised downwards.”
What’s going on here? The green curves in Figure 2 look reasonable in the sense that they go through the data, which really is all I have to work with here. The real weirdness comes in Figure 1, not so much with the curve going above 0 but with the steep declining slope which leads to huge negative impacts when extrapolating beyond the range of the points on the graph. This seems to be coming from the assumption that the curve is a quadratic (that is, a curve of the form y = ax – bx^2). But why must it be quadratic? Where’s that coming from? The answer (I’m pretty sure): nowhere at all. It’s just an extrapolation.
If you want to extrapolate, I think it would make more sense to extrapolate within the contexts of the individual reports rather than to draw a curve through the estimates obtained by different authors at different times.
One problem which Tol didn’t note was the role of the changing minus signs in interpreting the estimates that were not garbled. In particular, his estimate of a big positive impact at 1 degree is a clear outlier in his analysis. Did he look into that in the original paper? I took a look, and here’s what he wrote, back in 2009:
Given that the studies in Table 1 use different methods, it is striking that the estimates are in broad agreement on a number of points—indeed, the uncertainty analysis displayed in Figure 1 reveals that no estimate is an obvious outlier.
In this way, a misclassification of a couple of points can affect the interpretation of a third point.
Tol also wrote:
The fitted line in Figure 1 suggests that the turning point in terms of economic benefits occurs at about 1.1 degrees Celsius warming (with a standard deviation of 0.7 degrees Celsius).
This turning point has disappeared in his new Figure 2, so, again, I do think the new analysis has changed his conclusions in a real way.
P.S. Discussion of global warming has of course become politicized (see this post by Bob Ward for some background), and indeed Tol used the scare-word “shrill” in his 2009 article. So I should probably emphasize that in that paper he also wrote of “considerable uncertainty about the economic impact of climate change … negative surprises are more likely than positive ones. … The policy implication is that reduction of greenhouse gas emissions should err on the ambitious side.”