Each fall semester, Meteorology students in my “Introduction to Weather Analysis” course at Penn State University conduct a semester-long forecast verification study as part of the class. Students work in groups, with each group assigned a different city.
With AccuWeather having recently introduced its 45-day forecasts in August, I thought it would be interesting and informative for this year’s students to be some of the first to quantitatively assess the skill of this new product.
With that in mind, from September 2 to December 3 (a 93-day period), my students retrieved the maximum (high) temperature forecasts from AccuWeather.com for 15 cities: Denver, Glasgow, Bismarck, Minneapolis, Marquette, Buffalo, Caribou, Boston, Philadelphia, Cleveland, Chicago, Kansas City, Memphis, Louisville and Cape Hatteras. The students then computed the absolute value of the difference between the forecasts and the observed high temperature, and averaged those “absolute errors” for each forecast day to arrive at mean absolute errors for the Day-1, Day-2, Day-3, … , Day-45 forecasts.
All forecasts were retrieved at 7:30 a.m. ET using zip codes appropriate to the observing site at which the forecasts were verified (typically, the airports). For quality assurance, I also independently retrieved the forecasts and the observed maximum temperatures.
To assess the skill of the AccuWeather forecasts, the historical National Weather Service averages (“normals”) derived from the period 1981-2010 were also used as “forecasts” – this “climatology forecast” is commonly used as a basis for comparison.
The results for Minneapolis, Boston and Memphis (below) are representative of the rest of the cities. Depending on the location, AccuWeather becomes less accurate (on average) than climatology at anywhere between 9 and 11 days lead time. The average of the results for the 15 cities demonstrates that beyond this time frame, the AccuWeather forecasts actually have “negative skill” – that is, climatology, on average, is more accurate than the deterministic AccuWeather forecasts.
To experienced forecasters, these results are not surprising, though they were an eye-opening first-hand lesson for many students who were being forced to consider the limits of predictability in a formal way for the first time.
Another way to assess the skill of the AccuWeather forecasts with respect to climatology is to simply compute the percentage of forecast errors of a certain magnitude, and plot the frequency distribution for various forecast days. I chose error ranges of 0-5oF, 6-10oF, 11-15oF, and 16oF and larger, corresponding (informally) to “good,” “poor,” “bad,” and “very bad” forecasts.
The distribution of errors (see below) for all three-day forecasts clearly shows the utility of short-range temperature forecasts over climatology, with 85% of all AccuWeather forecasts within 5oF of the observed temperature, but just 2% with errors of 11oF or greater.
At a lead time of 15 days (see below), however, climatology has a greater percentage of forecasts with errors of 0-5oF and 6-10oF, while Accuweather has a greater percentage (32%) of forecasts than climatology (22%) in the “bad” and “very bad” categories.
Though these results are based on a limited dataset that spanned only one season (essentially, meteorological autumn) and only maximum temperatures were considered, they are remarkably consistent with the experience of seasoned mid-latitude forecasters: Beyond about 10 days, climatology (on average) is as good as, if not better than, any deterministic forecast.
The author, Jon Nese, is a professor of meteorology at Penn State University and associate head of its undergraduate program. Nese was an on-camera analyst at The Weather Channel from 2002-2005.