In March, a study by James Brander, et al. was published online by the Journal of Quantitative Analysis in Sports. The authors sought to estimate a typical player’s career arc, something we might use as a reference for our projections. The authors made a few decisions that I think are questionable and that I suspect impacted their results appreciably.
To start, they focused on per-game performance rather than a per-minute basis. They noted that minutes played per game is a major driver of points scored per game, but argue that points per game is superior for two reasons:
First, they note that stamina may play a role in an older player getting more minutes. It might have some influence, of course, but I would argue that it’s not the primary factor. I don’t think endurance is the reason why as rookies, Steven Stamkos played 14 minutes 56 seconds per game, Jakub Voracek played 12:40 per game, Jamie Benn played 14:42 per game, Logan Couture played 10:16 per game, Max Pacioretty played 12:37 per game, James van Riemsdyk played 12:58 per game, etc.
They also suggest that what I’m calling a bug might really be a feature, that having usage be a significant factor is a good thing if we’re trying to measure performance and coaches give their better players more playing time. Personally, I’d prefer not to convolve the two effects; separately analyzing playing time and scoring per minute would seem to be obviously superior — especially since we have some evidence that teams may be systematically giving younger players less credit than they deserve.
A second concern I have is with their decision not to adjust for era effects. They look at the fourteen seasons from 1997-98 through 2011-12, noting that long-run variations can make comparing across different eras difficult but assert that “the years covered in our analysis should be comparable.”
It’s true, scoring hasn’t changed all that much over the period in question. But it hasn’t been constant by any means — scoring spiked 18% in 2005-06, largely as a result of a dramatic increase in the number of penalties called. Since the authors do not attempt to correct for this, it will introduce considerable bias; when the five lowest-scoring seasons all occur in the first half of their study, on average, this will make player point totals look worse in their early seasons and better in their late seasons.
This effect can be clearly observed in the authors’ analysis of plus-minus; plus-minus is much less susceptible to this kind of era effect, and indeed their analysis of plus-minus suggests a peak age nearly a year and a half younger than what they conclude by looking at points per game. When an aging curve has a difference of just 6 percent between performance at age 24 and a peak at age 28, accounting for league-wide changes of a few percent will be important.
Finally, I have concerns with the authors’ treatment of selection bias. They recognize that good players tend to have longer careers than bad ones, and that you get the wrong answer if you do not in some manner account for the fact that players without much talent don’t generally play in the NHL at age 19 or 39. Unfortunately, they don’t recognize that ability is not the only factor that impacts career length; randomness plays a role in selection bias as well.
Everyone has some days, months, and even years when the bounces happen to go their way. The shots off the post go in, the player avoids nagging injuries, his linemates play well. The marginal player who runs hot for a year likely finds another NHL job the next year — and cools off, producing a data point of a player getting much worse from one year to the next. An analogously marginal player who ran cold for a year is much less likely to get an offer for the following year, so his expected increase is missing from the data. The result is a systematic skew which can significantly impact the results of a study. I would have much preferred to see the authors avoid this problem by either factoring in non-NHL data or accounting for the role of these random fluctuations at the NHL level.
“Players don’t peak until age 28 or 29” makes a catchy headline, and this has gotten the authors a lot of attention. But given what a flat plateau they estimate the aging curve to be, the exact location of the peak is very sensitive to the data treatment, and seemingly minor decisions can have a big impact on the outcome of the analysis. If you start with their estimate that forwards peak at 27.6 years old, slice off a year for the effect of usage and a year and a half for the overlooked era effects, you arrive at an estimate that is much more in line with what the online analytical community has concluded.
Eric Tulsky is a statistical analyst who has done work for the Nashville Predators and other teams. You can find more of his work at his blog, Outnumbered.