Football Outsiders and QB Score

Our book has many, many end notes. And these are recommended reading since some of what we think are the most humorous comments in the book are tucked away in these notes.The following, though, is not one of the funny comments (at least, it wasn’t meant to be funny). On page 254 of our book (the 18th note of Chapter Nine) we state the following (with one typo corrected):

Our QB Score measure is similar to the metric developed by Allen Barra (1986). Barra multiplies interceptions by 50 and subtracts this from gross passing yards. He then takes this result and divides by the number of passes thrown. We would like to note that these measures of a quarterback’s productivity should not be considered the final word on this subject. The work offered by Aaron Schatz (2005) and the people at FootballOutsiders.com, published annually in the Pro Football Prospectus, goes beyond our measures of Net Points, Wins Produced, and QB Score. For those interested in measures with more complexity, their work is recommended.

Football Outsiders does have more complex measures. Where QB Score – which is explained HERE and HERE, is based on the analysis of aggregate season data, Football Outsiders builds their metrics on the analysis of play-by-play data. Consequently, their data set is much larger and one might expect, provides a better view of a quarterback’s productivity.

Previously I addressed the ranking of quarterbacks in 2005 and the issues of consistency in quarterback performance. In this post I wish to address two issues: How different would the rankings of quarterbacks be if one went from QB Score (or Net Points or Wins Produced) to the metrics of Football Outsiders? More importantly, would the consistency story we tell about quarterbacks be different if we relied on the Football Outsiders measures?To begin, let’s briefly explain two measures Football Outsiders employs: Value Over Average (VOA) and Points Above Replacement (PAR).

To begin with, VOA is a measure of a player’s performance per play while PAR is an aggregate measure.Each needs a bit more explanation. VOA is defined by Football Outsiders as follows:

VOA breaks down every single play of the NFL season to see how much success offensive players achieved in each specific situation compared to the league average. It uses a value based on both total yards and yards towards a first down, based on work done by Pete Palmer, Bob Carroll, and John Thorn in their seminal book, The Hidden Game of Football. On first down, a play is considered a success if it gains 45% of needed yards; on second down, a play needs to gain 60% of needed yards; on third or fourth down, only gaining a new first down is considered success.

Football Outsiders doesn’t stop with VOA. They also report DVOA, which is defined as follows:

By adjusting each play based on the defense’s average success in stopping that type of play over the course of a season, we get DVOA, or Defense-adjusted Value Over Average.

Beyond VOA and DVOA, Football Outsiders also reports PAR and DPAR – or Defense adjusted Points Above Replacement. PAR is explained in some detail at Football Outsiders, an explanation one can read by following this link. For our purposes, one simply needs to know that PAR is an aggregate measure – like Wins Produced or Net Points – while VOA is a per-play metric – like Net Points per Play.

As noted, VOA and PAR rely on play-by-play data. So calculating the Football Outsider measures takes much more effort than one would expend in the calculation of QB Score. Is all this effort worth it?

To answer this question, let’s first look at how the ranking of quarterbacks differs between VOA and QB Score per Play. Football Outsiders reports evaluations of quarterbacks from 2000 to 2006. Focusing on quarterbacks who attempted 224 passes, I looked at the correlation between the rankings from the Football Outsiders measures and the rankings one would have using QB Score, Net Points, and Wins Produced. The results are reported HERE.

As one can see, the correlation between VOA and QB Score per Play is 95%. So these measures are not telling the exact same story, but its close. To illustrate the similarities, I looked at the top five quarterbacks in QB Score per play from 2000 to 2005, which one can see HERE. In four of these six seasons the top five in QB Score per Play and VOA was identical. In the two years there was a difference, four of top five quarterbacks were the same with the one difference ranked 6th in VOA.

Now QB Score per play does not adjust for the quality of the opponent, although I suppose you could make such an adjustment. DVOA does take this step, and consequently the difference between DVOA and QB Score per Play is greater. Specifically the correlation between these two metrics is only 90%.

A similar story can be told when we compare QB Score to Football Outsider’s aggregate metrics, PAR and DPAR. The consistency between PAR and aggregate QB Score is the same as what we observed between VOA and QB Score per Play. Again we see a correlation of 95%. When we look at DPAR and QB Score the correlation again drops, this time to 92%.

So this exercise reveals that there is a great deal of consistency between the Football Outsiders metrics and the metrics we report in The Wages of Wins.

What of the more important story? We argued that quarterbacks are not very consistent across time. When we look at VOA and DVOA we see the same story. Specifically, looking at a sample of quarterbacks who threw at least 224 passes in successive seasons we find that only 12% of a quarterback’s VOA in a current season is explained by last season’s VOA. For DVOA the explanatory power is 13%. These findings, along with what we report in the book, can be seen HERE.

In sum, the predictive power of VOA and DVOA is about the same as what we observe for the measures reported in The Wages of Wins.

None of this should be surprising. Again, a quarterback’s performance depends upon his teammates and coaches. In the book we noted that the quality of the opponent might also be important, but even when Football Outsiders adjusts for that factor predictive power stays basically the same.

So what does this exercise tell us about football? Like baseball and basketball, analysis in football relies upon player statistics. But unlike baseball and basketball, a quarterback’s performance depends immensely upon his teammates. Hence, in the end, the statistics tracked for quarterbacks cannot tell us if a specific signal caller is great, or his teammates are great. Or, from the other perspective, the stats can’t tell us if a quarterback is bad, or if his teammates are bad. And that is the story we tell whether one uses QB Score, Net Points, Wins Produced, VOA, or DVOA.

In my next post, I am going to discuss what all this means for player evaluation in the NFL.

– DJ

Posted in: Football Stories