A New QB Score

One of the updates in the paperback edition of The Wages of Wins is a new QB Score model. The QB Score model in the hardcover edition was as follows:

QB Score = Yards – 3*Plays – 50*All turnovers

In the paperback edition, QB Score is now:

QB Score = Yards – 3*Plays – 30*All turnovers

Obviously the change is with respect to the value of turnovers.  In the paperback we explain

1. why the value of turnovers was changed.

2. why this change really doesn’t matter.

Why QB Score has been changed?

As detailed in “Back to Back Evaluations on the Gridiron”, a paper published in Statistical Thinking in Sports (edited by Jim Albert and Ruud Koning), QB Score is derived from two models.  The first model looks at how many points an offense scores and the second examines how many points a defense allows.  The independent variables used in these models include the statistics tracking how a team acquires the ball, how the team moves the ball across the field of play, a team’s ability to maintain possession of the ball, and how a team scores.

When the model was initially created it included return yards on kick-offs, punts, and interceptions; as well as the opponent’s punt average.  After The Wages of Wins was published in May of 2006, though, Kevin Quinn (a professor of economics at St. Norbert’s College) noted that Football Outsiders had data on the average starting position of each team (thanks Kevin and Football Outsiders).

With data in hand the two aforementioned models were re-estimated.  The results were a bit better (explanatory power in each model increased by about 5%).  And as detailed in Tables One and Two, the impact of turnovers is now a bit less. 

Tables One and Two: The Value of Yards, Plays, and Turnovers

The statistics are valued both in terms of points and wins.  Originally an interception was found to cost a team (in terms of its impact on points scored and points allowed) about 3.7 points and 0.11 wins.  The new model indicated that the cost is only about 2.7 points and 0.08 wins.  A similar change was seen for fumbles lost.

QB Score was constructed by calculating the value of each statistic in terms of one yard.  Originally one interception was worth -44.7 yards while a fumble lost cost -47.6 yards.  A play, independent of yards, cost a team -2.7 yards.  These values were simplified to 50 and 3 to create the QB Score model.

The new model indicates that a play has about the same value in terms of yards.  But an interception now costs a team 34.5 yards while a fumble lost cost a team 36.4 yards. Given these new values, the new QB Score model now uses 30 as the value of all turnovers.

Does any of this matter in our evaluation of quarterbacks?

The answer to this question is a surprising “No”. To see this, all quarterbacks who participated in at least 100 plays in a season were evaluated from 1994 to 2006.  Across this time period, there are 607 observations of a quarterback participating in 100 plays in one year.  These quarterbacks were evaluated with the old model – in terms of QB Score, Net Points, and Wins Produced – and the new model (again with QB Score, Net Points, and Wins Produced).  The correlation between the old and new model was then calculated. These correlations are reported in Tables Three and Four.

Tables Three and Four: The Correlation between the Old and New QB Score

First issue I want to note is that moving from Net Points (or Wins Produced) to the simple QB Score measure doesn’t really matter much.  The correlation between the Old QB Score and the Old Net Points (or Old Wins Produced) is 0.99.  Likewise there isn’t much difference going to the new models either.  The Old QB Score and the New QB Score have a 0.97 correlation.

When we look at this on a per play basis – which is how we normally look at this – the correlation is even stronger.  The Old QB Score per play and the New QB Score per play have a 0.9875 correlation, or out to just two decimals, a 0.99 correlation.

In sum the changes to QB Score are not going to have much impact on our evaluations of quarterbacks.

A New RB Score

Of course if there is a new QB Score there is also going to be a new RB Score.  The old RB Score was as follows:

RB Score = Yards – 3*Plays – 50*Fumbles Lost

The new RB Score will be

RB Score = Yards – 3*Plays – 30*Fumbles Lost

As I did last year, each week I will evaluate the quarterbacks.  This year I will also add in an evaluation of the running backs as well.

A Disclaimer

The page detailing the final QB Score rankings for 2006 begins with the following disclaimer: “As we note in the book, statistics in the NFL do not necessarily represent a quarterback’s ability. So these evaluations should not been seen as conclusive evidence that one quarterback is “better” than another.”

Quarterbacks (and running backs) are quite inconsistent across time (unlike basketball players who are quite consistent across time).  This point about quarterbacks was made in the following posts:

Consistent Inconsistency in Football

Football Outsiders and QB Score

The Value of Player Statistics in the NFL

In The Wages of Wins we argue that the inconsistency on the gridiron is a reflection of the impact teammates and coaches have on player performance in the NFL.  A quarterback might have “good” statistics because he is “good”, his teammates are “good”, and/or he plays in a “good” system. The statistics cannot tell us who is ultimately responsible for the outcome we observe.  Hence the primary purpose of tracking statistics – it allows us to assign responsibility for outcomes – is not achieved in football.  This observation should always be kept in mind when we look at statistical evaluations of football players.

– DJ

Posted in: Football Stories