The weekly football column is all about “synergy”. But before we get to that, let’s look at the weekly QB Score and RB Score rankings:
Okay, we see the numbers. Here are my football thoughts for the week.
When we look at basketball and football players, we think of the word “synergy”. Players in these sports must work together for a team to be successful. This characteristic of basketball and football appears to make the evaluation of individuals difficult. Are the numbers a player posts due to his efforts? Or is it just his teammates?
One way to think about this issue is to consider the consistency of performance over time. If a player’s numbers changes as his teammates and coaches change, we suspect a player’s numbers are not about him. In contrast, if numbers stay fairly constant even as teammates and coaches change, we suspect the role of teammates (and coaches) is not that important.
In basketball, we tend to see consistency. We can see this in systematic regression analysis. We can also see this when we look at the career performance of specific players like Kobe Bryant, Jason Kidd, LeBron James, Yao Ming, Tim Duncan, Kevin Garnett, and Paul Pierce. Despite coaches and teammates changing, year after year these players are extremely good (and we can see the same pattern with “bad” players).
When we turn to football, though, we see a very different story. Regression analysis tells us that the numbers posted by quarterbacks are extremely inconsistent over time. We can also see this pattern when you look at individual players. Brett Favre will be phenomenal one year (2004) and below average the next (2005). And you can see the same pattern when you look at many other signal callers.
Changes in Manning and Brady
One player who has defied this tendency is Peyton Manning. An average quarterback will post a QB Score per play of 1.85. Across the past eight seasons, the lowest mark Manning has posted is 2.36 (2001). Six times he has posted marks that are over 3.00.
After seven games in 2007 we saw the same story. Manning had posted a QB Score per play of 3.78, a mark that ranked second in the NFL (Tom Brady was #1). And then something happened. Marvin Harrison finally stopped playing after Week Seven. Since Harrison went to the sideline Manning has had a couple of good games (most recently this past Thursday against Atlanta). But overall he has been below average, posting a QB Score per play of 1.53 since Harrison went completely away.
Let’s review. Manning plays six games where he looks like Manning has always looked. And then Harrison – who Manning has been able to throw to since he entered the league – exits the field. Since Harrison left, the great Manning is suddenly below average. This leads us to ask, is the greatness of Manning just about Harrison?
And Manning is not the only quarterback where we ask such a question. Tom Brady is in his 7th season as a starting quarterback for the Patriots. Across his first six seasons he posted a QB Score per play of 2.10, with a career high of 3.01 in 2005. After ten games in 2007, though, Brady has a mark of 4.72. To put that mark in perspective, Manning posted a QB Score per play in 2004 of 4.78, the highest mark observed since 1994 (when my data set begins).
How did Brady go from being a “good” quarterback to a quarterback who rivals the numbers of Manning at his best? One suspects the key is Randy Moss. Why Moss? Well, back in 2004, Daunte Culpepper, with Moss on his team, posted a 3.50 QB Score per play in 2004. After Moss left Minnesota, Culpepper was well below average (before he got hurt). In sum, Moss — when he is motivated — tends to make quarterbacks look really good.
What We Don’t Know About Quarterbacks
And so we see one aspect of the problem facing those charged with the task of evaluating quarterbacks. How do you separate the quarterback from his receivers? A quarterback cannot complete any passes without his receivers. So is the completed pass because of the receiver’s talent, or the quarterback’s abilities?
The problem is not just with receivers. What about the impact of the offensive line? Or the running game? Or the play calling of the coaches? All of these factors impact the numbers we assign to the quarterback. And all of these factors are independent of the quarterback’s talent.
Given this problem, are we sure Tony Romo is worth $67.5 million? Or is his above average play simply because he gets to throw to Terrell Owens (who is probably going to retire before Romo’s contract is up)?
Or what about the much maligned Joey Harrington? Harrington has been about average in Atlanta this year, despite having little talent around him. If Harrington got to throw to a motivated Moss (or Harrison, or Owens), would Harrington be a Pro Bowl quarterback?
A few weeks ago I brought up this very issue in talking about talent evaluation with an NBA executive. Let me repeat what I said then. Every NBA executive should be thrilled they don’t work in the NFL. The NBA has very good data that allows decision-makers to both explain the past and forecast the future. In contrast, the NFL’s data doesn’t tell us who was responsible for what we observed, and it doesn’t allow for adequate forecasting.
And before anyone imagines the NFL has some mysterious data source that does allow decision-makers in football to accurately evaluate talent and predict future performance, remember the pattern we see in the performances of Favre, Manning, etc… If people “knew” why these players were good or bad, I do not think we would see such wild fluctuations in player performance. And current performance would have some statistical relationship to current pay (it doesn’t, but that’s a story for another day). Instead, we find past performance and current pay are related (again, a story for another day). This suggests that decision-makers do consider what they see in the past in evaluating talent. But they find, as we report in The Wages of Wins, past performance in the NFL is simply no guarantee of future returns.
– DJ
For more on QB Score, RB Score and what these metrics mean see
Consistent Inconsistency in Football
Football Outsiders and QB Score
The Value of Player Statistics in the NFL
Wages of Wins Journal 
Gareth Lewin
November 25, 2007
So could it be that the way we evaluate the stats is wrong?
You’ve shown a few times that you can do pretty go analysis of NBA players purely from box score values.
Maybe the same is not true for the NFL? Maybe we need more stats to be tracked (YAC for example), that would allow for better analysis?
If that is true, what stats (except for YACs) do you think would be useful? Obviously we can’t have a stat like “How smart was the throw”. Perhaps more of a breakdown of what happened at what down etc?
dberri
November 25, 2007
Gareth,
I think FootballOutsiders does all this. Their stats aren’t any more consistent, though.
Brian
November 25, 2007
A few thoughts:
Even with YAC accounted for, QB performance is relatively inconsistent. Some other stats that would be useful would be o-line protection stats, receiver drop rates, and a receiver “open” stat of some sort. But even stats like that are inter-dependent on QB and other players’ performances.
The Brady-Moss connection underscores the synergy effect of NFL players. But inconsistency alone does not indicate a synergy effect. The NFL regular season only has 16 games, with an average of about 25-30 pass attempts for a QB. But many of those attempts are throw-aways, Hail Mary passes, or short screens and draws. Compare that to an NBA season of 82 games. Most players probably have more shot attempts a quarter of the way through the season than a starting NFL QB does in a full season. Because of the relatively small number of repetitions, there is a lot of variance due to luck in QB performance, so we should not expect to see the same amount of consistency as we do in other sports.
Strength of schedule is terribly important in the NFL compared to other sports due to its short and severely imbalanced schedule, and this may also explain some inconsistency. QBs can face vastly different pass defenses over the course of a season. For example, the Houston Texans are in a division with 3 of the best defenses in the league this year-IND, JAX, and TEN. Matt Shaub has to play each of them twice. In comparison, the Seahawks play in the NFC West against generally anemic pass defenses.
Statisticians and economists customarily evaluate things with OLS (conventional linear) regression. This appears to work well in basketball, where each player serves roughly the same functions: dribble, pass, shoot, rebound, guard. In football the players’ roles are so specialized that linear/additive combinations of player contribution don’t make sense. (I’ve tried it with *very* limited success.)
To get something like WP48 for football, we’d need some sort of non-linear interaction model. Say Tom Brady and Randy Moss’s skills (and those of all the other players on the field) could be boiled down into one number, the model might be:
Points created = B1 *(TB*RM) + B2*TB + B3*RM
instead of just = B1*TB + B2*RM, which is how most of us think of things now.
With so many players on the field, it would be infinitely difficult to solve for a single player’s contribution. We’d still need a lot of assumptions and simplifications.
One last point, Football Outsiders doesn’t really do this. Their DPAR system assigns “success” points to a player that accounts for game situations (e.g. a 2-yd gain by a RB on 4th and 1 is good, not bad). But it doesn’t account for the synergy of other players around him.
Okapi
November 25, 2007
Related to this post there was an article in the New York Times this weekend on whether running backs are overrated since their success is to a large extent an artifact of the offensive line–
mrparker
November 25, 2007
Football is all about use of weapons. The great players take advantage of having other great players around them.
There is no other way to analyze the game. Its not that Peyton sucks without Harrison. He sucks when his 2nd option wouldn’t make most pro rosters and his oline is down to 60 percent.
Owen
November 26, 2007
Somewhat interesting article in the Times yesterday, saying the new wisdom is that running backs are basically interchangeable. Seems to echo some of your themes…
“Running the Ball: Anyone Can Do it.”
It sort of makes you wonder what does matter in the NFL. It seems like general talent level and depth is the most important thing rather than the superstar approach in the NBA.
fool
November 28, 2007
Okay, so past success doesn’t guarantee future results in the NFL. What about success at the COLLEGE level?
I frequent the blogs at the Houston Chronicle’s website. for nearly two years now the arguments have continued to rage about the Texans’ 2006 draft, wherein they had the first pick and took DE Mario Williams instead of RB Reggie Bush (the “consensus #1” at the national level) or QB Vince Young (the overwhelming preference locally, since Young is a Houstonian and played at Texas).
Yet subsequently it looks like neither Bush nor Young have been all that great, and Mario Williams is doing okay but hardly lighting it up. Probably the best results have come from a couple of RBs at the bottom of the first round and top of the second, Joseph Addai and Maurice Jones-Drew.
The annals of the NFL draft are replete with examples of utter and horrific busts, players who were heralded coming out of college but not only didn’t make a good showing, but were utterly bumfuzzled on the NFL gridiron. And this doesn’t seem to have improved, even though the cash paid to some of these top picks equals the GDP of many third-world countries.
Can’t the NFL figure out a better way to project who in the draft class is going to be a stud, and who will be a bust?