My latest column for the Huffington Post discusses the research on quarterbacks and the NFL draft I recently published with Rob Simmons. This research has generated quite a bit of interest. Malcolm Gladwell discussed this in a New Yorker article in December of 2008. This article then became part of Gladwell’s What the Dog Saw, and Steven Pinker – in his review of this book – singled out the story of quarterbacks and the NFL draft as one particular tale he didn’t like.
Pinker’s reaction led to quite a bit of discussion at Gladwell’s blog. And it led to two posts in this forum.
The Inconsistent Quarterback Story Told Again in Less than 3,000 Words
Steven Pinker, Malcolm Gladwell, and Me
Brian Burke – of the Advanced NFL Stats – also participated in this discussion last year. And last month he offered two more posts on the topic.
Are Top Draft QBs Any Better Than Late Round Picks?
Steven Pinker vs. Malcolm Gladwell and Drafting QBs
Brian asked if I could comment on these latest stories. And after a delay of a few weeks, I finally found the time to write down some thoughts.
Where We Agree
Let me begin by noting two points where I think Brian and I agree. First, Brian and I both believe one needs to look at per-play performance. In other words, we know that quarterbacks taken earlier in the draft will see more time on the field. So if you look at aggregate performance metrics, quarterbacks taken earlier will tend to look better. But if these quarterbacks are truly offering more – and not just playing more – then we should see evidence in the per-play numbers.
We also appear to agree that the correlation between draft position and performance shouldn’t be very large. A quarterback’s performance – as Brian notes – is dependent upon more than just the quarterback’s ability. The performance of teammates and coaches also plays a significant role. All of these interaction effects would be difficult for scouts to forecast. So even if the scouts were focusing on the correct factors in evaluating a quarterback’s ability, there would still be a problem.
Given this point, one wonders what this discussion is about. Brian and I agree on the fundamental issue: Draft position doesn’t do a great job of predicting future performance. Although it is true we are in more agreement than some might suspect, there are some issues with Brian’s latest approach to this issue that I think I should note.
The Story of the Research
Before I get to my reaction to Brian’s latest, though, let me briefly tell the story of the research Rob and I have published. The impression one gets from much of the discussion people have offered is that Rob and I simply looked at the correlation between draft position and performance. Once we saw low correlation coefficients, we wrote our paper.
If one reads the paper, though, one sees there is much more to our tale. The question that motivated our story was “what determines where a quarterback is selected in the NFL draft?” The list of factors we considered included a quarterback’s height (in inches), his body mass index (and BMI squared), his Wonderlic score, time in the 40 yard dash, whether or not he played in the Division I-AA or Football Championship Subdivision (as opposed to the Football Bowl Subdivision or what was known as Division I-A), and various performance metrics. Our list of performance metrics included career plays, completion percentage, interceptions per attempt, yards per attempt, the NFL’s QB rating, and various Wages of Wins metrics (i.e. Wins Produced per play, Net Points per play, and QB Score per play).
What we found is that draft position is a function of height (taller quarterbacks are drafted first), BMI (and BMI squared), the Wonderlic score (quarterbacks with higher scores get drafted first), 40 yard dash times (faster quarterbacks get drafted first), playing FCS football (playing for an FCS team means you get drafted later), career plays (more plays gets you drafted earlier so it is better to stay in school), and interceptions per attempt (more interceptions and you get drafted later). We also found – in different estimations – that the NFL’s QB rating and the Wages of Wins metrics mattered. But completion percentage and yards per attempt did not statistically impact draft position.
Okay, all of this looks good. Quarterbacks are being evaluated with respect to performance on the field as well as factors measured at the NFL combine.
We then, though, looked at how these factors predicted future performance. We considered a variety of performance measures (i.e. Wins Produced per 100 plays, the NFL’s QB rating, completion percentage, interceptions per attempt, and passing yards per attempt). We also looked at performance at different points in a quarterback’s career. In sum, we ran many, many regressions. And here is our basic result (directly quoted from the paper): In all of our formulations, we never found that the combine factors, or the college performance with respect to Wins Produced per 100 plays or QB rating, had a significant impact—of the expected sign—on NFL Wins Produced per play or NFL QB Rating at any level of experience in the NFL.
Now we did find that completion percentage in the NFL was statistically related to college completion percentage (although the explanatory power was somewhat low). Completion percentage in college, though, did not predict where a quarterback was taken. And the factors that did predict where a quarterback was taken did not predict future performance.
Quarterback and Draft Position
In presenting our research we began with the story of draft position and quarterback performance. But the actual time-line of the research is reversed. Once we had evidence that the factors that drive draft position were not related to future performance we wondered “does draft position predict future performance?” It is this research that Brian discusses, but I am not sure I agree with his approach. What follows are some reactions to his two posts.
Sample and Draft Order
Beyond the issues discussed above, our study also looked at the link between quarterback performance and where a quarterback was selected in the draft from 1970 to 2007. Brian has argued that the 1970s should not be included in the data set. He has also argued that where a quarterback is selected isn’t as important as the order in which the quarterbacks were chosen because the former is impacted by team need. The order – Brian argues — should tell us more about how decision-makers ranked the quarterbacks.
Darren Rovell recently asked me to update our analysis for an article he posted at his CNBC blog. Darren’s article – Is it Time to Abolish the NFL Combine? — offers analysis that incorporates both of Brian’s points. Specifically, Darren reports the following two tables:
Table One: Performance of Quarterbacks by Draft Pick
Table Two: Correlation between Draft Position and Performance
These two figures get at both of Brian’s suggestions. And as one can see, our story remains the same. When we look at performance from 1980 to 2009 we still see that quarterbacks selected from 11-50 outperform those taken from 1-10. Furthermore, when we look at draft order and performance – as opposed to draft position and performance – we still see very low correlations.
Evaluating Players Who Didn’t Play (or Play Much)
So why are Brian finding somewhat different results? The key is what to do with quarterbacks who didn’t play or didn’t play much. The correlations reported in our article (and in the analysis posted by Darren) restricted the sample of quarterbacks to those who attempted at least 100 passes (what we did in the book) or participated in at least 100 plays (what I did for Darren). In other words, the analysis required that a quarterback play in about four games per season.
The basic argument for this restriction is that the numbers from a quarterback who didn’t play much in his career are probably not reflective of a quarterback’s value. To illustrate how the first games of a quarterback’s are misleading, guess the identity of the quarterback who posted the following numbers in his first 73 pass attempts: 384 passing yards, 31 rushing yards (on five attempts), seven interceptions, one touchdown, and a QB Score of -29.
These numbers indicate this quarterback is very bad. Luckily for this quarterback, he got another chance to play. And eventually, John Elway – yes, this is what Elway did in his first 73 attempts – got much better. For quarterbacks taken later in the draft, though, their career probably ends after these first 73 attempts.
Now is it likely that all quarterbacks who play badly will become Elway? No, and that isn’t the point we are making by restricting our study to quarterbacks who actually spent time on the field. What we are arguing is that a quarterback who really hasn’t played much is not generating numbers that tell us very much. And since we want to know how draft position relates to performance, we need to include performance measures in our analysis that are indicative of a quarterback’s actual ability.
In Brian’s view, though, these players who didn’t play much or never played at all were simply not good enough. Consequently, Brian either a) includes players in his analysis who had less than 100 plays (or pass attempts) and/or b) assigns a relatively low value to players who never played.
The approach taken by Brian increases the correlation between draft position and performance. By assuming that quarterbacks taken later would be poor quarterbacks, or by including the numbers generated by lower drafted quarterbacks who didn’t play much, Brian biased his evaluation of quarterbacks taken later downwards. And consequently, the correlation between draft position and performance is increased (although not by much).
In the end, when you assume those drafted in later rounds are poor players, you suddenly discover a somewhat stronger relationship between draft position and performance. But it’s important to emphasize how this analysis is constructed. Brian has essentially assumed his answer.
And there is reason to think quarterbacks who never played – or played very little – are not evaluated perfectly. Quarterbacks are not like other players on a football team. A back-up at almost every other position gets to play some in each game. Teams, though, want to play their starting quarterback. Consequently, during the week a back-up quarterback doesn’t get many reps in practices. And even if the back-up was getting reps, practice – where defensive players don’t get to keep hitting the quarterback – is not the same as a real game. Therefore, we should not be surprised by stories where a quarterback is traded or cut by one team only to later excel for another franchise (i.e. Kurt Warner). It simply is hard to evaluate how a quarterback who only gets a few practice reps is going to actually perform week after week as a starting quarterback.
Another Look
The focus on quarterbacks taken later in the draft, though, can obscure the basic story Rob and I are telling. Let me offer another look at this issue that highlights the basic problem decision-makers have in evaluating future signal callers. ESPN.com reports that the NFL drafted 14 quarterbacks in 2010. ESPN.com also lists between 40 to 50 quarterbacks who could have been selected. And there are even more quarterbacks who played football for an FBS or FCS school who are not listed. One suspects that there are more than 14 quarterbacks who played college football last year who had the physical skills to play in the NFL. But because the NFL only needs less than 100 quarterbacks, many of these players do not get the opportunity.
Let’s, though, focus on those that do get an opportunity. In 2009 the NFL selected nine quarterbacks (of the more than 40 ESPN listed as potential draft picks). The first of these nine was Matt Stafford. He received a contract worth $41.7 million in guaranteed money from the Detroit Lions. Four picks later the Jets selected Mark Sanchez. His contract had $28 million in guaranteed money. And then with the 17th pick in the draft, Josh Freeman went to the Tampa Bay Buccaneers. Freeman signed a contract with a guarantee of $10.2 million.
Such a pattern is hardly unique to the 2009 draft. A few positions in the NFL draft are literally worth millions of dollars. But does the NFL know enough to award contracts in this fashion?
Again, we have already seen that many factors that get a quarterback drafted are not related to future performance. But we can also look at the link between draft position and performance. This time, though, let’s completely ignore the late round picks and only focus on quarterbacks taken in the first sixty picks. And let’s consider quarterbacks who averaged at least 50 plays per season (as opposed to 100). Finally, we will also consider order of selection as well as a quarterback’s draft position. Looking at data from 1980 to the present, here are the correlations between draft position and performance:
Table Three: Evaluating the Top 60 Selections from 1980 to 2008
As one can see, our story is still the same. Where a quarterback is taken in these first 60 picks tells us quite a bit about the quarterback’s pay. But it doesn’t tell us much about his performance in the NFL.
And again, this shouldn’t be that surprising. There appear to be significant issues with respect to how quarterbacks are evaluated on draft day. Many of the factors that drive draft position are not related to future performance. Furthermore – and this is a point often made in this forum (as well as in The Wages of Wins and Stumbling on Wins) – performance of quarterbacks in the NFL is hard to predict even when we are using past NFL performance to make the prediction. Given all these issues, we should not be surprised by the stories of Tim Couch, Akili Smith, David Carr, Joey Harrington, Alex Smith, JaMarcus Russell, etc. Each of these quarterbacks was supposed to be a future stars on draft day. But as fans of the teams that selected these players, the star performances envisioned on draft day never actually appeared.
– DJ
The WoW Journal Comments Policy
Wages of Wins Journal 
bduran
May 26, 2010
If QB metrics are inconsistent because the position depends so much on team performance, is it really surprising that QBs taken in the first 10 spots, i.e. by the 10 worst teams, do worse? It seems like you would expect 11-50 to do better than 1-10, because picks 1-10 are all made by bad teams whereas some of the picks in 11-50 are made by the average to great teams.
Jeff
May 26, 2010
Not having read the paper, is anything done to control for endogeneity concerns regarding top QB picks? They tend to go to really terrible teams with terrible offensive lines, terrible running backs, terrible receivers, terrible defense . . .
Roethlisberger did not seem to do very much as a rookie, but he played for a team that was solid in most facets of the game.
dberri
May 26, 2010
Jeff and bduran,
I think you have hit on one point Rob and I are making. A quarterback’s performance depends on his teammates.
One thing I would note… we looked at performance after three, four, five, six, seven, and eight years. Surely that is enough time for anyone to build a team around any quarterback. And again, draft position and performance are not related.
Once again, we think this is also because much of what people focus upon on draft day is not related to future performance.
Dre
May 26, 2010
Dr. Berri,
So Brian’s analysis was a bit confusing for me. I was wondering if you could comment on a few things.
His first chart looks very noisy, but he puts a downward arrow through it, is this really justified?
His by position(e.g. first QB selected) and by draft round seem subject to less data in later info problems. E.G. the 13-14th selected QB are low justifying the downward arrow, but there was only one of each compared to several in earlier.
Also some language based things. His APY he says each pick should decrease it by .006, this seems really small compared to what his graphs seem to show. Second he mentions two seemingly contrasting points. One is by era the average raises so he adjusts QBs accordingly midway through their careers. He then lists age as a possible factor for degradation.
I may go over the article a bit more later, but those things seemed a little off to me.
bduran
May 26, 2010
dberri,
Thanks for clearing that up. The only other thing I can think of is maybe by comparing rounds and controlling for draft order. Something like doing a comparison of all QBs drafted in the first ten spots in each round, then the second, then the third and seeing if this changes anything. Of course, if you’re looking at QBs six, seven, and eight years out I would agree that that probably answers the question well enough.
ilikeflowers
May 26, 2010
[Off Topic]
Check out these APM and StdErr values for this basketball-reference blog post.
Almost all of the APM’s are essentially meaningless.
Tommy_Grand
May 26, 2010
Professor,
If (for some bizzare reason) you could look at only one metric before drafting a QB for your NFL team, would you look at college completion percentage?
Alex
May 26, 2010
Dr. Berri – I agree with your take on the ‘assuming the answer’ issue, but what do you think of Brian’s analysis suggesting that the low-playing-time players would need to be average or slightly better to eliminate the correlation?
dberri
May 26, 2010
Tommy,
Completion percentage in college does appear to be related to completion percentage in the NFL. So this is something decision-makers should consider. The other stats are difficult to interpret since there are differences in the talent employed in college and the pros.
Alex,
I sent Brian some additional comments on his work. On that point, I simply noted that we would expect very little correlation if we assume quarterbacks who didn’t play were average. And we expect a higher correlation if we assume these quarterbacks were very bad. So it just seemed to me that all we were seeing is the distribution of the data (i.e. a movement from the lower range of the distribution back to average). I was not very surprised by this particular argument. It seemed fairly clear that this is what you would have to see.
Devin
May 26, 2010
This seems like you’re assuming your conclusion though. For quarterbacks who wash out of the league, NFL decision makers must think they are some of the worst players in the league, or else they would be playing. By valuing them as if they performed as one of the worst players in the league, as NFL GM’s do, the correlation becomes much stronger than just looking at the lower round quarterbacks who do make it. Only by assuming that NFL GM’s don’t know what they’re doing can you value them as an average player, but that’s just a circular argument.
dberri
May 26, 2010
Devin,
Your point would have more validity if that was all we did. We started the analysis by showing the problems decision-makers had in evaluating the information available on draft day. Again, what we see from the combine — and much of what we see from the player statistics — are not related to future performance. That is the key finding in this study. So we have evidence — independent of the study of draft position and performance — that decision-makers are making mistakes on draft day.
TK
May 26, 2010
I’m super-conflicted by the various stories that can be told.
Analysis over 8 years surely evens out the team quality, but there’s no doubting that QBs taken 1-10 are disproportionately ikely to spend their formative years with a bad team.
Plus, the very fact that the pay scale is so skewed can warp the decision-making of coaches and GMs to force young QBs into the lineup earlier (or later) than might be optimal.
I’m also sympathetic to Brian’s argument that leaving out the abject failures (fewer than 100 plays) overweights the later round successes. After all, no first rounder (even JaMarcus) ever flames out without throwing 100 passes, so ALL of the first round busts are included in the sample, but by only considering P>100, you’re ejecting a great many later round busts from the pool, no?
Still puzzled…
Dan
May 27, 2010
Any analysis that you do has to make some assumption about the quarterbacks who were drafted but did not play. Brian was explicit about his assumption (he assigned them a certain level of performance). Your assumption is implicit in your method: you are effectively assuming that the quarterbacks who did not have enough attempts to be included in your regression were just as good, on average, as the quarterbacks at the same draft position who did play enough to qualify for your sample. By excluding late second round quarterbacks like Marques Tuiasosopo who didn’t play much, you are assuming that the ability of all second round quarterbacks (Tuiasosopo included) is reflected in the performance of the late second round quarterbacks who did play sufficiently (like Charlie Batch). This seems very implausible – it would require teams’ decisions about who to play at QB to be completely unrelated to the quarterbacks’ ability. There’s a reason why Batch got into the lineup more than Tuiasosopo, and why Chad Henne has been playing while Brian Brohm has not.
These assumptions can have a big effect on your conclusions, as Brian’s posts showed. You can also just look at the number of QBs who didn’t have enough attempts to qualify to see how important these assumptions are. Using the PFR draft querier, you can see that more than half of all quarterbacks drafted since 1970 (through 2007) didn’t even get 100 career passing attempts, including just 3% of first rounders, 29% of second rounders, and 74% of seventh rounders. Obviously your conclusions will depend on what you do about all of these quarterbacks whose performance you can’t measure, and the more optimistic your assumptions about the non-players, the better it will look to draft a quarterback in the second round (or the seventh) instead of the first.
Westy
May 27, 2010
I have to agree with Dan. It would seem reasonable to me that, in fact, you’re seeing an even greater selection bias in the late rounds as QBs chosen there must be extremely good to overcome the (very present) bias towards players in whom more is invested. Thus, only the cream of the crop reach the level at which they’re measured in this study.
How consistent is QB play? Is it safe to say that you do agree there is a difference between top and average performers? Or is it almost entirely dependent on the team? Presuming you agree there is a difference, research to determine what attributes make a good QB good would seem to be one of the most important research topics for NFL teams.
Mookie
May 27, 2010
Don’t count out Alex Smith just yet. He may not have lived up to expectations his first four years, but the new Spread-like offense is making him thrive. By adding two solid offensive linemen in the first round, and having Crabtree for the whole year next year, a probowl is definitely not out of reach.
EdG
May 27, 2010
I think Elway’s stats can also be looked at another way. Top picks are thrown into the game before they’re ready to play because they’re top picks. Their stats suffer accordingly. Later picks get a lot longer to prep for their first game and the ones who never get good enough are thrown out of your analysis. That’s a huge flaw.
Dan
May 27, 2010
Another way to run the analysis is to define what counts as a “good” career for a QB and then to look at what percentage of players drafted in a certain range turned out to have good careers. For instance, if we define a “good” as having a career adjusted yards/attempt* over 6.0 and at least 1,000 career pass attempts, then 44% of the first rounders taken from 1980-2005 have had good careers (24/54), while only 23% of QBs taken in the second or third round during those years have been good (12/53). So a first rounder is almost twice as likely as a quarterback taken in the 2nd or 3rd round to have a good career (and the ratio is similar if we change the standards for a “good career”). Maybe this is just because a bunch of second and third round quarterbacks who could’ve had good careers never got a fair chance, but we can’t know that.
* adjusted yards per attempt is defined at pro-football-reference.com as (pass yards + 20 x TDs – 45 x INTs)/(pass att). The players just above the 6.0 cutoff are Drew Bledsoe, Jim Harbaugh, and Michael Vick; just below it are Chris Miller, Vinny Testaverde, and Jake Plummer.
Ty
May 28, 2010
Quarterbacks are inextricably woven into the fabric of their team. Offensive scheme, coaches, wideouts, offensive line, running game, defense, whether the home crowd is supportive or throws beer bottles when you step on the field . . . all of these factors play enormous roles in the development of a quarterback. It’s not so much whether the quarterback is “good” or not, as whether or not he’s “right”.
It’s true that Jamarcus was drafted #1 overall for his size and his arm–not whether or not he could play quarterback. He’s a perfect example of the fallacies your study showed. But, I’m not sure Peyton Manning makes it through that organization alive, let alone Jamarcus. Meanwhile, the Aaron Rodgerses and Joe Flaccos of the world fall to great teams, are handled well, and are enjoying great success.
What I’m getting at is that it might be that the 49ers should have taken Aaron Rodgers’ proven production over Alex Smith’s physical tools–but even if they had, the two quarterbacks might have had each other’s careers.
Peace
Ty
dberri
May 28, 2010
Ty,
I think you are correct. And hopefully the Lions do well with Matt Stafford. Again, I don’t think anyone could know if Stafford was worth what they paid him after the draft. But maybe he will develop into someone worth the money. As a Lions fan, all you can do is hope.
Ty
June 1, 2010
David–
Thanks for the reply–and yes, you hit it spot-on; all we can do is hope. His career numbers didn’t look great at Georgia, and his rookie season numbers don’t look great either. But there were times last year when he was the only thing working for the Lions, and it became clear that the Lions’ chance for victory was zero without him, and nonzero with him.
Only time will tell, of course–but for now, these few minutes are the hook we hang our hopes on:
http://www.thelionsinwinter.com/2010/03/matthew-stafford-against-browns-one.html
Peace
Ty
Jim
June 3, 2010
Great Read!
The Shazad
June 8, 2010
By ignoring players without 100 pass attempts, this analysis is introducing a critical assumption that biases the result. This much is clear by how much the correlation changes when making explicit assumptions about the performance of the QBs who don’t get playing time in games.
Given that a QB has plenty of playing time in a season, it appears that his performance is independent of draft position. But this is still a conditional probability, and to make the claim that “draft position doesn’t do a great job of predicting future performance,” we must remove the conditional regarding playing time. For if the players who don’t get playing time are below average, then draft position certainly does predict performance.
Yes, draft position affects playing time. Yes, there are factors outside of a QB’s ability that affect playing time. Yes, you can find anecdotal evidence of good QBs who were underplayed. But none of that proves that players who don’t receive playing time are just as good as those who do. And that point is necessary if one is to claim that draft position and performance are essentially independent.
I recognize that you don’t want to try to make up data where there is none. You don’t know how good Colt Brennan really is, so you don’t want to include made up stats for him in the data set. But the absence of an assumption in this case is still a de facto assumption that has a profound impact on the result of the analysis.
Westy
June 8, 2010
What he said.