The Wages of Wins discusses how performance can be measured in both the NBA and NFL. The Wages of Wins Journal, though, almost exclusively focuses on the NBA. Why isn’t performance in the NFL discussed more frequently? The answer to this question can be illustrated by comparing the play of Jay Cutler and Kyle Orton.
Cutler and Orton Defy the Pundits
The Chicago Bears finished the 2008 season with a 9-7 record, a mark that fell just short of qualifying for the playoffs. In discussing Chicago’s problems, people tended to focus on the team’s quarterback. As Table One reports, Kyle Orton – the Bears starting quarterback in 2008 — was ranked 25th (out of 32) quarterbacks in both the NFL’s QB Rating system and the Wages of Wins metrics (i.e. QB Score, Net Points, Wins Produced).
Table One: Final Quarterback Rankings for 2008
In the offseason it became clear that Jay Cutler – a player who ranked 7th in Net Points per Play (and Wins Produced per 100 plays or WP100) – was available. So the Bears sent Kyle Orton – plus two first round draft picks and a third round pick – to the Broncos for Cutler. Fans of the Bears rejoiced at this move. And fans of the Denver Broncos became very, very angry. In the pre-season the views of both groups of fans were confirmed. The Bears finished the exhibition season with a 3-1 mark, while the Broncos – led by a less than impressive Orton – finished 1-3. Many NFL pundits were heard expressing the conventional wisdom: You simply don’t trade away a “franchise” quarterback.
And then the real games were played. As December begins, the Broncos are 7-4 while the Bears are 4-7. When we look at each quarterback’s stats – reported in Table Two – we see that the 2008 result has been essentially reversed. Orton now ranks 9th in the NFL in Wins Produced per 100 plays (Wins100) while Cutler is ranked 25th.
Table Two: Week Twelve Quarterback Rankings in 2009
The reversal in the ranking of these two quarterbacks is hardly unique. Nine of the quarterbacks ranked in the top 10 this year qualified for the rankings last year. Of these nine, only four – Drew Brees, Peyton Manning, Philip Rivers, and Matt Schaub – were ranked in the top ten at the end of last year. And we see the same story at the bottom of the rankings. Seven of the players ranked in the bottom ten qualified for the rankings last year. Of these, only two – JaMarcus Russell, and Derek Anderson – ranked in the bottom ten in 2008.
Despite such inconsistency, fans of the NFL – and apparently at least some decision-makers – can be impressed by a quarterback’s past numbers. Consequently, the Bears can be tempted to give up three draft picks and a starting quarterback for an apparent “franchise” signal caller. And the Chiefs can give up a second round pick and significant dollars for Matt Cassel (currently ranked 26th).
The problem facing decision-makers in the NFL is the numbers – which are often cited – don’t tell us very much about the future performance of a quarterback. A quarterback’s statistics depend on his teammates and the quality of his coaching. Change the teammates and coaches and you often see the numbers change as well. Unlike basketball – where player statistics are remarkably consistent from season to season – football numbers suffer from very significant interaction effects. This means those numbers – which told us that Cassel and Cutler are “great” quarterbacks – may not tell us much about what these quarterbacks will do when these players change teams.
And it’s important to note that this isn’t just some numbers or some quarterbacks. Less than 25% of a quarterback’s completion percentage and passing yards per attempt are explained by what the quarterback did with respect to these statistics last season. Less than 10% of touchdowns per pass attempt this season are explained by last year; and when we turn to interceptions per attempt, explanatory power falls to less than 2% (these results come from an examination of 399 quarterbacks who played consecutive seasons from 1994 to 2007). When we turn to measures such as QB Score, the NFL’s quarterback rating, or the numbers at FootballOutsiders.com, again we see inconsistency (explanatory power is less than 20%).
Such results tell us that what we see from Cutler and Orton in 2008 and 2009 should not be surprising. Predicting performance of quarterbacks in the NFL is simply very difficult (and this is not just the story I tell, but also the story told by Brian Burke at Advanced NFL Stats).
This is really a fascinating story. But the story was essentially told in The Wages of Wins. And I told it again during the 2006, 2007, and 2008 NFL season. Consequently, this is what I said towards the end of my discussion of the final quarterback rankings in 2008: “…the measurement of performance in football really only tells one story. The interaction effects in football cause the performance statistics to be inconsistent. So the players we see perform well today are not necessarily going to perform well tomorrow. Although I like telling that story, it’s really about all I ever say about the NFL. Consequently, this very long post … might be my last post on football.”
Looking at the NFL Draft Again
But now another aspect of this story has sparked some interest. Rob Simmons and I recently wrote an academic article examining the relationship between where a quarterback is selected in the draft and how he performs in the NFL. For many the results were surprising. As Rob and I report, where a quarterback is taken in the draft is not related to how that quarterback performs in the NFL.
Once again… it’s difficult to predict the future performance of NFL quarterbacks. On draft day NFL decision-makers have an even more difficult challenge. People in the NFL must project how well a quarterback will play in the NFL before he ever plays with — and against — NFL talent. Now if predicting performance of actual NFL quarterbacks is hard, what should one expect to see when it comes to projecting performance of quarterbacks that are not in the NFL?
Well, here is what Rob and I found.
1. We did find several factors that predict where a quarterback will get drafted. Specifically, we find that taller, faster, and smarter (i.e. better Wonderlic scores) quarterbacks get drafted first.
2. The factors that predict draft performance, though, don’t predict NFL performance.
3. Given this result, we shouldn’t be surprised that where a quarterback is drafted doesn’t predict how well a quarterback will perform in the NFL.
This is how point #3 was described a few days ago:
… here is a sample of what we found. After a quarterback has played five seasons in the NFL (minimum 500 career plays), here are the correlation coefficients between draft position and various career statistics:
Completion Percentage: -0.01
Passing Yards per Pass Attempt: -0.02
Touchdowns per Pass Attempt: -0.12
Interceptions per Pass Attempt: 0.00
QB Score per Play: -0.01
Net Points per Play: -0.02
Wins per Play: -0.02
QB Rating: -0.06
Directly below this data — and I mean, directly below this data – I wrote the following sentences:
Our data set runs from 1970 to 2007 (adjustments were made for how performance changed over time). We also looked at career performance after 2, 3, 4, 6, 7, and 8 years. In addition, we also looked at what a player did in each year from 1 to 10. And with each data set our story looks essentially the same. The above stats are not really correlated with draft position.
We should note that although draft position and performance are not related – and our story is the same regardless of when we look at the relationship — draft position and salary are clearly correlated. To illustrate, JaMarcus Russell has collected millions of dollars to play quarterback in the NFL. But he clearly has not performed at a level consistent with all those dollars. And a similar story can be told about David Carr, Ryan Leaf, Tim Couch, Joey Harrington, etc… Quarterbacks who are drafted early clearly get paid more. They just don’t seem to perform any better.
Reacting to Some Reactions
There have been a few reactions to this result that I would like to address. Here is a sample of what I have seen.
1. A problem with reading comprehension
Let me start with a response that suggests people don’t always read what’s being said. Despite the sentences I highlighted above, I have read statements like the following (this is comment #10 on Jason Lisk’s post at Pro-Football Reference.com from one of the bloggers that Steven Pinker cited):
The Berri choice to exclude QBs who didn’t play five years in the league is a pretty fundamental error to make.
Hmmm… pretty fundamental error? Perhaps a more fundamental error is not reading a single paragraph that, once again, appeared directly beneath the results I posted.
2. Per-play vs. Aggregate Measures, Part One
Beyond the issue of reading comprehension skills is the objection some people have voiced to how we examined the correlation between draft position and NFL performance. Rob and I focused on per play measures — such as completion percentage, yards per pass attempt, interceptions per pass attempt, touchdowns per pass attempt, NFL’s quarterback rating, QB Score per play, Wins Produced per play, and Net Points per play – in examining the link between draft position and NFL performance (again, at a host of different points in a quarterback’s career).
People have argued, though, that it’s better to look at aggregate measures such as total touchdown passes or total yards. Such examinations show a stronger correlation between draft position and performance (although not that strong). And these examinations show that “better” quarterbacks – where “better” is defined in terms of total touchdowns or total yards – tend to be picked first (again, this is not a strong tendency). Of course, one could define quarterbacks in terms of total interceptions thrown and show the opposite. Quarterbacks chosen first in the draft throw more interceptions, and since interceptions are not good, this means quarterbacks taken first tend to be “worse”.
The results with respect to interceptions — and passing yards and touchdowns — are driven by the fact quarterbacks taken first tend to play more. So by focusing on the aggregate measures one is really looking at the link between one decision (a team liked the quarterback on draft day) and another (the team decided it will play the quarterback it liked on draft day).
The persistence of draft day evaluations in the NFL is reminiscent of a study by Colin Camerer and Roberto Weber offered in a 1999 article looking at the NBA draft. The Camerer-Weber article looked at the factors that predicted minutes per game in the NBA. What they found was that draft position could still predict playing time – even after performance was controlled for – years into a player’s career. It wasn’t that performance didn’t predict playing time. No, the important finding was that draft position – independent of NBA performance – predicted playing time. Such results suggest that NBA teams had trouble ignoring sunk costs in making decisions.
This is essentially what Jason Lisk reported (in a less sophisticated study) with respect to quarterbacks and the NFL draft. Even after controlling for performance, Lisk reported that draft position predicted a quarterback’s playing time.
Such a story confirms the approach Rob and I took in our examination of quarterbacks and the NFL draft. Aggregate numbers are biased because draft position is an independent predictor of playing time. Therefore, one should focus on per-play metrics.
3. Per Play vs. Aggregate Measures, Part Two
One doesn’t need to consider the bias in playing time, though, to defend the choice of per play measures. In evaluating players in sports we tend to focus on measures that consider how many opportunities given the player. For example, in baseball we tend to look at batting average, on-base percentage, slugging percentage, OPS, ERA, etc… In basketball we tend to focus on per-minute measures. And in football, the basic quarterback rating measure is entirely defined in terms of performance per pass attempt.
We tend to think quarterbacks are “better” when they have a higher completion percentage and throw fewer interceptions per pass attempt. Draft position, though, doesn’t predict these measures (or any of the per play measures reported above). But if teams were getting it “right” on draft day, shouldn’t the quarterbacks taken first have a higher completion percentage, or get more yards per pass attempt, or throw fewer interceptions per pass attempt, or produce more wins per play, etc…?
4. Draft Position and Never Playing
Steven Pinker had one more reaction to the construction of our study. Pinker – in the New York Times – noted that lower drafted quarterbacks don’t “merit many plays”. And this somehow establishes that teams are drafting correctly. Again, though, this is using one evaluation to justify another. We expect that NFL teams are going to discount players who were already discounted.
For us to study the link between draft position and performance, we can only consider players who actually performed. It’s possible that those quarterbacks who never performed were really bad quarterbacks. But since they never played, we don’t know that (and Pinker also doesn’t know this). What we do know is that for those quarterbacks who did play, draft position and performance aren’t related.
Another way to think about this is to consider the careers of Kurt Warner and Tom Brady. The numbers tell us that Warner and Brady are among the best quarterbacks of the past decade. Yet both quarterbacks were passed over by teams on draft day (Warner was never selected and Brady was a 6th round draft choice). Are we to believe that Warner and Brady were the only quarterbacks passed over who could really play? It seems likely that at least some of the quarterbacks who never played really could have contributed to an NFL team. But once again, we will never know, since these quarterbacks never played.
And one should add once again… draft position and salary are clearly related. Teams pay much more for a quarterback taken with one of the first ten slots in the draft. But the evidence doesn’t indicate that these quarterbacks perform better than those taken later in the first round, second round, third round, etc….
5. Reacting to an Odd Interpretation of Our Results
All that being said, let me say what we are not saying. Jason Lisk – in the blog post linked to above — notes that past NFL performance predicts future playing time. Such a result is not surprising. Past performance predicts future salaries in the NFL (hence Cassell gets a big payday after last season in the NFL). How Lisk interpreted these results, though, was somewhat odd. Here is what Lisk said towards the end of his post:
If you believe that the only reason Carson Palmer has played a lot more than Gibran Hamdan is because Palmer was drafted alot higher, then you can accept Gladwell’s position.
I certainly don’t recall Malcolm Gladwell saying that draft position was the “only” (this is Lisk’s word) predictor of future playing time. What Gladwell argued – and what we argued – is that draft position couldn’t predict future performance. At no point have I ever argued that NFL decision-makers don’t consider past performance in determining playing time or salaries. In fact – as noted above – we have argued that NFL teams do consider past performance. Unfortunately, past performance is a poor predictor of the future. Hence, it’s not clear that the acquisitions of Cutler or Cassell will ever generate the returns envisioned when those players were acquired.
So we agree with Lisk when he argues that past performance predicts future performance. Where we don’t agree is with the assertion that at some point we argued something else.
Another Study Confirming Our Story
Let me close with a comment left by fellow economist Kevin Quinn at Malcolm Gladwell’s blog (you have to go through a large number of comments to get to Quinn’s thoughts):
I am a sports economist and have investigated the predictability of eventual NFL performance by QBs based on the information available just before the draft. While my approach and methods differed somewhat from those employed by… Dave Berri, my results essentially confirm his findings.
Kevin co-authored a working paper that examined the NFL draft and came to – as Kevin notes – a very similar conclusion (across a smaller sample then Rob and I considered). Again, this result –given what we see when we look at the consistency of performance in the NFL – is not surprising.
And hopefully this extremely lengthy post answers all the reactions to the study Rob Simmons and I published (and yes, this post is less than 3,000 words – although not very far below this mark).
– DJ
The WoW Journal Comments Policy
For more on the Wages of Wins football metrics see
Consistent Inconsistency in Football
Football Outsiders and QB Score
The Value of Player Statistics in the NFL
Wages of Wins Journal 
JoseMesaIsDead.com
December 6, 2009
Dave, I agree wholeheartedly with your analysis, but your comment about “reading comprehension” is a bit of a misfire, I think. Here’s the line from your explanation:
“We also looked at career performance after 2, 3, 4, 6, 7, and 8 years.”
The commenter was referring to the fact that the #5 is missing from that sentence…
dberri
December 6, 2009
No, I think I read it correctly. The analysis I presented was from five years. The commentator then said I made an error by not including anyone who hadn’t played five years. But that isn’t true. We looked at career performance from 2 through 8, and season performance from 1 throught 10.
Even if your reading is correct, though, that is still pretty bad. The five year analysis is what is being presented.
ilikeflowers
December 6, 2009
I’m not sure if you’re joking or not but just in case:
“The Berri choice to exclude QBs who didn’t play five years in the league is a pretty fundamental error to make.”
Assuming that the poster is primarily concerned with excluding QB’s who played less than five years, I think that looking at the numbers through years 2, 3, and 4 is pretty much the opposite of excluding them.
Kevin
December 6, 2009
I certainly appreciate your working so hard for a blog, responding to critiques from other bloggers. You really do the medium justice.
There seem to be 4 big factors with which a quarterback’s performance can interact.
1) strength of offensive line, for which there is no (?) direct, objective criterion.
2) strength of coach/offensive coordinator/system, for which there can be only highly complex (noisy and unreliable?) criterion.
3) speculatively, strength of running game, for which there can be some reliable criterion, though this may just be a proxy for overall quality of offensive line, and the interaction is bi-directional (better passing game means more opportunities for exploitive rushing gains, and better running game means more efficient passing game)
4) quality of receivers, which is fairly reliable, perhaps (some function of (drops vs reception), fumbles, and yards per reception).
I would love to see that model! Any sport which fields a high number of untracked positions which generate no stats is hell to model, I wager. Football qualifies, as at least 5 players in the offense of 11 leave no directly traceable mark in the boxscore.
Add to that, far fewer scoring attempts and fewer games than basketball, and you have a real beast…
John Giagnorio
December 6, 2009
Dave,
This is one of the better posts you’ve ever written. The NBA posts are nice, but I’d definitely like to see more of this sort of thing if you have the time :)
You kind of address this above, but do you have any specific thoughts regarding this bit from Lisk’s article:
“We haven’t accounted for the myriad of late round picks where the initial scouting met the performance teams were seeing in practice, and they never got any extended opportunity to play outside of the practice squad and pre-season contests.”
To me, it’s unclear if this is necessary. I’m sure there are plenty of late rd quarterbacks who aren’t very good, but there must be some who never get the chance – like NBA teams signing Joe Smith over and over again while Nick Fazekas never gets a shot – who could contribute. It doesn’t seem like there’s an easy/fair way to account for this in the type of study you did.
simon
December 6, 2009
Wow that’s heck of a response after all that Pinker fiasco. Thank you for the posting.
Robert Simmons
December 6, 2009
As noted a@@hole Paul Krugman said “[when] the students have missed the point so badly it must be [your] fault.”
http://krugman.blogs.nytimes.com/2008/06/29/stock-flow-equilibrium-wonkish-and-trivial
The commenter thinking you had excluded QBs with less than 5 years was justified, because your writing was insufficiently clear.
Also, your explanation for not including certain players reminds of the parable of the drunk looking for his keys under the street light.
kevin
December 6, 2009
How many failed QBs does it take to get a correlation between draft position and merit? The world may never know…
Man of Steele
December 6, 2009
Kevin,
I basically agree, although the guys at footall outsiders found that success running the game was more highly correlated with passing success than with wins, or something of the sort, from which they concluded that the pass sets up the run, rather than the other way around. I’m not sure if that’s correct, but it bears keeping in mind.
Daniel
December 6, 2009
Care switching Vince Young and Kerry Collins in your rankings? It’s pretty clear that, barring a VY injury, Collins will not be the Titan’s QB again this season.
dberri
December 6, 2009
After Week 12 Vince Young hadn’t thrown enough passes to qualify. If I post the rankings again this year, Young will appear.
Fred
December 6, 2009
Many NFL teams are enamored with picking tall quarterbacks with rocket arms with their high picks in the draft. But accuracy and decision-making are what make a quarterback good (aside from having a good line, good receivers, a believable running threat to improve success on play action, a good scheme, etc.). It’s almost as if NFL teams make a pick hoping that they’re going to land the 6’4″ rocket-arm guy who has the accuracy and decision-making skills to succeed, instead of trying to figure out who has the best chance of success regardless of their raw physical attributes.
Bjorn
December 6, 2009
My main question is one that doesn’t seem to come up in most of the critiques: did you consider draft position to be an ordinal variable, or an interval variable? If you considered it an interval variable, why did you do so? (conventional wisdom would dictate that there is a much greater difference between, say, the 1st and 2nd draft pick than the 91st and 92nd) And if it was considered an ordinal, how did you determine correlation?
(My university stats classes were a long time ago, so forgive me if I’m missing something obvious. I also couldn’t find the original paper to read…anyone have a link?)
Kevin
December 7, 2009
Hey, I’ve posted twice before, I just had one more thought (since finals are finally finished!). The only part of this which will really interest critics is this:
“For us to study the link between draft position and performance, we can only consider players who actually performed. It’s possible that those quarterbacks who never performed were really bad quarterbacks. But since they never played, we don’t know that (and Pinker also doesn’t know this). What we do know is that for those quarterbacks who did play, draft position and performance aren’t related. ”
Is that precisely true? You might have tested some assumptions, for instance, that the performance of players not given a chance to perform might have been normally distributed around the 25 percentile, the 10th percentile, the 30th percentile, or whatever you like. That way you could look at how good (or rather, how bad) those unused QBs needed to have been for a correlation between draft position and real QB value to appear.
That’s not as much work as it seems. Just find an exemplar and plug in his value, for kicks, or find find exemplars, (1 percentile, 20 percentile, etc.) and plug their values in, assuming a normal distribution (or skewed, assuming that diamonds in the rough are indeed rare).
But that is a lot to ask for free. I know I’m not going to do it. Again, I really appreciate your work Prof Berri.
Oh, and since I’m trained as a social psychologist, teasing out moderator variables is interesting to me (is that boring to you, as an economist?). For instance, you’ve made a behavioral claim that football is noisy and messy, and predictions about future performance are nearly impossible to judge from draft criteria. However (I mentioned this previously) a powerful moderator is degree of investment, or a species of loss aversion, which influences judgments so that 1) those highly invested in with a valuable draft pick and a large salary are given undeserved chances and 2) those who are not highly invested in and are in the too close to call slush pile (which is large in football, given that there are so few QB spots to fill) are frequently discarded without care.
Great stuff, but I really think you could solve the unused QB problem by testing a range of assumptions. If you want… I know I would read it!
Notimetoread
December 7, 2009
It seems like what Kevin is calling for is a “Vice President of Common Sense” (http://sports.espn.go.com/espn/page2/story?page=simmons/060519), proposed by Bill Simmons. It’s interesting that he was talking about NFL drafting at the time (how the Texans passed on Reggie Bush to take Mario Williams). Haven’t most NFL pundits now turned this around, with Williams having more of an impact than Bush? But I haven’t followed analysis of non-QB and defense numbers.
What Kevin wrote is also consistent with the possibility that NFL front offices have too many people who voice similar opinions. The difference between herd mentality and the wisdom of crowds is whether the errors are independent or not. If they are independent, then errors, in the aggregate, cancel each other out (for an example, see CASP – critical assessment of structure prediction for proteins.)
They hold a yearly contest whereby scientists can submit their best protein structure models; as it happens, the best performers tend to be “meta-engines”, in essence averaging the results of multiple computational prediction algorithms. The catch is that these algorithms must use different models. Again, if the models are similar, then errors in each would reinforce each other, rather than cancel out. It seems like there is some relevance to how sports related front offices might be structured.
As for Dr. Berri’s response to responses, it always surprises me that the readers mistake which variables are being measured and which are being varied.
1) For the R. Simmons who blames Berri for not being clear: this is a strange comment. Berri was looking at the performance of QBs over as long a time frame as possible. To answer the question of whether QBs improved (in terms of metrics), how much they earned, how many snaps they were given, Berri looked at the same quarterback at different points in his career. While one could argue for including all drafted quarterbacks who had played at least 2 years, this is only an issue with methodology and is not certainly a “fundamental” error.
2) As for Bjorn’s comments about interval vs ordinal variable, I believe this is a proper example “begging the question.” You assumed, as true, the very thing that Berri wanted to test. The hypothesis was, If there is in fact such a difference between first round and sixth round QBs, then there should be some obvious correlation between their draft position and performance. Perhaps one might expect higher passing percentage, more touchdowns, more interceptions, fewer blocked passes, and so on for a first rounder.
But as Berri pointed out, he little to no correlation between performance and draft position. It is also amazing that such middling quarterbacks keep their jobs and can play for so long. Berri noted that this could be a sunk-costs mentality, where no one wants to admit they erred in handing out $15 mil signing bonuses to an untested rookie. As a matter of fact, it remains difficult to predict which veteran QB would do well the following year.
And there are issues with the use of counting stats in such evaluations (of course players who play more with have more good and bad things happen to them: more TDs, more passing yards, more INTs, etc.) It is strange the people hold on to this point: wanting to use aggregate (counting) stats rather than rate-stats. The simplest example I can think of is from baseball: who is the more impressive power hitter? Someone who hits 350 HRs after having played fewer than 1o years, or someone who has 350 HRs after a 20 year career? The implicit evaluation is a rate-stat; HRs divided by time. Berri does a similar thing with his WP48 measure and his per-play stat. And of course, baseball analysts do the same thing by using plate-appearances/HR. Here’s another look at the use of counting stats. One might make the following nonsense argument by arguing that a veteran is better than, the same player, as a rookie, because the veteran would have more HRs, doubles, triples, and hits. This argument might make more sense by arguing whether the vet is the same player he once was, perhaps by using the rate stats (e.g. did his PA/HR ratio increase over time? Has he walked/SO/GIDPed more over the same number of opportunities?)
Westy
December 7, 2009
So obviously we still all think there’s a difference between ‘good’ and ‘bad’ quarterbacks, right? Surrounded by stellar teammates, some players will perform better, and those same players likely will perform better surrounded by poor players.
So if you don’t think QB score predicts future performance well, what does predict (measure) performance well?
Kevin
December 7, 2009
@ notimetoread:
VP of common sense? Not quite. What would this person’s background be anyway? With the way most people think of common sense, it would probably be some retired lineman or something like that, with a big voice and a hillbilly accent.
Actually, football GMs know that they don’t know (or are close to it). That’s why first round draft picks are not as desirable in football as they are in other sports; the Texans tried to unload that supposed no-brainer pick (Reggie Bush overall) because they knew they’d rather go with Williams (I lived in Houston at the time, I remember). Nobody wanted that pick, though! WRs are also a big problem position. Big attitudes coming in to the league, and a big waste of money too, quite often. Smart GMs seem to grab up those 2nd-6th round picks and get linemen, lots of linemen. You get them cheaper, they’re seldom a big franchise face, hence less feeling that you have to retain them when they crap out on you. Let some other fool figure out who the good name positions are.
I don’t disagree with Berri’s basic idea. Clearly drafting for football is a quantitative mess. It’s really hard to make good predictions about future success if past success is unreliable. Even if you do make decent predictions, injury changes everything, faster in football than any other (American?) sport, it seems.
But my point was, you can guess that those excluded from analyses are probably below average, on average, and still make some persuasive case that the correlation is weak (or at least is so small that it doesn’t impress anyone). Or, perhaps not. N = ? I haven’t read the paper. So rather than everybody saying, “I don’t know, do you know, I know you don’t know!” someone (who cares enough) can just make an assumption that everyone finds reasonable and test the assumptions. It may go some distance towards nailing the coffin on this whole business…
todd2
December 7, 2009
The Steelers and Seahawks played a Super Bowl a few years ago with the highest o-line salaries in the league. Maybe there’s a connection.
Steve
December 7, 2009
I think you’re being disingenuous here. The idea that because an individual quarterback never got a chance to play we don’t know that that particular quarterback wasn’t good enough is possibly true.
Your decision to lop off such a large part of the sample *is* relevant. It’s not just oh well, they never got a chance to play, so we can’t infer anything.
You’re dealing with a major issue of selection bias. We’re assuming a few major things:
1) QBs taken in the first few rounds are going to be given the benefit of the doubt because of the team’s investment in them, and are going to be given a higher amount of playing time than their ability, or at least performance, deserves.
2) QBs taken in the later rounds are going to get very little benefit of the doubt. Therefore, those who do get a chance to play are far likelier to earn that playing time and end up being good.
That leads to one of two conclusions: either QB play is extremely fungible and there is a massive inefficiency in the NFL player acquisition market, or your analysis is woefully incomplete.
So, I agree, *given that the quarterback makes it on to the field a lot*, draft position is unimportant.
You’re trying to say your analysis proves that draft position and performance are unrelated.
Until you address why all quarterbacks (regardless of when drafted, or whether they are drafted at all) make it or don’t, you haven’t proven anything, or really added much to the discussion that some pretty basic logic (if you’ve made it, it doesn’t really matter where you were picked) would have accomplished.
Steve
December 7, 2009
Oh, and regarding the Brady and Warner point—there are surely lots of other quarterbacks who, given the right opportunity, would have had fine careers.
But you absolutely cannot conclude that there are enough to suggest that draft position has no correlation with quality until you’ve examined them. Given the financial and competitive motivations involved, it again would be a massive inefficiency in the marketplace were there to be enough late-round QBs now in coaching or plumbing or banking that should be playing football. You haven’t proven anything.
If you have an idea of what makes a good QB, and you can find hidden gems that fit the profile and were drafted in the later rounds, perhaps we can look into those next. Until then, your research continues to be technically sound and yet unable to pass the smell (or even a basic logic) test.
Robert Simmons
December 7, 2009
Notimetoread, I don’t disagree with his methodology as much as I do with his bitchiness about people misunderstanding his explanation of it. I understood what he meant, but it’s easy for me to see why someone wouldn’t. And, to be consistent/honest, I must admit that you’re misunderstanding of me was primarily my fault, but I don’t know where I said anything about a fundamental error, or that the commenter at pro-footballreference.com was correct. My contention is that the commenter was justified in his understanding of the methodology, incorrect though he is.
pennaguy
December 8, 2009
Sorry to go off-topic a bit, but does anyone know of a study of NFL trades which evaluates draft-pick value vs. veteran player value?
Vince
December 18, 2009
From 1970 to 2007, 34 quarterbacks were taken in the top 5 picks of the draft. These include Peyton Manning, John Elway, Troy Aikman, Terry Bradshaw, Bert Jones, Steve McNair, Donovan McNabb, Philip Rivers, Carson Palmer, Jim Everett, Jim McMahon, and Eli Manning (that’s 12 QBs ranging from good to Hall of Fame, over 1/3 of the picks). Are you saying that these 34 QBs were no better than any random set of 34 drafted QBs, they just happened to get more playing time? In other words, that about a third of all drafted QBs are that good, but many of them just never get the chance to play?
During the same time period, 33 quarterbacks were taken in the 5th round. Out of those 33, Mark Brunell and Steve Grogan have had the best careers. Are you saying that there are about 10 other QBs among those 5th rounders who would have been about as good as the QBs on that list of top 5 picks, if only they had been given the playing time? 6th rounders look a little better than 5th rounders, but there are 51 of them and only 4 look like they could belong on the list with the 12 top 5 picks: Tom Brady, Mark Rypien, Matt Hasselbeck, and Marc Bulger. Are you saying that there are about another 13 6th round picks who could’ve performed at that level, if only they’d had the opportunity to play? In other words, for every Brady, Brunell, or Bulger – a late round gem who becomes a good or great QB – there are about 4 late round gems who get drafted but languish away unnoticed and drop out of the league?
Steve Sailer
December 19, 2009
Prof. Berri’s approach to logic could be extended to height of quarterback. We can use Prof. Berri’s thinking to prove that there’s no correlation between height and quarterback success.
The teams with the best record in 2009, Indianapolis and New Orleans, have quarterbacks who are, respectively, 6’5″ and 6’0″: Peyton Manning and Drew Brees. Therefore, we can conclude that height has zero correlation with NFL per-play performance.
Granted, a guy who is 6’5″ is maybe 100 times more likely to become an NFL starting quarterback than a guy who 6’0″, but the 6’5″ guys who are starting quarterbacks do about the same on a per play basis in the NFL as the starting quarterbacks who are 6’0″, so there is no correlation between height and success in the NFL at quarterback!
The logic is airtight!
Steve Sailer
December 19, 2009
Clearly, the enormous over-representation of tall quarterbacks in the NFL is solely caused by bias.
What other possible explanation could their be?
Steve Sailer
December 19, 2009
Moreover, Dr. Berri’s logic proves there must a whole bunch of 5’7″ quarterbacks who would be huge successes in the NFL if only there weren’t bias against short quarterbacks. Look at how Doug Flutie made it to the Pro Bowl when he was finally given a chance.
Not to mention a whole bunch of really skinny quarterbacks who aren’t being given a chance, too. And paraplegic ones, for that matter!
TK
January 4, 2010
Oy… at the risk of causing Steve Sailer to yammer on and on… can we get a final 2009 QB Score post?
(And this request comes with a pledge: I’ll only post once on this thread!)
dberri
January 4, 2010
TK,
I will try and get those posted by the end of the week.
Harvey Jerkwater
January 27, 2010
I have to agree with Steve above — this argument is making a huge switch in data sets and pretending the switch didn’t happen. The question is about all QBs *drafted*; the analysis is about all QBs who’ve *played.* Those are *not* the same groups. By simply ignoring the players who were drafted but did not play, the data is hugely skewed.
The assumption in the article is that players are given playing time based almost purely upon draft position. Do training camp and preseason not factor in? Is it not possible, even likely, that the duds of the lower rounds simply never see the field? Are we not seeing the effects of pre-season filtering?
This argument cannot be settled by the numbers. It just can’t. We don’t know how good the filtering of bad QBs in training camp is. What we can say from the data with confidence that is if a QB is considered by his coaching staff to be good enough to start, his draft position probably won’t matter. What we cannot say is that draft position for QBs is irrelevant in predicting the QBs’ skills. We simply can’t make such a sweeping statement from the data.