My last post focused on media’s decision to give the 2009 Rookie of the Year award to Derrick Rose. The post began by noting that Rose was essentially an average point guard in 2008-09. It then proceeded to offer the following argument:
So why was he named Rookie of the Year?
The key issue is point score per game. Rose had the second highest scoring average among rookies. And since the leading scorer – O.J. Mayo – played on a losing team (and was also drafted after Rose), we should not be surprised that most of the media focused on the point guard from Chicago.
Explaining the Vote
After writing this sentence I thought it might be a good idea to investigate my claim. Essentially I am arguing that the media primarily focuses on scoring in voting for the Rookie of the Year. And beyond scoring, draft position and playing for a winner also matters.
My argument is bolstered by past studies – detailed in The Wages of Wins and elsewhere — of how talent is evaluated in the NBA. However, none of these past studies – at least, none that I am aware of – looked specifically at voting for the Rookie of the Year. So I thought I would spend a bit of time seeing if the conjecture I offered was supported by some empirical evidence.
The study begins with the voting process. The Rookie of the Year award is decided as follows: Each member of the sports media asked to vote for this award names three rookies. The top choice from this trio receives five voting points. Second place is worth three points. And the third choice receives a single point. After all the votes are in, the voting points of each rookie placed on a ballot are added together. The rookie with the most votes gets a trophy. And everyone else gets to wonder why they didn’t get a trophy.
With voting explained, we now need to look at the data. As is the case with all awards given out by the NBA, voting data for the Rookie of the Year can be found at Patricia’s Various Basketball Stuff, a great NBA website maintained by Patricia Bender. According to Patricia’s website, from 2003 to 2009, 62 rookies received at least some consideration for this award (prior to 2003, Patricia Bender only reports the players who received a first place vote). What we wish to identify is the factors that caused the voting pattern we are observing.
Let’s start with a very simple model. Specifically, let’s regress voting points on the three variables identified earlier: points scored per game, team wins, and draft position. To this list I am going to add two more factors: number of games played and the size of the market where a rookie plays his home games.
As Table One indicates, regressing voting points on this list of factors reveals that the conjectures noted in the Derrick Rose article are supported by the evidence.
Table One: Estimation Results for Rookie of the Year Voting Model
Points scored per game, team wins, and draft position are all statistically linked to voting points, with points scored having the largest impact. In addition, games played also matters. Market size, though, was not statistically significant (I am going to explain what “statistically significant” means at the end of the column).
Our simple model explains 75% of the variation in voting points. What if we considered, though, the other box score statistics? To answer this question, adjusted field goal percentage, free throw percentage, rebounds, steals, assists, blocked shots, and turnovers were added to the original model (all of the non-shooting data was per-game and measured relative to position played). Including all of these factors boosts our explanatory power to… 75%. Okay, explanatory power looks the same although if you go out a few decimal points you see a slight increase. In addition, of the new variables added we see a statistical link between voting points and shooting efficiency, steals, personal fouls (a negative link), and maybe assists (at the 10% level for assists). Rebounds, turnovers, and blocked shots were not found to be statistically significant. And of all these factors, points scored per game easily has the biggest impact on voting points.
Just Like Coaches
For those who read the Wages of Wins, this story should sound familiar. In our book we discuss voting for the All-Rookie teams, which is done by the coaches. We report that a simple model — with points scored as the sole measure of performance — explained 76% of the variation in the coaches’ voting. Adding all the other box score statistics to the model only boosted explanatory power to 77%. And just like we saw with the sports media, points scored was the dominant performance factor.
Such a result should probably not surprise. One suspects that how the sports media see the players is heavily influenced by how the coaches see the players. So when coaches tell us that Kevin Durant and Derrick Rose are great players, one should expect the media to adopt this perspective. And when non-scorers do not get as much attention from coaches, we should expect the sports media to offer similar evaluations.
On Statistical Significance
Okay, that ends my story for today. What follows is a brief discussion of statistical significance. If you are not interested in statistics, this final section will not be the best thing you will read today. But since I used the word “statistical significance” in the column, I thought I would offer a brief introduction to the concept.
Let me start with a huge qualification (and something I think I have said in the past). You really can’t teach econometrics in a blog (at least, I don’t think I can). Having taught econometrics in the past I know this is a subject that requires a fair amount of class time. And after putting the time in the classroom (and I mean, taking more than one class), you then have to then put in additional time to gain experience. This means doing research that is reviewed by other people who understand statistics and econometrics (in other words, other people who have published research that utilized econometrics). After you put in all this time you will probably reach a point where you realize there is still much for you to learn (yes, this stuff is not real easy).
All that being said, let me try and clarify what is meant by the words “statistical significance”. In running a regression we are estimating the statistical relationship between the dependent variable (in this case, voting points) and each independent variable (for example, points scored). That relationship is captured by a coefficient (which tells us the direction and magnitude of the relationship) and a standard error. Although people tend to focus on the coefficient, the standard error is extremely important. This is because it’s the standard error that tells us whether or not our coefficient is statistically different from zero.
What does that mean? Before we can talk about the direction and magnitude of a relationship, we have to first establish whether or not a relationship even exists. If a coefficient was actually zero, then there would clearly be no relationship between the independent variable and dependent variable. In estimating the model, though, you are not going to see a coefficient that is actually zero. The number we do see, though, has to be something that can be differentiated from zero. And to make that differentiation, we compare the size of the coefficient to the size of the standard error.
To further explain this concept, let me fall back on a standard rule of thumb (keep in mind, an actual review of an article goes beyond this simple rule). Introductory textbooks will note that the general rule of thumb is that a coefficient has to be twice the size of the standard error for us to conclude the coefficient is statistically different from zero. So looking back at our simple regression, the coefficient for points scored per game is 6.33. The corresponding standard error is 0.74. The ratio of 6.33 to 0.74 exceeds 2, so we can now conclude that points scored per game has a statistically significant impact on voting points (although the strength of our conclusion depends on a host of other econometric issues that a reviewer would consider).
What about market size? The estimated coefficient is -0.19. One might interpret this result as evidence that rookies in larger markets receive fewer voting points from the sports media. But that is not the correct interpretation. The standard error for market size is 0.27. So the coefficient, in absolute terms, is not twice the value of the corresponding standard error. From this we can conclude that the empirical evidence suggests no relationship between market size and voting points.
It’s important to highlight my wording. The empirical evidence “suggests” a story. One could come back and say:
- what if we measured market size differently?
- what if we used a different functional form to estimate the model?
- what if we used a different method of estimation?
- what if we had more data?
and on and on….. In other words, even after you run a regression, there are still questions people could ask (and at meetings and in the journal review process these questions tend to get asked).
Even though we have questions, at this point it would be inappropriate to talk about the coefficient we have estimated for market size as being anything else than statistically insignificant. In other words, in interpreting the results we have, our present conclusion is that the link between market size and voting points is statistically insignificant. We do not say (and this point should be emphasized) the “coefficient is insignificant” and then proceed to tell additional stories about the link between these two variables.
One of my co-authors puts it this way to her students.
“When I teach econometrics I tell my students that a sentence that begins by stating a coefficient is statistically insignificant ends with a period.” She tells her students that she never wants to see “The coefficient was insignificant, but…”
Unfortunately I don’t always see people on-line following this advice. I have seen people report regression results but fail to note standard errors. Or standard errors are reported but the statistical insignificance of the results is ignored. Hopefully this brief discussion will help people understand what they are reporting and furthermore, what they are reading.
Let me close by noting there are a number of issues to consider in reviewing Table One. For example, the variables were logged so the estimated coefficients are actually elasticities. In addition, the estimation method [i.e TOBIT] was utilized because rookies who did not receive any votes were considered. In an article, each of these issues would be noted and explained. Since this is a blog post, though, I think I will avoid writing a few more paragraphs and just end the post.
– DJ
The WoW Journal Comments Policy
Our research on the NBA was summarized HERE.
The Technical Notes at wagesofwins.com provides substantially more information on the published research behind Wins Produced and Win Score
Wins Produced, Win Score, and PAWSmin are also discussed in the following posts:
Simple Models of Player Performance
What Wins Produced Says and What It Does Not Say
Introducing PAWSmin — and a Defense of Box Score Statistics
Finally, A Guide to Evaluating Models contains useful hints on how to interpret and evaluate statistical models.
Wages of Wins Journal 
Westy
May 1, 2009
I think that I somewhat differ with one of your conclusions. You note, One suspects that how the sports media see the players is heavily influenced by how the coaches see the players.
I’m not sure we can conclude this as I think there’s a logical alternative explanation and I would guess it might actually be the other way around. Thus, I guess I’m not completely surprised that these selections by the coaches so closely follow scoring.
I’d again blame the media. As much as coaches DO understand the game better than the media, they don’t have the time to understand much more than their own team. Thus, voting for players from other teams likely turns into voting for players they’ve heard are playing well. Well, where do they hear those things? From the media. So I don’t doubt that these coaches are being influenced by the media, which is being driven by scoring. In essence, it turns into a popularity contest.
I would not be surprised that if these same coaches spent a week with all the rookies in practice, they’d come out with a much different ranking that does not match their current one. Or likewise, if they were picking players to actually build a team they would coach, they would pick different players.
I think what this shows is not necessarily that the coaches don’t judge basketball talent well, but that they don’t have time to do so in voting for these teams.
Unfortunately, that ultimately reinforces the fact for the media that the most ‘important’ players are the highest scorers.
Horsecow
May 1, 2009
Is there any way we could see how the estimated results compared to the actual results, at least for 2008-09? Was the model accurate in picking Derrick Rose, or did OJ Mayo squeak by due to his scoring?
Oren
May 1, 2009
Westy,
I would have thought that coaches would scout(or at the very least get a very indepth scouting report for) opposing teams in order to prepare to play them.
I think Battier was quoted as saying during one playoff series that he could suit up for the other team and would be perfectly fine in their system. He’d know all the plays and everything.
Maybe in the NCAAs, teams don’t always take the time to scout out their opponent, but I’d be shocked if that’s the case in the NBA.
Westy
May 1, 2009
Oren,
I totally agree that coaches would scout (or at the very least get a very indepth scouting report for) opposing teams . However, do they refer back to these when they’re casting their votes? How well do they research their picks, or do they just go on name recognition? At the end of the year, they have 29 scouting reports. They’ve only played 15 of those teams twice, and are thus much less familiar with those. I would be surprised if they’re able to readily maintain all that information at the end of the year. Let alone remember the specifics on only a few of the players across the league. Not surprisingly, unless they did do further research at the time of their voting, the names that would bubble to the top in their mind would certainly seem to be those most familiar.
Now I’m sure some coaches do a better job researching it and taking the vote seriously than others. I would personally love to see the actual votes. But I wonder, is somebody like Jerry Sloan, who is dealing with a knee injury, scouting for his first round opponent the Lakers, and managing the travel that will ensue with that, really taking the time to make sure he’s voting in a manner reflective of how he would actually rank/manage these players if they were on his team?
Will
May 1, 2009
Dave, I love this explanation of ‘statistically significant’. I thought I understood this concept well, but I had no clue about the relationship between a coefficient and standard error and how to read it. I for one would like to see more of this type of analysis… but I realize I’m probably in the (severe) minority there. I can’t tell you how much time I spend (daily) trying to think of ways to argue in favor of win score with my friends, and any sort of deeper understanding of the process can only help. Thanks.
Italian Stallion
May 1, 2009
I’m not saying it should be this way, but I think the sports media, coaches, fans, and other players etc… all think about the players in terms of their potential too. It’s not just about their scoring and other stats in their rookie year.
Is there really anyone that doubts that Rose and Durant are going to be great players in this league soon (well excluding people that don’t have any skill at watching games, never played basketball, and don’t think high usage scoring matters at all)?
I’m not sure how you could measure the perceptions of the people who make the selection about the overall talent and potential of the players, but if you could, I’d bet it would rate as a statistically significant aspect of who gets the award. It might even rate higher than scoring. It just so happens that the most talented players are usually aggressive about scoring right from the start because that’s the way they are geared mentally. But you can tell which ones have real talent that will blossom in other ways.
Matt Walters
May 1, 2009
Italian Stallion,
No one is arguing that Rose, Durant, or anyone else can never become great players, but your presumption that such an outcome is beyond “doubt” is silly. My suspicion is that you, like many sportswriters and coaches, already consider these players great (at least for youngsters) and look askance at WoW analysis because it contradicts these assumptions. I think you will be able to formulate more productive criticisms of the model when you stop interpreting WoW as an attack on your “skill at watching [and experience playing] games.”
As for potential and its impact on rookie of the year voting, I think the obvious argument (besides your own acknowledgment that measuring perceived potential is impossible) is that coaches and sportswriters tend to view scoring volume as a primary indicator of potential; i.e. the concept of “overall talent and potential of the players” is inevitably drawn from players’ scoring totals and thus any distinction is meaningless. But no one really needs to make that argument anyway, because the Rookie of the Year award should go to the best rookie, not the one with the most potential. Presumably there will be a host of other awards awaiting that player when he blossoms in the future- why slight the man who did in favor of the man who might?
Sam Cohen
May 1, 2009
IS- why wouldn’t you just consider draft position to be a good proxy for perceptions of overall talent and potential? It sure seems to me that teams think mostly along those lines in making draft picks, particularly higher draft picks. And if that’s the case, then Professor Berri’s simple model already includes this factor.
JAW
May 2, 2009
DBerri, in terms of evaluating how coaches thinks what happens when you try to explain minutes played with each of the variables. I have to imagine that this will look much more like PAWS
dberri
May 2, 2009
JAW,
We will talk about minutes in our next book. For now… it doesn’t look like PAWS.
biggles
May 2, 2009
Hi, I’m having difficulty substituting values into the regression: I’m getting what I think are nonsense results (fitted log-response around -18 for the top players). Would you be willing to provide an example or two: perhaps Rose and Mayo, as Horsecow suggests?
dberri
May 2, 2009
biggles,
Did you take natural logs (LN)? Also, points are adjusted for position.
The model does predict Rose would get more voting points than Mayo.
Michael
May 3, 2009
“Rookie of the Year award should go to the best rookie, not the one with the most potential. ”
Define best.
Anon
May 3, 2009
statistically the most productive?
Michael
May 3, 2009
See my dictionary defines best (adj.) as:
1. Surpassing all others in excellence.
2. Most satisfactory.
3. Greatest.
4. Most highly skilled.
Seems like the subjectivity of the word matches the subjectivity of the award doesn’t it. One could argue that by excelling (by rookie standards) in a larger market like Chicago and making the post season in his first year that Rose is the ‘greatest’ rookie. Anyone watching him play could also be forgiven for calling him the ‘most skilled’ rookie as well.
I think the basic problem with this argument(that Rose was not the ‘best’ rookie) is that it defines best as ‘statistically the most productive by the wins produced measure.’ Obviously this is false because the award is not given using wins produced as a parameter, it is given based on other more subjective criteria and observations. Basically if you are saying that Rookie of the year should be awarded to the ‘best’ rookie, then arguing that Rose was not rookie of the year because wins produced shows that other rookies were more productive, is false because you are changing the definition of the premise (i.e moving from ‘best’ to ‘most productive.’)
If you think Love (or whoever) should have been rookie of the year then fine. But to suggest that a subjective award was given for flawed reasoning (ie potential) because the result didn’t adhere to your chosen metric seems off to me, especially if you then use a definition (‘best’) and which begs the question.
(I.E ‘Rookie of the year should go to the best Rookie.’
‘Derrick Rose did not produce the most wins according to wins produced’
‘Derrick Rose is therefore not the best Rookie’)
Matt Walters
May 3, 2009
Ay dios mio. Anon is right.
simon
May 3, 2009
Michael,
The best in sports usually means most productive, or at least somewhat close to it. You and I both know “skilled” doesn’t mean anything if it’s not used productivity.
The biggest problem is, there isn’t any well known metric system that suggests Rose was anything special in productivity this season.
WP48: 0.104
PER: Rose ranked 10th among the rookies with 16.
Adjusted +/-: -6.7 one of worst on his team.
Rolando Rating: -1.8 again not very good
So outside his scoring total and the fact that he was fortunate enough to have solid teammates, it’s hard to find evidence of Rose “excelling,” even by rookie standard. He looked great, I agree, with his jet-like speed and aggressive style of play, but in terms of actual productivity…not so great by any measure.
One thing you suggested – or at least you’ve suggested that’s how the voters see it – is that playing in a larger market is somehow tougher. I highly doubt there’s any evidence that players’ (or just rookies’) productivity is lower when they play in a larger market. Do you subscribe to theory yourself?
On a somewhat related note, Ben Gordon’s still looking for that $60~70 mil over five years.
biggles
May 3, 2009
Thanks Prof. Berri; I forgot that Excel uses log10 by default. Mea culpa for using Excel.
The models looks to fit well, with a few outliers — it’s (rightly) surprised that anyone voted for Robin Lopez, while it expected Michael Beasley would do better (maybe the media have higher standards for the #2 pick, or maybe they actually did look at stats besides points for him). Just looking at the top rookies, it seems that position-adjusted points does the lion’s share of explaining the vote, which is plausible enough. (In contrast, number of games, for example, mostly serves to separate to narrow the contenders to those who played in nearly all games.)
Of course, none of this excludes the possibility that other simple models might fit equally well (claiming causality when controlled experiments are impossible almost always requires a leap of faith somewhere…)
Phil
May 3, 2009
The general perception is that Rose had a strong start. Prof Berri wrote a post highlighting his strong start according to WP. After hitting the notorious “rookie wall”, I also believe he finished strong, or at least that is a widespread perception.
Do people tend to give more emphasis to their initial and/or most recent impressions? Rose might be a beneficiary, in either case. Just a notion.
dberri, did you consider factoring NCAA tourney success at all? Rose’s team made it to the final. It is an award based on perception, and having positive exposure could surely help. Coaches watch the final four, at the very least.
simon, I have a feeling Ben Gordon is going to become quite familiar with the term “Mid Level Exception”, as are quite a few others that feel they are due a bigger contract. Take a bow, Sean Marion.
simon
May 3, 2009
The Raptors would welcome Shawn Marion back with open arms at that price. :)
Westy
May 8, 2009
I thought I’d add this interesting quote by John Hollinger in regards to (in this case the all-defensive team) postseason awards voting:
“I refuse to believe the league’s coaches submitted those ballots for the All-Defensive Team. Pardon me while I wonder out loud how many of them outsourced the job to the team’s media directors or some other person down the food chain. Based on the voting, I’ll go with “most or all.” You can’t tell me with a straight face that one of the league’s coaches voted Travis Outlaw second-team all-defense or that one of them put David Lee on the first team (!) or that three of them felt Samuel Dalembert was worthy of a second-team vote at center. The other big clue is how star-heavy the final results were. Four of the top five players in the MVP voting made the first team. The other (Dwyane Wade) was a second-team choice. They didn’t pick an All-Defensive Team; they picked an all-endorsement team.“