Back in 2005 the Atlanta Hawks held the second pick in the NBA draft. With point guards Chris Paul and Deron Williams on the board, the Hawks opted for forward Marvin Williams.
Last year, while both Paul and D.Williams were posting above average campaigns, M. Williams was decidedly below average. And this is true regardless of which summary statistic you like. Per 48 minutes he posted the following:
Win Score: 5.3
NBA Efficiency: 17.7
Game Score: 11.8
To put these numbers in perspective, an average power forward would have posted the following marks per 48 minutes.
Win Score: 10.3
NBA Efficiency: 23.4
Game Score: 15.0
On Game Score
As we can see… oops, wait a minute. I probably need to explain Game Score.
John Hollinger created the Player Efficiency Rating (PER). Game Score is Hollinger’s simplified version of his complex PER model. It’s calculated as follows:
Game Score = PTS + 0.4*FGM + 0.7*ORB + 0.3*DRB + STL + 0.7*AST +0.7*BLK – 0.7*FGA – 0.4*Missed FTA – 0.4*PF – TO
PER, which is a per-minute measure, involves more than this simple equation, but the results are almost identical. For the 458 players who appeared in 2006-07, Game Score per-minute and PER had a 0.99 correlation. In essence, Game Score and PER are telling the same story.
When you look at the Game Score formulation you will note that a player receives credit for each point scored, as well as an additional credit for each field goal made. A player is also charged for each field goal he takes.
When you analyze these values it becomes apparent that a player who makes 30% of his two point field goal attempts will still come out ahead. Let’s quickly review the numbers. If a player made 30 out of 100 shots from within the arc he would score 60 Game Score points, receive an additional credit of 12 for making 30 shots, and be charged 70 for his attempts. In sum, his Game Score would rise by 2 (60+12-70). And if he took 200 shots his Game Score would rise by 4 (120+24-140). In other words, the more he shoots – despite the fact that by any reasonable assessment this player is not very good at shooting – the better he looks.
And the story is the same from beyond the arc. From three point range the player only has to convert on at least 21% of his shots to come out ahead. And again, once he passes this threshold, the more he shoots, the better he looks.
Marvin Makes a Hypothetical Deal
For those who have read the WoW Journal for awhile, you know I made this argument in November of 2006. So this has been said before. What I wish to do in this column is apply the lesson we learn from Game Score to the Marvin Williams we saw in 2006-07.
Williams took 706 shots from the field last season, of which 45 came from beyond the arc. From this distance he converted on 11 shots, for a conversion rate of 24.4%. Such a mark is below average, but still exceeds the threshold noted above. From within the arc Williams made 44.6% of his attempts. Again, this is below average, but this mark also exceeds the Game Score thresholds already detailed.
If we consider all that Williams did from the field – or his adjusted field goal percentage – we see a mark of 44.1%. This means that per field goal attempt, Williams averaged 0.88 points, and again that is well below average.
Turning to rebounds, we also see problems. The average power forward would grab per 48 minutes 3.7 offensive rebounds and 7.7 defensive rebounds. Williams, though, only captured 1.9 offensive boards and 5.6 defensive rebounds per 48 minutes. So on the boards, he was below average as well.
Now let’s imagine the following scenario. The coaches for Atlanta come to Williams and note his below average performance with respect to Game Score. And since Williams spends so much of his playing time at power forward, the Atlanta coaches ask Williams if he could boost his productivity by grabbing more rebounds, especially on the defensive end.
But Williams is smart. He has studied Game Score and understands how this is calculated. Consequently, he comes back to the coaches with an offer. Why not re-design the offense so that his field goal attempts double? As Table One illustrates, even if his below average shooting efficiency is unchanged, doubling his field goal attempts would cause his Game Score to rise to 17.4. And this level would easily pass the average mark for a power forward.
Table One: The Marvin Williams Experiment
Williams goes on to deflate the coaches’ focus on defensive rebounds. Since each defensive rebound increases Game Score by only 0.3, focusing on this factor does not generate much of a return. In fact, if Williams tripled his defensive rebounds – so that he was now averaging 13.2 rebounds per game – his Game Score would only rise to 15.2. Yes, that would be above average. But not nearly as good as Williams just doubling his shot attempts.
We should note that if Williams doubled his shot attempts, his per game scoring mark – even with shooting efficiency and free throws unchanged – would rise to 22.8. In other words, he would be among the games scoring leaders. And since scoring is the driving force behind play pay, the offer Williams is hypothetically making shouldn’t just make his coaches happy, it should also make Williams a much richer person.
Again, this story is hypothetical. We do not know that Williams is being evaluated in terms of Game Score or PERs. But this scenario does highlight the basic issue with the Game Score approach. This model tries to incorporate the idea of “usage” in the evaluation of players. Because it is believed – although in my view not systematically proven (see what Martin Schmidt said in June of 2006) – that more shots leads to less efficiency, Game Score is constructed to give players extra credit for taking shots. Unfortunately, this model ends up teaching us that inefficient scorers can increase their value by simply taking more shots. And the increase in value dwarfs a substantial increase in rebounds.
When we turn to NBA Efficiency (a metric that is also highly correlated with Game Score and PERs), we don’t see exactly the same story. Yes, an inefficient scorer will increase his NBA Efficiency mark with more shots. But because defensive rebounds are worth the same as points, tripling defensive rebounds results in a higher NBA Efficiency level than just doubling shots.
Of course, that shouldn’t make us happy with NBA Efficiency. Think about baseball for a moment. If a hitter had a 0.200 batting average, we would say he is below average. We would not think that such a player would be helping his team by doubling his at-bats. Giving more at-bats to an inefficient hitter should hurt a baseball team, not help.
When we look at Win Score, that’s the story we see. Williams posted a per-48 minute Win Score of 5.3 last season, which was below average. If he doubled his shot attempts, his per-48 minute mark would fall to 3.5. In contrast, if he grabbed more rebounds he would be well above average.
Reconciling Game Score and Win Score
Earlier I noted that Game Score and PERs were highly correlated. What of Game Score and Win Score? Here are the two metrics side-by-side.
Win Score = PTS + REB + STL + ½*BLK + ½*AST – FGA – ½*FTA – TO – ½*PF
Game Score = PTS + 0.4*FGM + 0.7*ORB + 0.3*DRB + STL + 0.7*AST +0.7*BLK – 0.7*FGA – 0.4*Missed FTA – 0.4*PF – TO
If we look at per-minute performance in Win Score and Game Score from 2006-07, we find a 0.82 correlation.
What if we change how Game Score regards shooting efficiency? If we drop the extra reward for field goals made (the 0.4*FGM term) and change the value of a field goal attempt from -0.7 to -1.0, the correlation between per-minute Game Score and Win Score rises to 0.93.
What if we also change the return to defensive rebounding? If we also change the value on defensive rebounds from 0.3 to 0.7 – or the same for offensive rebounds in Game Score – the correlation between the two metrics now rises to 0.98.
And if we say that each rebound is worth a point, then the correlation rises to 0.996.
In sum, the real difference between Game Score and Win Score is how it treats shooting efficiency and defensive rebounds. One doesn’t have to agree that each rebound is worth a point (although my regressions say they are). But if we agree that defensive rebounds are worth at least as much as offensive rebounds, and inefficient shooters should not be rewarded for just increasing their shot attempts, then we have essentially resolved all the substantial differences between Game Score and Win Score.
The Importance of All This
And if all that happens, what will that do for us? My answer returns to baseball. Earlier I talked about a player with a 0.200 batting average. Batting average has been around since the 19th century, and it has been criticized as inadequate since at least the early 20th century (see Alan Schwarz’s The Numbers Game). And yet, during every single televised baseball game, it’s batting average that the announcers note. Remember, this is a measure that says a home run is equal to a bunt single. Still it’s cited in every single game.
Now when I see this, I always throw my shoe through the television set. In fact, we keep a large supply of televisions on hand for every game I watch. Whenever batting average is mentioned, I throw my shoe through the screen. And then we spend a few minutes hooking up a new TV. Yes, this takes up time. But I don’t want to hold my anger within. It needs to be expressed.
Okay, no one does this (I hope). Although baseball keeps citing measures that might be “wrong”, life goes on. And I feel the same way about measures in basketball. John Hollinger can spend the rest of his life (hopefully a long one) calculating PERs, writing about PERs, etc… And this should not make any difference to anyone. This is just basketball. We are not talking about global warming or the latest macroeconomic model.
We should remember, global warming and macroeconomic models matter. If we get those wrong, then very bad things happen to very real people. If Marvin Williams gets to collect more money because he takes more inefficient shots, well that’s good for Marvin Williams and his family.
In essence, I think these stories are “interesting.” But I don’t consider this particular issue to be “important.” And that distinction sports fans should always keep in mind.
– DJ
Our research on the NBA was summarized HERE.
The Technical Notes at wagesofwins.com provides substantially more information on the published research behind Wins Produced and Win Score
Wins Produced, Win Score, and PAWSmin are also discussed in the following posts:
Simple Models of Player Performance
What Wins Produced Says and What It Does Not Say
Introducing PAWSmin — and a Defense of Box Score Statistics
Finally, A Guide to Evaluating Models contains useful hints on how to interpret and evaluate statistical models.
Wages of Wins Journal 
Ben Guest
December 16, 2007
DJ,
You and others have pointed this out many times. I think the more important question is why people still refuse to acknowledge this huge flaw in PER (and in similar player measurements). As you’ve said before, its ironic that PER purports to measure efficiency yet doesn’t take into shooting efficiency into account…
dberri
December 16, 2007
Ben,
Yes, I have said this before. But I only recently noticed the trade-off between rebounds and shot attempts. And that seemed kind of interesting.
As for people still using this metric…
well, it does match perceptions. Players who score will always score high in PERs. Plus, as I noted, people do not change their minds very quickly. Batting average is still with us, and that has obvious problems. The NFL’s QB Rating also has problems, but that is also not going away.
Polar Bear
December 17, 2007
This is a very good column. Compelling, easy to follow explanation of why PER is flawed.
Jason
December 17, 2007
I think there are a couple of reasons why PER remains a cited method for player eval.
The first is that it corresponds reasonably well with player perception at the high end of the game. Players who are widely regarded as very good do well by the method. There are relatively few surprises among this group and relatively few “stars” who don’t perform well. The list may not coincide with everyone’s eval, but no one is surprised by the list in general. It meshes with belief where people examine it, and *for the most part* these beliefs identify the very good players (though there are surprises that it misses).
I doubt very much that people think as carefully about what it means to be an average player, what it means to be slighly above average, somewhat below average. In terms of the difference between a moderately poor (say 33 win) and moderately good (say 47 win) team, these differences for several players are likely important, but I suspect that most people don’t put much thought into why a team wins and loses beyond those at the top of the game, the best performers on the team. And of course, there’s some validity to that since most production comes from a few people, so that the measure mis measures folk below the top doesn’t translate into popular opinion as easily. So there’s very little reason for most people to think about it as critically when looking at how Magliore and Gadzuric measure up against each other. The measure is either accepted based on its reasonably accurate assessment of LeBron and Garnett and passed down the line or rejected with a resounding “stats don’t mean anything-you can’t measure basketball with stats [BTW, did you see AI went OFF for 47 points!].”
So in general, I don’t think there is much critical evaluation of what the measure is and what it does other than that it corresponds well with All-Star balloting.
And from this, there’s one critical factor: the measure is easily accessible on ESPN’s website, meaning that it takes little hunting to find out a player’s PER. Look it up, there it is. That’s marketing for you. Economists must know that marketing works.
Animal
December 17, 2007
I am sure you have seen this, but I think this is what Jason is talking about
http://ballhype.com/story/the_foibles_of_formulas_a_quiz/
merl
December 17, 2007
Just looked at the above ballhype quiz.
It seems like the ‘star’ players are going to attract alot more attention, so would it be reasonable to assume that their turnovers are going to be higher by virtue of the extra attention that they attract?
If that’s the case, would it be worthwhile to make the weighting on turnovers in some way contingent on minutes played? I mean, if someone is going to rack up lots of turnovers and also get alot of burn from their coach, that has to say something doesn’t it?
I understand that this is diluting your model, but David Lee beating Tim Duncan seems wrong, and Timmy’s higher turnover rate seem like a big reason why.
Jason
December 17, 2007
There might be something to that. Per minute, Lee turned the ball over less often than Duncan. But as a percentage of touches? It’s tough to say. If assists + shots + rebounds are a good approximation of a big man’s touches (or at least gives a good idea relative to other big men) then their turnover:’touch’ ratios were remarkably similar. Is that coincidence though or is this something that actually shows that they’re more alike in their ability to keep ahold of the ball and Duncan simply had the ball more often, hence more chance to lose the ball?
However, I don’t know if there’s as much support for the notion that stars, in general have their turnovers inflated by attracting attention. If stars are regarded as stars largely based on their scoring abilities, then the relationship between PPmin and turnovers per min should be rather good. In fact, it isn’t good at all. There’s virtually no relationship between scoring and turnovers. It’s even worse if you evaluate it according to points per game, assuming that the “stars” are identified as stars by their scoring average (represented in PPG) and draw more attention as such. That relationship has an Rsq of 0.0101. While players who handle the ball do turn the ball over more to some degree, the relationship between turnovers and other things seems weak.
dberri
December 17, 2007
Jason,
In the immortal words of Johnny Carson.. “I did not know that.”
Okay, I am dating myself a bit. But what you found is interesting.
merl
December 18, 2007
Yes, the idea of ‘touches’ is what I’m thinking of. Turnovers should really be something that’s expressed in terms of total touches (I don’t know how much different PTS + AST + REB is from total touches, or even if touches is a stat that’s freely available. Commentators sometimes mention it, so I guess it’s available to someone).
I guess I’m changing the thrust of my thinking from ‘attention from the other team’ (which is too hard to track) to ‘number of opportunities to turn the ball over’, or touches / possessions with touch.
If someone plays 30 minutes per game, but touches the ball on every second posession, wouldn’t you expect them to have a higher per minute turnover rate than someone who plays 10 minutes per game and only touches the ball once in every six posessions?
Does that make sense?
Jason
December 18, 2007
There appears only the very smallest of relationships between turnover rate and ‘touch’ rate. Generally, you need to have the ball to turn the ball over, but the variation in turnover rates is not explained by variation in ‘touch’ rates as far as I can see. It may be that Duncan and Lee are relatively equal in skill at not turning the ball over, and Duncan’s higher *total* is merely a factor of more time with the ball, but in general, the variation in player’s ability to hold onto the ball exceeds the variation in their touches.
Hatertots
December 20, 2007
DJ,
I enjoyed reading your explanation of PERs shortcomings, but I feel compelled to point out that Marvin Williams primarily plays small forward for the Atlanta Hawks and not power forward as this article states.
dberri
December 20, 2007
Hatertots,
82games.com says he primarily plays PF. But the story would be the same if he played small forward. He was below average at SF last year as well.