Last week Henry Abbott – of TrueHoop – noted the conflicting stories told about Kevin Durant.
Durant was voted Rookie of the Year by the media and to the All-Rookie First Team by the coaches. John Hollinger’s Player Efficiency Rating – which Abbott argued was “the most respected single number to express a player’s total contributions”— says that Durant was above average his rookie season and ranked him as the 20th best player last season.
And then there is the adjusted plus-minus story. As Wayne Winston notes at his blog – see HERE and HERE — from the perspective of this measure Durant is clearly below average.
So which story is correct?
Three Perspectives on Durant
The answer to this question begins with the observation that PER is probably not “the most respected” measure of performance in the NBA. One only has to read Wayne Winston’s Mathletics to see the problems with Hollinger’s performance measure (or one can read what was said in this forum three years ago). The reason Durant has such a lofty PER is because the takes a large number of shots. A player only has to exceed very low levels of shooting efficiency for his PER to rise with more field goal attempts. Hence, top scorers tend to look good according to PER, even if – as Durant was his rookie season – the player can’t shoot very well.
Okay, PER has problems. So does adjusted plus-minus paint a better picture? Adjusted plus-minus ignores the box score statistics and simply looks at how a team does when a player is on the court. The “adjusted” part involves attempts to control for the impact of the player’s teammates and the team’s opponent. Theoretically, everything a player does – and that includes on-the-ball defense – should be incorporated in his adjusted plus-minus number. Hence, this measure should tell us more than what we see in the box score.
As every proponent of this measure notes, though, adjusted plus-minus is a very “noisy” measure. What does this mean? Typically there is a large standard error associated with the number assigned to any player. Hence, it is difficult to be sure whether many players have a positive or negative impact on outcomes. Furthermore, there’s very little consistency – at least relative to the box score numbers – in a player’s adjusted plus-minus numbers across time. As we very briefly discuss in our next book (due out next March), a player’s adjusted plus-minus value in this season is not highly correlated with what he did last year.
JC Bradbury notes that a performance metric should be both correlated with outcomes (i.e. typically wins in sports) and consistent over time. The former is important because we wish to know if a player is having a positive or negative impact on outcomes. The latter is important because inconsistent measure suggests you are not accurately capturing a player’s impact. PER fails the first test (it’s not highly correlated with team wins). And adjusted plus-minus has a problem on the second count.
And that brings us to Wins Produced. This measure is both connected to team wins and relatively consistent across time. Durant’s Wins Produced for his rookie season confirms the adjusted plus-minus story. In 2007-08, Durant produced 0.7 Wins Produced and posted a 0.012 WP48 [Wins Produced per 48 minutes]. Because Durant could score, though, his PER was above average and he was named Rookie of the Year. Wins Produced, though, indicated that Durant was not the best rookie. In fact, he wasn’t even one of the better players on a bad Thunder team.
In 2008-09, though, the Wins Produced story changed. As is often the case, young players get better. And Durant went from being below average to being the most productive player on the Thunder (yes, Wins Produced is relatively consistent across time, but young players – as noted – can get better).
Table One: Oklahoma City Thunder in 2008-09
Once again, the box score does not capture an individual player’s on-the-ball defense. If a player is a better defender than his teammates, his Wins Produced will understate his value. And if he is a worse defender, then Wins Produced will overstate his value.
With respect to Durant, adjusted plus-minus insists that he is a poor player. So it’s possible that Durant is just a really bad defensive player. Of course, it’s possible that the noise in the model is also producing this result (although Winston says that in this case, that doesn’t seem likely).
So is it defense or noise? Should we believe Wins Produced or adjusted plus-minus? Well, Winston and I definitely agree we shouldn’t believe PER. But on the “true” value of Durant, I think Winston and I disagree (somewhat). At least, I am not convinced Durant’s supposed problems on defense trump his box score numbers.
Reviewing the Thunder
Let me close this post by noting that there is more to the Thunder than Durant. Here is the team’s potential depth chart (Wins Produced and WP48 numbers from 2008-09, unless noted otherwise):
Potential First String
PG: Russell Westbrook [3.7 Wins Produced, 0.066 WP48]
SG: James Harden [rookie]
SF: Kevin Durant [10.5 Wins Produced, 0.175 WP48]
PF: Jeff Green [1.8 Wins Produced, 0.031 WP48]
C: Nenad Krstic [-1.0 Wins Produced, -0.042 WP48]
Potential Second String
PG: Shaun Livingston [0.5 Wins Produced, 0.107 WP48; 0.111 WP48 in 2006-07]
SG: Kyle Weaver [2.4 Wins Produced, 0.097 WP48]
SF: Thabo Sefolosha [4.3 Wins Produced, 0.144 WP48]
PF: Nick Collison [5.8 Wins Produced, 0.151 WP48]
C: Etan Thomas [17.3 Wins Produced, 0.125 WP48 for career]
Again, the Wins Produced numbers say Durant is the team’s most productive player. And he is the only above average performer on the first string. Off the bench, though, Oklahoma City has four players who have been above average and one player that is quite close (average WP48 is 0.100). This suggests the Thunder’s rotation is backwards and wins are going to be left on the bench in 2009-10.
One should note, though, that however this team did its rotation, in the Western Conference the Thunder will probably miss the playoffs. But if the bench players saw the court more often, it’s possible the Thunder could come closer to the post-season. At least, that’s the Wins Produced story.
– DJ
The WoW Journal Comments Policy
Our research on the NBA was summarized HERE.
The Technical Notes at wagesofwins.com provides substantially more information on the published research behind Wins Produced and Win Score
Wins Produced, Win Score, and PAWSmin are also discussed in the following posts:
Simple Models of Player Performance
What Wins Produced Says and What It Does Not Say
Introducing PAWSmin — and a Defense of Box Score Statistics
Finally, A Guide to Evaluating Models contains useful hints on how to interpret and evaluate statistical models.
Wages of Wins Journal 
Man of Steele
October 15, 2009
Dr. Berri,
Thanks for dealing with the ways in which individual defense is misvalued in WP (as in many other metrics). I think most of the reasonable arguments put forth in this blog against your method have focused on this probem, so it’s nice to have a comment from you on the topic rather than simply letting us pool our ignorance. It seems that something like adjusted +/- could be helpful for evaluating defense, if only we could excise all the noise from it so that we actually knew what we were measuring.
…
I wonder many players would be dramatically affected by the incorporation of some sort of defense-based evaluation in WP? I would guess probably no more than about 10% of the players in the league. If you think about it, defense minus blocks, steals, and defensive rebounds is basically just positioning and on-ball defense. There are probably relatively few players who are good enough (or bad enough) at on-ball defense to warrant a significant change in their WP. Stated another way, how should on-ball defesne be weighted against shooting efficiency, rebounds, assists steals, blocks, fouls, and turnovers? I would suppose that some players, like perhaps a Bruce Bowen or a Shane Battier, would come out looking a bit better, but our picture of them might not be completely different.
Perhaps one warrant for such a prediction would the simple observation that in general, only players who defend opposing teams’ best players have a large impact. So if Bruce Bowen or somebody shuts down Chris Paul, that’s huge. If somebody shuts down Melvin Ely, that’s not going to help their team beat the Hornets very much. So very few players would get significant defensive credit as a result of the Pareto principle – because a very limited number of players perform significantly on defense (of course, I guess that owuld mean that my estimation of 10% should have been 20%).
brgulker
October 15, 2009
Dr. Berri,
Didn’t Durant’s position also change during his sophomore campaign in the NBA?
I can’t seem to recall if he was moved from SG to SF or vice-versa — so obviously, I can’t determine if him playing at SF this year is a good or bad thing for the Thunder.
Matt Walters
October 15, 2009
One way to determine whether on-the-ball defense is actually worth incorporating into a future model would be to evaluate each player’s offensive performances against the same defender relative to his average performance. If a defender is able to consistently stymie his opponent’s production (or is consistently unable to do so) independent of defensive box score measures already included in the model, then WoW critics’ claims that the model is not accurately capturing a player’s contributions may have merit.
I suspect it would be difficult to gather sufficient data to draw any sound conclusions this way, but it’s better than just repeatedly asserting “Half of basketball is defense!!!” as Winston does without trying to examine the impact of on-ball defense alone.
Blake
October 15, 2009
Dr. Berri,
Do you feel that opponent’s Win Score is a good gauge of defensive value? QueenCityHoops has compiled the data to show opponent WS/min.
Shane Battier for instance limits his opponents at SG to a WS/min of .080, where expected production would be .151. That would seem to indicate that Battier, whose on-ball defense is notable rather than steals or blocks, is a very effective defender not just by conventional wisdom but by the numbers.
Here is the site, if anyone’s interested: http://queencityhoops.com/playerPage.php?team=HOUSTON&player=Shane+Battier
Tball
October 15, 2009
Matt,
Are quality stats kept on who is scored upon? If a SG loses his man on a switch so that the PF is guarding him, then the PF gets isolated, beaten off the dribble, but misses a layup because the C forced him to alter the shot, who is getting credit for that missed shot? If the stats aren’t being kept, then all we’re doing is wishing into the wind.
It might be interesting to see wins produced for a team’s opponent (e.g., what is the WP48 of PGs when playing the Suns over the course of the season? Are they as good as Steve Nash?). The players we are most interested in evaluating are the players that avearge 30 minutes per game for their team and generally play one position. This data might be useful in bringing team defense and individual defense into better focus.
dberri
October 15, 2009
Matt and Blake,
What you suggest is done at Buck’s Diary. From what I understand, opponent’s measures — like other plus-minus data — are also inconsistent. Still, it is an interesting approach.
Another approach is to allocate the opponent’s stats by how well a player plays defense on his team. I tried that once in a post on the Spurs. As I recall– as Man of Steele suggested — it didn’t change much.
One issue with defense is that in the NBA, defense is a team activity. Players do not play by themselves out there. So separating the player from his teammates is going to be a challenge.
Matt Walters
October 15, 2009
It suddenly occurs to me that a better and easier way to do accomplish this would be to calculate each player’s positional opponents’ WP48 in the aggregate. For example, one could determine the average WP48 of PFs matched up against Kevin Garnett and compare that to their WP48 when not playing against him. While this is not a perfect measure, if it produced consistent, significant results (say, KG’s opposing PFs saw their WP48s decline by 20% when playing against him) it would be possible to assign individual defensive adjustments to WP beyond the current “team average” system.
This seems like so simple a solution I’m sure you have already thought of it and rejected it for some reason that escapes me at the moment, Dr. Berri. Is this the case?
todd2
October 15, 2009
Pete Newell said it years ago, there are ten players and only one ball. There are a lot of skills involved in the game that are needed off the ball. I love the prof’s work, but it can’t take into account these “intangibles.” WP rewards good decisions and limiting mistakes.
Matt Walters
October 15, 2009
Oops, seems like I can’t post fast enough. Please ignore my redundant ponderings.
MRPARKER
October 15, 2009
ive found per difference to have significance
Italian Stallion
October 15, 2009
I don’t have any data to back this up, but intuitively I think it’s possible that when a player has impressive box score stats and has simultaneously been shown to be negative for the team on an adj +/- basis, it could say more about the coaching staff and other players on the team than the player himself.
As noted, the one exception would be a player that is woeful on defense. And from what I gather (I haven’t seen a lot of stats), Durant is a fairly poor defensive player right now. If that’s true, It probably accounts for part of the difference between the two methodologies.
However, I think another possible problem is that the coaching staff, his teammates, and Durant himself still haven’t figured out how to incorporate his huge talents into an effective “team effort”.
When one player is much more talented and skilled on offense than his teammates, I think there is a natural tendency for that player to become such a strong focus of the offense, it actually hurts his efficiency and reduces the effectiveness of his teammates. (remember we are talking about an extreme here, not a couple of shots one way or the other)
Both Michael Jordan and Kobe Bryant have spoken about learning this with time.
In the years to come, perhaps the challenge will be to implement a system that gets as much efficient play out of Durant as possible WITHOUT hurting his teammates.
Josh Smith
October 15, 2009
It should be noted that with upwards of 400 players in the NBA we should expect roughly 20 of them to have true APM values outside of the 95% confidence interval. Given what we know about Durant’s box score stats, I think it is reasonable to believe that there is a good chance he is one of them.
todd2
October 16, 2009
There have been some great players that have been poor defenders. Lary Bird and Magic Johnson come to mind. And watch Tim Duncan, he’s listed as a forward so he doesn’t have to draw the tougher defensive assignments. Good coaches know how to hide guys like these. Kevin Durant is long and light—not quick enough to defend on the perimeter and not strong enough to to defend on the interior. He does cause some serious mismatch problems offensively. It’s up to his staff to find some way to compensate on the defensive end.
DSMok1
October 16, 2009
Several people have created adjusted +/- that separate defense and offense– recently Illardi and Barzilai ( http://82games.com/ilardi2.htm ). Would it be useful to supplement WP with the defensive side of the metric?
Of course, the statistics problems in doing so are innumerable…
Michael
October 16, 2009
Professor, out of interest have you had any discussions with basketball-reference about them including your metric on their site?
That would be a boon for your work if they would.
markcuban
October 16, 2009
Actually, the missing element is coaching. Why did the OKC coach put him in the situations he was in and with who else on the floor ? who was on the court playing against him at the time ?
When a player is handpicked for stardom, the coach will often try to maximize the minutes and situation of the player , rather than playing the best lineup to win the game. That hurts in both adj pm and win shares.
Or the coach just isnt very good, and cant match up the right lineups or call the best plays to put the player in a position to succeed.
or finally, the more minutes a player plays, the more numbers they get. The more marketable they get.
I dont know the goal of either coach last year..
What I do know is that without exception, once they get to a positive adj pm, perennial all stars in the NBA earn positive Adj PM year after year .
I can go through 9 years of data and the all stars stand out like a sore thumb because they are always positive and more often than not they have a high impact (they help their team when it counts.).
To your point. 1 or 2 years of Adj PM is too noisy. 5 or more is not.
John Giagnorio
October 16, 2009
If adjusted +/- takes 5 or more years to reach a reliable conclusion about a player, how useful is it for decision making? It seems like a very long time to wait for an evaluation.
X
October 16, 2009
Mavs could be sitting on cap room and 4-5 years of Adjusted +/- data on Durant in summer 2011 or 12.
Man of Steele
October 17, 2009
markcuban,
So if coaches put players handpixked for stardom in situations that benefit the player but don’t lead to wins (can’t imagine what those situations would be) … that makes the player good? Isn’t the point of being a good player to help the team win?
Todd
October 20, 2009
“Once again, the box score does not capture an individual player’s on-the-ball defense. If a player is a better defender than his teammates, his Wins Produced will understate his value. And if he is a worse defender, then Wins Produced will overstate his value. ”
While I appreciate the attempt to admit a weakness in the wins produced model, I do not think that this is it. In fact this statement is far more relevant to PER, which does nothing to account for defense.
With win score, the problem is not that individual defense is not measured, but that individual defense is attributed to someone other than the defender. If a player is a good defender and also an even better defensive rebounder, his defensive contribution may still be deflated. The fact that defensive accomplishments are attributed to rebounders, and all players are compared to a prototype at his position creates noise in the win score statistic.
I also think it is unfair to judge PER based on it’s ability to predict wins as Hollinger openly admits that it is only measuring offense. I think it is a fair criticism that PER rewards high volume shooting for the sake of high volume shooting.