A couple of weeks ago I was asked to explain how Wins Produced differs from John Hollinger’s Player Efficiency Rating (PER). The post I wrote in response to this question was eventually noted by Malcolm Gladwell, who asked Hollinger to respond.
On Wednesday night at TrueHoop, an e-mail from Hollinger addressing my critique was posted. Unfortunately, Hollinger did not quite address the problem I noted with his method of player evaluation.
From my earlier post
Hollinger argues that each two point field goal made is worth about 1.65 points. A three point field goal made is worth 2.65 points. A missed field goal, though, costs a team 0.72 points.
Given these values, with a bit of math we can show that a player will break even on his two point field goal attempts if he hits on 30.4% of these shots. On three pointers the break-even point is 21.4%. If a player exceeds these thresholds, and virtually every NBA played does so with respect to two-point shots, the more he shoots the higher his value in PERs. So a player can be an inefficient scorer and simply inflate his value by taking a large number of shots.
Hollinger’s reply to this argument
Berri leads off with a huge misunderstanding of PER — that the credits and debits it gives for making and missing shots equate to a “break-even” shooting mark of 30.4% on 2-point shots. He made this assumption because he forgot that PER is calibrated against the rest of the league at the end of the formula.
Actually, if we took a player was completely average in every other respect for the 2006-07 season — rebounds, free throws, assists, turnovers, etc. — and gave him a league-average rate of shots, and all of them were 2-pointers, and he shot 30.4%, he’d end up with a PER of 7.18. As long-time PER fans know, that would make him considerably worse than nearly every player in the league.
To end up with a league-aveage PER of 15,00, the actual break-even mark in this case is 48.5%, which is exactly what the league average is on 2-point shots this season.
My response to his response
Let me summarize. I am arguing that if a player shoots inefficiently he can increase his PERs ranking by simply taking more shots. Hollinger responds by noting that a player can increase his PERs value by shooting more efficiently. You will note that this is not actually a response to what I am saying.
Let me try and illustrate with an example. Say a player shoots 40% from two point range. If he takes 10 shots his PERs ranking will rise by 6.6 (4*1.65) and decline by 4.32 (6*0.72). So the net gain is 2.28. If he doubles his shot attempts to 20 he will see his PERs ranking rise by 13.2 (8*1.65) and decline by 8.64 (12*0.72). Now his net gain is 4.56. The player is still shooting 40%, which is below average. But his PERs ranking increases the more shots he takes.
PERs, and other metrics like NBA Efficiency and Points Created, tell us that an inefficient shooter becomes more valuable the more shots he wastes. But that is not consistent with winning basketball games. Inefficient shooting does not simply become a good thing because you do more of it.
The rebound question, again
Hollinger goes on to argue that Wins Produced over-values rebounds. This argument was addressed earlier HERE and HERE. For now I will simply re-state the point that you cannot argue that we value any statistic incorrectly because the player rankings do not conform to your prior beliefs. You can only argue that this is “incorrect” because you have a “better” model with an alternative value.
And what makes a model “better”? A model would be preferred if it explains wins better and provides better forecasts of future player and team performance. Explanatory and predictive power are not the only criteria we have in evaluating models. But they are on the list. And this list does not include whether or not the model conforms to your prior beliefs.
Commenting on someone else’s work
Hollinger states at the onset of this e-mail “I’ve been trying real hard not to say anything about Wages of Wins, because it would only come across as self-serving to knock the work of another person in the field.”
This statement is disingenuous. In The Wages of Wins we quote extensively from Hollinger’s critique of the plus-minus approach. So Hollinger has addressed issues he has had with other approaches.
And in fact, if you are going to utilize a metric of performance you have to at least address why the available alternatives were not chosen. This is something we discuss in the book when we review NBA Efficiency and plus-minus. We did not offer a discussion of PERs, though, hence the need for the recent post.
– DJ
Wages of Wins Journal 
Joel W
November 24, 2006
David,
It seems like you are neglecting a couple issues that Hollinger addressed, and also neglecting the idea that rebounds might not actually be as valuable as your regression says, but that they might be correlated with something else importnt: defensive skill.
Hollinger says “Secondly, regarding Gladwell’s comments on Berri’s ranking of players like Ben Wallace and Dennis Rodman (as well as several other less famous players who also got piles of rebounds), the issue isn’t the discovery of some new Holy Grail of basketball analysis, but rather the result of an assumption in Berri’s formula that distorts their value.
Namely, that his formula equated a rebound as being just as valuable as a steal, and a missed shot just as damaging as a turnover.
While this made his sums work in the aggregate, for individuals it threw things way out of kilter, as it overrewarded rebounders and overpenalized guys who miss shots.
And it seems clearly mistaken, both because missed shots can be rebounded while turnovers can’t, and because a defensive rebound is merely the completing piece of a sequence that began by forcing a missed shot.”
While I think your point on missing shots is correct, Hollinger does seem to have a point on the differing value of a turnover and a rebound. And I don’t think you can just say “but our regression says so, so we can’t change it.” If the theory rejects the model you have used for your regression, then shouldn’t you look to see if that is correct?
It seems to me that your model correctly captures the value of Rodman and Wallace but for the wrong reasons. Isn’t it possible that blocks and offensive rebounds, as well as defensive rebounds, correlate strongly with other skills, namely defense and effort. Dennis Rodman always just seemed to be working harder and faster than every other player (hello greenies) on the court to get rebounds, missed balls, etc. If you want to say your model is more predictive because of this, I could accept it, but it does seem like there is a theoretical flaw in the approach.
***
Further, your model does seem flawed in that it rewards players who take too few shots to maximizie their team’s efficiency while maximizing their own efficiency. See: Garnett, Kevin. It would clearly be better for the TWolves if he went Teen-wolf on everybody and took more shots. His efficiency would go down, but his value to his team would go up since even the marginal KG shot is better than the average shot of other players on the Twolves. The more shots he took, the better they would be until the marginal value of his shot was equal to the value of him passing to a teammate and giving them the shot.
disappointmentzone
November 24, 2006
On what grounds are you making the claim that the TWolves would be clearly better if KG decided to shoot a bunch? In the TWolves’ four wins this season KG took 9, 16, 15, and 12 shots, or 13 spg. In the six losses KG took 15, 15, 18, 18, 23, and 17 shots, or 17.7 spg.
Second, there isn’t much evidence to suggest that if KG took more shots his efficiency would go down. In fact, the evidence suggests that it would probably stay the same, perhaps even go up slightly. Berri wrote a nice post about shot attempts and efficiency a few months ago. It’s worth reading.
You are also going about logical thinking in the wrong way (and so is Hollinger to some extent). You fit theories to data, not the other way around. If missed shots and turnovers regress identically, you don’t nudge the data to fit the fact that missed shots don’t look like turnovers. It’s not: “here’s a theory, now what data can we find to support it?” It’s: “here’s some data, now what theory can we draw from it?”
The model for the regression was a regression to wins, if I’m not mistaken. What would you prefer the data be regressed to? If the aim is to explain wins by way of statistics, the best way to do that is to connect the statistics to wins.
I’m not sure what to make of Hollinger saying: “While this made his sums work in the aggregate, for individuals it threw things way out of kilter, as it overrewarded rebounders and overpenalized guys who miss shots.”
In order to established that a formula threw something out of kilter for individuals, you need to have a better baseline for evaluating individuals. Again, on what grounds does Hollinger make the claim that Berri is over rewarding rebounders? What should rebounders be rewarded? Perhaps Hollinger has something in mind, but PER sure isn’t it.
Jason
November 24, 2006
It’s very strange when people start talking about data as “assumptions.” It’s my understanding that it’s not the assumption that a rebound is as valuable as a turnover or steal, but rather that the average value of a rebound can be derived mathematically and that this value in isolation is the same as the value of a steal or turnover.
Deriving the value and making sense of the derivation are different things.
I do not understand how a formula that works in agregate can set the value of a rebound too high or overstate the cost of a missed shot. Either the values were derived in error–in which case the agregate should not work–or they were not. In the latter case the value is the value, regardless of how this is later interpreted with respect to the game.
It seems that Hollinger in some way is confusing the value of a rebound with who gets *credit* for that value. They do not have to be the same thing. I’ve voiced this before. I do suspect that there are cases where the credit of a rebound should not entirely go to the lone person who gets credited with it. While I don’t presently know of a better way to award the credit, it seems to me that WP does not have a mechanism for rewarding guards who force missed shots as all credit goes to the guy who eventually gets the rebound. Ben Wallace could not have grabbed as many defensive rebounds if there were not as many missed shots. Additionally, guards who do not play defense are not penalized for this in WP in the same way that a forward who does not rebound is penalized. If someone can blow by the point guard for basket after basket, there are no defensive rebounds to be had. This will work out correctly in the *value* of a rebound, but since the position correction seems to mean that failing to rebound as a guard isn’t as costly as it is for a center. (I say *seems* because I haven’t checked numbers on this and I could be in error as a result.) The center denied opportunities for defensive boards because his guards allow the opposition open looks from 17 feet that they don’t miss very often does get penalized for failing to grab the rebounds not available to him.
But this is about the credit due for the actions and not the value of such actions. David is quite correct that without the actual defensive rebound, there’s no positive effect of whatever defense was played. That end is necessary and barring a better mechanism for awarding credit, it seems that giving credit to the guy who grabs the board seems reasonable as a starting point. And it’s possible that these synergistic effects aren’t large, as if they were, the consistency in WP from year to year among players when teammates change should be low. That it’s apparently not low indicates that such effects are smaller rather than the rule.
George
November 24, 2006
WP is not nearly as regression derived as some people like to suggest. Logic was used to assign a rebound equal to a possession, a possession equal to one point, the value of blocks, personal fouls, etc. The team adjustment causes all of this to come out in the wash, but that could be done with any system, just add a team adjustment and it will more accurately predict team records.
I have a serious question that you may have answered, maybe not, how does WP compare to Dean’s individual win-loss records? As far as I recall those were conceptually similiar.
dberri
November 24, 2006
George,
I seem to remember running a regression. So where did you hear that this is not regression derived?
And your argument about tacking on a team adjustment on to any system is also incorrect.
Evan
November 25, 2006
Anyone who follows this debate knows how you’ve thoroughly destroyed his arguments.
Evan
November 25, 2006
Let me clarify:
Anyone who follows this debate knows how you’ve thoroughly destroyed Hollinger’s arguments.
Erich
November 25, 2006
I am all for competition and statistical validation. That being said, I do not like the way this is playing out. From the combative conversation to the lack of cooperative learning, basketball’s statistical analysis fans are losing on all fronts.
It seems childish to have to go through a third party (Gladwell) and the language used so far has appeared to me as combative rather than constructive.
Given that we are in the early stages of understanding basketball by the numbers, any methodology should still be evolving, and therefore we should not fiercely defend it nor bash another evolving model.
I’d like to see a cooperative competition to advance all of basketball understanding. If each system (Beech, Berri, Hollinger) were to make open predictions, basketball analysis as a community can analyze the results and suggest continuous refinement to each model, furthering our total understanding as individuals and as a group.
I believe each model brings something to the table now. I like how Hollinger was able to produce in-depth preseason projections. Even if he was forced by ESPN’s department requirement, his projections ranked all NBA teams and were stated confidently.
Out of Berri’s model, I have enjoyed his easy-to-employ formula in analyzing NBA draft prospects and NCAA basketball. His formula generated my love for Paul Millsap which has so far been validated with his outstanding play.
Overall, basketball analysis is still in its infancy
We are all seeking the same thing. Lets take a deep breath, shake hands, and put this competition on the court. Develop rudimentary prediction systems and let the numbers play themselves out. Engage in constructive criticism with open minds and open communication. Basketball analysis can only benefit from such an arrangement.
Mark T
November 28, 2006
I don’t have a professional ax to grind here. I just read this stuff. But, just evaluating arguments as arguments, which my training as a lawyer ought to be useful in doing, it seems clear to me that Berri’s response to Hollinger on the issue of allegedly overvalung rebounds see “the rebound question again” is completely correct. The Hollinger criticism is not a valid argument; it’s just a more elaborate way of saying “you’re methodology is wrong because it delivers results that I don’t agree with.” This is the kind of argument one hears in punditry on a variety of subjects, most more important than this one, to our collective detriment.
I think Jason’s analysis above is much more thoughful and points the way to the next generation of this kind of analysis.
Ian
December 2, 2006
I must be missing something. It seems to me that according to this system, the entire value of the change in posession that results from a defensive rebounder is being assigned TO the rebounder, despite the fact that the rebounder itself couldn’t get that rebound unless an opposing player missed the shot, which has to at least be partially attributed to the defense played on that player. Thus, defense+defensive rebound=value of one change of possession. A steal alone=value of one change of possession, so how can a steal=defensive rebound in value?
Jason
December 3, 2006
That’s more or less true. The defensive rebound gets assigned to one person regardless of what caused the missed shot, who boxed out for the rebound. Many things go into it, but one person gets the credit. Just as a point of reflection, a steal is credited to one person when the other 4 players who have their men bottled up such that the guy with the ball doesn’t have any options and allows himself to be picked. Perhaps these aren’t equal, but all stats in some way reflect more than the single event.
But that’s a critique that tries to rationalize the value rather than obtain the value. And the critique is positioned at the level of the individual credit rather than the value as it goes to the team where a defensive rebound and a steal are both changes of possession without a score and could reasonably assumed to be equal. The ‘critique by rationalization’ doesn’t actually address whether or not the values credited to individuals in the model are or are not valid *as they relate to a forecasting model.* If they are not valid, then there should not be a lower level of consistency in the wins produced between players and players changing supporting casts should not bring their actual value since a large part of the ‘credit’ is misassigned. If the values are acceptable then it should work as a reasonable forecast even if rationally we know that there’s more to each of the stats. But that’s an emprirical matter of how the model fits to the data rather than one of rationalization. It’s important to keep the distinction in mind.
Rob Allen
December 5, 2006
I just did a search for the word “league” on this page and found that the only hits were in Hollinger’s response. I think that’s the thing you’re missing. PER is a comparison and doesn’t rise or fall in a vacuum.
Caleb
March 5, 2007
If a missed shot is NOT rebounded by a particular player, what happens to it? It seems like the value lost is only some fraction of the value of a posession (and thus the value of getting that rebound is only a fraction of a posession).
Doesn’t WP in fact consider that rebound to be as valuable as a full posession?
I could be wrong, as I am no statistician…
Shane Makayla
May 3, 2007
Great…
A spanish translation about this topic might be found in here
http://www.buscaparejaenlinea.es
Matthias Stacey
May 5, 2007
You’re the master man…
http://www.mexicoweb.com.mx/busca/buscador.cgi?query=%22%3E%20%3Ca%20href%3D%22http%3A%2F%2Fwww.ordercredit.info%22%3ESmall%20Business%20Credit%20Card%20Application%3C%2Fa%3E
Sullivan Naima
May 6, 2007
I love this article…
Will refer to my friends
http://www.blazingro.com
sportsreport.de
January 13, 2009
First of all: I love what you are doing.
at the end of the day this looks like a lot of fun.
(better than using the same economic datasets again and again)
where do you guys get the stats / data from? I mean Hollinger has Elias Sportsbureau but where do you get it from ?
Have you ever thought about weighting ?