John Hollinger Responds

Posted on November 24, 2006 by


A couple of weeks ago I was asked to explain how Wins Produced differs from John Hollinger’s Player Efficiency Rating (PER). The post I wrote in response to this question was eventually noted by Malcolm Gladwell, who asked Hollinger to respond.

On Wednesday night at TrueHoop, an e-mail from Hollinger addressing my critique was posted. Unfortunately, Hollinger did not quite address the problem I noted with his method of player evaluation.

From my earlier post

Hollinger argues that each two point field goal made is worth about 1.65 points. A three point field goal made is worth 2.65 points. A missed field goal, though, costs a team 0.72 points.

Given these values, with a bit of math we can show that a player will break even on his two point field goal attempts if he hits on 30.4% of these shots. On three pointers the break-even point is 21.4%. If a player exceeds these thresholds, and virtually every NBA played does so with respect to two-point shots, the more he shoots the higher his value in PERs. So a player can be an inefficient scorer and simply inflate his value by taking a large number of shots.

Hollinger’s reply to this argument

Berri leads off with a huge misunderstanding of PER — that the credits and debits it gives for making and missing shots equate to a “break-even” shooting mark of 30.4% on 2-point shots. He made this assumption because he forgot that PER is calibrated against the rest of the league at the end of the formula.

Actually, if we took a player was completely average in every other respect for the 2006-07 season — rebounds, free throws, assists, turnovers, etc. — and gave him a league-average rate of shots, and all of them were 2-pointers, and he shot 30.4%, he’d end up with a PER of 7.18. As long-time PER fans know, that would make him considerably worse than nearly every player in the league.

To end up with a league-aveage PER of 15,00, the actual break-even mark in this case is 48.5%, which is exactly what the league average is on 2-point shots this season.

My response to his response

Let me summarize. I am arguing that if a player shoots inefficiently he can increase his PERs ranking by simply taking more shots. Hollinger responds by noting that a player can increase his PERs value by shooting more efficiently. You will note that this is not actually a response to what I am saying.

Let me try and illustrate with an example. Say a player shoots 40% from two point range. If he takes 10 shots his PERs ranking will rise by 6.6 (4*1.65) and decline by 4.32 (6*0.72). So the net gain is 2.28. If he doubles his shot attempts to 20 he will see his PERs ranking rise by 13.2 (8*1.65) and decline by 8.64 (12*0.72). Now his net gain is 4.56. The player is still shooting 40%, which is below average. But his PERs ranking increases the more shots he takes.

PERs, and other metrics like NBA Efficiency and Points Created, tell us that an inefficient shooter becomes more valuable the more shots he wastes. But that is not consistent with winning basketball games. Inefficient shooting does not simply become a good thing because you do more of it.

The rebound question, again

Hollinger goes on to argue that Wins Produced over-values rebounds. This argument was addressed earlier HERE and HERE. For now I will simply re-state the point that you cannot argue that we value any statistic incorrectly because the player rankings do not conform to your prior beliefs. You can only argue that this is “incorrect” because you have a “better” model with an alternative value.

And what makes a model “better”? A model would be preferred if it explains wins better and provides better forecasts of future player and team performance. Explanatory and predictive power are not the only criteria we have in evaluating models. But they are on the list. And this list does not include whether or not the model conforms to your prior beliefs.

Commenting on someone else’s work

Hollinger states at the onset of this e-mail “I’ve been trying real hard not to say anything about Wages of Wins, because it would only come across as self-serving to knock the work of another person in the field.”

This statement is disingenuous. In The Wages of Wins we quote extensively from Hollinger’s critique of the plus-minus approach. So Hollinger has addressed issues he has had with other approaches.

And in fact, if you are going to utilize a metric of performance you have to at least address why the available alternatives were not chosen. This is something we discuss in the book when we review NBA Efficiency and plus-minus. We did not offer a discussion of PERs, though, hence the need for the recent post.

– DJ