As Dave has blogged earlier – we employ a team statistical adjustment (i.e. team defense) measure as part of the Wins Produced metric. This measure incorporates the opponent’s field goals made, opponent’s turnovers, and team rebounds into our analysis of individual players.
Now this may come as a shock to you but we have critics. I know, but it’s true. One critique is that we only present Wins Produced with the team adjustment but not without the team adjustment. Specifically we have been asked “… how well does Wins Produced with the team adjustment predict team wins? For the sake of intellectual honesty please make this information public.”
Dave provided an answer to this question, but I thought I would chime in with a few more thoughts. Let me begin with a quick linear regression review. One of the main assumptions of the linear regression model “… is that the dependent variables can be calculated as a linear function of a specific set of independent variables, plus a disturbance term.” (Kennedy p. 43, A Guide to Econometrics 3rd ed.) If this assumption is violated you have a specification error. One type of specification error is an omission of a relevant independent variable (i.e. the team adjustment in question). If we did not include the team adjustment measure then our econometric model would be mis-specified. Consequently our coefficient estimates would be biased and our inferences would be inaccurate!
Think about it this way – for years people believed that there was a “face on Mars”. Examining fuzzy pictures from the Viking Orbiter 1 space craft we see a fuzzy picture of a hill that is “face-like.” From that fuzzy picture some have concluded that this hill is an artificially shaped alien artifact. The problem is that the resolution of the 1976 Viking Orbiter was not strong enough to provide a clear picture of this hill. With the
2001 Mars Global Surveyor’s higher resolution Mars Orbiter Camera the “face-like” hill is well…. just a hill.
In the same manner if we estimated Wins Produced with the team defense adjustment our estimates would be like the original fuzzy pictures and could lead to inaccurate conclusions. Thus “[f]or the sake of intellectual honesty …” I would respond that by looking at how well Wins Produced without the team adjustment is like drawing conclusions from fuzzy pictures since the model is mis-specified. Hence, by not estimating a mis-specified model we are being as intellectually honest as possible.
I suspect that this explanation will not satisfy all of those who “know we are wrong or are hiding something up our sleeves”. Such is life. We have learned over the last year that you can’t make everyone happy. We do feel fairly confident that our fully specified model – based on the theoretically sound proposition that wins are determined by offensive and defensive efficiency — provides an accurate picture of player performance in the NBA. And although that picture might just be a hill in economics, it is our happy little hill (okay, my metaphor is getting away from me).
- Stacey
3 responses so far ↓
Guy // April 27, 2007 at 10:52 am
Stacey: The concern that has been raised about the team adjustment is not that it is incorrect or does not belong in the model. The concern is that once you include the team adjustment, a model with very different coefficients for key offensive variables will correlate with wins just as well as your current model does. For example, Dan Rosenbaum has shown that a model with these changes has an identical correlation with team wins:
field goals missed = -0.7
free throws missed = -0.35
offensive rebounds = 0.7
defensive rebounds = 0.3
(For that matter, points per game with a team adjustment correlates just as well.) As a result, the correlation with team wins cannot itself demonstrate the accuracy of your model at the individual player level. Moreover, Rosenbaum finds that this revised version of your model actually does a better job than Wins Produced at predicting FUTURE wins, the most important standard to meet. While this doesn’t prove that WP is “wrong,” it does leave you in the position of having no evidence that it is any more correct than a number of other possible linear weight models.
I think it would be great if you and David engaged this debate and responded directly to these concerns. Or, I can understand why you might want to continue ignoring them. But what is the point of writing a post like this, that raises the issue again but fails to acknowledge or address the actual concerns being raised about your method? Perhaps it reflects an honest misunderstanding on your part of the argument made by your critics, but it certainly appears to be a willful refusal to address legitimate issues.
Rosenbaum’s analysis, and other relevant comments, are here: http://sonicscentral.com/apbrmetrics/viewtopic.php?t=1232&postdays=0&postorder=asc&start=0
Okapi // April 28, 2007 at 12:06 pm
I enjoyed this post, but I’m not sure statistical analyses of basketball are tractable with respect to Martian metaphors.
Stacey // May 1, 2007 at 3:16 pm
Okapi,
I was going to use the conspiracy theory regarding a second shooter with the fuzzy picture in the “grassy knoll” with the death of President Kennedy, but quickly gave up on that idea, as I thought that might be offensive. Hence I turned to the red planet.