Answering a Critic

Posted on November 26, 2006 by


NOTE TO VISITORS FROM GLADWELL.COM: If you are looking for my response to Hollinger, please look at the post entitled John Hollinger Responds.  

A few weeks ago I was asked at The Wages of Wins Journal to comment on how the Wins Produced metric differed from John Hollinger’s Player Efficiency Rating. As I noted, the question had been originally asked by Malcolm Gladwell as he prepared to write his review in The New Yorker. After I posted my answer a few days ago, Gladwell also decided to comment on my PERs critique at his blog.

Interestingly, the comments at Gladwell’s blog have not been directed to the difference between Hollinger’s methods and the metric we offer in The Wages of Wins. Rather, a group of people inspired by Dan Rosenbaum have taken the opportunity to attack our work.

As JC Bradbury — of Sabernomics fame — noted at, these comments – posted at and in other forums on the Internet – have been “nasty.” Given the source and tone of these comments, we have had a debate at WOW about whether we should respond. Stacey says no we shouldn’t. Marty says we should. This leaves me as the tiebreaker, and since I have nothing else to write about on Sunday (and I do try and post each day), let me try and offer a response without being “nasty.”

The Rosenbaum Critique:

At Rosenbaum posted a summary of his critique:

Wins Produced is a metric that (a) professes to be regression-based, but is only marginally so, (b) misapplies its own logic when it derives (rather than estimates) its linear weights, (c) proposes explaining team wins as a barometer when ANY metric with a team adjustment (no matter how bizarre) would explain team wins just as well, and (d) only performs microscopically better than points per game at explaining how teams do when particular line-ups are on the floor.

Let me respond to each point.

To point (a) and (b) – yes, the Wins Produced model is based on regression analysis. In fact, the estimated weights are the result of several regressions. So this particular critique is simply factually incorrect. We did attempt to explain the intuition behind our results, which might give the impression you could have reached the same conclusions without the regressions. I do not think, though, that this is actually true.

To point (c) – Rosenbaum has fixated on the team adjustments as the secret to the Wins Produced model. I disagree that any model with a team adjustment could explain wins as well and offer the same ability to forecast future player and team performance. As we note in the book, the team adjustment we use do not dramatically impact the rankings of players derived from our model. So his story about the team adjustment is incorrect.

To point (d) – this statement refers to an approach Rosenbaum offered several months ago to test our model. What he did is regress the player rankings from his replication of Winston-Sagarin (described below) on player rankings derived from Wins Produced and other models. In other words, he evaluated how well each model predicted his model.

We are going to have to plead ignorance to this evaluation method. We tried to create a model that would take the statistics tracked for individual players and explain team wins, as well as future player and team performance. We did not know that the “Holy Grail” (a term often used in the APBRmetrics community) of player performance models was the one that explained Rosenbaum’s player rankings the best. If that is the standard, though, then I guess we are going to have to live with a model that comes up short.

The Winston-Sagarin Model

What exactly is the Winston-Sagarin model? It is difficult to describe accurately in a simple post, but I think a good description would be that the Winston-Sagarin model is a sophisticated version of plus-minus. Such an approach ignores all of the traditional player statistics and focuses solely on how a team does with and without a specific player. Having exchanged e-mails with Wayne Winston, I think such an approach is interesting, although perhaps not the best approach for our research.

Although Winston did send a few e-mails to me when our book first hit the market, the exchange was always congenial and we basically left the discussion acknowledging the pluses and minuses (sorry about that) of our two approaches. I would note that Winston and Rosenbaum are not working together. Winston, working with Jeff Sagarin, originated this approach which was purchased by the Dallas Mavericks (who are owned by one of Winston’s former students). Rosenbaum replicated this model later on and sold his approach to the Cleveland Cavaliers. At no point were all the details of this approach published in a refereed journal, a point that further hampers are ability to use this in our research.

The Wages of Wins Story, Again

It is important that I once again re-state the basic argument we make in The Wages of Wins. This quote I think captures this essential story:

“People in the game often claim to know instinctively how to measure intangibles, but salaries suggest otherwise. Teams pay for little more than the glory statistics (points, rebounds and, to a lesser extent, assists).

Although steals, blocks, shooting percentage and an ability to avoid turnovers are crucial to a team’s performance, players proficient in these aspects are rarely rewarded with bigger paychecks.”

Interestingly, the author of this statement was not one of the authors of The Wages of Wins. The author is Dan Rosenbaum, who published this statement in The New York Times in April of 2005. 

The Importance of Player Evaluation in the NBA

Let me close by making a point I have made a few times in the last few days. Basketball is a game invented by James Naismith. As Sports noted about ten days ago, the game Naismith developed was based on a 19th century game called “Duck on a Rock.” A century later people are spending the Thanksgiving weekend debating how to best measure a player’s performance in “Duck on a Rock.” When you put it this way it is clear this is not a very important issue.

We never claimed that the Wins Produced metric by itself was “important.” I do believe this measure is a good representation of what a player does on a basketball court. Consequently, with this representation in hand, we can investigate various topics in economics. And I think some of our work in economics can be thought of as “important” (at least to us).

Beyond what we do in economics, we can also tell some stories that might be of interest to fans of basketball. For the most part, that is what I try and offer in this forum. And as long as I think there is an audience for these stories, I will keep writing.

– DJ