On Wednesday, Henry Abbott of TrueHoop posted a link to an article by Jason Friedman of the Houston Press. Freidman’s very lengthy article was about the statistical focus of Daryl Morey (general manager of the Houston Rockets). In essence, Morey is described as part of a new generation of decision-makers in basketball, a generation that relies more heavily on statistical analysis.
In this article was an interesting quote from Morey about the value of box score statistics.
“I think often what you’ll find when you’re getting negative comments (on statistics), they’re basing it on what they’re used to being available, which is a regular box score. And there’s no question that anything in the box score is highly misleading. So if you’re basing your opinions on that – the box scores that they hand out at the games – you’re going to have an appropriate negative opinion of what you can understand using analytics. I would even have a negative opinion of [statistical analysis] if that’s all I’d ever seen.”
To illustrate the failings of the box score statistics, the article discusses the impact of Shane Battier. Here is Abbott’s summary of this part of the story.
What we’re getting from this is that Battier is doing “the little things.” Or “playing smart.” Or, essentially, doing the things that win basketball games — whatever they are — but don’t show up in the traditional box score.
The Shane Battier Story Via the Standard Box Score View
Okay, a bit of background on Shane Battier. With Battier the Grizzlies averaged 48 wins from 2003-04 to 2005-06. Without Battier last year, the Grizzlies were the worst team in the NBA. This suggests that Battier might have had some value.
But when we look at the box score data, it’s hard to find this value. At least, that’s what we are being told.
Let’s start with scoring, the box score statistic that is most frequently cited when discussing an NBA player. The average small forward will score 19.9 points per 48 minutes. The best Battier has ever done in his career is 17.4, and that was in his rookie season. For his career he only averages 15.2 points per 48 minutes. So Battier is a below average scorer. And for many, that makes him a below average player.
Of course there is more to the box score than just scoring. Let’s turn to a summary measure like NBA Efficiency. The average small forward will post a per 48 minutes NBA Efficiency mark of 20.3. For his career Battier’s per 48 minute mark is 18.8. So NBA Efficiency says Battier is below average also.
Okay, NBA Efficiency is too simple. Let’s look at John Hollinger’s Player Efficiency Rating, a more complicated measure of player performance. The average player has a PER mark of 15.0. For his career, though, Battier only posted a PER of 14.2. Again he is below average.
If this were all you looked at with respect to the box score, I guess you would have to conclude the box score data is pretty worthless. Clearly we need to go beyond the box score data to figure out a player’s value.
A Different Approach
Well, maybe not. Let’s take a different approach. What we could do is regress team wins on offensive and defensive efficiency. Such a regression tells us that 94% of wins are explained by a team’s efficiency marks. To put that in perspective, only about 90% of team wins in baseball are explained by runs scored and runs allowed. In other words, the link between the current stats and current wins is a bit stronger in basketball than it is in baseball.
In a moment I will return to the comparison between basketball and baseball. But for now, I want to note that from our analysis of the link between the efficiency metrics and wins we can derive the value – in wins – of each of the box score statistics. And those values are used to construct the two measures cited in The Wages of Wins – Wins Produced and Win Score.
What do we learn when we look at Battier’s Win Score? For the answer we turn to Table One.
Table One: Shane Battier’s Career
Per 48 minutes the average small forward posts a Win Score of 7.3. Except for Battier’s rookie season he has bested this average his entire career. Let me repeat this point. Win Score, a measure based entirely on box score statistics, tells us that Battier is above average.
If we delve into the numbers we can see why. First of all, although Battier doesn’t shoot much, he is an efficient scorer. And the aforementioned regression is quite clear on this point. Shooting efficiency matters in the NBA. Or to put it another way, inefficient shooting definitely hurts a team’s chances to win (the valuation of shooting efficiency by NBA Efficiency and PER is a point I have made before in more detail).
Battier’s value, though, goes beyond shooting efficiency. When we look at steals and turnovers we see another area where Battier helps. For a typical small forward, if we subtract turnovers from steals we get -1.1. In other words, typically a player will commit more turnovers than he will get steals. Battier, though, is not a typical player. Steals minus turnovers for Battier in his career is 0.2. This is a 1.3 swing in possessions for Battier in his career. It’s important to note that Battier is not just a below average producer of points (in terms of totals, not efficiency) and a below average rebounder. But his ability to hit shots efficiently, generate steals, and avoid turnovers — all stats found in the box score — tell us that he is an above average player.
As we can see in Table One, the story we tell about Battier from the box score depends on how we view the data. When we rely on scoring — or scoring dominated metrics like NBA Efficiency and PER — we see a below average player. But when we consider Battier in terms of efficiency, we see a player that is above average and a key player in the success the Grizzlies had from 2003-04 to 2005-06.
Back to Baseball
In closing this post I want to make two more comments about baseball data. As noted, the link between current stats and current wins is a bit stronger in basketball. It’s also the case that the box score statistics in basketball have a stronger predictive power than box score data from baseball. The year-to-year correlation in Win Score per minute in basketball is 0.82. In baseball the year-to-year correlation in a metric like OPS is only 0.57 (a similar story is told for linear weights).
Of course the box score data in basketball doesn’t capture all a player does on defense. But the same charge can be made against baseball data. On-base-percentage, slugging percentage, OPS, and linear weights don’t tell us about a baseball player’s defense. But these various baseball metrics still tell us a great deal about a baseball player’s value.
Summarizing the Story
So here’s the story I am telling today. Box score data in basketball is at least as good – and I think it’s better – than data in baseball. The problem is that box score data in basketball is not well understood. Too often the only stat people look at is scoring. And scoring, by itself, doesn’t explain much of wins.
Metrics like NBA Efficiency consider more statistics, but this measure is dominated by a player’s scoring. Inefficient scorers can increase their NBA Efficiency value by simply taking more shots. A similar story can be told about PER. When we look at the box score statistics via the measures, again we can be misled.
Faced with this problem we are told to ignore the box score statistics. But a simpler solution is to simply heed the lesson we have learned about wins and efficiency. It’s well understood that wins are determined by offensive and defensive efficiency. If we simply take this relationship and apply it to the analysis of individual players we can see that players like Battier are truly valuable. And we can see this in the very box score statistics reported in every newspaper.
– DJ
Our research on the NBA was summarized HERE.
The Technical Notes at wagesofwins.com provides more information on the published research behind Wins Produced and Win Score
Wins Produced, Win Score, and PAWSmin are also discussed in the following posts:
Simple Models of Player Performance
What Wins Produced Says and What It Does Not Say
Introducing PAWSmin — and a Defense of Box Score Statistics
Wages of Wins Journal 
APBRmetrics
November 2, 2007
First, I think Battier might have played some power forward last season, so his 7.8 Win Score might have been a pretty close to an average Win Score for such a player.
Second, this is a pretty bold argument, since it seems to say that not only do traditional GMs not understand what determines wins, but even a statistically inclined GM like Daryl Morey doesn’t understand.
Third, what evidence is there that Win Score or Wins Produced is any better at predicting the future than other metrics? The only work on this that I know of is a paper presented at Harvard by David Lewin and Dan T. Rosenbaum that showed that Wins Produced predicts future wins worse than Points per Game, Minutes per Game, and NBA Efficiency if a team adjustment is added to those metrics.
Now, as far as I know, the paper for that presentation is not publicly available, so we should be skeptical about this evidence until we see the full details. But at least there is an attempt there to present evidence about how well various metrics predict the future. Here the only evidence that Wins Produced does a good job is how well it explains current team wins. But I think you pointed out once that pretty much any metric with a team adjustment can explain current wins as well as Wins Produced. The real test would seem to be how well they predict the future.
mrparker
November 2, 2007
I think he did a pretty good job of predicting the outcome of the Allen Iverson trade. I became more convinced after that.
Paulo
November 2, 2007
APBRmetrics, with all the predictive models out there, I think that WP is the one that does the best job even without team adjustments. Unfortunately, predicting the future using WP is a very delicate beast, since minutes played and injury are very much in play (and up to a certain point, erratic).
I agree that the typical box score is valuable in determining the performance of players. In fact, this goes without saying, this is the only tangible way to do so, without scouting. WP has been very good in summarizing this, even without factoring in defense (see Bowen, Ben Wallace, Battier).
APBRmetrics
November 2, 2007
mrparker
Good point about Iverson, Wins Produced did a good job with that one. But that’s just one example (or two, if you count both teams). And I think Mr. Berri would even admit that anything can happen in small sample like that.
And, for that matter, I think that case is included in the study discussed above. What that study suggests is that if there were three blogs exactly like this one, except that they were dedicated to predictions based on these other metrics (one for Points per Game, one for Minutes per Game, and one for NBA Efficency – all with team adjustments), those other three blogs over the past few decades would have gotten things right more often than the Wages of Wins blog.
In other words, just like this blog, those blogs would have their diehards, excepts that those diehards would be right a little more often than the diehards on this blog – assuming the study is credible.
And I think the study somehow assumes that each of the metrics does an equally good job of predicting minutes played (and thus injuries) or how players get better or worse over their careers. So those can’t be the reasons why Wins Produced doesn’t seem to predict the future as well as a metric based upon Points per Game or Minutes per Game or NBA Efficiency.
mrparker
November 2, 2007
the key to that statement is “if team adjustments”
Color me a skeptic, but that qualifier alone elicits a “duh”.
We’ll try an exercise. When Kobe Bryant is traded we’ll see which method produces the closest approximation to win percentage going forward.
dberri
November 2, 2007
APBRmetrics,
Are you speaking for the forum, or is this one person talking?
Okay, probably not important.
Here is the problem with your argument. The team adjustment used to calculate Wins Produced is a specific variable. Specifically, the Team Adjustment used in calculating Wins Produced is calculated as follows:
Team Adjustment = [(2FGM(opp.)*- 0.032 + TO(opp.)*0.033 + TOTM*-0.032 + REBTM*0.033)/Minutes Played Team]*48
Now what is this? These statistics represent team defense. We don’t have data on these for individual players. But we do have this for the team. And in specifying our wins model, team defense has to be included.
I would add that team defense is not simply tacked onto the model in some ad hoc fashion. The variables are simply part of offensive and defensive efficiency. The wins model used in The Wages of Wins simply regresses wins on offensive and defensive efficiency. That model has been defined theoretically and explains 94% of wins.
Still, without any theoretical reason, people want to define the wins model differently. Specifically, the argument has been advanced that team defense, added to any variable, can explain wins just as well. Let’s test that hypothesis.
I regressed wins on NBA Efficiency and the Team Adjustment. I did the same thing for points scored and the Team Adjustment.
When you do this you find that NBA Efficiency and the Team Adjustment only explains 56% of wins. Points and the Team Adjustment explains 49% of wins. Obviously that is not the same explanatory power as offensive and defensive efficiency.
Rosenbaum and Lewin presented 12 slides at Harvard (still no paper) that said that NBA Efficiency and “a team adjustment” explains wins just as well as Wins Produced. Clearly they must have added a different variable to NBA Efficiency to do this, and simply called their different variable “a team adjustment.” Nowhere in their writings, though, do they tell us how this different variable is constructed.
In essence, what they did is extremely misleading. You can’t take the name from a defined variable and apply it to some other variable defined differently. And you can’t just throw a variable into a model unless you have a theoretical reason for doing so.
I would add, that these slides also show that adjusted plus-minus also has trouble explaining both itself and future wins. Rosenbaum calls this “noisy.” Another way of saying this is “your model can’t predict very well.” Yet this model that can’t predict very well is supposedly “the model.” At least, for 18 months Rosenbaum has been using a model that can’t predict to evaluate all other models. Again, this is a very suspect approach.
I would end this lengthy comment by noting that I was not criticising Morey. Morey reached the correct conclusion on Battier. I was simply showing that with box scored data you could reach the same conclusion.
Mountain
November 2, 2007
Of course the post from “APBRmetrics” is one person, speaking for themselves, presumably active in the named community. As another participant there I listened to the Rosenbaum presentation and thought it was interesting food for some further thought but I agree that more detail in a written paper will help and I am open to listening to commentary from Berri too on his method, the general metrics comparison topic and how to advance the field of public knowledge.
On Battier the discussion could go deeper into playoffs (how does he rate on WP and PAWS/min in playoffs). And perhaps whether he is a good player, a certain type of role player that needs to be on a team with high usage offensive players of acceptable performance that allow him to focus on what he is good at or if he could be a different as good or greater Battier than we’ve seen in a different role. That is pretty hypothetical though at this point. Battier is a role player being used pretty well for that purpose. But to me to say he is a success story he will have to win in the playoffs sometime and be judged as making a positive contribution to that victory.
Mountain
November 2, 2007
2003-4 playoffs Battier less minutes played, lower than regular season on FG%, FT%, reb rate about the same, way different and worse ST/TO ratio.
2004-5 playoffs Battier about same minutes played, lower than regular season on FG%, way lower on 3pt % and FT%, reb rate way up , worse ST/TO ratio.
2005-6 playoffs Battier minutes played down a bit, lower than regular season on FG%, lower on 3pt % and way lower on FT%, reb rate up , worse ST/TO ratio.
2006-7 playoffs Battier minutes played about the same, about the same on FG%, 3pt% and above regular season on FT%, reb rate way down, ST/TO ratio positive and better than regular season.
Maybe he was above average in this last of 4 palyoff experiences or did the rebound disappearance override?
I could probably compute WP and PAWS/min later but perhaps Dave or someone else set up with that can provide it.
Westy
November 2, 2007
Dave,
You note that “I would end this lengthy comment by noting that I was not criticising Morey. Morey reached the correct conclusion on Battier.”
In another quote from Morey he notes the following:
“Give me an example of a player who defies the numbers.
DM: I think the player that there’s a constant debate about in sort of the analytical basketball community is a player like Allen Iverson. He’s one that takes a lot of shots—they don’t always go in—but led a team to the Finals. So I think he’s someone definitely that I think is much better than an analytical look might show.
That’s crazy, because an area that hasn’t been studied enough –it’s been discussed a lot and no one’s come up with a definitive approach—is a trade-off of usage and efficiency. An analytical approach might tell you to have an entire team of Andres Biedrins because all he gets are lay-ups and dunks, or a player like Anderson Verejao, where of course an entire team of those would be a disaster because they have no one who can actually create a shot, no one who can break down a defense. So that’s just an example that trade-off is one that still—we’ve done a lot to look at that trade-off here—but I think in the community it’s still pretty nascent.”
I’m not sure what statistical tools Morey is using, but he seems to disagree with you. Obviously WP is not the only statistical engine that likes Battier. Based on the above, and considering the time and intellect Morey has put into developing the best statistical analysis possible, do you think he’s wrong?
Nephtuli
November 2, 2007
Of course the box score data in basketball doesn’t capture all a player does on defense. But the same charge can be made against baseball data. On-base-percentage, slugging percentage, OPS, and linear weights don’t tell us about a baseball player’s defense.
Yes, but in baseball they are constantly coming up with better and better defensive metrics. UZR, Baseball Prospectus’ FRAA/FRAR, HardballTimes’ Defensive Win Shares are just a few examples. It also seems that while these stats are lacking, they are more capable at determining a player’s defensive contribution because defense is baseball is generally a discrete situation involving one player.
So while box scores in basketball might be more accurate portrayals of a player’s value than in baseball, advanced statistics in basketball seem to be lacking at evaluating a major part of the game. Obviously it’s better than just watching the players, but stats in baseball overall are probably better able to paint a more complete picture.
Rasta
November 2, 2007
Great column. Battier is one of a number of players that defy PER, Efficiency, and other conventional metrics.
Two comments I’d like to add:
First, among the many “little things” that Battier does that doesn’t show up in the boxscore is taking a charge. according to 82games, Battier drew 50 charges last year, and 48 the year before. He ranked in the top 10 in this category each year. How many additional wins does this account for? I’m guessing about 1.5.
My second comment is a continuation from yesterday, which has to do with comparing Battier versus the average SF or versus his actual opponent.
Here’s a summary of Battier’s opponent (at SF) during 2006-07 per 48 minutes:
15.8 FGA
4.3 FTA
0.91 points per shot
6.7 rbs
3.3 ast
3.1 TO
17.7 points
All of these categories are lower than the “average SF”, which tells me that Battier played exceptional defense. In fact, when comparing Battier against his opponent, his approximate wins produced increases from 6.1 to 12.1.
Jason
November 2, 2007
The team results with him and without him would seem to indicate that WP is doing a reasonable job of showing Battier’s value. It’s possible that a team full of guys doing those ‘little things’ (which seems better defined as ‘things that aren’t going to gather All-star votes’ and/or ‘things that PER and NBA efficiency don’t value heavily’).
What it suggests to me as a subjective interpretation of the empirical data is that a guy who doesn’t make mistakes is worth a whole lot. By mistakes, I’d include missing baskets (which usually results from ‘creating’ tough low percentage shots) and turning the ball over. There may be a diminishing returns on this particular type of player (can you have a team full of ‘low mistake’ players and survive?) but we have little data on this.
mrparker
November 2, 2007
Jason,
I think the Portland team that won the title in the late 70s is this type of low mistake team.
There was an article about this team earlier this year.
Mountain
November 2, 2007
I calculate Battier’s PAWS/min in 06-07 playoffs as slightly below league average for regular season at -.004. Some slippage is typical though in playoffs compared to regular season so maybe he is still average or possibly even a bit above for that setting. But it appears only a slight distinction if any.
In 05-06 he rated at -.049 on PAWS/min certainly below average. In 04-05 he rated -.005.
In first playoff out his was -.130.
2 playoffs near average, one bad, one terrible. Not that strong a playoff performance record, at least on individual mostly offense basis.
Have positional playoffs averages been computed for PAWS/min?
Mountain
November 2, 2007
Battier is a valuable regular season player but his playoff record is not as strong.
Looking at bigger picture his team +/- on in playoffs is a big negative each of 4 years. His on/off differential is strongly negative 3 of 4 years.
His 1 on 1 defense was good 2 years, average 1, poor 1.
Has he done enough to get a lot of praise? To make the trade for him a big deal? To justify his salary? I think he is more average than good in the most critical timeframe for performance and that is being easy on him. Unless playoffs #5 changes that, I think his description should be amended to reflect the playoff record.
Jason
November 2, 2007
The sample size of playoffs is pretty small. If it’s demonstrated that his performance in that sample is statistically significantly different from his regular season stats, there’s a case. But I suspect much of the ‘clutch time’ stats including playoffs are but the statistical noise that shows up more pronounced in small samples. Rewarding or penalizing someone on such a basis will result in a rather skewed reward system that is unlikely to reflect future performance accurately.
dberri
November 2, 2007
This is a great discussion, but I have to serve as a substitute teacher today (another professor is at a meeting presenting research). So I can’t comment much.
I response to Mountain… I have playoff data but I haven’t looked at the position adjustment for that data set. All I have ever looked at is whether a player got better or worse, and when you compare a player to himself you don’t need a position adjustment.
I will add that Battier has never gotten out of the first round of the playoffs, so what we are seeing is a very small sample.
Mountain
November 2, 2007
Small sample size does deserve caution but I don’t agree with setting playoffs largely aside on that basis, just using it with caution and other data. Sum up Battier’s playoff years and he has 588 minutes of playoff experience. Yes that is fairly small but it is the most important measure of his performance to me. His PAWS/min for all those minutes is -.013. Regular season performance is good but look at the tape and see what he does and does not do in the playoffs that makes his rating significantly lower. Is it the competition level, the pressure? Is Battier good in regular season because he is better than others or to some large degree because he is more conscientious in his effort then than others? When better players with focus clash in the playoffs his team has come out losers in first round every year and his numbers appear a bit below average on PAWS/min and not that impressive on other measures either.
Not picking on Battier out of disrespect or dislike. But again the more you look, the more there is to consider. Morey’s case for Battier or Berri concurrence are 2 perspectives. This playoff angle adds a different view for any that think it should be incorporated.
TK
November 2, 2007
Not only is the sample small in terms of number of games and minutes. But even those numbers come from a single team each year (because he’s always been on one and done teams). So really, those numbers from three series are largely (not completely) about how he played against three guys. A dozen games worth of data would be small enough if it came against a dozen different teams. But against three teams? The determinative value seems even smaller…
Mountain
November 2, 2007
Alright Dave if today is particularly busy. I’ll check back later in case you post other comments on the topics.
Positional playoffs averages for PAWS/min probably would make a very interesting article at some point.
I’ve appreciated the opportunity to read your articles and perhaps have more interaction here in the future. The more I get in feedback on feedback the more effort I make in the future.
Mountain
November 2, 2007
TK makes a good point 19 games and 4 opponents isnt very broad a dataset but you could theoretically adjust the PAWS/min to the quality of team defense or even counterpart defense to make the playoff data more fair across player experiences. Not sure who covered Battier in those series but he faced quality team defenses twice and average to below average ones twice.
dberri
November 2, 2007
Okay, I have time for another comment.
Westy brought up a good point from the Morey article that I wanted to chime in on.
I think Morey is using adjusted plus-minus as his primary tool. This is the method developed by Wayne Winston and Jeff Sagarin (and I would add, although some proponents of adjusted plus-minus have an almost irrational hatred of WoW, Winston and I get along fine).
When I read Morey’s comment on Iverson what I thought of was the job Morey has to do. If you read Moneyball you learned that the general manager of a team is not a the “king.” To get anything done, he has to persuade others to his point of view. In sum, he has to be diplomatic.
One of the problems I think Paul DePodesta had with the Dodgers is an unwillingess or inability to diplomatically make his case. Eventually the troops rebelled and he was gone.
When you read the comments of any general manager or coach on statistics, you should remember that these people have to diplomatically make their arguments. They can’t say stuff like “staring isn’t going to work.”
Now if all you do is write a book, you have a bit more leeway.
One last note… Mountain is looking at all this and wondering “if you had time to write this much to Westy, why not look up playoff PAWSmin.” Well, that would take longer. At least, I think it would.
Mountain
November 2, 2007
I can’t control or really influence time allocation much. I might push here and there but it is up to everyone to do and not do what they want or have to.
Playoff PAWSmin would take time, I understand that. I’ve broken thru and calculated it in this case. I can do so again in the future if I choose. If I can read a comprehensive, well-written article from the method’s author that would be great. I’ll unwrap that present if it comes.
Morey does seem likely to be a user, perhaps even heavy user, of adjusted +/-. I’m quite interested in the topic of pure +/- vs “overall +/- with a statistical +/- component”. I wonder if he is using pure or a hybrid.
I wonder Dave if you have any thoughts you are willing to share about pure +/- vs hybrids and if you have ever computed a hybrid or have any temptation to do so or to comment of the existing one presented to a large degree but not totally. But that can come as the time of your choosing too of course.
Mountain
November 2, 2007
And before I go I want to ask have you taken or considered taking WP or the other metrics to a players on the court adjusted basis (as I mentioned in passing above)? I get the impression you might not go that direction because it might emphasize player interactions more than I think I hear you believe they should be but would welcome any clarification on that topic as well if it seems worthwhile to you.
Brian
November 2, 2007
Berri’s writing and arrogant handling of other thinkers in this area is really quite irritating.
For starters, to think that Battier’s departure was one of the main reasons behind Memphis’s atrocious season is absurd…but I do agree that he’s a pretty helpful player, even if the reasoning isn’t well presented in this post.
Berri says that Hollinger’s PER is bad because it puts him as slightly below average…but Hollinger’s comments on Battier showcase one of the big flaws with Wins Produced. The main reason for Battier’s PER dip in 2007 was due to the fact that he would pass up relatively high percentage shots far too frequently – which means instead of the possession culminating in a 40% 3-point attempt, it culminated in a much lower percentage attempt from Rafer Allston, for example.
In WP’s blind rejection of quantity of production in lieu of efficiency, this tendency to only take the super-wide open shots improves Battier’s score by quite a bit. The teammate who has to take the lesser quality shot that he passed up is the one who will, over time, have his WP score drop.
The problem that this reveals is that sometimes a shot that might on a regular basis decrease one’s WP score…is still, in that context, the best option for the team. Because Battier shot relatively infrequently last year and only went for it when absolutely most likely to make a basket, Wins Produced sees this as ideal behavior. PER recognizes that if you are a decent shooter, then you need to try for more opportunities, because those opportunities are often higher percentage options than the alternative. TMac and Yao desperately needed outside scoring help at times, and Battier should have provided more of it.
In WP, if a player shoots and misses, then they are penalized the number of points that is likely the possession would have gained if not for that shot (although ideally this should account for context – a shot near the end of the shot clock is the LAST option available, so missing in such contexts shouldn’t cause a penalty, since there was no trade-off with another scoring option at that point).
So why not the reverse? If a player has a decent opportunity and passes it up for a worse option in a bad TMac shot or Allston 3 – what Hollinger said was characteristic of Battier last season – shouldn’t he be penalized in his WP score for not shooting? Obviously this is impossible to practically do, but it showcases a problem inherent to WP, of penalizing bad shots but not penalizing the lack of good shots.
Mountain
November 2, 2007
In rough terms, starting anew, I’d probably take a split the difference near the middle approach on the FG% break even level between PER and WP.
Chirstopher
November 2, 2007
Define “decent opportunity” and maybe we can talk. Otherwise, I call purple crack.
Example: “TMac and Yao desperately needed outside scoring help at times, and Battier should have provided more of it.”
Battier shot 42.1% from 3pt range! Seems like he’s doing fine. This is the long-range threat you wanted.
Example: “In WP’s blind rejection of quantity of production in lieu of efficiency, this tendency to only take the super-wide open shots improves Battier’s score by quite a bit.”
Uh, there is a large body of empirical evidence linking efficiency to wining. Show me your empirical evidence re: quantity.
Palamida
November 2, 2007
I’d like to comment regarding Battier’s ability to “take a a charge”.
As far as i know, An offensive foul , stat wise is just that – a PF counted for the person who commited the violation. Although some charges are simply what they are (The Attacking player’s “fault”), Others are clearly the result of good positioning and /or understanding of the game. If The player who “recieved” the charge would be credited for a steal (as he should), some player’s WP marks would naturally be better, and even considerably better. Battier ” drew” 48 charges in 2005/06. Raja Bell for example led the league in “drawn offensive fouls” in 2005/06 with 76 charges in 79 games. that could mean almost a steal per game! (thanks 82games for that data). and that’s how some defensive contribution by some players that gets overlooked can be reinstated. As for charges that aren’t really “drawn”, that same thing goes for many other stats for example: a totally uncontested rebound, a steal which is in a fact just a bad pass that was easily caught and not “stolen” etc. However we can argue that over the course of a season\career, it’s negligible.
Josh Coleman
November 2, 2007
Brian said:
For starters, to think that Battier’s departure was one of the main reasons behind Memphis’s atrocious season is absurd…but I do agree that he’s a pretty helpful player, even if the reasoning isn’t well presented in this post.
Uhh…Grizzlies blogger here to chime in and say the Berri is absolutely correct that the departure of Shane Battier, who was replaced by rookie G/F Rudy Gay and F/C Stromile Swift, was a big factor in the Grizzlies’ dismal season last year. This was not just for his statistical contributions, but for his defensive ability, leadership and contributions to team chemistry. The trade that sent Battier to Houston should pay off in the long run for Memphis, while providing the Rockets with a player to help them “get over the hump” in the playoffs. But to think that the Grizzlies wouldn’t have been significantly better with his veteran presence last season, especially the first 20 games, when Pau Gasol was out with an injury, is simply an uninformed opinion that doesn’t deserve consideration, in my opinion.
But what do I know? I’ve only watched every game the team has played over the past 5 seasons.
Great article, as always Dave.
Guy
November 2, 2007
Dave:
There is no question about the value of “box score” statistics at the team level. If we know how much a baseball team outscored it’s opponents, we can predict its W-L record very accurately — the standard error is about 4 wins given a 162-game season. For a basketball team, the point differential predicts the W-L record even more accurately (because the SDs for points and points allowed is much smaller, relative to the mean).
In fact, if the test is explaining team wins, you can throw out most of the information in the box score. All you really need is runs/points scored and allowed. That allows you to predict W-L as well as you’ll ever be able to (given the limitations of a 162-game or 82-game “sample” of performance), since the remaining 6-10% of variance in win% is just random binomial variation.
However, the important question about these kinds of statistics is not predicting team wins and losses. The question is whether and to what extent they accurately assess the contributions of INDIVIDUAL players. That’s what all the debates about boxscore stats, including WP, are really about.
It it quite easy for boxscore stats to work at the team level yet be very misleading at the player level. For example, it would be easy to create a baseball productivity metric that evaluates hitters purely on the basis of runs scored per plate appearance, and evaluates pitchers based on runs allowed per inning. My two metrics would account for 90% of the variance in team wins, which is virtually perfect (the remaining variance being random). Yet everyone who understood baseball would immediately see I hadn’t measured much at all — I had disregarded the contributions of hitters who drove in those who scored, I had given no defensive credit to fielders, etc. etc.
Establishing correlation with team wins and losses is a useful first step — if your metric fails that test, it probably isn’t measuring what you want. But it cannot possibly tell you whether you’ve measured the contributions of individual players correctly.
Brian
November 2, 2007
“Battier shot 42.1% from 3pt range! Seems like he’s doing fine. This is the long-range threat you wanted.”
The point that Hollinger made about Battier which I am bringing up here, and that you missed very badly, is that he didn’t shoot those 3s enough. I was using this as an indication why pure efficiency isn’t always the same as most productive option. I don’t know what “42% is high!” is supposed to mean in response to that.
APBRmetrics
November 2, 2007
Dr. Berri,
You say that “this argument has been advanced that team defense, added to any variable, can explain [current] wins just as well [as Wins Produced].” Could you please specify where that claim was ever made? I have followed this debate fairly closely and don’t remember anyone making that claim.
You also say that “nowhere in [Rosenbaum’s and Lewin’s] writings, though, do they tell us how this [team adjustment] is constructed.” But this post at APBRmetrics spells out how this team adjustment probably was constructed (if this is what is used in the presentation) and makes it clear that it is different than the team adjustment that Wins Produced used. This post was from more than six months ago, so I think it might be a little misleading to say that Mr. Rosenbaum and Mr. Lewin have been “extremely misleading” about this.
You also say that “you can’t just throw a variable into a model unless you have a theoretical reason for doing so.” But I think it is fair to say that lots of non-academics make decisions without writing down a formal mathematical model. I think Mr. Rosenbaum and Mr. Lewin are trying to capture NBA decision-making with these Points per Game, Minutes per Game, and NBA Efficiency metrics (all with team adjustments). So I think the model here is that NBA GMs factor team efficiency perfectly into their player evaluations, but within a team they use Points per Game, Minutes per Game, or NBA Efficiency to evaluate players. I think that would get you the models that Mr. Rosenbaum and Mr. Lewin propose.
And regardless of how they get to the models, it is still the case that the do a better job predicting the future than Wins Produced. And I think the logic of why is easy to see. Wins Produced assumes that a player should be indifferent between a turnover and a missed shot. But the former results in the opposing team taking possession 100% of the time, while the latter results in the opposing team taking possession about 70% of the time. So the player clearly prefers missing a shot to turning the ball over, but Wins Produced doesn’t account for this, leading to rebounds being grossly overvalued and scorers being grossly undervalued.
In addition, has anyone provided deeper critiques of Adjusted Plus-Minus than Mr. Rosenbaum? Given that, it is not surprising to see that the one-year-at-a-time version of Adjusted Plus-Minus performed so poorly. A careful and fair reading of Mr. Rosenbaum’s writings would have suggested that he might come to this conclusion. He has often argued that several years of data or adding in box score data can greatly improve this methodology and that mis-applying this metric can make things worse rather than better.
Also, the predictive power of Adjusted Plus-Minus tells us nothing about its descriptive power. It can easily be the case that there are not enough minutes for any one player for it to be a good predictor, while simultaneously over a whole sample of players and seasons there being plenty of signal to evaluate various metrics with Adjusted Plus/Minus, i.e. with how the team does when specific players are in the game.
Finally, many of these issues have been discussed in a much greater level of depth over at the the APBRmetrics message board. Feel free to come over and join the discussion. I think we would benefit and so would you.
APBRmetrics
November 2, 2007
I tried to be too clever in the previous post with the links. Here is the first link.
http://sonicscentral.com/apbrmetrics/viewtopic.php?t=1232&postdays=0&postorder=asc&start=30&sid=054808549203d75556bea0058a3aab0b
Here is the second link.
http://sonicscentral.com/apbrmetrics/viewforum.php?f=1
Harold A.
November 3, 2007
APBRmetrics writes, Wins Produced assumes that a player should be indifferent between a turnover and a missed shot.
Very good point. Win Score formula is:
Points + Rebounds + Steals + ½Assists + ½Blocked Shots – Field Goal Attempts – Turnovers – ½Free Throw Attempts – ½Personal Fouls
It would seem a turnover and missed shot are penalized the same. That’s weird b/c they should vary by the % of shots that motivate offensive rebounds. Hmmm.
Guy
November 3, 2007
APBRMetrics:
It appears that Rosenbaum’s team adjustment is different than that employed in WP. Rosenbaum uses regression to capture whatever part of efficiency differential a given variable doesn’t account for. So of course every metric, coupled with his team adjustment, perfectly predicts efficiency differential. His team adjustment is indeed a “fudge factor” (as he admits). But the WP adjustment is a fixed, defined variable. It’s not fair to compare the two. I’m not faulting Rosenbaum, as when he posted that analysis I don’t think Dave had yet revealed the defensive adjustment formula. But Rosenbaum needs to do an apples-to-apples comparison, using similar defensive adjustments, if his analysis summarized in the powerpoint hasn’t already done so.
APBRmetrics
November 3, 2007
Guy, it looks like Mr. Rosenbaum addressed this in that post in March.
http://sonicscentral.com/apbrmetrics/viewtopic.php?t=1232&postdays=0&postorder=asc&start=30&sid=054808549203d75556bea0058a3aab0b
“What this essentially does is account for everything that predicts team efficiency that isn’t in the original metric. Now it would be possible to do this by including variables for those things (and this is what I think Berri does), but this way does the same thing.”
Note that what Mr. Rosenbaum is calling a “team adjustment” is a combination of Dr. Berri’s team defense, fouls, and assists/blocks adjustments. It looks to me like how Dr. Berri handles fouls is slightly different than what Mr. Rosenbaum assumes, but otherwise this is an apples-to-apples comparison.
If the two methods of deriving the overall Wins Produced team adjustment result in the same number for the adjustment (except for this slight difference in how fouls are handled), then it doesn’t matter which method was used.
dberri
November 3, 2007
APBRmetrics,
Guy is correct on this. What Rosenbaum did is quite incorrect.
Let me explain it this way. Here is a basic regression model.
Y = a1 + a2*X + e
When we evaluate a model, we want to know how X explains Y. e, or the residual (or error term), is not used to evaluate the explanatory power of a model.
What Rosenbaum is doing is saying that points scored, plus the residual, explains as well as Wins Produced. But all models, once you include the residual, explain the dependend variable equally well. So this just an extremely poor argument and not a proper use of econometrics.
And I would add that the team adjustment is fully explained in The Wages of Wins. If you read the thread you see that he knows what this is. Another story people tell is that we did not reveal all the details of Wins Produced in our book. This is not true. Wins Produced is explanined across two chapters. The chapters also have several pages of end notes. Every effort was made to explain how this was constructed.
Guy
November 3, 2007
The two methods are not the same. Rosenbaum basically “cheats” by creating an adjustment that captures everything about efficiency differential that his first variable (points, PER, whatever) fails to measure. And he acknowledges this. That’s just not the same as Berri’s team adjustment. (Rosenbaum also assumed that Berri’s team adjustment included factors, such as an adjustment for assists, that it apparently doesn’t include.) In any case, if Rosenbaum can construct team adjustments using the same variables that Berri uses, and still finds that the alternative metrics better predict future wins — and perhaps he can — that will tell us a lot.
APBRmetrics
November 3, 2007
One additional point that Mr. Rosenbaum makes in a handful of posts in this thread is that the team adjustment for Points per Game is team efficiency (or a weighted combination of the team variables that make up team efficiency) plus a factor that undoes the effect of Points per Game (just like Dr. Berri’s team adjustment undoes assists). So I guess in this case, it is also possible to write down a formal mathematical model. And overall this metric is much simpler than Wins Produced and predicts the future better, so it would seem like Points per Game with its team adjustment is clearly preferred to Wins Produced.
APBRmetrics
November 3, 2007
I think Mr. Rosenbaum all along has made exactly the same point that Dr. Berri makes above about explaining current wins not being a valid way of evaluating these metrics. He makes that point in this very thread. The real test would seem to be predicting future wins and “adding in this residual” would seem to be just fine if that is the way used to evaluate these metrics.
APBRmetrics
November 3, 2007
Guy,
I am a little puzzled as to why the Wins Produced team adjustment (that does include an adjustment for blocks/assists – see the technical notes) is the only valid team adjustment that any metric can use. I think the point you are trying to make is that it isn’t good to use the residual in constructing a metric if the end goal is to explain current team wins. I don’t see how anyone can disagree with that point.
But it seems to me that Dr. Berri wants to make a broader point when he argues that NBA decision-makers are “irrational.” It seems that he wants to argue that they are using the wrong model to predict the future.
But for that purpose, Points per Game with its team adjustment is simpler than Wins Produced and it predicts the future better. So by Dr. Berri’s own logic, an NBA decision-maker who used Wins Produced and not Points per Game with a team adjustment would be “irrational.”
dberri
November 3, 2007
APBRmetrics,
The “irrationality” argument we make in The Wages of Wins (and in an article coming out this month) is not based on a comparison between Wins Produced and NBA Efficiency (although this comparison supports the point we make in the book).
When we look at free agent salary and voting for the all-rookie team we find that scoring dominates these decisions. For salaries, factors like shooting efficiency, steals, and turnovers don’t matter at all. These factors do impact wins, hence decision-makers are not acting according to the dictates of instrumental rationality.
To understand this argument, you need to understand what we mean by “instrumental rationality.” I encourage you to read the book for the full definition.
I would thank you for pointing me to where Rosenbaum defined his team adjustment. Frankly I thought — after 18 months of his arguments (which often ended in a personal attack on me) he had done something that could at least be defended on some grounds. But when you use the residual to boost explanatory power you are violating basic econometrics. I understand why his 12 power point slides he presented at Harvard failed to define what he was doing. I doubt he would have found a very warm reception if he fully explained to his audience what he meant by a “team adjustment.”
Okapi
November 3, 2007
It seems to me as if dberri is regressing win totals against point differentials and calling it a theoretically defined model. I would think that’s a borderline tautology. It doesn’t validate how he then splits up those points into Points + Rebounds + Steals + ½Assists + ½Blocked Shots – Field Goal Attempts – Turnovers – ½Free Throw Attempts – ½Personal Fouls. You would get the same R^2 of 0.94 even if you split up the points just by players and ignored every other element of the box score. You could just add up everyone’s points scored and include a team term for points surrendered and get the R^2 of 0.94. Is this what “Rosenbaum” is saying?
The way dberri disaggregates everything seems intuitively appealing, and I’d be inclined to bet on (over other linear combinations of box score data) its robustness in an out of sample test as players switched teams. (Predicting a team’s performance next year when the roster is unchanged would seem to just fold into its point differentials last year regardless of how the point differentials are assigned to individual players, right?)
APBRmetrics
November 3, 2007
Dr. Berri,
Your are making a valid point if Mr. Rosenbaum’s presentation was about explaining CURRENT team wins.
But Mr. Rosenbaum and Mr. Lewin are examining how different metrics predict (or explain) FUTURE team wins, and for that there is no violation of any econometrics or statistics assumptions. Mr. Rosenbaum and Mr. Lewin are not adding in the residual for that regression. They are adding in the residual from a regression from an earlier year, so there is no violation in doing this.
So instead of making the same argument that Mr. Rosenbaum has repeatedly made – that explaining CURRENT team wins is not a valid measuring stick – it would seem more useful to respond to the argument actually being made.
And yes, NBA decision-makers may be making decisions sub-optimally, but the Wages of Wins arguments only fail to reject that steals, shooting efficiency, assists, etc. have an effect on various NBA decisions. That is not the same thing as saying that they “don’t matter at all.” I think that is why in standard statistics/econometrics, we are taught that “failing to reject the null hypothesis” is not the same as “accepting the null hypothesis.”
And more generally, the point here is that NBA decision-makers would, in some sense, be more “rational” when they are trying to PREDICT player value if they used Points per Game with its team adjustment than if they used Wins Produced.
And it leads to an interesting question. Why does Wins Produced predict FUTURE team wins worse that Points per Game with its team adjustment? How is it possible that one can throw away rebounds, assists, steals, turnovers, missed shots, fouls, and still do better than Wins Produced? It would seem like Wins Produced would run circles around such a limited metric in terms of predicting FUTURE team wins, but it doesn’t. Why?
APBRmetrics
November 3, 2007
Fixing a typo. This sentence in the earlier post should read.
And yes, NBA decision-makers may be making decisions sub-optimally, but the Wages of Wins arguments only fail to reject that steals, shooting efficiency, assists, etc. have NO effect on various NBA decisions.
dberri
November 3, 2007
APBRmetrics,
Let me put this simply…
We do not use the residual to boost the explanatory power of a model in the current period. And we do not use the residual to make out of sample forecasts either. Rosenbaum is welcome to try and publish this work in an economics journal. But I think he will find any referee, if this referee understands what he did, will reject this argument.
JC
November 3, 2007
I’m sorry, but I want to step in here to say that this is ridiculous. The only thing consistent about APBRmetrics argument is that whatever Dave Berri says is wrong. I have a strong feeling who this anonymous person is (don’t we all), and you don’t have to be a genius with tracking IP addresses to figure this out. Dave, feel free to ignore this clown, as no one thinks he’s making a worthwhile point. The day that Rosenbaum actually writes up his critique as an academic-quality paper, the discourse can proceed. There is the objective forum where the debate is supposed to take place. Until that time, I think it’s clear who is winning the argument and who is being evasive.
Please, keep up the good work Dave. I hate to see you waste your time with sophists.
Okapi
November 3, 2007
What exactly is Rosenbaum’s model? By incorporating the residual into his model is he fitting the intercept? Bringing an extra variable that correlates w/ that residual and then reconstructing the model? Is he just doing something trivial that is really just showing how win totals (and point differentials) of NBA teams are autocorrelated so that it’s difficult to improve upon a model that takes last year’s performance to predict next year’s? Does dberri beat Rosenbaum in predictive accuracy when players shift teams since it appears dberri is systematically assigning individual credit whereas Rosenbaum is arbitrarily doing so? Thank you.
APBRmetrics
November 3, 2007
Dr. Berri and JC,
This is a conversation about a methodology. Of course, such a conversation could be restricted to academic journal articles, but there is a wider community here that doesn’t have access to those journal articles. So what is the harm of having that conversation here out in the open?
(And by the way, I agree that it would be better if I could reveal who I am, but sometimes this is the only way to have a healthy conversation about methodology like we are having here. So I will remain “anonymous.”)
Let me write down a model that helps explain the issue here.
WIN(t) = a + b*X(t-1) + e(t),
where WIN(t) is wins in year t and
X(t-1) is a function of points per game for the given player in year t-1, points per game aggregated up for the entire team in year t-1, and team efficiency in year t-1. (Alternatively, team efficincy could be replaced by own team and opponent versions of points, field goal attempts, free throw attempts, reobund, and turnovers in year t-1.)
It looks to me that this is the key regression in this discussion. Where is the violation here?
dberri
November 3, 2007
I agree with JC. Write the academic article. Clearly explain what you are doing so everyone can see it. What I see is really bad econometrics. This will fool people who do not know econometrics. But I do not think it will work in an academic journal. You are free to demonstrate that I am wrong.
APBRmetrics
November 3, 2007
Fair enough. Since I am assuming it is OK for me to drop pretenses here, this has been helpful as I am writing up this paper. It is helpful in figuring out where I can take short-cuts and where I need to carefully explain things. As I have done in the past, I was planning to point out that these metrics cannot be used to explain CURRENT wins, but I will be sure to emphasize that point more. Also, I will spend more time explaining why the econometrics are OK here for explaining (or predicting) FUTURE wins. This conversation has helped me think of a couple different ways of doing that.
And one final note. I apologize for being so impatient. From the very beginning, I think I have wanted to have this conversation that we started to have today. And I have tried to do it both privately and publicly and gotten frustrated and you have gotten frustrated and we all know where that has led. So I apologize for the part that I have played in all of that.
Guy
November 3, 2007
I think there could still be a constructive discussion here*, though it won’t be easy.
To “APBR,” I would say: You are right that it’s not illegitimate to use year T residuals to predict year T+1 wins. You’re basically saying, whatever metric X can’t explain at the team level, I will allocate to players based on playing time. That’s not much different in theory from WP, which essentially uses regression to predict those residuals using team defensive variables. (And I was obviously mistaken about the assists adjustment, which I gather is separate from the defensive adjustment.) However, when comparing metrics’ ability to predict future performance, you do need to give WP the same benefit as the other metrics in terms of including residuals. Then you have a level playing field.
To dberri, I would say: First, Rosenbaum has not hidden his approach at all. In fact, I posted a summary of his case and a link to the very same post back in APRIL, on this website: https://dberri.wordpress.com/2007/04/26/the-face-on-mars-and-wins-produced/. Second, Rosenbaum is not suggesting that use of residuals is legitimate if the question is predicting current wins. Indeed, his whole point is that it proves nothing to do this. He obviously believed that the WP defensive adjustment amounts to virtually the same thing as his residuals-based version. To the extent you feel that’s not true, you should certainly make that case. But it’s silly to accuse him of “using the residual to boost explanatory power,” when his original point was precisely that everything predicts team wins if you include the adjustment.
And even if Rosenbaum has misunderstood the details of the WP team adjustments, his fundamental point still seems right: ANY of these metrics, adjusted by use of opposition points scored and other team defensive stats, is going to have a very high correlation with current wins. So these correlations can’t tell us how well they are measuring value at the player level. And it’s trivial to demonstrate that severely flawed metrics can accurately predict team wins (as I argued above in #30). So let’s move the debate to where it needs to go, which is how well these metrics predict outcomes in the future, especially when players change teams or play with different teammates.
* There can be a constructive discussion, that is, if JC’s unfortunate advice to remain in the bunker and ignore your critics is ignored.
APBRmetrics
November 3, 2007
Guy,
I have, BTW, been allocationg all of the residuals when constructing Wins Produced, so right now it is on a level playing field. Although this conversation made me think that I might have to go back and create it precisely the way it is specified in the Technical Notes (except for the position adjustments which I will not be able to perfectly replicate). I agree that doing it the way I currently do gives Wins Produced a better chance of performing well, but I don’t want to get dismissed just because what I am calling Wins Produced isn’t exactly Wins Produced.
dberri
November 3, 2007
Guy,
I am going to make this point and then do my errands for today. Rosenbaum has been attacking my work, and the work of my co-authors, for 18 months. These attacks have often been quite personal and at times, were done anonymously. He can say he did this because he was frustrated, but that does not excuse the behavior. Scholars simply do not behave in this fashion.
Now that I see what he has done in its entirety, I stand by my evaluation. This is not good work. We do not use residuals to evaluate models, either within sample or out of sample. You are free to reach a different conclusion.
Although I do not think this work is any good, he is free to write up his argument in its entirety and submit this to a journal. That is the way scholars have their work evaluated.
Okapi
November 3, 2007
“Rosenbaum”‘s model appears to be
WIN(t) = a + b*X(t-1) + e(t),
where WIN(t) is wins in year t and X(t-1) is a function of points per game for the given player in year t-1, points per game aggregated up for the entire team in year t-1, and team efficiency in year t-1.
It seems to me as if “Rosenbaum”‘s model and deberri’s models would have roughly the same predictive power if players don’t switch teams as the models would collapse into taking last year and projecting it forward. The test of which is better would be predictions of team performance when players with inefficient scoring and high rebounding switched teams. (e.g. the Iverson trade last year, though that’s only one data point.) The models would make very different predictions in those instances. I suspect dberri’s model would do better, but it’s of course a purely empirical matter.
Okapi
November 3, 2007
I’m not sure if this is what “Rosenbaum” is doing, but a default benchmark model would seemingly be taking the team’s point differential and distributing it to players in accordance with their minutes played or points scored. No regressions or anything necessary. When players switch teams they get their prior team’s performance assigned to them in this manner. From a team’s new roster each player’s assigned performance in the prior year can be aggregated. And then future performance predicted. Then dberri’s model could be compared to this baseline prediction to see how much his scheme for assigning credit improves upon this.
dberri
November 3, 2007
Okapi,
Let’s say a model based only on points scored and a residual is the greatest forecasting model ever. What message would you give to a decision-maker? Only look at points scored in evaluating a player? The problem the decision-maker has is using the residual in evaluating a player. There is no way a decisi0n-maker would know this. The decision-maker does know rebounds, steals, turnovers, etc..
This is just one practical problem with this approach. The other is an empirical issue, or more precisely, a question of proper methodology. You simply do not use a residual to evaluate the predictive power of a model. This is true if you are looking at a model with-in sample. And it is true if you look out of sample.
If what Rosenbaum suggests is valid, then he has a whole mess of papers he can write. Because I am willing to bet the “Rosenbaum residual” approach can out-forecast virtually any model published in economics on any topic.
Okapi
November 3, 2007
Dr. Berri,
Isn’t using the residuals for out-of-sample predictive power the same thing as having a moving average component in an autoregressive moving average model?
I’m not trying to defend Rosenbaum’s folderol, just making sure I understand everything.
Thanks, as always, for continuing to run a very interesting blog and interacting with the commenters.
Guy
November 3, 2007
Dave:
I can see you’re not interested in dialogue with Dan Rosenbaum. That’s too bad for the rest of us, but I understand there have been harsh words (on both sides) and it’s certainly your prerogative to take that stance.
Still, if you’re going to critique his analysis, we should clarify the debate. The important issue is how should metrics be evaluated: their ability to predict current wins or future wins? The legitimacy of using residuals depends entirely on how you answer the first question. If the measure of accuracy is predicting current wins, then using residuals is inappropriate as we all agree (including Dan). But if the evaluation turns on predicting NEXT year’s wins, then I don’t see what’s wrong with using this year’s residuals. It’s really just a small improvement on what you do: measure that part of team performance you can’t account for using individual player stats, divide the credit according to playing time, and assign back to the players. If you disagree, you should explain why, not just say “residuals are verboten.”
(Incidentally, Bill James does the same thing in his Win Shares system for baseball. After going as far as he can with individual stats, if a team has won more or fewer games than it “should” have given the sum of those individual statistics, he just allocates that to the players. Now, if James said “Look how accurate my Win Shares system is: if I total the WS of the players on a team and divide by three, it exactly equals the wins of that team,” that would be a ludicrous argument. But if players’ Win Shares ratings predict what happens NEXT year, that’s a legitimate validation. )
In the end I think most analysts will agree that predicting future wins is the more important test here. So the key question is whether Rosenbaum’s analysis, showing that WP predicts future wins less well than several other metrics, holds up to scrutiny. Since I assume you won’t trust his data, it would probably be wise for the WOW authors to do such an analysis on their own.
Christopher
November 3, 2007
I was going to leave a long comment on the usefulness of Battier. But we are long passed that. So I’ll be brief. Last year Battier was 10th in 3pM and 12th in 3p%. Not sure where you want to find improvement. You already have a top tier player in this regard plus everything else he does. Again, purple crack.
Pete
November 3, 2007
Rosenbaum is ruining it for all of us with his supercilious demeanor, possible pseudononymous posting, and unwillingness to engage in unemotional debate. :-(
Westy
November 3, 2007
Arghh, this is frustrating.
What started somewhat constructively seems to have regressed. The bottom line, as Guy says, is determining the extent that these models “tell us how well they are measuring value at the player level.”
Let me be frank, these ‘advanced’ basketball statistics are a step in the right direction. We all agree that points scored, etc. are overrated, especially by the media and the common fan.
But these systems are not perfect. Adjusted plus minus, while often seemingly predicting valuable ‘glue’ guys well, is right now too ‘noisy’. And Dave, as much as you say you are set with the formula the way it is, I refuse to accept that the evaluation at the individual player level is optimized by giving players full credit for a defensive rebound or no credit for a shot attempt that is offensively rebounded. I think there’s room to improve on so many fronts and I look forward to that happening.
And really I can’t shake the feeling that the best work in these areas is being done privately by NBA teams using proprietary in-house analysis and we don’t even know about it.
Pete
November 3, 2007
I refuse to accept that the evaluation at the individual player level is optimized by giving players full credit for a defensive rebound or no credit for a shot attempt that is offensively rebounded.
Do the shots of certain players get offensively rebounded more so than others? If so, model should adjust for it. If not, it’s just random and model wouldn’t need to reflect. I don’t see why a player shouldn’t get full defensive rebounding credit.
Pete
November 3, 2007
We all agree that points scored, etc. are overrated, especially by the media and the common fan.
Not just by the common fan but by Jon Hollinger as well.
The Franchise
November 4, 2007
Hollinger’s PER, though, is an intermediate step forward from the NBA player ratings, which are not particularly useful.
Guy
November 4, 2007
Westy/post 62: Amen.
Kent
November 4, 2007
Wow, nearly 70 comments. Who would have thought that Shane Battier was such a controversial figure!
Okapi
November 4, 2007
If you model NBA team wins or point differentials (this would be your dependent variable) based on some aggregation of individual player statistics (e.g. points, minutes, shooting %) as your independent varaibles, then points surrendered would be captured by the residual. If some players switch teams and you wanted to predict the new team performance next year then equally assigning that residual to individual players (I guess it would be on a game – by – game basis as the intercept would reflect the average # of pts surrendered if the dependent variables are points scored by individual players) seems to be legitimate. Perhaps I’m misunderstanding, but I don’t see why this is not a proper use of econometrics. Sometimes I’ll forecast variable “y” using variable “x” and where, say, half the forecast error from the prior month will, on average, persist from this forecasting period to the next. In this case I’m using the prior period residual for forecasting purposes.
Kent
November 4, 2007
Rosenbaum,
When your model is based on points per game for the given player do you allocate the residuals according to points per game such that a higher scorer gets a higher percentage of the residual?
Pete
November 4, 2007
The Rosenbaum residuals are a mockery of statistics.
Kent
November 4, 2007
Just like Wins Produced, Morey looks at effective field goal percentage, turnovers, rebounding, and free throws–
http://www.houstonpress.com/2007-11-01/news/outside-the-box
Kent
November 4, 2007
One last message …
This article indicates Morey uses plus/minus to evaluate Battier– http://www.houstonpress.com/2007-11-01/news/shane-s-game/
Interesting that Wins Produced confirms this.
Pete
November 5, 2007
The only work on this that I know of is a paper presented at Harvard by David Lewin and Dan T. Rosenbaum that showed that Wins Produced predicts future wins worse than Points per Game, Minutes per Game, and NBA Efficiency if a team adjustment is added to those metrics
I think this is a fallacy of authority. Rosenbaum is citing Harvard as validation instead of putting the statistics on the table. Harvard Schmarvard. A bad paper with bad econometrics presented at Harvard is still bad.
Harold A.
November 5, 2007
Pete, please tone down the shrill rhetoric and let’s try to have a more constructive dialogue. We’re all after the same thing here and there is absolutely no reason to be mean-spirited in what you rite.
Harold A.
November 5, 2007
JC, you too should tone down your tone. It’s inappropriate to call someone a “clown” just because he contends that out of sample testing is necessary.
Pete
November 6, 2007
Harold A.,
It’s not as if Hollinger’s famous efficiency ratings have been subject to out of sample testing either. Even if there hawn’t been out of sample testing, at worst, wins produced is a really good descriptive scheme.
dberri
November 6, 2007
Okay, I will say it again. The out of sample “test” of Wins Produced that has been described is not an appropriate out of sample test.
Let me try and put it this way. I have heard of models that can explain within sample, but can’t explain out of sample. And such models — and Wins Produced is not one of these — would be a problem.
The “residual” model, though, is a model that cannot explain within sample. But we are supposed to be impressed that because the residual was used, it could predict out of sample.
I have never heard of a model with low explanatory power within sample but high predictive power out of sample. Of course, I have never heard of someone using the residual to forecast. So perhaps this is why.
In sum, this is not a serious test. I am sorry many of you have not studied enough statistics to see this.
Guy
November 6, 2007
Dave: I don’t think you’re understanding what Rosenbaum is doing. His “models” have PERFECT explanatory power within sample. (I say “models” because he is expressly not advocating the use of any of these as predictive models.) His whole point is that a lot of very different models can achieve the same success as WP in terms of in-sample prediction at the team level. That includes a version of WP with radically changed coefficients. His point isn’t that those revised coefficients are “right” and the original coefficients are “wrong;” only that correlation with current team wins cannot itself validate the coefficients (since radically different coefficients yield the same overall correlation).
I see nothing at all wrong with using Yr1 residuals to predict Yr2 performance. That’s essentially what you do: you use team defensive variables to predict what the core WP metric cannot. You have no idea which players really deserve credit for the defensive success, so you allocate it by minutes. And there’s nothing wrong with that, if it improves your predictive power.
And that’s the real issue: predicting future wins is the critical barometer here. You say that WP doesn’t have a problem predicting out of sample, but how do you know that? Rosenbaum’s data indicates you have a pretty serious problem, if it’s correct. If you can refute that, you should (instead of denigrating the statistical knowledge of others).
dberri
November 6, 2007
Guy,
There is just no way I can teach you basic statistics in the comments section of a blog.
We do not use the residual to evaluate explanatory power. So his models do not “have perfect explanatory power” within sample.
I will review the issues people look at in evaluating a model. First and foremost, do you have a theory. Rosenbaum has argued wins can be defined strictly by points scored. What is the theory that suggests this? And what is the theory behind NBA Efficiency?
Secondly we look at such issues as significance of coefficients and explanatory power within sample (i.e r-squared).
And then we look at ability to forecast.
Rosenbaum has proposed models that do not have a theory and cannot explain within sample. He is then using the residual from these models to forecast. This is simply not good analysis.
I am sorry you can’t see this. But again Guy, as much as JC Bradbury, Rod Fort, and Skip Sauer have tried, we cannot teach you basic statistics and research methodology on a blog.
And with that statement, I am closing down comments on this post. Feel free to comment elsewhere. But for once, I get the last word on a blog post.