Statistical Analysis vs. Human Experts

Posted on September 28, 2007 by


Can statistical analysis best human judgment?  This is one of the arguments that we make in The Wages of Wins.  And we have seen that this one argument has been greeted by a bit of skepticism from some of our readers (and others who have never read our book).  Typically, the sentiment expressed is as follows: How can a statistical model yield decisions that are better than those reached from a lifetime of experience and personal observation?  Surely the skeptics argue – GM’s in the NBA are better evaluators of playing talent than a statistical model built by some economists who have never played college basketball?  The skeptics seem to conclude that these economists are just arrogant, egotistical idiots.  Yet, this is our argument – and it appears we are not alone in arguing that statistical analysis can improve upon the lessons learned from life experience.

The Financial Times recently ran a piece on two political scientists (Andrew Martin & Kevin Quinn) who claim they can predict how Supreme Court justices will vote.  Like us, they met some skepticism.  One of the skeptics, Ted Ruger (a law professor from the University of Pennsylvania), challenged Martin & Quinn to compare the accuracy of two different methods of predicting Supreme Court cases.  Martin & Quinn would use their model, and Ruger would rely on 83 legal experts.  Each attempted to predict opinions for cases that would come before the Supreme Court in 2002 (and had yet to be decided).  I bet you will never guess how this ended?  Yes, the statistical model correctly predicted the Supreme Court opinion 75% of the time, while the legal experts were only correct 59.1% of the time.

Likewise, a regression based model is demonstrating superior assessment of whether criminals will re-offend.  The Rapid Risk Assessment for Sexual Offender Recidivism model gives a criminal a score (similar in concept to a credit score) and if a criminal has a point score of four or greater then there is a 55% chance of committing another sex offense.  The state of Virginia uses this model in part when making parole decisions (along with human discretion).

In another completely different area, Mark Nissen, professor at the Naval Postgraduate School concludes that the shift is toward computer based systems where human discretion is eliminated in procurement decisions.

Here’s the rub – we are perceived by some as being arrogant because we advocate a decision making model that is based on statistical analysis.  Yet there is a growing pile of evidence that in many areas that shows experts who ignore statistical analysis tend to make poorer decisions.  Given this result, shouldn’t we conclude it’s more than a bit arrogant for “experts” to ignore what can be learned from statistical analysis?

– Stacey

Here is the Financial Times reference:

The Financial Times (September 1st/2nd) “How computers killed the expert”  p. 1 & 2 of Life & Arts.

Posted in: General