Supreme Court decisions predicted by online computer program

As the U.S. Supreme Court moves into its new term, litigants, attorneys and the public will be closely watching its docket and speculating about its decisions. Now, thanks to the Supreme Court Forecasting Project at Washington University in St. Louis, court watchers everywhere will be able to log on to the Internet and obtain a forecast of how individual cases are likely to be decided.

The project accurately predicted decisions in 75 percent of the cases heard by the Court in its last term.

“People can go to our web site and input some information about a Supreme Court case, such as where the case comes from, who the litigants are, the issue at hand, and so forth, and the model will produce a forecasted decision based on a statistical model,” said Andrew Martin, project collaborator and an assistant professor of political science in Arts & Sciences at Washington University in St. Louis.

Supreme Court cases are now predictable, thanks to new computer model.
Supreme Court cases are now predictable, thanks to new computer model.

Launched last year by researchers at Washington University in St. Louis and Harvard University, the Supreme Court Forecasting Project began as a friendly interdisciplinary competition that pitted political science methodologies against the individualized expertise of a panel of legal experts. For every case argued in the Court’s last term, the project compared two different methods of predicting the outcome − one relying on a computer model; the other on opinions of legal experts. Overall, the statistical model accurately predicted 75 percent of the court’s decisions, while the experts as a group were correct on only 59.1 percent of the cases.

“The fact that this statistical model did better than the experts as a group − and the group included law professors and deans of law schools and people that practice before the Supreme Court − shows that the social scientific methodology can be very successful at predicting what’s going to happen in the future,” Martin said.

The project Web site, located at wusct.wustl.edu, includes background on the project and the predictions and actual outcomes for all of the cases heard in the court’s 2002 Term. Forecasts for most of the important cases scheduled for hearing in the current term will be added to the site as details become available.

Other project researchers include Pauline T. Kim, professor of law at Washington University; Theodore Ruger, associate professor of law at Washington University; and Kevin M. Quinn, an assistant professor of government at Harvard University. Their results, and a discussion of the implications of the project for understanding the Court, will be published in the Columbia Law Review in May 2004.

For decades, political scientists and legal scholars have held heated debates about how and why the Supreme Court reaches a particular decision in the important and often controversial cases it hears each session.

“The Supreme Court is an extremely important institution in American politics,” said Ruger. “Its decisions impact a diverse array of vital economic, social and structural questions. A method that succeeds in recognizing patterns in its decision-making is of considerable interest to many parties.”

Although legal scholars tend to claim the intellectual high ground when it comes to knowledge of the Supreme Court, social scientists have begun challenging that supremacy by developing complex statistical models that gauge how decisions are influenced by myriad social, political and legal factors. Much of the analysis hinges on how the court has decided previous cases and how a particular case meshes with the conservative or liberal leanings of individual justices.

“What social scientists are really good at is looking in the past and trying to understand what happened, but the Supreme Court Forecasting Project is innovative in that instead of looking backward, it looks forward,” Martin said. “It uses the tools of modern social science to figure out what this important institutions is going to do in the future.”

More often than not, the model and the experts reached similar predictions about how a particular case would be decided, although the way in which they reached their predictions was quite different. For instance, in the undergraduate University of Michigan affirmative action cases that came before the court last term (Gratz v. Bollinger), both the statistical analysis and the experts accurately predicted that the Supreme Court would strike down the admissions policies. But, in the companion case challenging the University of Michigan Law School’s affirmative action policy, one legal expert correctly predicted that the Law School’s policy would be upheld, while the statistical model forecast an identical outcome to the undergraduate case.

In other cases, the model and expert predictions clearly diverged. For example, for Eldred v. Ashcroft, a case about intellectual property, the statistical model accurately forecasted an affirmance of the lower court’s decision, while both experts incorrectly forecast reversal.

The project team hopes their work will encourage a greater interdisciplinary dialogue between political scientists and legal academics who study the Court.

“Obviously, there is much more to the law than merely the ‘affirm’ or ‘reverse’ outcome,” Kim said. “Nevertheless, the fact that the statistical model did better than the experts at predicting outcomes last term suggests that there is something that the legal experts can learn from the methods of political science.”

Although results from the legal experts panel provided a nice benchmark for the accuracy of the project’s statistical analysis program, the panel component will not be repeated for the 2003 term, due mostly to the cost and complexity of identifying panelists and gathering their predictions. Although the project’s computer model will not be competing directly with a panel of experts, Martin expects the online availability of real-time predictions to spur increased interest in the forecasting project.

“When the Supreme Court agrees to hear a case, people will be able to come to our site and get some sense of how the case is likely to be decided,” Martin said. “And, as decisions are made, people can come back to the site and see if our forecast was accurate.”