The folks at FiveThirtyEight have created a set of pages displaying voluminous information about their track record, i.e. how good their own forecasts have been on matters like U.S. elections and NFL games. The information is presented via very well designed interactive graphics:
FiveThirtyEight (or its founder, Nate Silver) are famous for making some spot-on predictions, but an interesting lesson you can immediately derive from perusing these statistics is that their performance in many forecast categories is far from perfect. But then, forecasting performance shouldn’t only be compared with perfection. It should be compared with other benchmarks indicating how well it is possible, in practice, perform. For particularly tough types of forecasts, performance that is reliably just above chance can be impressive.
Perhaps most remarkable here are two things:
- The rigour with which they track and analyse their performance
- The fact that they share this information so openly
Is this relevant to intelligence organisations? It is easy to dismiss this kind of thing, for a number of reasons. One is to do with the type of forecasts being made. Presidential elections are very black and white – a candidate either wins or she doesn’t, usually at a specific time. Such questions “resolve,” as they say. Many forecasting challenges in the intelligence space aren’t like this, and too much is lost in translation when you try to reduce them to resolvable questions. Also, for obvious reasons, organisations will often have good reasons for not being as transparent as FiveThirtyEight.
Friedman and Zeckhauser have made a pretty compelling case for doing something along these lines in Why Assessing Estimative Accuracy is Feasible and Desirable.
Abstract: The US Intelligence Community (IC) has been heavily criticized for making inaccurate estimates. Many scholars and officials believe that these criticisms reflect inappropriate generalizations from a handful of cases, thus producing undue cynicism about the IC’s capabilities. Yet there is currently no way to evaluate this claim, because the IC does not systematically assess the accuracy of its estimates. Many scholars and practitioners justify this state of affairs by claiming that assessing estimative accuracy would be impossible, unwise, or both. This article shows how those arguments are generally unfounded. Assessing estimative accuracy is feasible and desirable. This would not require altering existing tradecraft and it would address several political and institutional problems that the IC faces today.