Prediction markets can be a remarkably effective way to divine the wisdom of crowds.
Prediction markets of course only work for predictions – or more generally for what I call “verifiable” questions. A verifiable question is one for which it is possible, at some point, to determine the answer definitively. For example, predicting the winner of the Oscar for best picture. This is what allows the prediction market to determine how much each player wins or loses.
The problem is that many issues we want addressed are not verifiable in this sense.
For example, decisions. Would it be better to continue to negotiate with Iran over its nuclear ambitions, or should a military strike be launched? We can speculate and debate about this, but we’ll never know the answer for sure, because one path will be taken, and the other never taken, and so we’ll never know what would have happened had we taken the other path.
Wouldn’t it be good if we had something like a prediction market, but which works for non-verifiable issues?
Amazon.com book ratings are an interesting case. Whether a book is or is not a good one is certainly a non-verifiable issue. Yet Amazon has created a mechanism for combining the views of many people into a single collective verdict, e.g. 4.5 stars. At one level the system is just counting votes; Amazon users vote by choosing a numerical star level, and Amazon averages these. But note that Amazon’s product pages also allow users to make comments, and reply to comments; and these comment streams can involve quite a lot of debate. It is plausible that, at least sometimes, a user’s vote is influenced by these comments. So the overall rating is at least somewhat influenced by collective deliberation over the merits of the book.
Amazon’s mechanism is an instance of a more general class, for which I’ve coined the term “deliberative aggregator“. A deliberative aggregator has three key features:
- It is some kind of virtual forum, thereby allowing large-scale, remote and asynchronous participation.
- It supports deliberation, and its outputs in some way depend on or at least are influenced by that deliberation. (That’s what makes it “deliberative.”)
- It aggregates data of some kind (e.g. ratings) to produce a collective viewpoint or judgement.
YourView is another example of a deliberative aggregator. Yourview’s aggregation mechanism (currently) is to compute the “weighted vote,” i.e. the votes of users weighted by their credibility, where a user’s credibility is a score, built up over time, indicating the extent to which, in their participation on YourView, they have exhibited “epistemic virtues,” i.e. the general traits of good thinkers.
Many other kinds of deliberative aggregators would be possible. An interesting theoretical question is: what is the best design for a deliberative aggregator? And more generally: what is the best way to discern collective wisdom for non-verifiable questions?
Tim van Gelder 
Tim,
Excellent and compelling concept I had never considered. I have just experienced the DA phenomenon first-hand, in a message thread I have ben participating in on a popular rock climbing blog’s discussion board.
In “sport” rock climbing, where bolts are placed to offer safety in the case of a fall, there has been a HUGE ongoing debate about the secondary practice of artificially creating “holds” (via drilling holes in the rock, or chipping them in with a chisel) to facilitate moving over expanses of rock that seemed unclimbable without such modifications. The vast majority of climbers are very hostile to this practice and hold the dogmatic view that it is patently “unethical.”
I initiated this discussion thread by posting that the blogger’s report demonstrated that this view might be mistaken; that “hold modification” might in fact be justified in certain contexts.
This board allows people to vote “thumb up” or “thumbs down” for any particular response. The results are automatically weighed and either a positive or negative (or zero) is assigned to any post with such votes (e.g., 3 thumbs down and 1 thumbs up results in a “-2” rating).
As expected, my post initially received a steadily increasing negative rating. However, after many more subsequent posts, where I took on various members of the opposing view, as well as put forth much more detailed argumentation in support of my standpoint, I noticed a shift in the rating of my initial post (which tends to get the most ratings due to its location at the top of the thread). It seems then, that the “overall merits” of my initial post were in fact affected by the “collective wisdom” of the growing dialogue located in subsequent commentary.
I look at this as a positive phenomenon; one in which the prevalent viewpoint is indeed shaped by the dialogue and exchanges of the participants, rather than simply a knee-jerk expression of pre-existent beliefs that may in fact lack sufficient justification.
John
Thanks, Tim – and I like the term you’ve coined.
There would appear to be potential applications for teaching and learning at secondary and tertiary levels; a tool to help students collectively learn ‘how to think’ and problem solve.
Best
Victoria Heathcote
In a broader sense. If you cannot verify
1) what soes it matter
2) how can you validate the methods
I expect Amazon has a verifiable goal.(s) Drive page views, time on site, sales increases and market share.
I believe you meant to say “a way to divine”.
Of course Tim, nothing can really be “verified”. Abduction and induction are both forms of conjecture, rather than logically justifiable forms of argument. This might seem like nit-picking but foundationalism does lead to all sorts of biases.
“I may be wrong and you may be right, and by an effort, we may get nearer to the truth” stated on page 225 of volume two “The Open Society and Its Enemies” remained for the rest of his life as Karl Popper’s core expression of critical rationalism.
In Popper’s view, phrasing critical rationalism in this way “I may be wrong and you may be right” prevents dogmatism as he hasn’t said “I may be right and you may be wrong”. It is up to the individual whether or not to try to adopt this attitude. Acceptance of any theory in Popper’s view ought to be consciously tentative only, though one may believe that a theory is preferable to certain alternative theories.
With respect to criticism, valid inferences are not tools of exploration of the world but rather are tools for exploring our existing conjectures. We attempt to clarify and criticize rather than ground our conjectures. The source of conjectures is open.
What metric or ground could one apply for nearness to the truth, no matter how dearly we would like certainty? At least falsifiability and criticism gives us a fighting chance of reducing our mistakes.