Think of a collection of people as having a kind of collective mind. How can you find out what that collective mind believes?
That may sound like a fanciful philosophical question, but it has very real, even urgent applications. For example the IPCC is a collection of hundreds of scientists, and they put out reports supposedly embodying their consensus position – i.e. what they believe not as individuals but as a body of scientists.
There are of course already various methods for determining what people believe collectively; the IPCC have their own approach. Such methods have various strengths and weaknesses. For example, the IPCC approach is claimed to be riddled with political conflict.
A little while back, at Austhink, we came up with an alternative approach, which worked successfully in its first application. We have used it a number of times since with various organisations, calling it the “Wisdom of Groups” method.
Here is a write-up of the first deployment.
___________________________
A few years back, the National Centre for Education and Training on Addiction and the South Australian Department of Health and Human Services put together a 3 day “Summer School” on the topic of addictions, inequalities and their interrelationships, with a view to providing guidance for policy makers. They said that 20% of the South Australian budget is used to deal with problems of addiction, so this is a major issue. They hoped to come up with a kind of Position Statement, which would summarise the consensus, if any, that the group of 50 or so participants reached during the Summer School.
They contacted Austhink hoping that we’d be able to help them with one aspect of it, namely making any debate/discussion/rational deliberation more productive. So initially the idea was that live argument mapping facilitation would be used with the whole group to help them work through some issues. But it became clear that they were open to ideas about how the Position Statement would be developed, and our involvement was increased to one of (a) developing a process for developing a Position Statement representing the group consensus, and (b) helping facilitate the overall Summer School to produce that Statement.
So we suddenly found ourselves faced with a very interesting challenge, which boiled down to:
- how do you figure out what, if anything, 50 participants with diverse backgrounds, interests, professional specializations, ideologies etc.. agree on:? i.e., how do you actually come up with a draft Position Statement?
- how do you rationally deliberate over that draft?
- how do you measure the degree to which the participants do in fact agree with any given aspect of that Statement – i.e., the extent to which the resulting draft Position Statement does in fact represent the consensus of the group?
This challenge has a lot in common with problems of democratic participation, of the sort that Deliberative Polling is intended to deal with.
Our approach, in a nutshell, was this:
Phase 1: Developing a Draft Statement
The first two days were occupied mostly with presentations by a range of experts (this had already been set up by the organizers; we had to work around that). We divided the Position Statement into three categories:
- Definitions and Values
- Empirical Realities; and
- Directions.
At the end of the first day, participants filled out a worksheet asking them to nominate 5 distinct propositions in the Definitions and Values category, propositions which they regarded as true and worth including in any Position Statement. On the second day, they filled out similar workshops for Empirical Realities and Directions. Then, for each category, Paul Monk and I spent a few hours synthesizing the proposed propositions into a set of around 10 candidate statements for inclusion in the Position Statement. This involved sorting them into groups and then extracting the core proposition from each group. So this resulted in a set of about 32 statements. Note, however:
- This process was democratic, in that it treated everyone’s contribution pretty much equally.
- Nevertheless, no one of these statements was put forward by a majority of people. It simply wasn’t clear to what extent these statements represented the consensus view of the whole group.
- Third, and somewhat parenthetically, it is worth noting that, in most cases, it was apparent to Paul and I that most participants had only a very partial and idiosyncratic understanding of the material they had been presented with. The synthesized statement sets, however, were (in our opinion) very good “takes” on the material. In other words, unsurprisingly, 50 brains really are a lot better than one (in most cases). The trouble is synthesizing the thinking of 50 brains.
Phase 2: Deliberation
Half of day three was devoted to deliberating over selected statements, using real-time argument mapping with the Reasonable software. A whole-group session introduced the participants to the approach, and made some progress on a particular issue. In another session there were two groups with separate facilitators; each chose their own issues to debate.
Phase 3: Consensus
In the final phase all participants used an online questionnaire to register their attitudes towards each of the 32 propositions. Each participant was asked, for each statement, to choose Agree/Disagree/Abstain, Include/Exclude/Abstain, and was able to offer comments. The online system automatically and immediately collated the results, producing graphical (bar chart) displays of the level of consensus.
In the final session, the results were reviewed. We found that there was a surprisingly high level of consensus on almost all propositions; that, in other words, the draft Position Statement did in fact represent a consensus opinion of the group. Note also that the Position Statement is accompanied by a precise specification of the extent to which the group in fact thought that each component statement was (a) true, and (b) worth including.
The level of consensus regarding the Position Statement developed through the process is particularly noteworthy in light of the fear, expressed to us prior to the Summer School, that there would be such disagreement between the major groupings of participants (roughly, the “addictions” people, and the “inequalities” people), that there would be literally nothing (or nothing worth saying) that they could agree on.
We think that this technologically-augmented (in two ways: argument mapping software+projection, and online questionnaire) process could well be deployed again in a wide range of contexts in which groups get together and need to figure out what they think on a complex issue; and in particular, need to figure out what, if anything, they can agree on to form a basis for policy.
Tim van Gelder 
That is fantastic! We always complain about the idea of “decision making by committee” but you’ve managed to find a way to use the technology to bring a serious number of people to consensus. Well done, Tim.
It’s also interesting to me that you were able to do it with, effectively, binary choices instead of likert scales, and still reach consensus. Obviously the initial work you did developing the Draft Statement paid off.
Do you have any thoughts for how you might increase participants understanding of “the field and the material they had been presented with”? Would that have been helpful to have more people ‘get it’ better, or counterproductive in that there may have been more divergent opinions that might have stalled the consensus process?
Thanks for posting this Tim!
Re “Do you have any thoughts for how you might increase participants understanding of “the field and the material they had been presented with”?”
Well, one thing that seems remarkably inefficient is the standard practice of having experts deliver presentations, typically aided by lots of PowerPoint slides dense with bullet points, charts etc..
Here’s a kind of cart-before-the-horse suggestion that might actually work: don’t bother with trying to get people to have a better understanding of the field prior to doing the consensus-opinion-determining activity. Rather, do that activity, and hand people the results. If they were to largely just accept the collective understanding, in most cases they’d be far better off.
I heartily agree, and am surprised and delighted to find someone else willing to go out on this non-intuitive limb that meetings are not an efficient way to present information. I believe information should be presented beforehand, after which meetings are for 1) massaging social connections, and 2) deciding or creating something.
The problem becomes convincing institutions to set up their processes so that information is mastered beforehand. Are you familiar with the “brainwriting” technique of Paul Paulus?
In brainwriting, ideas are written rather than spoken, then passed among group members. If tool such as your software is being used, previous ideas necessarily are considered before one knows where and how to append one’s own.
http://www.ideatree.us/view-brainwriting.php5
i think there is a goldmine in here. I have to say that I am coming to rely less and less that anyone really has a strong handle on any answers to really tough problems. I had this driven home this morning by Steven Landsburg (The Big Questions, Chapter 8, Diogenes Nightmare). If you think you believe what you say you believe, you owe it to everyone you are in discourse with to read this chapter. I think Tim should make this Chapter, if not the whole book a prerequisite to participating in one of his workshops.
I think the implicit move towards finding a consensus, however carefully achieved and recorded, contains some dangers. My brief view of the above example, is to note that the intro says…
“… South Australian Department of Health and Human Services put together a 3 day “Summer School” on the topic of addictions, inequalities and their interrelationships, with a view to providing guidance for policy makers. They said that 20% of the South Australian budget is used to deal with problems of addiction, so this is a major issue. They hoped to come up with a kind of Position Statement, which would summarise the consensus, if any, that the group of 50 or so participants reached during the Summer School”.
I’d be happier with it all if the SA Dept. were challenged on why they felt the need to do this (wooly in this example, at least to my reading) and what precisely they thought they might do with a result.