Feeds:
Posts
Comments

YourView Links

[This post is in response to a request from a colleague for “the best thing to introduce someone to YourView”.]

The goal of the YourView project was to develop a platform for identifying collective wisdom, focusing on major public issues, aiming to remedy some of the defects of democracy.  The platform was a deliberative aggregator, i.e. it supported large-scale deliberation and its aggregated outputs reflected that deliberation.  The project was active from around 2011 to 2014, with its moment of glory being when the platform was used as part of Fairfax Media’s coverage of the 2013 Federal Election.

The YourView platform is still alive and can be explored, though there has been no real activity on the site since 2014.  There is a collection of pages and links about the YourView project.

The best theoretical overview is Cultivating Deliberation for Democracy, particularly the second half.  The “The Zone” interview by Fairfax’s Michael Short is a good read.

Note: this is a draft section of a larger guide.  Comments welcome. 

What is reasoning? Everyone has an intuitive sense, though many would struggle if asked to define it.

A dictionary is usually a good starting point. Merriam-Webster defines reasoning as the process of thinking about something in a logical way in order to form a conclusion or judgment.  

This is OK as far as it goes, but we need to expand and sharpen it quite a bit.  To do this, let’s look at some simple examples.   

Reasoning as a mental activity

Suppose Daniel, someone you know and trust, tells you that a person he knows, Marie, is married.  You now know Marie is in fact married.  Put differently, you are now confident that the claim Marie is married is true.  

Now Daniel asks: does Marie have a husband? Think about that before reading on.

 

If you’re like most people, you would have quickly thought something like Of course Marie has a husband – she’s married! But then you may well have reflected a bit more.  Why would Daniel ask about this, if the answer is so obvious?  What’s the trick?  

The “trick,” of course, is that Marie could be married to a woman, and so have a wife.  Marie might be lesbian and live in a state allowing same-sex marriage.  Or Marie might in fact be a straight man.  Marie’s being married doesn’t prove she has a husband – though she probably does.    

Your thinking here involved considering various ideas – Marie’s being married, Marie’s being a lesbian, Marie’s being a man, and perhaps others – and arriving at a judgement about Marie’s having a husband.  This is reasoning in the Merriam-Webster sense.

In slightly technical terminology, we say that you considered various claims, and also your confidence in the truth of these claims:

Claims Your confidence in their truth
Marie is married. Certain
Marie is a lesbian, married to a woman. Remote possibility.  
Marie is a man, married to a woman. Remote possibility.

and, given the logical relationships among these claims, you arrived at a level of confidence in another claim:

Claim Your confidence in its truth
Marie has a husband. Probable

So in this sense, reasoning is a mental activity; it is:

  • Understanding the logical relationships, if any, among claims; and  
  • Adjusting your confidence in those claims accordingly.  

However this is not the full story.

Reasoning is also the network of claims

Sometimes the word “reasoning” is used to refer not to the mental activity but to the claims themselves.

For example here are the various claims in the above example, with a few extra words (but, and, so) used to indicate logical relationships among them:

Marie reasoning

This is the sense of “reasoning” we are using when we say things like Show me your reasoning! or The reasoning in the article is flawed.  

Reasoning in this sense is like a social network, except claims replace people, and logical relationships replace personal relationships.  Note that just as some people in a social network have no relationship with to each other, some of the claims in the reasoning might not be logically related at all.

Thus, “reasoning” has two different meanings: the mental activity, and the network of claims.  These are of course closely connected; the network is what the mental activity is about.  

Reasoning can be presented in prose, or in a diagram

A network of logically related claims is an abstract thing.  We always need some way to show or present the network, so that our minds can see and follow it.

The standard way to do this is to express the claims in prose (writing or speech).  Examples abound; just look at the opinion page of any newspaper.  Here’s an example of some reasoning expressed in standard prose:

Religion Reasoning.png

It’s not set out rigidly as in the list for the Marie example above, but it is still expressing logically related claims, aimed at getting you to agree that not all religions deserve equal respect.  (Plus, it’s more fun to read.)

Representing reasoning in text is so common, and so normal, that most people hardly even realise that that is what they are doing.  There is however an alternative.  We can represent a network of claims diagrammatically.  

Here’s a diagram for the religion example:

Religion diagram.png

Note that arrows are used instead of the words like but, and and so in the Marie example; and the claims are arranged left to right in a logical order, though one quite different to the order in which they appeared in the original text.

There are lots of different ways to diagram reasoning, depending on what conventions you choose to adopt.  The diagram above is very minimalist.  In this course, we’ll be using a few different types of diagramming.  

To understand somebody’s reasoning, we must model it.

As mentioned, people almost always present reasoning in ordinary prose.  Consequently, we (the readers) have to interpret the prose in order to understand what their reasoning is.  Sometimes this is simple and effortless.  Other times, it is very difficult.  Often it is not at all obvious exactly what the reasoning is, and we have to make our best guess.  

In this course, such “guesses” or interpretations are called models of reasoning.  The diagram above presents a model of the reasoning in the religion text.

Coming up with this model required:

  • Figuring out what claims were being made as part of the reasoning.  For example, the sentence “Jedi knights, for example?” was interpreted as making the claim It is appropriate to ridicule Jedi Knights.
  • Figuring out what logical relationships, if any, these claims are supposed have to each other.  The arrows show these logical relationships.   Notice that nothing in the original text explicitly specified these particular relationships.  They are a matter of interpretation.  

Now, you may not agree with the model expressed in the diagram.  You may think that the author’s reasoning was different.  You might be right; but that would just highlight that you are coming up with your own model of the reasoning, and that coming up with such models is what we always have to do when we read or listen to prose presentations.  

Religion Model.png

An argument map displays a model of reasoning

In practice, a reasoning model is usually displayed diagrammatically.  In the graphic above, the middle representation, the list of claims and their relationships, was included for a couple of reasons.  First, it emphasises the point that reasoning (in one sense) is a set of claims with logical relationships.  Second, it makes visually clear that the same reasoning can be expressed in prose or displayed in a diagram.  

A diagram displaying a model of reasoning in some text can be thought as a kind of map.   A good analogy here is the classic subway map.  The subway map does not show the subway system exactly as it is in reality.  Rather, it portrays certain aspects of the subway system.  Similarly, a diagram of the reasoning expressed in a piece of prose cannot display the reasoning itself; it can only ever show a model of the reasoning.  

A diagram displaying a model of the reasoning expressed in a text is called an argument map.  

Religion Map.png

Recap

We’ve just covered a fair bit of theory, so here is a brief recap.  We defined reasoning as

  1. In one sense, a mental activity, in which we understand the logical relationships among claims, and adjust our confidence in the truth of those claims accordingly.  
  2. In another sense, a network of claims defined by logical relationships.    

Reasoning in the second sense must always be expressed or displayed in some way so that we can see what it is and apply our reasoning capacities to it.  Almost always, reasoning is laid out in prose (speech or writing).  However it is also possible to present reasoning diagrammatically.  A diagram will usually be much better than prose in specifying exactly what the reasoning is.  

Often, it is not easy to identify the reasoning somebody has expressed in prose.  We need to make our best guess as to what that reasoning is; in other words, we need to come up with a model of the reasoning. An argument map is a diagram displaying such a model.

 

 

 

 

 

Many people who are new to argument mapping look for a convenient software tool.  Similarly, instructors in critical thinking or informal logic would often like to point their beginner students to a suitable tool for basic argument diagramming.  Ideally that tool would:

  • Be easy to use
  • Be good enough for simple maps
  • Not require installation
  • Not require a specific operating system (Windows, Mac) or browser
  • Not require creating an account on (yet another) website
  • Integrate seamlessly with other tools already being used
  • Be free

After much searching around over the years, my current view is that the tool best meeting these conditions is Microsoft SmartArt.  Nearly everyone already has, or has access to, Word and PowerPoint, and many use them almost everyday.  Most would be surprised to know that they have built in a passable facility for quickly creating simple argument maps.

Of course I’d known about SmartArt, and the possibility of using it for argument mapping, for years.  However for most of that time I’d written it off as being superficially attractive but too limited and frustrating to use.  Recently I’ve changed my tune.  As described below, if you pick the right template and persevere a little bit, you’ll find that SmartArt can do a reasonable job.  It is certainly not ideal, but it may be the best – or rather, the least bad – option currently available.

If you’re not familiar with SmartArt there are introductory videos on Youtube, such as this one.

In what follows I’ll assume that argument maps are (basically; see below) hierarchies or tree structures.  This is convenient because all SmartArt templates are based on hierarchies, represented in editing mode as indented lists.  Some of the SmartArt templates are explicitly classified as “Hierarchy”:

smartart

The templates I find work best are Labeled Hierarchy and Table Hierarchy.  (Tip: don’t bother with the one called Hierachy.)

Here’s a very simple argument map in Labeled Hierarchy format:

simple trump map

This is using the default colour scheme.  A little adjusting using the usual formatting commands results in a map with a more standard colouring:

simple trump map colour

This template has a few drawbacks.  For example, the lines joining the arguments to the contention really should be separate arrows.  Overall, however, it is a pretty classy diagram, and it only takes a minute or two to create.

A neat feature of SmartArt is you can easily change the template while keeping the content the same.  Here’s the same map (minus the labels) in Table Hierarchy format:

tableformat

I’ve included in this image the editing panel at left.  This is only visible when the SmartArt graphic is selected.

As you’d expect, arguments can be nested indefinitely deeply:

tableformat2

The SmartArt algorithm is “space filling” so that no matter how many nodes there are in the argument, the map will fit into whatever space you specify for the SmartArt graphic.  The SmartArt graphic can be resized by simple dragging operations.  If you want to create a really complex map, you can set a large custom size for your Word page or your PowerPoint slide, and add as many boxes as you like.

Any experienced argument mapper reading this will no doubt be thinking something like:

Fine, but what about multi-premise arguments (a.k.a. linked arguments)?

The reality is that hierarchical argument maps are not actually simple tree structures.  The technical name for the kind of structure that argument maps have is hi-tree; see this paper for explanation, and a description of layout algorithms for hi-trees.  SmartArt is based on simple tree structures, and so in principle cannot properly represent reasoning. However the Table Hierarchy format allows a pretty good approximation:

linked

Note the nested linked arguments.

To create the bar which binds premises into a linked argument, you just create an empty node in the hierarchy, and resize and recolour it appropriately.

In theory, there’s no limit to how complex Table Hierarchy maps could get.  In practice, the map above is towards the upper limit for SmartArt argument maps.  The biggest problem you start to encounter is that modifying the map structure starts to become a challenging exercise in hierarchical puzzle-solving.  You can’t just drag and drop objects to add to, or modify, a map; all editing of structure is done in the left hand panel (see graphic above) as operations on an indented list.  This is easy in simple cases but becomes frustrating and time-consuming in more complex maps.

Even in simple cases, using SmartArt to create argument diagrams takes a certain amount of familiarity with SmartArt manipulation and Word formatting more generally.  I haven’t tried to cover these topics in this post.  If you’re an instructor recommending SmartArt as a diagramming tool, you’d probably want to have/get that familiarity yourself and then give your students some guidance.  Guidelines might include:

  • Use the right template (NOT the one called “Hierarchy”)
  • Specify font size across the whole graphic to specific size rather than allowing the algorithm to set font sizes
  • When needed, resize the whole graphic so text fits nicely in boxes

These are very simple operations to carry out when you’re familiar with them.

Also you probably should provide students with semi-prepared graphics for them to use as starting points.

For more advanced users…

For anyone who wants to get more serious about argument mapping, but still wants

  1. To stay within the Word environment; and
  2. Something free

there is the CASE-mapping Word Add-in we created to support a specific variety of argument mapping:

4787835

It comes with an instruction manual, but the support material that is currently publicly available is quite limited, so you need to have a pretty good idea what you’re doing to find this useful.

 

The question of who actually wrote the works attributed to “William Shakespeare” is a genuine conundrum.  In fact it may be the greatest “whodunnit” of all time.

Although mainstream scholars tend to haughtily dismiss the issue, there are very serious problems with the hypothesis that the author was William Shakspere of Stratford upon Avon. However all other candidates also have serious problems.  For example Edward de Vere died in 1604, but plays kept appearing for another decade or so.  Hence the conundrum.

Recently however this conundrum may have been resolved.  A small group of scholars (James, Rubinstein, Casson) have been arguing the case for Henry Neville.  A new book, Sir Henry Neville Was Shakespeare, presents an “avalanche” of evidence supporting Neville.  Nothing comparable has been available for any other candidate.

Suppose Rubinstein et al are right.  How can the relevant experts, and interested parties more generally, reach rational consensus on this?  How could the matter be decisively established?  How can the process of collective rational resolution be expedited?

A workshop later this month in Melbourne will address this issue.  The first half will involve traditional presentations and discussion, including Rubinstein making the case for Neville.

The second half will be attempting something quite novel.  We will introduce a kind of website – an “arguwiki” where the arguments and evidence can be laid out, discussed and evaluated not as a debate, in any of the standard formats, but as a collaborative project.  The workshop will be a low-key launch of the Shakespeare Authorship Arguwiki; and later, all going well, it will be opened up to the world at large.  Our grand ambition is that the site, or something like it, may prove instrumental in resolving the greatest whodunnit of all time, and more generally be a model for collective rational resolution of difficult issues.

The workshop is open to any interested persons, but there are only a small number of places left.

Register now.  There is no charge for attending.

 

Missing Pieces – The Skill of Noticing Events that Didn’t Happen
Spotting the Gaps – What Does it Take to Notice the Missing Pieces?

Summary

This pair of  short pieces were published a week apart by distinguished decision theorist Gary Klein. Their very-similar titles promise insight into how critical thinkers can be better at noticing absent evidence – things which are not present, or didn’t happen, but which might be just as “telling” for or against various hypotheses as their more salient “present” counterparts.   The advice he provides boils down to two points.  (1) Be experienced.  Experience sets up (often unconscious) expectations, whose violations might capture our attention, or at least create an uneasiness which prompts us to wonder what we’re missing.  (2) Have an active, curious mindset.  This “goes behind what we can see and hear, and starts puzzling when an expected event fails to materialize.”

Comment

I have plenty of respect for Klein, but these are disappointing pieces.  They mainly just rehash anecdotes from his earlier work.  He says very little about how experience or an active mindset actually work to help us notice what’s missing, or how to achieve either of these things.  In fact “an active curious mindset” seems to be little more than a redescription of the ability to notice things – barely more satisfying than saying “pay attention!” or “look around for what’s missing!”.  In studying these pieces, I engaged an active curious mindset.  I noticed what was missing: anything of any great insight or use.  Which I know from experience is unusual in Klein’s case.

Meta-analysis has become an indispensable part of modern science.  By pooling data from many studies, and using special mathematical techniques, meta-analysis answers more questions, with more power and precision, than is possible either with single studies or informal reviews.

Currently, however, meta-analysis is a closed activity. It is performed by small, funded teams operating behind closed doors and with the results only becoming available in technical journal articles which are often stuck behind pay-walls. This closed approach has serious problems, as we discuss below.

Change is overdue, and indeed on its way.  There are increasing calls to make meta-analysis projects more accessible, transparent, collaborative, and frequently updated (“living”) – or in a word, more open.

What would truly open meta-analysis be like as a social practice?  How would it fit into other practices such as journal publication?  What technological support does it need?  How well could it work?

We are confident that meta-analysis would benefit greatly from being conducted far more openly than it typically is today. We draw inspiration from the way the wiki transformed encyclopedia production, and open-source transformed software production.  Meta-analysis is, to be sure, a technical matter, but so is writing an encyclopedia article about leukemia or producing an operating system.

We see many serious challenges, but no fundamental barrier to having crowds collaborate in posing useful questions, identifying suitable studies, extracting key data, selecting and applying analytical methods, and deriving insights from the results.

We use the term “open collaborative meta-analysis” (OCMA) to indicate that in this open alternative, MA projects welcome all comers to not only view the data and findings, but to also contribute their data, their labor and their insights, regardless of whether they be career scientists working on the topic, undergraduates learning the ropes, or interested members of the public.

What’s wrong with how meta-analysis is currently done?

Thousands of meta-analyses appear every year, many of high quality, profoundly contributing to scientific knowledge.  However the current approach has some serious problems, caused in part by its closed nature.

One is painfully apparent to anyone who has actually done a meta-analysis: they take a lot of tedious work. When this work is shouldered solely by a small, funded team, it is slow and expensive. This reduces the number of projects that get undertaken, and narrows their scope. Important issues, such as the role of moderators, are neglected, and information inherent in the vast pool of primary studies is under-exploited.

The current approach can also harm meta-analytic findings, because, first, there is no opportunity, during the analysis, for external critical review of the innumerable judgements the team must make, such as what risk of bias rating to give a particular study.  When teams have only themselves as critics, they are more likely to make clerical mistakes or technical errors. Sometimes they may even be tempted by dubious, self-serving choices.  Second, it is hard for small teams, with their limited resources and networks, to identify all the studies which meet their inclusion criteria.  A recent review has found biomedical meta-analyses to be “consistently incomplete” in their evidence base. Third, a MA project typically comes to a halt when the small team has drawn their conclusions and drafted their publications, even though new relevant studies continue to appear.  This means that a project’s findings can quickly go out of date.

Consequently, all too often a meta-analysis’s findings are limited, wrong or misleading, notwithstanding the competence and diligence of the small team behind it.

The situation gets worse when we consider sets of meta-analyses in a given area. The closed and competitive nature of the current approach means that teams are often unaware that other teams are addressing the same or very similar questions – or they are aware, but push on regardless.  The result is redundant analyses and even confusion when different analyses present overlapping and conflicting findings. The closed approach, with its lack of transparency in the meta-analytic process, hinders clarification of why these differences exist and how they might be corrected.

Finally, there is a growing movement towards synthesizing meta-analyses into even larger studies. This is difficult to do when meta-analyses themselves are so poorly disclosed.

What is the open alternative?

The essence of OCMA is that meta-analyses are conducted as public collaborations.  Anyone can initiate a MA, and anyone can contribute to an existing project, in a range of ways. Projects are ongoing; they evolve over time as more studies become available, problems are corrected, analytical methods improve, and new questions are asked.

In this way meta-analysis projects benefit from much broader input than is possible in the standard approach, and both the scientific community and the public benefit from projects and their findings being so easily accessible, correctable, and continually updated.

A well-designed online platform will be needed for OCMA to work.  Since an online platform is, in one sense, just software code running on servers, the platform can itself become an open development project. Similarly, OCMA as a scientific practice, with its workflow, norms, roles, and sanctions, can be governed by by the community of users, much as the practices of open encyclopedia production are governed by the Wikipedia community.

OCMA is very general, applicable in any area of science.  We envisage a single platform capable of supporting analyses not just in biomedical science but in education and many other fields, though it may be more practical to have a number of specialised OCMA platforms.

OCMA changes the way people come together to share and collaborate, not the theory of meta-analysis. OCMA processes and platforms would support whatever range of statistical methods the scientific community deems appropriate.

Is there anything like this out there already?

OCMA, as we conceive it, does not yet exist.  There have been important developments pushing in broadly similar directions, but they all lack one or more key ingredients of true open, collaborative meta-analysis.  Space does not allow exhaustive comparisons, but we can illustrate with reference to some of the most comparable efforts:

  • openMetaAnalysis is biomedicine-specific, limited in functionality, and not easy to use.
  • Covidence presents the kind of platform interface quality OCMA needs, but was designed for traditional small-groups, and only supports gathering and coding of studies, not the full analytic process.
  • metaBUS makes results relatively easily accessible to the public, but depends on “curation” work by a cadre of technical specialists and currently at least is restricted to the field of human resource management.
  • The Systematic Review Data Repository makes data sets and systematic reviews available, but does not support open collaboration in the meta-analysis process
  • Live cumulative network meta-analysis is as yet only a concept, and focuses a very technical form of biomedical meta-analysis; it would not be suitable for the vast majority of meta-analysis projects.

More generally, as compared with existing developments, OCMA is to varying degrees more crowd-oriented, collaborative, widely applicable across scientific fields and user-friendly.

But have you thought of…?

OCMA faces formidable challenges. We describe some here, and sketch some possible solutions.  However we recognise that these are difficult problems, and that our prototyping exercise may well throw up many new ones.

Why would anyone bother contributing?

There are many different kinds of motivation for participating in open projects such as open encyclopaedias, open science projects, and open software development.  Different people would contribute to OCMA for their own mix of reasons.

For example, a researcher may want to put her MA-in-progress up on the open platform in order to gain the benefits of crowd involvement, such as contributions of labour, and double-checking of judgements.  Authors of relevant studies will often be motivated to ensure that their studies are included and treated appropriately. Other researchers may want to participate through interest, collegiality, and concern for correctness.

An important challenge is to allow researchers to get recognition for their contributions to open projects.  This problem has already arisen in other contexts of open knowledge production. OCMA would need to include mechanisms for reliably documenting and perhaps even assessing a researcher’s contributions. In parallel, the wider scientific community would need to evolve ways of accepting such documentation in performance evaluations.

Would OCMA analyses get published?  How?

At least initially, OCMA would not affect how meta-analytic results get published.  OCMA might support standard publications as follows. An OCMA platform would enable a researcher to “freeze” an instance of a suitably-developed project.  The researcher can take its findings and present them in an article which is then subject to a journal’s normal review processes and standards. The OCMA community is acknowledged as a kind of contributor, but the researcher takes final responsibility for completeness and correctness.

In the longer term, OCMA may give rise to an alternative to standard publishing for meta-analyses. There is currently much dissatisfaction with scientific publishing, and many people are exploring ways to improve or sidestep the standard journal publication processes.  A critical issue is how research gets authorised or endorsed by the scientific community.  We envisage that OCMA would develop practices, supported by the OCMA platform, for indicating when projects are sufficiently well-developed that they are at least as, if not more, authoritative than traditional journal publications.  These practices may be supported by rigorous quality tests comparing OCMA analyses with suitable benchmarks such as Cochrane reviews on the same topics.

Nuisance

An obvious problem is that if anyone can come in make changes, then malicious users could vandalise projects; users with vested interests may try to manipulate findings; and well-meaning users might just “stuff things up.”

This is a version of a standard early objection to Wikipedia. However Wikipedia has proven that nuisance is manageable. It has developed a range of responses, including the ability to revert changes; page watchlists; blocking vandals; and clean-up bots. Such methods can also be used in OCMA.

Also, an OCMA site will be less likely to attract nuisance users. The OCMA site would be relatively dry and technical in nature, and have a relatively small community of viewers and editors. We expect that a sufficiently large proportion of visitors will be reasonably competent and well-intentioned that nuisance could be managed adequately.

Stability and Customization

A key challenge will be to reconcile two apparently conflicting requirements.  On one hand, the whole idea is that projects continually evolve as users make incremental changes, or do exploratory “what ifs.”  On the other, users will want the project to remain fixed or stable for various purposes such as publishing findings.  One technical solution may be to enable signed-in users to save a configuration for a particular project.  A visitor can then view either the master version, one of their saved configurations, or one which has been shared with them by somebody else.

 


Thanks to the following for their input into this document:

  • Professor Robert Badgett, Preventive Medicine and Public Health, University of Kansas School of Medicine
  • Professor John Hattie, Director, Melbourne Education Research Institute, University of Melbourne
  • Professor Julian Elliott, Head of Clinical Research, Infectious Diseases, Alfred Hospital and Monash University; Senior Research Fellow at the Australasian Cochrane Centre
  • Dr. Charles Twardy, Senior Data Scientist, NTVI; Affiliate Professor, George Mason University

All Wikipedia Roads Lead to Philosophy is a brief discussion of the initially surprising fact that if you click the first link in any Wikipedia article, you’ll eventually arrive on the page for Philosophy.  It is worth trying yourself to experience why this happens: most first sentences on Wikipedia pages relate the page topic to some larger topic, e.g. “Geranium argenteum (silvery crane’s bill) is an ornamental plant…”.   This is a very cute way of revealing something important about how knowledge is organised, and how we explain things to each other.