Feeds:
Posts
Comments

Many people who are new to argument mapping look for a convenient software tool.  Similarly, instructors in critical thinking or informal logic would often like to point their beginner students to a suitable tool for basic argument diagramming.  Ideally that tool would:

  • Be free
  • Be easy to use
  • Be good enough for simple maps
  • Not require installation
  • Not require a specific operating system (Windows, Mac) or browser
  • Not require creating an account on (yet another) website
  • Integrate seamlessly with other tools already being used
  • Be free (yeah, this seems to keep coming up)

After much searching around over the years, my current view is that the tool best meeting these conditions is Microsoft SmartArt.  Nearly everyone already has, or has access to, Word and PowerPoint, and many use them almost everyday.  Most would be surprised to know that they have built in a passable facility for quickly creating simple argument maps.

Of course I’d known about SmartArt, and the possibility of using it for argument mapping, for years.  However for most of that time I’d written it off as being superficially attractive but too limited and frustrating to use.  Recently I’ve changed my tune.  As described below, if you pick the right template and persevere a little bit, you’ll find that SmartArt can do a reasonable job.  It is certainly not ideal, but it may be the best – or rather, the least bad – option currently available.

If you’re not familiar with SmartArt there are introductory videos on Youtube, such as this one.

In what follows I’ll assume that argument maps are (basically; see below) hierarchies or tree structures.  This is convenient because all SmartArt templates are based on hierarchies, represented in editing mode as indented lists.  Some of the SmartArt templates are explicitly classified as “Hierarchy”:

smartart

The templates I find work best are Labeled Hierarchy and Table Hierarchy.  (Tip: don’t bother with the one called Hierachy.)

Here’s a very simple argument map in Labeled Hierarchy format:

simple trump map

This is using the default colour scheme.  A little adjusting using the usual formatting commands results in a map with a more standard colouring:

simple trump map colour

This template has a few drawbacks.  For example, the lines joining the arguments to the contention really should be separate arrows.  Overall, however, it is a pretty classy diagram, and it only takes a minute or two to create.

A neat feature of SmartArt is you can easily change the template while keeping the content the same.  Here’s the same map (minus the labels) in Table Hierarchy format:

tableformat

I’ve included in this image the editing panel at left.  This is only visible when the SmartArt graphic is selected.

As you’d expect, arguments can be nested indefinitely deeply:

tableformat2

The SmartArt algorithm is “space filling” so that no matter how many nodes there are in the argument, the map will fit into whatever space you specify for the SmartArt graphic.  The SmartArt graphic can be resized by simple dragging operations.  If you want to create a really complex map, you can set a large custom size for your Word page or your PowerPoint slide, and add as many boxes as you like.

Any experienced argument mapper reading this will no doubt be thinking something like:

Fine, but what about multi-premise arguments (a.k.a. linked arguments)?

The reality is that hierarchical argument maps are not actually simple tree structures.  The technical name for the kind of structure that argument maps have is hi-tree; see this paper for explanation, and a description of layout algorithms for hi-trees.  SmartArt is based on simple tree structures, and so in principle cannot properly represent reasoning. However the Table Hierarchy format allows a pretty good approximation:

linked

Note the nested linked arguments.

To create the bar which binds premises into a linked argument, you just create an empty node in the hierarchy, and resize and recolour it appropriately.

In theory, there’s no limit to how complex Table Hierarchy maps could get.  In practice, the map above is towards the upper limit for SmartArt argument maps.  The biggest problem you start to encounter is that modifying the map structure starts to become a challenging exercise in hierarchical puzzle-solving.  You can’t just drag and drop objects to add to, or modify, a map; all editing of structure is done in the left hand panel (see graphic above) as operations on an indented list.  This is easy in simple cases but becomes frustrating and time-consuming in more complex maps.

Even in simple cases, using SmartArt to create argument diagrams takes a certain amount of familiarity with SmartArt manipulation and Word formatting more generally.  I haven’t tried to cover these topics in this post.  If you’re an instructor recommending SmartArt as a diagramming tool, you’d probably want to have/get that familiarity yourself and then give your students some guidance.  Guidelines might include:

  • Use the right template (NOT the one called “Hierarchy”)
  • Specify font size across the whole graphic to specific size rather than allowing the algorithm to set font sizes
  • When needed, resize the whole graphic so text fits nicely in boxes

These are very simple operations to carry out when you’re familiar with them.

Also you probably should provide students with semi-prepared graphics for them to use as starting points.

For more advanced users…

For anyone who wants to get more serious about argument mapping, but still wants

  1. To stay within the Word environment; and
  2. Something free

there is the CASE-mapping Word Add-in we created to support a specific variety of argument mapping:

4787835

It comes with an instruction manual, but the support material that is currently publicly available is quite limited, so you need to have a pretty good idea what you’re doing to find this useful.

 

The question of who actually wrote the works attributed to “William Shakespeare” is a genuine conundrum.  In fact it may be the greatest “whodunnit” of all time.

Although mainstream scholars tend to haughtily dismiss the issue, there are very serious problems with the hypothesis that the author was William Shakspere of Stratford upon Avon. However all other candidates also have serious problems.  For example Edward de Vere died in 1604, but plays kept appearing for another decade or so.  Hence the conundrum.

Recently however this conundrum may have been resolved.  A small group of scholars (James, Rubinstein, Casson) have been arguing the case for Henry Neville.  A new book, Sir Henry Neville Was Shakespeare, presents an “avalanche” of evidence supporting Neville.  Nothing comparable has been available for any other candidate.

Suppose Rubinstein et al are right.  How can the relevant experts, and interested parties more generally, reach rational consensus on this?  How could the matter be decisively established?  How can the process of collective rational resolution be expedited?

A workshop later this month in Melbourne will address this issue.  The first half will involve traditional presentations and discussion, including Rubinstein making the case for Neville.

The second half will be attempting something quite novel.  We will introduce a kind of website – an “arguwiki” where the arguments and evidence can be laid out, discussed and evaluated not as a debate, in any of the standard formats, but as a collaborative project.  The workshop will be a low-key launch of the Shakespeare Authorship Arguwiki; and later, all going well, it will be opened up to the world at large.  Our grand ambition is that the site, or something like it, may prove instrumental in resolving the greatest whodunnit of all time, and more generally be a model for collective rational resolution of difficult issues.

The workshop is open to any interested persons, but there are only a small number of places left.

Register now.  There is no charge for attending.

 

Missing Pieces – The Skill of Noticing Events that Didn’t Happen
Spotting the Gaps – What Does it Take to Notice the Missing Pieces?

Summary

This pair of  short pieces were published a week apart by distinguished decision theorist Gary Klein. Their very-similar titles promise insight into how critical thinkers can be better at noticing absent evidence – things which are not present, or didn’t happen, but which might be just as “telling” for or against various hypotheses as their more salient “present” counterparts.   The advice he provides boils down to two points.  (1) Be experienced.  Experience sets up (often unconscious) expectations, whose violations might capture our attention, or at least create an uneasiness which prompts us to wonder what we’re missing.  (2) Have an active, curious mindset.  This “goes behind what we can see and hear, and starts puzzling when an expected event fails to materialize.”

Comment

I have plenty of respect for Klein, but these are disappointing pieces.  They mainly just rehash anecdotes from his earlier work.  He says very little about how experience or an active mindset actually work to help us notice what’s missing, or how to achieve either of these things.  In fact “an active curious mindset” seems to be little more than a redescription of the ability to notice things – barely more satisfying than saying “pay attention!” or “look around for what’s missing!”.  In studying these pieces, I engaged an active curious mindset.  I noticed what was missing: anything of any great insight or use.  Which I know from experience is unusual in Klein’s case.

Meta-analysis has become an indispensable part of modern science.  By pooling data from many studies, and using special mathematical techniques, meta-analysis answers more questions, with more power and precision, than is possible either with single studies or informal reviews.

Currently, however, meta-analysis is a closed activity. It is performed by small, funded teams operating behind closed doors and with the results only becoming available in technical journal articles which are often stuck behind pay-walls. This closed approach has serious problems, as we discuss below.

Change is overdue, and indeed on its way.  There are increasing calls to make meta-analysis projects more accessible, transparent, collaborative, and frequently updated (“living”) – or in a word, more open.

What would truly open meta-analysis be like as a social practice?  How would it fit into other practices such as journal publication?  What technological support does it need?  How well could it work?

We are confident that meta-analysis would benefit greatly from being conducted far more openly than it typically is today. We draw inspiration from the way the wiki transformed encyclopedia production, and open-source transformed software production.  Meta-analysis is, to be sure, a technical matter, but so is writing an encyclopedia article about leukemia or producing an operating system.

We see many serious challenges, but no fundamental barrier to having crowds collaborate in posing useful questions, identifying suitable studies, extracting key data, selecting and applying analytical methods, and deriving insights from the results.

We use the term “open collaborative meta-analysis” (OCMA) to indicate that in this open alternative, MA projects welcome all comers to not only view the data and findings, but to also contribute their data, their labor and their insights, regardless of whether they be career scientists working on the topic, undergraduates learning the ropes, or interested members of the public.

What’s wrong with how meta-analysis is currently done?

Thousands of meta-analyses appear every year, many of high quality, profoundly contributing to scientific knowledge.  However the current approach has some serious problems, caused in part by its closed nature.

One is painfully apparent to anyone who has actually done a meta-analysis: they take a lot of tedious work. When this work is shouldered solely by a small, funded team, it is slow and expensive. This reduces the number of projects that get undertaken, and narrows their scope. Important issues, such as the role of moderators, are neglected, and information inherent in the vast pool of primary studies is under-exploited.

The current approach can also harm meta-analytic findings, because, first, there is no opportunity, during the analysis, for external critical review of the innumerable judgements the team must make, such as what risk of bias rating to give a particular study.  When teams have only themselves as critics, they are more likely to make clerical mistakes or technical errors. Sometimes they may even be tempted by dubious, self-serving choices.  Second, it is hard for small teams, with their limited resources and networks, to identify all the studies which meet their inclusion criteria.  A recent review has found biomedical meta-analyses to be “consistently incomplete” in their evidence base. Third, a MA project typically comes to a halt when the small team has drawn their conclusions and drafted their publications, even though new relevant studies continue to appear.  This means that a project’s findings can quickly go out of date.

Consequently, all too often a meta-analysis’s findings are limited, wrong or misleading, notwithstanding the competence and diligence of the small team behind it.

The situation gets worse when we consider sets of meta-analyses in a given area. The closed and competitive nature of the current approach means that teams are often unaware that other teams are addressing the same or very similar questions – or they are aware, but push on regardless.  The result is redundant analyses and even confusion when different analyses present overlapping and conflicting findings. The closed approach, with its lack of transparency in the meta-analytic process, hinders clarification of why these differences exist and how they might be corrected.

Finally, there is a growing movement towards synthesizing meta-analyses into even larger studies. This is difficult to do when meta-analyses themselves are so poorly disclosed.

What is the open alternative?

The essence of OCMA is that meta-analyses are conducted as public collaborations.  Anyone can initiate a MA, and anyone can contribute to an existing project, in a range of ways. Projects are ongoing; they evolve over time as more studies become available, problems are corrected, analytical methods improve, and new questions are asked.

In this way meta-analysis projects benefit from much broader input than is possible in the standard approach, and both the scientific community and the public benefit from projects and their findings being so easily accessible, correctable, and continually updated.

A well-designed online platform will be needed for OCMA to work.  Since an online platform is, in one sense, just software code running on servers, the platform can itself become an open development project. Similarly, OCMA as a scientific practice, with its workflow, norms, roles, and sanctions, can be governed by by the community of users, much as the practices of open encyclopedia production are governed by the Wikipedia community.

OCMA is very general, applicable in any area of science.  We envisage a single platform capable of supporting analyses not just in biomedical science but in education and many other fields, though it may be more practical to have a number of specialised OCMA platforms.

OCMA changes the way people come together to share and collaborate, not the theory of meta-analysis. OCMA processes and platforms would support whatever range of statistical methods the scientific community deems appropriate.

Is there anything like this out there already?

OCMA, as we conceive it, does not yet exist.  There have been important developments pushing in broadly similar directions, but they all lack one or more key ingredients of true open, collaborative meta-analysis.  Space does not allow exhaustive comparisons, but we can illustrate with reference to some of the most comparable efforts:

  • openMetaAnalysis is biomedicine-specific, limited in functionality, and not easy to use.
  • Covidence presents the kind of platform interface quality OCMA needs, but was designed for traditional small-groups, and only supports gathering and coding of studies, not the full analytic process.
  • metaBUS makes results relatively easily accessible to the public, but depends on “curation” work by a cadre of technical specialists and currently at least is restricted to the field of human resource management.
  • The Systematic Review Data Repository makes data sets and systematic reviews available, but does not support open collaboration in the meta-analysis process
  • Live cumulative network meta-analysis is as yet only a concept, and focuses a very technical form of biomedical meta-analysis; it would not be suitable for the vast majority of meta-analysis projects.

More generally, as compared with existing developments, OCMA is to varying degrees more crowd-oriented, collaborative, widely applicable across scientific fields and user-friendly.

But have you thought of…?

OCMA faces formidable challenges. We describe some here, and sketch some possible solutions.  However we recognise that these are difficult problems, and that our prototyping exercise may well throw up many new ones.

Why would anyone bother contributing?

There are many different kinds of motivation for participating in open projects such as open encyclopaedias, open science projects, and open software development.  Different people would contribute to OCMA for their own mix of reasons.

For example, a researcher may want to put her MA-in-progress up on the open platform in order to gain the benefits of crowd involvement, such as contributions of labour, and double-checking of judgements.  Authors of relevant studies will often be motivated to ensure that their studies are included and treated appropriately. Other researchers may want to participate through interest, collegiality, and concern for correctness.

An important challenge is to allow researchers to get recognition for their contributions to open projects.  This problem has already arisen in other contexts of open knowledge production. OCMA would need to include mechanisms for reliably documenting and perhaps even assessing a researcher’s contributions. In parallel, the wider scientific community would need to evolve ways of accepting such documentation in performance evaluations.

Would OCMA analyses get published?  How?

At least initially, OCMA would not affect how meta-analytic results get published.  OCMA might support standard publications as follows. An OCMA platform would enable a researcher to “freeze” an instance of a suitably-developed project.  The researcher can take its findings and present them in an article which is then subject to a journal’s normal review processes and standards. The OCMA community is acknowledged as a kind of contributor, but the researcher takes final responsibility for completeness and correctness.

In the longer term, OCMA may give rise to an alternative to standard publishing for meta-analyses. There is currently much dissatisfaction with scientific publishing, and many people are exploring ways to improve or sidestep the standard journal publication processes.  A critical issue is how research gets authorised or endorsed by the scientific community.  We envisage that OCMA would develop practices, supported by the OCMA platform, for indicating when projects are sufficiently well-developed that they are at least as, if not more, authoritative than traditional journal publications.  These practices may be supported by rigorous quality tests comparing OCMA analyses with suitable benchmarks such as Cochrane reviews on the same topics.

Nuisance

An obvious problem is that if anyone can come in make changes, then malicious users could vandalise projects; users with vested interests may try to manipulate findings; and well-meaning users might just “stuff things up.”

This is a version of a standard early objection to Wikipedia. However Wikipedia has proven that nuisance is manageable. It has developed a range of responses, including the ability to revert changes; page watchlists; blocking vandals; and clean-up bots. Such methods can also be used in OCMA.

Also, an OCMA site will be less likely to attract nuisance users. The OCMA site would be relatively dry and technical in nature, and have a relatively small community of viewers and editors. We expect that a sufficiently large proportion of visitors will be reasonably competent and well-intentioned that nuisance could be managed adequately.

Stability and Customization

A key challenge will be to reconcile two apparently conflicting requirements.  On one hand, the whole idea is that projects continually evolve as users make incremental changes, or do exploratory “what ifs.”  On the other, users will want the project to remain fixed or stable for various purposes such as publishing findings.  One technical solution may be to enable signed-in users to save a configuration for a particular project.  A visitor can then view either the master version, one of their saved configurations, or one which has been shared with them by somebody else.

 


Thanks to the following for their input into this document:

  • Professor Robert Badgett, Preventive Medicine and Public Health, University of Kansas School of Medicine
  • Professor John Hattie, Director, Melbourne Education Research Institute, University of Melbourne
  • Professor Julian Elliott, Head of Clinical Research, Infectious Diseases, Alfred Hospital and Monash University; Senior Research Fellow at the Australasian Cochrane Centre
  • Dr. Charles Twardy, Senior Data Scientist, NTVI; Affiliate Professor, George Mason University

All Wikipedia Roads Lead to Philosophy is a brief discussion of the initially surprising fact that if you click the first link in any Wikipedia article, you’ll eventually arrive on the page for Philosophy.  It is worth trying yourself to experience why this happens: most first sentences on Wikipedia pages relate the page topic to some larger topic, e.g. “Geranium argenteum (silvery crane’s bill) is an ornamental plant…”.   This is a very cute way of revealing something important about how knowledge is organised, and how we explain things to each other.

Eliezer Yudkowsky claims that Nate Silver erred when calculating that the probability of Trump getting the nomination was 2%.  Silver’s calculation was based on Trump’s needing to pass through six stages and there was only 50% chance of passing each stage.  Yudkowsky believes that Silver should have used the conditional probability of passing each stage given that Trump had passed the previous stages.  For the sixth stage, for example, the probability that he would pass that stage may well be judged much higher than 50%, given that he had succeed in five previous stages.
space
Yudkowsky’s analysis seems relevant to explaining why people – allegedly – commit a basic error of probabilistic judgement, which is to fail to multiply the probabilities of a chain of independent events and hence to overestimate the probability that all events occur.  A standard illustration of this is (as I recall) something like the 10 lock problem.  A safe has 10 locks.  The burglar has a 90% chance of picking each lock.  What is the probability he breaks the safe?  .9^10 = approx .34, and apparently people tend estimate a much higher figure.  This might be explained, in a handwavy way, by saying they anchor on .9, and fail to sufficiently adjust.
 space
However, a more “ecological” approach might seek to understand people’s judgements in terms of how events in fact unfold in the “real” world, or the world they evolved in.  While it is possible to artificially define a situation in which the probability of cracking each lock is, by stipulation, .9, what would happen in the real world is that if you watched somebody trying to crack a safe, and they’d cracked 9 of 10 locks already, you’d think that the safebreaker is so good that they are almost certain to crack the last one.  In other words, you would – in a somewhat Bayesian manner – update your estimate of the safebreaker’s skill as each lock is cracked, and hence the probability of cracking the remaining locks.
 space
When a mathematically unsophisticated person is asked for their answer to the 10 safe problem, it is plausible (this is a testable conjecture) that they imagine the burglar starting with the first lock, probably cracking that, proceeding to the next lock, and so on.  (It seems unlikely they would mentally simulate the entire sequence.)  We know that decision making by mental simulation is a very common strategy (Klein).  This is not quite decision making, but it is similar; the RPD perspective suggests the decision maker mentally simulates one approach to see if it is likely to work, and similarly the naive subject may (start to) mentally simulate the sequence of lock cracking.
 space
This suggests two things.  First, people’s higher than “normative” estimate might be explained by their (in some vague sense) conditionalising the probabilities.  They intuitively judge that the chance of cracking the later locks in the sequence is greater than .9.  Second, depending on the environments they are normally in, this might be the right thing to do.  Or, put another way, the “fallacy” is to resist the idea of purely independent events, and to be quasi Bayesian.  Maybe they are being more rational, in some ecological sense, than the smarty-pants psychologists who try to trip them up.
space
space
space
space

The Only Rule Is It Has To Work is a great story about a minor league baseball team being taken over by two sabermetricians. It addresses one of the big questions of our time: to what extent can statistics and data science replace intuitive human judgement?  This is obviously Moneyball territory, but if it was just a repeat of that great book and movie, it wouldn’t be all that interesting. I won’t spoil the story, but there’s an important lesson here, and its more about communication than it is about analytics. For the quick version, see the author’s article in the New York Times, What Happens When Baseball-Stats Nerds Run a Pro Team?

Follow

Get every new post delivered to your Inbox.

Join 615 other followers