Feeds:
Posts
Comments

Missing Pieces – The Skill of Noticing Events that Didn’t Happen
Spotting the Gaps – What Does it Take to Notice the Missing Pieces?

Summary

This pair of  short pieces were published a week apart by distinguished decision theorist Gary Klein. Their very-similar titles promise insight into how critical thinkers can be better at noticing absent evidence – things which are not present, or didn’t happen, but which might be just as “telling” for or against various hypotheses as their more salient “present” counterparts.   The advice he provides boils down to two points.  (1) Be experienced.  Experience sets up (often unconscious) expectations, whose violations might capture our attention, or at least create an uneasiness which prompts us to wonder what we’re missing.  (2) Have an active, curious mindset.  This “goes behind what we can see and hear, and starts puzzling when an expected event fails to materialize.”

Comment

I have plenty of respect for Klein, but these are disappointing pieces.  They mainly just rehash anecdotes from his earlier work.  He says very little about how experience or an active mindset actually work to help us notice what’s missing, or how to achieve either of these things.  In fact “an active curious mindset” seems to be little more than a redescription of the ability to notice things – barely more satisfying than saying “pay attention!” or “look around for what’s missing!”.  In studying these pieces, I engaged an active curious mindset.  I noticed what was missing: anything of any great insight or use.  Which I know from experience is unusual in Klein’s case.

Advertisements

Meta-analysis has become an indispensable part of modern science.  By pooling data from many studies, and using special mathematical techniques, meta-analysis answers more questions, with more power and precision, than is possible either with single studies or informal reviews.

Currently, however, meta-analysis is a closed activity. It is performed by small, funded teams operating behind closed doors and with the results only becoming available in technical journal articles which are often stuck behind pay-walls. This closed approach has serious problems, as we discuss below.

Change is overdue, and indeed on its way.  There are increasing calls to make meta-analysis projects more accessible, transparent, collaborative, and frequently updated (“living”) – or in a word, more open.

What would truly open meta-analysis be like as a social practice?  How would it fit into other practices such as journal publication?  What technological support does it need?  How well could it work?

We are confident that meta-analysis would benefit greatly from being conducted far more openly than it typically is today. We draw inspiration from the way the wiki transformed encyclopedia production, and open-source transformed software production.  Meta-analysis is, to be sure, a technical matter, but so is writing an encyclopedia article about leukemia or producing an operating system.

We see many serious challenges, but no fundamental barrier to having crowds collaborate in posing useful questions, identifying suitable studies, extracting key data, selecting and applying analytical methods, and deriving insights from the results.

We use the term “open collaborative meta-analysis” (OCMA) to indicate that in this open alternative, MA projects welcome all comers to not only view the data and findings, but to also contribute their data, their labor and their insights, regardless of whether they be career scientists working on the topic, undergraduates learning the ropes, or interested members of the public.

What’s wrong with how meta-analysis is currently done?

Thousands of meta-analyses appear every year, many of high quality, profoundly contributing to scientific knowledge.  However the current approach has some serious problems, caused in part by its closed nature.

One is painfully apparent to anyone who has actually done a meta-analysis: they take a lot of tedious work. When this work is shouldered solely by a small, funded team, it is slow and expensive. This reduces the number of projects that get undertaken, and narrows their scope. Important issues, such as the role of moderators, are neglected, and information inherent in the vast pool of primary studies is under-exploited.

The current approach can also harm meta-analytic findings, because, first, there is no opportunity, during the analysis, for external critical review of the innumerable judgements the team must make, such as what risk of bias rating to give a particular study.  When teams have only themselves as critics, they are more likely to make clerical mistakes or technical errors. Sometimes they may even be tempted by dubious, self-serving choices.  Second, it is hard for small teams, with their limited resources and networks, to identify all the studies which meet their inclusion criteria.  A recent review has found biomedical meta-analyses to be “consistently incomplete” in their evidence base. Third, a MA project typically comes to a halt when the small team has drawn their conclusions and drafted their publications, even though new relevant studies continue to appear.  This means that a project’s findings can quickly go out of date.

Consequently, all too often a meta-analysis’s findings are limited, wrong or misleading, notwithstanding the competence and diligence of the small team behind it.

The situation gets worse when we consider sets of meta-analyses in a given area. The closed and competitive nature of the current approach means that teams are often unaware that other teams are addressing the same or very similar questions – or they are aware, but push on regardless.  The result is redundant analyses and even confusion when different analyses present overlapping and conflicting findings. The closed approach, with its lack of transparency in the meta-analytic process, hinders clarification of why these differences exist and how they might be corrected.

Finally, there is a growing movement towards synthesizing meta-analyses into even larger studies. This is difficult to do when meta-analyses themselves are so poorly disclosed.

What is the open alternative?

The essence of OCMA is that meta-analyses are conducted as public collaborations.  Anyone can initiate a MA, and anyone can contribute to an existing project, in a range of ways. Projects are ongoing; they evolve over time as more studies become available, problems are corrected, analytical methods improve, and new questions are asked.

In this way meta-analysis projects benefit from much broader input than is possible in the standard approach, and both the scientific community and the public benefit from projects and their findings being so easily accessible, correctable, and continually updated.

A well-designed online platform will be needed for OCMA to work.  Since an online platform is, in one sense, just software code running on servers, the platform can itself become an open development project. Similarly, OCMA as a scientific practice, with its workflow, norms, roles, and sanctions, can be governed by by the community of users, much as the practices of open encyclopedia production are governed by the Wikipedia community.

OCMA is very general, applicable in any area of science.  We envisage a single platform capable of supporting analyses not just in biomedical science but in education and many other fields, though it may be more practical to have a number of specialised OCMA platforms.

OCMA changes the way people come together to share and collaborate, not the theory of meta-analysis. OCMA processes and platforms would support whatever range of statistical methods the scientific community deems appropriate.

Is there anything like this out there already?

OCMA, as we conceive it, does not yet exist.  There have been important developments pushing in broadly similar directions, but they all lack one or more key ingredients of true open, collaborative meta-analysis.  Space does not allow exhaustive comparisons, but we can illustrate with reference to some of the most comparable efforts:

  • openMetaAnalysis is biomedicine-specific, limited in functionality, and not easy to use.
  • Covidence presents the kind of platform interface quality OCMA needs, but was designed for traditional small-groups, and only supports gathering and coding of studies, not the full analytic process.
  • metaBUS makes results relatively easily accessible to the public, but depends on “curation” work by a cadre of technical specialists and currently at least is restricted to the field of human resource management.
  • The Systematic Review Data Repository makes data sets and systematic reviews available, but does not support open collaboration in the meta-analysis process
  • Live cumulative network meta-analysis is as yet only a concept, and focuses a very technical form of biomedical meta-analysis; it would not be suitable for the vast majority of meta-analysis projects.

More generally, as compared with existing developments, OCMA is to varying degrees more crowd-oriented, collaborative, widely applicable across scientific fields and user-friendly.

But have you thought of…?

OCMA faces formidable challenges. We describe some here, and sketch some possible solutions.  However we recognise that these are difficult problems, and that our prototyping exercise may well throw up many new ones.

Why would anyone bother contributing?

There are many different kinds of motivation for participating in open projects such as open encyclopaedias, open science projects, and open software development.  Different people would contribute to OCMA for their own mix of reasons.

For example, a researcher may want to put her MA-in-progress up on the open platform in order to gain the benefits of crowd involvement, such as contributions of labour, and double-checking of judgements.  Authors of relevant studies will often be motivated to ensure that their studies are included and treated appropriately. Other researchers may want to participate through interest, collegiality, and concern for correctness.

An important challenge is to allow researchers to get recognition for their contributions to open projects.  This problem has already arisen in other contexts of open knowledge production. OCMA would need to include mechanisms for reliably documenting and perhaps even assessing a researcher’s contributions. In parallel, the wider scientific community would need to evolve ways of accepting such documentation in performance evaluations.

Would OCMA analyses get published?  How?

At least initially, OCMA would not affect how meta-analytic results get published.  OCMA might support standard publications as follows. An OCMA platform would enable a researcher to “freeze” an instance of a suitably-developed project.  The researcher can take its findings and present them in an article which is then subject to a journal’s normal review processes and standards. The OCMA community is acknowledged as a kind of contributor, but the researcher takes final responsibility for completeness and correctness.

In the longer term, OCMA may give rise to an alternative to standard publishing for meta-analyses. There is currently much dissatisfaction with scientific publishing, and many people are exploring ways to improve or sidestep the standard journal publication processes.  A critical issue is how research gets authorised or endorsed by the scientific community.  We envisage that OCMA would develop practices, supported by the OCMA platform, for indicating when projects are sufficiently well-developed that they are at least as, if not more, authoritative than traditional journal publications.  These practices may be supported by rigorous quality tests comparing OCMA analyses with suitable benchmarks such as Cochrane reviews on the same topics.

Nuisance

An obvious problem is that if anyone can come in make changes, then malicious users could vandalise projects; users with vested interests may try to manipulate findings; and well-meaning users might just “stuff things up.”

This is a version of a standard early objection to Wikipedia. However Wikipedia has proven that nuisance is manageable. It has developed a range of responses, including the ability to revert changes; page watchlists; blocking vandals; and clean-up bots. Such methods can also be used in OCMA.

Also, an OCMA site will be less likely to attract nuisance users. The OCMA site would be relatively dry and technical in nature, and have a relatively small community of viewers and editors. We expect that a sufficiently large proportion of visitors will be reasonably competent and well-intentioned that nuisance could be managed adequately.

Stability and Customization

A key challenge will be to reconcile two apparently conflicting requirements.  On one hand, the whole idea is that projects continually evolve as users make incremental changes, or do exploratory “what ifs.”  On the other, users will want the project to remain fixed or stable for various purposes such as publishing findings.  One technical solution may be to enable signed-in users to save a configuration for a particular project.  A visitor can then view either the master version, one of their saved configurations, or one which has been shared with them by somebody else.

 


Thanks to the following for their input into this document:

  • Professor Robert Badgett, Preventive Medicine and Public Health, University of Kansas School of Medicine
  • Professor John Hattie, Director, Melbourne Education Research Institute, University of Melbourne
  • Professor Julian Elliott, Head of Clinical Research, Infectious Diseases, Alfred Hospital and Monash University; Senior Research Fellow at the Australasian Cochrane Centre
  • Dr. Charles Twardy, Senior Data Scientist, NTVI; Affiliate Professor, George Mason University

All Wikipedia Roads Lead to Philosophy is a brief discussion of the initially surprising fact that if you click the first link in any Wikipedia article, you’ll eventually arrive on the page for Philosophy.  It is worth trying yourself to experience why this happens: most first sentences on Wikipedia pages relate the page topic to some larger topic, e.g. “Geranium argenteum (silvery crane’s bill) is an ornamental plant…”.   This is a very cute way of revealing something important about how knowledge is organised, and how we explain things to each other.

Eliezer Yudkowsky claims that Nate Silver erred when calculating that the probability of Trump getting the nomination was 2%.  Silver’s calculation was based on Trump’s needing to pass through six stages and there was only 50% chance of passing each stage.  Yudkowsky believes that Silver should have used the conditional probability of passing each stage given that Trump had passed the previous stages.  For the sixth stage, for example, the probability that he would pass that stage may well be judged much higher than 50%, given that he had succeed in five previous stages.
space
Yudkowsky’s analysis seems relevant to explaining why people – allegedly – commit a basic error of probabilistic judgement, which is to fail to multiply the probabilities of a chain of independent events and hence to overestimate the probability that all events occur.  A standard illustration of this is (as I recall) something like the 10 lock problem.  A safe has 10 locks.  The burglar has a 90% chance of picking each lock.  What is the probability he breaks the safe?  .9^10 = approx .34, and apparently people tend estimate a much higher figure.  This might be explained, in a handwavy way, by saying they anchor on .9, and fail to sufficiently adjust.
 space
However, a more “ecological” approach might seek to understand people’s judgements in terms of how events in fact unfold in the “real” world, or the world they evolved in.  While it is possible to artificially define a situation in which the probability of cracking each lock is, by stipulation, .9, what would happen in the real world is that if you watched somebody trying to crack a safe, and they’d cracked 9 of 10 locks already, you’d think that the safebreaker is so good that they are almost certain to crack the last one.  In other words, you would – in a somewhat Bayesian manner – update your estimate of the safebreaker’s skill as each lock is cracked, and hence the probability of cracking the remaining locks.
 space
When a mathematically unsophisticated person is asked for their answer to the 10 safe problem, it is plausible (this is a testable conjecture) that they imagine the burglar starting with the first lock, probably cracking that, proceeding to the next lock, and so on.  (It seems unlikely they would mentally simulate the entire sequence.)  We know that decision making by mental simulation is a very common strategy (Klein).  This is not quite decision making, but it is similar; the RPD perspective suggests the decision maker mentally simulates one approach to see if it is likely to work, and similarly the naive subject may (start to) mentally simulate the sequence of lock cracking.
 space
This suggests two things.  First, people’s higher than “normative” estimate might be explained by their (in some vague sense) conditionalising the probabilities.  They intuitively judge that the chance of cracking the later locks in the sequence is greater than .9.  Second, depending on the environments they are normally in, this might be the right thing to do.  Or, put another way, the “fallacy” is to resist the idea of purely independent events, and to be quasi Bayesian.  Maybe they are being more rational, in some ecological sense, than the smarty-pants psychologists who try to trip them up.
space
space
space
space

The Only Rule Is It Has To Work is a great story about a minor league baseball team being taken over by two sabermetricians. It addresses one of the big questions of our time: to what extent can statistics and data science replace intuitive human judgement?  This is obviously Moneyball territory, but if it was just a repeat of that great book and movie, it wouldn’t be all that interesting. I won’t spoil the story, but there’s an important lesson here, and its more about communication than it is about analytics. For the quick version, see the author’s article in the New York Times, What Happens When Baseball-Stats Nerds Run a Pro Team?

Anyone familiar with this blog knows that it frequently talks about argument mapping.  This is because, as an applied epistemologist, I’m interested in how we know things.  Often, knowledge is a matter of arguments and evidence.  However, argumentation can get very complicated.  Argument mapping helps our minds cope with that complexity by providing (relatively) simple diagrams.

Often what we are seeking knowledge about is the way the world works, i.e. its causal structure.  This too can be very complex, and so its an obvious idea that “causal mapping” – diagramming causal structure – might help in much the same way as argument mapping.  And indeed various kinds of causal diagrams are already widely used for this reason.

What follows is a reflection on explanation, causation, and causal diagramming.  It uses as a springboard a recent post on blog of the Lowy Institute which offered a causal explanation of the popularity of Russian president Putin.  It also introduces what appears to be a new term – “causal storyboard” – for a particular kind of causal map.


 

In a recent blog post with the ambitious title “Putin’s Popularity Explained,” Matthew Dal Santo argues that Putin’s popularity is not, as many think, due to brainwashing by Russia’s state-controlled media, but to the alignment between Putin’s conservative policies and the conservative yearnings of the Russian public.

Dal Santo dismisses the brainwashing hypothesis on very thin grounds, offering us only “Tellingly, only 34% of Russians say they trust the media.” However professed trust is only weakly related to actual trust. Australians in surveys almost universally claim to distrust car salesmen, but still place a lot of trust in them when buying a car.

In fact, Dal Santo’s case against the brainwashing account seems to be less a matter of direct evidence than “either or” reasoning: Putin’s popularity is explained by the conservatism of the public, so it is not explained by brainwashing.

He does not explicitly endorse such a simple model of causal explanation, but he doesn’t reject it either, and it seems to capture the tenor of the post.

The post does contain a flurry of interesting numbers, quotes and speculations, and these can distract us from difficult questions of explanatory adequacy.

The causal story Dal Santo rejects might be diagrammed like this:

putin1

The dashed lines indicate the parts of the story he thinks are not true, or at least exaggerated. Instead, he prefers something like:

putin2
However the true causal story might look more like this:

putin3.jpg

Here Putin’s popularity is partly the result of brainwashing by a government-controlled media, and partly due to “the coincidence of government policies and public opinion.”

The relative thickness of the causal links indicate differing degrees to which the causal factors are responsible. Often the hardest part of causal explanation is not ruling factors in or out, but estimating the extent to which they contribute to the outcomes of interest.

Note also the link suggesting that a government-controlled media might be responsible, in part, for the conservatism of the public. Dal Santos doesn’t explicitly address this possibility but does note that certain attitudes have remained largely unchanged since 1996. This lack of change might be taken to suggest that the media is not influencing public conservatism. However it might also be the dog that isn’t barking. One of the more difficult aspects of identifying and assessing causal relationships is thinking counterfactually. If the media had been free and open, perhaps the Russian public would have become much less conservative. The government-controlled media may have been effective in counteracting that trend.

The graphics above are examples of what I’ve started calling causal storyboards. (Surprisingly, at time of writing this phrase turns up zero results on a Google search.) Such diagrams represent webs of events and states and their causal dependencies – crudely, “what caused what.”

For aficionados, causal storyboards are not causal loop diagrams or cognitive maps or system models, all of which represent variables and their causal relationships.  Causal loop diagrams and their kin describe general causal structure which might govern many different causal histories depending on initial conditions and exogenous inputs.  A causal storyboard depicts a particular (actual or possible) causal history – the “chain” of states and events.  It is an aid for somebody who is trying to understand and reason about a complex situation, not a precursor to quantitative model.

Our emerging causal storyboard surely does not yet capture the full causal history behind Putin’s popularity. For example it does not incorporate any additional factors, such as his reputed charisma. Nor does it trace the causal pathways very far back. To fully understand Putin’s popularity, we need to know why (not merely that) the Russian public is so conservative.

The causal history may become very complex. In his 2002 book Friendly Fire, Scott Snook attempts to undercover all the antecedents of a tragic incident in 1994 when two US fighter jets shot down two US Army helicopters. There were dozens of factors, intricately interconnected. To help us appreciate and understand this complexity, Snook produced a compact causal storyboard:

148914-151707.png

To fully explain is to delineate causal history as comprehensively and accurately as possible. However, full explanations in this sense are often not available. Even when they are, they may be too complex and detailed. We often need to zero in on some aspect of the causal situation which is particularly unusual, salient, or important.

There is thus a derivative or simplified notion of explanation in which we highlight some particular causal factor, or small number of factors, as “the” cause. The Challenger explosion was caused by O-ring leaks. The cause of Tony Abbott’s fall was his low polling figures.

As Runde and de Rond point out, explanation in this sense is a pragmatic business. The appropriate choice of cause depends on what is being explained, to whom, by who, and to what purpose.

In an insightful discussion of Scott Snook’s work, Gary Klein suggests that we should focus on two dimensions: a causal factor’s impact, and the ease with which that factor might have been negated, or could be negated in future. He uses the term “causal landscape” for a causal storyboard analysed using these factors. He says: “The causal landscape is a hybrid explanatory form that attempts to get the best of both worlds. It portrays the complex range and interconnection of causes and identifies a few of the most important causes. Without reducing some of the complexity we’d be confused about how to act.”

This all suggests that causes and explanations are not always the same thing. It can make sense to say that an event is caused by some factor, but not fully explained by that factor. O-ring failure caused the Challenger explosion, but only partially explains it.

More broadly, it suggests a certain kind of anti-realism about causes. The world and all its causal complexity may be objectively real, but causes – what we focus on when providing brief explanations – are in significant measure up to us. Causes are negotiated as much as they are discovered.

What does this imply for how we should evaluate succinct causal explanations such as Dal Santo’s? Two recommendations come to mind.

First, a proposed cause might be ill-chosen because it has been selected from underdeveloped causal history. To determine whether we should go along, we should try to understand the full causal context – a causal storyboard may be useful for this – and why the proposed factor has been selected as the cause.

Second, we should be aware that causal explanation can itself be a political act. Smoking-related lung cancer might be said to be caused by tobacco companies, by cigarette smoke, or by smoker’s free choices, depending on who is doing the explaining, to whom, and why. Causal explanation seems like the uncovering of facts, but it may equally be the revealing of agendas.

A colleague Todd Sears recently wrote:

I thought I’d write to let you know that I used an argument map last night to inform a public conversation about whether to change our school budget voting system from what it is (one meeting and you have to be physically present to vote), to the (of all things!) Australian Ballot system (secret ballot, polls open all day, and absentee ballots available).

So, I went through the articles, editorials, and opinion pieces I could find on the matter and collapsed those into a pretty simple argument, which it is. Simple reasoning boxes get the job done.  Our voters had never really seen this kind of visualization.  It’s nice to be able to see an argument exist in space, and to demonstrate by pointing and framing that a “yea” vote needs to buy into the green points, but also that they need to reconcile the red points, somehow. It had very good response.

Ultimately, the AB motion was defeated by five votes.  Still, it was a good example of a calm, reasonable, and civil dialogue.  A nice change from the typical vitriol and partisan sniping.

Here is his map (click to view full size version):

Bethelmap

When I suggested that readers of this blog might find his account interesting or useful, he added:

Let me clarify what I did because it wasn’t a classic facilitation.

1. I reviewed all of the on-line Vermont-centric AB content I could find in the more reputable news sources, and put a specific emphasis on getting the viewpoints of the more vociferous anti-AB folks in my town so that I could fairly represent them.

2. I created a map from that information and structured it in a way that spread out the lines of reasoning in an easily understandable way. I could have done some further abstraction and restructured things, or made assumptions explicit using the “Advanced” mode, but chose to focus on easily recognized reasoning chains.

3. I sent the map out to the entire school board, the administrators, a couple of politicians, the anti-AB folks and some of the other more politically engaged people in town.

4. The session was moderated by the town Moderator, who set out Robert’s Rules of Order. Then discussion began. In fact, the first ant-AB speaker had my map in his hand and acknowledged the balance and strength of both sides of the argument.

5. I let the session run its course, and then explained what I did and how I did it, and then reviewed the Green and Red lines of the debate, explaining that a vote for or against means that the due diligence has to be done in addressing the points counter to your own position, and I demonstrated how this should be done. Though I was in favor of AB, I maintained objectivity and balance, rather than a position of advocacy one way or another.

Overall the session was very civil, informed, and not one point was made (myriad rhetorical flourishes aside) that was not already on the map. Many variations on similar themes, but nothing that hadn’t been captured.

And followed up with:

BTW, just 30 minutes ago I received an e-mail which said this:

Hi,

I love the map of the issues around Australian Ballot that you sent out. Is there an easy way to make such a map? We are tackling some issues that have our faculty at Randolph Union High School pretty evenly split and I think two such maps would be a powerful way for my colleague and I who are leading this change to communicate. It looks as if it was created in PowerPoint. If you are too busy to elaborate that’s fine too.

Thanks for your leadership on the Australian Ballot issue. I appreciate it.