Meta-analysis has become an indispensable part of modern science.  By pooling data from many studies, and using special mathematical techniques, meta-analysis answers more questions, with more power and precision, than is possible either with single studies or informal reviews.

Currently, however, meta-analysis is a closed activity. It is performed by small, funded teams operating behind closed doors and with the results only becoming available in technical journal articles which are often stuck behind pay-walls. This closed approach has serious problems, as we discuss below.

Change is overdue, and indeed on its way.  There are increasing calls to make meta-analysis projects more accessible, transparent, collaborative, and frequently updated (“living”) – or in a word, more open.

What would truly open meta-analysis be like as a social practice?  How would it fit into other practices such as journal publication?  What technological support does it need?  How well could it work?

We are confident that meta-analysis would benefit greatly from being conducted far more openly than it typically is today. We draw inspiration from the way the wiki transformed encyclopedia production, and open-source transformed software production.  Meta-analysis is, to be sure, a technical matter, but so is writing an encyclopedia article about leukemia or producing an operating system.

We see many serious challenges, but no fundamental barrier to having crowds collaborate in posing useful questions, identifying suitable studies, extracting key data, selecting and applying analytical methods, and deriving insights from the results.

We use the term “open collaborative meta-analysis” (OCMA) to indicate that in this open alternative, MA projects welcome all comers to not only view the data and findings, but to also contribute their data, their labor and their insights, regardless of whether they be career scientists working on the topic, undergraduates learning the ropes, or interested members of the public.

What’s wrong with how meta-analysis is currently done?

Thousands of meta-analyses appear every year, many of high quality, profoundly contributing to scientific knowledge.  However the current approach has some serious problems, caused in part by its closed nature.

One is painfully apparent to anyone who has actually done a meta-analysis: they take a lot of tedious work. When this work is shouldered solely by a small, funded team, it is slow and expensive. This reduces the number of projects that get undertaken, and narrows their scope. Important issues, such as the role of moderators, are neglected, and information inherent in the vast pool of primary studies is under-exploited.

The current approach can also harm meta-analytic findings, because, first, there is no opportunity, during the analysis, for external critical review of the innumerable judgements the team must make, such as what risk of bias rating to give a particular study.  When teams have only themselves as critics, they are more likely to make clerical mistakes or technical errors. Sometimes they may even be tempted by dubious, self-serving choices.  Second, it is hard for small teams, with their limited resources and networks, to identify all the studies which meet their inclusion criteria.  A recent review has found biomedical meta-analyses to be “consistently incomplete” in their evidence base. Third, a MA project typically comes to a halt when the small team has drawn their conclusions and drafted their publications, even though new relevant studies continue to appear.  This means that a project’s findings can quickly go out of date.

Consequently, all too often a meta-analysis’s findings are limited, wrong or misleading, notwithstanding the competence and diligence of the small team behind it.

The situation gets worse when we consider sets of meta-analyses in a given area. The closed and competitive nature of the current approach means that teams are often unaware that other teams are addressing the same or very similar questions – or they are aware, but push on regardless.  The result is redundant analyses and even confusion when different analyses present overlapping and conflicting findings. The closed approach, with its lack of transparency in the meta-analytic process, hinders clarification of why these differences exist and how they might be corrected.

Finally, there is a growing movement towards synthesizing meta-analyses into even larger studies. This is difficult to do when meta-analyses themselves are so poorly disclosed.

What is the open alternative?

The essence of OCMA is that meta-analyses are conducted as public collaborations.  Anyone can initiate a MA, and anyone can contribute to an existing project, in a range of ways. Projects are ongoing; they evolve over time as more studies become available, problems are corrected, analytical methods improve, and new questions are asked.

In this way meta-analysis projects benefit from much broader input than is possible in the standard approach, and both the scientific community and the public benefit from projects and their findings being so easily accessible, correctable, and continually updated.

A well-designed online platform will be needed for OCMA to work.  Since an online platform is, in one sense, just software code running on servers, the platform can itself become an open development project. Similarly, OCMA as a scientific practice, with its workflow, norms, roles, and sanctions, can be governed by by the community of users, much as the practices of open encyclopedia production are governed by the Wikipedia community.

OCMA is very general, applicable in any area of science.  We envisage a single platform capable of supporting analyses not just in biomedical science but in education and many other fields, though it may be more practical to have a number of specialised OCMA platforms.

OCMA changes the way people come together to share and collaborate, not the theory of meta-analysis. OCMA processes and platforms would support whatever range of statistical methods the scientific community deems appropriate.

Is there anything like this out there already?

OCMA, as we conceive it, does not yet exist.  There have been important developments pushing in broadly similar directions, but they all lack one or more key ingredients of true open, collaborative meta-analysis.  Space does not allow exhaustive comparisons, but we can illustrate with reference to some of the most comparable efforts:

  • openMetaAnalysis is biomedicine-specific, limited in functionality, and not easy to use.
  • Covidence presents the kind of platform interface quality OCMA needs, but was designed for traditional small-groups, and only supports gathering and coding of studies, not the full analytic process.
  • metaBUS makes results relatively easily accessible to the public, but depends on “curation” work by a cadre of technical specialists and currently at least is restricted to the field of human resource management.
  • The Systematic Review Data Repository makes data sets and systematic reviews available, but does not support open collaboration in the meta-analysis process
  • Live cumulative network meta-analysis is as yet only a concept, and focuses a very technical form of biomedical meta-analysis; it would not be suitable for the vast majority of meta-analysis projects.

More generally, as compared with existing developments, OCMA is to varying degrees more crowd-oriented, collaborative, widely applicable across scientific fields and user-friendly.

But have you thought of…?

OCMA faces formidable challenges. We describe some here, and sketch some possible solutions.  However we recognise that these are difficult problems, and that our prototyping exercise may well throw up many new ones.

Why would anyone bother contributing?

There are many different kinds of motivation for participating in open projects such as open encyclopaedias, open science projects, and open software development.  Different people would contribute to OCMA for their own mix of reasons.

For example, a researcher may want to put her MA-in-progress up on the open platform in order to gain the benefits of crowd involvement, such as contributions of labour, and double-checking of judgements.  Authors of relevant studies will often be motivated to ensure that their studies are included and treated appropriately. Other researchers may want to participate through interest, collegiality, and concern for correctness.

An important challenge is to allow researchers to get recognition for their contributions to open projects.  This problem has already arisen in other contexts of open knowledge production. OCMA would need to include mechanisms for reliably documenting and perhaps even assessing a researcher’s contributions. In parallel, the wider scientific community would need to evolve ways of accepting such documentation in performance evaluations.

Would OCMA analyses get published?  How?

At least initially, OCMA would not affect how meta-analytic results get published.  OCMA might support standard publications as follows. An OCMA platform would enable a researcher to “freeze” an instance of a suitably-developed project.  The researcher can take its findings and present them in an article which is then subject to a journal’s normal review processes and standards. The OCMA community is acknowledged as a kind of contributor, but the researcher takes final responsibility for completeness and correctness.

In the longer term, OCMA may give rise to an alternative to standard publishing for meta-analyses. There is currently much dissatisfaction with scientific publishing, and many people are exploring ways to improve or sidestep the standard journal publication processes.  A critical issue is how research gets authorised or endorsed by the scientific community.  We envisage that OCMA would develop practices, supported by the OCMA platform, for indicating when projects are sufficiently well-developed that they are at least as, if not more, authoritative than traditional journal publications.  These practices may be supported by rigorous quality tests comparing OCMA analyses with suitable benchmarks such as Cochrane reviews on the same topics.

Nuisance

An obvious problem is that if anyone can come in make changes, then malicious users could vandalise projects; users with vested interests may try to manipulate findings; and well-meaning users might just “stuff things up.”

This is a version of a standard early objection to Wikipedia. However Wikipedia has proven that nuisance is manageable. It has developed a range of responses, including the ability to revert changes; page watchlists; blocking vandals; and clean-up bots. Such methods can also be used in OCMA.

Also, an OCMA site will be less likely to attract nuisance users. The OCMA site would be relatively dry and technical in nature, and have a relatively small community of viewers and editors. We expect that a sufficiently large proportion of visitors will be reasonably competent and well-intentioned that nuisance could be managed adequately.

Stability and Customization

A key challenge will be to reconcile two apparently conflicting requirements.  On one hand, the whole idea is that projects continually evolve as users make incremental changes, or do exploratory “what ifs.”  On the other, users will want the project to remain fixed or stable for various purposes such as publishing findings.  One technical solution may be to enable signed-in users to save a configuration for a particular project.  A visitor can then view either the master version, one of their saved configurations, or one which has been shared with them by somebody else.

 

Thanks to the following for their input into this document:

  • Professor Robert Badgett, Preventive Medicine and Public Health, University of Kansas School of Medicine
  • Professor John Hattie, Director, Melbourne Education Research Institute, University of Melbourne
  • Professor Julian Elliott, Head of Clinical Research, Infectious Diseases, Alfred Hospital and Monash University; Senior Research Fellow at the Australasian Cochrane Centre
  • Dr. Charles Twardy, Senior Data Scientist, NTVI; Affiliate Professor, George Mason University