Systematic reviews

Selecting relevant studies

The following steps should be done when selecting relevant studies for inclusion into your review:

  1. De-duplicate citations and pilot test eligibility criteria
  2. Screen titles and abstracts identified via searches using eligibility criteria
  3. Obtain full-text articles for all potentially relevant studies
  4. Select full-text articles for inclusion in a review using eligibility criteria
  5. Report results of the selection process based in the PRISMA statement

Source

Appraising studies

Key steps to consider when appraising studies include:

  1. Note the type of study design in included studies based on a review question
  2. Identify a valid quality assessment tool appropriate for a review
  3. Two or more reviewers test the tool with a sample of included studies (for inter-rater reliability)
  4. Reviewers carryout critical appraisal of included studies independently and in duplicate
  5. Tabulate and summarize results of critical appraisal

Critical appraisal tools
The following table presents a selection of appraisal tools (this is not meant as an exhaustive list).

Study design Critical appraisal tool
Systematic reviews AMSTAR
Assessment of Multiple Systematic Reviews (AMSTAR) is a 37-item assessment tool used to assess the methodological quality of systematic reviews.
Randomized control trials (RCTs)

CONSORT
The Consolidated Standards of Reporting Trials (CONSORT) Statement is a detailed document which outlines an explanation and elaboration of the CONSORT statement for reporting randomized controlled trials.

Cochrane Risk of Bias Tool
The Cochrane Risk of Bias Tool is used by the Cochrane Collaboration to assess the risk of bias for randomized controlled trials. Bias is assessed as a judgment (high, low, or unclear) for individual elements from five domains (selection, performance, attrition, reporting and other). 
Non-randomized control trials TREND statement (PDF)
The TREND statement complements the widely adopted CONSORT statement developed for randomized controlled trials.

Ottawa-Newcastle Scale
The Newcastle-Ottawa Scale (NOS) was developed to assess the quality of non-randomized studies with its design, content and ease of use directed to the task of incorporating the quality assessments in the interpretation of meta-analytic results. The goal of this project is to develop an instrument providing an easy and convenient tool for quality assessment of non-randomized studies to be used in a systematic review.
Observational studies

STROBE
Observational studies in epidemiology (cohort, case-control studies and cross-sectional studies). 
Cohort studies CASP: Cohort Studies (PDF)
Critical Appraisal Skills Program (CASP): Cohort Studies is a methodological checklist which provides key criteria relevant to cohort studies.
Case control studies CASP: Case Control Studies (PDF)
Critical Appraisal Skills Program (CASP): Case Control Studies is a methodological checklist which provides key criteria relevant to case control studies.
Clinical practice guidelines

AGREE II
The Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument assesses both the quality of the reporting, and the quality of some aspects of recommendations. It provides an assessment of the predicted validity of a guideline, that is the likelihood that it will achieve its intended outcome. It does not assess the impact of a guideline on patients outcomes.

GRADE
The Grading of Recommendations of Assessment, Development and Evaluation (GRADE) is a common, sensible and transparent approach to grading quality (or certainty) of evidence and strength of recommendations. 
Qualitative studies CASP: Qualitative Research (PDF)
Critical Appraisal Skills Program (CASP): Qualitative Research is a methodological checklist which provides key criteria relevant to qualitative research studies.

McMaster Critical Review Form - Qualitative Studies (PDF)
McMaster Critical Review Form for Qualitative studies contains a generic quantitative appraisal tool, accompanied by detailed guidelines for usage.

For more critical appraisal tools, see the following resources:

Considering systematic review software

There are different types of systematic review software that you can consider, depending on the type of review you are conducting.

  • Covidence​
    Covidence imports references, removes duplicates, screens full text, and allows you to crate data extraction forms. It also supports screening on mobile devices, and is recommended by Cochrane.
    The Library has a subscription to Covidence. Consult our Covidence research guide to learn more.
  • DistillerSR
    ​    DistillerSR systematic review software manages, tracks, and streamlines the screening, data extraction, and reporting processes of your systematic reviews and literature reviews, letting you focus on delivering the best possible evidence-based research, faster.
  • EPPI Reviewer 4
    EPPI-Reviewer 4 has the functionality to help manage your systematic review through all stages of the process from bibliographic management, screening, coding and right through to synthesis.
  • Rayyan
    Rayyan is a 100% free Web application to help systematic review authors perform their job in a quick and easy fashion. Authors create systematic reviews, collaborate on them, maintain them over time and get suggestions for article inclusion. There is also a mobile app available.
  • Systematic Review Data Repository
    SRDR is a powerful and easy-to-use tool for the extraction and management of data for systematic review or meta-analysis. It is also an open and searchable archive of systematic reviews and their data.

Contact

For more information, please contact:

Systematic Review Toolbox

The SR ToolBox aims to help reviewers find appropriate software tools based on how they provide support for the systematic review process.