Now that you’ve chosen the studies to include in your review you need to check if they have followed sound scientific principles and the outcomes are valid.

Critical appraisal has been described as "the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context" (Burls, 2009).

• Has the research been conducted in a way that minimises bias? (Is it trustworthy?)

• If so, what does the study show? (What is its value?)

• What do the results mean for a particular context in which a decision is being made? (Is it relevant to your research question?)

Traditional literature reviews and scoping reviews require a fairly general critical analysis, but for a systematic review, each study will have to be analysed. That sounds difficult, but fortunately there have been a number of tools developed for different study types in different disciplines (mainly health) to guide you through the process of determining the validity and accuracy of the information in individual studies.

Not all the studies you have retrieved in your searches will have the same evidence rigor, and many researchers have developed tables to explain this. For example, systematic reviews and meta analyses rank very highly because they have synthesised a wide range of evidence. Case studies rank lowly because the results are in isolation from other evidence. Here are some examples of ranking the levels of evidence:

• Brian Haynes 6S Hierarchy of Evidence Based Resources (2009) - pictured at right and widely used in health research.
• Oxford Centre for Evidence-Based Medicine (OCEBM) (2011). Levels of Evidence
• Joanna Briggs Institute (JBI) (2013). Levels of Evidence 
• National Health and Medical Research Council (NHMRC). Levels of Evidence 

If you're not completing a full systematic review and need to quickly analyse the studies you've retrieved, here are some considerations:

• Relevance - Compare the study to your search framework (such as PICO).
• Results - Were statistical tests applied and did they indicate the findings were statistically significant?
• Applicability to your research question - Did the researchers contribute to an answer to your original question? Is your study group similar to this or quite different? 
• Quality of study – There are many tools out there for a rapid critical appraisal of the study quality. For example, one you could use if you were reviewing controlled clinical trials is the RAMMbo appraisal method (Salisbury, Glasziou, & Del Mar, 2007):

R - Recruitment – were the subjects chosen in the study representative of the target population? Were there enough subjects to make the study valid?

A - Allocation – was the trial randomised?   

M - Maintenance – Was the status of the control group and study group maintained throughout the trial?  Were they treated the same way apart from the intervention?  

M - Measurement (Blinding, Objective measures) – were the outcomes measured objectively and the subjects blinded to the intervention?  Was bias eliminated as much as possible?      

Salisbury, J., Glasziou, P., & Del Mar, C. (2007). Evidence-based practice workbook: Bridging the gap between health care research and practice (2nd ed.). Oxford: Blackwell/BMJ Books.

Another handy tool for quick appraisal of health articles is from the Medical Research Council in the UK. Understanding Health Research takes you through a series of questions about a particular article, highlighting the good points and possible problem areas. You can print off a pdf summary at the end of your checklist.

There are many useful tools to help you assess the value of various types of studies, and again, this aspect is strong in the health sciences.

• Buccheri and Sharifi (2017) outlined the uses, strengths and weaknesses of various appraisal tools for nursing research in their paper, Critical appraisal tools and reporting guidelines for evidence‐based practice. Worldviews on Evidence‐Based Nursing, 14(6), 463-472. https://doi.org/10.1111/wvn.12258
• The "How to read a paper" series from The BMJ links to articles that explain how to read and interpret different kinds of research.
• Trisha Greenhalgh's book, How to read a paper, is an extension of this, with the 2019 6th edition available online through the Library.

Critical appraisal tools from the NHS, Greater Glasgow and Clyde in Scotland is interactive and includes links to all sorts of resources on how to identify the study type and build your critical appraisal skills, as well as the tools themselves.

Other freely available critical appraisal tools include:

• Assessment of Multiple Systematic Reviews (AMSTAR) checklist
• Centre for Evidence-based Veterinary Medicine, University of Nottingham. Appraisal Tools
• COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN)
• Critical Appraisal Skills Programme (CASP) - Critical appraisal tools for Systematic Reviews, Randomised Controlled Trials, Cohort Studies, Case Control Studies, and more.
• Eco-Evidence - Includes analyser software you can download which helps in the evaluation of evidence from environmental science literature. You will need to register to use the tools.
• GATE (Graphic Appraisal Tool for Epidemiology)
• International Centre for Allied Health Evidence: Critical Appraisal Tools (University of South Australia)
• Joanna Briggs Institute - critical appraisal tools - A range of appraisal tools for all different study types.
• McMaster University critical review forms - Includes both qualitative and quantitative study designs especially for occupational therapy.
• PEDro scale - Physiotherapy evidence appraisal tool.
• Standard quality assessment criteria for evaluating primary research papers from a variety of fields - Canadian paper by Kmet, Lee & Cook.
• Center for Evidence-Based Management (CEBMa) - some checklists from this Centre based in The Netherlands.