Now that you’ve chosen the studies to include in your review you need to check if they have followed sound scientific principles and the outcomes are valid.
Critical appraisal has been described as "the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context" (Burls, 2009).
Has the research been conducted in a way that minimises bias? (Is it trustworthy?)
If so, what does the study show? (What is its value?)
What do the results mean for a particular context in which a decision is being made? (Is it relevant to your research question?)
Traditional literature reviews and scoping reviews require a fairly general critical analysis, but for a systematic review, each study will have to be analysed. That sounds difficult, but fortunately there have been a number of tools developed for different study types in different disciplines (mainly health) to guide you through the process of determining the validity and accuracy of the information in individual studies.
Not all the studies you have retrieved in your searches will have the same evidence rigor, and many researchers have developed tables to explain this. For example, systematic reviews and meta analyses rank very highly because they have synthesised a wide range of evidence. Case studies rank lowly because the results are in isolation from other evidence. Here are some examples of ranking the levels of evidence:
If you're not completing a full systematic review and need to quickly analyse the studies you've retrieved, here are some considerations:
R - Recruitment – were the subjects chosen in the study representative of the target population? Were there enough subjects to make the study valid?
A - Allocation – was the trial randomised?
M - Maintenance – Was the status of the control group and study group maintained throughout the trial? Were they treated the same way apart from the intervention?
M - Measurement (Blinding, Objective measures) – were the outcomes measured objectively and the subjects blinded to the intervention? Was bias eliminated as much as possible?
Salisbury, J., Glasziou, P., & Del Mar, C. (2007). Evidence-based practice workbook: Bridging the gap between health care research and practice (2nd ed.). Oxford: Blackwell/BMJ Books.
Another handy tool for quick appraisal of health articles is from the Medical Research Council in the UK. Understanding Health Research takes you through a series of questions about a particular article, highlighting the good points and possible problem areas. You can print off a pdf summary at the end of your checklist.
There are many useful tools to help you assess the value of various types of studies, and again, this aspect is strong in the health sciences.
Critical appraisal tools from the NHS, Greater Glasgow and Clyde in Scotland is interactive and includes links to all sorts of resources on how to identify the study type and build your critical appraisal skills, as well as the tools themselves.
Other freely available critical appraisal tools include:
Charles Sturt University acknowledges the traditional custodians of the lands on which its campuses are located, paying respect to Elders, both past and present, and extend that respect to all First Nations Peoples.
Charles Sturt University is an Australian University, TEQSA Provider Identification: PRV12018. CRICOS Provider: 00005F.