Library Guides: Evidence-Based Practice: What is Being Appraised?

Elements of critical appraisal

Students are often asked to evaluate resources using the CRAP Test: Currency, Reliability, Authority, and Purpose.

But in the Evidence-Based Practice process we need to go beyond that to also critically appraise certain aspects of the evidence. Even when we have found a study that is the right sort of study for our clinical question, and is of the appropriate level in the hierarchy, we still need to question its relevance, validity, and conclusions. We need to ask:

Does the study address a clearly focused question?
Did the study use valid methods to address the question? (How well was the study done?)
What do the results mean and are they statistically valid (and not just due to chance)?
Are the (valid) results of the study important?
Can I apply these (valid and important) results to my patient/population?

The CASP (Critical Appraisal Skills Program) website puts this another way. It talks about the need to evaluate ...

relevance
trustworthiness
results.

... but it also emphasises that the process of, and tools for, critical appraisal might vary according to the type of study/research.

Relevance

The best way to gauge the relevance of a primary study is to refer to your PICO. Remember that PICO stands for:

Patient / Population / Problem
Intervention
Comparison
Outcome.

You will rarely get an exact match, so you need to decide if the study matches closely enough to the elements of your PICO to assist with your clinical decision.

Trustworthiness (Bias and Confounding Factors)

The quality of primary research often depends on the extent to which the methodology deals with bias and confounding factors:

Bias

"Bias is the degree to which the result is skewed away from the truth" . Bias can occur in many ways, including: how subjects are selected; how they are allocated to groups; and how results are measured. The best way to prevent bias is to keep details of interventions hidden from participants and also researchers (Glasziou, Salisbury, & Del Mar, 2007, p. 75). This is called blinding.

Confounding factors

"Confounding factors are patient features and other possible causal factors, apart from the one that is being measured, that can affect the outcome of the study." These can be eliminated or reduced by ensuring that groups are matched as closely as possible at the start of the study, and managed in the same way in the course of the study (Glasziou et al., 2007, p. 76).

RAMMbo

One way to quickly appraise a study's trustworthiness is to use the RAMMbo mnemonic:

R	Recruitment	Were the subjects in the study representative of the target population? Were there enough subjects to make the study valid?
A	Allocation	Were the subjects randomly (and "blindly") allocated to groups?
M	Maintenance	Was the status of the study group and the control group maintained throughout the trial? Were they treated the same way apart from the intervention?
Mbo	Measurement (blinding; objective measures)	Were the subjects (and researchers) blinded to the intervention and were the outcomes measured objectively? Was bias eliminated as much as possible?

Source: Glasziou, P., Del Mar, C., & Salisbury, J. (2007). Evidence-based practice workbook: Bridging the gap between health care research and practice (2nd ed.).

Results (Meaning and Value)

Questions to ask here include:

Were statistical tests applied and did they result in a significant impact from the intervention?
Could the result have been due to chance, or to something else in the study which was not controlled?
What does the study show? What does it mean and what is its value?
What do the results mean for a particular context in which a decision is being made?