Conditions

Understanding evidence quality

The basics

If evidence is rated as LOW quality, this does not mean it is totally unreliable!

For practical purposes, the spectrum of evidence quality going down from HIGH to MODERATE to LOW tells us that there is increasing uncertainty about the size of the treatment effect (i.e., there could be more or less benefit). LOW doesn’t usually mean that the treatment is unlikely to work or that we should not work with this evidence.

VERY LOW QUALITY evidence might well mean there is genuine uncertainty about whether a treatment works. Where this is the case, we give further information to help clarify.

How is evidence quality appraised on GP Evidence?

more

We use the GRADE system of evidence quality rating1,2, which is used by NICE and Cochrane among others.

Where NICE or Cochrane have performed a GRADE rating, this is reported here on this website. Where a GRADE rating has not been done, we perform our own assessment according to the GRADE structure.

GRADE provides a structured way to assess evidence quality. Though it is designed to be objective and reproducible, there is a degree of subjectivity involved in making judgements.

How does it work?

In a nutshell:

Randomised Controlled Trials and Meta-analyses are regarded as HIGH quality evidence, but can be downgraded for a variety of reasons.

Observational Research is initially rated as LOW quality, but can be upgraded for a variety of reasons.

Factors which can lead to a downgrading of evidence quality are:

  • Risk of bias in study design
    • inadequate randomisation, allocation concealment or blinding of treatments
    • poorly matched comparison groups
    • inadequate blinding or uncertain assessment of outcome measurement
    • high levels of loss to follow-up
    • bias in outcome detection or reporting
    • other sources of bias, e.g., publication bias
  • Inconsistency of evidence (heterogeneity)
    • where different trials within one systematic review give varying estimates of treatment effect
  • Poor applicability of evidence (indirectness)
    • where the evidence in the trials is derived from a population significantly different from the people to whom it will be applied
  • Imprecision
    • wide confidence intervals around the estimate of effect

Factors which can lead up an upgrading of observational research are:

  • A large treatment effect
  • A dose-response effect
    • where greater benefit is observed with greater doses
  • Where plausible biases (i.e., those which are expected), would overall tend to reduce the estimate of treatment effect, but one is still seen despite them.

 

References and links

1)GRADE working group website. Contains explanations, tutorials, links to literature.

2)Goldet G and Howick J. Understanding GRADE: an introduction. Journal of Evidence-Based Medicine 2013; 6: 50-54.