NHMRC additional levels of evidence and grades for recommendations STAGE 2 CONSULTATION
NHMRC additional levels of evidence and grades for recommendations
for developers of guidelines
STAGE 2 CONSULTATION
Early 2008 – end June 2009
Introduction
The National Health and Medical Research Council (NHMRC) in Australia has, over recent years,
developed a suite of handbooks to support organisations involved in the development of evidence-
ased clinical practice guidelines
(www.nhmrc.gov.au/publications/synopses/cp65syn.htm).
Reflecting the general impetus of the past decade, these handbooks focus predominantly on
assessing the clinical evidence for interventions. As a consequence, the handbooks present ‘levels
of evidence’ appropriate mainly for intervention studies. However, feedback from guideline
developers received by the NHMRC has indicated that the levels of evidence used by the
NHMRC for intervention studies have been found to be restrictive. This is particularly so where
the areas of study do not lend themselves to research designs appropriate to intervention studies
(i.e. randomised controlled trials).
As an interim measure to a review of the handbooks, this paper presents a forward-thinking
approach to classifying levels of evidence, and grading evidence recommendations, which should
e relevant to any clinical guideline (not just those dealing with interventions).
The NHMRC is committed to addressing this issue and any other relevant concerns when all of the
handbooks are reviewed.
This consultation draft has been developed based on a Pilot Program on ‘NHMRC additional
levels of evidence and grades for recommendations for developers of guidelines’, which was
initially released for public consultation in 2005, until mid-2006 with feedback sought until
30 June 2007 on their usability and applicability.
Levels of evidence
Guidelines can have different purposes, dealing with clinical questions such as intervention,
diagnosis, prognosis, aetiology and screening. To address these clinical questions adequately,
guideline developers need to include different research designs. This consequently requires different
evidence hierarchies that recognise the importance of research designs relevant to the purpose of the
guideline. A new evidence hierarchy was subsequently developed in 2005 by the NHMRC Guideline
Assessment Register (GAR) consultants. This hierarchy assigns levels of evidence according to the
1
type of research question, recognising the importance of appropriate research design to that question.
As well as the cu
ent NHMRC levels of evidence for interventions, new levels have been developed
for studies relevant for guidelines on diagnosis, prognosis, aetiology and screening.
This consultation framework outlines the expanded levels of evidence, and provides additional
information in the form of table notes, a study design glossary and a summary of how the levels of
evidence and other NHMRC dimensions of evidence should be used (see Part A).
Grades of recommendations
To further assist guideline developers to make judgments on the basis of the body of evidence
elevant to a research question, a grading system for recommendations has been developed (see
Part B). This takes the form of an evidence matrix, which lists the evidence components that should
e considered when judging the body of evidence. The grade of a recommendation is based on an
overall assessment of the rating of individual components of the evidence matrix.
Feedback
A draft of the revised levels of evidence and new grading system for recommendations was posted
on the NHMRC website from 2005 until mid-2007, with feedback being sought internationally on
their usability and applicability. Several guideline development teams, with guidance from their
GAR consultant, tested the revised levels and grades of evidence in guidelines that were developed
during the pilot period. The website feedback and the practical experience of guideline developers
support the clinical utility and academic rigour of the new levels of evidence, and the grades of
ecommendation.
This revised levels of evidence and grading system for recommendations which has been amended
ased on submissions received from the first pilot consultation, is now released for a second stage
consultation from early 2008 to 30 June XXXXXXXXXXSubmissions on the Stage 2 consultation draft can be
submitted to the NHMRC at: XXXXXXXXXX, with the heading: Att Project Officer, Levels &
Grades public consultation Stage 2.
NHMRC or other guidelines that are developed using this consultation framework must include a
statement at the front of the document explaining that the guidelines were developed using this
consultation framework, which blends the official NHMRC levels with the ‘interim’ levels of
evidence and grading system for recommendations.
Authors
Kristina Coleman, Sarah No
is, Adele Weston - Health Technology Analysts Pty Ltd
Karen Grimmer-Somers, Susan Hillier - Division of Health Sciences, University of South
Australia
Tracy Merlin - Adelaide Health Technology Assessment (AHTA), Discipline of Public
Health, University of Adelaide
Philippa Middleton, Rebecca Tooher - ASERNIP-S
Janet Salisbury - Biotext
Acknowledgments:
(a) Feedback has been provided during this document’s development phase from the following:
Marita Broadstock – New Zealand Health Technology Assessment
Suzanne Dyer – NHMRC Clinical Trials Centre
Paul Glasziou – Oxford University, United Kingdom
Brian Haynes – McMaster University, Canada
Paul Ireland – National Institute of Clinical Studies
2
mailto: XXXXXXXXXX
Nicki Jackson –Deakin University
Sally Lord and Les Irwig – University of Sydney
Skye Newton and Janet Hiller – University of Adelaide
Andrew Oxman – Oslo, Norway (GRADE Working Group)
(b) Significant contribution to the revision of the pilot for the Stage 2 Consultation was
provided during July-December 2007 by members of the NHMRC Guideline Assessment
Register (GAR) panel organisations:
• Division of Health Sciences,
University of South Australia
1) Quality Use of Medicines and
Pharmacy Research Centre
XXXXXXXXXXDr Agnes Vitry
XXXXXXXXXXMs Simone Rossi
XXXXXXXXXXCentre for Allied Health Evidence
XXXXXXXXXXProf Karen Grimmer-Somers
XXXXXXXXXXDr Susan Hillier
XXXXXXXXXXDr Caroline Smith
XXXXXXXXXXDr Saravana Kumar
XXXXXXXXXXDr Nicola Massy-Westropp
XXXXXXXXXXMr Peter Lekkas
• Faculty of Health Sciences, The
University of Adelaide:
1) Australian Research Centre for
Health of Women and Babies
(ARCH), Discipline of Obstetrics &
Gynaecology, School of Paediatrics
and Reproductive Health
- Prof Caroline Crowther
- Ms Philippa Middleton
- Dr Rebecca Tooher
XXXXXXXXXXAdelaide Health Technology
Assessment (AHTA), Discipline of
Public Health, School of Population
Health and Clinical Practice
XXXXXXXXXXProf Janet Hiller
XXXXXXXXXXMs Tracy Merlin
XXXXXXXXXXDr Peng Bi
XXXXXXXXXXMs Skye Newton
• Health Technology Analysts Pty Ltd
- Dr Adèle Weston
- Dr Sarah No
is
- Dr Kristina Coleman
• Biotext Pty Ltd
XXXXXXXXXXDr Janet Salisbury
XXXXXXXXXXDr Hilary Cadman
XXXXXXXXXXDr Fiona Mackinnon
The work on this project is being managed by the Evidence Translation Section, and
supported by National Institute of Clinical Studies Officers of the NHMRC.
3
PART A
Implementing NHMRC dimensions of evidence including new ‘interim’ levels of
evidence
This part of the document outlines how individual studies included in a systematic literature
eview should be assessed using the NHMRC dimensions of evidence and provides levels of
evidence appropriate for the most common types of research questions. The basic principles of
systematic reviewing and assessing evidence are set out in the NHMRC handbook series on the
development of clinical practice guidelines (NHMRC 2000ab).
Dimensions of evidence for assessing included studies
Each included study in a systematic review should be assessed according to the following three
dimensions of evidence:
1. Strength of evidence
a. Level of evidence: Each study design is assessed according to its place in the research
hierarchy. The hierarchy reflects the potential of each study included in the systematic
eview to adequately answer a particular research question, based on the probability that its
design has minimised the impact of bias on the results. See page 6–10 of How to use the
evidence: assessment and application of scientific evidence (NHMRC 2000b).
The cu
ently available NHMRC levels of evidence for intervention studies (NHMRC
2000b), together with the new levels of evidence for questions on diagnosis, prognosis,
aetiology and screening are shown in the evidence hierarchy in Table 1.
. Quality of evidence (risk of bias): The methodological quality of each included study
is critically appraised. Each study is assessed according to the likelihood that bias,
confounding and/or chance may have influenced its results. The NHMRC toolkit
How to review the evidence: systematic identification and review of the scientific
literature (NHMRC 2000a) lists examples of ways that methodological quality can
e assessed. In cases where other critical appraisal approaches may be required, there
are a number of alternatives. The NHMRC Guideline Assessment Register consultant
can advise on the choice of an alternative to supplement and/or replace those in the
NHMRC handbook (see Table 2).
c. Statistical precision: The primary outcomes of each included study are evaluated to
determine whether the effect is real, rather than due to chance (using a level of significance
expressed as a P-value and/or a confidence interval). See page 17 of How to use the
evidence: assessment and application of scientific evidence (NHMRC 2000b).
2. Size of effect
This dimension is useful for assessing the clinical importance of the findings of each study (and
hence clinical impact)