1Centre for Health Economics, University of York, York YO10 5DD, UK
2Department of Economics and Related Studies, University of York, York, UK
3Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, The Netherlands 4Department of Health Technology & Services Research, University of Twente, Enschede, The Netherlands
Correspondence to: S Griffin susan.griffin@york.ac.uk Additional material is published online only. To view please visit the journal online (http://dx.doi. org/10.1136/bmj.h5987) Cite this as: BMJ 2015;351:h5987 doi: 10.1136/bmj.h5987 Accepted: 13 October 2015
How to estimate the health benefits of additional research
and changing clinical practice
Karl Claxton,
1,2Susan Griffin,
1Hendrik Koffijberg,
3,4Claire McKenna
1A simple extension of standard
meta-analysis can provide quantitative
estimates of the potential health
benefits of further research and of
implementing the findings of existing
research, which can help inform
research prioritisation and efforts to
change clinical practice
Decisions about undertaking further research and at what point evidence should be implemented are important questions for research prioritisation and health policy. The results of standard meta-analysis can be extended to provide a quantitative assessment of the potential health benefits of gathering additional evidence and of implementing the findings in a way that consideration of statistical significance cannot.1
Health impact of additional research
The result of a meta-analysis describes both the central estimate of effectiveness within a range of plausible val-ues (for example, a confidence interval) and the likeli-hood of observing particular values (for example, P values or error probability).2 This distribution of values
describes the uncertainty about the relative effective-ness of an intervention.3 Decisions to commission
fur-ther research imply that this uncertainty is unacceptable or important enough to justify the research costs. When the range of plausible values can support more than one intervention this uncertainty has health conse-quences, because for any treatment choice there will be a chance that another intervention would have improved health outcomes to a greater extent. The importance of the uncertainty is indicated by the scale of these health consequences. The chance that an inter-vention is not the most effective, how much less effec-tive the intervention is likely to be, and the number of patients facing this uncertain treatment choice are all
factors that contribute to the health consequences of this uncertainty.
The chance that an intervention is not the most effec-tive is indicated by the standard results of meta-analy-sis. By applying a relative measure of effect (for example, an odds ratio for mortality) to an estimate of baseline risk of an adverse outcome, it is possible to estimate the amount of health that is potentially lost by adopting a less effective intervention. The web appen-dix provides a worked example. In this way, the number of patients facing the uncertain treatment choice allows the estimated consequences of uncertainty to be expressed as the health outcomes that could have been achieved for the patient population.
Further research can reduce (although not entirely eliminate) uncertainty and its associated health conse-quences, because the range of values about a central estimate of effect do not exclude the null effect or possi-ble harm but instead render it increasingly unlikely.14
Therefore, an estimate of the potential adverse health consequences of uncertainty represents an upper bound on the potential benefits of further research. It can be used to (1) prioritise proposed research, (2) con-sider whether the potential benefits are likely to justify the costs, and (3) indicate when existing evidence is sufficient to warrant implementation because further research is unlikely to be worthwhile.45
Health impact of changing clinical practice
Health outcomes can be improved by ensuring that the findings of existing research are more fully adopted within clinical practice.6 By combining the central
esti-mate of relative effect from meta-analysis with baseline risk estimates of the particular outcome, we can esti-mate the expected (absolute) benefits of implementing research findings. When combined with an estimate of the incidence of disease or clinical events, and the cur-rent levels of use of the available interventions, the expected population health benefits of implementation efforts can also be estimated (the web appendix pro-vides a worked numerical example). This estimate allows a direct comparison of undertaking further research versus implementation of current evidence as two different ways to improve health outcomes.
Of course, further research might also influence clin-ical practice and contribute to implementation efforts. However, a clear distinction is necessary because fur-ther research is not the only, or necessarily the most effective, way to change clinical practice. In view of other mechanisms such as more effective dissemina-tion of evidence, guidelines, incentives, and sancdissemina-tions, conducting research to influence clinical practice might be an inappropriate use of scarce research capacity to generate greater health benefits elsewhere,
Summary pointS
A simple extension of standard meta-analysis can provide quantitative estimates of the potential health benefits of further research and of implementing the findings of existing research
This strategy can help identify research proposals that are likely to be worthwhile and offer the greatest value to be commissioned
These estimates can also help determine when research findings should be implemented and when existing evidence is sufficient because further research is not worthwhile
doi1 02.00;6/bmj.h15987 | BMJ 2015;101h15987 | the bmj
ReseaRch Methods & RepoRting
2
and could disadvantage those patients participating in the research.6
Example: use of streptokinase in early thrombolysis to treat myocardial infarction
In this example, we consider clinical trials that investi-gated early thrombolysis using streptokinase (versus no thrombrolysis) after acute myocardial infarction. Full details of trials are detailed in a reference at the end of the web appendix. The sequence of these clinical trials illustrates how the values of further research and of implementation can be derived from the results of meta-analysis.7-10 Figure 1 lists the sequence of trials
showing the effects of streptokinase on mortality in a forest plot.79 A random effects meta-analysis of these
trials strongly favoured streptokinase with an odds ratio of mortality of 0.76 (95% confidence interval 0.67 to 0.85).
In figure 2 , a cumulative meta-analysis shows how this evidence accumulated over time: the effect on mor-tality and its associated uncertainty are updated with each subsequent trial.9 The forest plot demonstrates
that the balance of evidence quickly favoured
streptoki-nase and the degree to which uncertainty about its effectiveness generally declined as subsequent trials reported. For example, following the European 3 trial, the probability that streptokinase was more effective exceeded 0.95 in both the trial and random effects meta-analysis. By the time the ISIS-2 trial was reported, any remaining uncertainty was effectively resolved. A number of important questions emerge (box).
What are the potential health benefits of subsequent trials?
Although the balance of evidence quickly favoured streptokinase in this example, there remained some uncertainty about its effectiveness at each point in this sequence of trials—that is, streptokinase use following acute myocardial infarction might increase, rather than reduce, mortality. The expected health consequences of this uncertainty depend not only on the chance that streptokinase was not effective, but also on how much less effective than no thrombolysis it was likely to be and the incidence of patients eligible for early thrombolysis.
The chance that streptokinase was not effective is indicated by the confidence intervals for the odds ratios
Fletcher 1959 Dewar 1963 European 1 1969 European 2 1971 Heikinheimo 1971 Italian 1971 Australian 1973 Frankfurt 2 1973 NHLBI SMIT 1974 Frank 1975 Valere 1975 Klein 1976 UK Collab 1976 Austrian 1977 Australian 2 1977 Lasierra 1977 N Ger Collab 1977 Witchiz 1977 European 3 1979 ISAM 1986 GISSI-1 1986 Olson 1986 Schreiber 1986 Durand 1987 White 1987 Bassand 1987 Vlay 1988 Kennedy 1988 ISIS-2 1988 Pooled summary 0.16 (0.01 to 1.73) 0.47 (0.11 to 1.94) 1.46 (0.69 to 3.10) 0.64 (0.45 to 0.90) 1.25 (0.64 to 2.42) 1.01 (0.51 to 2.01) 0.75 (0.44 to 1.31) 0.38 (0.18 to 0.78) 2.59 (0.63 to 10.60) 0.96 (0.29 to 3.19) 1.06 (0.39 to 2.88) 3.20 (0.30 to 34.59) 0.88 (0.57 to 1.35) 0.56 (0.36 to 0.87) 0.81 (0.44 to 1.48) 0.22 (0.02 to 2.53) 1.22 (0.80 to 1.85) 0.78 (0.20 to 3.04) 0.42 (0.24 to 0.72) 0.88 (0.68 to 1.14) 0.80 (0.72 to 0.90) 0.83 (0.21 to 3.29) 0.21 (0.02 to 2.07) 0.59 (0.12 to 2.86) 0.18 (0.04 to 0.81) 0.48 (0.15 to 1.51) 0.42 (0.03 to 5.30) 0.63 (0.29 to 1.36) 0.75 (0.68 to 0.82) 0.76 (0.67 to 0.85) 0.1 0.5 1 1.5 2.0 2.5 3.0 3.5
Study and year
Favours
streptokinase Favourscontrol
Odds ratio (95% CI) Odds ratio (95% CI)
1/12 4/21 20/83 69/373 22/219 19/164 26/264 13/102 7/53 6/55 11/49 4/14 48/302 37/352 25/112 1/13 63/249 5/32 25/156 124/859 628/5860 5/28 1/19 3/35 2/79 5/52 1/13 12/191 791/8592 Streptokinase 4/11 7/21 15/84 94/357 17/207 18/157 32/253 29/104 3/54 6/53 9/42 1/9 52/293 65/376 31/118 3/11 51/234 5/26 50/159 142/882 758/5832 5/24 4/19 4/29 12/93 10/55 2/12 17/177 1029/8595 Control No of deaths/total
Fig 1 | Random effects meta-analysis of sequence of trials of early thrombolysis using streptokinase, with the endpoint of mortality
Fletcher 1959 Dewar 1963 European 1 1969 European 2 1971 Heikinheimo 1971 Italian 1971 Australian 1973 Frankfurt 2 1973 NHLBI SMIT 1974 Frank 1975 Valere 1975 Klein 1976 UK Collab 1976 Austrian 1977 Australian 2 1977 Lasierra 1977 N Ger Collab 1977 Witchiz 1977 European 3 1979 ISAM 1986 GISSI-1 1986 Olson 1986 Schreiber 1986 Durand 1987 White 1987 Bassand 1987 Vlay 1988 Kennedy 1988 ISIS-2 1988 – – – 0.85 (0.24 to 1.50) 0.83 (0.40 to 1.45) 0.86 (0.51 to 1.33) 0.83 (0.57 to 1.16) 0.76 (0.52 to 1.05) 0.80 (0.55 to 1.15) 0.80 (0.58 to 1.13) 0.82 (0.60 to 1.12) 0.83 (0.62 to 1.14) 0.83 (0.65 to 1.09) 0.80 (0.63 to 1.02) 0.79 (0.64 to 0.99) 0.78 (0.63 to 0.97) 0.83 (0.67 to 1.03) 0.82 (0.67 to 1.00) 0.78 (0.63 to 0.97) 0.80 (0.66 to 0.95) 0.80 (0.68 to 0.93) 0.80 (0.68 to 0.93) 0.79 (0.68 to 0.92) 0.79 (0.67 to 0.92) 0.77 (0.65 to 0.90) 0.77 (0.65 to 0.89) 0.77 (0.65 to 0.88) 0.76 (0.64 to 0.88) 0.76 (0.67 to 0.85) 0.1 0.5 1 1.5 2.0
Study and year
Favours
streptokinase Favourscontrol
Cumulative odds ratio (95% CI) Cumulative odds
ratio (95% CI)
Fig 2 | Cumulative random effects meta-analysis of sequence of trials of early thrombolysis using streptokinase, with the endpoint of mortality
in figure 2, in particular, where the confidence interval spans the line of unity (where the intervention effect lies in favour of control). How much less effective it may be is estimated by combining the range of odds ratios that exceed one with an estimate of the baseline mortal-ity risk following acute myocardial infarction. Here, we use the range of values for the baseline odds from a pooled analysis of the control arms of the trials to esti-mate the baseline mortality risk. We can derive uncer-tainty in the relative effect size by taking random samples from the distribution of relative effect size (that is described by the meta-analysis), and combining them with random samples from the previously estimated baseline mortality risk. Each sample represents one possible “true” outcome supported by the evidence (see web appendix for a worked example).
Repeating this process can ensure a population of sampled values that describes a distribution of the health consequences of uncertainty. An estimate of the patient population size is required that includes those who have acute myocardial infarction and who reach hospital in time to be eligible for early thrombolysis (incidence in the United Kingdom was estimated to be 204 500 patients per year8).
Figure 3 shows the resulting distribution of health consequences of uncertainty at three points in the cumulative meta-analysis (that is, when the European 1,
NHLBI SMIT (the National Heart Lung and Blood Insti-tute streptokinase in myocardial infarction trial), and European 3 trials were reported). Following the Euro-pean 1 trial, early in the sequence of trials, the balance of evidence favoured streptokinase with a 69% chance that the enzyme was effective and reduced mortality (that is, there was a 69% chance that streptokinase use would have no adverse mortality consequences). However, there was also a 31% chance that streptokinase was not effective and mortality would have been lower if the enzyme had not been used. Examining this 31% chance reveals that the adverse health consequences are not uniform. For example, there was a greater chance of more limited adverse consequences (20% chance of zero to 15 000 deaths per year) than a smaller chance of greater adverse consequences (11% chance of more than 15 000 deaths per year). The mean of this distribution provides an estimate of the expected adverse health sequences of uncertainty and, by implication, the con-sequential value of conducting further research at this point (6264 deaths averted per year).
Uncertainty and potential benefits from further research decrease as subsequent trials report. For exam-ple, following the NHLBI SMIT trial, the chance that streptokinase was more effective rose to 91% (that is, a 91% chance of zero adverse mortality consequences). The chance of more limited consequences was lower (8.77% chance of zero and 15 000 deaths per year) and the chance of greater consequences was almost elimi-nated (0.21% chance of more than 15 000 deaths per year). Consequently, the potential benefit of conducting further research fell to averting 306 deaths per year, based on the mean of this distribution of outcomes. By the time the European 3 trial was reported, there was much less uncertainty (98% chance that streptokinase was more effective), and the potential benefits decreased to an estimated mean of 27 deaths averted per year. Once the ISIS-2 trial was reported, uncertainty was effectively eliminated and the potential benefits for further research were negligible.
What is the expected health benefit of implementing research findings?
In this example, we have examined the potential harms due to the uncertainty about whether use of streptoki-nase reduces mortality. Examining the expected bene-fits indicates the value of implementing the findings of existing research. In this example, the balance of evi-dence quickly favoured streptokinase (that is, on aver-age the benefits (odds ratio <1) exceed the harms (odds ratio >1)); thus, one would expect a reduction in mortal-ity by ensuring streptokinase use following acute myo-cardial infarction was widely implemented.
Figure 4 illustrates the expected benefits of imple-menting what existing evidence suggests is the most effective intervention and the potential benefits of con-ducting further research at the same three points in the cumulative meta-analysis as figure 3. Streptokinase use in early thrombolysis was expected to avert 7609 deaths per year following the European 1 trial, 6138 deaths per year following the NHLBI SMIT trial, and 7125 following
Important questions to address as evidence accumulates
• How does the potential benefit of further research change as evidence accumulates?
• Could health outcomes be improved most by conducting further research or by ensuring the findings of existing research change clinical practice?
• How should other aspects of health outcome and the additional costs of the intervention be considered?
• When should the findings of research be implemented?
• At what point is it unlikely that additional research is worthwhile? • How can research be prioritised and which proposed research should be
commissioned?
• Are sufficient resources being devoted to evaluative research?
Additional deaths per year from using streptokinase
Probability 0 0 0.2 0.4 0.6 0.8 1.0 ≤14 999 15 000 to 29 999 30 000 to89 999 ≥90 000
European 1 (mean deaths=6264) NHLBI SMIT (mean deaths=306) European 3 (mean deaths=27)
Fig 3 | Distribution of consequences of uncertainty for streptokinase use following acute myocardial infarction, according to selected trials in example meta-analysis. Numbers in brackets are the expected (mean) number of deaths that could be avoided if uncertainty were
eliminated in the decision to use streptokinase at the point of each trial
doi1 02.00;6/bmj.h15987 | BMJ 2015;101h15987 | the bmj
ReseaRch Methods & RepoRting
4
the European 3 trial. However, it was not until European 3 that both a single trial and the cumulated evidence reported a statistically significant result that favoured streptokinase. Although clinical practice did respond to this evidence with the widespread use of streptokinase, before European 3 the expected benefits of implement-ing streptokinase use for acute myocardial infarction were already substantial (5696 deaths averted per year). This analysis suggests that waiting to implement research findings until the traditional rules of statistical significance are achieved could come at considerable cost to patient outcomes.1
When should research findings be implemented?
In our example, figure 4 shows that the expected bene-fits of encouraging the implementation of what existing evidence suggests is the most effective intervention exceed the potential benefits of conducting further research. However, the early implementation of an intervention based on the balance of evidence may not be appropriate if its widespread use means that the type of research required (for example, with adequate con-trol) becomes impossible or more costly.
For example, in 2001, the National Institute for Health and Care Excellence (NICE) approved acetyl-cholinesterase inhibitors for the treatment of mild and moderate Alzheimer’s disease in the UK’s health service, despite considerable uncertainty about the long term effects on institutionalisation. As a conse-quence of widespread access to these drugs following NICE approval, a large ongoing publicly funded trial, with much longer follow-up than the existing trials and the only trial with a primary endpoint of institu-tionalisation, was unable to complete recruitment. Therefore, the benefit of early implementation needs to be balanced against the potential benefits for future patients from further research—it may be better to delay implementation until further research can be completed that will offer benefits to patients over many years.
Furthermore, efforts to implement research findings could be costly and, if successful, it could become diffi-cult and even more costly to reverse if subsequent research finds that the intervention is not sufficiently
effective. The question of when research findings should be implemented requires nuanced trade-offs, which can be better informed with a quantitative assessment of these two different ways to improve health outcomes.4
other aspects of outcome to consider
The trials in our example reported mortality as a pri-mary endpoint, so the benefits of further research and implementation have been expressed as deaths averted. In other contexts (such as Alzheimer’s disease), end-points associated with morbidity might include events (such as the need for nursing home care) or continuous measures of performance (such as improvements in the mini-mental state examination 9). In these circum-stances, the benefits of research and implementation can be expressed as the number of events avoided or, if on a continuous scale, the improvement per patient or the number of patients with a clinically important improvement.
However, the mortality and morbidity endpoints reported in trials commonly reflect only limited aspects of outcome. Linking these endpoints to a more com-plete measure of health outcome could be informed by meta-analysis of secondary endpoints or by other sources of evidence, while reflecting any additional uncertainty that might be introduced.6
In our example, evidence of expected survival and quality of life after myocardial infarction could be used to translate mortality consequences in figure 3 into life years and quality adjusted life years (QALYs) gained. The translation of health effects to a more generic mea-sure of outcome enables a comparison of the health benefits of further research and implementation efforts across diverse clinical areas. Greater improvements in effectiveness will be needed to justify any additional costs associated with an intervention, which could be reflected by a minimum clinical difference in effect. This additional hurdle will tend to reduce the benefits of implementation and further research.4
is further research worthwhile?
Estimates of the potential health benefits of research start to indicate its relative priority compared with other areas of research. Whether the costs of research are jus-tified by the potential benefits, can be evaluated by examining whether the resources required might gener-ate similar, or gregener-ater, expected health benefits if made available to the NHS? Or, equivalently, whether more NHS resources would be required to generate similar or more health benefits than the proposed research? For example, recent research suggests that it costs the NHS £114 000 (€161 000; $172 000) to avert one death, £25 000 to gain one life year, and £13 000 to gain one QALY.11
This type of quantitative assessment allows research proposals that are likely to be worthwhile to be identi-fied and those that are likely to offer the greatest value to be commissioned from available research funds. This assessment can also indicate whether sufficient resources are being devoted to research—that is,
European 1
NHLBI SMIT
European 3
0.25 0.50 0.75
Value of research = 6264 deaths averted
Value of implementation = 7609 deaths averted
Value of implementation = 6138 deaths averted
Value of implementation = 7125 deaths averted
Value of implementation = 7609 deaths averted
Value of implementation = 6138 deaths averted
Value of implementation = 7125 deaths averted
Value of research = 307 deaths averted
Value of research = 27 deaths averted
Value of research = 27 deaths averted
1.00 1.25 1.50
Favours streptokinase Favours control
Odds ratio mortality (95% CI)
Fig 4 | Expected benefits of implementation (of streptokinase use following acute myocardial infarction) and potential benefits of further research, according to selected trials in example meta-analysis shown in figure 3. Solid line=95% credible interval; dashed line=90% credible interval
whether potentially worthwhile proposals cannot be commissioned because other proposals are more valu-able, and whether patient outcomes could be improved overall by transferring resources from service provision to research.
Conclusions
Systematic reviews and meta-analyses often conclude that further research is needed. The simple extension of standard meta-analysis illustrated in this article could be routinely reported and would provide quanti-tative assessments of the health benefits of further research and of implementing the findings of existing research. These estimates can help inform the prioriti-sation and commissioning of research by identifying where and what type of research would be most valu-able. For example, the Patient Centred Outcomes Research Institute in the United States is responsible for allocating $3.8bn to comparative effectiveness research as part of recent healthcare reforms, and is already using the principles of this type of analysis to inform its priorities.12
These estimates can also inform other important questions of health policy: for example, a review into medical innovation and technology by the Department of Health and Office of Life Sciences is underway in the UK. Other questions include whether the earlier wide-spread use of an intervention should be encouraged based on the balance of existing evidence and whether sufficient resources are being devoted to evaluative research.
Contributors: All authors contributed equally to the conception of the research study. KC and CM wrote the first draft and led the writing of the paper. SG carried out the analysis with input from KC, CM, and HK. SG and HK read and commented on drafts of the paper. All authors have approved the final version. KC is the guarantor.
Funding: This research was supported by the Patient Centered Outcomes Research Institute (PCORI) in the USA. Initial findings were presented at the PCORI methodology workshop for prioritising specific research topics in Washington, DC, December 2012. HK was also supported by the Netherlands Organization for Scientific Research
(NWO, grant 916.11.126). Neither organisation was involved in the study.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: support from PCORI and the Netherlands Organization for Scientific Research for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Provenance and peer review: Not commissioned; externally peer reviewed.
1 Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. J Health Econ 1999;18:341-64.
2 Sutton AJ, Abrams KR. Bayesian methods in meta-analysis and evidence synthesis. Stat Methods Med Res 2001;10:277-303. 3 Whitehead A. Meta-analysis of controlled clinical trials. John Wiley &
Sons, 2002.
4 Claxton K, Palmer S, Longworth L, et al. Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development. Health Technol Assess 2012;16:1-323.
5 Colbourn TE, Asseburg C, Bojke L, et al. Preventive strategies for group B streptococcal and other bacterial infections in early infancy: cost effectiveness and value of information analyses. BMJ 2007;335:655. 6 Claxton K, Griffin S, Koffijberg H, et al. Expected health benefits of
additional evidence: principles, methods and applications. Research paper 83. Centre for Health Economics, University of York, 2013. 7 Antman EM, Lau J, Kupelnick B, et al. A comparison of results of
meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA 1992;268:240-8.
8 Boland A, Dundar Y, Bagust A, et al. Early thrombolysis for the treatment of acute myocardial infarction: a systematic review and economic evaluation. Health Technol Assess 2003;7:1-136. 9 Lau J, Antman EM, Jimenez-Silva J, et al. Cumulative meta-analysis of
therapeutic trials for myocardial infarction. N Engl J Med 1992;327:248-54.
10 Yusuf S, Collins R, Peto R, et al. Intravenous and intracoronary fibrinolytic therapy in acute myocardial infarction: overview of results on mortality, reinfarction and side-effects from 33 randomized controlled trials. Eur Heart J 1985;6:556-85.
11 Claxton KP, Martin S, Soares MO, et al. Methods for the estimation of the NICE cost effectiveness threshold. Health Technol Assess 2015;19. 12 Fleurence RL, Meltzer DO. Toward a science of research prioritization?
The use of value of information by multidisciplinary stakeholder groups. Med Decis Making 2013;33:460-2.
© BMJ Publishing Group Ltd 2015
