Expert Elicitation to Populate Early Health Economic Models of Medical Diagnostic Devices in Development

(1)

EXPERT ELICITATION TO POPULATE EARLY HEALTH ECONOMIC

MODELS OF MEDICAL DIAGNOSTIC DEVICES IN DEVELOPMENT

Wieke Haakma

1 _{, Laura Bojke}

2 _{, Lotte Steuten}

1 _{, Maarten IJzerman}

1 1. University Twente, Health Technology and Services Research, Enschede, Netherlands

2. University of York, Center for Health Economics, York, United Kingdom

Introduc)on

There is an increasing interest to es.mate the poten.al clinical value and likely cost-‐eﬀec.veness of diagnos.c and therapeu.c technologies during early development stages to guide further developments. [1,2] Yet, early stages of development are typically characterized by large uncertainty and popula.ng health-‐economic models with empirical data is not always feasible due to limited availability of data. Elicita.on of expert opinions is viewed as an appropriate alterna.ve and may serve as the input for early health economic models. [3]

Objec)ve

In the present study we explore whether expert elicita.on is a valid approach to characterize uncertainty regarding the diagnos.cs performance of photoacous.c imaging in breast cancer. As PAM is s.ll in the transla.onal stage (ﬁgure 1) and the prototype is s.ll in development, there is no clinical informa.on available.

Basic

research Transla.on research research Clinical Market access

Decision uncertainty

Figure 1 Flowchart of product development [1]

Methods

Diﬀerent methods have been applied to evaluate medical technologies in early stages of development e.g. Analy.c Hierarchy Process (AHP) [4], and expert elicita.on. Expert elicita.on is intended to link an expression of an experts’ beliefs into a sta.s.cal format and has been used a lot in Bayesian sta.s.cs because of the need to formulate priors.

We have chosen to use expert elicita.on as a method to formulate the knowledge and beliefs of experts about the future performance of PAM and to quan.fy this informa.on into probability distribu.ons.

Sample of experts

Twenty radiologists, specialized in the examina.on of MR images of breasts, from both academic and non academic hospitals in the Netherlands, were invited to par.cipate in this study as experts.

Calibra.on method

The purpose of calibra.on is to receive a rela.ve weigh.ng index for each expert. The weight of each individual expert was determined based on clinical background.

Ra.ng of tumor characteris.cs

Radiologists are asked to indicate the performance of PAM and MRI for diﬀerent tumor characteris.cs used in the examina.on of images of breasts.

Tumor characteris.cs are: (1) mass margins, (2) mass shape, (3) mass size, (4) vasculariza.on,

(5) localiza.on, (6) oxygen satura.on and (7) mechanical proper.es

Tumor characteris)cs

Indicate importance for discrimina)on between benign and

malign )ssue

Indicate performance

of MRI and PAM MRI PAM Allocate 100% Range 0-‐100

Figure 2 Elicita)on of tumor characteris)cs

Elici.ng distribu.ons

A spreadsheet-‐based (Excel) exercise was designed to elicit the TPR and TNR. Experts received a face-‐to-‐face interview of 30 to 45 minutes in which the similar data regarding PAM was presented to each individual radiologist.

Pooled data of MRI was provided based on four studies where MRI was used in a diagnos.c se_ng. For this a 2*2 table was used, where it is suﬃcient to es.mate the TPR and TNR as the false posi.ve rate (FPR), and false nega.ve rate (FNR), will follow from that.

Mathema.cal approach

Parameters

Expert panel

Calibra.on

method

• Individual face-‐to-‐face interviews

• 18 (non) academic

radiologists • True posi.ve rate _{• True
nega.ve
rate}

• Years of experience • Number of MRI's examined • Other areas

Credible

interval

• Mode •  Lower boundary • Upper boundary

Presen.ng

experts'

beliefs

• Pert approach • Beta distribu.on

Bias

• Provide data in similar way • Explain uncertainty • Provide feedback

Synthesis

method

• Linear opinion pooling

Figure 3 Elicita)on procedure

Figure 5 Importance tumor characteris)cs and performance MRI and PAM

Results

Of the 20 radiologists, two radiologists were unable to aiend. One radiologist was excluded due to his lack of compliance with the method.

Ra.ng tumor characteris.cs

Radiologists indicated that they did not have suﬃcient data about the added value of oxygen satura.on and the mechanical proper.es.

Sensi.vity and speciﬁcity

Three out of seventeen radiologists indicated that it was too early to make these es.ma.ons due to the absence of data from clinical trials.

Probability distribu.on

Experts were asked to indicate the mode (figure 3a) the lower and the upper boundaries (figure 3b) within a 95% credible interval. With the PERT approach the mean (µ), standard devia.on (σ), alpha (α) and beta (β) can be obtained of which the probability distribu.on (figure 3c) can be determined.

Linear opinion pooling was used to obtain an overall probability distribu.on, where p(Ѳ) is the probability distribu.on for the unknown parameter Ѳ and where w_i is the radiologists’ i’s weight summing up to 1.

a b c

Elici.ng the mode, than the upper and lower boundaries and by using the PERT approach a probability distribu.on was obtained.

0 0.2 0.4 0.6 0.8 1 0 50 100 150 200 250 Pro ba bi lit y TPR Mode -‐0.01 0.01 0.03 0.05 0 50 100 150 200 250 Pro ba bi lit y TPR

Lower and upper boundaries 0 0.02 0.04 0.06 0 50 100 150 200 250 Pro ba bi lit y TPR Probability distribu)on 1 Mass margins 2 Mass shape 3 Vasculariza.on 4 Mechanical proper.es 5 Mass size 6 Loca.on mass 7 Oxygen satura.on 0 5 10 15 20 25 30 35 0 10 20 30 40 50 60 70 80 90 100 1 2 3 4 5 6 7 Sc or e tum or c hr ac te ris) cs Sc or e MRI a nd P AM Tumor characteris.cs Score MRI Score PAM 4) 2) 3) 5) 1)

Conclusions

§ Experts es.mated the mode of the sensi.vity and speciﬁcity of PAM to be 75.6% and 66.5%, which is lower than MRI (90.1% and 69.5%).

§ Experts expressed diﬃcul.es es.ma.ng the performance of PAM based on limited data regarding PAM. § To improve the validity of radiologists’ es.ma.ons in this study, it is desirable to elicit priors for speciﬁc

tumor types, since radiologists indicated to base their es.ma.ons on an aggregate expecta.on about how PAM will visualize the various tumor types.

§ Further clinical trials should be commissioned to indicate whether these results are valid and expert elicita.on could be used in early technology assessment. Before that, the use of the elicited priors in health economic models requires careful considera.on.

References

1. IJzerman MJ, Steuten LMG. Early assessment of medical technologies to inform product development and market access. A review of methods and applica.ons. Appl. Health Econ & Health Policy. 2011; 9(5): 331-‐347.

2. Vallejo-‐Torres L, Steuten LMG, Buxton MJ, Girling AJ, Lilford RJ, Young T. Integra.ng health economics modeling in the product development cycle of medical devices: A Bayesian approach. Int. J. Technology Assessment in Health Care. 2008;24(04):459-‐64.

3. Bojke L, Claxton K, Bravo-‐Vergel Y, Sculpher M, Palmer S, Abrams K. Elici.ng distribu.ons to populate decision analy.c models. Value in Health. 2010 Aug;13(5):557-‐64.

4. Hilgerink MP, Hummel MJM, Manohar S, Vaartjes SR, IJzerman MJ. Assessment of the added value of the Twente Photoacous.c Mammoscope in breast cancer diagnosis. Med Devices. Evidence & Research. 2011; 4: 107-‐114

Figure 6 shows that there is considerably heterogeneity between radiologists.

The sensi.vity ranged from 58.9% to 85.1% with a mode of 75.6%. The speciﬁcity ranged from 52.2% to 77.6% with a mode of 66.5%.

Figure 6 Probability distribu)on of es)ma)ons of TPR of 14 radiologists

0.00000 0.02000 0.04000 0.06000 0.08000 0.10000 0.12000 0.14000 0.16000 0.18000 0.20000 0 50 100 150 200 250 Pro ba bi lit y TPR Expert 1 Expert 2 Expert 3 Expert 4 Expert 5 Expert 6 Expert 7 Expert 8 Expert 9 Expert 10 Expert 11 Expert 12 Expert 13 Expert 14 Experts overall

Expert Elicitation to Populate Early Health Economic Models of Medical Diagnostic Devices in Development