Addressing uncertainty in MCDA for healthcare decisions:
A scoping review of methods
Henk Broekhuizen, MSc
1
; Karin Groothuis-Oudshoorn, PhD
1
; Janine van Til, PhD
1
; Marjan Hummel, PhD
1
; Maarten IJzerman, PhD
1
(1) Health Technology and Services Research, University of Twente, Enschede, the Netherlands
The setting of healthcare decisions
Our review identified five approaches to take uncertainty into account in MCDA. The approach most used in health
care was deterministic sensitivity analysis. This approach will most likely suffice for most health care policy
decisions because of its low complexity and straightforward implementation. Further research is needed to identify
when to take into account uncertainty and which approach is most useful for decision makers.
Objectives
Multi criteria decision analysis (MCDA) aims to support decision-making where decisions are
based on multiple criteria. The use of MCDA in HTA priority-setting and reimbursement
decisions is growing, but mostly limited to research projects. A factor that might influence
acceptance is a perceived difficulty to value an MCDA’s outcome when its inputs and outputs
contain uncertainties. When this is the case, decision makers might not feel confident in
accepting or rejecting its outcome.
The objective of this study is to review how uncertainty is taken into account in MCDA
methods in general, and to discuss which of the approaches is appropriate for healthcare
decision making.
Results
The search strategy identified 569 abstracts, mostly from non-healthcare journals (Figure 1). 3% were published in
healthcare-related journals. A large variety of MCDA methods was found, confirming earlier indications of a
heterogeneous MCDA nomenclature. Some combinations of MCDA method and approach to deal with uncertainty
were identified often, such as Fuzzy AHP. Approaches identified were
-
Deterministic framework (31%)
-
Probabilistic framework (15%)
-
Bayesian framework (6%)
-
Fuzzy set theory (45%)
-
Grey theory (3%)
Methods
A scoping literature review was conducted using the Scopus and Pubmed databases.
Identified abstracts were categorized by MCDA method used. Then, approaches to deal with
uncertainty were identified by two independent reviewers. The most recent methodological
article per approach was read to identify methodological details.
Poster presenter:
Henk Broekhuizen, MSc.
PhD student at University of Twente
Contact information: h.broekhuizen@utwente.nl www.utwente.nl/mb/htsr/Staff/broekh uizen/
Deterministic framework
Bayesian framework
Probabilistic framework
Fuzzy set theory
Grey theory
Model parameters are varied manually, and the impact on model outcomes is assessed. The effect on outcomes can be shown in simple line plots or a tornado diagram.
The uncertainty around model parameters is estimated with probability distributions that reflect reality. Impact on model outcomes can be assessed by varying all parameters simultaneously based on their probability distribution.
Fuzzy sets are distinguished from regular sets in that elements in a set have a degree of membership instead of a binary (yes/no) membership [3]. Proponents of fuzzy set theory argue that human judgment is often fuzzy, and that assessments in decision analytic models should incorporate this. We found fuzzy set theory was combined a suprisingly large number of times (n=174) with the MCDA method AHP. In most of these studies, the conventional socalled crisp judgement scale was replaced with fuzzy triangular numbers to indicate the fuzzyness of judgements. 0.00 0.10 0.20 0.30 0 .0 0 0 .0 5 0 .1 0 0 .1 5 0 .2 0 0 .2 5 0 .3 0 0 .3 5 Risk-benefit plane
Sum of weighted benefits
S u m o f w e ig h te d r isks Duloxetine Venlafaxine Bupropion
Verbal judgement Saaty’s fundamental scale Triangular fuzzy set
Extremely preferred 9 (9,9,9) Very strongly to extremely
preferred
8 (7,8,9)
Very strongly preferred 7 (6,7,8)
Strongly to very strongly preferred 6 (5,6,7) Strongly preferred 5 (4,5,6) Moderately to strongly preferred 4 (3,4,5) Moderately preferred 3 (2,3,4)
Equally to moderately preferred 2 (1,2,3)
Equally preferred 1 (1,1,1) Adapted from [4]
0
1
Belief Plausibility Doubt Disbelief Uncertainty Adapted from [6] 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5Drug A Drug B Drug C
Worst case Interval Best case Adapted from [2] Source: [1] Adapted from [8] Bayesian networks
A core idea of the Bayesian framework is the updating of prior belief with acquired data. The latter is captured in Bayes’ theorem: 𝑃 𝐴 𝐵 = 𝑃 𝐵 𝐴 𝑃 𝐴𝑃(𝐵) . An example of using the Bayesian framework for MCDA is the construction of a so-called Bayesian net, which is a directed graph illustrating (conditional) links between model parameters [2].
Goal
Criterion 1 Alternative i Criterion n Uncertain criteria Criterion n+1 Criterion n+x Bayesian net Criterion n+1 Criterion n+x Factor 1 Factor k Factor k+1References
[1] H Broekhuizen, CGM Groothuis-Oudshoorn, AB Hauber, JP Jansen and MJ IJzerman, "Integrating elicited patient preferences and clinical trial data in a quantitative model for benefit-risk assessment", 25th Annual DIA EuroMeeting, 2013 [2] N Fenton and M Neil, “Making Decisions: Using Bayesian Nets and MCDA”, Knowledge-Based Systems, vol. 14, pp. 307-325, 2000
[3] L Zadeh, “Fuzzy sets,” Information and control, vol. 353, pp. 338–353, 1965.
[4] P Pitchipoo, P Venkumar, and S Rajakarunakaran, “Fuzzy hybrid decision model for supplier evaluation and selection,” Int J of Production Research, vol. 51, no. 13, pp. 3903–3919, Jul. 2013. [5] W Ma, W Xiong, and X Luo, “A Model for Decision Making with Missing , Imprecise , and Uncertain Evaluations of Multiple Criteria,” Int J of Intelligent Systems, vol. 28, pp. 152–184, 2013. [6] RU Kay, “Fundamentals of the Dempster-Shafer theory and its applications to system safety and reliability modelling,” Int J of Reliability, Quality and Safety Engineering, pp. 173–185, 2007. [7] JL Deng, “Control problems of grey systems,” Systems & Control Letters, vol. 1, no. 5, pp. 288–294, Mar. 1982.
[8] EK Zavadskas and A Kaklauskas, “Multi-Attribute Decision-Making Model by Applying Grey Numbers,” Informatica, vol. 20, no. 2, pp. 305–320, 2009.
[5] W Ma, W Xiong, and X Luo, “A Model for Decision Making with Missing , Imprecise , and Uncertain Evaluations of Multiple Criteria,” Int J of Intelligent Systems, vol. 28, pp. 152–184, 2013. [6] RU Kay, “Fundamentals of the Dempster-Shafer theory and its applications to system safety and reliability modelling,” Int J of Reliability, Quality and Safety Engineering, pp. 173–185, 2007. [7] JL Deng, “Control problems of grey systems,” Systems & Control Letters, vol. 1, no. 5, pp. 288–294, Mar. 1982.
[8] EK Zavadskas and A Kaklauskas, “Multi-Attribute Decision-Making Model by Applying Grey Numbers,” Informatica, vol. 20, no. 2, pp. 305–320, 2009.
Figure 1: Research areas in which abstracts were found, estimated with the All Science Journal Classification (ASJC).
Table 1: Distribution of themes over various MCDA methods.
Dempster-Shafer theory
The evidential reasoning method Dempster-Shafer theory is meant to deal with unknown, interval valued, multifaceted or ambiguous information [5]. Experts make probability mass statements over frame of discernment Θ that are mapped with mass function 𝑚: 2Θ → [0,1]. Assigning mass to the whole set 2Θ is a measure of residual ignorance. Lower and upper bounds of evidential support are termed belief and plausibility. Probability masses can be combined with Dempster’s rule of combination. The degree of conflict between the judgments of experts can be assessed. Finally, probability masses assigned to preferences and performances can be combined with this rule to make statements about alternatives’ performances.
Grey numbers are numbers whose exact value is not known [7]. They are instead represented with ranges, for example grey numbers 𝐺1 ∈ (−1,5)or 𝐺2 ∈ [3, ∞]. Black numbers are totally unknown, e.g. 𝐵 ∈ [−∞, ∞] , and white numbers represent perfect knowledge; e.g. white number 𝑊 ∈ [15,15]. Greyness as a concept can also be applied to the ambiguity present in decisions, where most decisions are grey; i.e. under some but not complete uncertainty.
Online seminar about MCDA in healthcare
The recordings from our recent online seminar, entitled ‘The Basics and Application of Multi Criteria Decision Analysis in Healthcare Decision Making’ are available online via www.utwente.nl/mb/htsr, or you can contact me (Henk) for a link via email. In the seminar we gave a short introduction into the rationale behind MCDA, gave practical examples of its application and delved into a number of methodological issues to consider when choosing a method.
