University of Groningen
Mortality prediction models in the adult critically ill
HEALICS Consortium; Keuning, Britt E.; Kaufmann, Thomas; Wiersema, Renske; Granholm,
Anders; Pettila, Ville; Moller, Morten Hylander; Christiansen, Christian Fynbo; Forte, Jose
Castela; Snieder, Harold
Published in:
Acta Anaesthesiologica Scandinavica
DOI:
10.1111/aas.13527
IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from
it. Please check the document version below.
Document Version
Publisher's PDF, also known as Version of record
Publication date:
2019
Link to publication in University of Groningen/UMCG research database
Citation for published version (APA):
HEALICS Consortium, Keuning, B. E., Kaufmann, T., Wiersema, R., Granholm, A., Pettila, V., Moller, M.
H., Christiansen, C. F., Forte, J. C., Snieder, H., Keus, F., Pleijhuis, R. G., & van der Horst, I. C. C. (2019).
Mortality prediction models in the adult critically ill: A scoping review. Acta Anaesthesiologica Scandinavica.
https://doi.org/10.1111/aas.13527
Copyright
Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).
Take-down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.
Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the number of authors shown on this cover page is limited to 10 maximum.
Acta Anaesthesiol Scand. 2019;00:1–19. wileyonlinelibrary.com/journal/aas
|
1Received: 13 May 2019
|
Revised: 7 October 2019|
Accepted: 4 December 2019 DOI: 10.1111/aas.13527R E V I E W
Mortality prediction models in the adult critically ill: A scoping
review
Britt E. Keuning
1| Thomas Kaufmann
2| Renske Wiersema
1|
Anders Granholm
3| Ville Pettilä
4| Morten Hylander Møller
3,5|
Christian Fynbo Christiansen
6| José Castela Forte
1,7| Harold Snieder
8|
Frederik Keus
1| Rick G. Pleijhuis
9| Iwan C. C. van der Horst
1,10|
HEALICS consortium
1Department of Critical Care, University of Groningen, University Medical Center Groningen, Groningen, The Netherlands 2Department of Anesthesiology, University of Groningen, University Medical Center Groningen, Groningen, The Netherlands 3Department of Intensive Care, Copenhagen University Hospital, Rigshospitalet, Copenhagen, Denmark
4Division of Intensive Care Medicine, Department of Anesthesiology, Intensive Care and Pain Medicine, University of Helsinki and Helsinki University Hospital, Helsinki, Finland
5Centre for Research in Intensive Care, Copenhagen University Hospital, Rigshospitalet, Copenhagen, Denmark 6Department of Clinical Epidemiology, Aarhus University Hospital, Aarhus, Denmark
7Bernoulli Institute for Mathematics, Computer Science and Artificial Intelligence, University of Groningen, Groningen, The Netherlands 8Department of Epidemiology, University of Groningen, University Medical Center Groningen, Groningen, The Netherlands
9Department of Internal Medicine, University of Groningen, University Medical Center Groningen, Groningen, The Netherlands 10Department of Intensive Care, Maastricht University Medical Center+, Maastricht University, Maastricht, The Netherlands
This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.
© 2019 The Authors. Acta Anaesthesiologica Scandinavica published by John Wiley & Sons Ltd on behalf of Acta Anaesthesiologica Scandinavica Foundation.
Correspondence
Iwan C. C. van der Horst, Department of Intensive Care, Maastricht University Medical Center+, Maastricht University, Maastricht, The Netherlands.
Email: iwan.vander.horst@mumc.nl
Funding information
This research received no specific grant from any funding agency from any sector.
Background: Mortality prediction models are applied in the intensive care unit (ICU) to stratify patients into different risk categories and to facilitate benchmarking. To ensure that the correct prediction models are applied for these purposes, the best performing models must be identified. As a first step, we aimed to establish a system-atic review of mortality prediction models in critically ill patients.
Methods: Mortality prediction models were searched in four databases using the fol-lowing criteria: developed for use in adult ICU patients in high-income countries, with mortality as primary or secondary outcome. Characteristics and performance measures of the models were summarized. Performance was presented in terms of discrimination, calibration and overall performance measures presented in the original publication. Results: In total, 43 mortality prediction models were included in the final analy-sis. In all, 15 models were only internally validated (35%), 13 externally (30%) and 10 (23%) were both internally and externally validated by the original research-ers. Discrimination was assessed in 42 models (98%). Commonly used calibration measures were the Hosmer-Lemeshow test (60%) and the calibration plot (28%).
1 | INTRODUCTION
Outcome prediction models, severity scales and risk scores are prog-nostic tools to estimate the probability for a pre-specified outcome.1 These prognostic tools use variables (eg about the severity of illness) to predict outcome, often mortality, in a specific patient population such as the critically ill. In the intensive care unit (ICU), mortality prediction models may be applied to stratify patients in different risk categories and to facilitate benchmarking using standardized mortality rates. An accurate mortality prediction model provides a stratification of the risk of an outcome at a population level. These models generally provide a numerical estimate of that risk based on estimates from previous pop-ulations.2 Per definition, all mortality prediction models are best suited for use at a population level and not for individual prognostication, as uncertainty for individual patients remains high.3,4
Several models are widely known and broadly applied such as the Acute Physiology and Chronic Health Evaluation (APACHE) I-IV, the Mortality Prediction Model (MPM) and the Simplified Acute Physiology Score (SAPS) I-III,5 whereas others like the Intensive Care National Audit & Research Centre (ICNARC) are used solely in one country.6 Previous literature has only reviewed commonly used models, models with different outcome than mortality or dis-ease- or organ-specific prognostic models.3-5,7,8 To the best of our knowledge, no study has systematically assessed which mortality prediction models have been developed and validated for broad co-horts of adult critically ill patients.
1.1 | Rationale and objective
The objective of this study was to provide an overview of available mortality prediction models in adult critically ill patients as a step-up towards future head-to-head comparison of model performance through systematic external validation.
2 | METHODS
2.1 | Protocol and registration
This scoping review was performed following our protocol (Appendix S1) and was reported in accordance with the PRISMA-ScR checklist.9
Notably, we aimed to publish the protocol on PROSPERO, but dur-ing the process it showed that PROSPERO currently does not ac-cept registrations for scoping reviews, literature reviews or mapping reviews.
2.2 | Search strategy
We conducted a systematic search of MEDLINE, EMBASE, Web of Science and The Cochrane Central Register of Controlled Trials (CENTRAL) to identify relevant ICU mortality prediction models (Appendix S1). Mortality was chosen as the outcome of interest, as prediction models were originally developed to identify patients with high mortality risk. For all databases, except the CENTRAL da-tabase, the search period encompassed a period starting from the 1st January 2008 to the 21st April 2019. We used snowballing, that is, searching references and related articles, to identify additional prediction models that were published before 2008.
One author ran the search, after which the screening of records and data extraction were performed in duplicate. All records were screened based on title and/or abstract. Papers clearly irrelevant to the purpose were excluded. The remaining articles were screened for eligibility. Consulting a third opinion solved disagreements. More detailed information is presented in the protocol (Appendix S1).
2.3 | Eligibility criteria
To be considered eligible, mortality prediction models had to meet the following criteria: (a) originally developed specifically for use in adult critically ill patients as defined by the included studies, (b) representing Calibration was not assessed in 11 models (26%). Overall performance was assessed
in the Brier score (19%) and the Nagelkerke's R2 (4.7%).
Conclusions: Mortality prediction models have varying methodology, and valida-tion and performance of individual models differ. External validavalida-tion by the original researchers is often lacking and head-to-head comparisons are urgently needed to identify the best performing mortality prediction models for guiding clinical care and research in different settings and populations.
Editorial Comment
In this review, mortality prediction models in intensive care have been identified. Characteristics and performance of 43 individual models are summarized according to docu-mentation in the original publications so that validation and predictive performances can be compared.
broad groups of ICU patients (with large diversity of admission diagno-ses, eg non-diabetic patients, medical admissions, surgical admissions, etc), (c) availability of the original article in English and (d) mortality at any time as (primary or secondary) outcome of interest.
Prediction models were excluded (a) when developed for low- or middle-income countries, as characteristics of ICU patients in these countries often substantially differ from those in high-income coun-tries and, epidemiological data from low-income councoun-tries have been frequently unavailable,10,11 (b) when developed as a digital model or derived from a machine-learning algorithm, since code and data avail-ability are not requirements in all journals. Since our utmost goal is to make a head-to-head comparison of available mortality prediction models using an independent external validation cohort, the code or data necessary to retrieve the underlying prediction model formula are required to reproduce the prediction models. (c) When the develop-ment of multiple customized prediction models was described in one article, but no final model was proposed, the prediction models were excluded. Finally, (d) we excluded prediction models specifically devel-oped for subgroups of intensive care patients such as those with sep-sis, trauma, cardiac and neurological patients. Studies not specifying inclusion of these subgroups within a wider, general ICU population were considered to be eligible. Prediction models developed in a med-ical or surgmed-ical ICU were included.
2.4 | Data extraction
If multiple mortality outcomes (eg at different time points) were used, we used the primary outcome in the original publication (or the first mortality outcome if the primary outcome was not mortality) to describe the performance of the prediction model.
Details on the development process of the mortality prediction models included were shown, as well as the number of variables in-cluded in the prediction models, mortality rate in each development setting and method of handling of missing data. To give an overview of the performance of all mortality prediction models, for example, values from discrimination, calibration and overall performances measures12 for mortality were presented for development and in-ternal or exin-ternal validation cohorts in the original publication (if available).
The discrimination measure presented was the C-statistic (area under the receiver operating characteristic curve [AUROC]), calibra-tion measures presented were goodness-of-fit tests like the Hosmer-Lemeshow (HL) test, calibration plot and calibration slope, and the overall performance measures presented were the Nagelkerke's R2 and the Brier score.12
Preferable values from external validation were presented if both internal and external validation values were present in the
original publication. If not available, values of internal validation cohorts were presented. External validation was defined as using a separate individual dataset for validation of the mortality prediction model (ie no split sampling of a dataset also used for the develop-ment of the model).
Citations of original publications were screened for internal and/ or external validation articles and shown as being present (+) or ab-sent (−). A list of variables sought for in the identified articles can be found in Appendix S1.
3 | RESULTS
The selection of sources of evidence can be found in the flowchart (Figure 1). Articles evidently developed for specific groups of pa-tients (ie sepsis, trauma, cardiac, neurological papa-tients) were ex-cluded based on the title and/or abstract. Evaluating 99 full-text articles for eligibility resulted in exclusion of another 39 articles, leaving 60 articles that were screened for original publications. Eventually, 43 relevant mortality prediction models reported in 38 publications were extracted and included in the final analysis.
3.1 | Characteristics of the included mortality
prediction models
Characteristics of the mortality prediction models and underlying derivation cohorts are presented in Table 1. In all, 19 mortality pre-diction models (44%) were developed using prospectively collected data specifically gathered for the development of the prediction model,6,13-27 whereas 24 (56%) were developed using either retro-spective data28-44 or prospective data previously collected for other purposes.45-49 The start of data collection for the development co-horts spanned 36 years (1979-2015), and the duration of the cohort studies varying from 2 months up to 10 years for each cohort. Two mortality prediction models (4.7%) did not report the timespan dur-ing which their development cohort was assembled.22,33 In all, 31 mortality prediction models (74%) were developed in a single coun-try,14,18-27,29,31,33-45,47,49 six (14%) in neighbouring countries (two or more)6,13,28,30,32,46 and five (12%) were developed in multiple coun-tries worldwide.15-17,48 The number of patients included in the de-velopment databases ranged from 232 to 731 611 patients with a median of 4,895 (IQR 528-35 878). The minimum age at which pa-tients were included was 15 years (2.3%).35 In all, 11 mortality pre-diction models (26%) did not specify age.6,13,23,25,29,31,36,38,42,46 The number of variables included in the mortality prediction models var-ied from 5 up to 5695, with a median of 16 (IQR 9-24).
3.2 | Outcome measures
The timing of mortality outcome varied between the stud-ies. Hospital mortality was the most frequently used
primary outcome in 29 (67%) mortality prediction mod-els.6,13-19,21,22,24,27,28,30-33,35,36,38,41-43,45,46 Other primary out-come variables were ICU mortality (7%),23,26,34 28-day mortality (4.7%),39,44 90-day mortality (4.7%),48,49 3- to 28-day mortality (4.7%),40 30-day mortality (2.3%),47 180-day mortality (2.3%),20 6-month mortality (2.3%),25 15-year mortality (2.3%),37 and 6- and 12-month mortality (2.3%).29
Secondary outcomes were 1-month mortality after ICU admis-sion (4.7%),24,31 hospital mortality (4.7%),29,34 ICU mortality (2.3%),45 3-month mortality after ICU admission (2.3%),31 6-month mortality after ICU admission (2.3%),31 9-month mortality (2.3%),47 1-year mortality (2.3%)45 and length of stay (2.3%).24 Of the 43, 37 mor-tality prediction models (86%) did not prognosticate any secondary outcome.6,13-23,25-28,30,32,33,35-44,46,48,49
Hospital mortality rates of the development cohorts varied from 6.9% to 48% and were not reported for nine mortality prediction models (21%).6,15,18,29,33,40,42
For 21 mortality prediction models (49% of 43), data were collected within the first 24 hours after patient admission to the ICU.6,13,14,17-19,24,26,27,30,31,34,38,39,42,44,47-49 For 11 prediction models (26%), data on ICU admission were collected,16,23,25,28,32,35,36,41,43,45,46 whereas for the remaining prediction models data timing varied from 24 days before admission up to 5 days after patient admission to the ICU.
Handling of missing data was not reported in 11 mortal-ity prediction models (26%),23,25,26,31,33,38,39,41,45,46,49 20 pre-diction models (47% of 43) excluded records with missing data,6,14,16,19,21,24,27,28,30,32,34,40,42-44 six prediction models (14%) imputed values with normal or mean values15,17,18,20,22,29 and four prediction models (9.3%) reported no missing data.13,35-37 The re-maining two prediction models (4.7%) excluded patients when more than a certain percentage of the data was missing (>5% or >25%).47,48
3.3 | Discrimination, calibration and overall
performance measures
Discrimination, calibration and overall performance measures are presented in Table 2. Of the 43 mortality prediction models, 15 (35%) were only internally validated,23,26,28-31,33,38-41,44,46,48 13 (30%) only externally,16,19-21,25,35,36,42,43,47 10 (23%) were both inter-nally and exterinter-nally validated,6,13-15,17,18,22,32,34,37 and 5 prediction models (12%) were not validated at all.24,27,45,49 In all, 15 prediction models (35%) included a description of an external validation in their original publication.13,16,20-22,25,34-36,42,43,47
Discrimination was expressed as the AUROC in 42 of the 43 mor-tality prediction models original publications (98%). Only the APACHE II model did not report an AUROC value in the original publication.19 In the development cohorts, the lowest discrimination was AUROC 0.72 (95% CI 0.71-0.74),48 and the highest AUROC 0.91 (95% CI not speci-fied).30 In the validation cohorts, the lowest AUROC was 0.58 (95% CI not specified),44 and the highest AUROC 0.95 (0.91-0.99).23
Calibration measures were expressed by various statistical mea-sures. The HL goodness-of-fit test was used in 26 mortality prediction
T A B LE 1 C ha ra ct er is tic s o f t he d ev el op m en t o f t he 4 3 m or ta lit y p re di ct io n m od el s Mor tal it y pr ed ic tion m ode l Yea r pu bl is hed D ev elopm en t da ta bas e C oh or t a ss em bl y pe rio d IC U po pu la tio n N umb er o f va ria ble s a O ut co me H os pi tal mo rt al it y ra te i n e ac h de ve lopm en t se tti ng D at a c ol le ct io n H an dl in g o f m issi ng d at a Pri m ar y Se co nda ry IC N A RC H ar ris on e t a l 6 20 07 21 6 62 6 Pros pe ct iv e D ec em ber 19 95 -A ug us t 2 00 3 G en er al , a du lt p a-tie nt s i n E ng la nd , W al es a nd I re la nd 16 H os pi ta l m or ta lit y – N ot r ep or te d W or st v al ue s an d t ot al u rin e ou tp ut i n i ni tia l 24 h i n I C U Ex cl us io n IC N A RC-II Fe rr an do -V iv as et a l 13 20 17 15 5 23 9 Pros pe ct iv e 01 /01 /2 01 2-31/ 12 /2 012 G en er al , a du lt p a-tie nt s i n E ng la nd , W al es a nd I re la nd 23 H os pi ta l m or ta lit y – 32 0 64 /1 55 2 39 (2 0.7 % ) W or st v al ue s an d t ot al u rin e ou tp ut i n i ni tia l 24 h i n I C U N o m is si ng d at a A PA C H E I V Zi m m er m an e t al 14 20 06 66 2 70 Pros pe ct iv e 01 /01 /2 00 2-31/ 12 /2 00 3 G en er al , a du lt (≥ 16 y ) p at ie nt s i n th e U SA 14 2 H os pi ta l m or ta lit y – 9, 01 3/ 66 ,2 70 b (1 3. 6% ) W or st v al ue s i n in iti al 2 4 h i n ICU Ex cl us io n SA PS I II M or en o e t a l 15 20 05 13 4 28 b Pros pe ct iv e 14 /1 0/ 20 02 -15 /1 2/ 20 02 G en er al , a du lt (≥ 16 y ) p at ie nt s w or ld w ide 20 H os pi ta l m or ta lit y – N ot r ep or te d IC U adm is si on ± 1 h Im pu ta tio n o f no rmal valu es M PM 0 -I I Le m es ho w e t a l 16 19 93 12 61 0 Pros pe ct iv e 17 /0 4/1 98 9-31 /0 7/ 19 90 ( da ta se t I) a nd 30 /0 9/ 19 91 -27 /1 2/ 19 91 ( da ta se t II) G en er al , a du lt (≥ 18 y ) p at ie nt s in E ur op e a nd t he US A 15 H os pit al m or ta lit y – 26 32 /1 2 61 0 (2 0.9 % ) IC U a dm is si on Ex cl us io n M PM 24 -I I Le m es ho w e t al 16 ,2 1 19 93 10 3 57 Pros pe ct iv e 17 /0 4/1 98 9-31 /0 7/ 19 90 ( da ta se t I) a nd 3 0/ 09 /1 99 1-27 /1 2/ 19 91 ( da ta se t II) G en er al , a du lt (≥ 18 y ) p at ie nt s in E ur op e a nd t he US A 13 H os pi ta l m or ta lit y – 22 61 /1 0 35 7 (2 1. 8% ) A t 2 4 h i n I C U Ex cl us io n SA PS I I Le G al l e t a l 17 19 93 83 69 Pros pe ct iv e 30 /0 9/ 19 91 -28 /0 2/ 19 92 G en er al , a du lt (≥ 18 y ) p at ie nt s in E ur op e a nd No rt h-A mer ic a 17 H os pit al m or ta lit y – 18 24 /8 36 9 b (2 1. 8% ) W or st v al ue s i n in iti al 2 4 h i n ICU Im pu ta tio n o f no rmal valu es A PA C H E III K na us e t a l 18 19 91 78 48 b Pros pe ct iv e M ay 1 98 8-N ov em be r 19 89 G en er al , a du lt (≥ 16 y ) p at ie nt s i n th e U SA 26 H os pi ta l m or ta lit y – N ot r ep or te d W or st v al ue s i n in iti al 2 4 h i n ICU Im pu ta tio n o f no rmal valu es A PA C H E I I K na us e t a l 19 19 85 50 30 Pros pe ct iv e 19 79 -19 82 G en er al , a du lt (≥ 16 y ) p at ie nt s i n th e U SA 18 H os pit al m or ta lit y – 99 3/5 03 0 (1 9. 7% ) W or st v al ue s i n in iti al 2 4 h i n ICU Ex cl us io n (Co nti nue s)
Mor tal it y pr ed ic tion m ode l Yea r pu bl is hed D ev elopm en t da ta bas e C oh or t a ss em bl y pe rio d IC U po pu la tio n N umb er o f va ria ble s a O ut co me H os pi tal mo rt al it y ra te i n e ac h de ve lopm en t se tti ng D at a c ol le ct io n H an dl in g o f m issi ng d at a Pri m ar y Se co nda ry SU PP OR T K na us e t a l 20 19 95 43 01 Pros pe ct iv e Ju ne 1 98 9-Ju ne 1 99 1 G en er al , a du lt (≥ 18 y ) p at ie nt s i n th e U SA 15 18 0-da y m or ta lit y – 20 72 /4 30 1 (4 8. 2% ) A ft er 3 d ay s Im pu ta tio n o f no rmal valu es , m is si ng d at a at d ay 3 w er e im pu te d w ith da y 1 valu es M PM 48 -I I Le m es ho w e t a l 21 19 94 20 49 Pros pe ct iv e 17 /0 4/1 98 9-31 /0 7/ 19 90 G en er al , a du lt (≥ 18 y ) p at ie nt s i n th e U SA 13 H os pi ta l m or ta lit y – 30 7/ 20 49 b (1 5. 0% ) A t 4 8 h i n I C U Ex cl us io n M PM 72 -I I Le m es ho w e t a l 21 19 94 1497 Pros pe ct iv e 17 /0 4/1 98 9-31 /0 7/ 19 90 G en er al , a du lt (≥ 18 y ) p at ie nt s i n th e U SA 13 H os pi ta l m or ta lit y – 41 8/ 1497 b (2 7. 9% ) A t 7 2 h i n I C U Ex cl us io n TR IOS Ti m si t e t a l 22 20 01 893 Pros pe ct iv e N ot r ep or te d ( va lid a-tio n d at as et i n M ar ch 1999 ) G en er al , a du lt (≥ 16 y ) p at ie nt s, ho sp ita liz ed > 48 h in F ra nc e 32 H os pi ta l m or ta lit y – 26 8/ 89 3 ( 30 .0 % ) Fi rs t 3 d ay s i n ICU Im pu ta tio n o f no rmal valu es M or ta lit y R is k Sco re D óle ra -M or en o et a l 23 20 16 84 4 Pros pe ct iv e Jan uar y 20 13 -A pr il 20 14 G en er al , a du lt pa tie nt s i n S pa in 6 IC U m or ta lit y – 91 /8 44 ( 10 .8 % ) IC U a dm is si on N ot r ep or te d M or ta lit y M ult ifa ct or M od el Li e t a l 24 20 17 50 0 Pros pe ct iv e 01 /0 3/2 01 4-30 /0 4/ 20 14 G en er al , a du lt (≥ 18 y ) p at ie nt s i n C hin a 36 H os pit al m or ta lit y M or ta lit y 30 d ay s af te r I C U ad m is -si on , L O S 10 2/ 50 0 ( 20 .4 % ) Fi rs t 2 4 h i n I C U Ex cl us io n M or ta lit y Pro gn os tic M od el H ad iq ue e t a l 25 20 17 50 0 Pros pe ct iv e No ve m ber 2 01 3-A pr il 20 14 M ed ic al , a du lt p a-tie nt s i n t he U SA 44 6-m on th m or ta lit y -18 0/5 00 (3 6. 0% ) IC U a dm is si on , SQ w ithin 12 -2 4 h o f ad m is si on N ot r ep or te d M or ta lit y Pre di ct io n M od el Fi ka e t a l 26 20 18 40 0 Pros pe ct iv e Jan uar y 20 12 -J ul y 20 13 G en er al , a du lt (≥ 18 y ) p at ie nt s i n G re ece 12 IC U m or ta lit y – 13 1/ 40 0 ( 23 .8 % ) W or st v al ue s i n in iti al 2 4 h i n ICU N ot r ep or te d A PA C H E II-A PM Nem ati fa rd e t a l 27 20 18 30 4 Pros pe ct iv e Ju ne 2 01 4-N ov em be r 20 16 G en er al , a du lt (≥ 16 y ) p at ie nt s in I ra n 19 H os pi ta l m or ta lit y – 96 /3 04 ( 31 .6 % ) W or st v al ue s i n in iti al 2 4 h i n ICU Ex cl us io n (Co nti nue s) T A B LE 1 (Co nti nue d)
Mor tal it y pr ed ic tion m ode l Yea r pu bl is hed D ev elopm en t da ta bas e C oh or t a ss em bl y pe rio d IC U po pu la tio n N umb er o f va ria ble s a O ut co me H os pi tal mo rt al it y ra te i n e ac h de ve lopm en t se tti ng D at a c ol le ct io n H an dl in g o f m issi ng d at a Pri m ar y Se co nda ry A PA C H E III -A PM Nem ati fa rd e t a l 27 20 18 30 4 Pros pe ct iv e Ju ne 2 01 4-N ov em be r 20 16 G en er al , a du lt (≥ 16 y ) p at ie nt s in I ra n 27 H os pi ta l m or ta lit y – 96 /3 04 ( 31 .6 % ) W or st v al ue s i n in iti al 2 4 h i n ICU Ex cl us io n A N ZRO D 0 Pa ul e t a l 28 20 17 73 1 61 1 Ret ros pe ct iv e 01 /01 /2 00 6-31/ 12 /2 01 5 G en er al , a du lt (≥ 16 y ) p at ie nt s i n A us tr al ia a nd N ew Zea la nd 11 H os pi ta l m or ta lit y – 69 5 03 /7 31 6 11 b (9 .5% ) IC U a dm is si on Ex cl us io n MMI Min e t a l 29 20 17 35 4 15 4 b Ret ros pe ct iv e Jan uar y 20 03-D ec em be r 20 13 M ed ic al , v et er an IC U p at ie nt s i n t he US A 569 5 A ll-ca us e m or -ta lit y a t 6 - a nd 12 -m ont hs pos t-hos pit al di sc ha rg e H os pit al m or ta lit y N ot r ep or te d W or st v al ue s of 2 4 h b ef or e an d 2 4 h a ft er ad m is si on Im pu ta tio n o f m ea n valu es A N ZRO D Pa ul e t a l 30 20 13 30 4 14 9 Ret ros pe ct iv e 01 /01 /2 00 4-31/ 12 /2 00 9 G en er al , a du lt (≥ 16 y ) p at ie nt s i n A us tr al ia a nd N ew Zea la nd 38 H os pi ta l m or ta lit y – 34 3 69 b (11 .3 % ) W or st v al ue s i n in iti al 2 4 h i n ICU Ex cl us io n C us to m iz ed A PA C H E I V B rin km an e t a l 31 20 13 77 6 16 Ret ros pe ct iv e 01 /01 /2 00 8-01 /0 7/ 201 1 N on -C A BG , ad ul t c rit ic al ly i ll pa tie nt s i n t he N et he rla nds 14 2 H os pit al m or ta lit y M or ta lit y at 1 , 3 a nd 6 m on th s af te r I C U ad m is si on 12 1 86 /7 7 61 6 b (1 5. 7% ) Fi rs t 2 4 h i n I C U N ot r ep or te d M PM 0 -III H ig gi ns e t a l 32 20 05 74 5 78 Ret ros pe ct iv e O ct ob er 20 01 -M arc h 20 04 G en er al , a du lt (≥ 18 y ) p at ie nt s i n th e U SA , C an ad a an d B ra zi l 16 H os pit al m or ta lit y – 10 2 92 /7 4 57 8 (1 3. 8% ) IC U a dm is si on Ex cl us io n NQ F-IC OM m or t Ph ili p R . L ee In st itu te 33 20 16 40 3 95 Ret ros pe ct iv e N ot r ep or te d G en er al , a du lt (≥ 18 y ) p at ie nt s i n th e U SA 17 H os pit al m or ta lit y – N ot r ep or te d 1 h p rio r t o I C U ad m is si on t o 1 h af te r a dm is si on N ot r ep or te d OA SI S Jo hn so n e t a l 34 20 13 39 0 70 Ret ros pe ct iv e 01 /01 /2 00 7-15 /0 9/ 20 11 G en er al , a du lt (≥ 16 y ) p at ie nt s i n th e U SA 10 IC U m or ta lit y H os pit al m or ta lit y 45 71 /3 9 07 0 b (11 .7 % ) W or st v al ue s an d t ot al u rin e ou tp ut i n i ni tia l 24 h i n I C U Ex cl us io n C O PE-4 D uk e e t a l 35 20 13 35 8 78 Ret ros pe ct iv e 01 /0 7/ 20 04 -30 /0 6/ 20 06 G en er al , a du lt (≥ 15 y ) p at ie nt s i n A us tr al ia 6 H os pit al m or ta lit y – 44 15 /3 5 87 8 (1 2. 3% ) IC U a dm is si on (m ec ha ni ca l v en -til at io n du ring IC U a dm is si on ) N o m is si ng d at a (Co nti nue s) T A B LE 1 (Co nti nue d)
Mor tal it y pr ed ic tion m ode l Yea r pu bl is hed D ev elopm en t da ta bas e C oh or t a ss em bl y pe rio d IC U po pu la tio n N umb er o f va ria ble s a O ut co me H os pi tal mo rt al it y ra te i n e ac h de ve lopm en t se tti ng D at a c ol le ct io n H an dl in g o f m issi ng d at a Pri m ar y Se co nda ry RD W -S AP S H un zi ke r e t a l 45 201 2 17 9 22 Ret ros pe ct iv e Januar y 20 01 -D ec em be r 20 08 G en er al , a du lt (≥ 18 y ) p at ie nt s i n th e U SA 15 H os pit al m or ta lit y IC U mor ta lit y, 1-ye ar m or ta lit y 20 07 /17 9 22 b (11 .2 % ) IC U a dm is si on N ot r ep or te d CO PE D uk e e t a l 36 20 08 17 8 80 Ret ros pe ct iv e 01 /0 7/ 20 04 -30 /0 6/ 20 05 G en er al , a du lt p a-tie nt s i n A us tr al ia 5 H os pi ta l m or ta lit y – 21 86 /1 7 88 0 (1 2. 1% ) IC U a dm is si on (m ec ha ni ca l v en -til at io n du ring IC U a dm is si on ) N o m is si ng d at a PR EDIC T H o e t a l 37 20 08 11 9 30 Ret ros pe ct iv e 19 89 -20 02 G en er al , a du lt (≥ 16 y ) p at ie nt s i n A us tr al ia 6 15 -y ea r m or ta lit y – 82 9/ 11 9 30 b (6 .9 % ) Fi rs t 5 d ay s i n ICU N o m is si ng d at a H igh -R is k Se le ct ion Sy st em Ia pi ch in o e t a l 46 20 06 824 8 Ret ros pe ct iv e O ct ob er 19 94 -F eb ru ar y 19 95 G en er al , a du lt pa tie nt s ( >2 4 h i n IC U ) i n E ur op e 16 H os pi ta l m or ta lit y – 16 17/ 82 48 b (1 9. 6% ) IC U a dm is si on N ot r ep or te d G V-SA PS I I Li u e t a l 47 20 16 48 95 Ret ros pe ct iv e 20 01 -20 08 N on -d ia be tic , a du lt (≥ 18 y ) p at ie nt s i n th e U SA 20 30 -d ay m or ta lit y 9-m on th m or ta lit y 64 9/ 48 95 (1 3. 3% ) Fi rs t 2 4 h i n I C U W he n > 5% ex cl us io n, < 5% no t r ep or te d M O D S/ N EMS K ao e t a l 38 20 16 43 21 Ret ros pe ct iv e 01 /01 /2 00 9-30 /1 1/ 20 12 G en er al , a du lt p a-tie nt s i n C an ad a 32 H os pi ta l m or ta lit y – 98 6/ 432 1 (2 2. 8%) Fi rs t 2 4 h i n I C U N ot r ep or te d SM S-IC U G ra nh ol m e t a l 48 20 18 40 86 Ret ros pe ct iv e 23/ 12 /2 00 9-30 /0 6/ 20 16 G en er al , a du lt (≥ 18 y ), a cu te ly ad m itt ed p at ie nt s w or ld w ide 7 90 -d ay m or ta lit y – 140 3/ 40 86 (3 4. 3% ) W or st v al ue s i n in iti al 2 4 h i n ICU M ul tipl e im pu ta -tio ns , e xcl us io n w he n > 25 % P- m od el U m eg ak i e t a l 39 20 10 35 05 Ret ros pe ct iv e 01 /01 /2 00 7-31/ 12 /2 00 7 G en er al , a du lt (≥ 20 y ) p at ie nt s i n Jap an 10 M or ta lit y a t 28 d ay s a ft er th e f irs t I C U day – 33 6/ 350 5 b (9. 6% ) Fi rs t 2 4 h i n I C U N ot r ep or te d B C V m od el H ua ng e t a l 40 20 13 16 24 Ret ros pe ct iv e 01 /01 /2 00 6-01/ 12 /2 00 8 G en er al , a du lt (≥ 18 y ) p at ie nt s i n Ta iwa n 6 D ai ly p ro ba bi lit y of m or ta lit y fr om d ay 3 t o da y 2 8 p os t-IC U a dm is si on – N ot r ep or te d D ai ly c om pl et e bl oo d c ou nt Ex cl us io n (Co nti nue s) T A B LE 1 (Co nti nue d)
Mor tal it y pr ed ic tion m ode l Yea r pu bl is hed D ev elopm en t da ta bas e C oh or t a ss em bl y pe rio d IC U po pu la tio n N umb er o f va ria ble s a O ut co me H os pi tal mo rt al it y ra te i n e ac h de ve lopm en t se tti ng D at a c ol le ct io n H an dl in g o f m issi ng d at a Pri m ar y Se co nda ry B C V/ A PA C H E I I m ode l H ua ng e t a l 40 20 13 16 24 Ret ros pe ct iv e 01 /01 /2 00 6-01/ 12 /2 00 8 G en er al , a du lt (≥ 18 y ) p at ie nt s i n Ta iwa n 24 D ai ly p ro ba bi lit y of m or ta lit y fr om d ay 3 t o da y 2 8 p os t-IC U a dm is si on – N ot r ep or te d D ai ly c om pl et e bl oo d c ou nt , A PA C H E I I s co re in t he f irs t 2 4 h in I C U Ex cl us io n C RE EK St ac ho n e t a l 41 20 08 52 8 Ret ros pe ct iv e A pr il 20 03 -J anuar y 20 04 M ed ic al , a du lt (≥ 18 y ) p at ie nt s i n G er m any 8 H os pi ta l m or ta lit y – 87/ 52 8 (1 6. 5% ) IC U a dm is si on N ot r ep or te d SA PS -R V iv ia nd e t a l 42 19 91 351 Ret ros pe ct iv e 01 /01 /1 98 6-31 /1 0/1 98 8 G en er al , a du lt p a-tie nt s i n F ra nc e 5 H os pi ta l m or ta lit y – N ot r ep or te d W or st v al ue s i n in iti al 2 4 h i n ICU Ex cl us io n SA PS -E V iv ia nd e t a l 42 19 91 351 Ret ros pe ct iv e 01 /01 /1 98 6-31 /1 0/1 98 8 G en er al , a du lt p a-tie nt s i n F ra nc e 7 H os pi ta l m or ta lit y – N ot r ep or te d W or st v al ue s i n in iti al 2 4 h i n ICU Ex cl us io n 25 O H D D ey o-C har ls on C om or bidit y In de x M ah at o e t a l 49 20 16 310 Ret ros pe ct iv e 01 /0 6/ 20 12 -30 /0 5/ 20 15 G en er al , a du lt (≥ 18 y ) p at ie nt s i n th e U SA 18 90 -d ay m or ta l-ity a ft er I C U ad m is si on – 59 /3 10 (1 9. 0% ) Fi rs t 2 4 h i n I C U N ot r ep or te d D EL AW ARE St ac ho n e t a l 43 20 08 271 Ret ros pe ct iv e A pr il 20 03 -J anuar y 20 04 Su rg ic al , a du lt (≥ 18 y ) p at ie nt s i n G er m any 9 H os pi ta l m or ta lit y – 67 /2 71 ( 24 .7 % ) IC U a dm is si on Ex cl us io n Si m pl ifie d M or ta lit y S co re G oa g e t a l 44 20 18 232 Ret ros pe ct iv e Ju ne 2 01 5-Fe br ua ry 20 16 M ed ic al , a du lt (≥ 18 y ) p at ie nt s i n Ko re a 8 28 -d ay m or ta lit y – 72 /232 b (3 1.1 % ) W ith in 2 4 h o f IC U a dm is si on Ex cl us io n A bb re vi at io ns : A N ZR O D , A us tr al ia n a nd N ew Z ea la nd R is k O f D ea th ; A PA C H E, A cu te P hy si ol og y a nd C hr on ic H ea lth E va lu at io n; A PM , a dd uc to r p ol lic is m us cl e; B C V, b lo od c el l v ar ia bi lit y; C O PE , c rit ic al ca re o ut co m e p re di ct io n e qu at io n; C RE EK , c rit ic al r is k e va lu at io n b y e ar ly k ey s; D EL AW A RE , D en se L ab or at or y W ho le B lo od A pp lie d R is k E st im at io n; G V, g lu co se v ar ia bi lit y; I C N A RC , I nt en si ve C ar e N at io na l A ud it R es ea rc h C en tr e; I C U , i nt en si ve c ar e u ni t; L O S, l en gt h o f s ta y; M M I, m ul ti-m or bi di ty i nd ex ; M O D S, m ul tip le o rg an s d ys fu nc tio na l s co re ; M PM ; m or ta lit y p re di ct io n m od el ; N EM S, n in e eq ui va le nt s n ur si ng m an po w er u se s co re ; N Q F-IC O M m or t, n at io na l q ua lit y f or um I C U o ut co m es m od el ( m or ta lit y) ; O A SI S, o xf or d a cu te s ev er ity o f i lln es s s co re ; P RE D IC T, p re di ct ed r is k, e xi st in g d is ea se s an d i nt en si ve c ar e t he ra py ; R D W , r ed c el l d is tr ib ut io n w id th ; S A PS , s im pl ifi ed a cu te p hy si ol og y s co re ; S M S-IC U , s im pl ifi ed m or ta lit y s co re f or t he i nt en si ve c ar e u ni t; S Q , s ur pr is e q ue st io n; S U PP O RT , st ud y t o u nd er st an d p ro gn os es a nd p re fe re nc es f or o ut co m es a nd r is ks o f t re at m en ts ; T RI O S, t hr ee -d ay r ec al ib ra tin g I C U o ut co m es . aW he n ( pa rt s o f) o th er m or ta lit y p re di ct io n m od el s w er e u se d a s v ar ia bl es i n a m or ta lit y p re di ct io n m od el ( eg t he C ha rls on C om or bi di ty I nd ex a nd A PA C H E I II a s v ar ia bl e i n t he M or ta lit y P ro gn os tic M od el ), v ar ia bl es i nc lu de d i n t he se s pe ci fic m or ta lit y p re di ct io n m od el s w er e a ls o t ak en i nt o a cc ou nt . bEs tim at ed b as ed o n i nf or m at io n i n o rig in al p ub lic at io n. T A B LE 1 (Co nti nue d)
T A B LE 2 Pe rf or m an ce o f t he 4 3 m or ta lit y p re di ct io n m od el s Mor tal it y pr ed ic tio n m od el V al id at ed ? a A U RO C ( 95 % C I) D ev el op me nt co ho rt b C alib ra tio n D ev el op me nt co ho rt b O ver al l per for ma nc e D ev el op me nt co ho rt b Ty pe o f v al id at io n co ho rt i n o rig in al pu bl ic at io n A U RO C ( 95 % C I) V al ida tio n co ho rt C alib ra tio n V al ida tio n co ho rt O ver al l per for ma nc e V alid at io n co ho rt In ter nal ly Ex ter nal ly IC N A RC H ar ris on e t a l 6 + Dat a s pli tt ing + − − − In te rn al val id at io n dat as et 0. 87 ( n. s. ) − B rie r s co re : 0.1 32 IC N A RC-II Fer ra nd o-V iv as et a l 13 + Boot st ra pp in g + Orig in al pu bl ic at io n 0. 89 (0 .89 -0 .89 ) − B rie r s co re : 0.1 03 Ex te rn al v al id at io n dat as et 0. 89 (0. 88 -0. 89 ) C al ib ra tio n p lo t pr es ent B rie r s co re : 0.1 08 A PA C H E I V Zi m m er m an e t a l 14 + Dat a s pli tt ing + − − − Ex te rn al v al id at io n dat as et 0. 88 ( n. s. ) H L X 2: 1 6. 8 ( P = .0 8) − SA PS I II M or en o e t a l 15 + Cros s-va lid at io n + − − − In te rn al val id at io n dat as et 0. 85 ( n. s. ) H L H -s ta tis tic : 1 0. 6 (P = .3 9) H L C -s ta tis tic : 1 4. 3 (P = .1 6) C al ib ra tio n p lo t pr es ent − M PM 0 -I I Le m es ho w e t a l 16 − + Orig in al pu bl ic at io n 0. 84 ( n. s. ) H L C -s ta tis tic : 6. 2 ( P = .6 2) − Ex te rn al v al id at io n dat as et 0. 82 ( n. s. ) H L C -s ta tis tic : n .s . (P = .3 3) − M PM 24 -I I Le m es ho w e t a l 16 ,2 1 − + Orig in al pu bl ic at io n 0. 84 ( n. s. ) H L C -s ta tis tic : 4. 9 ( P = .7 6) − Ex te rn al v al id at io n dat as et 0. 84 ( n. s. ) H L C s ta tis tic : 1 2. 9 (P = .2 3) − SA PS I I Le G al l e t a l 17 + Dat a s pli tt ing + 0. 88 (0. 87 -0. 90 ) H L H -s ta tis tic : 3. 70 ( P = .8 8) − In te rn al val id at io n dat as et 0. 86 (0. 84 -0. 88 ) H L H s ta tis tic : n .s . (P = .1 0) − A PA C H E III K na us e t a l 18 + Dat a s pli tt ing + − − − In te rn al val id at io n dat as et 0. 90 ( n. s. ) c − − A PA C H E I I K na us e t a l 19 − + − − − − − − − SU PP OR T K na us e t a l 20 − + Orig in al pu bl ic at io n 0.7 9 (n .s .) − − Ex te rn al v al id at io n dat as et 0. 78 ( n. s. ) C al ib ra tio n p lo t pr es ent − M PM 48 -I I Le m es ho w e t a l 21 − + Orig in al pu bl ic at io n 0. 81 ( n. s. ) H L C -s ta tis tic : 11 .7 ( P = .3 1) − Ex te rn al v al id at io n dat as et 0. 80 ( n. s. ) H L C s ta tis tic : 8 .4 (P = .5 9) − M PM 72 -I I Le m es ho w e t a l 21 − + Orig in al pu bl ic at io n 0.7 9 (n .s .) H L C -s ta tis tic : 11 .6 ( P = .3 1) − Ex te rn al v al id at io n dat as et 0.7 5 (n .s .) H L C s ta tis tic : 1 0. 4 (P = .4 1) − (Co nti nue s)
Mor tal it y pr ed ic tio n m od el V al id at ed ? a A U RO C ( 95 % C I) D ev el op me nt co ho rt b C alib ra tio n D ev el op me nt co ho rt b O ver al l per for ma nc e D ev el op me nt co ho rt b Ty pe o f v al id at io n co ho rt i n o rig in al pu bl ic at io n A U RO C ( 95 % C I) V al ida tio n co ho rt C alib ra tio n V al ida tio n co ho rt O ver al l per for ma nc e V alid at io n co ho rt In ter nal ly Ex ter nal ly TR IOS Ti m si t e t a l 22 + Boot st ra pp in g + Orig in al pu bl ic at io n 0. 79 (0. 77 -0. 82 ) H L C -s ta tis tic : 5. 6 ( P = .7 0) − Ex te rn al v al id at io n dat as et 0. 83 ( 0. 78 -0 .8 7) − − M or ta lit y R is k Sc or e D ól er a-Mo reno et a l 23 + Dat a s pli tt ing − − − − In te rn al val id at io n dat as et 0.9 5 (0 .9 1-0.9 9) Li ke lih oo d r at io t es t X 2: 2 96 .8 c − M or ta lit y M ul tif ac to r Mo de l Li e t a l 24 − − 0. 84 (0. 80 -0. 87 ) H L X 2: 1 2. 3 (P = .1 4) C al ib ra tio n p lo t pr es ent − − − − − M or ta lit y Pr og no st ic Mo de l H ad iq ue e t a l 25 − + Orig in al pu bl ic at io n 0. 83 (0. 80 -0. 87 ) H L s ta tis tic : 6 .5 (P = .5 9) − Ex te rn al v al id at io n dat as et 0. 84 (0. 81 -0. 88 ) H L s ta tis tic : 9 .2 (P = .3 3) − M or ta lit y Pr edic tio n M od el Fi ka e t a l 26 + Dat a s pli tt ing − − − − In te rn al val id at io n dat as et 0. 85 (0. 73 -0. 97 ) H L X 2: 4 .9 ( P = .7 7) − A PA C H E I I-A PM N em at ifa rd e t a l 27 − − 0. 85 (0. 81 -0. 90 ) − − − − − − A PA C H E III -A PM N em at ifa rd e t a l 27 − − 0. 87 (0. 82 -0. 91 ) − − − − − − A N ZRO D 0 Pa ul e t a l 28 + Dat a s pli tt ing − 0. 85 (0. 85 -0. 86 ) H L C -s ta tis tic : 45 9. 3 B rie r s co re : 0.0 69 A dj us te d B rie r s co re : 0. 19 6 In te rn al val id at io n dat as et 0. 85 (0 .85 -0 .85 ) H L C -s ta tis tic : 26 4.9 C al ib ra tio n p lo t pr es ent B rie r s co re : 0.0 69 A dj us te d B rie r sc or e: 0. 190 MMI Min e t a l 29 + Dat a s pli tt ing − − − − In te rn al val id at io n dat as et 6-m on th m or ta lit y: 0. 86 (0. 85 -0. 86 ) 12 -m ont h m or ta lit y: 0 .8 4 (0. 83 -0. 84 ) − B rie r s co re : 0. 21 c,d (Co nti nue s) T A B LE 2 (Co nti nue d)
Mor tal it y pr ed ic tio n m od el V al id at ed ? a A U RO C ( 95 % C I) D ev el op me nt co ho rt b C alib ra tio n D ev el op me nt co ho rt b O ver al l per for ma nc e D ev el op me nt co ho rt b Ty pe o f v al id at io n co ho rt i n o rig in al pu bl ic at io n A U RO C ( 95 % C I) V al ida tio n co ho rt C alib ra tio n V al ida tio n co ho rt O ver al l per for ma nc e V alid at io n co ho rt In ter nal ly Ex ter nal ly A N ZRO D Pa ul e t a l 30 + Dat a s pli tt ing − 0. 91 ( n. s. ) H L C -s ta tis tic : 18 9. 5 H L H -s ta tis tic : 174 .1 C ox c al ib ra tio n re gre ss io n sl op e: 1 B rie r s co re : 0.0 65 In te rn al val id at io n dat as et 0. 90 ( n. s. ) H L C -s ta tis tic : 10 4. 9 H L H -s ta tis tic : 111 .4 C ox c al ib ra tio n re gr es si on s lo pe : 0.9 8 C al ib ra tio n p lo t pr es ent B rie r s co re : 0.0 66 C us to m ize d A PA C H E I V B rin km an e t a l 31 + Boot st ra pp in g − 0. 88 (0. 88 -0. 88 ) C al ib ra tio n p lo t pr es ent B rie r s co re : 0.0 9 In te rn al val id at io n dat as et − − − M PM 0 -III H ig gi ns e t a l 32 + Dat a s pli tt ing + 0. 83 (0. 82 -0. 83 ) H L s ta tis tic : 1 1. 5 (P = .1 7) − In te rn al val id at io n dat as et 0. 82 (0. 82 -0. 83 ) H L s ta tis tic : 1 1. 6 (P = .3 1) − N Q F-IC O M m or t Ph ili p R . L ee In st itu te 33 + Dat a s pli tt ing − − − − In te rn al val id at io n dat as et 0. 82 (0. 81 -0. 83 ) H L C s ta tis tic : 1 2. 0 (P = .2 8) H L H s ta tis tic : 1 6. 9 (P = .0 8) C al ib ra tio n p lo t pr es ent − OA SI S Jo hn so n e t a l 34 + Dat a s pli tt ing + Orig in al pu bl ic at io n − − − Ex te rn al v al id at io n dat as et 0. 90 ( P < .0 00 3) e H L X 2: 1 9. 6 e B rie r s co re : 0.0 48 e C O PE-4 D uk e e t a l 35 − + Orig in al pu bl ic at io n − − − Ex te rn al v al id at io n dat as et − ( 0. 82 -0 .8 3) H L H -s ta tis tic : 1 4. 8 (P = .0 6) C or re la tio n o f ca lib ra tio n p lo t R 2: 0.9 9 C al ib ra tio n p lo t pr es ent − RD W -S AP S H un zi ke r e t a l 50 − − 0.7 7 (n .s .) Q ua si L ik el ih oo d un de r t he In dep en denc e m ode l Cr iter io n (Q IC ) X 2: 1 .8 3 − − − − − (Co nti nue s) T A B LE 2 (Co nti nue d)
Mor tal it y pr ed ic tio n m od el V al id at ed ? a A U RO C ( 95 % C I) D ev el op me nt co ho rt b C alib ra tio n D ev el op me nt co ho rt b O ver al l per for ma nc e D ev el op me nt co ho rt b Ty pe o f v al id at io n co ho rt i n o rig in al pu bl ic at io n A U RO C ( 95 % C I) V al ida tio n co ho rt C alib ra tio n V al ida tio n co ho rt O ver al l per for ma nc e V alid at io n co ho rt In ter nal ly Ex ter nal ly CO PE D uk e e t a l 36 − + Orig in al pu bl ic at io n − ( 0. 83 -0 .8 4) H L X 2: 2 3. 1 (P < .0 1) − Ex te rn al v al id at io n dat as et − ( 0. 83 -0 .8 4) H L X 2: 2 6. 9 ( P < .0 1) − PR EDIC T H o e t a l 37 + Boot st ra pp in g + − − − In te rn al val id at io n dat as et 0.7 6 (0 .7 5-0.7 7) C al ib ra tio n p lo t pr es ent N ag el ke rke 's R 2: 0 .2 55 H ig h-Ri sk S el ec tio n Sy st em Ia pi ch in o e t a l 46 + Dat a s pli tt ing − 0. 81 ( n. s. ) H L X 2: n .s . (P = .2 1) − In te rn al val id at io n dat as et 0. 81 ( n. s. ) H L X 2: n .s . ( P = .2 2) − G V-SA PS I I Li u e t a l 47 − + Orig in al pu bl ic at io n 0. 83 (0. 81 -0. 84 ) − − Ex te rn al v al id at io n dat as et 0. 82 (0. 81 -0. 83 ) − − M O D S/ N EMS K ao e t a l 38 + Boot st ra pp in g − 0.7 9 (n .s .) − − In te rn al val id at io n dat as et 0.7 6 (n .s .) H L X 2: 5 .4 8 (P = .3 2) c − SM S-IC U G ra nh ol m e t a l 48 + Boot st ra pp in g + 0.7 2 (0 .7 1-0.7 4) H L X 2: 9 .0 (P = .3 4) c C al ib ra tio n s lo pe : 0.9 9 C al ib ra tio n p lo t pr es ent N ag el ke rke 's R 2: 0 .1 91 In te rn al val id at io n dat as et 0.7 3 (n .s .) C al ib ra tio n s lo pe : 0.9 9 C al ib ra tio n p lo t pr es ent N ag el ke rke 's R 2: 0 .1 93 P-m ode l U m eg ak i e t a l 39 + Cros s-va lid at io n − 0. 87 (0. 85 -0. 90 ) H L X 2: 1 4. 5 (P = .0 7) − In te rn al val id at io n dat as et 0. 90 (0. 88 -0. 92 ) H L X 2: 1 3. 5 ( P = .1 0) − B C V m od el H ua ng e t a l 40 + Dat a s pli tt ing − 0.7 9 (0 .7 6-0. 81) H L X 2: 8 .7 (P = .3 7) − In te rn al val id at io n dat as et 0.7 6 (0 .7 1-0. 81) H L X 2: 1 1. 1 ( P = .1 9) − B C V/ A PA C H E I I m ode l H ua ng e t a l 40 + Dat a s pli tt ing − 0. 80 ( 0. 78 -0 .8 3) H L X 2: 6 .2 (P = .6 3) − In te rn al val id at io n dat as et 0. 78 ( 0. 73 -0 .8 3) H L X 2: 5 .4 ( P = .72 ) − C RE EK St ac ho n e t a l 41 + Cros s-va lid at io n − 0. 86 ( n. s. ) H L C -s ta tis tic : 10 .7 ( P = .2 2) H L H -s ta tis tic : 10 .1 ( P = .2 6) B rie r s co re : 0.0 96 In te rn al val id at io n dat as et 0. 83 2 (n. s. ) − − SA PS -R V iv ia nd e t a l 42 − + Orig in al pu bl ic at io n − − − Ex te rn al v al id at io n dat as et 0.7 6 (n .s .) − − (Co nti nue s) T A B LE 2 (Co nti nue d)
Mor tal it y pr ed ic tio n m od el V al id at ed ? a A U RO C ( 95 % C I) D ev el op me nt co ho rt b C alib ra tio n D ev el op me nt co ho rt b O ver al l per for ma nc e D ev el op me nt co ho rt b Ty pe o f v al id at io n co ho rt i n o rig in al pu bl ic at io n A U RO C ( 95 % C I) V al ida tio n co ho rt C alib ra tio n V al ida tio n co ho rt O ver al l per for ma nc e V alid at io n co ho rt In ter nal ly Ex ter nal ly SA PS -E V iv ia nd e t a l 42 − + Orig in al pu bl ic at io n − − − Ex te rn al v al id at io n dat as et 0.7 9 (n .s .) − − 25 O H D D ey o-C ha rlso n C om or bi di ty In de x M ah at o e t a l 49 − − 0. 75 (0. 67 -0. 83 ) − − − − − − D EL AW ARE St ac ho n e t a l 43 − + Orig in al pu bl ic at io n 0. 86 (0. 80 -0. 91 ) H L s ta tis tic : n .s . (P = .2 8) C al ib ra tio n p lo t pr es ent − Ex te rn al v al id at io n dat as et 0. 81 (0. 75 -0. 87 ) H L s ta tis tic : 0 .4 4 (P = n .s .) C al ib ra tio n p lo t pr es ent − Si m plif ie d M or ta lit y Sc or e G oa g e t a l 44 + Dat a s pli tt ing − − − − In te rn al val id at io n dat as et 0. 58 ( n. s. ) − − A bb re vi at io ns : A N ZR O D , A us tr al ia n a nd N ew Z ea la nd R is k O f D ea th ; A PA C H E, A cu te P hy si ol og y a nd C hr on ic H ea lth E va lu at io n; A PM , a dd uc to r p ol lic is m us cl e; A U RO C ; a re a u nd er t he r ec ei vi ng op er at in g c ur ve s; B C V, B lo od C el l V ar ia bi lit y; C I, c on fid en ce i nt er va l; C O PE , C rit ic al c ar e O ut co m e P re di ct io n E qu at io n; C RE EK , C rit ic al R is k E va lu at io n b y E ar ly K ey s; D EL AW A RE , D en se L ab or at or y W ho le B lo od A pp lie d R is k E st im at io n; G V, g lu co se v ar ia bi lit y; H L, H os m er -L em es ho w ; I C N A RC , I nt en si ve C ar e N at io na l A ud it R es ea rc h C en tr e; I C U , i nt en si ve c ar e u ni t; M M I, M ul ti-m or bi di ty I nd ex ; M O D S, M ul tip le O rg an s D ys fu nc tio na l S co re ; M PM ; m or ta lit y p re di ct io n m od el ; N EM S, N in e E qu iv al en ts N ur si ng M an po w er u se S co re ; N Q F-IC O M m or t, N at io na l Q ua lit y F or um I C U o ut co m es m od el (m or ta lit y) ; n .s ., n ot s pe ci fie d; O A SI S, O xf or d A cu te S ev er ity o f I lln es s S co re ; P RE D IC T, P re di ct ed R is k, E xi st in g D is ea se s a nd I nt en si ve C ar e T he ra py ; R D W , r ed c el l d is tr ib ut io n w id th ; S A PS , S im pl ifi ed A cu te P hy si ol og y S co re ; S M S-IC U , S im pl ifi ed M or ta lit y S co re f or t he I nt en si ve C ar e U ni t; S U PP O RT , S tu dy t o U nd er st an d P ro gn os es a nd P re fe re nc es f or O ut co m es a nd R is ks o f T re at m en ts ; T RI O S, Th re e-da y R ec al ib ra tin g I C U O ut co m es . aCita tio ns o f o rig in al p ub lic at io ns w er e s cr ee ne d o n i nt er na l a nd /o r e xt er na l v al id at io n a rt ic le s a nd s ho w n a s b ei ng p re se nt ( +) o r n ot p re se nt ( −) . W he n i nt er na l v al id at io n w as p re se nt , t he m et ho d o f in te rn al v al id at io n u se d i n t he o rig in al p ub lic at io n w as p re se nt ed . W he n e xt er na l v al id at io n i n t he o rig in al p ub lic at io n w as p re se nt , or ig ina l p ubli ca tio n w as a dd ed i n t he c ol um n. bD ev el op m en t c oh or t i nd ic at es t he c oh or t i n w ho m t he p re di ct io n m od el w as d ev el op ed , s om et im es a ls o r ef er re d t o a s t ra in in g c oh or t. cN ot c le ar w he th er t he v al ue w as d er iv ed f ro m t he d ev el op m en t o r v al id at io n d at as et i n t he o rig in al p ub lic at io n, o r v al ue w as d er iv ed f ro m t he d ev el op m en t a nd v al id at io n d at as et t og et he r. dN ot c le ar w he th er t hi s v al ue i s c al cu la te d f or t he 6 -m on th m or ta lit y o ut co m e o r 1 2-m on th m or ta lit y. eN ot c le ar w he th er t he v al ue w as d er iv ed f ro m t he i nt er na l o r e xt er na l v al id at io n d at as et i n t he o rig in al p ub lic at io n. T A B LE 2 (Co nti nue d)
models (60%).14-17,21,22,24-26,28,30,32-36,38-41,43,46,48 Calibration plot was expressed for 12 prediction models (28%),13,15,20,24,28,30,31,33,35,37,43,48 and two prediction models (4.7%) presented the calibration slope value.30,48 Finally, one prediction model (2.3%) used the likelihood ratio test chi-squared value,23 and one prediction model (2.3%) used the Quasi likelihood under the Independence Criterion.45 In 11 pre-diction models (26%), calibration was not assessed.6,18,19,27,29,42,44,47,49
Overall performance was expressed as the Brier score in eight mortality prediction models (19%),6,13,28-31,34,41 and as Nagelkerke's R2 in two prediction models (4.7%).37,48
4 | DISCUSSION
4.1 | Main findings
In this scoping review, we presented a contemporary overview of 43 mortality prediction models used in adult ICU patients in high-income countries. We found varying methodology, and the valida-tion and performance of individual predicvalida-tion models differ. Only 23 mortality prediction models of the 43 (53%) were externally vali-dated. This overview provides a basis for head-to-head comparison of existing mortality prediction models through systematic external validation, with the ultimate goal to identify the most suitable pre-diction model for a certain cohort of patients.
4.2 | Summary of evidence
In previous literature, the maximum number of ICU mortality pre-diction models reviewed was 12,7 which is considerably less than the 43 prediction models identified by this review. Where we in-cluded all developed prediction models specifically designed to assess mortality, other reviews regarding ICU mortality prediction models focused mainly on commonly used models like the APACHE, SAPS and MPM,3-5 or identified models with different outcome than mortality (eg organ dysfunction) or disease- or organ-specific prog-nostic models.4,5,7,8 Additionally, only Siontis et al and Strand et al applied a systematic search to identify the models and discussed the validation of the models.5,8 Where we included all developed mortality prediction models, Strand et al did only include predic-tion models when the search for the specific scoring system yielded more than 50 citations.5 Siontis et al. conducted an evaluation of validated tools for hospitalized patients to predict all-cause mor-tality. However, their analysis included specific patient groups (eg heart or liver patients) rather than general ICU patients as included in the current review.8
Model performance is affected by the choice of outcome.31,50 Most mortality prediction models used hospital mortality as outcome measure.6,13-19,21,22,24,27,28,30-33,35,36,38,41-43,45,46 In gen-eral, longer fixed-time outcome measures used in some mod-els20,24,25,29,31,37,39,40,44,45,47-49 are currently recommended.51 To elaborate, hospital mortality is dependent on discharge practices
and availability of post-ICU care, and is therefore a subjective measure. Furthermore, critical illness affects patients after hos-pital discharge.
The time span during which the mortality prediction models gathered their data varied from short (eg upon ICU admission or during the first initial hour of admission to the ICU) to long (eg during the first 24 hours of admission). Concerning complexity (time consumption) and missing data problems, it may be better in some situations to use a simpler model with less missing data than a more complex model built from a dataset with more miss-ing data which achieves a slightly better performance.52 Longer collection periods may lead to more complete data, as incomplete-ness is often substantial for biochemical variables for patients with short-duration admissions (ie less than 24 hours). However, sampling rate affects predictions.53 This limitation is considered less important in models with shorter data collection. Similarly, the treatments administered during the first 24 hours in the ICU obvi-ously also affect predictions.
4.3 | Comparison of performance
We reported the performance of mortality prediction models in terms of discrimination, calibration and overall performance values. Direct comparison of prediction models predictive performances is not possible, as the development cohorts differed substantially from one another. As a consequence, prediction models cannot be consid-ered interchangeable. Comparisons that are not done head-to-head in external samples independent of all models developed are at high risk of being misleading and may lead to inappropriate conclusions and resource use.12
Of 43, 26 (60%) mortality prediction models used the HL good-ness-of-fit test for calibration.14-17,21,22,24-26,28,30,32-36,38-41,43,46,48 The HL test is commonly used, despite being frequently non-significant for small data cohorts and nearly always significant for large data co-horts.54-57 When only the HL test is reported without any calibration plot or table comparing predicted and observed outcome frequencies, inadequate information regarding calibration is provided.1
Many ICU mortality prediction models are available and compar-atively assessing their performance is a crucial task.4 In all, 25 articles compared the performance of the new model with existing models but used the same cohort of patients that was used in the devel-opment of the ‘novel’ model.6,13,14,16-18,20,22,24,26-30,32,34,40-47,49 This methodology is inherently biased in favor of the ‘novel’ model.54,57 Comparisons between prediction models should therefore only be executed in independent external validation samples not used to de-velop any of the models.
4.4 | Machine-learning algorithms
Mortality prediction models developed as an electronic model or derived from a machine-learning algorithm such as AutoTriage58
were excluded in our manuscript since code and data availability are not requirements in all journals and this is necessary to re-produce the specific prediction model. However, code availability appears to be a rising trend.59 Machine-learning-based prediction models seem to achieve increasingly higher accuracies and are be-coming more dynamic,60 although they still have to include a suf-ficiently large development and validation cohort to adequately assess performance and the risk of overfitting. However, a recent systematic review concluded that machine learning did not have superior performance over logistic regression for clinical predic-tion models.61
The association between mortality and variables may have changed since the original mortality prediction models were developed, for ex-ample, as a result of advancements in diagnostics and therapeutics.62 Mortality alone however is rarely the only outcome measure for inter-ventional studies in ICU patients, and many trials, especially in sepsis, include an organ dysfunction score as part of ongoing patient assess-ment so that effects on morbidity can also be evaluated.3
Misuse of mortality prediction models can lead to inappropriate use of resources and potentially even mismanagement of patient care due to incorrect stratification.57 Awareness of the differences in model design, the variance of predictions across different ICU settings and the effect of heterogeneity in populations are of utmost importance.
4.5 | Limitations
Some limitations of this study need to be addressed. First, having restricted our search to the period from 2008, relevant mortal-ity prediction models might have been overlooked. Even though some of the most widely used mortality prediction models pre-cede the screening period, we identified 16 prediction models that were published before 2008, but optimally searches have no time limit.63 Second, we only included mortality prediction models originally developed for use in the ICU. Mortality predic-tion models not originally developed for mortality predicpredic-tion in the ICU could still be valuable clinically. Third, in some original publications, it was unclear whether the presented discrimination, calibration and/or overall performance values were derived from the development cohort or from the validation dataset. We aimed to clarify these, but certain values might reflect another dataset from the original publication. Fourth, we only provided a system-atic overview of all developed mortality prediction models in adult critically ill patients. We did not perform a systematic review of every retrieved model complete with all consecutive internal and external validations, as results from different external validations in different cohorts are not directly comparable due to differences in populations, case-mix and settings. We restricted the scope of this review to only identify whether internal or external validation had been performed as a measure of thoroughness of develop-ment of the identified models. For this reason, only screening of citations of the original articles was done to identify internal and/ or external validation articles. Therefore, we should address that
our assessment on mortality prediction models not being inter-nally and/or exterinter-nally validated might be incomplete if validation in different publications was missed. A systematic search spe-cifically designed for retrieving validation papers is advised when systematically reviewing the internal and external validations of mortality prediction models.64
4.6 | Unanswered research questions
Although we retrieved many developed mortality prediction mod-els that can be used as a step towards future head-to-head com-parison, with the results of this scoping review it is not possible to make a recommendation on what mortality prediction models to use and it was not our intention to do so. External validation in-volving direct head-to-head comparisons in independent cohorts is needed to unravel the comparable performance of individual models. Although we provide a systematic overview of mortal-ity prediction models and describe whether these were internally and/or externally validated, it was not desirable to give an over-view of all external validations of the prediction models since this would require a specific search strategy for each model. Moreover, we would have liked to assess risk of bias using the recently devel-oped PROBAST score.1 However, this was not feasible because of the number of prediction models.
5 | FUTURE PERSPECTIVES
To identify the most suitable mortality prediction model for a cer-tain patient cohort, ideally a head-to-head comparison of available models should be performed through systematic external validation using prospectively obtained datasets and appropriate statistical methods. The eventual aim will be to use this review to identify, up-date and implement the best performing mortality prediction mod-els in daily practice. We are in the process of validating the found prediction models in independent contemporary cohorts to provide external validation of these models. Second, the process should be performed in different cohorts as heterogeneity of ICU patients ex-ists on multiple levels, that is, patient level, hospital level, region and country level.65 The best mortality prediction model in one setting is not necessarily the best performing prediction model in another setting. Third, it is worth mentioning that ICU patients have reduced long-term survival and impaired quality of life after ICU discharge compared to the general population.66 Future research should also look at determinants of poor outcomes in ICU survivors to help guide long-term follow-up.67
6 | CONCLUSIONS
In this review, 43 mortality prediction models have been studied. The validation and performance of individual prediction models
