UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)
Intensive care unit benchmarking
Prognostic models for length of stay and presentation of quality indicator values
Verburg, I.W.M.
Publication date
2018
Document Version
Other version
License
Other
Link to publication
Citation for published version (APA):
Verburg, I. W. M. (2018). Intensive care unit benchmarking: Prognostic models for length of
stay and presentation of quality indicator values.
General rights
It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons).
Disclaimer/Complaints regulations
If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible.
to predict adult ICU
length of stay?
A systematic review
Ilona W.M. Verburg, Alireza Atashi, Saeid Eslami, Rebecca Holman, Ameen Abu-Hanna, Evert de Jonge, Niels Peek and Nicolette F. de Keizer
Abstract
Objective: We systematically reviewed models to predict adult intensive care
unit (ICU) length of stay.
Data Sources: We searched the Ovid Excerpta Medica database (EMBASE) and
Medical Literature Analysis and Retrieval System Online databases (MEDLINE) for studies on the development or validation of ICU length of stay prediction models.
Study selection: We identified 11 studies describing the development of 31
prediction models and three describing external validation of one of these models.
Data extraction: Clinicians use ICU length of stay predictions for planning ICU
capacity; identifying unexpectedly long ICU length of stay; and benchmarking ICUs. We required the model parameters to have been published and for the mod-els to be free of organizational characteristics and to produce accurate predictions, as assessed by the squared Pearson's correlation coefficient (R2) across patients for planning and identifying unexpectedly long ICU length of stay and across ICUs for benchmarking, with low calibration bias. We assessed the reporting quality using the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies.
Data synthesis: The number of admissions ranged from 253 to 178,503. Median
ICU length of stay was between 2 and 6.9 days. Two studies had not published model parameters and three included organizational characteristics. None of the models produced predictions with low bias. The value of R2 was 0.05 to 0.28 across patients and 0.01 to 0.64 across ICUs. The reporting scores ranged from 49/78 to 60/78 and the methodological scores from 12/22 to 16/22.
Conclusion: No models completely satisfy our requirements for planning,
iden-tifying unexpectedly long ICU length of stay, or for benchmarking purposes. Physicians using these models to predict ICU length of stay should interpret them with reservation.
2
2
2.1
Introduction
I
ntensive care units (ICUs) provide complex and expensive care and hospitalsface pressure to improve efficiency and reduce costs [23, 54]. Since costs are strongly related to ICU length of stay, shorter ICU length of stay generally equates to lower costs [23, 24]. Hence, models predicting ICU length of stay can play an important role in examining the efficiency of ICU care. We identified three main reasons for clinicians to predict ICU length of stay [35, 36]: 1) planning the number of beds and members of staff required to fulfill demand for ICU care within a given hospital or geographical area; 2) identifying individual patients or groups of patients with unexpectedly long ICU length of stay to drive direct quality improvement; and 3) enabling case-mix correction when comparing average length of stay between ICUs (benchmarking). The requirements of an ICU length of stay prediction model differs between these situations. A model for planning purposes or to identify individuals or groups with unexpectedly long ICU length of stay needs to predict ICU length of stay reliably for individual patients. When benchmarking the quality or efficiency of ICU care and benchmark reports are based on summary measures of differences between expected and observed length of stay [55], a prediction model needs to predict total ICU length of stay accurately across ICUs.A range of models to predict case-mix adjusted ICU length of stay have been published. However, their clinical utility is unclear [25, 55] and there is no consensus on which is the best [16, 26, 29]. Predicting ICU length of stay accurately is difficult for three reasons. Firstly, statistical methods often assume a Gaussian distribution, but ICU length of stay is generally right skewed [56]. Secondly, patients admitted to an ICU form a heterogeneous group with a wide range of complex health issues, each of which may have a different association with ICU length of stay [25]. Thirdly, the association between severity of illness and ICU length of stay differs for ICU survivors and ICU non-survivors [56]. If not correctly addressed, these points could lead to wildly inaccurate or biased predictions of ICU length of stay, thus negating their utility [56].
In this study, we systematically review reporting and methodological quality of models for predicting ICU length of stay and assess their suitability for planning ICU resources, identifying unexpectedly long ICU length of stay, and benchmarking. We examine characteristics most relevant to clinicians assessing the suitability of a published model to predict ICU length of stay in their own hospital or group of hospitals.
2.2
Methods
2.2.1 Search strategy and inclusion and exclusion criteria
We searched the Ovid Excerpta Medica database (EMBASE) and Ovid Medical Literature Analysis and Retrieval System Online (MEDLINE) databases from database inception until October 31th 2014 by searching all fields and including citations in-progress, which are not indexed with Medical Subject Headings (MeSH) headings. The search query consisted of three sub-queries, with synonyms and combined with 'AND', on intensive care, length of stay and prediction. We present the detailed search strategy in appendix 2.A, table 2.7. We included all original papers describing the development and/or validation of a prediction model for ICU length of stay in adult patients. We excluded duplicate studies, papers not written in English and studies, which were later updated by the same research group.
When deciding whether to include a paper, the authors (IV, AA) classified the papers by reading the title and abstract, then compared and discussed their results until they reached consensus about the eligibility of the paper based on the title and abstract. IV manually reviewed the references in papers found. The authors (IV and AA) read the full text of the included papers and independently scored the items for each model. If necessary, disagreement in all steps was reconciled by discussion with other authors (NdK, SE).
2.2.2 Assessment of methodological and reporting quality
We used the consensus based Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies (CHARMS) [57] to assess the quality of reporting in the studies. This checklist was based on previously published reporting guidelines, systematic reviews, methodological literature and pilot versions discussed within the Cochrane Prognosis Methods Groups. Although the checklist is relatively new, it has already been used in scientific reviews [58–60]. We extended the checklist with three items of importance in prediction models [61] and four items on how the prediction model handled four specific subgroups of ICU patients: 1) patients readmitted to the ICU; 2) patients transferred from or to another ICU; 3) patients who survived their ICU stay versus patients who died on the ICU; 4) and patients who underwent cardiac surgery. Although the checklist presents items to report on, no score and no weight per item was reported. To circumvent this, we assess the methodological quality of prediction models using eleven items that we considered as important, based on literature [57, 61]. We assigned item scores of zero (Not), one (Partly) or two (Yes) points and calculated a total score for reporting and methodological quality. There were 39 items in the reporting score for a total of 78 possible points, and 11 items in the methodological score for a total of 22 possible points. All items are presented in appendix 2.B, table 2.8.
2
2
2.2.3 Definition of utility in predicting ICU length of stay
In order to assess the models we defined four requirements on prediction models in advance. The first three were applicable to models for all purposes. We regarded a model as suitable if: a) the parameters required to predict ICU length of stay have been published; b) it does not include any organizational characteristics; c) it has a low level of bias, demonstrate by at least ('moderate' calibration [62]. Moderate calibration is achieved if the mean observed values equal the mean predicted values for groups of patients with similar predictions [62]. Moderate calibration is important when benchmarking to avoid distortion of benchmarks if mean predicted ICU length of stay differs greatly between hospitals. Our fourth requirement was that a model produces accurate predictions. Our definition of 'accurate predictions' is different for models used to plan resources or identify patients with unexpectedly long ICU length of stay and for models used for case-mix correction in benchmarking. We regard a model as suitable for planning resources or identifying patients with unexpectedly long ICU length of stay if it has an squared Pearson's correlation coefficient (R2) of at least 0.36 (strong) across patients. We regard a model as suitable for case-mix correction in benchmarking if it has an R2 of at least 0.36 (strong) across ICUs [63].
2.3
Results
We identified 3,818 unique articles in Embase and Medline and selected 25 (0.7%) based on the title and abstract. Following inspection of the full text, we included 13 (52.0%) studies and identified one additional study from the references of previously included papers [32]. We excluded three studies [29, 37, 64], because a more recent version of the model was available [28]. Hence, we included 11 studies describing the development of 31 models [25, 26, 28, 32, 55, 56, 65–69]. We present the inclusion process in figure 2.1.
Most studies presented one model, but four [25, 26, 55, 56] each presented between two and 12 models. Three studies [26, 56, 67] described the external validation and one a second order recalibration [26] of the Acute Physiology And Chronic Health Evaluation (APACHE) IV [28] model for ICU length of stay. We present information on geographical location, time period of data collection, observed mean and median ICU length of stay, numbers of parameters estimated of ICUs and included patients in table 2.1. Data was collected between 1995 [68] and 2011 [56] over periods of one month [68] to eleven years [66] in Europe [55, 56, 68], USA [26, 28, 67], Southern America [65], Egypt [69] and worldwide [32]. The data come from 253 [69] to 178,503 [66] admissions to three [69] to 275 [32] ICUs. The number of parameters estimated ranged from one [32, 65] to 151 [56] and the average number of admissions per parameter estimated from 82 [68] to 16,560 [32].
Studies reviewed based on title and abstract: 3,818
?
Review full text for relevance: 25
?
Articles included based on full text: 15
?
Articles eligible for review: 17
Articles included from references listed by articles included: 2 -Model development: 15 Model validation: 12 ) P P P P P PPq
Figure 2.1: Flowchart representing the result of search and the number of articles excluded and eligible for review.
We present patient exclusion criteria and candidate predictors considered for inclusion in table 2.2 and table 2.3. Studies used exclusion based on observed value of ICU length of stay, such as incomplete or unknown ICU length of stay, ICU length of stay less than four [25, 26, 28, 37, 56, 64, 66, 67] or six [55] hours, ICU length of stay greater than 48 hours [68] or 60 days [25, 66]. Three studies included ICU organizational characteristics, such as geographic location, ICU level, number of beds or teaching hospital status [25, 55, 66].
In table 2.4, we present how researchers handled patients, who were readmitted to the ICU, transferred between ICUs, underwent cardiac surgery or died before ICU discharge. The researchers' strategies for handling readmissions were: including only a patient's first admission to the ICU [25, 26, 28, 32, 56, 66]; including readmission as predictor in the model [67]; defining ICU length of stay as the sum of all ICU length of stay of a patient's ICU admission [65]; and including readmissions as separate data records [55]. Three studies excluded patients based on transfer to [56, 67] or from another ICU [28, 56, 67] and six excluded one or more groups of cardiac surgery patients [26, 28, 55, 56, 67, 69]. Researchers handled patients, whose ICU stay ended in death by: including death as a predictor [25, 55, 66], excluding patients dying within an hour of ICU admission [69] or developing different models for ICU survivors and non-survivors [56].
2
2
T able 2.1: Coun try , p erio d of data colle ction, outcome v alues, n um b e r of predictors and sample size for mo del dev elopmen t studies. Reference Data collec tion coun tryData collection perio
d Num b er of ICUs included P ercen tage of patien ts excluded Num b er of included admissions Mean (median) ICU length of sta y (da ys) Num b er of parameters estimated Clermon t, 2004 [68] 11 Europ ean coun tries 1 1995 49 -989 -12 P erez, 2006 [65] Colom bia 1997 to 1998 20 6% 1 ,528 20 (13.0) 2 1 Zimmerman, 2006 [28] USA 2002 to 2003 104 12% 69 ,652 3.9 (2.0) 131 Rothen, 2007 [32] 35 coun tries 3 2002 275 -16 ,560 2.0 1 Moran, 2008 [66] A ustral ia and New Zealand 1993 to 2003 99 12% 178 ,503 3.6 (2.9) 79 Moran, 2012, mo del 1-12 [25] A ustralia and New Zealand 2008 to 2009 131 4% 89 ,330 3.6 (2.9) 26 Niskanen, 2009, mo del 1-2 [55] Finland 2000 to 2005 23 22% 37 ,718 3.9 (2.0) 9 V asilevsiks, 2009, mo del 2 [26] 4 USA 2001 to 2004 35 -6 ,684 4.0 (2.0) 24 V asilevs iks, 2009, mo del 3 [26] USA 2001 to 2004 35 -6 ,684 4.0 (2.0) 2 Kramer, 2010 [67] USA 2002 to 2007 83 10% 12 ,640 4.2 (2.14) 5 106 Al T ehewy , 2010 [69] Egypt 2004 to 2005 3 42% 253 5.1 (4.0) 1 V erbur g, 2014, mo del 1-8 [56] Netherlands 2011 83 1% 6 32 ,667 4.2 (1.7) 151 111 Unsp ecified coun tries participating in the Europ ean So ciet y of In tensiv e Car e Medicine (ESICM). 2Hospital da ys after first da y of ICU admission. 3List of 35 co un tries (47): Argen tina; A ust rali a; A ustria; Belgium; Brazil; Bulgaria; Canada; Cuba; Czec h Republic; De nmar k; France; German y; Greece; Hong K ong;Hungary; India; Ireland; Israel; Italy; Luxem bourg; Mexico; Netherlands; Norw ay; P oland; P or tugal; R ussian Federation; Serbia; Slo vakia; Slo venia; Spain; Sw eden; Switzerland; T urk ey; Uni ted Kingdom; United States. 4V asil evsiks, 2009 mo del 2: co variates included from Mortalit y Probabilit y Mo dels (MPM0) II I; mo del 3: co va riates included from Simplified A cute Ph ysiology score (SAPS) II mo del. 5Mean predicted remaining ICU sta y af te r da y 5 w as 6.87 for the subgroup of admi ssions with ICU length of sta y longer than fiv e da ys. 6P er cen tage of excluded admissions after applying the AP A CHE IV inclusion criteria.
Table 2.2: Summary of patient exclusion criteria, for development studies. Reference Based on outcome Based on age ICU readmissions T ransf ers from another ICU T ransf ers to another ICU Surviv al statu s Burns Cardiac surgery Other 1 Clermont, 2004 [68] x Perez, 2006 [65] x x Zimmerman, 2006 [28] x x x x x x x x Rothen, 2007 [32] x Moran, 2008 [66] x x x x Moran, 2012, model 1-12 [25] x x x x Niskanen, 2009, model 1 [55]2 x x x x x Niskanen, 2009, model 2 [55] Vasilevsiks, 2009, model 2 [26]3 Vasilevsiks, 2009, model 3 [26] Kramer, 2010 [67] x x x x x x Al Tehewy, 2010 [69] x x x x x Verburg, 2014, model 1-8 [56] x x x x x x x x
1Other subgroups consist of severity of illness; admission type; admission diagnose;
unknown discharge location or date; trauma; dialysis; and unknown Glasgow Coma Score.
2Niskanen, 2009 model 1: outcome measure truncated at 30 days;
model 2: log-transformed outcome measure used.
3Vasilevsiks, 2009 model 2: covariates included from Mortality Probability Models (MPM0) III;
2
2
Table 2.3: Summary of predictor variables included in the models, for development studies.
Reference Ov erall sev e rit y of illness 1 A dmission source Age Mec hanical v e n tilation Glasgo w Coma Score Comorbidities Hospital length of sta y b efore ICU admission Organizational Coun try sp ecific In teraction terms 2 Other predictors Clermont, 2004 [68] x x x x Perez, 2006 [65] x Zimmerman, 2006 [28] x x x x x Rothen, 2007 [32] x Moran, 2008 [66] x x x x x x x Moran, 2012, model 1-12 [25] x x Niskanen, 2009, model 1 [55]3 x x x x x Niskanen, 2009, model 2 [55] x x x x x Vasilevsiks, 2009, model 2 [26]4 x x x x x Vasilevsiks, 2009, model 3 [26] x Kramer, 2010 [67] x x x x x x x x Al Tehewy, 2010 [69] x Verburg, 2014, model 1-8 [56] x x x x x x x
1Overal severity of illness consist of Acute Physiology Score (APS);
Acute Physiology and Chronic Health Evaluation (APACHE) II score; APACHE III score; Simplified Acute Physiology Score (SAPS) II score; SAPS III score; SAPS probability of mortality; and Mortality Probability Models (MPM)
2Interactions with age; APACHE III score; Therapeutic Intervention Scoring System (TISS) score;
SAPS II score; mechanical ventilation; gender; calendar year; hospital type; ICU death; and yearly number of admissions.
3Niskanen, 2009 model 1: outcome measure truncated at 30 days;
model 2: log-transformed outcome measure used.
4Vasilevsiks, 2009 model 2: covariates included from Mortality Probability Models (MPM0) III;
T able 2.4: The handling of subgroups of patien ts in eac h of the mo d els for predicting ICU length of sta y . Reference T reatmen t of patien ts readmitted to the ICU Exclusion of patien ts transferred b et w een ICUs Exclusion of patien ts admitted to th e ICU follo wing cardiac surgery 1 P atien ts, wh o died b efore ICU disc harge Clermon t, 2004 [68] -P erez, 2006 [65] Summed o v er 2 -Death as one of the states Zimmerman, 2006 [28] Excluded F rom another ICU CABG -Rothen, 2007 [32 ] Excluded -Moran, 2008 [66] Excluded -Predictor: death Moran, 2012, mo del 1-12 [25] Exclude d -Predictor: death Niskanen, 2009, mo del 1-2 [55] Included -Op e n heart Predictor: death V asilevsiks, 2009 mo del 2 [26] 3 Excluded -CABG -V asilevs iks, 2009 mo del 3 [26] -CABG -Kramer, 2010 [67] Predicted T o and from another ICU CABG -Al T ehewy , 2010 [69] -CABG; cardiac v alv e or heart transplan t Excluded: deaths within first hour; deaths cardiopulmonary arrest within 4 hours V erbur g, 2014, mo del 1-8 [56] Excl uded T o and from another ICU CABG; all electiv e su rgery Mo del for whole group and separate mo dels for surviv ors and non-surviv ors 1CABG=coronary artery by ass grafting; ele ctiv e=electiv e surgery 2The sum of ICU length of sta y of all admissions of a patien t 3V asil evsiks, 2009 mo del 2: co variates included from Mortalit y Probabilit y Mo dels (MPM0) II I; mo del 3: co va riates included from Simplified A cute Ph ysiology score (SAPS) II mo del.
2
2
Two studies related ICU length of stay to in-hospital mortality [32, 55]; and presenting expected and observed ICU length of stay separately for survivors and non-survivors [28, 67]. Five studies neglected mortality in their model.
We present results on model evaluation and performance reported by the original authors in table 2.5. The authors used a random sample from the development data set of [25, 26, 28, 55, 65–67], bootstrap methods [25] or data from different time periods [69] or ICUs [68]. The total number of admissions for validation ranged from 460 [68] to 46,517 [28] and the average number per parameter from 35 [26] to 2,843 [55]. The Pearson correlation coefficients ranged from 0.05 [26, 69] to 0.28 [55] across patients and from 0.01 [26] to 0.62 [28] across ICUs. Differences between the mean observed and mean predicted ICU length of stay ranged between 0.01 and 4.7 days and were not statistically significant for seven models. Two studies test these differences for subgroups of the covariates included in their model [26, 28, 70]. Four studies presented a calibration curve [26, 28, 55, 67], but none regression coefficients.
Table 2.6 described the suitability of models according to our predefined require-ments. No models met all of our requirements for models for planning the number of beds and members of staff required or identifying individual patients or groups of patients with unexpectedly long ICU length of stay to drive direct quality improvement. The APACHE IV model [28] and a second order recalibration with updated model parameters of this model [26] fulfilled most of our requirements for models for benchmarking, presented in table 2.6. However, the requirement for moderate calibration was not fulfilled.
We present external validation studies on the APACHE IV model in appendix 2.C table 2.9. The number of included admissions ranged between 4,611 and 32,667 admissions to between 35 and 83 ICUs in the USA and the Netherlands between 2001 and 2011. The average number of patients per parameter was between 35 and 249. The R2 was moderate (0.16 to 0.18) and strong (0.43 to 0.44) across ICUs. The difference in days between the observed and predicted mean length of stay was larger than for the internal validation of the model.
We present the scores assigned to the methodological quality items in appendix 2.B, table 2.8. The overall reporting quality scores ranged from 49 [69] to 60 [55] and median score of 55. The overall methodological quality scores ranged from 12 [25, 32, 66, 68] to 16 [65] points and a median score of 14. Further items extracted from each study are described in appendix 2.D, table 2.10.
T able 2.5: V alidation of prediction mo del p erformance b y the original authors. Mo del v alidation Mo del p erformance Reference V alidation me tho d Num b er of patien ts included in v alidation set R 2 across
ICU (validation set)
R 2 across patien ts (v alidation set) Difference mean observ ed and mean predicted (v alidation set) in da ys (bias) Recalibration plot presen ted Clermon t, 2004 [68] 12 other ICUs 460 -0.50 1 No P erez, 2006 [65] Random sample (pps 2) of 50% stratified b y ICM co de 3 1 ,531 -0.01 to 6.69 1,4 No Zimmerman, 2006 [28] Simple random sample of 40% 46 ,517 0.62 0.22 0.08 1 Y es 5 Rothen, 2007 [32] -0.40 6 No Moran, 2008 [66] Random sample (pps 2) of 20% stratified b y y ear of admission 44 ,625 -0.18 -7 No Moran, 2012, mo del 1-12 [25] Random sample (pps 2) of 20% stratified b y y ear of admission 22 ,333 -0.18 to 0.20 0.2 to 4.7 Y es Niskanen, 2009, mo del 1 [55] 8 Simple random sample of 40% 25 ,586 0.57 0.27 0.01 1 Y es 5 Niskanen, 2009, mo del 2 [55] Simple random sample of 40% 25 ,586 0.64 0.28 0.76 Y e s V asilevsiks, 2009, mo del 2 [26] 9 Simple random sample of 40% 4 ,611 0.28 0.10 0.01 1 Y es 5 V asilevs iks, 2009, mo del 3 [26] Simple random sample of 40% 4 ,611 0.01 0.05 0.02 1 Y es Kramer, 2010 [67] Simple random sample of 50% and differen t time p er io d 12 ,904 0.43 6 0.18 6 0.02 1 and 0.61 10 Y es Al T ehewy , 2010 [69] T w o ICUs and differen t time p erio d -0.05 -No V erbur g, 2014, mo del 1-8 [56] Bo otstrap (100x) 32 ,667 -0.09 to 0.15 -Y es 1Difference tes ted as not significan t differen t from 0 (t-test, Mann Withney U test or Chi 2-go odness of fit test). 2pps=probabilit y prop ortional to size. 3ICM=In tensiv e Care N ational A udit and Rese arc h Cen tre co ding metho d. 4Presen ted per da y, per system: 0.01 (1da y)-6.69 (30da ys), not significan t. 5Recalibration rep orted as accurate, go od or prefect, app endix 2.C, table 2.9. 6P erformance ba sed on the dev elopmen t set, and presen ted as a median. 7Figure 1, presen ts ra w and predicted mortalit y. 8Niskanen, 2009 mo del 1: outcome truncated at 30 da ys; mo del 2: log -transformed outcome measure used. 9V asil evsiks, 2009 mo del 2: co variates included from Mortalit y Probabilit y Mo dels (MPM0) II I; mo del 3: co va riates included from Simplified A cute Ph ysiology score (SAPS) II mo del. 10 P er formance rep orted for admissions with ICU LoS>5 da ys. Difference w as 0.02 for in ternal and 0.61 for external validation.
2
2
T able 2.6: Suitabilit y of mo dels predicting ICU LoS for planning resources; iden tifying ind ividual patien ts or group s of patien ts with unexp ecte dly long ICU length of sta y; and b enc hmarking, b ased on the defined requiremen ts. Reference Mo del parameters published No organiza-tional or coun try sp ecific predictors Only predictors at ad mission or first 24 hours included Mo derate calibration (for all subgroups) Strong R 2 across ICUs Strong R 2 across patien ts Suitable for planning and iden tifying long ICU length of sta y Suitable for b enc hmar-king Clermon t, 2004 [68] Y es Y es No No No No No No P erez, 2006 [65] Y es Y es Y es No No No No No Zimmerman, 2006 [28] Y es Y es Y es No 1 Y es No No No Rothen, 2007 [32] Y es Y es Y es No No No No No Moran, 2008 [66] Y es No Y es No No No No No Moran, 2012, mo del 1-12 [25] No No Y e s No No No No No Niskanen, 2009, mo del 1 [55] 2 Y es No Y es No Y es No No No Niskanen, 2009, mo del 2 [55] Y es No Y es No Y es No No No V asilevsiks, 2009, mo del 2 [26] 3 Y es Y es Y es No 1 No No No No V asilevs iks, 2009, mo del 3 [26] Y es Y es Y es No 1 No No No No Kramer, 2010 [67] Y es Y es No No Y es No No No Al T ehewy , 2010 [69] Y es Y es Y es No No No No No V erbur g, 2014, mo del 1-8 [56] No Y es Y es No No No No No 1Mo dera te recalibration w as tested and differences w ere not significan t for sev era l subgroups. 2Niskanen, 2009 mo del 1: outcome m easure truncated at 30 da ys; mo del 2: log-transformed outcome measure used. 3V asil evsiks, 2009 mo del 2: co variates included from Mortalit y Probabilit y Mo dels (MPM0) II I; mo del 3: co va riates included from Simplified A cute Ph ysiology score (SAPS) II mo del .2.4
Discussion
In this systematic review, we focussed on the utility of models for predicting ICU length of stay and assessed their suitability for planning ICU resources, identifying unexpectedly long ICU length of stay, and benchmarking ICUs. We included eleven studies on model development and three studies externally validating the APACHE IV model [28]. We concluded that no models fulfilled all of our requirements for planning ICU resources or identifying patients with unexpectedly long ICU length of stay. The original [28] and a second order recalibration of the [26] APACHE IV model fulfilled most of our requirements for benchmarking. However, these models did not fulfill our requirement for moderate calibration. As no models fulfilled our requirements, physicians choosing to use them to predict ICU length of stay should interpret the predications with reservation. Benchmarking incorrect predictions can have large consequences, especially when benchmarking results are published and those without specialist statistical knowledge use them to judge hospitals. As patient characteristics in individual hospitals over time may remain more similar than between hospitals, benchmarking ICU length of stay to an historical benchmark may be acceptable with these models.
In addition, as healthcare and hospital policies differ between countries and over time, we recommend validating a model using recent local data before using it to predict individual or to benchmark hospitals on ICU length of stay.
Four main aspects of reporting and methodological quality in studies reporting on the development of prediction models for ICU length of stay could be improved. The first is the exclusion of patients based on observed ICU length of stay. Excluding a few extreme values might enable researchers to obtain a model with a reasonable fit for the majority of patients. However, sub-optimal patient care could lead to prolonged ICU length of stay and, when benchmarking, truncating ICU length of stay or excluding patients with prolonged ICU length of stay can lead to biased performance results [71]. The second is the handling of ICU non-survivors. The association between severity of illness and ICU length of stay differs for ICU survivors and non-survivors [56]. It seem sensible to develop different models for these two groups. However, higher mortality rates could lead to shorter average ICU length of stay, but it is undesirable to reduce ICU length of stay at the cost of increasing mortality. Thirdly, for benchmarking composite indicators incorporating information on ICU length of stay and mortality may be preferable to length of stay as a single outcome measure. Fourthly, some researchers used ordinary linear regression to predict ICU length of stay. This can lead to predictions of ICU length of stay which are negative and, as such, conceptually incorrect [56]. Logarithmic or other transformations of ICU length of stay or other regression models can overcome these problems. Our study has two main strengths over previous reviews of prediction models for ICU length of stay. Firstly, this is the first systematic review of ICU length of stay prediction models for the general ICU population. Secondly, we systematically assessed the studies
2
2
in light of three clinical applications and using established frameworks [57, 61, 72, 73]. Previously, researchers compared the Acute Physiology and Chronic Health Evaluation (APACHE) IV [28], mortality probability [74] and simplified acute physiology score [75] models and found that the APACHE IV scoring system is most frequently used to predict ICU length of stay [28]. A systematic review in cardiac surgery patients identified several models for unexpectedly long ICU length of stay [76], but we found no studies which defined ICU length of stay in this way for the general ICU population. This may be because cardiac surgery is performed in relatively small numbers of specialist hospitals and patients generally have a shorter and less variable ICU length of stay than other patients.
Our study also has Five main weaknesses. Firstly, we did not require studies to have a minimum sample size. Little consensus exists on the sample size required to prevent overfitting when constructing a prediction model for a continuous variable [77–79]. The minimum mean number of admissions per predictor is 82, and hence sufficient, according to the sparse literature. Secondly, we did not place restrictions on the period in which the data were collected, when in the ICU stay patient characteristics were measured or the point from which ICU length of stay was measured. Two studies report on data collected in or before 1999 [65, 68], two [67, 68] used characteristics that changed over time and one [67] predicted prolonged ICU length of stay beyond a threshold value. These models may not be suitable for current implementation, but throw light on patient characteristics that should be included in prediction models for ICU length of stay. Although these models are substantially different to the other models included in this review, we believe that their inclusion does not influence our final conclusions because we do not recommend them for either purpose. Thirdly, we did not examine differences in performance according to whether ICU length of stay is defined in fractional or billing days. Fourthly, we only included studies that developed or validated prediction models for ICU length of stay. We did not include all studies evaluating associations between patient characteristics, such as Sequential Organ Failure Assessment (SOFA) score, physiological or laboratory values, and ICU length of stay. Fifthly, we did not consider the utility of models for the early identification of individuals with a high risk of excessively long ICU length of stay.
Constructing a good model for ICU length of stay would require specialized statistical and clinical knowledge [28, 71–73, 80, 81]. For instance, as several studies [68] have reported an association between daily SOFA scores and ICU length of stay, researchers could explore using these data for day-on-day planning. They could also examine novel statistical methods, such as joint modelling [82] and competing risks [83] to predict ICU length of stay [84–86]. However, we do not expect that these methods will result in substantially better models for predicting ICU length of stay if they only consider patient characteristics. We expect that data on hospital characteristics and ICU policies and practice will also be required. However, including this type of characteristics will make it more difficult to use the resulting models for benchmarking purposes.
2.5
Conclusion
No previously published models satisfy our three general requirements for pre-diction models for ICU length of stay or our specific requirements for models to plan resource allocation and to identify patients with unexpectedly long ICU length of stay or our specific requirements for models for benchmarking purposes. Physicians considering using these models to predict ICU length of stay should interpret them with reservation until a validation study using recent local data has shown that they obtain moderate calibration and produce accurate predictions.
2
2
Appendix 2.A: Search query
Table 2.7: Search queryIntensive Care Unit Admission duration Prognostic model
ICU length of stay prognostic
intensive care prognose
critical care predictive
critically ill prediction
predict predictor
Ovid EMBASE and Ovid MEDLINE were searched from database inception until 31-10-2014: ("length of stay"[All Fields] AND (prognostic [All Fields] OR prognose[All Fields] OR predictive[All Fields] OR prediction[All Fields] OR predict[All Fields] OR
predictor[All Fields])) AND ("intensive care"[All Fields] OR "critical care"[All Fields] OR "critically ill" [All Fields] OR ICU [All Fields])
App
endix
2.B:
A
dopted
domains
and
(k
ey)
items
of
the
used
CHARMS
[57]
c
hec
k
lis
t
T able 2.8: A dopted domains and (k ey) items of the used chec klist [57] accompanied wi th the rep orting-and metho dological score. -(1 of 5) Mo del dev elopmen t Mo del v alidation Clermont, 2004[68] Perez, 2006[65] Zimmerman,2006 [28] Rothen,2007 [32] Moran,2008 [66] Moran,2012 m1-12[25] Niskanen,2009 m1[55] Niskanen,2009 m2[55] Vasile vsiks,2009 m1[26] Vasilevs iks,2009 m2[26] Vasilevs iks,2009 m3[26] Kramer,2010 m1[67] AlT ehewy, 2010[69] Verbu rg,2014 m1-8[56] Vasile vsiks,2009 [26] Kramer,2010 [67] Verbur g,2014 [56] Total scorek eyitem Source of data 1 y y p p y y y y y y y y y n y y n 28 Particip ants P articipan t e ligibilit y and recruitmen t metho d 2 y y y y y y y y y y y y p y y y y 33 P articipan t d escription p y y p y y y y p p p y p y p y y 27 Study dates y y y y y y y y y y y y y y y y y 34 Outc ome(s) to be pr edicte d Definition and metho d for meas uremen t of outcome p p y p y y n n y y y y p y y y y 26 W as the same outcome definition used in all patien ts? y y y y y y y y y y y y y y y y y 34 T yp e of ou tcome y y y y y y y y y y y y y y y y y 34 W ere candidate predictors part of th e outcome? y y y y y y y y y y y y y y y y y 34 Spread is rep orted for p rimary outcome measure n n n n y y y y n n n n n y n n y 12 Candidate pr edictors Num b er and typ e of predictors y y y y y y y y y y y y y y y y y 34 Definition and metho d for measuremen t of predi ctors y y y y y p y y y y y y y y 27 Timing of p redictor measu remen t y y y y y y y y y y y y y y y y y 34 Handling of predictors in the mo delling y y y y y y y y y y y y y y y y y 34 y=fulfilled (y es); n=not fulfilled (no ); p=partly fulfilled (partly); u=unkno wn, not men tioned2
2
T able 2.8: A dop ted domains and (k ey) items of the used chec klist [57] accompanied with the rep orting-and metho dological score -(2 of 5). Mo del dev elopmen t Mo del v alidation Clermont, 2004[68] Perez, 2006[65] Zimmerman,2006 [28] Rothen,2007 [32] Moran,2008 [66] Moran,2012 m1-12[25] Niskanen,2009 m1[55] Niskanen,2009 m2[55] Vasile vsiks,2009 m1[26] Vasilevs iks,2009 m2[26] Vasilevsi ks,2009 m3[26] Kramer,2010 m1[67] AlT ehewy, 2010[69] Verbur g,2014 m1-8[56] Vasile vsiks,2009 [26] Kramer,2010 [67] Verbur g,2014 [56] Total scorek eyitem Sample size Num b er of participan ts and outcomes/ev en ts y y y y y y y y y y y y p y y y y 33 Num b er of outcomes/ev en ts in relation to the n um b er of pred ictors (Ev en ts P er V ariable) 1 p y p y p p p p p p p p p p p p p 19 Missing data Num b er of participan ts with an y missing v alue y n n n n n n n y y y n y y n n y 14 Num b er of participan ts with missing data for eac h predictor p n n n n n n n p p p n p y p n y 10 Handling of missing data y n n n y y y y y y y y y y y y y 28 Mo del development Mo delling metho d y y y y y y y y y y y y y y y y y 34 Mo delling assumptions satisfied n y p n p p p p n n n p n p n p p 11 Metho d for selection of pre dictors for inclusion p n y y p p y y y y y y n y 21 Initial predictors/v ariables are rep orted 2 y y y y y y y y y y y y y y 28 Metho d for selection of pr edictors and criteria used y y y y n p y y y y y y y y 25 Shrinkage of predictor w eigh ts or regression co efficien ts n n n n n n n n n n n n n n 0 Rep orting of mo del deriv ati on and calibration pro cess is sufficien t for the results to b e repro duced 2 p p y y p p y y y y y y p n 21 y=fulfilled (y es); n=not fulfilled (no ); p=partly fulfilled (partly); u=unkno wn, not men tionedT able 2.8: A d opted domains and (k ey) items of the used chec klist [57] accompanied with the rep orting-and metho dological score -(3 of 5). Mo del dev elopmen t Mo del v alidation Key items Clermont, 2004[68] Perez, 2006[65] Zimmerman,2006 [28] Rothen,2007 [32] Moran,2008 [66] Moran,2012 m1-12[25] Niskanen,2009 m1[55] Niskanen,2009 m2[55] Vasilevsiks ,2009 m1[26] Vasilevs iks,2009 m2[26] Vasilevs iks,2009 m3[26] Kramer,2010 m1[67] AlT ehewy, 2010[69] Verbu rg,2014 m1-8[56] Vasilevsi ks,2009 [26] Kramer,2010 [67] Verbu rg,2014 [56] Total scorek eyitem Hand ling sp ecific p atient sub gr oups 3 Readmissions 2 n y y y y y y y y y y y n y y y y 30 T ransf ers 2 n n y y y y y y y y y y n y y y y 28 Non-surviv ors 2 n y n n y y y y n n n n y y n n y 16 Cardiac surgery 2 n n y n n n y y y y y y y y y y y 24 Mo del p erformanc e Calibration and Discrimination y y y n y y y y y y y y y y y y y 32 Measures with confidence in terv als n n n n n n y y n n n n n n n n n 4 Classification measures and use of a-priori cut p oin ts n n n n n n n n n n n n n n n n n 0 Mo del evaluation Metho d used for testing mo del p erformance: dev elopmen t dataset only or separate external v alidation 1 y y y n y y y y y y y y y y y y y 32 In case of p o or v alidation, whether mo del w as adjusted or up dated n n n n n n n n n n n n n n 0 Public ation of the develop ed mo dels Final and other m ultiv ariable mo dels presen ted 1 y y y y y n n y y y y y y n 22 An y al ternativ e presen tation of the final prediction mo dels n n n n n n n n n n n n n n 0 Comparison of the distri bution of predictors y y n n n n n n y y y n n n y n n 12 y=fulfilled (y es); n=not fulfilled (no ); p=partly fulfilled (partly); u=unkno wn, not men tioned
2
2
T able 2.8: A dop ted domains and (k ey) items of the used chec klist [57] accompanied with the rep orting-and metho dological score -(4 of 5). Mo del dev elopmen t Mo del v alidation Clermont, 2004[68] Perez, 2006[65] Zimmerman,2006 [28] Rothen,2007 [32] Moran,2008 [66] Moran,2012 m1-12[25] Niskanen,2009 m1[55] Niskanen,2009 m2[55] Vasile vsiks,2009 m1[26] Vasilevs iks,2009 m2[26] Vasilevsi ks,2009 m3[26] Kramer,2010 m1[67] AlT ehewy, 2010[69] Verbur g,2014 m1-8[56] Vasile vsiks,2009 [26] Kramer,2010 [67] Verbur g,2014 [56] Total scorek eyitem Interpr etation and discussion of the eligible studi es In terpretation of p resen ted mo d els y y y y y y y y y y y y y y y y y 34 Comparison with other stud ies, discu ssion of generalizabilit y , strengths and limitations. y y y y y y y y n n n y y y n y y 26 Metho dolo gic al quality items Study consist of a cohort study or registry instead of a randomized design (s ource of data) y y y y y y y y y y y y y y y y y 34 Study consist of a prosp ectiv e study design (sour ce of data) y y u u n n u u n n n n p n n u 5 P atien ts are not excluded bas ed on outcome v ariable (participan ts) n y n y n n n n n n n n y n n n n 6 Selectiv e inclusion based on data a v ailabilit y did not to ok place (participan ts) y u u u n n n n n n n n n n n n n 2 sample size (n) in dev elopmen t set is sufficien t relativ e to the n um b er of v ariables in the final mo del (sample size) y y y y y y y y y y y y y y 28 y=fulfilled (y es); n=not fulfilled (no ); p=partly fulfilled (partly); u=unkno wn, not men tionedT able 2.8: A d opted domains and (k ey) items of the used chec klist [57] accompanied with the rep orting-and metho dological score -(5 of 5). Mo del dev elopmen t Mo del v alidation Clermont, 2004[68] Perez, 2006[65] Zimmerman,2006 [28] Rothen,2007 [32] Moran,2008 [66] Moran,2012 m1-12[25] Niskanen,2009 m1[55] Niskanen,2009 m2[55] Vasilevsiks ,2009 m1[26] Vasilevs iks,2009 m2[26] Vasilevs iks,2009 m3[26] Kramer,2010 m1[67] AlT ehewy, 2010[69] Verbu rg,2014 m1-8[56] Vasilevsi ks,2009 [26] Kramer,2010 [67] Verbu rg,2014 [56] Total scorek eyitem Metho dolo gic al quality items-c ontinue d Sp ecific treatmen t for this subgr oup to ok place: readmissions n y y y y y y y y y y y n y y y y 30 Sp ecific treatmen t for this subgr oup to ok place: transfers n n y y y y y y y y y y n y y y y 28 Sp ecific treatmen t for this subgr oup to ok place: non-surviv ors n y n n y y y y n n n n y y n n y 16 Sp ecific treatmen t for this subgr oup to ok place: cardiac surgery n n y n n n y y y y y y y y y y y 2 4 V alidation to ok place using an ind ep enden t v alidation datase t (mo del ev aluation) y y y n y y y y y y y y y y y y y 32 Mo del is repro ducible (results of the de v elop ed mo dels) y y y y n n n n y y y y y n 18 R ep orting sc or e 50 52 53 45 54 52 58 60 57 57 57 56 49 58 43 44 50 Metho dolo gic al sc or e 12 16 14 12 12 12 14 14 14 14 14 14 15 14 10 10 12 y=fulfilled (y es); n=not fulfilled (no ); p=partly fulfilled (partly); u=unkno wn, not men tioned 1One or more me tho dological scores are gi ven to this key ite m 2A ddit io nal items added to the chec klist from a score framew ork dev elop ed for reviewing mo dels to predi ct mortalit y in very premature infan ts 3A ddit io nal domain with four key items to the chec klist to assess ho w patien ts of four subgroups whic h could influence ICU LoS w ere treated.
2
2
App
endix
2.C:
External
v
alidation
T able 2.9: External v alidation of the A cute Ph ysiology and Chroni c Health Ev aluation (AP APCHE) IV prediction mo de l for ICU length of sta y . ReferenceData collection location
Data collection y ears Num b er of ICUs included Num b er of patien ts included A v erage n um b er of patien ts p er parameter R 2 across ICUs R 2 across patien ts Bias 1
Recalibration plot rep
orted External v alidati on V asilevs iks, 2009 [26] USA 2001 to 2004 35 4 ,611 35 0.44 0.18 0.19 Y es Kramer, 2010 [67] USA 2002 to 2007 83 11 ,903 91 -6.22 No V erbu rg, 2014 [56] Netherlands 2011 83 32 ,667 249 -0.16 -No Second order recalibrati on V asilevs iks, 2009 [26] USA 2001 to 2004 35 6 ,684 Dev; 51 Dev; 0.42 0.20 0.07 2 Y es 4 ,611 V al 35 V al Dev=dev elopmen t; V al=v alidation 1Difference in mean observ ed and predicted length of sta y in da ys (bias). 2Difference tes ted as not significan t differen t from 0, if tested by t-test, Mann Withney U-test or χ 2-go odness of fit test.
App
endix
D:
Extracted
data
for
eac
h
of
the
included
studies
T able 2.10: Extracted data for eac h of the included studies for the domains source of data, participan ts, outcome(s) to b e predicted and candid ate predictors, of the sev en teen in c lud e d studies in this literature review -(1 of 4). Source of data P artici pa nts Outcome(s) to be predicted Reference Study de sign 1 Selection of ICUs 2 Sample sc heme patien t selection Num ber of exclusion criteria Definition outcome 3 Gran ularit y outcome 4 Clermon t, 2004 [68] Pros cohort V ol in v w orking on sepsis -rel ate d problems; ESICM Cons 1 ICU LoS -P er ez, 2006 [65] Pro s cohort Srs of ICUs stratified Cons 2 L oS post -Zimmerman, 2006 [28] Cohort AP A CHE system installed Cons 9 ICU LoS Min Rothen, 2007 [32] Cohort Database SAPS 3 study Cons 1 SR U -Moran, 2008 [66] Ret cohort of pros collected data ANZICS, ≥ 150 yearly admissions Cons 9 ICU LoS Hours Moran, 2012 m1-12 [25] R et cohort ANZICS Cons 5 ICU LoS Hours Niskanen, 2009 m1 [55] Cohort Finnish consortium Cons 6 ICU LoS -Niskanen, 2009 m2 [55] Cohort Finnish consortium Cons 6 ICU LoS -V asil evsiks, 2009 m1 [26] Ret cohort V ol in v hospitals in California USA (CALICO) Pps (hospital size) 7 ICU LoS Min V asil evsiks, 2009 m2 [26] Ret cohort V ol in v hospitals in California USA (CALICO) Pps (hospital size) 7 ICU LoS Min V asil evsiks, 2009 m3 [26] Ret cohort V ol in v hospitals in California USA (CALICO) Pps (hospital size) 7 ICU LoS Min Kramer, 2010 m1 [67] Ret cohort AP A CHE system installed Cons 8 ICU LoS-5 Min Al T ehewy ,2010 [69] Ret cohort; pros and in terv en tion 5 -Syst 20% 6 ICU L oS -V erburg, 20 14 m1-8 [56] -V ol part NICE Cons 9 ICU LoS One decimal V asile vsiks, 2009 m4 [26] Ret cohort V ol in vi ta ti on all hospitals in California USA (CALICO) Pps (hospital size) 7 ICU LoS Min Kramer, 2010 m2 [67] Ret cohort AP A CHE system installed Cons 8 ICU LoS-5 Min V erburg, 20 14 m9-10 [56] -V olun tary participating NICE Co ns 9 ICU LoS One decimal2
2
T able 2.10: Extracted data for eac h of the included studies for the domains source of data, participan ts, outcome(s) to b e predicted and candidate predictors, of the sev en teen included studies in this literature review -(2 of 4). Candidate predictors Missing da ta Reference Preselection based on literature and kno wl edge Timing of measuremen t of predictors T ransformation of con ti nuous variables Handling missing values 6 P ercen tage of missing values 7 Clermon t, 2004 [68] Y es A dmission, daily Categorizing using cutp oin t Imputation by lin in terp olation (SOF A sore) 11% Missing ov erall (in particular SOF A score) P er ez, 2006 [65] Y es A dmission No -Zimmerman, 2006 [28] Y es A dmission, first 24h Spline -Rothen, 2007 [32] Y es A dmission No -Moran, 2008 [66] First 24h Categorizing using cutp oin t; square terms Excl (incomplete GCS; hospital outcome na) -Moran, 2012 m1-12 [25] Y es First 24h Square terms Excl (incomplete GCS; al lph ys value s na); replace imputa ti on Gaussian distribution -Niskanen, 2009 m1 [55] Y es First 24h, eac h da y No Excl (disease category; ho spi ta l outcome; TISS; or SAPS II score na) -Niskanen, 2009 m2 [55] Y es First 24h, eac h da y No Excl (disease category; ho spi ta l outcome; TISS; or SAP S II score na) -V asile vsiks, 2009 m1 [26] Y es A dmission, first 24h Spline Excl (ICU LoS; or patie nts conditions acrross eac h risk-adjustmen t mo del na) -V asile vsiks, 2009 m2 [26] Y es A dmission, first 24h Categorizing using cutp oi nt Excl (ICU LoS; or patien ts co ndit ion s acrross eac h risk-adjustmen t mo del na) -V asile vsiks, 2009 m3 [26] Y es F ir st 24 h Log Excl (ICU LoS; or patie nts conditions acrross eac h risk-adjustmen t mo del na) -Kramer, 2010 m1 [67] Y es First 24 h, da y 5 Spline Excl (ph ys is na); mis GCS as pred -Al T ehewy ,2010 [69] Y es First 24 h No Excl (to m uc h data on sev erit y of illness na) 42.8% V erburg, 20 14 m1-8 [56] Y es A dmission, first 24h Spline, cycli cal cosin us and sin us terms Excl ( predi ctor variable na) 1.3% 8 V asile vsiks, 2009 m4 [26] Y es A dmission, first 24h Spline Excl (ICU LoS; or patie nts conditions acrross eac h risk-adjustmen t mo del na) 0.6% Missing ICU LoS Kramer, 2010 m2 [67] Y es A dmission, first 24h Spline Excl (ph ys is na); mis GCS as pred -V erburg, 20 14 m9-10 [56] Y es A dmiss io n, first 24h Spline Excl (mi ssing predictor) 1.3%**T able 2.10: Extracted data for eac h of th e included studies for th e domains source of data, participan ts, outcome(s) to b e predicted and candidate predictors, of the sev en teen included studies in this literature review -(3 of 4). Mo de ldev elopmen t Reference Mo delling metho d 9 T ransformation of the outcome 10 Rep orting on mo del assumptions V ar ia ble selection based on analysis Clermon t, 2004 [68] Mark ov t33d ICU -No P er ez, 2006 [65] Mark ov t30d hospital Chec ki ng Mark ov chain assumptions No Zimmerman, 2006 [28] Ols t30d ICU Explanation of the use of splines No Rothen, 2007 [32] A verage No -No Moran, 2008 [66] Ols L og; excl >60d ICU and >365 hosp Residual analyses N o Moran, 2012 m1-12 [25] 12 Metho ds 11 No and log Residual analyse s Com binations of variables and their in tera ctions w ere asses sed for effect Niskanen, 2009 m1 [55] Ols t30d ICU Relationship based on linearit y Bac kw ard (p=0.1) and forw ard (p=0.05) selection Niskanen, 2009 m2 [55] Ols Log Relationship based on linearit y Bac kw ard (p=0.1) and forw ard (p=0.05) selection V asile vsiks, 2009 m1 [26] Ols t30d ICU -No V asile vsiks, 2009 m2 [26] Ols t30d ICU -No V asile vsiks, 2009 m3 [26] Ols t30d ICU -No Kramer, 2010 m1 [67] Ols t30d ICU Explanation of the use of splines Bac kw ard (p<0.05) Al T ehewy ,2010 [69] Ols Log -No V erburg, 20 14 m1-8 [56] 8 Metho ds 12 No and log and t30d ICU -Bac kw ard (p=0.1) V asile vsiks, 2009 m4 [26] Ols t30d ICU -Kramer, 2010 m2 [67] Ols t30d ICU Explanation of the use of splines -V erburg, 20 14 m9-10 [56] Ols t30d ICU
