UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)
Intensive care unit benchmarking
Prognostic models for length of stay and presentation of quality indicator values
Verburg, I.W.M.
Publication date
2018
Document Version
Other version
License
Other
Link to publication
Citation for published version (APA):
Verburg, I. W. M. (2018). Intensive care unit benchmarking: Prognostic models for length of
stay and presentation of quality indicator values.
General rights
It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons).
Disclaimer/Complaints regulations
If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible.
outcome-based quality
indicators for intensive
care units
Ilona W.M. Verburg, Evert de Jonge, Niels Peek and Nicolette F. de Keizer
Abstract
Objective: Often used quality indicators for intensive care unit (ICU) care are
based on in-hospital mortality, readmission to the ICU, and ICU length of stay. These outcome measures are known to be interrelated and influencing each other at patient level. Our aim was to examine whether there is also an association between unit-level, case-mix adjusted quality indicators based on these outcome measures in the general ICU population as well as within specific subgroups.
Design: ICUs participating in the Dutch National Intensive Care Evaluation
registry (NICE).
Setting: All 83 Dutch ICUs
Patients: A total of 59,809 ICU patients admitted in the year 2015. Interventions: None
Measurements and Main Results: The standardized in-hospital mortality
ratio (SMR) ranged from 0.5 to 1.7; within 48 hours after ICU standardized read-mission ratio (SRR) ranged from 0.0 to 2.3; and standardized ICU length of stay ratio (SLOSR) ranged from 0.7 to 1.6. We expressed association through Pearson's correlation coefficient and considered p-values smaller than 0.01 as statistically significant. For the total ICU population we found no significant associations. We found a positive association between (SMR) and (SLOSR) for admissions with low-mortality risk, (r=0.25; p=0.024), and a negative association between these indicators for admissions with high-mortality risk (r=-0.49; p<0.001).
Conclusion: Overall, quality indicators for in-hospital mortality; readmission
to the ICU; and ICU length of stay showed no association at ICU population level, but we did find associations with opposite directions in subgroups with low versus high mortality risk. We recommend users of quality information to take all indicators into account as they capture different aspects of ICU performance. Fur-thermore, we suggest to report quality indicators for patient subgroups, especially low-risk and high-risk patients groups.
5
5
5
5
5
5.1
Introduction
I
n recent years attention to quality of healthcare as expressed by quality indica-tors has increased [41, 47]. Treatment at the intensive care unit (ICU) is verycomplex and delivered in a highly technical and labor-intensive environment. Along with these developments the cost of intensive care has increased substan-tially resulting in a high proportion of the health care expenditure accountable to ICUs [12].
To assess and improve the effectiveness of ICU care, in-hospital mortality rates are often used as principal quality indicator. Two other often used, easily quantifiable, quality indicators to assess the efficiency of ICU care are based on readmission to the ICU and ICU length of stay. ICU discharge is often based on subjective criteria and discharge policy could be influenced by the availability of bed and staff resources [34, 108, 129]. Premature discharge may lead to an increase in patients readmitted to the ICU or higher post-IC mortality rates [39, 42, 130–132]. Improving discharge decision making may reduce resource consumption, prevent readmissions to the ICU, shorten ICU length of stay, and hence reduce costs [133]. Several studies have shown that patients having an ICU readmission are more severely ill, show longer ICU length of stay and higher in-hospital mortality rates compared to patients not having an ICU readmission [37, 38]. Furthermore, ICU length of stay differs for survivors compared to non-survivors [30, 56]. This study focuses on the association between these quality indicators on unit-level in the context of benchmarking the performance of ICUs.
Quality indicators could be positively correlated, that is, ICUs performing well on one of the quality indicators may also perform well on other quality indicators. This means that the quality indicator does not only provide information about the related process, on efficiency or effectiveness, but it also provides information about quality in general. Quality indicators could also be negatively correlated, meaning that performing well on one of the quality indicators often comes at the expense of performance on the others. It could be the case that these quality indicators are not correlated because the quality indicators capture independent aspects of performance. Finally, it is relevant to know whether ICUs perform similarly across their entire patient population or differently in specific subgroups. The aim of this study was to examine whether there is an association between unit-level, case-mix adjusted quality indicators based on in-hospital mortality; readmission to the ICU; and ICU length of stay in the general Dutch ICU popula-tion as well as within specific subgroups of ICU patients.
5.2
Methods
5.2.1 The Dutch National Intensive Care Evaluation registry
The Dutch National Intensive Care Evaluation foundation (NICE) registry has been active since 1996 and, in 2015, all 83 Dutch ICUs delivered data to the registry. The NICE registry [13] collects demographical, physiological and diagnostic data from the first 24 hours of all patients admitted to participating ICUs, including all variables used in the Acute Physiology and Chronic Health Evaluation (APACHE) IV model to predict probabilities of in-hospital mortality [16]. Collected data is checked by registry staff for internal consistency, by performing onsite data quality audits and training local data collectors [134].
The data collected by the registry are officially registered and collected in ac-cordance with the Dutch Personal Data Protection Act. The medical ethics committee of the Academic Medical Center waived the need for medical ethics for this study (registration number W17_162).
5.2.2 Study data
We included consecutive ICU admissions with initial ICU admission between January 1st 2015 and January 1st 2016. Figure 5.1 presents a flow chart of patient inclusion. We excluded cardiac surgery patients, as patients admitted for cardiac surgery in general have a short, generally fixed, ICU length of stay. We applied the APACHE IV exclusion criteria [47] and in addition we excluded patients who were discharged to another ICU, as their observed ICU length of stay was truncated, and we excluded patients with missing values for model covariates. A readmission to the ICU was defined as a second ICU admission within 48 hours of ICU discharge from the initial ICU admission, in the same hospital within the same hospital stay. A time period of 48 hours was chosen, since readmissions within this period have a stronger relationship with ICU interventions, such as mechanical ventilation, and discharge circumstances, than later readmissions [135, 136]. When considering readmissions to the ICU we excluded patients who were not at risk for readmission to the ICU during the same hospitalization period: patients who died during their initial ICU admission; and patients who were discharged from the ICU to a location outside the hospital (e.g. to home, nursery home or another hospital) [135].
Associations were examined for the entire cohort and for the following subgroups: medical admission, urgent surgery or elective surgery; groups based on severity of illness defined using the calibrated APACHE IV probability of mortality (low: smaller than 0.3, medium: larger or equal to 0.3 and smaller than 0.7, or high: larger or equal to 0.7); and several homogeneous diagnostic subgroups, i.e. patients admitted for community acquired pneumonia (CAP), sepsis, or out of hospital cardiac arrest out of hospital cardiac arrest (OHCA). The precise definitions of all subgroups are presented in appendix 5.A, table 5.3.
5
5
5
5
5
ICU admissions without cardiac surgery between January 1st 2015 and January 1st 2016:
72,797 admissions; 83 ICUs
?
-Exclusion criteria regarding in-hospital mortality and ICU length of stay
APACHE IV exclusion criteria: 11,440 (15.7%)
qAge less than 16 years: 241
qICU length of stay shorter than 4 hours: 2,977 qHospital length of stay longer dan 365: 2,739 qDied before ICU admission: 82
qICU readmissions, within the same hospital
admission: 4,827
qAdmission from critical care unit or ICU: 3,203 qMissing APACHE IV diagnose: 980
qBurns: 137
qTransplantations: 178
qUnknown hospital discharge date: 160 qUnknown admission type: 585
Modeling ICU LoS: 3,480 (4.8%)
qUnknown gender: 7
qICU length of stay unknown: 33 qUnknown Glasgow coma scale: 1,436 qDischarged to another ICU: 1,344
Total number of admissions excluded: 12,988 (17.8%) Number of included admissions for
in-hospital mortality and ICU
length of stay (min to max per hospital): 59,809 (205 to 2,282)
?
-Additional exclusion criteria for readmission to the ICU
qPatient died during ICU admission: 5,611 (9.3%) qDischarged from hospital (to home or other
hospital): 5,702 (9.5%)
Total number of admissions excluded: 11,303 (18.9%) Number of included admissions for
readmission to the ICU (min to max per hospital): 48,496 (148 to 1,974)
Figure 5.1: Flowchart of admissions included for the quality indicators for in-hospital mortality, readmission to the ICU and ICU length of stay.
5.2.3 Quality indicators
Quality indicator scores may be influenced by differences in patient characteristics at admission. Therefore, we performed case-mix correction to adjust for these differences. Furthermore, the included ICUs differ in size. For ICUs with a small number of admissions, frequentist estimators of quality indicator scores would be less reliable than for larger ICUs. For this reason we adjusted the scores for reliability using empirical Bayes estimators, effectively shrinking unreliable scores (i.e. of small ICUs) towards 1 [137, 138]. The empirical Bayes estimators were implemented by fitting mixed effects regression models with random intercept for ICU [137, 138].
To avoid that subgroup analyses with small numbers of admissions would lead to non-convergence of the mixed effects regression analyses, we applied a two-step approach. First we fitted fixed effect models to calculate predicted values for each of the three outcome measures at patient level. These models were used for case-mix correction in the entire cohort and in subgroups. In a second step, the (log odds of the) predicted outcome was included as fixed covariate in mixed effect
models for each of the three outcome measures and each subgroup.
For all outcome measures case-mix correction models and model performances are presented in appendix 5.B.
Mortality: As quality indicator for in-hospital mortality after or during ICU
admission we used the standardized in-hospital mortality ratio (SMR). To estimate the reliability adjusted SMR, we first recalibrated the APACHE IV probability of mortality [56] to our dataset using logistic regression analysis. Secondly, for each subgroup we performed logistic regression with the log-odds of the probability of mortality as fixed covariates and ICU as random intercept. To estimate the reliability adjusted risk adjusted rates we subsequently added the log-odds of the overall mortality rate to the random intercept per ICU and applied the inverse logit of the result in order to calculate the reliability adjusted risk adjusted rates for each ICU. We divided the risk adjusted rates by the overall mortality rate to obtain the reliability adjusted SMR values.
Readmission to the ICU: As quality indicator for readmission to the ICU we used
the standardized readmission ratio (SRR). To estimate the reliability adjusted SRR, we first estimated the expected probability of readmission by developing a logistic regression model on the entire cohort, in a similar way as described in [135]. Secondly, we estimated the SRR using random intercepts per ICU as is done for the SMR.
ICU length of stay: As quality indicator for ICU length of stay we used the
standardized ICU length of stay ratio (SLOSR). To estimate the reliability adjusted SLOSR, we first estimated the expected ICU length of stay, in fractional days, using ordinary least square regression with a log-link function, as we did in a previous study [56]. Secondly, we applied ordinary least square regression with the expected ICU length of stay as fixed covariate and ICU as random intercept.
5
5
5
5
5
We added the random intercept per ICU to the average length of stay for each subgroup to obtain the reliability adjusted average ICU length of stay and divided this value by the average length of stay for the subgroup to get the reliability adjusted SLOSR.
5.2.4 Statistical analysis
The overall performance for the models for in-hospital mortality and readmission to the ICU was assessed by the scaled Brier score [139] and the discriminative ability by the concordance index (C-statistic). The overall performance of the model for ICU length of stay was assessed by the squared Pearson's correlation
coefficient (R2). Model calibration was assessed by creating 50 equally-sized
subgroups of predicted values.
To present associations between quality indicators, we plotted pairs of quality indicator values (measured at unit level) against each other. We examined the association between pairs of quality indicator values by Pearson's correlation coef-ficients and calculated corresponding p-values using the Student's t-distribution. Because small numbers of patients per ICU may result in unreliable estimates of quality indicator values, we also performed a second analysis using Spearman's rank correlation coefficient which is less sensitive to outliers. We considered p-values smaller than 0.01 as statistically significant.
We performed all analyses using the R statistical software, version 3.3.2 [97]. We used linear models (LM) and generalized linear models (GLM) for regression analysis [140] and the 'splines' package for calculating restricted cubic splines [119]. We used the lme4 package [118] to fit linear mixed-effects models (LMMs) and generalized linear mixed-fixed models (GLMMs) by respectively the functions 'lmer' and 'glmer' and estimated the random effects by the function 'ranef'. The 'cor.test' function was used to examine Pearson's and Spearman's correlation coefficients and corresponding p-values [141].
5.3
Results
5.3.1 Study data and quality indicators
In total 72,797 patients were admitted to 83 Dutch ICUs, for other reasons than admission after cardiac surgery. For in-hospital mortality and ICU length of stay as outcome measure, after applying the exclusion criteria 59,809 (82.2%) patients, ranging from 205 to 2,282 patients per ICU were included. From these patients 48,496 (81%) were included for readmission as outcome measure, see figure 5.1 and table 5.8.
Table 5.1 presents crude percentages of the outcomes and case-mix adjusted quality indicators, for the entire cohort and each subgroup. Crude in-hospital mortality was 14.3% ranging from 2.3% to 26.4% and the recalibrated SMR ranged from 0.6 to 1.7. The crude percentage of ICU readmissions was 2.4% ranging from 0.0% to
5.5% and the SRR ranged from 0.7 to 2.1. The overall mean ICU length of stay was 3.3 ranging from 1.7 to 6.1 and the SLOSR ranged from 0.7 to 1.3.
5.3.2 Statistical analysis
Results on the performance of the prediction models are shown in appendix 5.B, table 5.7 and figures 5.4 to 5.6. Figure 5.2 presents the pairs of SMR, SRR and SLOSR plotted against each other for the entire cohort. Table 5.2 presents Pearson's correlation coefficients for the entire cohort as well as for subgroups. No significant associations were identified at cohort level, but a significant association was found for the low mortality risk subgroup (i.e., recalibrated APACHE IV probability of mortality smaller than 0.3). Within this subgroup, units with a higher SMR had a higher SLOSR (r=0.25; p=0.024). Conversely, a negative association (r=-0.486; p<0.001) between SMR and SLOSR was found when selecting admissions with high mortality risks (i.e., recalibrated APACHE IV probability of mortality larger than 0.7). Figure 5.3 presents the pairs of SMR and SLOSR for these subgroups based on probability of mortality.
Appendix 5.C, table 5.9 presents Spearman's rank correlation coefficients for the entire cohort as well as for subgroups. These results are compatible with the results from the primary analysis.
5
5
5
5
5
T able 5.1: V alues for the crude outc omes and for the qu alit y indicators standardized mortalit y ratio (S MR); standardized readmission ratio (SRR) and standardized ICU len gth of sta y ratio (SLOSR). Ov erall v alues are sho wn accompanied with minim um and maxim um ranges o v er the ICUs. In-hospital mortalit y Readmission to the ICU within 48 hours after ICU disc harge ICU length of sta y P atien t sub group Crude o v erall in-hospital mortalit y rate (min to max) (%) min to max SMR Crude ICU readmission rate (min to max) (%) min to max SRR Ov erall median ICU length of sta y (min to max) min to max SLOSR All ICU admiss ions 14.3 (2.3 to 26.4) 0.6 to 1.5 2.4 (0.0 to 5.5) 0.7 to 2.1 1. 2 (0.9 to 2.8) 0.7 to 1.3 CAP 16.2 (0.0 to 100) 0.7 to 1.6 3.1 (0.0 to 33.3) 0.9 to 1.2 1.8 (0.4 to 20.8) 0.7 to 1.4 Sepsis 27.2 (0.0 to 55.3) 0.8 to 1.3 2.6 (0.0 to 14.3) 0.8 to 1.4 2. 3 (0.7 to 11.4) 0.8 to 1.3 OHCA 48.7 (0.0 to 100) 0.8 to 1.2 2.9 (0.0 to 100) 0.0 to 2.0 3.0 (0.3 to 17.4) 0.8 to 1.2 A dmission typ e Medical 19.5 (11.1 to 31.0) 0.7 to 1.3 2.6 (0.0 to 9.3) 0.6 to 2.2 1.8 (1.0 to 2.9) 0.7 to 1.3 Urgen t Surgery 15.8 (2.3 to 30.4) 0.6 to 1.4 2.9 (0.0 to 11.5) 0.8 to 2.1 1.7 (0.8 to 4.7) 0.8 to 1.2 Electiv e surgery 3.3 (0.0 to 10.5) 0.6 to 1.6 2.0 (0.0 to 8.8) 0.9 to 1.3 0.9 (0.8 to 2.9) 0.4 to 2.3 Probabilit y of mortalit y <0.3 6.6 (1.6 to 11.7) 0.7 to 1.5 2.3 (0.0 to 4.6) 0.8 to 1.9 1.0 (0.8 to 2.8) 0.6 to 1.4 ≥ 0.3 and <0.7 47.7 (23.1 to 73.3) 0.7 to 1.2 3.9 (0.0 to 33.3) 0.6 to 2.9 3.5 (1.6 to 6.2) 0.8 to 1.2 ≥ 0.7 78. 4 (33.3 to 100) 0.5 to 1.7 3.2 (0.0 to 100) 0.0 to 1.0 2.2 (0.6 to 9.2) 0.7 to 1.7 AP A CHE IV=A cute Ph ysiology and Chronic Health E v aluation IV; CAP=comm un it y acquired pneumonia; OHCA=out of hos pital cardiac arrestSRR SMR SLOSR SRR SMR SLOSR 0.0 0.0 0.8 1.2 0.4 0.0 1.6 1.2 0.8 0.4 0.0 0.4 0.8 1.2 1.6 0.0 0.4 0.8 1.2 1.6 1.2 0.8 0.4 2.0 0.0 1.6 1.2 0.8 0.4 2.0 A. B. C.
Figure 5.2: Pairs of quality indicators plotted against each other on ICU level. Figure A: standardized in-hospital mortality ratio (SMR) against standardized ICU readmission within 48 hours ratio (SRR). Figure B: standardized length of stay ratio (SLOSR) against SMR. Figure C: SLOSR against SRR.
5
5
5
5
5
1.6 1.2 0.8 0.4 0.0 1.6 1.2 1.2 1.2 0.8 0.8 0.8 0.4 0.4 0.4 0.0 0.0 0.0 0.0 0.0 SLOSR SMR SLOSR SMR SLOSR SMR 1.2 0.8 0.4 1.2 0.8 0.4 A. B. C.Figure 5.3: Quality indicators SMR and SLOSR plotted against each other on ICU level for patients grouped based on severity of illness defined using the probability of mortality. Figure A: probability of mortality smaller than 0.3. Figure B: probability of mortality larger or equal to 0.3 and smaller than 0.7. Figure C: probability of mortality larger or equal to 0.7.
Table 5.2: Pearson's correlation coefficients with corresponding p-values for the pairs of quality indicators standardized mortality ratio (SMR); standardized readmission ratio (SRR) and standardized ICU length of stay ratio (SLOSR).
SMR and SRR SMR and SLOSR SLOSR and SRR Patient subgroup Coefficient p-value Coefficient p-value Coefficient p-value All ICU admissions −0.02 0.877 0.11 0.312 −0.15 0.170 CAP 0.14 0.214 −0.10 0.358 0.10 0.381 Sepsis 0.14 0.199 −0.11 0.330 −0.09 0.429 OHCA −0.11 0.323 −0.18 0.109 0.02 0.862 Admission type Medical −0.05 0.664 0.12 0.264 −0.06 0.614 Urgent Surgery 0.11 0.321 0.04 0.725 −0.12 0.298 Elective surgery 0.17 0.114 0.26 0.018 0.20 0.077 Probability of mortality <0.3 0.08 0.481 0.25 0.024 −0.14 0.195 ≥0.3 and <0.7 −0.11 0.330 −0.07 0.521 −0.05 0.663 ≥0.7 −0.22 0.045 −0.49 0.000 −0.07 0.517 CAP=community acquired pneumonia, OHCA=out of hospital cardiac arrest
P-values were calculated using the Student's t-distribution and p-values smaller than 0.01 were viewed as statistically significant.
5.4
Discussion
In this study we examined the association between SMR, SLOSR and SRR as individual quality indicators for benchmarking ICU quality of care. We did not find significant correlations between SRR and SMR or SRR and SLOSR. This means that ICUs with lower than expected readmission rates did not have lower or higher mortality, or shorter or longer length of stays, than expected. We also did not find a significant correlation between SMR and SLOSR for the general ICU population. However, subgroup analyses based on probability of mortality showed significant associations in opposite directions. Among the low-risk patient population there was a positive association between SMR and SLOSR, i.e. better performance on SMR (lower mortality) was associated with better performance on SLOSR (shorter ICU length of stay). Among the high-risk patient population the association between SMR and SLOSR was negative, i.e. better performance on SMR (lower mortality) was associated with worse performance on SLOSR (longer ICU length of stay).
Other studies found that patients that survived their initial ICU stay and are readmitted to the ICU showed higher mortality rates and longer length of stay [39, 133, 142] compared to patients not readmitted to the ICU. However, studies using quality indicators after correction for case-mix reported similar results as ours. Two earlier studies found no association between quality indicators for
5
5
5
5
5
ICU patients after case-mix correction [39, 142]. For all hospitalized patients one study found no correlation between SMR and readmission rate, but length of stay was positively correlated with SMR [143]. These results may differ since they are not specific for ICU patients, hospital length of stay was used and no-case mix correction was applied for length of stay. Two other studies on hospitalized patients also found that a decrease in length of stay may not necessarily lead to increases in hospital readmission rates [42] and that hospitals with different levels of performance on mortality showed similar results on hospital readmission rates [144].
Interestingly, in our pre-defined subgroup analyses, an association between SMR and SLOSR was identified. In patients with high mortality risk, a high SMR was associated with a shorter case-mix corrected ICU length of stay. In contrast an opposite association was found in low-risk patients. Although results from sub-groups should interpreted cautiously, such different directions in the associations between SMR and SLOSR depending on mortality-risk could be plausible. High quality of care could result in both lower mortality and earlier ICU discharge, explaining the positive association in low-risk patients. However, patients who die may die later in ICUs with high quality of care compared to ICUs with lower quality of care. Thus, if mortality is high, this may paradoxically decrease the mean ICU length of stay, explaining the negative association between SMR and SLOSR in high-risk populations.
For the total cohort of ICU patients the risk of bias in our study was low and the generalizability was high, since many patient level data were available and because all Dutch ICUs participated in this study. Although we do not know the level of residual confounding, we believe that it is small because case-mix correction was applied using a well-established mortality prediction model (APACHE IV) and the models for predicting readmissions and length of stay were developed on our data using a broad set of predictive variables, all measured during the first 24h of ICU admission.
ICU discharge decisions often do not only depend on a patient’s recovery, but on organizational circumstances such as availability of beds on the general ward and the need to free up ICU beds for other patients. Nevertheless, we have deliberately chosen not to include ICU and hospital level covariates in our prediction model for ICU length of stay, since our study has been performed in the context of benchmarking where one does not want to adjust for but detect organizational differences. Besides, in a previous study, we found that after correcting for patient characteristics including ICU characteristics did not significantly improve ICU LoS predictions [145].
Calibration of all three models was satisfactory, but we did find low values of the scaled Brier score and C-statistic for readmission predictions. This indicates that there was variation in readmission rates that could not be explained by case-mix factors; we do not believe that it has influence our findings. Our model to predict the probability of being readmitted within the first 48 hours after ICU discharge
relies on patient data reflecting severity of illness available from the first 24 hours of ICU admission [135]. It may be better to take severity of illness upon ICU discharge into account [37, 133] but these data are not available in our registry. A limitation of our approach is that some quality indicator values were estimated from small subgroups. This may have resulted in unstable estimates. As a so-lution, for each subgroup we decided to apply reliability adjustment to shrink the observed indicator values to the overall average. A sensitivity analysis using the more robust rank correlation coefficient and analysis without adjustment for reliability showed similar results as the main analysis.
Our criteria for patient inclusion lead to several limitations. Firstly, the exclusion of patients that were transferred to another ICU or are discharged from the hospital in order to get palliative care elsewhere may have biased the SMR values. These patients are known to have higher mortality rates [26]. Using long term mortality as outcome measure compared to in-hospital mortality may be useful [146]. We found that the number of transferals in the Dutch ICU system was 1.2% after applying all other patient exclusion criteria. Secondary analyses showed that using SMR based on three months mortality instead of in-hospital mortality did not change the results regarding associations between SMR, SRR and SLOSR (results not shown). Secondly, no information was available regarding readmission to the ICU for patients discharged from the hospital, which may have biased our estimates of SRR values. Thirdly, death or hospital discharge within 48 hours could be competing risks with readmission to the ICU and was not taken into account. Patients who die or are discharged from the hospital within 48 hours after ICU discharge may have lower probability to be readmitted within 48 hours after ICU discharge compared to patients who are not discharged from the hospital within 48 hours after ICU discharge.
5.5
Conclusion
We identified no significant association between quality indicators for in-hospital mortality, readmission to the ICU within 48 hours after ICU discharge and ICU length of stay at ICU population level. Differential associations were found between performance on mortality and length of stay within different risk strata. We recommend users of quality information to take all indicators into account when judging or monitoring ICU quality of care as they capture different aspects of ICU performance. Furthermore, we suggest that users of quality information also receive quality data for patient subgroups, especially low-risk and high-risk patients groups.
5
5
5
5
5
5.6
Acknowledgements
We acknowledge all participating ICUs in the National Intensive Care Registry for their participation and hard work to collect and improve their processes based on these data. We would like to thank Rebecca Holman for statistical advise during the process of writing this paper.
Appendix 5.A: subgroups of patients
Table 5.3: Definitions of patients subgroups included in the study.
Subgroups of ICU
admissions Definition in this study Community acquired
pneumonia (CAP)
APACHE IV admission diagnose pneumonia: aspiration; bacterial; fungal; parasitic; viral; other. Hospital admission prior to ICU admission less than 48 hours.
Sepsis APACHE IV admission diagnose sepsis by infection site: cutaneous/soft tissue; gastrointestinal; Gynecologic;
pulmonary; renal/urinary tract; other location; or unknown location
Out of hospital cardiac arrest (OHCA)
Cardio pulmonary resuscitation in the 24 hours before ICU admission or APACHE IV diagnose cardiac arrest (with or without respiratory arrest; for respiratory arrest see Respiratory System).
Patient admitted directly from home or from an emergency department in their own hospital or other hospital.
Admission type
Medical All admissions not admitted directly from operating or recovery room.
Urgent Surgery Immediate surgery where resuscitation, stabilization and physiological optimization occur simultaneously or immediately prior to surgery.
Elective surgery Surgery at a time that both patient and surgeon schedule or early surgery scheduled within 24 hours after surgery indication.
Probability of mortality
<0.3 Admissions with recalibrated APACHE IV probability smaller than 0.3.
≥0.3 and <0.7 Admissions with recalibrated APACHE IV probability larger or equal than 0.3 and smaller than 0.7.
≥0.7 Admissions with recalibrated APACHE IV probability larger or equal than 0.7.
5
5
5
5
5
Appendix 5.B: case-mix correction models and model
performances
Regression model in R for in-hospital mortality prediction
f o r m u l a M o r t a l i t y <- [ i n −h o s p i t a l d e a t h ] ~ L o g i t ( [APACHE IV p r o b a b i l i t y ]) + [ a d m i s s i o n t y p e ] * r c s ( [APACHE I I I s c o r e ] )
f i t <- glm ( formula=f o r m u l a M o r t a l i t y , data=d a t a s e t , family=binomial ( ) )
Table 5.4: Recalibration of the APACHE IV probability of mortality by logistic regression and in-hospital mortality as dependent variable.
Patient characteristics Coefficient (Wald-type confidence interval) p-value (Chi-squared test) Intercept −2.22 0.015
Logit probability of mortality 0.92 <0.001
Admission type (reference elective surgery)
Medical 1.00 0.339
Urgent −2.09 0.319
Spline AP III score
First spline coefficient 0.02 0.431 Second spline coefficient 0.01 0.976 Third spline coefficient 0.04 0.948 Fourth spline coefficient −0.19 0.770 Medical x spline AP III score
First spline coefficient −0.01 0.653 Second spline coefficient 0.11 0.662 Third spline coefficient −0.59 0.448 Fourth spline coefficient 0.83 0.242 Urgent surgery x spline AP III score
First spline coefficient 0.07 0.258 Second spline coefficient −0.60 0.174 Third spline coefficient 1.79 0.160 Fourth spline coefficient −1.51 0.158
Regression formula for spline variables for APACHE III (AP III) score:
0.02 ∗ [AP III score] + 6.93 ∗ 10−7∗([AP III score] − 20)3
+4.62 ∗ 10−6∗([AP III score] − 39)3−1.93 ∗ 10−5∗([AP III score] − 53)3
+1.75 ∗ 10−5∗([AP III score] − 71)3−3.49 ∗ 10−6∗([AP III score] − 118)3
−0.01 ∗ [AP III score] ∗ [Medical] + 1.15 ∗ 10−5∗([AP III score] − 20)3∗[Medical]
−6.13 ∗ 10−5∗([AP III score] − 39)3∗[Medical] + 8.64 ∗ 10−5∗([AP III score] − 53)3∗[Medical]
−4.05 ∗ 10−5∗([AP III score] − 71)3∗[Medical] + 3.84 ∗ 10−6∗([AP III score] − 118)3∗[Medical]
+0.07 ∗ [AP III score] ∗ [Urgent surgery] − 6.23 ∗ 10−5∗([AP III score] − 20)3∗[Urgent surgery]
+1.86 ∗ 10−4∗([AP III score] − 39)3∗[Urgent surgery]
−1.57 ∗ 10−4∗([AP III score] − 53)3∗[Urgent surgery]
+3.48 ∗ 10−5∗([AP III score] − 71)3∗[Urgent surgery]
Regression model in R for prediction of readmission to the ICU f o r m u l a R e a d m i s s i o n <- [ r e a d m i t t e d t o t h e ICU w i t h i n 48 h o u r s ] ~ [ l o g i t ( [ p r o b a b i l i t y o f m o r t a l i t y ] ) ] + [ a g e ]+ [ p l a n n e d a d m i s s i o n ] + [ m e d i c a l a d m i s s i o n ]+ [ u r g e n t s u r g e r y a d m i s s i o n ] + [ v a s o a c t i v e drug ]+ [ c o n f i r m e d i n f e c t i o n ] + [COPD] + [CVA]+ [ m e c h a n i c a l v e n t i l a t i o n f i r s t 24 h]+ [ c a r d i o v a s c u l a r i n s u f f i c i e n c y ] + [ neoplasm ]+ [ c h r o n i c r e n a l i n s u f f i c i e n c y ] + [ c i r r h o s i s ]+ [ c h r o n i c d i a l y s i s ] + [ i m m u n o l o g i c i n s u f f i c i e n c y ]+ [ c h r o n i c r e s p i r a t o r y i n s u f f i c i e n c y ]+ [ h e m a t o l o g i c m a l i g n a n c y ] + [ a c u t e r e n a l f a i l u r e ]+ [ g a s t r o i n t e s t i n a l b l e e d i n g ]
f i t <- glm ( formula <- f o r m u l a R e a d m i s s i o n , data=d a t a s e t , family=binomial ( ) )
Table 5.5: Regression coefficients of logistic regression with patient readmitted to the ICU during the same hospital admission period as dependent variable.
Patient characteristics Coefficient (Wald-type confidence interval) p-value (Chi-squared test) Intercept −3.56 (-4.01 to -3.11) <0.001
Logit probability of mortality 0.13 (0.08 to 0.18) <0.001
Age 0.00 (0.00 to 0.01) 0.689 Planned admission −0.11 (-0.33 to 0.10) 0.311 Admission type (reference elective surgery)
Medical −0.18 (-0.4 to 0.04) 0.102 Urgent surgery −0.03 (-0.27 to 0.20) 0.774 Confirmed infection (yes) 0.16 (0.01 to 0.31) 0.041 Mechanical ventilation first 24 hours (yes) 0.36 (0.23 to 0.50) <0.001
Vasoactive drug use first 24 hours (yes) 0.05 (-0.09 to 0.18) 0.484 Chronic diagnoses (yes)
Cardiovascular insufficiency −0.18 (-0.52 to 0.15) 0.283 COPD 0.15 (-0.01 to 0.31) 0.068 Chronic renal insufficiency 0.00 (-0.27 to 0.27) 0.992 Chronic dialysis 0.22 (-0.28 to 0.72) 0.396 Chronic resperatoir insufficiency 0.00 (-0.28 to 0.27) 0.988 Cirrhosis 0.09 (-0.36 to 0.54) 0.689 Hematologic malignity 0.26 (-0.12 to 0.63) 0.187 Immunologic insufficiency 0.13 (-0.06 to 0.33) 0.187 Neoplasm −0.30 (-0.57 to -0.02) 0.033 Acute diagnoses (yes)
Acute renal failure −0.13 (-0.34 to 0.09) 0.255 CVA −0.14 (-0.42 to 0.15) 0.350 Gastro-intestinal bleeding 0.55 (0.24 to 0.87) 0.001
5
5
5
5
5
Regression model in R for prediction of ICU length of stay
As independent variables we included an initial group of patient characteristics and performed a stepwise backwards selection procedure, excluding the variable with the highest p-value at each step until all variables remaining in the model had p-value less than 0.1.
s p l i n e a p s <- r c s ( [APACHE IV aps s c o r e ] , parms=3) s p l i n e a g e <- r c s ( [ age ] , parms=3)
f o r m u l a L o S <- [ ICU length o f s t a y ] ~
[ g e n d e r ] + [ s p l i n e a p s ] + [ s p l i n e a g e ]+
[ a d m i s s i o n t y p e ] + [ m e c h a n i c a l v e n t i l a t i o n f i r s t 24 h]+ [ c o n f i r m e d i n f e c t i o n ] + [ v a s o a c t i v e drug ] + [ d i a b e t e s ]+ [COPD] + [ Lowest GCS f i r s t 24 h ] + [CPR] + [ cva ]+
[ i m m u n o l o g i c i n s u f f i c i e n c y ] + [ h e m a t o l o g i c m a l i g n a n c y ]+ [APACHE IV a d m i s s i o n d i a g n o s e ] + [ d y s r h y t h m i a ]+ [ g a s t r o i n t e s t i n a l b l e e d i n g ] + [ i n t r a c r a n i a l mass ]+ [ a c u t e r e n a l f a i l u r e ] + [ c h r o n i c r e n a l i n s u f f i c i e n c y ]+ [ c h r o n i c d i a l y s i s ] + [ c h r o n i c r e s p i r a t o r y i n s u f f i c i e n c y ]+ [ c a r d i o v a s c u l a r i n s u f f i c i e n c y ]
model . c a s e M i x <- glm ( formula <- formula _LoS , data=d a t a s e t , family=gaussian ( link= ’ l o g ’ ) ) model . c a s e M i x 1 <- step ( model . c a s e M i x )
Table 5.6: Regression coefficients of ordinary least square regression with intensive care unit length of stay as outcome measure - (1 of 2).
Patient characteristics Coefficient (Wald-type confidence interval) p-value (Chi-squared test) Intercept −1.41 (-1.56 to -1.25) <0.001 Gender: male 0.07 (0.05 to 0.10) <0.001 Admission type Urgent surgery −0.46 (-1.12 to 0.20) 0.169 Elective surgery −0.71 (-1.37 to -0.06) 0.033 Age
First coefficient of spline for age 0.01 (0.01 to 0.01) <0.001
Second coefficient of spline for age −0.01 (-0.02 to -0.01) <0.001
APACHE IV physiology score (APS)
First coefficient of spline for APS 0.04 (0.04 to 0.04) <0.001
Second coefficient of spline for APS −0.04 (-0.05 to -0.04) <0.001
Confirmed infection (yes) 0.31 (0.28 to 0.34) <0.001
Mechanical ventilation first 24 hours (yes) 0.55 (0.52 to 0.59) <0.001
Vasoactive drug use first 24 hours (yes) 0.29 (0.26 to 0.32) <0.001
Table 5.6: Regression coefficients of ordinary least square regression with intensive care unit length of stay as outcome measure - (2 of 2).
Patient characteristics Coefficient (Wald-type confidence interval)
p-value
(Chi-squared test) Chronic diagnoses (yes)
COPD −0.09 (-0.12 to -0.06) <0.001
Chronic renal insufficiency −0.06 (-0.11 to -0.01) <0.001
Chronic dialysis −0.30 (-0.43 to -0.17) <0.001
Chronic resperatory insufficiency 0.04 (0.00 to 0.09) 0.054 Cirrhosis −0.29 (-0.38 to -0.19) <0.001
Diabetes −0.04 (-0.07 to -0.01) 0.007 Hematologic malignity −0.05 (-0.12 to 0.01) 0.113 Immunologic insufficiency 0.10 (0.06 to 0.13) <0.001
Neoplasm −0.32 (-0.38 to -0.25) <0.001
Acute diagnoses (yes)
Acute renal failure 0.21 (0.18 to 0.24) <0.001
CPR −0.08 (-0.13 to -0.03) 0.003 Dysrhytmia 0.04 (0.01 to 0.07) 0.023 Intracranial mass effect 0.21 (0.15 to 0.27) <0.001
APACHE IV admission diagnose
Gastro-intestinal non-operative 0.03 (-0.03 to 0.10) 0.318 Genito-uritary non-operative −0.21 (-0.33 to -0.09) <0.001
Hematology non-operative and operative 0.21 (0.09 to 0.34) <0.001
Metabolic non-operative −0.18 (-0.32 to -0.03) 0.015 Musculoskeletal/skin non-operative 0.13 (-0.07 to 0.34) 0.207 Neurological non-operative 0.00 (-0.06 to 0.06) 0.977 Respiratory non-operative 0.23 (0.19 to 0.27) <0.001 Transplant operative 0.01 (-0.6 to 0.62) 0.966 Trauma non-operative 0.35 (0.27 to 0.42) <0.001 Cardiovasculair operative 0.58 (-0.08 to 1.24) 0.084 Gastro-intestinal operative 0.43 (-0.23 to 1.08) 0.204 Genito-uritary operative 0.10 (-0.58 to 0.78) 0.772 Metabolic operative 0.22 (-0.69 to 1.13) 0.637 Musculoskeletal/skin operative 0.08 (-0.61 to 0.76) 0.827 Neurological operative 0.60 (-0.06 to 1.26) 0.076 Respiratory operative 0.48 (-0.18 to 1.15) 0.155 Trauma operative 0.73 (0.07 to 1.39) 0.031
COPD=chronic obstructive pulmonary disease; CPR=cardiopulmonary reanimation Variables excluded by stepwise regression: hematologic malignity; resperatroy insufficiency; cardio vascular insufficiency; dysrhytmia
Regression formula for spline variables for APACHE III (AP III) score:
4.17 ∗ 10−2∗[AP III score] − 1.02 ∗ 10−5∗([AP III score] − 19)3
+1.50 ∗ 10−5∗([AP III score] − 40)3−4.77 ∗ 10−6∗([AP III score] − 85)3
Regression formula for spline variables for age:
5
5
5
5
5
Performance of the case-mix adjusted models
Table 5.7: Performance results for the predicted probability of mortality; predicted probability of readmission within 48 hours; and ICU LoS predictions on patient.
R2 on Scaled Calibration curve3 patient Brier C- α (confidence β (confidence
All ICU admissions level1 score2 statistics interval) interval)
Predicted probability of mortality 0.33 0.89 0.00 (0.00 to 0.01) 0.99 (0.97 to 1.01) Predicted probability of readmission 0.00 0.61 0.00 (0.00 to 0.01) 0.95 (0.79 to 1.12) Predicted ICU length
of stay
0.16 0.06
(-0.01 to 0.14)
0.99
(0.97 to 1.01)
1Squrared Pearson's correlation coefficient (R2) was used for continuous outcomes.
2For the dichotomous outcome measures the Brier skill score and C-statistics were used.
3Calibration is based on 50 subgroups of 2% percentiles of predicted outcome.
●●●●●●●●●●●●●●●●●●●●●● ●●●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● -P re d ic te d p ro b ab il it y o f in h os p it al m or ta li ty
Predicted probability of in hospital mortality
-0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 1.0 0.8
Figure 5.4: Calibration plot of predicted in-hospital mortality against observed in-hospital mortality, based on 2% percentiles of predicted probability of mortality.
● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● Predicted prob ability of readmission t o the ICU
Observed probability of readmission to the ICU
0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.00 0.01 0.02 0.03 0.04 0.05 0.06
Figure 5.5: Calibration plot of predicted probability of readmission to the ICU against observed probability of readmission to the ICU within 48 hours of ICU discharge, based on 2% percentiles of predicted probability of readmission to the ICU.
● ●● ●● ●●●●●● ●●●●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● 0 3 6 9 12
Mean observed ICU length of stay
Mean predicted ICU length of stay
0
6 9
12
3
Figure 5.6: Calibration plot of mean predicted ICU length of stay against mean observed ICU length of stay, based on 2% percentiles of predicted ICU length of stay.
5
5
5
5
5
Appendix 5.C: results
Table 5.8: Number of included ICUs and admissions for each subgroup and quality indicator. Overall values are accompanied with minimum and maximum ranges over the ICUs.
Patient subgroup
Number of ICUs
Number of admission (min to max) for length of stay and mortality
Number of admission (min to max) for readmission to the ICU All ICU admissions 83 59,809 (205 to 2,282) 48,496 (148 to 1,974) CAP 81 1,286 (1 to 113) 876 (1 to 59) Sepsis 83 2,843 (2 to 107) 2,135 (2 to 90) OHCA 78 2,150 (1 to 103) 1,110 (1 to 52) Admission type Medical 83 34,096 (49 to 1,137) 24,881 (36 to 838) Urgent Surgery 83 8,000 (14 to 357) 6,908 (13 to 299) Elective surgery 83 17,713 (12 to 918) 16,707 (12 to 911) Probability of mortality <0.3 83 46,971 (160 to 1,877) 40,858 (123 to 1,716) ≥0.3 and <0.7 83 8,671 (11 to 288) 6,137 (6 to 219) ≥0.7 83 4,167 (1 to 166) 1,501 (1 to 78)
CAP=community acquired pneumonia, OHCA=out of hospital cardiac arrest
Table 5.9: Spearman's rank correlation coefficients with corresponding p-values for the pairs of quality indicators standardized mortality ratio (SMR); standardized readmission ratio (SRR) and standardized ICU length of stay ratio (SLOSR).
SMR and SRR SMR and SLOSR SLOSR and SRR Patient subgroup Coefficient p-value Coefficient p-value Coefficient p-value All ICU admissions −0.01 0.925 0.09 0.407 −0.05 0.671 CAP 0.07 0.560 −0.13 0.238 −0.09 0.449 Sepsis 0.19 0.086 −0.09 0.402 −0.11 0.317 OHCA 0.00 0.994 −0.17 0.145 0.03 0.781 Admission type Medical −0.01 0.937 0.12 0.271 0.03 0.799 Urgent Surgery 0.10 0.375 0.03 0.818 −0.07 0.556 Elective surgery 0.19 0.087 0.31 0.004 0.20 0.075 Probability of mortality <0.3 0.08 0.500 0.29 0.009 −0.11 0.333 ≥0.3 and <0.7 −0.15 0.171 −0.15 0.190 −0.01 0.951 ≥0.7 −0.26 0.016 −0.41 0.000 −0.09 0.424
P-values were calculated using the Student's t-distribution and p-values smaller than 0.01 were viewed as statistically significant.
