The handle http://hdl.handle.net/1887/55893 holds various files of this Leiden University dissertation.
Author: Busweiler, L.A.D.
Title: Evaluating quality of care and setting future goals in oesophagogastric cancer treatment
Issue Date: 2017-11-29
Ch ap te r
3
Alternative methods displaying variation in hospital performance in oesophageal cancer surgery using funnel plots
L.A.D. Busweiler1,2,§, M.G. Schouwenburg1,2, §, S. Amodio3, J.L. Dikken2, M.I. van Berge Henegouwen3, M.W.J.M. Wouters1,4 and E.W. van Zwet3
1Dutch Institute for Clinical Auditing, Leiden
2Department of Surgery, Leiden University Medical Centre, Leiden
3Department of Medical Statistics and Bioinformatics, Leiden University Medical Centre, Leiden
4Department of Surgery, Academic Medical Centre, Amsterdam
5Department of Surgical Oncology, The Netherlands Cancer Institute - Antoni van Leeuwenhoek Hospital, Amsterdam
§The first two authors equally contributed to this manuscript
Submitted.
ABSTRACT
Background: Evaluating individual hospital results for quality improve- ment is challenging for resections for oesophageal cancer, as these are often performed in hospitals with low annual volumes and hospital vol- umes vary widely. The aim of this study was to evaluate different funnel plots that can be used for displaying variation in hospital performance.
Methods: All patients who underwent a curative resection for oe- sophageal cancer, registered in the Dutch Upper GI Cancer Audit (DUCA) between 2011-2014, were included. Two different approaches for funnel plots to assess casemix-adjusted hospital variation were evaluated.
Primary outcome was (major) postoperative complication rate. The first method evaluated hospital results within a set amount of time, the sec- ond method evaluated hospital results over a set number of patients. Two simulation experiments were performed to test the statistical properties of the two methods.
Results: A total of 3292 patients were registered by 22 hospitals. Hospital volume ranged between 84-417 patients. Method II was able to identify hospitals with lower annual hospital volumes as positive outliers. Two simulation experiments showed that method II has greater power to identify outliers compared to method I. The over- or underperformance of hospitals detected by method II was more severe.
Conclusion: When dealing with low hospital volumes, evaluating hospital results over a set number of patients instead of a set amount of time has shown to be a valid and reliable tool for identifying centres of excellence and underperformers subjected to further investigation. However, identi- fication of significant outliers remains a challenge.
INTRODuCTION
Insight in quality of care has become increasingly important in today’s healthcare system. Clinical auditing, quality registries and other re- lated quality improvement initiatives have been widely used in Western countries to measure and benchmark the quality of care delivered by individual hospitals e.g. the American College of Surgeons’ National Surgi- cal Quality Improvement Program (NSQIP)1-3, the National Clinical Audit (NCA) Program, including the National Oesophago-Gastric Cancer Audit (NOGCA), managed and developed by the Healthcare Quality Improve- ment Partnership (HQIP) in the United Kingdom4, the Swedish National Register for Oesophageal and Gastric Cancer5 and several audits like the Dutch Surgical Colorectal Cancer Audit (DSCA) supported by the Dutch Institute for Clinical Auditing6.
In the surgical field, where interventions and (short-term) outcomes of care are clearly linked, (risk adjusted) outcome measures like postoperative morbidity and mortality are often used to evaluate hospital performance.
In order to display such variation in hospital performance, a widely used graphical aid is the so-called funnel plot7. This plot shows the outcome of interest (vertical axis) per hospital using 95.0 per cent and/or 99.8 per cent confidence intervals that vary in relation to hospital volume (horizontal axis). Hospitals performing above or below this expected range are poten- tial outliers (both positive and negative) and can be subject for in-depth investigation. Funnel plots are mostly used to display variation in hospital performance on an (bi-) annual basis (‘classical approach’).
Identifying outliers becomes a challenge when evaluating high-risk and low-volume surgery, like resections for oesophagogastric cancer. In the Netherlands, this type of surgery is monitored in a nationwide audit, the Dutch Upper Gastrointestinal Cancer Audit (DUCA)8. Despite current
volume standards, these procedures are often performed infrequently at the vast majority of Dutch hospitals, and annual hospital volumes vary considerably8. The evaluation of the quality of care at hospitals with low annual volumes lacks statistical certainty, which is reflected in wide confidence intervals shown in traditional funnel plots. As a consequence low volume hospitals are rarely identified as an outlier, either positive or negative9. Improving statistical certainty by analysing hospital results over multiple years can be a solution. However, this will further increase differences in hospital volumes. Moreover, this could lead to out-dated results and a delay in the initiation of quality improvement initiatives.
Therefore, the aim of this study was to describe and evaluate additional statistical approaches using funnel plots that can be used for displaying variation in performance for different hospitals within the context of clini- cal auditing.
METHODS
Data source
The dataset was retrieved from the DUCA, a nationwide quality registry and continuous feedback system, providing individual hospitals with benchmarked information regarding various quality parameters for all patients undergoing surgery with the intention of resection for oesopha- geal or gastric cancer in the Netherlands since 20118. Patients with non- epithelial tumours and patients undergoing non-surgical treatment (such as definitive chemoradiotherapy) are not included in this registry. Under Dutch law, no ethical approval or informed consent was required for this study.
Patients and hospitals
All patients with oesophageal cancer who underwent a curative resec- tion, as decided by the surgeon at the end of the procedure, registered in the DUCA between 2011-2015, were included. Additionally, a minimum number of items per patient was required in order to consider a patient eligible for analysis. These items were: information on tumour location, date of birth, date of surgery, intent of surgery as defined at the end of the procedure (potentially curative resection, palliative resection or no resec- tion) and the patient’s vital status 30 days after surgery and/or at time of discharge. In order to investigate current hospital variation, all hospitals that were no longer performing oesophageal cancer surgery in the last year of the study period were excluded.
Outcome and casemix
Major postoperative complication rate, defined as any postoperative complication resulting in a reintervention, a prolonged hospital stay (>21 days) and/or postoperative mortality within 30 days after surgery or dur- ing the initial hospital admission, was chosen as primary outcome for this study.
To enable sound and reliable hospital comparisons, individual hospital results were adjusted for differences in casemix (non-modifiable patient and tumour specific risk factors that can influence the outcome)10. A se- lection of available casemix factors associated with major postoperative complication rates was made based on expert opinion and literature: sex, age, Body Mass Index (BMI), American Society of Anesthesiologists (ASA) classification, coexisting diseases (number of organ systems involved), overall weight loss before first consultation by the surgeon, clinical TNM stage (7th edition)11, tumour location, urgency of the procedure, aditional organ resection due to extensive tumour infiltration, use of immune suppresives, and year of surgery. All casemix factors were categorized
into discrete categories and entered in a multivariable logistic regression model. Missing casemix variables were imputed by their median or mode (most prevalent category). This was done separately for patients with a favourable and non-favourable outcome. Patients with missing data regarding a major postoperative complication were excluded. The regres- sion model was fitted on all patients from the period 2011-2015.
Hospital comparisons
Funnel plots were constructed using 95 per cent confidence intervals.
Individual hospital results were plotted as the ratio between the observed number of patients with a major postoperative complication and the ex- pected (based on casemix) number of patients with a major postoperative complication (Observed/Expected (O/E) outcome ratio). An O/E outcome ratio above 1 indicates a hospital performs below average, whereas an O/E outcome ratio below 1 indicates a hospital performs above average.
The width of the funnel (95 per cent confidence interval) is determined by the expected number of events, which depends both on the hospital volume and casemix. To facilitate interpretability, ‘effective volume’ was defined as the multiple of the expected number of events that best ap- proximates the hospital volume. This ‘effective volume’ was plotted on the x-axis.
Different methods for funnel plots
Two different methods for hospital comparisons were evaluated. For both methods, hospital-specific casemix adjusted major postoperative complication rates were analysed. The first method is most commonly used and evaluates hospital results over a fixed time period (Method I).
The second method evaluates hospital results based on a fixed number of patients (method II). For Method I, all patients who underwent surgery between 2014-2015 were included. For Method II, the 71 most recently
treated patients were selected for each individual hospital. These selec- tions were made in a way that the total number of included patients was approximately equal for both methods.
Simulation experiments
To evaluate the statistical properties of funnel plots using Method I and II, for each method two simulation experiments (a and b) were performed.
For the first simulation (a), the hospital with the largest volume (hospital
“L” with n=174) and the hospital with the lowest volume (hospital “S”
with n=33) in 2014-2015 were selected. The goal was to determine how the power to detect a difference in complication rate depends on the number of patients at a hospital when applying the different methods. A casemix model as estimated from all the data from 2011-2015 was used to compute the probability of a major complication for each patient treated in 2011-2015. Next, the risk of a major postoperative complication was increased for each patient by varying amounts referred to as the ‘effect’.
Finally, outcome was sampled at random for each patient according to his or her risk. By repeating this simulation procedure 10,000 times, power to detect this increased risk of a major postoperative complication was com- puted. This was performed by counting the number of times simulated performance of both centres (L and S) was located outside the funnel in all 10,000 simulations.
In the second simulation (b), the casemix model was re-estimated. Again, all data from 2011-2015 were used, but this time a random effect was added per hospital. The random effect distribution characterizes the performance heterogeneity between hospitals. Next, outcomes were simulated for all patients in the dataset by first sampling the centre ef- fects from the random effect distribution, and then sampling outcomes for each patient according to the centre the patient was treated at, and
his or her casemix characteristics. By repeating this procedure 10,000 times, a comparison could be made between the number of times the two different funnels plots detected over- or underperformance at particular hospitals. Moreover, a comparison could be made between the effect sizes at the hospitals that were detected as over- or underperformance.
Statistical analyses were performed in PASW Statistics version 23.0 (SPSS inc, Chicago, IL) and R.3.0.2
RESuLTS
Patients and hospitals
Between January 2011 and December 2015, a total of 3292 patients who underwent a curative resection for oesophageal cancer in the Netherlands were registered by 22 different hospitals. Five hospitals (116 patients) were excluded as they discontinued oesophageal cancer surgery during the study period. Twenty-two patients were excluded as the outcome of interest was missing. Patient and tumour characteristics of the included patients are shown in Table 1. Total volume per hospital ranged between 84-417 patients. The national average of major postoperative complica- tions was 30 per cent.
Set amount of time (Method I)
For the first method a total of 1553 patients with oesophageal cancer who underwent a resection between 2014-2015 were included in 22 different hospitals. O/E outcome ratios varied from 0.42-1.36 with the effective hospital volume ranging considerably from 29-178 (Figure 1a). Two hos- pitals performed significantly better than average. Hospital A had an O/E outcome ratio of 0.42 and hospital B had and O/E outcome ratio of 0.56.
Table 1. Patient and Tumour characteristics
N %
Total 3292
Sex
Male 2573 78.2%
Female 719 21.8%
Age (years)
0-64 1499 45.5%
65-74 1335 40.6%
≥75 458 13.9%
BMI (kg/m2)
<20 206 6.3%
20-24 1283 39.0%
25 - 29 1285 39.0%
≥30 518 15.7%
ASA classification
I-II 2503 76.0%
≥ III 761 23.1%
Unknown 28 0.9%
Coexisting diseases*
0 752 22.8%
1 883 26.8%
2 784 23.8%
3 473 14.4%
4 400 12.2%
Weight loss before surgery
<10 kg 2939 89.3%
10.1-15 kg 215 6.5%
>15 kg 138 4.2%
Clinical TNM stage
T0/Tis** 12 0.40%
I 475 14.4%
II 843 25.6%
III 1630 49.5%
IV 31 0.9%
X/unknown 301 9.1%
Set number of patients (Method II)
For the second method, the 71 most recently treated patients were in- cluded for all 22 hospitals (n=1562). O/E outcome ratios varied from 0.52 –1.36 with the effective hospital volume ranging from 58-84 (Figure 1b).
Hospital A remained a positive outlier (O/E outcome ratio: 0.52). Another hospital (C) was identified as positive outlier with an O/E outcome ratio of 0.54. Hospital B was no longer performing significantly better than average.
Table 1. (continued)
N %
Tumour location
Proximal and middle intrathoracic oesophagus 419 12.7%
Lower third intrathoracic oesophagus 2037 61.9%
Gastro Oesophageal Junction (GEJ) 836 25.4%
Urgency procedure
Elective 3282 99.7%
Acute 10 0.3%
Previous abdominal surgery
No 2286 69.4%
Yes 1006 30.6%
Additional resection due extensive tumour infiltration
No 3189 96.9%
Yes 103 3.1%
Year of surgery
2011 467 14.2%
2012 610 18.5%
2013 662 20.1%
2014 745 22.6%
2015 808 24.5%
ASA indicates American Society of Anesthesiologists; TNM, tumour node metastasis classification (7th edition).*Number of organ systems involved. **Including T0N0-2M0.
0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 1.10 1.20 1.30 1.40 1.50 1.60 1.70 1.80 1.90 2.00
0 20 40 60 80 100 120 140 160 180 200
Observed / expected
Effective Hospital volume Hospital Average 95% CI C
A
B
a.
0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 1.10 1.20 1.30 1.40 1.50 1.60 1.70 1.80 1.90 2.00
0 20 40 60 80 100 120 140 160 180 200
Observed / expected
Effective Hospital volume Hospital Average 95% CI A C
B
b.
Figure 1. Analysis over a set amount of time according to Method I (a) and a set amount of patients according to method II (b).
Simulation study 1
In Table 2 and Figure 2 power to detect an increased risk of a major postop- erative complication is shown, where the national average complication rate is 32 per cent. Hospital L (n=174) has a 46 per cent power to detect a complication rate of 41 per cent. For hospital S (n=33) this is only 6 per cent. To have 50 per cent power, hospital S would need to have a major postoperative complication rate of about 48 per cent is approximately 50 per cent in both hospitals. The slight difference in power at the two hospitals is due to casemix differences.
Simulation study 2
The random effect standard deviation was estimated at 0.33 on the log odds scale. Figure 3 illustrates results of the first two simulated datasets (b=1 and b=2). In the first column, the funnel plots according to method I and to method II using original data are shown. Table 3a and b show the descriptive statistics after 10,000 repetitions. When using method I, 2.68 hospitals are identified as an outlier on average while with method Table 2. The power at hospital L (n=174) and hospital S (n=33). The power is also shown when including the most recent 71 patients from both hospitals.
Major postoperative complication rate L (174) S (33) L (71) S (71)
0.32 0.01 0.03 0.02 0.02
0.34 0.03 0.02 0.02 0.03
0.37 0.09 0.02 0.04 0.06
0.39 0.23 0.04 0.09 0.13
0.41 0.46 0.06 0.18 0.23
0.44 0.71 0.10 0.30 0.36
0.46 0.89 0.16 0.45 0.52
0.49 0.97 0.24 0.62 0.68
0.51 0.99 0.34 0.76 0.81
0.54 1.00 0.45 0.87 0.90
0.56 1.00 0.56 0.94 0.95
II on average 2.82 hospitals are identified as an outlier (Table 3a). Hence, method II has slightly greater power. In addition, the effects in the hospi- tals that are detected as an outlier tend to be larger when using method II compared to method I (Table 3b). The maximum detected major post- operative complication rate was 0.58 for method I, and 0.67 for method II (Table 3b). The difference between the two methods is also shown in Figure 4. So, when method II detects a hospital as an outlier, the over- or underperformance tends to be more severe – and hence more relevant – compared to method I.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0.32 0.44 0.56
Probability of complication rare falling outside 95% CI
Complication rate Hospital L (174) Hospital S (33) Hospital L (71) Hospital S (71)
Figure 2. Power to detect an increased risk of a major postoperative complication. The solid lines represent the two hospitals with the largest volume (hospital L, N=174) and the smallest volume in 2014-2015 (hospital S, N=33). The dotted lines represent the same hospitals while including the 71 most recently treated patients (Method II). CI = confi- dence interval.
Table 3a. Descriptive statistics of the detection rate for method I and method II
Min 1st Qu. Median Mean 3rd Qu. Max
Method I 0.00 2.00 3.00 2.68 4.00 9.00
Method II 0.00 2.00 3.00 2.82 4.00 10.00
Table 3b. Descriptive statistics of the detected complication rates for method I and method II
Min 1st Qu. Median Mean 3rd Qu. Max
Method I 0.32 0.38 0.41 0.41 0.44 0.58
Method II 0.32 0.38 0.42 0.42 0.45 0.67
10 30 50
0.40.81.21.6
Expected
Observed/Expected
patients 2014-2015
10 30 50
0.51.01.5
Expected
Observed/Expected
b = 1
10 30 50
0.40.81.21.6
Expected
Observed/Expected
b = 2
18 22 26
0.60.81.01.21.4
Expected
Observed/Expected
71 patients
18 22 26
0.61.01.41.8
Expected
Observed/Expected
b = 1
18 22 26
0.61.01.4
Expected
Observed/Expected
b = 2
Figure 3. Simulation experiment example for two random repetitions. In the first column, the funnel plots according to methods I (a) and II (b) applied to the original data are shown. In columns 2 and 3, two simulations are shown. CI = confidence interval.
DISCuSSION
This study investigated two different approaches for funnel plots with real- world data that can be used to evaluate variation in hospital performance, and to identify those hospitals worthy for further investigation. Method II was able to identify hospitals with lower annual volumes as positive outli- ers. Two simulation experiments showed that method II has greater power to identify outliers compared to method I. In addition, when method II detects a hospital as an outlier, the over- or underperformance tends to be more severe.
0.30 0.35 0.40 0.45 0.50 0.55 0.60
02468
Complication rate
Density
method I method II
Figure 4. Density plot of the detected complication rates using method I versus method II in simulation experiment 2.
Funnel plots play an important role as a quality control tool in surgical performance assessment, as they are relatively simple to construct, easy to interpret, and overcome the limitations of ranking as stated by Spie- gelhalter7. Funnel plots have been widely used by surgical quality initia- tives and audits, such as the Bi-National Colorectal Cancer Audit (BCCA) in Australia and New Zealand12 and the National Oesophago-Gastric Cancer Audit (NOGCA)4 and the National Bowel Cancer Audit (NBCA) in the United Kingdom13. Truly divergent hospitals can be identified when plotting adjusted outcome of interest against individual hospital volume whilst still allowing for natural variation. In the ideal situation, this leads to identification of all outliers, independently of hospital volume. However, when using traditional funnel plots this is difficult for low volume hospi- tals, since these hospitals are subject to sampling variation and identifica- tion of positive or negative outliers becomes statistically challenging.
A plethora of studies has been published over the last years, corroborat- ing the positive correlation between volume and outcome14,15. In gen- eral, this has led to concentration of complex care towards (high-volume) specialized centres. This also applies for oesophageal cancer surgery in the Netherlands, with a minimum volume standard of 10 resections per year per hospital as of 2007, and 20 resections per year per hospital from 20118,16. On the other hand, there are numerous studies suggesting no such volume and outcome correlation1,17,18. Using hospital volume as the reason for referral has been criticized as individual low-volume hospitals can have excellent outcomes. However, providing substantial evidence for good performance is challenging for low-volume hospitals due to sta- tistical uncertainty19. In addition, these small hospitals can’t be identified as underperformers as they are underpowered. As shown in this study, the hospital with the smallest biannual volume would have to perform with a major postoperative complication rate of 56 per cent to have a detec- tion power of 50 per cent. In these situations, alternative approaches like
Method II can be used, as they allow low-volume hospitals to incorporate patients over a longer time period. This way a similar amount of patients can be included for each hospital resulting in identical confidence in- tervals for all hospitals and hence equal power to detect under- or over performance. By using these methods, hospital C could be identified as a positive outlier. Results of the simulation study showed that method II has (slightly) greater power to detect compared to method I. Moreover, results showed that when a centre is situated outside the funnel plot constructed by method II, the effect size tends to be greater compared to method I.
These alternative methods could therefore be a useful addition for bench- marking the quality of care in high-risk and low-volume surgery such as oesophageal cancer surgery. Especially when annual hospital volumes vary considerably between hospitals.
Currently, healthcare systems are shifting the focus from volume and productivity to improving value for patients –outcomes in relation to the costs of the care provided20, and outcome-based referral is advo- cated rather than volume based referral21. This is also incorporated in reimbursement systems of many countries. A wide spectrum of payment models has been introduced that financially reward providers based on their performance regarding specific outcome measures, such as pay-for- performance and bundled payment22. In this environment, it is critical to develop instruments that are sensitive and reliable enough to monitor performance for individual hospitals, either low or high volume. Results of this study show (method II) that hospital A and C are both low volume centres with a biannual volume of 33 and 38 patients respectively, and are performing significantly better than the national average. These results support outcome-based referral and payment models rather than solely focusing on high-volume centres.
This study also has limitations as many studies have shown that only reporting of rare outcomes (e.g. postoperative mortality) or small samples have a limited role in hospital-specific benchmarking9,19. As demonstrated in Figure 3, both Method I and II are underpowered. Even the largest cen- tre would need to have a complication rate of 46 per cent (compared to the national average of 32 per cent) to have sufficient power to be flagged as an outlier (i.e. statistical power of 0.9). These great differences in power diminished in method II, because patients can be included over a longer period of time. However, increasing the observational period in order to allow hospitals with an even smaller sample size to reach the set amount of patients/expected events would be less relevant to current performance, since surgical staff and techniques may change over time. On the other hand, a similar amount of patients’ needs to be treated before outliers can be identified. In the case of a bad-performing hospital this means that patient harm is equal for both low- and high-volume hospitals. Finally, an important disadvantage of traditional funnel plots as a quality control tool for hospital performance also applies for these alternative methods as performance is measured at a certain time point (cross sectional as- sessment). This makes it difficult to detect gradually changing trends and results are always out of date. This is in contrast with other case-by-case monitoring techniques (Variable Life Adjusted Display (VLAD) charts, cumulative sum (CUSUM) charts and cumulative funnel plots)23-26.
Worldwide programs focusing on concentration of high-risk surgery towards high-volume hospitals take place, with good results, also for oesophageal cancer. However, despite current volume standards that apply for oesophageal cancer surgery in the Netherlands annual hospital volumes vary considerably between hospitals. Therefore, alternative methods that provide us with additional information regarding varia- tion in hospital performance were described in this study. The methods presented in this study provide a valid and reliable tool for identifying
centres of excellence not solely based on hospital volume. Stimulating outcome-based referral in combination with volume standards could lead to further quality improvement. Additionally, the presence of truly divergent hospitals as identified by these methods, either as good or bad performers, will give further insight in underlying factors behind the qual- ity of care provided.
REFERENCES
1. Khuri SF, Daley J, Henderson W, et al. The Department of Veterans Affairs’ NSQIP:
the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care.
National VA Surgical Quality Improvement Program. Ann Surg 1998;228(4):491- 507.
2. Ingraham AM, Richards KE, Hall BL, et al. Quality improvement in surgery: the American College of Surgeons National Surgical Quality Improvement Program approach. Adv Surg 2010;44:251-67.
3. Cohen ME, Liu Y, Ko CY, et al. Improved Surgical Outcomes for ACS NSQIP Hospitals Over Time: Evaluation of Hospital Cohorts With up to 8 Years of Participation. Ann Surg 2016;263(2):267-73.
4. Digital NHSN. National Oesophago-Gastric Cancer Audit (NOGCA), http://content.
digital.nhs.uk/og (accessed 17-03-2017).
5. Linder G, Lindblad M, Djerf P, et al. Validation of data quality in the Swedish Na- tional Register for Oesophageal and Gastric Cancer. Br J Surg 2016;103(10):1326- 35.
6. Van Leersum NJ, Snijders HS, Henneman D, et al. The Dutch surgical colorectal audit. Eur J Surg Oncol 2013;39(10):1063-70.
7. Spiegelhalter DJ. Funnel plots for comparing institutional performance. Stat Med 2005;24(8):1185-202.
8. Busweiler LA, Wijnhoven BP, van Berge Henegouwen MI, et al. Early outcomes from the Dutch Upper Gastrointestinal Cancer Audit. Br J Surg 2016; 103(13):1855-1863.
9. Dimick JB, Welch HG, Birkmeyer JD. Surgical mortality as an indicator of hospital quality: the problem with small sample size. Jama 2004;292(7):847-51.
10. Osborne NH, Ko CY, Upchurch GR, Jr., et al. The impact of adjusting for reliability on hospital quality rankings in vascular surgery. J Vasc Surg 2011;53(1):1-5.
11. Edge SB, Byrd DR, Compton CC, et al. AJCC Cancer Cancer staging Manual (7th edn).
Springer: New York, 2010.
12. Teloken PE, Spilsbury K, Platell C. Analysis of mortality in colorectal surgery in the Bi-National Colorectal Cancer Audit. ANZ J Surg 2016;86(6):454-8.
13. Digital NHSN. National Bowel Cancer Audit (NBCA): http://content.digital.nhs.uk/
bowel (accessed 17-03-2017).
14. Birkmeyer JD, Dimick JB, Birkmeyer NJ. Measuring the quality of surgical care:
structure, process, or outcomes? J Am Coll Surg 2004;198(4):626-32.
15. Birkmeyer JD, Siewers AE, Finlayson EV, et al. Hospital volume and surgical mortal- ity in the United States. N Engl J Med 2002;346(15):1128-37.
16. Dikken JL, Dassen AE, Lemmens VE, et al. Effect of hospital volume on postop- erative mortality and survival after oesophageal and gastric cancer surgery in the Netherlands between 1989 and 2009. Eur J Cancer 2012;48(7):1004-13.
17. Kozower BD, Stukenborg GJ. Hospital esophageal cancer resection volume does not predict patient mortality risk. Ann Thorac Surg 2012;93(5):1690-6; discussion 96-8.
18. Kothari AN, Blanco BA, Brownlee SA, et al. Characterizing the role of a high-volume cancer resection ecosystem on low-volume, high-quality surgical care. Surgery 2016;160(4):839-49.
19. Seaton SE, Barker L, Lingsma HF, et al. What is the probability of detecting poorly performing hospitals using funnel plots? BMJ Qual Saf 2013;22(10):870-6.
20. Porter ME. What is value in health care? N Engl J Med 2010;363(26):2477-81.
21. Wouters MW, Krijnen P, Le Cessie S, et al. Volume- or outcome-based referral to improve quality of care for esophageal cancer surgery in The Netherlands. J Surg Oncol 2009;99(8):481-7.
22. Chee TT, Ryan AM, Wasfy JH, et al. Current State of Value-Based Purchasing Pro- grams. Circulation 2016;133(22):2197-205.
23. Steiner SH, Cook RJ, Farewell VT. Risk-adjusted monitoring of binary surgical outcomes. Med Decis Making 2001;21(3):163-9.
24. Steiner SH, Cook RJ, Farewell VT, et al. Monitoring surgical performance using risk- adjusted cumulative sum charts. Biostatistics 2000;1(4):441-52.
25. Steiner SH, Mackay RJ. Monitoring risk-adjusted medical outcomes allowing for changes over time. Biostatistics 2014;15(4):665-76.
26. Kunadian B, Dunning J, Roberts AP, et al. Cumulative funnel plots for the early detection of interoperator variation: retrospective database analysis of observed versus predicted results of percutaneous coronary intervention. Bmj 2008;336(7650):931-4.
