An ordinal prediction model for the diagnosis of non-obstructive coronary artery and multi-vessel disease in the CARDIIGAN cohort
Michael Edlinger, PhDa, Jakob Dörler, MDb, Hanno Ulmer, PhDa, Maria Wanitschek, MDb, Ewout W Steyerberg, PhDc,d, Hannes F Alber, MDe,f, Ben Van Calster, PhDg
a Department of Medical Statistics, Informatics, and Health Economics, Medical University Innsbruck, Austria
b University Clinic of Internal Medicine III - Cardiology and Angiology, Medical University Innsbruck, Austria
c Department of Public Health, Erasmus University Medical Centre, Rotterdam, the Netherlands ?
d Department of Biomedical Data Sciences, Leiden University Medical Centre, the Netherlands
e Department of Cardiology and Karl Landsteiner Institute for interdisciplinary science, Rehabilitation Centre Münster in Tyrol, Austria
f Department of Internal Medicine and Cardiology, Klinikum Klagenfurt, Austria
g Department of Development and Regeneration, KU Leuven, Belgium
Correspondence:
Ben Van Calster
Ben.VanCalster@kuleuven.be
Abstract
Objectives: In this study it was aimed to develop and internally validate an ordinal diagnostic prediction model, based on an existing binary one, in patients with suspected coronary artery disease (CAD) who were referred for elective invasive conventional coronary angiography.
Background: The severity of disease is relevant for the evaluation and the choice of treatment of patients. For some low risk patients the angiography might be avoided.
Methods: We included 4,888 patients from the Coronary Artery disease Risk Determination In Innsbruck by diaGnostic ANgiography (CARDIIGAN) cohort. We used cumulative logit modelling to estimate probabilities of five incrementally relevant disease categories (no CAD, non-obstructive stenosis, and one-, two- and three-vessel disease). We performed regression analyses with 11 predictors (age, sex, chest pain, diabetes, hypertension, dyslipidaemia, smoking, HDL, LDL, fibrinogen, and CRP). We evaluated discrimination and calibration with bootstrap resampling procedures.
Results: Age, sex and HDL cholesterol had a large prognostic effect. The model could not separate adjacent disease categories, but performed well for categories far apart and for the dichotomised evaluations (c = ..?) with low optimism. The value of the ordinal model is shown by comparison of the disease category predictions of a patient with the baseline probabilities.
Conclusions: The proposed ordinal diagnostic model includes 11 easily obtainable variables, which provides a detailed assessment of the extent of coronary artery disease. Further external validation is necessary before introduction in clinical practice is possible.
Introduction
Many cardiologic studies aim at unravelling the causes of coronary artery (CAD) and multi-vessel disease. The reference standard to diagnose these diseases is conventional coronary angiography, but this procedure can be harmful and involves considerable costs. Therefore, there are also attempts to find diagnostic predictors besides causes, since preselection of patients based on easily obtained information could be advantageous. Recently for example, it has been suggested that high-sensitivity C-reactive protein (hs-CRP) is positively associated with severity of CAD [ CITATION Lia \l 3079 ]. A different study found increased IL-6 levels, male gender, and diabetes as predictors of obstructive CAD [ CITATION Exp \l 3079 ]. Cho et al [CITATION Cho \l 3079 ] reported conventional risk factors (hypertension, diabetes mellitus, dyslipidaemia, triglycerides, HDL cholesterol, glucose, and insulin) and also hs-CRP and IL-6 as predictors from a multivariable model. And yet another study came to similar conclusions among patients with lower extremity peripheral artery disease [ CITATION Cho1 \l 3079 ]:
diabetes and the number of cardiovascular risk factors (i.e. hypertension,
diabetes, smoking, and dyslipidaemia) were associated with a higher probability of a CAD diagnosis. Among a group of patients with coronary angiography
performed because of suspected CAD, impaired renal function was related to CAD presence and extent [ CITATION Dog \l 3079 ]. However, these studies are not all methodologically sound and rather inappropriate for predictive estimation
purposes.
For proper diagnostic prediction, statistical multivariable modelling can be applied with performance evaluation [ CITATION Ste09 \l 3079 ]. Concerning CAD, Genders et al [ CITATION Gen1 \l 3079 ] recently presented a model including European patient data, which was then externally validated [ CITATION Edl67 \l 3079 ]. The main aim of this model was to predict obstructive stenosis among a group of patients with suspected CAD. The outcome measure was binary,
therefore patients with low probability of a stenosis might benefit in that they can be treated conservatively with lifestyle recommendations, medication
prescriptions, and regular check-ups instead of having to undergo the invasive procedure of coronary angiography. However, with CAD a distinction is often made between how many arteries are seriously affected, so besides non-
obstructive CAD, also one-, two- and three-vessel disease are distinguished. The severity of disease is actually also relevant for the choice of intervention besides
conservative treatment, including percutaneous coronary intervention (PCI) and coronary artery bypass surgery (CABG) [ CITATION Moh \l 3079 ][ CITATION Hea \l 3079 ]. So clinically speaking, modelling a dichotomous outcome (presence or absence of an obstructive stenosis) is a too restrictive approach. Being more specific about the grading of the disease could be advantageous for patient evaluation and treatment. Also from a methodological and theoretical point of view it makes sense to differentiate between more than two ordinal outcome categories, since this implies the use of more information, thus the use of the data is more efficient.
In other areas the use of an ordinal modelling has already been applied prosperously. In a study of patients with aneurysmal subarachnoid haemorrhage, a model was developed to predict clinical outcome after two months, using clinical features and neuro-imaging readily available on admission. The outcome variable was the modified Rankin Scale running from 0 (no symptoms) to 6 (death) [ CITATION Ris \l 3079 ]. In traumatic brain injury research the 5-point Glasgow Outcome Scale is often applied, usually measured at six months after injury. In this field, McHugh et al [ CITATION McH \l 3079 ] actually performed a simulation study to investigate the potential efficiency gains to be achieved with an ordinal analysis, as compared to a binary one. They concluded that the
required sample size of for example clinical trials could be reduced by over 40%.
A real-life situation has also been investigated and indeed pointed to a
substantially greater statistical power to detect a treatment effect with the same sample size, thus enabling the detection of smaller treatment effects [ CITATION Roo \l 3079 ]. And when evaluating the so called IMPACT studies on traumatic brain injury, the clinical relevance of ordinal analyses is touched upon [ CITATION Maa \l 3079 ]. Examples are given, like when a patient has a poor prognosis survival would be particularly relevant and on the other hand for a patient with good prospects complete recovery is the only improvement that can be achieved.
Such an approach makes more sense than just joining all patients together and dichotomising the outcome somewhere in the middle. A binary outcome can be too restricted for certain diseases, too much a simplification and not doing enough justice to the issues at hand. It has already been suggested to perform more elaborate research differentiating obstructive CAD by its severity, because it can be helpful in the decision-making process concerning the application of an angiography and the treatment strategy [ CITATION Edl67 \l 3079 ].
In this study we wanted to develop an ordinal diagnostic prediction model, based on an earlier presented binary one, in patients with suspected CAD and vessel disease who were referred for elective invasive conventional coronary angiography, and perform an internal validation of the model.
Materials and methods
The Coronary Artery disease Risk Determination In Innsbruck by diaGnostic ANgiography (CARDIIGAN) cohort has been described previously [ CITATION Edl67
\l 3079 ]. In short, during several years the inclusion was performed of 8,296 consecutive patients with chest pain or symptoms suggestive of CAD undergoing elective coronary angiography at a single-centre secondary and tertiary
cardiology clinic. After applying the in- and exclusion criteria, 4,888 patients without known previous CAD or other heart diseases and without a history of coronary revascularisation were available for the current study. Relevant data were recorded as in routine clinical practice in a prospective quality enhancement initiative. Patients gave their written informed consent for the coronary
angiography.
Data was gathered on basic characteristics, medical history, symptoms, laboratory results, and therapy decision right away when the catheterisation was done. Definitions and procedures have been summed up by Edlinger et al
[ CITATION Edl67 \l 3079 ]; the cut-off of stenosis was 70% and the left main artery weighed as three vessels (where ≥50% stenosis thereof was reset to
≥70%) and the three parts of the left anterior descending artery counted as one vessel. Thus, the outcome variable consisted of the following five categories: no CAD, non-obstructive stenosis, one-vessel disease, two-vessel disease, and three- vessel disease [ CITATION Lev \l 3079 ]. All study data was handled by the first author, who had complete access, and the main analysis done by the last author.
Baseline results are presented as proportions for categorical variables and means and standard deviations (or medians and interquartile ranges) for
continuous ones. Since nearly 2% of the clinical data was missing, concerning about a quarter of the participants, multiple imputation (20 times) was applied to avoid potential biases from this source [ CITATION LiP \l 3079 ][ CITATION Moo \l 3079 ][ CITATION Lit02 \l 3079 ], as described elsewhere [ CITATION Edl67 \l 3079 ]. We based the ordinal prediction model on earlier work with a binary model [ CITATION Edl67 \l 3079 ] and a priori used the same predictors, under the assumption that they would be relevant here. Model development was performed by combining the 20 imputed datasets according to Rubin's rules [ CITATION Whi \l 3079 ]. We used the cumulative logit model. Here, the ordinal outcome with 5 disease categories is characterised in terms of 4 cumulative category
contrasts. It is also called the proportional odds model. Basically the same odds are assumed for a prognostic factor when comparing each two sets of adjoining outcome categories. Thereby identical log-odds were assumed across the four cut points between the five disease categories [ CITATION Ana \l 3079 ]. The model served as input for a nomogram to depict the relative contribution of the
predictors to the diagnostic characterisation and the cumulative risk estimates of the various disease categories.
After establishing the apparent diagnostic performance ordinal model, we performed an internal validation by bootstrapping. We developed the model in 200 samples that were drawn with replacement per imputation dataset and tested each model an the imputed data set. We hence estimated the optimism of the prediction model [ CITATION Ste09 \l 3079 ]. Performance was evaluated by a ordinal concordance (c) statistic, to evaluate discrimination, and the calibration slope, to characterise the overall strength of predictor effect. Discrimination measures how well a prediction model distinguishes between patients with an outcome present and for those without. The ordinal c statistic was estimated based on the the observed disease category versus the expected outcome value from the predicted probability for each patient:
Value=0
∑
4
Pr (Value)∗Value
In addition, we estimated the c statistics for each dichotomisation of separate disease categories and also separately for the four dichotomisations of the five outcome groups [ CITATION Van1 \l 3079 ]. Calibration assesses the
correspondence between the observed and the predicted outcomes. It was assessed as the general calibration slope and the slopes for the four
dichotomised comparisons[ CITATION Van2 \l 3079 ][CITATION Van3 \l 3079 ]. The apparent as well as the optimism corrected performance was assessed. The slope at bootstrapping provided a shrinkage factor for the final model, which provides slightly less extreme probability estimates compared to the original model. The optimism-corrected dichotomous calibration slopes were used for recalibration of the ordinal prediction model.
The model predictions for the disease categories was compared with estimates based on the model. Relative risks are used to give an indication of the extent of change of the probability that applies to a certain disease category when estimated by the model for a particular set of predictor values.
The data preparation was performed with SPSS version 19.0, the multiple imputations with Stata/MP version 11.2, and the main analyses with R version 3.4.1 (including the VGAM, HMISC, MITOOLS, and RMS libraries).
Results
Of the total of 4,888 CAD suspected patients, 3,028 (62%) were male and age ranged from 18 to 89 years. Among these patients, 1,381 (28%) did not have CAD while 1,901 (39%) had at least one seriously affected artery (one-, two-, or three-vessel disease). For most predictors the value (or proportion affected) increased with each next category of disease severity, except for HDL cholesterol with (expectedly) an opposite tendency. For sex, age, hypertension,
dyslipidaemia, and HDL cholesterol the patients with two- and three-vessel disease look very similar. The data on smoking and the laboratory findings are not complete; with smoking status the situation is worst with a missing value for 13% of the patients (table 1).
In table 2 the estimates of the multivariable cumulative logit model (based on the multiply imputed data) are shown. Each disease category has its own intercept and the other coefficients apply to all categories. The absolute
magnitude of the intercepts increases with disease severity, as expected. Rather large effects appear for male sex (OR=3.02, 95% CI 2.67 to 3.40), age (per 10 years: OR=1.78, 95% CI 1.69 to 1.89) and HDL cholesterol (per 10 mg/dl:
OR=0.84, 95% CI 0.81 to 0.87), but small ones for hypertension (OR=1.16, 95%
CI 0.99 to 1.36), smoking (OR=1.21, 95% CI 1.08 to 1.36) and C-reactive protein (OR=1.13, 95% CI 0.95 to 1.35).
In figure 1 the nomogram is depicted in which the relative potential contribution of the various predictors to the model is manifest. Each predictor contributes at the maximum 100 points (scale at the top), but for example diabetes provides only about 20 points when present. Adding all points of the predictors up leads to a total sum of points scored (scale from 0 to 400 shown under the predictors in figure 1) and from this total one can read out the
estimated risk of having the disease of at least a certain intensity (bottom four bars). Let us say a patient has a total score of 250, then the risk of at least a non- obstructive stenosis (or worse) is between 0.90 and 0.95, the risk of one- or more vessel disease about 0.70, the risk of two- or three-vessel disease over 0.40, and the risk of three-vessel disease between 0.20 and 0.30.
The overall apparent discrimination of the model adds up to a c statistic of 0.71 (95% CI 0.69 to 0.73) (table 3). The dichotomous c statistics varied over
quite a large range between 0.54 (95% CI 0.50 to 0.58) for the distinction
between two- and three-vessel disease and 0.86 (95% CI 0.84 to 0.88) for the two most extreme disease categories. The model can discriminate well between the lowest (no CAD) and the other disease categories (c >0.70), but less so between the three adjacent categories of patients with stenosis (c between 0.54 and 0.59). This is reflected in the dichotomised c statistic of 0.77 (95% CI 0.75 to 0.78) between the unaffected and all other patients (category 0 vs. categories 1 to 4 taken together). In the current development model the apparent general calibration slope is, by definition, equal to 1.00 [ CITATION Ste09 \l 3079 ].
However, the slopes per dichotomisation deviate: on the one hand, the
probability of no CAD compared to the rest (category 0 vs. categories 1 to 4) is underestimated (slope=1.13), while the other risks are overestimated.
Overestimation is largest for three-vessel disease (category 4 vs. categories 0 to 3) with a slope of 0.90. Under- and overestimation is a consequence of assuming that the proportional odds apply.
The internal validation showed only a limited amount of optimism regarding discrimination (table 3). The overall c statistic had an optimism of 0.12%, too small to lead to a substantial different corrected value (0.71, 95% CI 0.69 to 0.73). For the dichotomised estimates there are roughly the same values, all within the range of 0.05% to 0.21% optimism, so the corrected c statistic in each case was practically the same as the apparent one. Calibration however appeared to be affected slightly more by optimism, with about 1%, and this resulted in somewhat lower slope values. The general calibration slope is 0.99, which means that the expected value of the outcome category is practically in accordance with the predicted value and little model shrinkage is necessary.
Since the dichotomised calibration slopes differed somewhat from the ideal value of 1.00, the model was recalibrated by the optimism-corrected dichotomous calibration slopes.
The bar charts in figure 2 exemplify application of the ordinal model for three theoretical patients. The first example patient is a rather young man with dyslipidaemia who has ever smoked, but was otherwise not very unfavourable regarding the other predictors. The probability he does not have a CAD is 44.6%
and not an obstructive coronary artery 79.0%. The relative risk of a CAD, compared to the baseline probability, is 0.77 (55.4% divided by 71.7%); the relative risk of a vessel disease is 0.54. If this would have been a woman with
otherwise the same scores, the estimates are a probability of 73.5% and 91.0%, and a relative risk of 0.37 and 0.23 respectively. The second example concerns a somewhat older man with hypertension, dyslipidaemia, and rather low HDL cholesterol; his cumulative risk of having at least an obstructive stenosis is 46.6%
and the highest relative risk is with one-vessel disease (relative risk of 1.33). The third patient is even worse off at a high age of 84 and various other high scores on the predictors, accumulating in a chance of 96.9% of having a CAD and of 50.7% of having a two- or three-vessel disease (with high relative risks of 2.44 and 3.07 respectively).
Discussion
The proposed ordinal diagnostic prediction model performed relatively well in distinguishing between more extreme / distant disease categories with limited statistical optimism. Especially age, sex, and HDL cholesterol had a large effect on the estimated probabilities and the potential of the model is promising.
The large number of patients led to a stable model. External validation of the model will still be needed before it can be introduced as an extra tool for cardiologists to decide on the necessity of performing conventional coronary angiography and on further treatment steps for such patients. We consider that, at a particular centre, the distribution of the five considered disease categories is roughly known. This is the starting point of the diagnostic process. The potential of this ordinal model is to estimate for each individual patient the chance that a certain disease category is present, based on the value of 11 predictors that are documented anyway. This can lead to better medical decision making. Also, apart from considering the model reproducibility, transportability across different but similar patient groups is worthwhile to further improve clinical decisions
[CITATION Deb1 \l 3079 ].
The diagnostic prediction model could not separate between the two- and three-vessel disease categories. These two groups are relatively small compared to the others and, clinically speaking, the distinction is not very important for treatment decision making. In addition, discrimination was also rather moderate between one- and two-vessel disease and between non-obstructive stenosis and one-vessel disease. This is not very surprising since this disease reflects the gradual ageing and deterioration of the cardiovascular system. Also, the cut-off was set at 70% stenosis based on visual inspection. So where one might think in terms of distinct disease categories, we are actually dealing with a continuum.
Further investigation should clarify whether the two- and three-vessel disease categories can better be collapsed, or one might even go one step further and group the three highest categories together into one. This way the 5-category outcome would be broken down into a 3-category classification, with no CAD, non-obstructive stenosis and any vessel disease; but then the possible utility of the model to help decide between treatment with PCI or with CABG would disappear. On the other hand, it appears that a more elaborate model like the one here, with more than two disease categories, does indeed enable better
model-building because of more informative and efficient use of the data.
Discrimination seems to be better between several single categories (far apart) and according to various dichotomisations of all patients than in the binary model published before (8). Further research can shed light on this matter and help to come to a definite conclusion about how many (and which) categories are best for the ordinal diagnostic model.
Here the proportional odds model was applied and, after recalibration, performed well. In methodological studies, other types of ordinal models can be evaluated and compared. For example, partial proportional odds modelling has been proposed, in which proportional odds are assumed for some predictors but not for others [ CITATION Ana \l 3079 ]. Thereby maybe a better fit might be accomplished, but for the present study the larger number of estimated
coefficients was undesirable since the model then is less parsimonious and thus more complicated in application. Such forthcoming research will show how the trade-off can be made and whether it is advantageous to proceed with more complicated ordinal models in the clinical field concerned here.
The strengths of the current study include the application, to our knowledge for the first time, of more than two ordinal disease categories in a diagnostic prediction model of CAD. This leads to the use of more available information and is more efficient and elaborate so more is gained from the data.
In addition, the clinical relevance is given because various treatment strategies are available depending on the extent of disease. It is also advantageous to have a large cohort of nearly 5,000 patients, so estimates can be rather precise and robust. The data was collected prospectively and is of good quality, including several routine laboratory parameters. This makes it a fruitful source for data modelling and eventually even model comparison studies in the future.
As for the limitations, obviously as with all model development, external validation still has to be performed. Such testing is necessary before a definite model can be presented in clinics for patient use. Another limitation is the possibility of verification bias since some relevant patients might not have been referred for angiography, especially relevant among less affected patients. Also other predictors not included in this study might be valuable, as suggested in other studies, for example carotid artery information [ CITATION Exp \l 3079 ] and amount of calcification of the coronary arteries [CITATION Gen \l 3079 ].
Sometimes missing data is seen as a limitation, even though they are inevitable in observational studies. However, through multiple imputation all resources available have been used and more valid and more precise results are thus achieved [CITATION Ste02 \l 3079 ]. Finally, one could argue that the data was gathered quite some time ago. There is however no reason to believe that predictor effects might have changed and few predictive diagnostic studies are yet available, so one has to make the most of it.
In conclusion, the diagnosis of coronary artery and vessel disease can be estimated with a limited number of easily obtained clinical variables with the use of an ordinal prediction model. Although external validation, and possibly
updating, is still needed, the model performance is quite satisfactory. Applying five ordinal disease categories has methodological and clinical advantages above just distinguishing between the presence and absence of an obstructive coronary artery disease.
References
1. Liang Y, Hou Y, Niu H, Lu M, Xue L, Sun Q. Correlation of high-sensitivity C- reactive protein and carotid plaques with coronary artery disease in elderly patients. Exp Ther Med 2015;10:275-278
2. Guaricci AI, Pontone G, Fusini L, De Luca M, Cafarelli FP, Guglielmo M, Baggiano A, Beltrama V, Muscogiuri G, Mushtaq S, Conte E, Guglielmi G, Andreini D,
Brunetti ND, Di Biase M, Bartorelli AL, Pepi M. Additional value of inflammatory biomarkers and carotid artery disease in prediction of significant coronary artery disease as assessed by coronary computed tomography angiography. Eur Heart J Cardiovasc Imaging 2016 doi:10.1093/ehjci/jew173
3. Cho S, Lee SH, Park S, Jee SH, Hong MK, Chung N, Cho SY, Jang Y. The additive value of multiple biomarkers in prediction of premature coronary artery disease.
Acta Cardiol 2015;70:205-210
4. Cho SW, Kim BG, Kim DH, Kim BO, Byun YS, Rhee KJ, Lee BK, Goh CW.
Prediction of coronary artery disease in patients with lower extremity peripheral artery disease. Int Heart J 2015;56:209-212
5. Doganer YC, Rohrer JE, Aydogan U, Barcin C, Cayci T, Saglam K. Association of renal function, estimated by four equations, with coronary artery disease. Int Urol Nephrol 2015;47:663-671
6. Steyerberg EW. Clinical prediction models; a practical approach to development, validation, and updating. New York: Springer, 2009.
7. Genders TS, Steyerberg EW, Hunink MG, Nieman K, Galema TW, Mollet NR, de Feyter PJ, Krestin GP, Alkadhi H, Leschka S, Desbiolles L, Meijs MF, Cramer MJ, Knuuti J, Kajander S, Bogaert J, Goetschalckx K, Cademartiri F, Maffei E, Martini C, Seitun S, et al. Prediction model to estimate presence of coronary artery disease:
retrospective pooled analysis of existing cohorts. Br Med J 2012;344:e3485
8. Edlinger M, Wanitschek M, Dörler J, Ulmer H, Alber HF, Steyerberg EW. External validation and extension of a diagnostic model for obstructive coronary artery disease: a cross-sectional predictive evaluation in 4,888 patients of the Austrian Coronary Artery disease Risk Determination In Innsbruck by diaGnostic
ANgiography (CARDIIGAN) cohort. BMJ Open 2017;7:e014467
9. Mohr FW, Morice MC, Kappetein AP, Feldman TE, Ståhle E, Colombo A, Mack MJ, Holmes DR, Morel MA, Dyck N van, Houle VM, Dawkins KD, Serruys PW. Coronary artery bypass graft surgery versus percutaneous coronary intervention in
patients with three-vessel disease and left main coronary disease: 5-year follow- up of the randomised, clinical SYNTAX trial. Lancet 2013;381:629-638
10. Head SJ, Davierwala PM, Serruys PW, Redwood SR, Colombo A, Mack MJ, Morice MC, Holmes DR, Feldman TE, Ståhle E, Underwood P, Dawkins KD, Kappetein AP, Mohr FW. Coronary artery bypass grafting vs. percutaneous
coronary intervention for patients with three-vessel disease: final five-year follow- up of the SYNTAX trial. Eur Heart J 2014;35:2821-2830
11. Risselada R, Lingsma HF, Molyneux AJ, Kerr RSC, Yarnold J, Sneade M, Steyerberg EW, Sturkenboom MCJM. Prediction of two month modified Rankin Scale with an ordinal prediction model in patients with aneurysmal subarachnoid haemorrhage. BMC Med Res Methodol 2010;10:86
12. McHugh GS, Butcher I, Steyerberg EW, Marmarou A, Lu J, Lingsma HF, Weir J, Maas AI, Murray GD. A simulation study evaluating approaches to the analysis of
ordinal outcome data in randomized controlled trials in traumatic brain injury:
results from the IMPACT Project. Clin Trials 2010;7:44-57
13. Roozenbeek B, Lingsma HF, Perel P, Edwards P, Roberts I, Murray GD, Maas AI, Steyerberg EW; IMPACT study group; CRASH trial collaborators. The added value of ordinal analysis in clinical trials: an example in traumatic brain injury.
Crit Care 2011;15:R127
14. Maas AI, Murray GD, Roozenbeek B, Lingsma HF, Butcher I, McHugh GS, Weir J, Lu J, Steyerberg EW; on behalf of the International Mission on Prognosis
Analysis of Clinical trials in Traumatic brain injury (IMPACT) study group.
Advancing care for traumatic brain injury: findings from the IMPACT studies and perspectives on future research. Lancet Neurol 2013;12:1200-1210
15. Levine GN, Bates ER, Blankenship JC, Bailey SR, Bittl JA, Cercek B, Chambers CE, Ellis SG, Guyton RA, Hollenberg SM, Khot UN, Lange RA, Mauri L, Mehran R, Moussa ID, Mukherjee D, Nallamothu BK, Ting HH. 2011 ACCF/AHA/SCAI guideline for percutaneous coronary intervention. A report of the American College of Cardiology Foundation/American Heart Association task force on practice guidelines and the Society for Cardiovascular Angiography and Interventions. J Am Coll Cardiol 2011;58:e44-122
16. Li P, Stuart EA, Allison DB. Multiple imputation: a flexible tool for handling missing data. J Am Med Assoc 2015;314:1966-1967
17. Moons KG, Donders RA, Stijnen T, Harrell FE. Using the outcome for imputation of missing predictor values was preferred. J Clin Epidemiol 2006;59:1092-1101
18. Little RJA, Rubin DB. Statistical analysis with missing data. Hoboken: Whiley- InterScience, 2002 (2nd edn.)
19. White IR, Royston P, Wood AM. Multiple imputation using chained equations:
issues and guidance for practice. Stat Med 2011;30:377-399
20. Ananth CV, Kleinbaum DG. Regression models for ordinal responses: a review of methods and applications. Int J Epidemiol 1997;26:1323-1333
21. Van Calster B, Vergouwe Y, Van Belle V, Looman CWN, Timmerman D,
Steyerberg EW. Assessing the discriminative ability of risk models for more than two outcome categories: a perspective. Eur J Epidemiol 2012;27:761-770
22. Van Calster B, Van Hoorde K, Valentin L, Testa AC, Fischerova D, Van Holsbeke C. Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study. Br Med J 2014;349:g5920
23. Van Hoorde K, Vergouwe Y, Timmerman D, Van Huffel S, Steyerberg EW, Van Calster B. Assessing calibration of multinomial risk prediction models. Stat Med 2014;33:2585-2596
24. Debray TPA, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg EW, Moons KGM.
A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015;68:279-289
25. Genders TSS, Coles A, Hoffmann U, Patel MR, Mark DB, Lee KL, Steyerberg EW, Hunink MGM, Douglas PS und investigators, CAD consortium and the PROMISE. The external validity of prediction models for the diagnosis of
obstructive coronary artery disease in patients with stable chest pain; insights
from the PROMISE trial. JACC Cardiovasc Imaging 2017 doi:10.1016/j.jcmg.2017.02.020
26. Sterne JA, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, Wood AM, Carpenter JR. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. Br Med J 2009;338:b2393
Table 1. Characteristics of the CARDIIGAN patient group (n=4,888) by disease categorya
no coronary artery
disease non-obstructive
stenosis one-vessel
disease two-vessel
disease three-vessel
disease n missing n=1,381 (28.3%) n=1,606 (32.9%) n=997 (20.4%) n=475 (9.7%) n=429 (8.8%)
Mean age in years (SD) 59.4 (11.2) 65.9 (9.7) 65.2 (10.3) 67.0 (9.9) 67.4 (10.7) 0
Male sex 45.0% 60.3% 73.1% 78.5% 78.6% 0
Chest pain 54.2% 57.7% 66.5% 70.3% 73.2% 0
Diabetes mellitus 9.0% 16.4% 16.8% 19.8% 25.2% 0
Hypertension 67.5% 79.3% 77.6% 83.6% 82.2% 0
Dyslipidaemia 59.7% 63.1% 64.3% 72.0% 68.5% 0
Ever smoked 41.0% 44.7% 49.4% 49.9% 51.7% 640
Mean HDL cholesterolb (SD) 60.9 (18.9) 57.0 (17.0) 54.0 (15.1) 52.0 (14.9) 52.2 (15.9) 312
Mean LDL cholesterolb (SD) 127 (34) 126 (36) 131 (40) 130 (38) 135 (40) 310
Median fibrinogenb (IQR) 341 (277; 408) 362 (297; 443) 368 (301; 455) 370 (304; 462) 384 (318; 480) 119
CRP >1.00 mg/dl 10.3% 12.6% 17.9% 16.0% 18.8% 96
a coronary artery disease cut-off at 70% stenosis, but for the left main artery at 50%; the left anterior descending, proximal and distal left anterior descending arteries add up to one vessel, the left main artery counts as three vessels
b in mg/dl
CARDIIGAN Coronary Artery disease Risk Determination In Innsbruck by diaGnostic ANgiography cohort, CRP C-reactive protein, HDL high-density lipoprotein, IQR interquartile range, LDL low-density lipoprotein, SD standard deviation
Table 2. Proportional odds ordinal regression estimates of predictors for a one disease category increase in the CARDIIGAN patient group (n=4,888)
coefficient (SE) odds ratio (95% CI) Intercept non-obstructive stenosis -7.17 (0.64) .
Intercept one-vessel disease -8.84 (0.65) . Intercept two-vessel disease -10.02 (0.65) . Intercept three-vessel disease -10.95 (0.65) .
Age (per 10 years) 0.58 (0.03) 1.78 (1.69 to 1.89) Sex (male vs. female) 1.10 (0.06) 3.02 (2.67 to 3.40)
Chest pain 0.55 (0.06) 1.73 (1.55 to 1.92)
Diabetes mellitus 0.45 (0.07) 1.57 (1.36 to 1.82)
Hypertension 0.15 (0.08) 1.16 (0.99 to 1.36)
Dyslipidaemia 0.37 (0.08) 1.45 (1.24 to 1.68)
Ever smoked 0.19 (0.06) 1.21 (1.08 to 1.36)
HDL cholesterol (per 10 mg/dl) -0.18 (0.02) 0.84 (0.81 to 0.87) LDL cholesterol (per 10 mg/dl) 0.047 (0.008) 1.05 (1.03 to 1.06) ln(fibrinogen) (mg/dl) 0.57 (0.11) 1.76 (1.43 to 2.17) C-reactive protein >1.00 mg/dl 0.12 (0.09) 1.13 (0.95 to 1.35)
CARDIIGAN Coronary Artery disease Risk Determination In Innsbruck by diaGnostic ANgiography cohort, HDL high-density lipoprotein, LDL low-density lipoprotein, ln natural logarithm, SE standard error
Table 3. Performance measures (with 95% CI) of the proportional odds diagnostic model in the CARDIIGAN patient group (n=4,888)
performance
apparent optimism corrected
Ordinal c 0.71 (0.69 to 0.73) 0.12% 0.71 (0.69 to 0.73) Dichotomousa c:
d0 vs. d1 0.71 (0.69 to 0.73) 0.12% 0.71 (0.69 to 0.73) d0 vs. d2 0.78 (0.76 to 0.80) 0.21% 0.78 (0.76 to 0.80) d0 vs. d3 0.85 (0.83 to 0.87) 0.19% 0.85 (0.83 to 0.87) d0 vs. d4 0.86 (0.84 to 0.88) 0.19% 0.86 (0.83 to 0.88) d1 vs. d2 0.59 (0.57 to 0.61) 0.14% 0.59 (0.56 to 0.61) d1 vs. d3 0.68 (0.66 to 0.71) 0.20% 0.68 (0.65 to 0.71) d1 vs. d4 0.71 (0.68 to 0.74) 0.21% 0.71 (0.68 to 0.73) d2 vs. d3 0.59 (0.56 to 0.62) 0.08% 0.59 (0.56 to 0.62) d2 vs. d4 0.63 (0.59 to 0.66) 0.12% 0.63 (0.59 to 0.66) d3 vs. d4 0.54 (0.50 to 0.58) 0.05% 0.54 (0.50 to 0.58) d0 vs. d1-4 0.77 (0.75 to 0.78) 0.17% 0.77 (0.75 to 0.78) d0-1 vs. d2-4 0.72 (0.71 to 0.73) 0.19% 0.72 (0.70 to 0.73) d0-2 vs. d3-4 0.73 (0.71 to 0.75) 0.17% 0.73 (0.71 to 0.74) d0-3 vs. d4 0.72 (0.69 to 0.74) 0.16% 0.72 (0.69 to 0.74)
General calibration slope 1.00 0.96% 0.99
Dichotomousa calibration slope:
d0 vs. d1-4 1.13 1.01% 1.12
d0-1 vs. d2-4 0.91 0.98% 0.90
d0-2 vs. d3-4 0.94 0.96% 0.93
d0-3 vs. d4 0.90 0.95% 0.89
a disease categories: (0) no coronary artery disease, (1) non-obstructive stenosis, (2) one-vessel disease, (3) two-vessel disease, (4) three-vessel disease
c concordance statistic, CARDIIGAN Coronary Artery disease Risk Determination In Innsbruck by diaGnostic ANgiography cohort, CI confidence interval, dx disease category x
Figure 1. Nomogram with the relative contribution of the predictors in the risk estimation of the proportional odds diagnostic model in the CARDIIGAN patient group (n=4,888), including the predicted cumulative risks of disease categoriesa
a the predicted risks are shown for a particular disease category and the categories with more severe disease taken together
CARDIIGAN Coronary Artery disease Risk Determination In Innsbruck by diaGnostic ANgiography cohort, CRP C-reactive protein, HDL high-density lipoprotein, LDL low-density lipoprotein
N.B. The range of the predictors is based on the 5th and 95th percentile of the original (not imputed) data ... (ME: this might give the impression that only these ranges of the predictors are included in the study, e.g. only patients with age between 40 and 80 years)
Figure 2. Stacked bar charts of the baseline probability distribution of the disease categoriesa compared to three patient examplesb of recalibrated diagnostic predictions and relative risksc in the CARDIIGAN patient group (n=4,888)
Baseline
probabilities Patient #1 probabilities
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
28.3%
44.6%
32.9%
34.4%
20.4%
14.4%
9.7%
3.8%
8.8% 2.8%
Baseline
probabilities Patient #2 probabilities
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
28.3%
15.5%
32.9%
37.9%
20.4%
27.1%
9.7% 11.0%
8.8% 8.5%
Baseline
probabilities Patient #3 probabilities
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
28.3%
3.1%
32.9%
18.9%
20.4%
27.3%
9.7%
23.7%
8.8%
27.0%
a disease categories (from bottom to top): no coronary artery disease, non-obstructive stenosis, 1-, 2-, and 3-vessel disease
b with age, sex, pain, diabetes, hypertension, dyslipidaemia, smoking, HDL, LDL, fibrinogen, CRP respectively:
Patient #1: 48, male, no, no, no, yes, yes, 61, 144, 372, no Patient #2: 67, male, no, no, yes, yes, no, 44, 117, 351, yes Patient #3: 84, male, yes, no, yes, yes, no, 43, 182, 258, no
c relative risks of the 5 disease categories (from bottom to top):
Patient #1: 1.58, 1.05, 0.71, 0.39, 0.32 Patient #2: 0.55, 1.15, 1.33, 1.13, 0.96 Patient #3: 0.11, 0.58, 1.34, 2.44, 3.07
