Predictive Computer Models Predictive Computer Models
for Medical Classification Problems for Medical Classification Problems
PhD progress report (2000.8 ~ 2003.10) Student : Chuan LU
Promoters: Prof. Dr. Ir. Sabine Van Huffel Prof. Dr. Ir. Johan Suykens Advisers : Prof. Dr. Dirk Timmerman
Prof. Dr. Ir. Joos Vandewalle
Overview Overview
PhD topic
Doctoral Programme:
courses, publication, meetings...
Research Work
Work plan and timing
PhD Topic PhD Topic
The development, statistical analysis and
clinical evaluation of a new class of predictive models which optimally extract information from patient data.
The attention is focused on intelligence machine learning methods such as neural
networks, kernel based algorithms, and their
integration with Bayesian framework.
Joint Research Activities Joint Research Activities
Classification of ovarian tumors
logistic regression (LR)
artificial neural networks (ANNs)
Bayesian least squares support vector machines (LS-SVMs)
Prediction of pregnancy of unknown location (PUL)
LR, LS-SVMs, relevance vector machines (RVMs)
Variable selection for medical classification problems:
(Bayesian framework)
The Doctoral Programme The Doctoral Programme
‘Direct Tuition’ – (520 h)
Doctoral Courses
Case Studies in Biomedical Data Processing (25x6=150 h)
Phd training course: Longitudinal Data,Incomplete Data, and Causal Inference (18x1=18 h)
Courses in master of statistics
Basic Concepts of Statistical Modeling (60x4=240 h)
Applied Statistical Models (60x4=240 h)
Seminars
Presentation at BioMed Seminar, SISTA (50 x 1 h)
The Doctoral Programme The Doctoral Programme
Other Study Activities and Achievements
Publications (1) – with first authorship
[1]‘Preoperative prediction of malignancy of ovarian tumors using least squares support vector machines’. Artificial Intelligence in Medicine, vol. 28, no. 3, Jul. 2003, pp. 281-306. (200 h)
[2] ‘Using Artificial Neural Networks to Predict Malignancy of Ovarian
Cancers’, in Proc. Of the 23rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society - EMBC2001, Istanbul,
Turkey, Oct. 2001, CD-ROM. (100 h)
[3] ‘Classification of ovarian tumor using Bayesian least squares support vector machines’, accepted for publication in the 9th Conference on Artificial
Intelligence in Medicine Europe (AIME 03), Oct 18-22, Cyprus. (100+60 h) [4] ‘Bayesian Least Squares Support Vector Machines for Classification of
Ovarian Tumors’, Internal Report 02-105, ESAT-SISTA, K.U.Leuven (Leuven, Belgium), 2002.
The Doctoral Programme The Doctoral Programme
Publications (2) – with coauthorship
[1] ‘Prediction of mental development of preterm newborns at birth time using LS-SVM’, in Proc. of ESANN'02, Bruges, Belgium, Apr. 2002, pp. 167-172.
[2] ‘Color Doppler and Gray-Scale Ultrasound Evaluation of the Postpartum uterus’, Ultrasound Obstet. Gynecol., vol. 20, 2002, pp. 586-591
[3] ‘Prospective evaluation of blood flow in the myometrium and in the uterine arteries in the puerperium’, Jan 2003, accepted for publication in Ultrasound in Obstetrics and Gynecology.
[4] ‘Subjective use of serum human chorionic gonadotrophin and progesterone levels for the investigation of pregnancies of unknown location : analysis of interuser variability and experience’, 2003.
[5] ‘Can ectopic pregnancies be predicted using serum hormone levels ?’, submitted, 2003.
[6] ‘A novel neural approach to inverse problems with discontinuities (the GMR neural network)’, in IJCNN'03, Portland, Oregon, Jul. 2003, pp. 3106-3111.
[7] ‘Direct Torque control of induction motors by use of the GMR Neural network’, in IJCNN'03, Portland, Oregon, July 20-24, 2003, pp. 2106-2111.
The Doctoral Programme The Doctoral Programme
Other Study Activities and Achievements
Participation in Scientific Meetings
9th Annual Meeting of the Belgian Statistical Society-BSS2001, Oostende, Oct 2001, Poster Presentation: ‘Prediction of Malignancy of Ovarian Tumors Using Logistic Regression and Artificial Neural Network Models’
the Advanced NATO Study Institute on Learning Theory and Practice (NATO- ASI LTP 2002), Leuven, Belgium, July 8-19, 2002, Poster Presentation:
‘Blackbox classifiers for preoperative discrimination between malignant and benign ovarian tumors’.
10th Annual Meeting of the Belgian Statistical Society-BSS2002, Kerkrade, The Netherlands, October 18-19, 2002 Poster Presentation: ‘Comparative study on variable selection for nonlinear classifiers ’
Poster presentation at study day of IAP network 2001, 2002 and 2003.
Belgian Day of Biomedical Engineering in Brussels, , October 17th 2003,
Poster presentation: ‘Variable selection using linear sparse Bayesian models for medical classification problems’.
The Doctoral Programme The Doctoral Programme
Other Study Activities and Achievements
Supervision of licentiate thesis (2x150+100=400 h)
Thesis supervision for Master of applied statistics: ‘Mathematical models for predicting the evolution of a pregnancy of unknown location’, 2003
Thesis supervision for Master of applied statistics: ‘Prediction of pregnancy evolution and ectopic pregnancy’ , 2002.
Thesis supervision for ERASMUS student from university de Picardie Jules Verne, France, 2003, Approches neuronales pour la résolution de problèmes inverses avec discontinuités
Research Research
- Building blocks - Building blocks
Explorative data analysis (EDA)
Probabilistic modeling techniques
Variable selection
Applications
Explorative Data Analysis Explorative Data Analysis
Gain insights into a data set: structure, imprtant
variables, outiers, and the model suggested by the data.
Techniques: scatterplots, boxplots, histograms, PCA, FA, CCA, biplots, etc.
New nonlinear techniques in EDA: kernel PCA, kernel
CCA, and nonlinear biplots. Uncovering the nonlinear
structure of the data, aid in nonlinear modeling such as
LS-SVM.
Explorative Data Analysis Explorative Data Analysis
Fig. Biplot of Ovarian Tumor data.
The observations are plotted as points (o - benign, x - malignant), the variables are plotted as vectors from the origin.
- visualization of the correlation between the variables
- visualization of the relations between the variables and clusters.
Explorative Data Analysis Explorative Data Analysis
Fig. Nonlinear Biplot for kernel PCA with RBF kernels
Data projected onto pairs of PCs (PCs with the largest correlation with y were selected for visualization) , computed by kernel trick.
Approximate decision boundary: ridge regression (y=1) using pairs of PCs
For kth variable, pseudosamples generated by: setting data mean as starting point, varying the value of variable k while fixing the others.
Variable trajectory: tracing the projection of the pseudosample onto
Probabilistic Modeling Probabilistic Modeling
Probabilistic modeling needed in medical Decis. Supp.
the uncertainty and different mis. class. cost.
Traditional statistical linear probabilistic classifiers:
Linear discriminant analysis (LDA)
Logistic regression (LR)
Bayesian MLPs
Bayesian + Kernel based modeling:
Bayesian LS-SVM classifiers (Suykens 1999, 2001, 2002)
Sparse Bayesian modeling and relevance vector machines (RVMs) (Tipping 2001,2003)
Recipe Recipe
Goal:
Linear model y=w
Tx Nonlinear model
Dealing with uncertainty
Model selection
Sparseness
Ingredients:
Kernel trick: x (x) higher dim. feature space
Bayesian framework
Bayesian Inference Bayesian Inference
Find the maximum a posterior (MAP) estimates of model parameters wMP
and bMP, using conventional LS-SVM training.
The posterior probability of the parameters can be estimated via marginalization using Gaussian probability at wMP, bMP
Assuming a uniform prior p(Hj) over all model, rank the model by the evidence p(D|Hj) evaluated using Gaussian approximation.
( , ) (
) (
( ,
( ,
) )
= p D H p) H
p p D H
D H H
p D
:
Infer hyperparameter Level 2
:
Level 3 ( ) (
( ) p D H p Hj j) ( )
p H D p D H
w b, , ,
p D w b( , , , ) ( ,( H p w b, ) , )H exp( J( , ))p w
D H b
D H P
: for rbf kernels,
(model kernel parameter, e.g.
hyperp
: arameters, e.g. regularization parame rte s)
H
:
infer , for given , Level 1w b H
Variable selection Variable selection
Importance in medical classification problems
economics of data acquisition
accuracy and complexity of the classifiers
gain insights into the underlying medical problem.
Filter approaches: filter out irrelevant attributes before induction occurs
Wrapper approaches: focus on finding attributes that are useful for performance for a specific type of model,
rather than necessarily finding the relevant ones.
Variable selection Variable selection
Heuristic search:
forward, backward, stepwise
hill-climbing, branch and bound…
Variable selection criteria:
Correlation, fisher score,
mutual information
Evidence in Bayesian framework
Classification performance, e.g. AUC
Sensitivity analysis: change in the objective function J by removing variable i: DJ(i)
Statistical chi-square test
Variable selection Variable selection
We focus on evidence (marginal likelihood) based method within the Bayesian framework
Forward / stepwise selection
Bayesian LS-SVM
Sparse Bayesian models
Accounting for uncertainty in variable selection
Application Application
- Ovarian tumor classification - Ovarian tumor classification
Problem
develop a reliable diagnostic tool to discriminate
preoperatively between benign and malignant tumors.
assist clinicians in choosing the appropriate treatment.
Data (from IOTA project)
Patient data collected at Univ. Hospitals Leuven, Belgium, 1994~1999
425 records, 25 features.
291 benign tumors, 134 (32%) malignant tumors.
Application Application
- Ovarian tumor classification - Ovarian tumor classification
Forward variable selection based on Bayesian LS-SVM
Evolution of the model evidence
10 variables were selected based on the training set (first treated 265 patient data) using RBF kernels.
Application Application
- Ovarian tumor classification - Ovarian tumor classification
Predictive power of the models given the selected variables
ROC curves on test Set (data from 160 newest treated patients)
Application Application
- Ovarian tumor classification - Ovarian tumor classification
Performance on test set with rejection based on, e.g.,
The rejected patients need further examination by human experts
Reject AUC Accuracy Sensitivity Specificity 10% (16) 0.9420 88.97 83.72 91.4
5% (8) 0.9343 87.50 82.61 89.8 0% (0) 0.9184 84.38 77.78 87.74 Reject AUC Accuracy Sensitivity Specificity
10% (16) 0.9420 88.97 83.72 91.4
5% (8) 0.9343 87.50 82.61 89.8
0% (0) 0.9184 84.38 77.78 87.74
| (P y 1| , , ) - 0.5 | x D H uncertainty
| (P y 1| , , ) - 0.5 | x D H uncertainty
Application-
Application- binary cancer classification binary cancer classification based on microarray data
based on microarray data
Variable selection using linear sparse Bayesian logit model, LOO CV accuracy
#Var RVM LS-SVM LR LDA
Leukemia all: 7129 0.9310 0.958 N/A N/A
sel: 4 1 1 1 0.986
Colon all: 2000 0.823 0.871 N/A N/A
sel: 5 0.984 1 1 1
#Var RVM LS-SVM LR LDA
Leukemia all: 7129 0.9310 0.958 N/A N/A
sel: 4 1 1 1 0.986
Colon all: 2000 0.823 0.871 N/A N/A
sel: 5 0.984 1 1 1
cancer no. samples no. genes task
leukemia 72 7192 2 subtypes
colon 62 2000 disease/normal
Application-
Application- brain tumor multiclass brain tumor multiclass
classification based on MRS spectra data classification based on MRS spectra data
4 types of brain tumors, 205x138 magnitude value
Variable selection using linear sparse
Bayesian logit model, 30 runs of random CV accuracy
#Var RVM LS-SVM LR LDAall: 138 69.95% 68.48% N/A N/A
±2.88% ±3.03% N/A N/A
sel:27 74.07% 75.34% 74.61% 75.05%
±2.82% ±3.55% ±3.64% ±3.47%
#Var RVM LS-SVM LR LDA
all: 138 69.95% 68.48% N/A N/A
±2.88% ±3.03% N/A N/A
sel:27 74.07% 75.34% 74.61% 75.05%
±2.82% ±3.55% ±3.64% ±3.47%
Work Plan and Timing Work Plan and Timing
Nov - Overview paper on Linear and nonlinear preoperative classification of ovarian tumors (chapter proposal accepted for edited book "Knowledge Based Intelligent System for Health Care.")
Dec – Jan 2003, paper on variable selection
Feb 2004 – model averaging
Model evaluation using IOTA data.
April 2004 – writing draft of thesis
Sept 2004 - Defense
