X-ray Angiography for
Image-Guided Coronary
Interventions
The research described in this thesis was carried out at the Department of Radiol-ogy & Nuclear Medicine, Erasmus MC, University Medical Center Rotterdam (the Netherlands). This work was supported by the IMAGIC project under the iMIT program of NWO-domein TTW (grant number 12703).
Advanced School for Computing and Imaging
This work was carried out in the ASCI graduate school. ASCI dissertation series number 410.
Financial support by the Dutch Heart Foundation for the publication of this thesis is gratefully acknowledged.
Additional financial support for the publication of this thesis was kindly provided by • the Department of Radiology & Nuclear Medicine, Erasmus MC,
• the ASCI graduate school,
• Medis Medical Imaging Systems bv, • Pie Medical Imaging BV.
Copyright c 2019 by Hua Ma. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechani-cal, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the author.
Cover design by Hua Ma. ISBN: 978-94-6375-752-2
for Image-Guided Coronary Interventions
Analyse van X-ray Angiografische Beeldreeksen voor
Beeldgeleide Interventies aan de Kransslagaders
Thesis
to obtain the degree of Doctor from the Erasmus University Rotterdam
by command of the rector magnificus Prof.dr. R.C.M.E. Engels
and in accordance with the decision of the Doctorate Board. The public defence shall be held on
Wednesday, 12 February 2020 at 11:30 hrs
by
Hua Ma
Promotor: Prof.dr. W.J. Niessen
Other members: Prof.dr.ir. A.F.W. van der Steen Dr.ir. J. Dijkstra
Dr. E.S. Regar Copromotor: Dr.ir. T. van Walsum
Colophon ii
1 Introduction 1
1.1 Coronary Arteries: Anatomy and Functions . . . 1
1.2 Coronary Artery Disease . . . 1
1.3 Treatment of Coronary Artery Disease . . . 2
1.3.1 Percutaneous Coronary Intervention . . . 3
1.3.2 X-ray Angiography for Image Guidance . . . 4
1.4 Challenges . . . 6
1.4.1 Challenges of Image Guidance for PCI . . . 6
1.4.2 Challenges of Image Analysis for X-ray Angiograms . . . 6
1.5 This Thesis . . . 7
1.5.1 Dynamic Analysis of X-ray Angiograms . . . 7
1.5.2 Purpose and Chapter Organization . . . 7
2 Layer Separation for Vessel Enhancement in Interventional X-ray Angiograms Using Morphological Filtering and Robust PCA 9 2.1 Introduction . . . 10
2.2 Method . . . 10
2.2.1 Separation of Breathing Structures . . . 11
2.2.2 Background Separation Using Robust PCA . . . 11
2.2.3 Image Processing Pipeline of XA Layer Separation . . . 12
2.3 Experiments . . . 12
2.4 Results . . . 15
2.5 Discussion and Conclusion . . . 15
3 Automatic Online Layer Separation for Vessel Enhancement in X-ray Angiograms for Percutaneous Coronary Interventions 19 3.1 Introduction . . . 20
3.1.1 Motivation . . . 20
3.1.2 Related Works . . . 21
3.1.3 Overview and Contributions . . . 22
3.2 Method . . . 23
3.2.1 Overview . . . 23
3.2.3 Separation of Vessel Layer via OR-PCA . . . 24
3.2.4 Summary . . . 28
3.3 Experiments . . . 29
3.3.1 Image Data . . . 29
3.3.2 Experiment 1: Parameter Tuning for OR-PCA . . . 31
3.3.3 Experiment 2: Downweighting the Past Data in OR-PCA, In-fluence of the Parameters . . . 34
3.3.4 Experiment 3: Comparison with Other Methods . . . 34
3.3.5 Experiment 4: Vessel Enhancement in Low-Contrast XA . . . . 36
3.3.6 Implementation . . . 36
3.4 Results . . . 36
3.4.1 Optimal Parameters for OR-PCA . . . 36
3.4.2 Influence of the History Parameters . . . 38
3.4.3 Comparison with Other Methods . . . 38
3.4.4 Vessel Enhancement in Low-Contrast XA . . . 42
3.4.5 Computation Time . . . 44
3.5 Discussion and Conclusion . . . 45
4 PCA-derived Respiratory Motion Surrogates From X-ray Angiograms For Percutaneous Coronary Interventions 55 4.1 Introduction . . . 56
4.2 Methods . . . 57
4.2.1 Preprocessing of XA Images . . . 57
4.2.2 Principal Component Analysis . . . 58
4.3 Experiments . . . 59
4.3.1 Image Data . . . 59
4.3.2 Ground Truth Data . . . 59
4.3.3 Retrospective Evaluation . . . 60 4.3.4 Prospective Evaluation . . . 60 4.4 Results . . . 61 4.4.1 Retrospective Analysis . . . 61 4.4.2 Prospective Analysis . . . 64 4.5 Discussion . . . 68 4.6 Conclusion . . . 70
5 Fast Prospective Detection of Contrast Inflow in X-ray Angiograms with Convolutional Neural Network and Recurrent Neural Network 73 5.1 Introduction . . . 74
5.2 Methods . . . 75
5.2.1 The CNN-based Method . . . 75
5.2.2 The RNN-based Method . . . 75
5.3 Experiments . . . 77
5.4 Results and Discussion . . . 78
6 Dynamic Coronary Roadmapping via Catheter Tip Tracking in X-ray Fluoroscopy with Deep Learning Based Bayesian Filtering 81
6.1 Introduction . . . 82
6.1.1 Clinical Background . . . 82
6.1.2 Dynamic Coronary Roadmapping . . . 82
6.1.3 Interventional / Surgical Tool Tracking . . . 83
6.1.4 Contributions . . . 85
6.2 Scenario Setup and Method Overview . . . 85
6.2.1 Offline Phase . . . 87
6.2.2 Online Phase . . . 87
6.3 ECG Matching for Roadmap Selection . . . 87
6.4 Bayesian Filtering for Catheter Tip Tracking . . . 88
6.4.1 Theory of Bayesian Filtering . . . 88
6.4.2 A Deep Learning based Likelihood . . . 89
6.4.3 Approximation of the Posterior with Particle Filter . . . 92
6.4.4 Summary . . . 93
6.5 Experimental Setup . . . 94
6.5.1 Data . . . 94
6.5.2 Data Split for Catheter Tip Detection and Tracking . . . 94
6.5.3 Experimental Settings for Training the Deep Network . . . 96
6.5.4 Setup for Evaluating Dynamic Coronary Roadmapping . . . . 97
6.5.5 Implementation . . . 97
6.6 Experiments and Results . . . 98
6.6.1 Training the Deep Neural Network . . . 98
6.6.2 Catheter Tip Tracking . . . 99
6.6.3 Dynamic Coronary Roadmapping . . . 105
6.6.4 Processing Time . . . 108
6.7 Discussion . . . 110
6.8 Conclusion . . . 116
7 Summary And Future Perspectives 117 7.1 Summary . . . 117 7.2 Future Perspectives . . . 120 Bibliography 123 Samenvatting 131 PhD Portfolio 137 Publications 139 Acknowledgment 141 Curriculum Vitae 143
Introduction
1.1
Coronary Arteries: Anatomy and Functions
The heart is the central organ of the human circulation system. It pumps blood through the blood vessels to provide the human body with oxygen and nutrients, and it carries away metabolic waste. The heart muscles receive their own supply of blood via the coronary arteries. These vessels branch off from the aorta near the point where the aorta and the left ventricle meet [7] (Fig. 1.1a). The coronary arteries wrap around the surface of the heart, with small branches entering into the heart muscle to supply it with blood [2]. Thus, the coronary arteries play a significant role to the heart, and so to speak, to human life.
Coronary arteries have two main branches: the left main and right coronary artery, each of which further divides into smaller branches (Fig. 1.1a).
The left main coronary artery (LM or LCA) supplies blood to the left ventricle and left atrium. It divides into two major branches (Fig. 1.1a): the left anterior descending artery (LAD) and the left circumflex artery (LCX). The LAD and its diagonal branches supply blood to the front and the left side of the heart, mainly the anterior ventricular septum and the major part of the anterior portion of the left ventricle. The LCX encircles the heart muscle and provides blood to the left atrium and the posterior-lateral aspect of the left ventricle that are mainly at the outer side and back of the heart [2, 75].
The right coronary artery (RCA) supplies blood to the right ventricle, the right atrium, and the SA (sinoatrial) and AV (atrioventricular) nodes. The RCA divides into smaller branches (Fig. 1.1a), such as the posterior descending artery (PDA) and the acute marginal artery, providing blood to the inferior part of the heart and the lateral portion of the right ventricle, respectively [75]. The RCA also supplies blood to the septum of the heart, together with the LAD.
A brief schematic view summarizing all major and small branches of the coronary arteries is illustrated in Fig. 1.1b, where the relative positions of coronary artery branches and aorta are shown.
1.2
Coronary Artery Disease
Coronary artery disease, also known as ischemic heart disease or atherosclerosis, is one of the leading causes of death in the world [88]. It involves a reduction of blood supply to the heart muscle due to the build-up of plaque in the coronary arteries. Plaque
(a) (b)
Figure 1.1: The anatomy of coronary arteries: (a) Coronary arteries in the anterior view of the heart [64] (labeled in red text). (b) the branches of the coronary arteries (the figure is adapted from [57]).
Figure 1.2: The formation of atherosclerosis caused by plaque built up inside blood vessels. (the figure is adapted from [86])
mainly consists of fat, cholesterol or calcium. Accumulation of these substances at the plaque narrows the lumen of coronary arteries, reducing the amount of blood flowing to the heart muscle. A rupture of the plaque may also cause the formation of blood clots in the coronary arteries that can lead to partially or complete obstruction of the vessel lumen. The consequence of a narrowed or blocked vessel lumen is that insufficient oxygen-rich blood can reach the heart muscles, which can cause angina or heart attack and may lead to heart failure or arrhythmias in long term [6]. Fig. 1.2 illustrates the formation of atherosclerosis in the vessel lumen.
1.3
Treatment of Coronary Artery Disease
The treatment of coronary artery disease includes medication and medical procedures. Drugs, such as cholesterol-modifying medications, may alleviate the symptoms of coronary artery disease by decreasing the amount of cholesterol in the blood, one of the major component materials of the plaque [5]. Furthermore, a patient suffering from severe cases may undergo medical procedures: a percutaneous coronary intervention or coronary artery bypass grafting.
Percutaneous coronary intervention (PCI), also known as Coronary Angioplasty, is a minimally-invasive procedure to treat narrowing of the coronary arteries. It was
(a)
(b)
(c) Figure 1.3: The PCI procedure: (a) insertion sites of guiding catheter, (b) injection of contrast agent through the guiding catheter, (c) balloon and stent expansion to widen the vessel lumen. (the figure is adapted from [1, 4])
first introduced by Andreas Gr¨unzig in Zurich, Switzerland in 1977 and has become worldwide-adopted since 1980s [73]. During this procedure, the stenosed vessel area is widened by the inflation of a balloon and a stent that are introduced through a long and thin catheter inserted in the blood vessel from a small skin incision.
Coronary artery bypass grafting (CABG) is used for more severe cases that are difficult to treat with PCI, e.g. coronary arteries with multiple stenosed sites [5]. In this procedure, a surgeon creates a graft to bypass the blocked vessel using arteries or veins from other parts of the body. Compared to PCI, CABG is superior for patients with multivessel disease [44], yet it is more invasive, as it requires to open the chest in order to reach the heart.
The target application of this thesis is the PCI procedure. Its minimally-invasive nature puts patients under lower risk, especially for patients in very old and very young age [97].
1.3.1
Percutaneous Coronary Intervention
Fig. 1.3 shows an overview of the PCI procedure. At first, a guiding catheter is in-serted in the blood vessel, e.g. via the groin or arm. The catheter is then manoeuvred through the aorta or brachial artery (depending on the insertion site) towards the os-tium of the coronary arteries. Through the catheter, X-ray opaque contrast agent can be injected to identify the lesions in the vessels, and a guidewire is manipulated by the interventional cardiologist to move to the stenosed site. Once the guidewire reaches the site of the lesion, a balloon catheter carrying a stent, a collapsed wire mesh tube,
(a) Monoplane (b) Biplane
Figure 1.4: Typical X-ray imaging equipment in a catheterization laboratory in a hospital with a Siemens Artis zee ceiling monoplane system (a) and biplane system (b). (images copyright: Siemens Healthineers AG [3])
is advanced over the guidewire towards the lesion site. The balloon is then inflated to expand the vessel lumen as well as the stent. Finally, the stent is deployed at the lesion to prevent the vessel from collapsing, and the balloon is retrieved.
1.3.2
X-ray Angiography for Image Guidance
The PCI procedure, as suggested by its name, is performed percutaneously, which means that the interventional cardiologist does not directly see the coronary arteries, vessel lesions and interventional tools in the patient’s body during the procedure. In a catheterization laboratory (the operation room where PCI is performed), X-ray angiography (XA) is the imaging technique that is commonly used during PCI for visual guidance of the procedure. The complete set-up is normally an integrated system that typically contains one or two C-arms, a patient table and monitors (Fig. 1.4). The C-arm is a C-shape equipment connecting an X-ray source with an X-ray detector that is mainly used for fluoroscopic imaging, and can be rotated during the procedure to acquire X-ray images from different angles. The acquired images are shown on the monitors together with other relevant patient information (e.g. ECG) to provide real-time visual feedback to the interventional cardiologist during PCI.
Fig. 1.5 illustrates two examples of X-ray angiographic images of left and right coronary arteries (Fig. 1.5c and 1.5d). The vessels are only visible with the use of contrast agent, showing as ”tree” structures in the images. When no contrast agent is injected, the vessels are not visible in the X-ray fluoroscopic images, while the guidewire has good visibility (Fig. 1.5a and 1.5b). The real X-ray angiographic images used for PCI are dynamic cine-angiograms with a frame rate ranging between 4 and 30 frames per second (fps). In typical angiograms, moving structures, such as diaphragm and vessels, and transition between contrast and non-contrast phase can be seen.
arter-Guidewire Guiding
catheter
(a) Without contrast agent
Guidewire Guiding catheter
(b) Without contrast agent
Guidewire Guiding
catheter
Vessel
Vessel
(c) Left coronary arteries
Guidewire Guiding catheter
Vessel Vessel
(d) Right coronary arteries
Figure 1.5: Examples of X-ray images for PCI. In (a) and (b), contrast agent has not been injected, guiding catheter (the long, dark, thin tube) and guidewire (the thin wire) can be seen in the images. In (c) and (d), contrast agent is used for visualizing the left (c) and the right (d) coronary arteries.
ies, and to find and assess the lesions on the vessels. Since guidewires and stents have better visibility in fluoroscopic images, cardiologists typically manipulate the instruments without continuously injecting contrast agent. The manipulation of in-struments therefore mainly relies on the operator’s mental map of the vessels and plaques’ location from the previous XA images with contrast agent.
1.4
Challenges
1.4.1
Challenges of Image Guidance for PCI
One of the challenges in PCI is that limited visual feedback is provided to the in-terventional cardiologist when manoeuvring guidewires towards the lesion site. As the navigation of instruments is guided with “vessel-free” fluoroscopic images, the operator needs to mentally reconstruct the vessels from the previous angiographic acquisitions, which relies largely on the skills and experiences of the operator, as it is, in general, a challenging task to accurately imagine the position of moving structures. To be more certain about the target location, cardiologists sometimes repeat-edly inject contrast agent to get a better mental image of the vessels and the le-sions, as the opacification of the vessels lasts only for a short period before contrast agent drains from the vessel lumen. However, contrast-induced side effects limit the amount of contrast agent that can be used on patients during PCI. Allergic reaction and nephrotoxicity have been reported as side effects that X-ray contrast agent may have [10, 105]. Especially the latter one, also known as contrast-induced nephropathy (CIN), which may result in chronical renal failure with all its relevant sequelae [105], has been associated to contrast volume [104].
Another challenge in performing PCI comes from the side effect of exposure to X-ray. The ionizing radiation of X-ray has detrimental effect on the exposed human tissues, including tissue reactions and increased risk for stochastic events, such as skin necrosis and radiation-induced cancer [58]. To spare the radiation dose to patients and operators, strategies, such as minimizing the fluoroscopy acquisition time, modulating the fluoroscopy dose per frame, decreasing the radiation detector magnification and reducing the frame rate, should be applied [40, 58]. As some of these strategies may hamper the quality of the acquired X-ray images, operators face a trade-off between a lower X-ray dose and abilities to resolve small vessels or motion details.
The above listed challenges of image guidance for PCI procedure could be ad-dressed with an improved system that can help interventional cardiologists to view relevant information that is needed for PCI, and that meanwhile controls the poten-tial risks the operators and the patients may have from the procedure. This thesis describes the works done during my PhD in order to address the challenges from the perspective of automatic image analysis approaches.
1.4.2
Challenges of Image Analysis for X-ray Angiograms
In addition to the challenges from the image guidance point of view, the physical nature of X-ray angiographic images also poses challenges for developing automatic and robust algorithms for X-ray image analysis. The major challenges are four-fold:
• X-ray images are the results of projection of 3D structures on a 2D plane, there-fore, the 3D information is lost during the image formation. The information of anatomical structures overlaps each other in X-ray images, making automatic analysis of those structures challenging.
• Different from X-ray projection radiography which is normally acquired in the anterior-posterior direction, X-ray angiography can be taken from arbitrary viewing directions, depending on the C-arm configuration. This means that X-ray images acquired from different view angles based on the same 3D struc-tures look different. Therefore, the image analysis approaches for XA need to be robust for images acquired with different C-arm angles.
• An X-ray angiogram is a cinematic clip instead of a static image, the structures of interest in the image sequence normally moves instead of staying still, such as diaphragm and coronary arteries. As the motion may be caused by the patient’s respiration or heartbeat, the moving structures may present respiratory or cardiac motion patterns. These two types of motions normally need to be taken into consideration for image guidance applications.
• The level of contrast agent in an X-ray angiogram does not stay constant. After the contrast injection, the amount of contrast in the field of view increases rapidly till its maximum, the vessels becomes fully opacified at this phase. After a short period, the contrast agent drains gradually from the target vessels along with the blood flow, the vessels becomes invisible again. These changes between the contrast and the non-contrast phase in an X-ray sequence require additional designs for some image guidance applications.
1.5
This Thesis
1.5.1
Dynamic Analysis of X-ray Angiograms
Despite of the previously mentioned challenges, opportunities may also come from the particular properties of X-ray angiographic image data. Different from many other medical imaging modalities which are static snapshots of anatomical structures, the cinematic nature of X-ray angiograms possesses a time dimension in the data, bringing the possibility of using the temporal information in the image analysis. The temporal information contains the changes of the same tissues at different time points, which may serve as a cue to link different frames instead of treating each image independently. For some problems, the temporal information is the key to overcome the obstacles.
In this thesis, I use the term dynamic analysis to call the image analysis ap-proaches that take advantage of the temporal, motion or inter-frame information in X-ray angiograms. Different dynamic analysis approaches will be introduced in this book to address the challenges in various aspects of the image guidance for PCI.
1.5.2
Purpose and Chapter Organization
The purpose of the research presented in this thesis is to develop and evaluate dynamic image analysis techniques that may improve image guidance for percutaneous coro-nary interventions. To this end, this thesis investigates on the following approaches:
1. Layer separation, a computational angiography approach Layer sepa-ration is a computational technique to improve vessel visibility in XA images by removing static and moving background structures. In Chapter 2, a layer separation approach using robust PCA was proposed to separate an XA angio-graphic sequence into three additive layers: a vessel layer, a breathing layer and a quasi-static background layer. In Chapter 3, we developed and evaluated an online layer separation method which dynamically separates streaming XA data into the three layers and have shown its potential on reducing the amount of contrast agent used for PCI.
2. Layer separation as a component for XA analysis Layer separation en-ables independent analysis of the layers it outputs, and can serve as a component in the image processing pipeline for various applications. Two example are pre-sented in this thesis. In Chapter 4, a PCA-based approach was proposed to extract respiratory motion surrogate from the breathing layer that has high correlation with the respiration movement. In Chapter 5, we adopted the vessel layer for automatic detection of contrast inflow in an XA sequence. An approach using a recurrent neural network (RNN) was proposed to classify whether an X-ray image is with contrast or not using features extracted from the enhanced vessel layer. Additionally, we also proposed a second method based on a convo-lutional neural network (CNN) to perform the frame classification.
3. Dynamic roadmapping, an augmented fluoroscopy approach Interven-tional tools are typically navigated in fluoroscopy mode with non-contrast-enhanced X-ray images, which forces the operator to rely on a mental recon-struction of anatomical structures. In Chapter 6, we developed and evaluated a novel dynamic coronary roadmapping approach to tackle the challenge. The fluoroscopic images are augmented with a dynamic motion-compensated vessel layer to provide real-time visual guidance during PCI, while in the meantime, reducing the use of contrast agent.
In Chapter 7, the thesis concludes with a summary on the methods and result, and a discussion on future research directions.
Layer Separation for Vessel
Enhancement in Interventional
X-ray Angiograms Using
Morphological Filtering and Robust
PCA
Abstract — Automatic vessel extraction from X-ray angiograms (XA) for per-cutaneous coronary interventions is often hampered by low contrast and presence of background structures, e.g. diaphragm, guiding catheters, stitches. In this pa-per, we present a novel layer separation technique for vessel enhancement in XA to address this problem. The method uses morphological filtering and Robust PCA to separate interventional XA images into three layers, i.e. a large-scale breathing structure layer, a quasi-static background layer and a layer containing the vessel structures that could potentially improve the quality of vessel extraction from XA. The method is evaluated on several clinical XA sequences. The result shows that the proposed method significantly increases the visibility of vessels in XA and out-performs other background-removal methods.
Based upon: H. Ma, G. Dibildox, J. Banerjee, W.J. Niessen, C. Schultz, E. Regar and T. van Wal-sum: Layer Separation for Vessel Enhancement in Interventional X-ray Angiograms Using Morpho-logical Filtering and Robust PCA. Workshop on Augmented Environments for Computer-Assisted Interventions (AE-CAI 2015), Lecture Notes in Computer Science, vol. 9365, pp. 104-113, 2015.
2.1
Introduction
Percutaneous coronary intervention (PCI) is a minimally invasive procedure for treat-ing patients with advanced coronary artery disease. It is usually performed under guidance of X-ray angiograms (XA) where coronary arteries are opacified with con-trast agent. Automatic processing of XA images, e.g. vessel extraction of coronary arteries, may serve as a basis for further processing, such as coronary motion analy-sis [78] and pre/intra-operative information fusion [14].
Hessian-based vessel enhancement filtering, e.g. Frangi vesselness filter [38], is commonly used for extraction of vessels in medical images. Applying such filters directly on interventional XA, however, often also enhances non-vascular structures, such as catheter segments and vertebral contours, due to their tubular or curvilinear structural appearances.
Related works have reported on methods to remove non-vessel structures or im-prove the visibility of vessels in XA images. In [15], a method that subtracts the median frame was used for removing static structures in XA, such as vertebral bod-ies. Schneider et al. [93] proposed a post-processing technique on vesselness images that combines a local probability map with local directional vessel information for artifact reduction and catheter removal. Layer separation methods provide an alter-native way of vessel enhancement. In [116], a multi-scale framework was developed to separate XA images into three layers based on different motion patterns such that coronary arteries are better visible in the fast motion layer. This method involves human-interactions to label corresponding control points in XA images for motion field estimation. In another study [118], a Bayesian framework was developed that combines dense motion estimation, uncertainty propagation and statistical fusion to achieve motion layer separation. Both layer separation methods require to compute motion field. Robust principal component analysis (Robust PCA) is a data decompo-sition technique that has e.g. been used for background modeling from surveillance video in [23]. In [43], Robust PCA was adopted for registration of DCE MR time series.
In this paper, we propose an automatic method to robustly separate foreground (contrast-enhanced vessels, guiding catheter tip) from (quasi) static background, such as vertebral bodies and guiding catheters in the aorta, while ignoring large-scale mo-tion such as diaphragm movement. Our contribumo-tions are three folds: 1) the develop-ment of a Robust PCA based layer separation method that does not require compu-tation of the motion field; 2) qualitative and quantitative evaluations on four clinical XA sequences; 3) comparison to other related background-removal approaches.
2.2
Method
The method enhances vessels in XA images by separating an image into three layers, i.e. a large-scale breathing layer, a quasi-static background layer and a foreground layer containing the vessels. To this end, our proposed method consists of two steps: first, separation and removal of large-scale breathing structures, such as diaphragm, from the original images, using morphological closing; second, separation of a
quasi-static background from the moving structures using Robust PCA. In the remainder of this section, we describe both steps in more details, followed by the integrated layer separation.
2.2.1
Separation of Breathing Structures
To obtain a separate layer containing large-scale structures, we remove small objects from the original image, including guiding catheters, guide wires, stitches and ver-tebral bodies. Similar to the approach in [66] (Chapter 4 of this thesis), we apply morphological closing to the image with a circular structuring element of 8.5 mm in diameter. Pilot experiments indicated that this size was adequate for a complete removal of vessels and guiding catheters from our images while not causing too much circular artifacts. An example of a resulting image is shown in Fig. 2.1b. Compared to the original image, the guiding catheter and coronary arteries are removed and ver-tebral contours are blurred, while structures that presents respiratory motion, such as the diaphragm and lung tissue, remain in the image (white area in the upper left part of the image). The resulting image that contains large scale structures which exhibit respiratory motion is called the breathing layer, and will later be subtracted from the original image to obtain the difference image (DI, Fig. 2.1c) of an XA frame for further processing.
(a) (b) (c)
Figure 2.1: Morphological closing operation on an XA image: (a) the original image, (b) image processed with morphological closing, (c) the difference image (DI) of (a) and (b).
2.2.2
Background Separation Using Robust PCA
Robust PCA decomposes a data matrix into two different sources: a low-rank ma-trix and a sparse mama-trix. Suppose that M is an m × n mama-trix to be decomposed, which contains n observations of m dimensional data in its columns. Robust PCA is formulated as the following optimization problem [23]:
minimize kLk∗+ λkSk1
where L is a low-rank matrix and S is a sparse matrix of the same size as M . kLk∗
denotes the nuclear norm of L and kSk1 is the L1 norm of S. λ is the tuning
pa-rameter of regularization. Source decomposition is achieved by solving this optimiza-tion problem. In this work, we use inexact Augmented Lagrange Multiplier (ALM) method [63] to solve the problem. Robust PCA can be applied for separation of the background layer of DI from the vessel layer. The background of an XA sequence is an image series with small changes of pixel intensity containing (quasi) static struc-tures, while the foreground, or the vessel layer, consists of moving objects. Thus, resizing the background image into a column vector and combining all these vectors from a background series together results in a low rank matrix. Likewise, the image series of vessel layer can be modeled as a sparse matrix, as either vessels or guiding catheters take up only a small part of the whole image content. Therefore, the back-ground layer and vessel layer of DIs can be separated by solving the Robust PCA problem.
2.2.3
Image Processing Pipeline of XA Layer Separation
The proposed layer separation algorithm consists of the following steps. All steps are illustrated in Fig. 2.2.
1. Given an XA sequence, apply morphological closing on each frame of the series, as described in Section 2.2.1. For each frame, subtract the morphological-closed image from the original image to obtain the DI.
2. Rearrange the DIs of the XA sequence to construct a matrix whose columns represent the frames. This matrix is considered as the input matrix M in Equation 2.1.
3. Solve the Robust PCA problem to obtain the background layer matrix L and vessel layer matrix S. Resize L and S to get the background layer and vessel layer of the previous size for each frame of the sequence.
2.3
Experiments
Fully anonymized imaging data were used in our experiments. Four XA image series that were acquired with Siemens AXIOM-Artis biplane system were analyzed. The frame rate of all sequences is 15 frames per second. The number of frames per series ranges from 55 to 169. From our data, the image matrix is 512 × 512 pixels for one of the series and 600 × 600 for the other three, with resolution 0.216 × 0.216 and 0.184 × 0.184 mm2, respectively.
To quantify the visibility of vessels in an image, the contrast-to-noise ratio (CNR) is used in the experiments. CNR is a measure of image quality based on contrast. Once the background and foreground of an image is defined, the definition of CNR can be formulated as:
CN R =|µF− µB| σB
Figure 2.2: The pipeline of the proposed layer separation method.
where µFand µBare the mean of foreground and background pixel values respectively,
and σB is the standard deviation of the background pixel values. This definition of
CNR measures the contrast between the foreground and background pixel intensities in relation to the standard deviation of the background pixel intensities. Larger CNR values imply a better contrast.
Two different versions of CNR are computed, using two different masks for defin-ing the foreground (vessel) and the background in XA images (Fig. 2.3). In mask 1, as shown in Fig. 2.3 column 1, a 4 mm-wide image area around the manually-labeled vessel centerline is defined as the foreground (the dark area inside white region); the background are its 3 mm-wide neighborhood area (white region surrounding the ves-sel). This mask can be used to assess the local contrast around vessels in XA. In mask 2, as shown in Fig. 2.3 column 3, everything outside the foreground is consid-ered background, which thus also evaluates the removal of the diaphragm, guiding catheters, etc.. In our experiments, we randomly select 5 frames once from each se-quence for the mask generation and compute the average CNR of the 5 frames.
We compare the performance of our approach to 3 other related methods. In [15], static background is eliminated by subtracting the median of the first 10 frames from each frame in the sequence. This method is referred to as MedSubtract 1. Second, we considered an advanced version of median subtraction by firstly removing the breathing layer using morphological closing and then subtracting the median. This is called MedSubtract 2 in the experiments. Third, a conventional PCA technique is explored. The breathing layer is first removed to generate the difference image and the background layer is later reconstructed with the first principal component using PCA. This is referred to as Normal PCA.
For the parameter λ in the formulation of Robust PCA, we use the value sug-gested in [23]. All experiments were implemented in MATLAB 2013b on an Intel Core i7-4800MQ 2.70 GHz computer with 16 GB RAM running Windows.
Figure 2.3: Two types of mask images. Background is defined as the white image region, foreground is defined as the dark area within the white part: (Column 1) Mask 1 for one frame in the four XA sequences; (Column 2) Mask 1 overlaid on the corresponding XA frames; (Column 3) Mask 2 for one frame in the four XA sequences; (Column 4) Mask 2 overlaid on the corresponding XA frames.
2.4
Results
Fig. 2.4 shows an example result of layer separation on one XA sequence. Note that in the original image (Fig. 2.4a), the presence of the diaphragm, the vertebral structures and the long guiding catheter segment makes extracting the vessels challenging. In the vessel layer image (Fig. 2.4d), those structures are removed, and the contrast between vessels and their neighborhood pixels is larger than in the original image.
Fig. 2.5 presents the comparison of our proposed method (Row 5) to three other
(a) (b) (c) (d)
Figure 2.4: An example of layer separation: (a) the original image, (b) breathing layer, (c) quasi-static background layer, and (d) vessel layer.
background-removal methods (Row 2-4) applied on four XA sequences. For each of the sequences, we selected a representative frame. It can be observed that all the four methods increase the visibility of vessels in XA with better contrast. However, the result of MedSubtract 1 method (Row 2) still presents artifacts in the foreground due to the motion of diaphragm, whereas our method successfully removes the diaphragm using morphological closing. Compared to MedSubtract 2 (Row 3) and Normal PCA methods (Row 4), the method based on Robust PCA performs better on removing quasi-static structures, such as the guiding catheter segment in aorta (column 1-3) and stitches (column 4).
The CNR values of XA sequences and vessel layers are illustrated in Fig. 2.6. Compared to the original XA, as shown in both Fig. 2.6a and Fig. 2.6b, all methods improve the CNR values. For CNR 1, when only local contrast around vessels is measured, Robust PCA method performs better than the other approaches for patient 1 and 2, but has slightly lower CNR than Normal PCA for patient 3 and 4. In the case that the removal of diaphragm and guiding catheter is considered, as what CNR 2 indicates, Robust PCA is superior in all four patients.
2.5
Discussion and Conclusion
We have developed an automatic method for layer separation of interventional XA images, to enhance vessel visualization. The method separate XA images into a breathing layer, a quasi-static background layer and a vessel layer using morpho-logical filtering and applying Robust PCA. The separation is evaluated on four XA
Figure 2.5: Example frames of foreground images obtained by different background-removal techniques applied on four XA sequences: (Column 1-4) The four different XA sequences, (Row 1) The original image, (Row 2) MedSubtract 1, (Row 3) MedSubtract 2, (Row 4) Normal PCA, (Row 5) our method using Robust PCA.
Patient ID 1 2 3 4 CNR 1 0 0.5 1 1.5 2 2.5 3 Original MedSubtract 1 MedSubtract 2 Normal PCA Robust PCA (a) CNR 1 Patient ID 1 2 3 4 CNR 2 0 1 2 3 4 5 6 Original MedSubtract 1 MedSubtract 2 Normal PCA Robust PCA (b) CNR 2
Figure 2.6: The average CNR over 5 randomly-chosen frames using two types of masks for the four XA sequences.
sequences, demonstrating better separation of the coronary arteries and reduced in-clusion of breathing or quasi-static structures compared to other approaches.
Fig. 2.5 shows that the proposed method is able to improve the visibility of vessels and performs better on representative frames of the four XA sequences. Fig. 2.6a shows that the Robust PCA method is advantageous over the two median subtraction methods on improving the local contrast, and has similar performance with Normal PCA. Fig. 2.6b, which displays the global CNR measure, shows that Robust PCA is superior on all four patients which indicates that the superiority of Robust PCA to other approaches is more on removing respiratory and quasi-static structures from XA to improve the contrast of vessels in the whole image. This advantage could potentially reduce the generation of spurious vessels when applying vessel extraction methods on XA.
Compared to original images, the Robust PCA method improves image quality in the vessel layer by removing breathing structures and background objects. Com-pared to the absolute-static background resulted from the median-subtraction-based methods, Robust PCA models a quasi-static background with small changes, which is more adaptive to the change of image content caused by coronary motion. Normal PCA also models a flexible background , which could be the reason why it has similar performance with Robust PCA. Compared to Normal PCA, Robust PCA produces less residuals of guiding catheter in the vessel layer after the removal of the back-ground layer. The regularization parameter of Robust PCA enables better flexibility of balancing between moving objects and background in layer separation. Compared to other related techniques e.g. in [116] [118], the main difference of the proposed method is that it does not rely on motion field, therefore, no motion field is required to extract before doing layer separation.
Several factors might have impact on CNR values. The masks defines the back-ground and foreback-ground, therefore the mask-related factors could directly influence the CNR values, e.g. the width of the foreground or background, whether or not including small vessels or the guiding catheter distal segment in the foreground. In addition,
the number of the selected frames for mask generation from each XA sequence might also be an important factor. More in-depth analysis of these factors is part of the future work.
In conclusion, we proposed a novel layer separation method based on morpho-logical operation and Robust PCA. We also demonstrated that the method improves the visibility of coronary arteries in XA and has advantages over several other related approaches. In the future, we will assess this technique in prospective settings and study its application in approaches that improve image guidance in XA guided car-diac interventions.
Acknowledgement This work was supported by Technology Foundation STW, IMA-GIC project under the iMIT program (grant number 12703).
Automatic Online Layer Separation
for Vessel Enhancement in X-ray
Angiograms for Percutaneous
Coronary Interventions
Abstract — Percutaneous coronary intervention is a minimally invasive procedure that is usually performed under image guidance using X-ray angiograms in which coronary arteries are opacified with contrast agent. In X-ray images, 3D objects are projected on a 2D plane, generating semi-transparent layers that overlap each other. The overlapping of structures makes robust automatic information processing of the X-ray images, such as vessel extraction which is highly relevant to support smart image guidance, challenging. In this paper, we propose an automatic online layer separation approach that robustly separates interventional X-ray angiograms into three layers: a breathing layer, a quasi-static layer and a vessel layer that contains information of coronary arteries and medical instruments. The method uses morphological closing and an online robust PCA algorithm to separate the three layers. The proposed layer separation method ran fast and was demonstrated to significantly improve the vessel visibility in clinical X-ray images and showed better performance than other related online or prospective approaches. The potential of the proposed approach was demonstrated by enhancing contrast of vessels in X-ray images with low vessel contrast, which would facilitate the use of reduced amount of contrast agent to prevent contrast-induced side effects.
Based upon: H. Ma, A. Hoogendoorn, E. Regar, W.J. Niessen and T. van Walsum: Automatic Online Layer Separation for Vessel Enhancement in X-ray Angiograms for Percutaneous Coronary Interventions. Medical Image Analysis, vol. 39, pp. 145-161, 2017.
3.1
Introduction
3.1.1
Motivation
Percutaneous coronary intervention (PCI) is a minimally invasive procedure for pa-tients with advanced coronary artery disease. In this procedure, a stent pre-mounted on a delivery catheter is advanced over a guide-wire and through a guiding catheter at the site of narrowing in a patient’s coronary arteries. Once the lesion site is reached, the delivery balloon is inflated and the stent is deployed against the coronary wall, as-suring optimal patency of the artery. As there is no direct eyesight on the target area, these procedures are commonly performed under image guidance using X-ray angiog-raphy (XA), where coronary arteries are visualized with X-ray contrast agent. During the intervention, clinicians use XA images to navigate catheters and guidewires inside the patients.
As XA images contain useful information on anatomy and instrument position, many works have been published on extracting relevant information to improve the image guidance for cardiac interventions. For example, Panayiotou et al. [78] have developed a retrospective motion gating technique of interventional X-ray images through vessel extraction. Also using the information of vessels, pre/intra-operative information fusion between CT angiography and XA have been reported [14, 89]. Apart from vessels in XA, there is interest to track structures such as the lungs, catheters and guidewires. Shechter et al. [96] have used the position of diaphragm as an indicator of respiratory phase and constructed a patient specific coronary motion model based on that. In [13], the position of guiding catheter tip has been related to the combination of respiratory and cardiac motion.
Since X-ray images are projections of 3D structures on a 2D plane, the image content can be interpreted as a composition of several opaque or semi-transparent structures, which have different appearances and motion patterns. The overlapping nature of the structures makes automatic analysis of XA challenging. Separating the structures from each other enables visualizing and analyzing different structures in-dependently, which would, therefore, potentially facilitate the information processing of XA. For example, vessel extraction using Hessian-based filtering method in XA is often hampered by non-vascular structures, such as guiding catheters, diaphragm border and vertebral body edges, because of their tubular or curvi-linear appear-ance in XA. Separating non-vascular structures would improve the visibility of vessels and promote automatic vessel extraction that would ultimately facilitate the image guidance during interventions.
In the context of this work, we interpret the process of separating those structures in XA images as a separation of a set of additive 2D layer images which add up to the original image, and each of them has different structures. The purpose of this work is to develop and evaluate a fast method that can run prospectively for the effective and efficient separation of the structures on different layers for XA sequences. Following the terminology from earlier works (in Section 3.1.2), we adopt the term “layer separation” to refer to the separation of structures and putting them in different layers.
3.1.2
Related Works
Existing methods for layer separation for X-ray fluoroscopic sequences can be cate-gorized into two approaches: motion-based and motion-free .
Motion-based layer separation methods treat each frame of an X-ray fluoroscopic sequence as the outcome of the motion of each layer. Hence, the key part of ob-taining the layers in these methods is estimating the motion of every layer. Various assumptions on the type of motion have been proposed. For instance, Close et al. [26] have estimated translation, rotation and scaling for each layer in a region of inter-est. The layers are computed by transforming each frame with the estimated motion and averaging the transformed frames. This method computed a total of four layers for a sequence. Zhu et al. [118] have proposed a two-layer separation scheme. They have developed a Bayesian framework that combines dense motion estimation, uncer-tainty propagation and statistical fusion to achieve layer separation. In a three-layer separation approach proposed in [116], a multi-scale framework has been developed based on different motion patterns for the static background, lung and vessels. In this work, a dense motion field of each layer has been constructed using thin plate splines. Fischer et al. [37] have further extended this method by introducing a regularization term for layers with a Bayesian model to aid layer separation. In particular, they have proposed to use a robust data term and edge-preserving regularization. In [12], a joint layer segmentation and parametric motion estimation scheme has been pro-posed for transparent image sequences. Similarly, Preston et al. [85] jointly estimated layers and their corresponding smooth deformation to model the non-smooth motion observed in a fluoroscopic sequence. A total variation based regularization was used to encourage sparsity of gradients within and across the layer images.
Unlike motion-based methods, motion-free approaches do not require estimating the motion of layers. Instead, they directly model the background layer or/and fore-ground (vessel) layer of an image sequence under certain hypotheses. One of the simplest ways of modeling the background of XA is computing the median of several frames in a sequence, and obtaining the foreground by subtracting the median image from the original frames [15]. This method worked well for the background that is entirely static, but generates artefacts when there are moving objects in the back-ground, e.g. diaphragm in XA. A more advanced method has been proposed in [103] in which they assumed that the vessel and the backgrounds generate independent sig-nals that are mixed in a sequence, so that the vessel-background separation becomes a blind source separation problem that is commonly solved by independent component analysis (ICA) [52].
Apart from ICA, robust principal component analysis (RPCA) is also a common approach for source decomposition. One of the most popular RPCA methods, prin-cipal component pursuit (PCP) [23], splits a data matrix into a low-rank component and a sparse component. It has been used for background modeling or foreground detection for surveillance videos [20]. In the field of medical image analysis, it found applications in reconstruction [76] and motion correction [43] in dynamic MRI. On the topic of layer separation for X-ray images, Ma et al. [65] (Chapter 2 of this the-sis) have used morphological closing to remove breathing structures from the images and adopted RPCA to separate a quasi-static layer and a vessel layer from XA. This
method could only be used in a retrospective setting, since it requires all frames of a sequence. Volpi et al. [109] have developed a method that worked in a prospective setting. The method used vesselness filtering [38] and RPCA to separate a foreground that contains interventional devices. They have implemented the foreground separa-tion by solving RPCA with a mini-batch of data: for each new coming mini-batch, the average of the low-rank component was estimated and used as the background for the next mini-batch. The limitation of this method is that the foreground separation of a mini-batch is delayed by the processing of the previous complete block of data.
Online robust PCA (OR-PCA) is an online extension of the original RPCA method, proposed in [35]. OR-PCA overcomes the limitation of RPCA-based methods by reformulating the nuclear norm in the RPCA formulation as an explicit low-rank factorization, so that it does not require to “see” the complete dataset or a mini-batch of data, but can process each single data sample one at a time. This setting enables online processing of streaming data. In [98], a closed-form solution for the subspace basis update in OR-PCA has been proposed and shown to achieve better performance in image alignment tasks. OR-PCA has been used in computer vision tasks, such as background subtraction [54] and foreground detection [53], but its application in the field of medical imaging has not been investigated yet.
3.1.3
Overview and Contributions
In this work we extended the method in [65] (Chapter 2 of this thesis) that only worked in a retrospective or “off-line” setting. To this end, we developed and evalu-ated an automatic motion-free online layer separation method for X-ray angiograms. The method robustly separates the layer that contains vessels and catheter tip from a (quasi) static background, while ignoring large-scale motion such as diaphragm movement. Our contributions are:
• We integrated OR-PCA in the layer separation scheme, enabling online layer separation for XA, which is a key ingredient for its potential application in a clinical workflow.
• Inspired by the work in [70], we proposed and analyzed three ways to downweight past information that is able to improve the layer separation performance using the original OR-PCA algorithm.
• We compared the proposed method with other related background-removal ap-proaches and evaluated the results visually and quantitatively on real patient XA data.
• We investigated the potential of improving the contrast of vessels in a low-contrast scenario using the proposed method with synthetic low-low-contrast XA sequences and real sequences acquired in a pig experiment in which various contrast levels were used.
Figure 3.1: The overview of online layer separation for an XA frame.
3.2
Method
3.2.1
Overview
The proposed method treats the intensity of an XA frame as the sum of three lay-ers, i.e., a “breathing” layer, a quasi-static layer and a layer that contains vessels. The method consists of two main steps: first, large-scale breathing structures, e.g. diaphragm, are separated and removed from the original XA frame, and second, smaller moving structures, e.g. vessels and guiding catheters, are separated from a quasi-static background using online robust PCA (OR-PCA). Fig. 3.1 provides an overview of the complete method, details are described in the remainder of Section 2.
3.2.2
Separation of Breathing Structures
To prevent artefacts remaining in the vessel layer due to respiratory motion, the layer that contains large-scale breathing structures, such as diaphragm, is removed from the original XA images in the first step.
The layer of breathing structures was obtained by removing “small” objects from the original X-ray angiographic frame. Depending on the field of view, those objects could include vessels, guiding catheters, guide wires, stitches and vertebral bodies. Following the approach in [65] (Chapter 2 of this thesis), as a preprocessing step, we applied a morphological closing operation to the XA image with a circular structur-ing element of 8.5 mm in diameter, in order to remove any tubular and curvilinear structures smaller than that size. An example of a resulting image is shown in Figure 3.2b, where the guiding catheter and vessels are removed and vertebral contours are blurred, while structures that are susceptible to breathing motion remain in the im-age (diaphragm and lung tissue are shown as the white area in the upper left part of the image). The resulting image is referred to as the “breathing layer” in this paper and was next subtracted from the original image to obtain the difference image (DI, Figure 3.2c) of the XA frame for further processing.
(a) (b) (c)
Figure 3.2: Morphological closing operation applied on an XA frame: (a) original frame, (b) image processed with morphological closing, (c) difference image (DI) (a-b).
3.2.3
Separation of Vessel Layer via OR-PCA
In this section, we briefly review the formulation of the online robust PCA method proposed in [35] and different subspace basis update strategies for solving the OR-PCA problem [35, 98]. Then we propose three different ways of coping with previous frames to improve on these methods.
3.2.3.1 Notation
Bold letters are used to denote vectors. With the difference image (DI) of an XA frame represented with a k × k matrix, we concatenated all pixels in this matrix to form a single column vector z ∈ Rp, where p = k2 is the dimension of the observed
sample. Likewise, we use x ∈ Rp to denote the quasi-static background of the XA
frame and e ∈ Rp represents the foreground. Hence, z = x + e. Let n denote the
number of frames in a sequence, t be the index of the sample/time instance of a frame and r denote the intrinsic dimension of the subspace underlying {xi| i = 1, 2, . . . n}.
Matrices are denoted by capital letters in the following sections. In particular, Z ∈ Rp×n is the matrix of a complete sequence of difference images (DIs), where
its column zi represents the i -th DI. Likewise, X and E are the background and
the foreground matrices with xi and ei the vector for the i -th background and the
i -th foreground. For an arbitrary real matrix M , let ||M ||1 = Pi,j |Mi,j| denote
the L1-norm of M , ||M ||F denotes the Frobenius norm ||M ||F =
qP
i,j |Mi,j|2, and
||M ||∗ = Pi σi(M ) denotes the nuclear norm, i.e., the sum of its singular values.
T r(M ) denotes the trace of a matrix. 3.2.3.2 Online Robust PCA
Robust PCA (RPCA) aims at estimating the subspace underlying the observed sam-ples. Among many popular RPCA methods, Principal Component Pursuit (PCP) [23] has been proposed to solve the RPCA problem by approximating the data matrix as
the sum of a low-rank matrix and a sparse matrix. The concepts of low-rank and sparsity have been implemented using the nuclear norm and the L1-norm of matrix
respectively. This formulation is suitable for the separation of the vessel layer from the DI of an XA frame, since the background has merely minor changes, which can be modeled as a low-rank matrix. In addition, the fact that vessels and guiding catheters take up only a small portion of the complete image content fits the requirement of sparsity.
3.2.3.2.1 The OR-PCA formulation
Different from the classical formulation in [23], PCP can be reformulated as Equa-tion (3.1) [35]: min X,E 1 2||Z − X − E|| 2 F+ λ1||X||∗+ λ2||E||1 (3.1)
where λ1and λ2are regularization coefficients. Through minimizing the cost function
(3.1) that contains the nuclear norm of the background X and the L1-norm (sparsity)
of the foreground E, the RPCA algorithm aims at obtaining the background (X) and foreground (E) that best approximate the XA sequence (Z). Because the nuclear norm couples all samples tightly, typical methods to solve Equation (3.1), such as Augmented Lagrangian Multiplier (ALM) [63], are often implemented in a batch manner, which limits its application in scenarios that deal with streaming data, e.g. X-ray cine angiography data during coronary interventions.
To overcome this problem, Feng et al. [35] have proposed to use an equivalent form of the nuclear norm:
||X||∗= inf L,R 1 2||L|| 2 F+ 1 2||R|| 2 F : X = LRT (3.2) where inf denotes the greatest lower bound of a subset of a partially ordered set, L ∈ Rp×r
is the basis of the low-dimensional subspace and R ∈ Rn×r can be seen as
the samples’ coefficient with respect to the basis. Substituting Equation (3.2) into (3.1), the RPCA problem can be reformulated as (3.3):
min L,R,E 1 2||Z − LR T − E||2F + λ1 2 (||L|| 2 F+ ||R|| 2 F) + λ2||E||1 (3.3)
Following [35], solving Equation (3.3) is equivalent to minimizing the following empirical cost function given a sequence Z consisting of n samples [z1. . . zn]:
fn(L) 4 = 1 n n X i=1 l (zi, L) + λ1 2n||L|| 2 F (3.4)
where the loss function l (z, L) for each sample is defined as:
l (z, L)= min4 r,e 1 2||zi− Lr − e|| 2 2+ λ1 2 ||r|| 2 2+ λ2||e||1 (3.5)
Note that Equation (3.4) enables the possibility of updating the basis L based on each individual sample. To handle streaming data in practice, in [35], the estimation of basis Lt is obtained through minimizing the following surrogate function of (3.4)
with respect to L for the t -th time instance:
gt(L) 4 = 1 t t X i=1 (1 2||zi− Lri− ei|| 2 2+ λ1 2 ||ri|| 2 2+ λ2||ei||1) + λ1 2t||L|| 2 F (3.6)
Also observe that the loss function (3.5) optimizes r (the coefficient of zi on the
basis L) and e (the sparse component of zi) to minimize the cost given a fixed basis.
Through an alternating optimization of r, e and L, Equation (3.4) can be solved in an online manner. The complete stochastic optimization scheme for solving the OR-PCA problem is described in Algorithm 1.
Algorithm 1 Stochastic optimization for OR-PCA [35]
Require: {z1, . . . , zT} (sequentially revealed data samples), λ1, λ2 ∈ R
(regulariza-tion parameters), L0 ∈ Rp×r, r0 ∈ Rr, e0 ∈ Rp, A0 = 0r×r, B0 = 0p×r (initial
solution), T (number of samples).
1: for t = 1 to T do
2: Reveal the sample zt.
3: Given Lt−1, project the new sample:
{rt, et} = argmin r,e 1 2||zt− Lt−1r − e|| 2 2+ λ1 2 ||r|| 2 2+ λ2||e||1 (3.7) 4: At← At−1+ rtrTt, Bt← Bt−1+ (zt− et)rTt
5: Update the basis Lt
Lt 4 = argmin L 1 2T r[L TL(A t+ λ1I)] − T r(LTBt) (3.8) 6: end for
7: return XT = LTRTT (the low-rank matrix), ET (the sparse matrix).
Note that the right-hand side of Equation (3.7) in Algorithm 1 is equivalent to the loss function (3.5) for the t -th sample. To solve it, Feng et al. [35] give a closed-form solution to alternatively update r and e until a convergence criterion is met. The update of Lt in Equation (3.8) is discussed in the next section.
3.2.3.2.2 Update the subspace basis Lt
To minimize the function (3.6) with respect to L, note that the termλ1 2||ri||
2 2and
Lt 4 = argmin L 1 2T r[L T L t X i=1 (rirTi + λ1 t I)] − T r(L T t X i=1 ((zi− ei)rTi )) (3.9)
Using the two intermediate variables Atand Bt that accumulate information of
past frames, Equation (3.9) is equivalent to (3.8) in Algorithm 1. Equation (3.8) is then solved by the block-coordinate descent method, i.e., each column of the basis L is updated sequentially while fixing the other columns (see Algorithm 2).
Algorithm 2 The basis update using block-coordinate descent [35]
Require: L = [l1, . . . , lr] ∈ Rp×r, A = [a1, . . . , ar] ∈ Rr×r, B = [b1, . . . , br] ∈
Rp×r. ˜A ← A + λ1I. 1: for j = 1 to r do
2: Update the j-th column of L. lj ← 1 ˜ Aj,j (bj− L˜aj) + lj. (3.10) 3: end for 4: return L.
Another way of solving Equation (3.8) is to derive a closed-form solution. Let the derivative of the right-hand side of (3.8) with respect to L be zero, we obtain
1
2L(At+ λ1I)
T +1
2L(At+ λ1I) − Bt= 0 (3.11) where At= At−1+ rtrTt, Bt = Bt−1+ (zt− et)rTt. As (At+ λ1I) is symmetrical, a
simple closed-form solution of (3.8) can be derived as
Lt= Bt(At+ λ1I)−1. (3.12)
This is equivalent to the form given in [98].
3.2.3.3 Downweighting the Past Information
The previous solutions for the subspace basis update treat all samples equally, which works well for scenarios where samples are independently drawn. For stream video data, however, adjacent frames have higher correlation than “distant” frames. Thus, it may be possible to improve the basis update by treating past frames with different weights, giving close-by frames higher impact to the result than the distant frames. Inspired by the work in [70] which has reported several possibilities to handle past data in an online dictionary learning problem, we propose three approaches to downweight past information for the OR-PCA algorithm. In Algorithm 1, as At and Bt contain
information of past frames, variations can be made to replace the following equation set on line 4 in Algorithm 1:
At← At−1+ rtrTt
Bt← Bt−1+ (zt− et)rTt
(3.13) A logical choice is to apply an exponential decay (ED) to “forget” past information as in (3.14):
At← (1 − )At−1+ rtrTt
Bt← (1 − )Bt−1+ (zt− et)rTt
(3.14) where is the decay rate and 0 < < 1. So for the t-th time instance, the weight for the i-th sample is (1 − )t−i.
Similar to [70], as a second option we consider supra-linear decay (SLD) approach: At← 1 −1t ρ At−1+ rtrTt Bt← 1 −1t ρ Bt−1+ (zt− et)rTt (3.15) where ρ is a tunable decay parameter and ρ > 0. At the t-th time instance, the weight for the i-th sample becomes itρ
. Note that: when ρ = 0, (3.15) turns into (3.13); when ρ = 1, (3.15) degrades to a linear decay.
Apart from ED and SLD that scale the past data, it is also an option to focus only on adjacent frames in a fixed-size window, so that frames within the sliding window are treated equally, whereas the frames outside the window from the earlier times are not considered for the basis update, as follows:
At← rtrTt Bt← (zt− et)rTt , t0= 1 At← At−1+ rtrTt Bt← Bt−1+ (zt− et)rTt , t0> 1 and t 6 t0 At← At−1+ rtrTt − rt−t0r T t−t0 Bt← Bt−1+ (zt− et)rTt − (zt−t0− et−t0)r T t−t0 , else (3.16)
where t0 is the window size (number of frames within the window). This approach is
referred to as “sliding-window (SW)”.
3.2.4
Summary
The proposed online layer separation method consists of the following steps, as shown in Figure 3.1.
1. Breathing layer separation When a new XA frame is obtained, the breathing layer is firstly extracted by applying morphological closing on that frame, as described in Section 3.2.2. Subsequently, the breathing layer is subtracted from the original frame to obtain the DI.
2. Quasi-static layer and vessel layer separation Transform the DI from a matrix to vector by concatenating each column of the matrix one after another. This vector is then separated into two components by the OR-PCA method, as described in Section 3.2.3. The sparse component is reshaped to form the vessel layer, the other component is constructed as the quasi-static layer.
Finally, as pixels belonging to contrast agent always have negative value in the vessel layer, pixels with positive value in the vessel layer are heuristically set to zero to suppress artefacts.
3.3
Experiments
3.3.1
Image Data
In this work, we used three types of data for evaluation: clinical X-ray angiograms, synthetic low-contrast XA and X-ray angiographic data of pigs with variations in contrast concentration.
3.3.1.1 Clinical X-ray Angiographic Data
Imaging data from clinical routine that were anonymized were used for our experi-ments. The data were acquired under standard clinical protocol from the Department of Cardiology at Erasmus MC in Rotterdam, the Netherlands. The 42 XA sequences are from 21 patients who underwent a PCI procedure and were acquired with Siemens AXIOM-Artis biplane system. The frame rate of all sequences is 15 frames per second (fps). The number of frames per sequence varies from 46 to 244. All 42 XA sequences have in total 4886 frames. 22 sequences have 512 × 512 pixels, 12 have 600 × 600 pixels, 2 have 776 × 776 and 6 have 1024 × 1024. Their corresponding pixel sizes are 0.216 × 0.216 or 0.279 × 0.279, 0.184 × 0.184, 0.184 × 0.184 and 0.139 × 0.139 mm2,
respectively. In all sequences, inflow and wash-out of contrast agent can be observed. 3.3.1.2 Synthetic Low-Contrast XA
The synthetic image data was used to simulate the condition that a reduced amount (50%) of contrast agent is administered, for the purpose of testing our online layer separation method on low-contrast XA. To create these synthetic XA sequences from the real ones, we used the off-line layer separation method in [65] (Chapter 2 of this thesis). The idea is that the real clinical XA sequence was firstly separated into three layers. The intensity of the vessel layer was then halved and added back to the other two layers to generate a new XA sequence that has half the amount of intensity compared to the original one, as shown in Equation (3.17):
Isynthetic= α Ivessel∗ + I ∗ static+ I
∗
breathing (3.17)
where Isynthetic denotes the synthetic XA sequence, Ivessel∗ , Istatic∗ and Ibreathing∗ are
the vessel layer, quasi-static layer and breathing layer separated using the method in [65] (Chapter 2 of this thesis), respectively, and α = 0.5. The synthetic sequence
(a) Clinical XA (b) Synthetic XA
Figure 3.3: An example frame of real clinical XA sequences and synthetic low-contrast XA sequences: (a) the real image, (b) the synthetic XA frame with 50% vessel contrast.
has the same number of frames, same image size and resolution as its original in the clinical dataset. An example of a synthetic low-contrast XA is shown in Figure 3.3b. Note that the vessels have less contrast to the background than the original image in Figure 3.3a. We created a low-contrast XA sequence from each clinical XA described in Section 3.3.1.1, which results in 42 synthetic XA sequences in total.
3.3.1.3 X-ray Angiograms of Pigs
Additionally, in vivo XA data were acquired during a pig experiment performed at the Erasmus MC in Rotterdam, the Netherlands. 4 XA sequences with different contrast concentration levels were obtained from 1 FBM (familiar-hypercholesteremia Bretonchelles Meishan) pig which underwent a catheterization procedure after 14 months of high-fat diet. The XA sequences were acquired using a Siemens AXIOM-Artis monoplane system. The frame rate of all sequences is 15 frames per second. The number of frames per sequence varies from 48 to 79. The 4 XA sequences have in total 238 frames. All sequences have 776 × 776 pixels corresponding to a pixel size of 0.184 × 0.184 mm2. In all images, the inflow of contrast agent can be observed. The
XA images were made during a manual injection of iso-osmolar X-ray contrast medium (Visipaque 320, GE Healthcare, Buckinghamshire, U.K), delivered through the guide catheter. The full-contrast images were acquired with a contrast concentration of 320 g/mL. For the 25%, 50% and 75% contrast concentration images, the contrast agent was diluted accordingly with a 0,9% sodium-chloride solution (saline). Prior to image acquisition, the guide catheter was flushed with the right concentration of the contrast agent.
with 75% contrast. This might be due to incomplete flushing of the guiding catheter so that the contrast agent from the previous injection dilutes the current contrast agent.
3.3.2
Experiment 1: Parameter Tuning for OR-PCA
OR-PCA has three parameters: the intrinsic rank of the subspace basis r and the regularization parameters λ1 and λ2. In [35, 98], both λ1 and λ2 were set to 1/
√ p, where p is the dimension of data. This value had been proposed in [23] as a general rule of thumb, but it can be slightly adjusted to achieve the best possible result. Javed et al. [53], for example, have empirically selected different values for λ1 and λ2
instead of 1/√p. Unlike the rule for choosing λ1and λ2, the choice for r depends more
on specific applications.
In order to find the optimal parameter setting for the layer separation application on the clinical XA data, we used the following way to quantify the outcome of layer separation with a certain set of parameters.
3.3.2.1 The Definition of Foreground and Background
We firstly defined the “foreground” and the “background” for the objective of opti-mization in Section 3.3.2.2. It is worth noticing that the foreground and the back-ground here are merely defined for computing the vessel contrast and thus should not be confused with the foreground and background’s definition coming from the layer separation scheme described in the previous sections.
We used masks to define the foreground and background. A 1 mm wide area around manually-labeled vessel centerlines was considered as the foreground (shown as the dark area in the mask in Figure 3.4a). This area falls entirely within the vessel, and thus is a good representative of pixels belonging to vessels. For background, we adopted two different masks for measuring “global” and “local” contrast. The first one highlights all pixels outside a 4 mm wide area around the vessel centerlines (the white area in the mask in Figure 3.4b). This mask can quantify the effect of the removal of diaphragm, guiding catheters, etc. and can be used in a global measurement of contrast. The local background is defined as a 3 mm wide neighborhood area around the dark area in the global mask (the white area in the mask in Figure 3.4c).
For each clinical XA sequence, we randomly selected 8-15 frames for mask genera-tion and contrast evaluagenera-tion. The number of selected frames depends on the sequence length. As the vessel contrast is of main interest in this paper and in practice, only the frames with contrast agent were selected. This way we also avoided choosing non-contrast frames from the beginning of a sequence where the online algorithm has not converged yet. In total, 444 frames were chosen from 42 sequences.
We also created the masks for the four pig XA sequences. From each pig XA data, we randomly chose 8-12 frames. In total, 38 frames were chosen for the mask creation. These masks are only used for evaluation of the contrast level in pig data, not for parameter optimization.
