Complexity prediction of automatic image registration: a case
study on motion-compensated DSA
Citation for published version (APA):
Albers, A. H. R., Suijs, E., & With, de, P. H. N. (2010). Complexity prediction of automatic image registration: a case study on motion-compensated DSA. In Proceedings of the 2010 IEEE International Conference on Biomedical imaging: from nano to Macro, 14-17 April2010, Rotterdam, The Netherlands (pp. 1205-1208). IEEE Press. https://doi.org/10.1109/ISBI.2010.5490211
DOI:
10.1109/ISBI.2010.5490211 Document status and date: Published: 01/01/2010
Document Version:
Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers)
Please check the document version of this publication:
• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website.
• The final author version and the galley proof are versions of the publication after peer review.
• The final published version features the final layout of the paper including the volume, issue and page numbers.
Link to publication
General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain
• You may freely distribute the URL identifying the publication in the public portal.
If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement:
www.tue.nl/taverne Take down policy
If you believe that this document breaches copyright please contact us at: openaccess@tue.nl
providing details and we will investigate your claim.
COMPLEXITY PREDICTION OF AUTOMATIC IMAGE REGISTRATION:
A CASE STUDY ON MOTION-COMPENSATED DSA
Rob Albers
a,b, Eric Suijs
band Peter H.N. de With
a,caEindhoven University of Technology, PO Box 513, 5600 MB Eindhoven, The Netherlands, bPhilips Healthcare, Interventional X-Ray, PO Box 10.000, 5680 DA Best, The Netherlands,
cCycloMedia Technology, PO Box 68, 4180 BB Waardenburg, The Netherlands.
ABSTRACT
Real-time video and Quality-of-Service aspects play an in-creasing role in the development of medical imaging systems. To avoid resource overload and to guarantee the throughput of dynamic applications, we present a method for complex-ity prediction of image registration and motion-compensation algorithms, which can have a highly dynamic nature at run-time. As a case study, we explore a medical imaging function to reduce motion-artifacts in X-ray Digital Subtraction An-giography (DSA). Complexity prediction is based on motion estimation, prior to the actual image registration. Experimen-tal results show that it is possible to model a dynamic content-dependent processing task with a high accuracy (95%, stan-dard deviation 5%), thereby facilitating a higher quality for remaining tasks and well defined options for QoS.
Index Terms— Medical Imaging, Quality-of-Service,
Image Motion Analysis, X-ray Imaging, Multiprocessing 1. INTRODUCTION
Further advances in medical imaging involve live viewing and editing of high-definition content, intelligent behavior recog-nition and computer analysis of diseases within the medical domain. With the growing complexity of processing and software-based implementations, the immanent limitations become more apparent: compute-intensive tasks require so much power from the platform that not all remaining tasks can be performed at the desired quality, leading to arbitrary task switching and thus ungraceful degradation of quality. Alter-natively, sometimes application-specific dedicated hardware is employed which constrains the general-purpose nature of the architecture.
In professional medical imaging, multiple imaging appli-cations are executed in parallel on the same system, thereby sharing platform resources. To optimize system resources and simultaneously preserve the image quality, real-time pro-cessing aspects and Quality-of-Service (QoS) considerations should be applied [1]. This allows close to average-case re-source allocation and graceful degradation of quality if dead-lines of background applications are occasionally missed.
This problem statement has already been addressed in several fields of applications, such as high-performance computing and multimedia [2]. We borrow the developed concepts for these fields and impose them on professional medical imag-ing and processimag-ing, where architectures have different cost constraints and flexibility are of key importance.
We restrict ourselves to X-ray interventional imaging with a case study on image registration for Digital Subtraction An-giography (DSA) [3]. Eye-hand coordination is an impor-tant aspect in clinical practice, requiring low-latency imag-ing. Automatic motion-artifact correction for X-ray DSA is a powerful image registration technique based on template matching [4]. With template matching, a search is executed to match parts of an image to a visual template. As motion-based template matching is highly variable with the image motion, the computing demands and latency vary accordingly, which influence the other applications running in parallel. To con-trol the quality and latency, an ahead-of-time knowledge of the required resource budget is required.
Let us discuss the above fluctuations in more detail. As some steps are content dependent, images with severe motion artifacts will require more processing time than images with only small motion artifacts. From a system architectural per-spective, when multiple applications are executed on the same medical system, functions with dynamic resource usage ham-per the latency and throughput of other critical applications when resources are shared. Currently, the system designer has no other alternative than to assign a worst-case budget of re-quired resources and over provision the processing platform. To overcome this drawback, we present a method for run-time complexity prediction of motion-compensated DSA, to ensure sufficient response time for medical interaction, while still using a processing platform, dimensioned for average-case resource requirements. The result is a predictive knowl-edge of the required resource budget.
Earlier work has shown that future systems employing these kind of methods can provide higher QoS by more ef-ficiently utilizing the available system resources [5]. The pro-posed solution enables close-to-average case resource alloca-tion and graceful degradaalloca-tion of quality. As this technique
D E ('*( '(7(&7 &$11< &21752/ 32,17 6(/(&7 '(/$81$< 75,$1 *8/$7( 75,$1*/(6 75,$1*/(6 7(03/$7( 0$7&+ 0$6. ,0$*( :$53 68%75$&7 '6$,0$*( 0$6. ,0$*( /,9(,0$*( 0$6. ,0$*(
Fig. 1. Flowchart for motion-compensated DSA with (a) control point selection and (b) registration and compensation of the mask and live X-ray images.
can be found in other application areas such as video coding or face recognition, our work is relevant for a broader area than medical imaging only.
This paper is organized as follows: in Section 2, we briefly present the DSA application. Complexity prediction is described in Section 3. Experimental results are explained in Section 4 and Section 5 concludes the paper.
2. DIGITAL SUBTRACTION ANGIOGRAPHY (DSA) In DSA, a sequence of X-ray projection images is acquired during the passage of a bolus of injected contrast agent through the vessels of interest [6]. By subtracting an image acquired prior to arrival of the contrast agent (the mask im-age), background structures in the contrast images are largely removed. However, due to patient motion, DSA images often show motion artifacts that may hamper proper diagnosis. In order to reduce these motion artifacts, the misalignment of the successive images in the sequence needs to be determined and corrected. This alignment operation is often referred to as image registration. Many semi- or even fully automatic registration techniques have been developed over the past two decades [3]. In [4], an approach has been developed for reg-istration of angiographic image sequences which is effective and computationally efficient.
The algorithm [4] consists of several steps (Fig. 1). Given an image sequence withN images of size M × M pixels, the registration of the mask image,I0(x, y) (Fig. 2(a)), with
re-spect to the contrast images,In(x, y), with n the image index
andn ∈ [1, N − 1]⊂ N, is accomplished by carrying out the following six steps:
1. Compute the gradient magnitude, ∇L , at scale σ, of an
Mσ× Mσdownsized version of the mask image,I0(x, y), of
the sequence, and extract potential artifact regions by means of thresholding at levelΘe.
2. Extract control points from the exposure region,RE, by
us-ing the thresholded gradient magnitude, ∇L , of the mask image,I0(x, y)
3. Given the set of control points,P = pi, construct a triangular
mesh, D(P ), by using the incremental Delaunay triangula-tion algorithm proposed by Watson [7] (Fig. 2(b)).
4. For every image,In(x, y), n ∈ [1, N − 1] ⊂ N, in the
se-quence, compute the displacements, dn(pi), of the selected
control points, pi ∈ P , by maximizing the energy of the histogram-of-differences similarity measure, in a K × L neighborhood, using hill-climbing optimization.
5. Warp the mask image by deforming every triangleijkin
the meshD(P ), by using the displacements dn(pi), dn(pj),
and dn(pk) of the constituent control points pi, pj, and pk,
and the linearly interpolated displacements at the remaining points,p = pi, pj, pk, and by using bilinear interpolation of
grey values.
6. Compute the motion-corrected subtraction image, corre-sponding to the contrast imageIn(x, y), n ∈ [1, N − 1] ⊂
N, by subtraction with the motion-corrected mask image I0(x, y) (Fig. 2(d,f)).
3. COMPLEXITY PREDICTION
To be cost effective, it is required that the applications make efficient use of the available resources on the platform. Ex-ecuting a set of tasks on a multi-core processor platform requires matching the needs of every application with the available platform resources. Due to changing runtime con-ditions with respect to e.g. user application requirements, or having multiple simultaneously executing applications competing for platform resources, there is a need for a run-time decision-making and QoS arbitration entity: the runrun-time
manager. Also, as an answer to the variable processing rates
of applications, performance prediction may be applied in the form of modeling of the execution of the essential processing tasks, to guide the application.
For accurate resource management, static analysis alone will not solve the problem. Instead, an approach is required that combines static and dynamic analysis. To this end, we have developed a model for the DSA algorithm [4]. We re-quire a quantitative metric for the amount of used resources, predicting the cost of the next frame in contrast to the current image frame. Parameters of the algorithm can help in estab-lishing this predictive metric. In the following, we present the predictive algorithm in more detail and propose two metrics for the prediction of the dynamic parts. Prediction is based on estimating the motion of the incoming image frames prior to the actual image registration.
For analysis of static processing, tasks belonging to the generation of the triangular mesh of control points (Tasks 1, 2, 3), and the final image warping and subtraction task (Task 5, 6) are identified as static, having a deterministic computation time. The model can be defined with a constant value.
With respect to dynamic tasks, registration of the mask image (Task 4) with the live image depends heavily on the motion error of the mask image with the processed live image data. As this task is based on template matching, the actual
D E
F G
H I
Fig. 2.(a) Mask image, (b) triangular mesh generation on mask, (c) unregistered DSA image without motion, (d) registered, (e) unregis-tered DSA image with severe motion, (f) regisunregis-tered.
computation time varies with the number of control points and the required amount of iterations for each matching pro-cedure. The amount of iterations for template matching is heavily correlated with the motion between mask image and live images. Global object motion will have a higher impact on the complexity than local object motion, as the motion er-ror and the number of matching iterations will be higher also. Let us now discuss the creation of the complexity
predic-tion model. For estimating the complexity of image
registra-tion, motion estimation seems to be a logical candidate. We have considered several options for motion estimation. First, we try to calculate the intensities of the pixel differences be-tween the mask image and the live image. But with this ap-proach, we are unable to discriminate between global differ-ences (object motion) and local differdiffer-ences (injection of con-trast medium). Therefore, as a second approach, we down-scale the image to8 × 8 pixels. We call this an image stamp. Intensity differences between the image stamp of the mask and live image are made insensitive for local injection of con-trast medium and possible noise. The difference images cor-relate to (relatively large) object motions only. As the number of control points are not equally distributed across the image, we multiply each pixeli in the image stamp with a special
weight factorα(i), reflecting the amount of control points in-volved with the image stamp. Factorα(i) is given by:
α(i) = 10 ×PPi
max, (1)
where i represents the current pixel value, Pi the number
of control points at pixel i, and Pmax corresponds to the
maximum amount of control points per pixel. The resulting weighted pixel values are calledg.
The similarity measureM used for template matching is a very important metric in obtaining a good registration re-sult. Many measures have been proposed for the registration of X-ray angiography images [3]. In our case study, template matching is performed based on the Energy of the Histogram (H) of Differences (EHD), specified by:
MEHD(d) = G
g=−G
H2(g), (2)
with d the displacement in windowG for weighted pixel val-uesg.
For predicting the computation complexity of template matching, we have to correlate the obtained image stamp to the actually measured computing complexity of template matching for a set of training sequences. We apply the His-togram of Absolute Differences (HAD) metric for this task, defined by: MHAD(d) = G g=−G |H|(g). (3)
The HAD is found to be an effective metric to correlate the actual computation time to the prediction model using motion estimation (See Fig. 3). Compared to the EHD metric, the HAD is only slightly less accurate, but computational more efficient [3]. Histogram HHAD consists of 20 bins with a
variance σ = 100. The first bins represent small motion errors, the last bins represents large motion errors.
Template-matching-based image registration requires a more or less constant amount of iterations cm for image
frames with a small amount of motion or no motion. In the case of medium-to-large object motion, the amount of iterations increases drastically. As medium and large object motion errors are located in the last bins of histogramHHAD,
these are directly correlated with the peaks in computing com-plexity for template matching. We use this knowledge in our complexity prediction model. The prediction algorithm is executed as follows:
(1) The algorithm Cp maps the amount of bin items from
histogramHHADto a prediction for the computational com-plexity of a single pixel in the image stamp, which is de-scribed by: Cp= ⎧ ⎨ ⎩ cp0 , (0 <= H(19) < 2), cpi , (2i <= H(19) < 2i + 2), (1 ≤ i ≤ 5). cp6 , elsewhere. (4)
[ [ D E F G
Fig. 3. Four image stamps. (a) Prediction model without motion, (b) with severe motion, (c) actual computation time without motion, and (d) with severe motion.
(2) Prediction coefficientscp(cp0throughcp6) are described
by:
cp0= cm/Ptotal,
cpi= cp0× (1 + i/10), (5)
wherePtotalis the total amount of control points, and
1 ≤ i ≤ 6. From the first frame of a particular X-ray image sequence, the constantcmis determined.
(3) The computational complexity for the entire difference imageCtotalis defined as the sum of the individual pixel
con-tributionsCp.
(4) For every subsequent image frame i of a particular se-quence, an image stamp is created, (5) similarity measure
MHADis computed from HistogramHHAD and (6) finally,
complexity predictionCtotalis defined as the sum of the in-dividual pixel contributionsCp.
4. EXPERIMENTAL RESULTS
We apply the Histogram of Absolute Differences (HAD) sim-ilarity measure, to define a complexity metric for template matching. As global object motion has the highest impact on the complexity, the method first removes local motion pat-terns with an image pyramid, to create a downscaled image stamp. Fig. 3 shows that the prediction model has a strong spatial correlation with the actual computation time for tem-plate matching, with and without motion. Computation time statistics are obtained by profiling the application on a multi-core platform1. In Fig. 4, we show the experimental results
applying complexity prediction. A data set of 15 patients of in total 400 video frames was used. The proposed complexity prediction model shows an average accuracy of 95%, with a standard deviation of 6%.
1A shared-memory, quad processor-core system, 2.33 GHz, 4 GB RAM.
&RPSXWDWLRQWLPH PV 0HDVXUHGWLPH 3UHGLFWLRQPRGHO ,PDJHIUDPHV
Fig. 4.Experimental results, showing the prediction vs. actual com-putation time of image registration.
5. CONCLUSIONS
We have presented a method for runtime prediction of the required resources of image registration for X-ray DSA, for which the computation complexity is highly content depen-dent. The system is based on modeling the required computa-tions, by employing a combination of (1) motion estimation, based on the Histogram of Absolute Differences (HAD), and (2) a mapping stage, where the estimation results are corre-lated to the actual computation times of template-matching-based image registration. Experimental results show an aver-age prediction accuracy of 95% with a standard deviation of only 6%. The algorithm is fast and accurate enough to be used for runtime quality control. The results in this paper show that it is possible to model a dynamic content-dependent pro-cessing task with a high accuracy, thereby facilitating higher quality for remaining tasks and well defined options for QoS.
6. REFERENCES
[1] M. Roitzsch and M. Pohlack, “Video quality and system re-sources: Scheduling two opponents,” Journal Vis. Commun.
Image R., vol. 19, no. 8, pp. 473–488, 2008.
[2] V. Nollet and D. Verkestt, “A Quick Safari Through the MPSoC Run-Time Management Jungle,” in IEEE/ACM/IFIP Workshop
on Embedded Systems for Real-Time Multimedia, ESTIMedia,
2007, pp. 41–46.
[3] E.H.W. Meijering, W.J. Niessen, and M.A. Viegever, “Retro-spective motion correction in digital subtraction angiography: a review,” Medical Imaging, IEEE Trans. on, vol. 18, no. 1, pp. 2–21, Jan. 1999.
[4] E.H.W. Meijering, K.J. Zuiderveld, and M.A. Viergever, “Image registration for digital subtraction angiography,” Int. J. Comput.
Vision, vol. 31, no. 2-3, pp. 227–246, 1999.
[5] R. Albers, E. Suijs, and P.H.N. de With, “Resource usage prediction for groups of dynamic image-processing tasks using markov modeling,” in Acoustics, Speech and Signal Processing.
ICASSP. IEEE Int. Conf. on, April 2009, pp. 1929–1932.
[6] W.R. Brody, “Digital subtraction angiography,” IEEE Trans. on
Nuclear Science, vol. 29, no. 3, pp. 1176–1180, 1982.
[7] D.F. Watson, “Computing the n-dimensional Delaunay tessel-lation with application to Voronoi polytopes,” The Computer
Journal, vol. 24, no. 2, pp. 167–172, 1981.
