Automated segmentation & classification on OCT lung imaging of asthma patients

(1)

Automated segmentation & classification on OCT

lung imaging of asthma patients

dept. Biomedical Engineering & Physics Amsterdam UMC, location

AMC, Amsterdam, the Netherlands

By: CR de Graaf

10645500

D.M. de Bruin

M.W.A. Caan

August 2019

(2)

Chapter 1 ABSTRACT

1.1 EN

Optical Coherence Tomography (OCT) is a penetrative imaging technique able to capture micrometer-resolution images in three dimensions. Applying OCT imaging through use of an endoscope, many medical fields could potentially improve non-invasive imaging. Diagnosing asthma patients in the Amsterdam UMC through OCT imaging of the bronchus is one. Bronchial OCT alone is not sufficient for diagnostics and improvements from digital image processing could aid in effectively and efficiently assessing OCT images. This paper researches whether an automatic segmentation algorithm is able to identify muscular tissue in bronchial OCT and can differentiate between asthmatic and non-asthmatic bronchial OCT images.

5 lung cancer patients underwent lobectomy and had samples analyzed through Desmin histology and ex-vivo OCT. 16 unique OCT pullbacks were gathered and positionally matched with histology using suture needles before sample analysis. From a different sample group 132 unique in-vivo OCT pullbacks were gathered from 9 healthy and 9 severely asthmatic participants. Raw OCT images were cor-rected for PSF and Roleoff, log-normalization and linear-polar space conversion. A proprietary linear filter called ALN (average layer normalization) and convolutional filter MF (median filter) were used for feature enhancement. Threshold masking and component connectivity were used for feature extraction with cross-sectional bronchial surface area as main outcome.

Best performing automatic image processing run was ALN50 MF3 with Pearson correlation to histology of 0.61 and R2 of 0.38. All segmented surface areas were oversized compared to manual segmentations. Differences between asthmatic and non-asthmatic groups were inconclusive due to inconsistent measurements. In-vivo OCT imaging has more failed automatic segmentations than ex-vivo OCT. Future work should include different tactics for feature extraction. preferably a thresholding system that looks at change in contrast

(4)

1.2 NL

Optische Coherentie Tomografie (OCT) is een penetratieve beeldvormende techniek welk beelden in drie dimensies op micrometer schaal kan vastleggen. Door OCT via een endoscoop mogelijk te maken kunnen meerdere medische sectoren potentieel vooruitgang boeken in de kwaliteit en mogelijkheden van niet-invasieve beeldvorm-ing. Het diagnosticeren van de bronchiale luchtwegen van astma patienten in het Amsterdam UMC is daar een van. Bronchiale OCT op zichzelf is niet genoeg om diagnostiek uit te voeren. Vooruitgang in digitale beeldverwerking kan mogelijk helpen in het effectief en effici¨ent beoordelen van OCT. Dit paper onderzoeks of een automatisch segmentatie-algoritme spierweefsel in de bronchiale luchtweg kan detecteren en of dit algoritme astmatische OCT beelden van niet-astmatische OCT beelden kan onderscheiden.

5 longkanker patienten ondergingen lobectomie waarna weefsel door Desmine-histologie en ex-vivo OCT zijn onderzocht. 16 unieke OCT ’pullbacks’ zijn verzameld en gemarkeerd met hechtnaalden om de positie tussen histologie en OCT coupes te kunnen matchen. Van een andere groep zijn 132 unieke in-vivo OCT pullbacks verzamelt van 9 gezonde en 9 ernstig astmatische deelnemers. Ruwe OCT beelden zijn gecorrigeerd voor PSF en Roleoff, log-genormaliseerd en omgevormd van een po-lair stelstel naar Cartesiaans stelsel. Een zelfgeschreven linear filter genaamd ALN (average layer normalization) en convolutioneel filter (mediaan filter) zijn gebruikt als artefact-herkenning. Vervolgens zijn de technieken threshold masking en com-ponent connectivity gebruikt voor artefact-extractie. Hierin was de cross-sectionele bronchiale oppervlakte de hoofdzakelijke uitkomstmaat.

Het best presterende automatische beeldverwerkende algoritme was met de pa-rameters ’ALN50 en MF3’. Deze bereikte een Pearson correlatie met de histologische standaard van 0.61 en een R2 _{van 0.38. Alle gesegmenteerde oppervlakken waren}

een overschatting van de grootte in vergelijking met de handmatige segmentatie. Verschillen tussen astmatische en niet-astmatische groepen waren niet eenduidig dankzij inconsistente oppervlakte metingen. In-vivo OCT beeldvorming heeft meer gefaalde oppervlakte metingen dan ex-vivo OCT. Toekomstig werk zou andere tac-tieken moeten includeren voor artefact-extracitie, prefereerbaar een thresholding techniek dat kijkt naar relatieve contrastverschillen.

(5)

Chapter 2 INTRODUCTION

Optical Coherence Tomography (OCT) is a penetrative imaging technique able to capture micrometer-resolution images in three dimensions, mostly used in ophthal-mology [1]. By use of an endoscope, OCT can potentially be used in different medical fields where high resolution penetrative imaging could greatly increase in-sight on in-vivo physiology. A strong case can be made for enabling high-resolution reconstructions of bronchial epithelium[5, 4]. This technique is already used in diag-nostics for asthma patients in the Amsterdam UMC. The thickness of the muscular lining in the lungs can be evaluated using bronchial OCT. After interpretation by one or more specialists the creation of an appurtenant treatment planis supported. The treatment may in some cases consist of an endoscopic intervention where the muscle tissue is reduced.

Bronchial OCT imaging alone however is currently not sufficient in diagnostics of severe asthma and invasive and time/resource intensive alternatives such as biopsy, MRI or CT is always performed. One of the leading issues with bronchial OCT is the difficulty in extracting desired measurements. Manual measurement of mucosal and submucosal surface area has been proven to work and correlate[4] with results found in histological examination from biopsy. Manual measurement however is both in early stages and remains time intensive.

Digital image recognition and classification has taken big leaps forward due to ever increasing processing power and the constant invention of new image processing techniques. Digital image recognition and classification may open new possibilities of aiding medical professionals in quickly and effectively assessing complicated imaging such as bronchial OCT. To achieve this state, it is vital to know if an algorithm can recognize the desired structures which medical professionals are looking for. This pa-per focuses on image processing specifically for asthma diagnostics. Through writing and evaluating a custom software toolkit, the following questions are researched:

• What is the accuracy of an automatic segmentation algorithm identifying mus-cular tissue from OCT bronchial imaging?

• Can an automatic segmentation algorithm differentiate between the presence and absence of asthma from OCT bronchial imaging?

(6)

Chapter 3 BACKGROUND

Several concepts that will be mentioned require specific domain knowledge. This section functions as a frame of reference and a stepping stone to additional literature and important concepts. Important concepts from the medical domain include OCT imaging, lung anatomy, diagnostics and histology. Important concepts from the image processing domain include linear and convolution type digital filters, threshold masking, segmentation and the polar-cartesian image space.

3.1 OCT

Optical Coherence Tomography is a non invasive imaging technique popular in oph-thalmology. A single-waveform light pulse is emitted onto biological tissue. Depend-ing on the scatterDepend-ing of the light inside this tissue, similar to ultrasound imagDepend-ing, a penetrative image can be reconstructed [6]. In more recent years, the technique is beginning to extend to imaging of the urinary tract and lung imaging through use of an endoscope. For the purpose of endoscopic imaging, an endoscope is outfitted with a fiber optic cable running through the scope and a rotating mirror at the tip. Light is pulsed through the scope and reflected into the tissue by the mirror. The light interacts with tissue through reflection, refraction and absorption and is subsequently partially scattered back onto the scope. The returning signal is carried through the fiber optic cable back to a computer, which parses this feedback and calculates the intensity at the lights positional origin. While the mirror at the tip of the scope rotates, a two dimensional image can be reconstructed. When the scope is being pulled back while the mirror is rotating, the helical shaped trace of the pullback movement can be used to translate the input data into a three-dimensional structure. Whereas points in a two-dimensional (flat) image are referred to as ’pix-els’ in an X by Y plane, points in a three-dimensional image are referred to as ’voxels’, needing an extra, third, parameter Z to indicate its position.

Several machines capable of OCT supporting endoscopic practise are available on the market. Depending on the manufacturer and machine, image formats may vary from paginated TIFF files, to DICOM standard images, or other potentially proprietary formats.

(7)

3.1.1 OCT correction mechanisms

Viewing raw OCT files will reveal a mostly black image due to the extreme outliers in contrast. Correction of contrast is needed to make it viewable for the human eye. This is done using a log-10 transformation on all pixel values.

To correct systemic bias, a raw OCT image is adjusted with algorithms for PSF and Roleoff [3]. PSF (point spread function) characteristics for a specific OCT machine define the usable resolution of an image. A function correcting the image for the specific optics of an OCT machine is applied.

A Roleoff adjustment function is applied to correct the drop in sensitivity for signals from deeper tissue. The drop in sensitivity from measuring deeper tissue (fur-ther from the measuring scope) is inherent to SD-OCT machines (spectral domain OCT). Examples of all corrections can be seen in figure 3.1. [10]

(a) log corrected image with correct psf and roleoff configuration

(b) log corrected image without psf and roleoff correction

(c) raw image without log, psf or roleoff correction

Figure 3.1: Single OCT image, cut into three equal segments, detailing the impact of OCT correction mechanisms

3.2 Lung anatomy and asthma

Different structures of the airway wall can be recognized with OCT imaging: the mucosa (epithelial lining), submucosa (smooth muscle tissue) and cartilage. The OCT scope is used to image the bronchial wall. This region mostly consists of soft tissue for which there is no alternative high resolution imaging solution other than biopt microscopy. An OCT pullback usually covers a depth of around 5-7cm of the bronchial wall. The tissue thickness and composition of the bronchial wall differs based on the measured depth. Depth of measurement should therefore be taken into consideration when comparing tissue samples between individuals. In the OCT pullbacks used in this paper, roughly 3 to 5 different tissue compositions can be distinguished.

(8)

Asthma has multiple effects on the lung physiology . It has been known since as early as 1922 that asthma correlates with an increase of smooth muscle tissue and subsequently: increased thickness of the bronchial airway wall. General treat-ment mainly focuses on removing the involuntary muscle contraction or the inciting auto-immune response through bronchodilators or anti-inflammatory medication. Severity of asthma can be indicated through lung function tests such as spirometry. One of the most important outcomes is the FEV1 (Forced Expiratory Volume in 1 second) [9].

3.3 Histology and desmin coloring

In cases of severe asthma (based on the frequency of exacerbations; i.e. episodes) a biopt can be taken from one of the upper bifurcations in the bronchus. This biopt is used for histological examination. The biopt is sliced into coupes, stained on desmin (highlighting muscle tissue) and magnified. The resulting image is used as a highly reliable method of estimating the amount of smooth muscle tissue ex vivo and can be used in conjunction with the FEV1 to estimate severity of asthma[4]. See figure 3.2 for an example.

Figure 3.2: Manual surface estimation of bronchial lumen as well as mucosa and submucosa layers in both desmin stained histological slices (A, B) and SD-OCT slices (C, D). Histological slices are ex-vivo after lobectomy. Source: [4]

3.4 Image processing techniques

Raw image data from an OCT machine is insufficient to segment (cut out) desired features. There is noise in the image, edges might be blurred and undesired features might interfere with the segmentation algorithm. Sometimes features can be made

(9)

visible that were previously completely invisible to humans (logaritmic normaliza-tion). An input OCT image needs to be altered before segmentation. This section will detail all the techniques that were used to do so.

3.4.1 False coloring

Image data consists of lists of values describing each pixel in an image. OCT image data only has one color channel (greyscale). This means one pixel is described by one value. In contrast, popular standards for color images usually have three color channels: three values per pixel. One value for the amount of red, one for the amount of green and one for the amount of blue. The pixel values for an OCT image vary greatly. Most pixel values lie so closely together that if viewed on a linear scale, even after log-normalization, details do not appear clearly. Proprietary solutions exist to create an artificial ’false’ scale (eg. Golden Image tm, see figure 5.1a). Multiple colors are used successively to increase the contrast in a single color scale. For example: in an image with pixel values between 0 and 1000, values 0-330 may be colored from black to blue, values 330-660 from blue to green and values 660 to 1000 from green to white. Now more details are apparent compared to a single scale where values 0-1000 are colored only from black to white. A false coloring scale meant to achieve similar results to Golden Image tm has been created. See figure 5.1b.

3.4.2 Digital filters

In order to perform image segmentation as described in section 3.4.3, the input image must contain certain characteristics that are both related to the desired measure-ment and are recognizable by a given feature extraction algorithm. When working with greyscale or color images, these feature extraction algorithms often depend on detecting differences in contrast such as lines, corners or complex shapes. To enhance the performance of feature extraction, the image can be first altered by digital filters such that important image characteristics are more obvious to the feature extraction algorithm.

OCT images contain speckle noise due to the scattering of a single-waveform light beam on microscopic granular (rough) surfaces [12]. Speckle noise makes the image looks grainy. Even though it looks similar to random noise, the source of the noise follows a predictable distribution of contrast variation. Speckle noise looks similar to salt-and-pepper noise: a very grainy image with random white and black dots. Techniques to improve image quality in salt-and-pepper noise are generally easier to implement than speckle noise as many pre-written algorithms are freely and openly available [7]. Comparatively, image adjustment of speckle noise is often based on modelling the phenomenon [13, 2]. This modelling approach has good perseverance of detail. When smaller details of image structures is not important however, median filters can reduce speckle noise effectively [8] whilst maintaining edges.

While using a digital filter may enhance desired image characteristics, other image details may be lost. A smoothing or blurring filter for example may be effective in reducing image noise such as salt-and-pepper noise, but will reduce the sharpness of the image. Smoothing filters can thus cause details such as borders and lines to

(10)

disappear that might be important to the feature extraction algorithm. Conversely, median filters will maintain borders, but can cause subtle contrast differences to disappear. Choosing the right filter for the task is imperative. Both linear and non-linear processing techniques have been explored.

Average Layer Normalization (ALN)

The average layer normalization (ALN) is a custom made linear filter similar to a mean filter. A linear filter is based on a linear relationship between the input signal and output signal, such as adding input values together. The ALN is based on the three-dimensional representation of tissue from an OCT-pullback. In the X and Y direction for an axial slice and the Z direction for the proximally directed pullback. Neighbouring axial slices from the unprocessed OCT-pullbacks are approximately 90µm apart. The output for a specific voxel is generated by averaging all the voxel values of a given distance in the Z-axis (neighbouring slices). When looking at a single output slice, the image will look blurred. Regions of contrast in a slice will appear brightest where all included neighbouring input layers have high contrast on the same X and Y location.

The ALN is designed to bring out the thickness of the airway wall from an OCT-pullback. The contrast of the airway wall in an OCT slice is high compared to the lumen and deeper tissue. By using the stacking layer filter, the airway wall contrast is improved. See figure 3.3b.

(a) ’Default’ linear projec-tion of log10 raw image

(b) Stacking layer filter of 20 slices on linear projection

(c) Median filter with kernel size 19 on linear projection

Figure 3.3: Effects of different filters on OCT slice + false coloring

Median filter (MF)

The median filter is a commonly used convolution filter for salt-and-pepper noise. A convolution filter, rather than a linear filter, relies on the information in neighbouring pixels to calculate the output, rather than a simple mathematical operation for each pixel. Consider figure 3.3a as the input image and 3.3c as the output. The input image has a lot of noise that looks similar to salt-and-pepper noise. The median filter is an iterative function that looks at each individual pixel. For every iteration/input pixel the neighbours in a certain grid around that pixel are selected as well. This grid (or matrix) around a pixel can be different shapes and sizes and is called the kernel (3.4 ). The median filter specifically works by looking at the kernel (i.e. surrounding pixels) around a certain pixel and taking the median value. This median value will

(11)

be the output of the pixel. Subsequently, this process of overlaying a kernel and calculating the median is repeated for each pixel in the image until an output image can be made. See figure 3.4.

Figure 3.4: Median filter convolution demo. Green is currently targeted pixel. Grey are all pixels within kernel of size 3. The median value of all pixels in the kernel = new value of targeted pixel

The figure shows how any median filter will have a kernel of size X and Y where all pixels in the kernel are sorted in a one-dimensional list by value size. The middle value of the list will be the output of the center of the kernel.

The output of a median filter has different characteristics than that of the stack-ing layer filter or smoothstack-ing filter. Median filters preserve edges, but lose detail of gradual slopes of contrast. Depending on the kernel size, these gradual changes in contrast are converted into sharper edges. The higher contrast regions from the airway wall are separated by the median filter from the lower contrast regions of the lumen and deeper tissue. For an example, see figure 3.3c.

3.4.3 Feature extraction

Feature extraction is the next step. The goal of feature extraction in image pro-cessing is to move away from image data and to convert image characteristics to measurement data which is better suited for analysis. The two techniques applied to OCT lung images in extracting the feature: ”airway wall” were binary threshold masking and 8-way connected component analysis.

Threshold masking

In threshold masking, an image is transformed from a continuous range of pixel values to a nominal subset of values. Most commonly an image is converted into a binary mask with values 0 and 1. Besides an input image, an input threshold must be given, corresponding to the pixel value which dictates the ’edge’ of contrast. Any pixel value lower than the threshold is converted to value 0 and any value higher than the threshold will be converted into 1. The resulting binary image is commonly referred to as a mask.

When an OCT image of lung tissue is input and the edge between the lumen, airway wall and deeper tissue is visible (enhanced by a digital filter) and homoge-neous (similar contrast value along the edge), threshold masking can separate the component from unwanted data. The resulting mask from this separation loses vis-ible details of the image, but can be used for size and shape analysis. This size and shape data is necessary to automatic the segmentation process.

(12)

Component connectivity

The connected component algorithm stems from graph theory, where it identifies unique subgraphs of connected nodes. It has been adapted to work in image process-ing to identify components and component characteristics from masks. To calculate the size and shape of components in an image mask, the connected components algorithm tests the neighbours of each individual pixel with value 1. When any neighbour is also a one, these pixels are ’connected’. That pixel is then also re-cursively tested for connected neighbours until the entire connected component is uncovered. A neighbour in a two-dimensional image is commonly defined as ei-ther one of two ways: 4-way connectivity 3.5a (only pixels directly up, down, left and right of the input pixel) and 8-way connectivity 3.5b (4 way plus all diagonal neighbours).

(a) 4-way (b) 8-way

Figure 3.5: Common types of connectedness. Green is the currently targeted pixel in a connectedness test. Grey squares are potential neighbours

Through this algorithm the surface area (amount of connected pixels) and shape (relative location of connected pixels) of a component can be stored.

3.4.4 Folding linear-polar space

A regular digital image is made up of many squares, e.g. pixels. Any specific pixel can be uniquely addressed using a Cartesian coordinate system. This Cartesian coordinate system is the most common way method of plotting everyday graphs

(a) Polar space where Y = Θ

(b) Linear space where unit of Y is in px

(13)

where points are indicated by parameters X and Y . Raw image data from the C7-XR OCT machine is addressed using this coordinate, with an added dimension Z. When considering how this raw image data relates to the real world however, some adjustment is necessary. Considering any two-dimensional slice from the OCT pullback, there are X and Y coordinates indicating the position of a pixel. The Y coordinate however is not a normal coordinate. Whereas X can be converted into a distance metric such as mm, Y cannot. Y is an angular coordinate. See figure 3.7a. An angular coordinate such as Y contains information on the orientation of the OCT scope while it is turning around. When Y is 0, all contrast information in X is aimed in a certain direction, for example to the right. When Y is 270 out of a resolution of 540 , which is 50% of the full resolution, all contrast information in X is aimed at the opposite direction to the left. This means that all contrast information for any Y and when X = 0 is of the exact same point in space. For a comparison of linear vs polar space, see figure 3.6.

A transformation between the two image spaces is necessary. Different techniques and software solutions already exist. In matplotlib, a python image and data plotting library, the PColorMesh function is available. This function takes an image and transforms it over a given ’MeshGrid’. See figure 3.7b. A MeshGrid is a Cartesian space of points in which corners of shapes can be expressed. Using sin and cosin, the polar coordinates can be expressed in Cartesian space by drawing circles with straight lines through them [11].

(a) Polar coordinates Θ (b) linear-polar transformation where red co-ordinates indicate points on the MeshGrid

(14)

Chapter 4 METHODS

4.1 OCT data

4.1.1 Format

All available OCT pullbacks were creating using the C7-XR St. Jude Medical Inc. Ilumien Optis(tm) system interfaced with a C7 Dragonfly catheter (Ø0.9 mm diam-eter) running software version E.4.1 (Build 10918). Exported images were stored in the RAW multi-frame recording mode, as paginated TIFF files containing 540 pages as image slices.

4.1.2 Origin

All data originated from a lung OCT tissue identification study by J.N.S. d’Hooghe and A.W.M. Goorsenberg [4]. Raw OCT imaging data was provided from this study, including:

• 16 unique ex-vivo OCT pullbacks on 13 different airways from 5 NSCLC ( non-small-cell lung carcinoma) lung cancer patients from the Amsterdam UMC lo-cation AMC. All pullbacks were positionally matched with histological coupes on 51 positions. Data on airway surface area from manually segmented OCT images on these 51 positionally matched coupes was provided, as well as the surface area from histology.

• 132 unique in-vivo OCT pullbacks from 9 healthy and 9 asthmatic participants from the Amsterdam UMC location AMC and UMC Groningen

In previous research[4], patients were included after being diagnosed with non-small cell lung cancer (NSCLC) and underwent bronchoscopic work-up and lobec-tomy. Both in-vivo OCT (pre-lobectomy) and ex-vivo OCT (resected lung) imaging was gathered. The resected lung was histologically prepared in HE and Desmin color staining on 4µm coupes.

4.2 Processing tools

Dedicated hardware in the Amsterdam UMC, location AMC was provided with local access to the data. Patient data included raw OCT imaging, localization metadata

(15)

Figure 4.1: Flowchart of OCT data collection, manipulation and observation. Top half shows previous work [4]. Bottom half shows runs discussed below.

and anonimized patient identifiers. No data ever left the hospital. A software toolkit called SegOct, consisting of around 3k lines of code was written in python for this research. It is able manage digital OCT image files exported as paginated TIFF, applying digital filters and performing feature extraction. All techniques described in the background are available in SegOct. The SegOct toolkit is published on a gitlab repository internal to the Amsterdam UMC. To write SegOCt, Python 3.6 was used in combination with OpenCV2[7], MatPlotLib and NumPy. NumPy was used to write all techniques from scratch, enhanced with pre-written algorithms from OpenCV2 to aid image processing performance. MatPlotLib was used to show and save images and draw graphs. All UI elements are from OpenCV2.

The first step after obtaining raw image files is to sort and load the raw data. Although carrying the .oct file extension, reviewing file metadata showed the C7-XR exported raw images as paginated TIFF files. The SegOct toolkit supports reading these images including a short description from the filename. The SegOct toolkit has several classes to perform different types of operations on the image, mainly: view, corrections, segment and fold. These classes enable the consecutive use of the image operations discussed in the processing runs below.

4.3 Analysis

Different processing runs were repeated and different techniques were considered. Below is explained how these relate to the research questions. For a full overview of all steps see figure 4.1. letters A, B and C are used to denote different parts of the full analysis.

4.3.1 Verify data preparation (A)

To verify that the exporting and importing of OCT pullback data went successfully and without compression, and to verify correct data preparation by the SegOct toolkit, the image as output from the C7-XR OCT machine was compared with psf, roleoff and log corrected image data from the SegOct toolkit. Images as viewed on

(16)

the C7-XR use a proprietary false coloring technique called ”Golden Imagetm” that is not available outside of the C7-XR machine. A different false coloring scheme was written in SegOct and manually compared side by side with the C7-XR dicom file output. Potential loss of detail and contrast differences would be visualized.

4.3.2 Identifying muscular tissue from OCT lung images (B)

The performance of segmenting muscle tissue from OCT lung images with the SegOct toolkit was observed. As a golden standard Desmin colored histological coupes were prepared after the vivo OCT scan. The histological coupes and ex-vivo OCT coupes were matched by localizing the suture needles that were added prior to scanning. The histological coupes were manually assessed by the authors from [4] to identify different airway wall layers.

The automatic segmenter in the SegOct software was tested using different set-tings. Each run (performing a full set of operations including: image modification, segmentation and observation for a dataset) had different settings for the average layer normalization (ALN) and the median filter, and subsequently the outcome measured.

First step in any run: the raw image input was ’rebuilt’ to human readable format. Contrast was adjusted with log-10. Raw images from the C7-XR scanner did not need PSF and Roleoff correction. Finally, after the log contrast adjustment, the image file was folded from Polar-space to Cartesian-space.

Second step: digital filtering. Either one of two techniques, or a combination of both, was attempted in a single run. Multiple runs were performed to attempt segmentation using different parameters. These different runs will be denoted by the parameter given to each of these techniques (for example: ALN20 MF3 = 20 layers averaged with a median filter of 3x3).

• ALN (Average Layer Normalization) with varying levels of overlap. overlap of 0 slices, 3 slices, 20 slices and 50 slices were attempted.

• MF (Median Filter) with varying levels of kernelsize. a kernel size of 3x3, 5x5 and 19x19 were attempted (kernel size must be uneven).

Third step: Feature extraction. A binary mask was made using threshold mask-ing. Different methods for determining the threshold were used.

• A manual approach where a hard-coded value can be entered based on rough estimations after tissue inspection. The histological coupes were used to assess the location of the desired tissue. The threshold was based on the average between this and random noise far away from the scope.

• A manual approach where the SegOct user had to roughly draw a line through the desired tissue region and a line outside of the tissue region, but close to the border. The average pixel values between the two line selections was calculated by the software and used as a threshold.

• An automatic approach where the estimation of a pixel value threshold was made by the segmenter by looking at the larger components left over from the median filter. This approach therefore only worked with the median filter.

(17)

After feature extraction, the images were classified through the connected com-ponents algorithm. The algorithm returned the surface areas in pixels2 _{of all}

con-nected components. The largest component containing ones (thus excluding the background) was considered to always wrap around the desired area.

The resulting surface area in pixels2 _{was converted to mm}2 _{by calculating a}

conversion factor. The conversion factor is based on the diameter of the scope in the polar-space converted image from SegOct, compared to the diameter of the scope in the output of the St Jude C7-XR, which had a conversion factor in the metadata. The surface areas from the SegOct runs were compared to the surface areas from the corresponding histological coupes using the Pearson correlation test.

4.3.3 Identifying differences in lung OCT tissue between

asthma and non-asthma patients (C)

The best performing (highest Pearson correlation) parameters between runs from part B were used on a different dataset. 9 healthy participants and 9 patients with severe asthma from the Amsterdam UMC location AMC and UMC Groningen were enrolled in a follow-up study to [4]. These 18 participants underwent (in-vivo) OCT lung endoscopy. In total 55 healthy and 77 asthmatic pullbacks were collected. All pullbacks were exported from the St Jude C7-XR OCT machine and imported into SegOct. Subsequently the best performing parameters from run B were used on a single run on all 132 pullbacks. Differences in cross-sectional bronchial surface area between the two groups were visualized graphically.

(18)

Chapter 5 RESULTS

participants male [%] mean age mean mean OCT [#] [years] FEV1 [%]* pullbacks [#]

ex vivo 4 75.0 66 67.0 12

in vivo

- healthy 9 66.7 46 116.0 ** 6

- asthma 9 22.2 41 78.6 8

Table 5.1: Demographic information on the participants of both the in-vivo and ex-vivo lung-OCT groups. There is no overlap in participants between groups. *FEV1 % of predicted value; mean lung function. **FEV1 data missing on 4 participants.

5.1 (A) Data preparation

(a) OCT slice from C7-XR dicom export in GoldenImage tm

(b) OCT slice from SegOct. Log-adjusted view

Figure 5.1: OCT slice from St Jude C7-XR in GoldenImage tm _{false coloring vs St}

(19)

5.2 (B) Identifying muscular tissue from OCT lung

images

The best performing automatic segmentation run was with ALN = 50 (Average Layer Normalization) and MF = 3 (Median Filter). See table 5.2. This gave a Pearson correlation of 0.61 and R2 _{of 0.38. This run is further elaborated in figure}

5.2b. In contrast. the manual OCT segmentation achieves a Pearson correlation of 0.93 and an R2 _{of 0.87. See figure 5.2a.}

(a) Manual OCT segmentation vs Histology. Surface area

(b) Automatic OCT segmentation vs Histology. Surface area

Figure 5.2: Ex-vivo group (B) comparison between cross sectional bronchial surface area of manual and automatic OCT segmentation on vertical axes. Manual histo-logical segmentation on horizontal axes as golden standard. All surface areas are in mm2.

(20)

Figure 5.3: Comparison of cross-sectional bronchial surface area between histology, manual OCT segmentation and automatic OCT segmentation. The Y-axis con-tains all 16 pullbacks combined sorted by depth. Pullback depth for histology was determined by matching the position of suture needles[4]

5.3 (C) Identifying differences in lung OCT tissue

between asthma and non-asthma patients

(21)

ALN 1 3 20 50 MF 1 Pearson 0 0 0 0 R2 0 0 0 0 MF 3 Pearson −0.55 0.05 0.42 0.61 R2 _0.31 _0.00 _0.17 _0.38 MF 5 Pearson −0.55 0.05 0.41 R2 _0.30 _0.00 _0.17 MF 19 Pearson −0.57 0.05 0.41 R2 _0.33 _0.00 _0.17

Table 5.2: Correlation between surface area from automatic OCT segmentations and Histology from different runs. ALN = Average Layer Normalization and was varied between 1 (off), 3, 20 and 50 slices above and below the OCT slice mapped with Histology. MF = Median Filter and was varied by changing the kernel size between 1 (off), 3, 5 and 19. Correlation is measured with a Pearson test where -1 = perfect negative correlation, 0 = no correlation and 1 = perfect positive correlation. R2 _is

provided to show variance around linear correlation line.

Figure 5.4: Comparison of cross-sectional bronchial surface area between first fifteen OCT pullbacks from the Healthy and Asthma groups. For every pullback, the bronchial surface area was calculated once every 10 slices (horizontal).

(22)

Chapter 6 DISCUSSION

6.1 Main findings

The OCT segmentation run with parameters ALN50 and MF3 showed the highest positive Pearson correlation of 0.61 and an R2 of 0.38 when compared to the his-tological golden standard. Small ALN values of 1 or 3 show no or even negative correlation. Use of the stacking layer mask on more than 3 layers at a time results in visible mismatching of the automatic segmentation vs the general wanted outline of the airway wall in OCT images 6.1b. All surface areas calculated by the best performing automatic segmenter are overestimated. See figure 5.3. Even though the automatic segmenter shows positive correlation (0.61) with histology, future improvements can be made as the correlation is still lower than manual (0.93).

Comparing the surface area of the OCT image airway wall between asthmatic and non-asthmatic patients using any automatic segmentation run does not seem different. There are a lot of missing segmentations unable to find the right surface area completely, as shown by the occurance of very small surface area measurements, generally under 2 mm2. Due to the amount of failed automatic segmentations in-vivo compared to ex-vivo OCT, conclusions on a significant difference remain inconclu-sive.

6.2 Interpretation

The average layer normalization and the median filter both have their uses in specific scenarios. The average layer normalization can be used to average out the airway wall in slices with a lot of small structures. These small structures can disrupt the thickness measurement (very rough segmentation borders). For the area measure-ment however, averaging multiple layers causes a malformation of the mask when examining a single layer. See figure 6.1b.

The median filter when used as a part of a thresholding technique seems to seg-ment out the airway wall correctly and without overestimation (figure 6.1a). Actual measurements on the surface area from the automatic segmentation do not correlate with the histological standard without a large ALN. The ALN1 MF3 segmentation run for example even shows negative correlation. It is important to note that after sampling different segmentations, some seem to undercut the airway wall, whilst other segmentations seem to overcut the wall. This could indicate that the current method of thresholding is an insufficient detector in the contrast difference of the

(23)

(a) OCT segmentation from ALN3 MF3 from correctly segmented example.

(b) OCT segmentation from ALN50 MF3 showing overestimation of the surface area

Figure 6.1: automatic OCT segmentations overlayed on log-normalized slice

airway wall in OCT images. A different approach looking at the intensity of contrast change ∆x could prove more accurate.

When not accounting for the linear-polar space transformation, a fairly strong inverse Pearsson correlation (approx. -0.82) was found between the automatic OCT segmentation and the histological segmentation. The exact reason remains specu-lation, but the removal of any correlation after adjusting for the linear-polar space indicates a relation between larger segmented regions when most of the mask area is close to the scope and smaller segmented regions when most of the mask area is far away from the scope. This correlation is supported by the characteristic that in manual OCT segmentation, when no airway wall was visible, the ’invisible’ but ex-pected surface area was included. Meanwhile in the automatic segmention without linear-polar space transformation, a larger surface area away from the scope repre-sented less surface area compared to the truth and the chunks of ’invisible’ surface area were excluded. Another theory is a relation between distal and proximal airway tissue, where distal airways are smaller and lie closer to the scope than proximal airways. Measurements closer to the scope are more detailed and less prone to noise, improving the odds of an accurate segmentation.

6.3 Strengths and limitations

The speed at which a segmentation is created is faster than manually possible. With a small ALN (for example ALN3 MF3 in figure 6.1a) the segmentation for a full uncompressed pullback took approx. 11 seconds. Without ALN, the full segmentation process took about 7 seconds. With minor tweaking, this process can expectedly be made real-time.

The false colored image from the OCT machine showed slight compression similar to guassian blur. The SegOct toolkit provides no compression.

There is no publicly available package that is capable of folding RAW OCT data between linear and polar space. The SegOct toolkit makes this conversion possible. Segmenting an OCT slice with a small ALN doesnt always work. There are too

(24)

many failed segmentations. When corrected for these failed segmentations there is still no correlation. Possibly the information from many slices (high ALN) is needed to provide a smoothed out image without unwanted artefacts. With a low ALN, the pearson correlation graph with the low ALN segmentation vs histological segmentation shows a clear separation. Generally, all OCT segmentations with a surface area below 2 mm2 _{tend to be of a wrong component, impacting performance.}

See figure 6.2.

Figure 6.2: Segmentation correlation graph of ALN3 MF3 automatic OCT vs his-tology

The approach to thresholding is less than ideal. It was difficult to base a threshold on evidence and manual threshold selection takes time. The automatic thresholder is based on image features present in every pullback: the intensity inside the scope. The contrast located on or around the scope has no proven relation to the biological tissue surrounding it. In edge cases the threshold causes the segmentation to be of a wrong image component. Due to none of the thresholding techniques having any effect on the outcome, automatic thresholding was used in generating all results. The segmentation is limited to selecting the second-largest component that was found with the connectivity algorithm in the image. The largest component is assumed to be the background. This is not always the case, aiding to some segmentations failing completely. When using component connectivity, gaps between airway wall tissue (for example a sudden increase in lumen diameter due to a bifurcation) will mean two components are found that both describe different parts of the airway wall. The smaller of the two components is now always left out.

Without a median filter the segmentation does not work and returns surfaces of 0. Something inherent to the segmentation algorithm was most likely made dependant on a feature extracted by the median filter during programming.

The systemic correction parameters PSF and Roleoff are specific to the manufac-turer and the machine. These numbers were provided, but remain magic numbers in the software.

The type of noise is speckle noise which has its own types of preprocessing other than median filters. A median filter is used as it is a good option for salt and pepper noise, which looks similar to speckle noise and maintaining small details is not necessary for the segmentation to be accurate. There are several downsides as details of gradual change in contrast are lost. Nonetheless, the median filter provides edge preservation capability, which is more important than image details. Median filtering might not work in other scenarios where the oct image contains a softer speckle noise with less sharp spikes in contrast.

(25)

For the linear-polar image transformation, a conversion factor had to be cal-culated. Calculating the dimension of scale was too mathematically challenging when converting between polar and Cartesian space and was omitted. Instead, the diameter of the scope was used to estimate surface area.

6.4 Future work

The focus of this paper was to compare the automatic OCT airway wall segmenters performance to proven standards. These standards were recorded as surface areas to recognize different airway wall structures from OCT imaging. The preferred measurement for airway wall thickness is depth, as surface areas can vary between patients with similar airway wall thicknesses. The airway wall thickness could have been extracted from original data, but was not available at this time.

Different techniques than threshold segmenting should be considered. Mainly focussing on change in contrast as a threshold, instead of a set univeral cut-off point. Raycasting type techniques could prove useful in measuring thickness using this change in contrast ∆x.

Python is not the fastest language, especially for image processing. Writing the toolkit in a different language (for instance C++), utilizing GPU acceleration or supporting multithreading will all expectably yield significant performance im-provements.

Finally, some images showed a slight measurement drift. Structures in two spe-cific pullbacks seemed to slightly rotate around the scope. this could be due to scope rotation, or another systemic measurement error. All of the discussed measurements were taken on a two-dimensional plane, but when considering measurement or seg-mentation techniques in the three-dimensional plane, rotational drift should be taken into account. -real time aspect and performance improvement possibilties

(26)

Bibliography

[1] Mehreen Adhi and Jay S Duker. “Optical coherence tomography–current and future applications.” In: Current opinion in ophthalmology 24.3 (May 2013), pp. 213–21. issn: 1531-7021. doi: 10 . 1097 / ICU . 0b013e32835f8bf8. url: http : / / www . ncbi . nlm . nih . gov / pubmed / 23429598 % 20http : / / www . pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC3758124.

[2] Desmond C. Adler, Tony H. Ko, and James G. Fujimoto. “Speckle reduction in optical coherence tomography images by use of a spatially adaptive wavelet filter”. In: Optics Letters 29.24 (Dec. 2004), p. 2878. issn: 0146-9592. doi: 10. 1364/OL.29.002878. url: https://www.osapublishing.org/abstract. cfm?URI=ol-29-24-2878.

[3] Anant Agrawal et al. “Characterizing the point spread function of retinal OCT devices with a model eye-based phantom.” In: Biomedical optics express 3.5 (May 2012), pp. 1116–26. issn: 2156-7085. doi: 10.1364/BOE.3.0011163. url: http://www.ncbi.nlm.nih.gov/pubmed/22567601%20http://www. pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC3342187.

[4] Julia N. S. d’Hooghe et al. “Optical coherence tomography for identification and quantification of human airway wall layers”. In: PLOS ONE 12.10 (Oct. 2017). Ed. by Marinko Sarunic, e0184145. issn: 1932-6203. doi: 10 . 1371 / journal.pone.0184145. url: https://dx.plos.org/10.1371/journal. pone.0184145.

[5] Ming Ding et al. “Measuring Airway Remodeling in Patients With Differ-ent COPD Staging Using Endobronchial Optical Coherence Tomography.” In: Chest 150.6 (Dec. 2016), pp. 1281–1290. issn: 1931-3543. doi: 10.1016/j. chest . 2016 . 07 . 033. url: http : / / www . ncbi . nlm . nih . gov / pubmed / 27522957.

[6] D Huang et al. “Optical coherence tomography”. In: Science 254.5035 (Nov. 1991), pp. 1178–1181. issn: 0036-8075. doi: 10 . 1126 / science . 1957169. url: http : / / www . ncbi . nlm . nih . gov / pubmed / 1957169 % 20http : / / www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC4638169% 20http://www.sciencemag.org/cgi/doi/10.1126/science.1957169. [7] _{Image Filtering — OpenCV 3.0.0-dev documentation. 2014. url: https://}

docs.opencv.org/3.0-beta/modules/imgproc/doc/filtering.html. [8] Jyoti Jaybhay and Rajveer Shastri. “Signal & Image Processing”. In: An

International Journal (SIPIJ) 6.3 (2015). doi: 10.5121/sipij.2015.6306.

url: https://pdfs.semanticscholar.org/d4d2/9b986c4ceb0c2bd476fa652068b254393f92. pdf.

(27)

[9] L.F Keglowich and P Borger. “The Three A’s in Asthma – Airway Smooth Muscle, Airway Remodeling & Angiogenesis”. In: The Open Respiratory

Medicine Journal 9 (2015), p. 70. issn: 1874-3064. doi: 10.2174/1874306401509010070. url: http://www.ncbi.nlm.nih.gov/pubmed/26106455%20http://www.

pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC4475688.

[10] Sang-Won Lee et al. “Optimization for Axial Resolution, Depth Range, and Sensitivity of Spectral Domain Optical Coherence Tomography at 1.3 m.” In: The journal of the Korean Physical Society 55.6 (Dec. 2009), pp. 2354–2360. issn: 0374-4884. doi: 10 . 3938 / jkps . 55 . 2354. url: http : / / www . ncbi . nlm.nih.gov/pubmed/23239900%20http://www.pubmedcentral.nih.gov/ articlerender.fcgi?artid=PMC3519436.

[11] _{matplotlib.pyplot.pcolormesh — Matplotlib 3.1.1 documentation. url: https:} //matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.pcolormesh. html.

[12] J. M. Schmitt, S. H. Xiang, and K. M. Yung. “Speckle in Optical Coherence Tomography”. In: Journal of Biomedical Optics 4.1 (Jan. 1999), p. 95. issn: 10833668. doi: 10.1117/1.429925. url: http://www.ncbi.nlm.nih.gov/ pubmed / 23015175 % 20http : / / biomedicaloptics . spiedigitallibrary . org/article.aspx?doi=10.1117/1.429925.

[13] Alexander Wong et al. “General Bayesian estimation for speckle noise reduc-tion in optical coherence tomography retinal imagery”. In: Optics Express 18.8 (Apr. 2010), p. 8338. issn: 1094-4087. doi: 10 . 1364 / OE . 18 . 008338. url: http://www.ncbi.nlm.nih.gov/pubmed/20588679%20https://www. osapublishing.org/oe/abstract.cfm?uri=oe-18-8-8338.

Automated segmentation & classification on OCT lung imaging of asthma patients