PERCEPTUAL INDUSTRIAL SURFACE QUALITY INSPECTION

(1)

PERCEPTUAL INDUSTRIAL SURFACE QUALITY INSPECTION

with application to Playmobil toy figures

MSc Thesis Herbert Kruitbosch

born January 19th, 1988 at Voorst, The Nederland

(2)

(3)

First supervisor:

Prof. dr. N. Petkov Second supervisor:

Dr. G. Azzopardi

(4)

(5)

ABSTRACT

Automated industrial quality inspection is an important factor in economizing production processes and improving the quality of products. Manual alternatives are often hard to implement in industrial settings or too expensive. This thesis proposes the use of perceptual image quality measures for this purpose and as part of a pipeline applies such a measure, structural similarity, to assess the quality of Playmobil toy figures. The pipeline also includes methods to select an ideal golden reference for comparison with a specific test sample, which significantly improves generalization. The pipeline ends by focussing on low-similarity regions of the toy figure and derives one value as a response to quan- tify the defectiveness. To test the pipeline we also implemented methods to create artificial data sets.

Ultimately we achieve a classification accuracy between 90% and 100%.

(6)

(7)

ACKNOWLEDGEMENTS

This research project was a challenging tasks, for which I very often required the efforts, intelligence, patience and understanding of others. In particular, I wish to express my sincere gratitude to both my supervisors, Dr. George Azzopardi and Prof. Dr. Nicolai Petkov, who encouraged me to look in different directions and offered invaluable assistance, guidance, motivation and support. I would like to thank all university staff and fellow-students who offered significant discussions and insights.

I would like to thank my parents, sister and grandmother for their confidence that I would complete my thesis during the long time it took. I would also like to thank Sjoerd for the countless philosophical discussions which improved my way of research. Lastly I am grateful to all the others who have helped me.

(8)

(9)

Introduction

Abstract

Human perception plays an important role in industrial quality inspection: fruits, vegetables, fabric and other products sell better when they look better. Manual inspection naturally allows for measuring different perceptual quality cues, however human workers may be expensive, subjective and unsuited for dangerous production environments. They may also not keep up with production speeds or experience fatigue during a long work day.

This chapter discusses the general problem of perceptual and surface quality inspection. Specifically, a problem of inspecting Playmobil toy figures is introduced, children’s increasing spending power induce a need for such systems.

1.1 Background

Human visual perception, called perception in this thesis, uses recognition memory which allows us to recognize different types of objects. For example, one person who has seen several dogs may recognize a new one even though it looks different. This illustrates that human perception is able to consider very different objects to be of the same class or category.

Additionally, perceived quality plays an important role in consumer decision making. For example, a study regarding consumer quality determination regarding fresh vegetables and fruits shows that during the sale perceived attributes are important, while during consumption experienced attributes are important (Ragaert et al., 2004). Other quality cues may include product familiarity and brand (Bredahl, 2004) and country of origin (Elliott and Cameron, 1994). Children are mostly influenced so- cially, by television, by family and friends and by brand recognition (Dotson and Hyatt, 2005). With the increasing spending power of children (McNeal, 1999), quality perception of toy figures may, however, also be a profitable attribute of toy figures.

Computer-based computations that mimic human perception can help industrial processes to asses how a product’s quality appeals to a consumer. This allows industries to perform competitively in a global economy, where traditionally quality control is performed by human experts (Mital et al., 1998).

Although machine vision systems may operate faster, cheaper and more consistently, their develop- ment is often not trivial and many such systems were developed only to suit one type of production process. Automated inspection may still pay off, as estimations indicate that only 80% of a human oper- ator’s decisions agree with the supervisor’s decisions (Juran, 1962; Sannen and Van Brussel, 2009) and industrial settings may be too dangerous for humans operate in.

As a consequence, machine vision quality inspection systems have been used intensively in fabric defect detection (Schneider et al., 2014; Kumar, 2008), soldering defect detection for printed circuit board (Benedek, 2011; Mar et al., 2011) and agricultural products (Bhatt and Pant, 2013; Patel et al., 2012; Zhu et al., 2007), but also for transparent parts (Mart´ınez et al., 2013), pharmaceutical tablet imprints (Moˇzina

(12)

2 1. Introduction

Figure 1.1:Example defects from (Tolba, 2011; Mart´ınez et al., 2013; Leemans et al., 1999; Moˇzina et al., 2013), defects are outlined red, red, green and red respectively. The first image shows defective fabric, the second a damaged transparent surface, the third an apple with bruises and the last a tablet with partially damaged inscription.

et al., 2013), granite surfaces (Shafarenko et al., 1997) and bank notes (Torres et al., 1998). Of course, this summation is not complete.

1.2 Problem definition

Quality inspection covers a broad set of problems, since many different definitions of quality or defective and many industrial processes exist. Some examples are shown in Figure 1.1. The general approach is that a product is identified as one of different quality or defect classes based on some physical property.

Figure 1.2 describes a generic visual quality inspection system. After image acquisition, features are extracted, in order to then be classified. Which features are extracted typically depends on the definition of quality and what is considered to be the ground truth. For example, fruits may be judged on how appetizing they look, whereas pharmaceutical tablets may be judged on the readability of imprints.

With respect to implementation, Figure 1.2 has many realisations. Classification can be sample-wise which accepts or rejects a whole sample or pixel-wise which localizes potential defects. The classification can use different samples for training. For example semi-supervised which only trains on acceptable samples, fully supervised which trains on acceptable and defective samples or unsupervised which uses no training data at all.

Quality inspection can classify a sample to be defective or not, but may also specifically mark defec-

Figure 1.2:Quality inspection system diagram. Some methods also classify into different types of defects and some methods are unsupervised.

(13)

1.3. Motivation 3

A B C D E F G

Figure 1.3:Golden references: different toy figure types of defect free Playmobil parts.

tive regions. The first typically is applied to products which can either be defective or of good quality as a whole, like fruits and pharmaceutical tablets. Applications for the latter are those that inspect surfaces which may be partially declined, such as textiles, lumber or metal.

A subtle, but important difference to note is that between a golden reference and a ground truth. The golden reference is a reference image of what an acceptable, industrial product is supposed to look like. A ground truth is a typically human created segmentation or classification of a data set, indicating what is expected from machine vision algorithms. In this particular case, the classification between acceptable or defective samples.

This thesis investigates different objective perceptual measures to assess the quality of industrial Playmobil parts. These parts have colourful, painted plastic surfaces in order to appeal to children.

Several golden reference images are shown in Figure 1.3. These parts have an acceptable appearance.

Due to variance in production quality, however, some of the parts may contain scratches or paint stains. Such examples are shown in Figure 1.4. This thesis investigates the detection of such defects while ignoring differences which are perceptually inconspicuous.

1.3 Motivation

The first section of this chapter discussed motivations for automated inspection in general. These include disadvantages of human inspection, like subjective judgement, fatigue, safety limitations, labour costs, inspection speed and effectiveness. Although no effective general purpose inspection systems exist, custom made solutions can pay off by improving product quality and economizing human labour.

Nevertheless, not all industrial plant owners can invest in automated quality control, while certain surface quality inspection systems may also apply to more general problems than their original, ren- dering custom made systems otiose. Hence generic perceptual quality inspection systems can increase economic effectiveness on a competitive market.

To the best of our knowledge, perceptual image quality measures have not been used for surface

Figure 1.4:Five defective regions in parts of the last and fore last part types shown in Fig. 1.3. Their locations on the toy figures are marked with the red rectangles shown right.

(14)

4 1. Introduction

quality inspection. Several of such measures are discussed by Lin and Jay Kuo (2011). Their current main use is comparison of original and distorted images to assess or steer compression algorithms.

Such measures should weigh the difference between a reference and test image with respect to human perception. Since industrial quality inspection may also need to measure such perceptual cues, this thesis postulates the use of these measures for generic surface quality inspection.

1.4 Overview

The next chapter shortly surveys several existing visual quality inspection systems and shows that generic systems are scarce or non-existent. Chapter 2 also discusses a framework for image quality measures. Subsequently, Chapter 3 describes the nature of the small Playmobil data set and proposes two techniques to artificially create acceptable and defective samples. Furthermore, the chapter proposes the use of orthogonal vector spaces to represent the golden references. Comparing potentially defective samples with their projection on this space focusses on defect induced differences and ignores differences due to acceptable production variance. For sample comparison the structural similarity (SSIM) is proposed.

Chapter 4 tests the proposed inspection set-up with respect to classification accuracy, comparing the the Euclidean distance with SSIM and comparing two projection methods. The outcomes are discussed in Chapter 5 and there is a conclusion in Chapter 6.

(15)

Chapter 2

Related work

Abstract

Machine vision is used intensively in industrial settings to reduce costs and increase accuracy of quality inspection tasks. This chapter discusses several methods proposed for several industrial use cases. Many of the techniques used in surface and texture quality inspection are based on local features, which are then classified as either defective or correct. In contrast many of the structural surface inspection methods use a golden reference as a template to match on test samples. As an overview, this chapter provides a table with the features and classification methods discussed.

In general, pixel-wise differences say very little about the humanly perceived difference between two images. Such perception often cannot be mimicked by the Euclidean distance. Hence perceptual quality measures exist, they try to mimic human perception and are classically used to measure how similar compressed images are to their original. This chapter discusses flaws of the Euclidean distance with respect to perception and suggests the use of perceptual image quality measures.

2.1 Compact survey

This chapter compactly compares existing visual quality inspection systems which try to find some kind of surface defect in either a flat surface, like textile, or a non-flat surface, like an apple. Other quality inspection systems may detect operational or dimensional quality, these are not discussed as they are beyond the scope of this study. Hardware capturing systems are also not discussed, nor are feedback systems between a computer vision algorithm and a capturing device.

2.1.1 Examples

This section discusses several quality inspection techniques used in an industrial setting. The types of products used are broad. Examples for general surfaces like metal, glass or textiles, electronics, agricultural products like fruits and meat and pharmaceutical tablets are given.

Many industrial products are in the form of some kind of surface, like metal, glass, textile, wood, etc. These type of surfaces may contain defects, like scratches and holes. This section discusses some methods to inspect their quality, starting with transparent surfaces.

Transparent surfaces may suffer from various defects, especially punctual (blisters, drops and bub- bles), linear (scratches and threads) and superficial (burrs, sinks, general superficial damage). Mart´ınez et al. (2013) proposed a defect detection system to mark defective regions on transparent surfaces. After image acquisition, the system segments the image using different threshold-based techniques applying histogram information of the image.

(16)

6 2. Related work

Homogeneous surfaces Tolba (2011) proposed the use of Log Gabor filter banks to create structure features, indicating the amount of lines and edges in certain directions in a certain window. Feeding the window’s features into a probabilistic neural network classifies whether the window is homogeneous like the rest of the surface or damaged. The result is a segmentation marking anomalies not similar to an otherwise homogeneous surface.

Random surfaces Granite is a randomly textured material and in order to assess its quality a segmentation method is proposed by Shafarenko et al. (1997). The method segments each region of the granite which has a homogeneous colour using the Watershed algorithm based on the LUV gradient. This gradient assigns to each pixel the maximum LUV-distance, the Euclidean distance in the LUV colour space, with its neighbours.

Due to local minima that are introduced by noise, the Watershed-method over-segments, which is resolved by merging adjacent segments with similar homogeneous colour. This bottom-up procedure stops when the adjacency graph of the segments has a maximum number of cycles. The maximum is determined by adjacency statistics of granite. Features of the segments can be extracted to determine which ones are perceived by humans as salient. This could then be used to assess the quality of granite surface.

Electronics One particular problem of solder joint detection is the specular behaviour of metal. Slight changes in light-camera-object orientation may introduce or remove specular effects. Bartlett et al.

(1988) introduce a rule-based solder defect inspection system. First the method applies masks on each solder joints to extract corners, the center and what is in between both. Then for all these subsets features, like average value, are extracted, and combined as one feature vector. These features are used in the rule-based system to classify different types of soldering defects.

A more recent method compares a perceptron, an artificial neural network (ANN), an ensemble of ANN’s and a SVM to classify between defective and functional solder joints (Luo et al., 2007). The input of these techniques consists of a subset from the 100 ˆ 44 ˆ 3 pixel image representation of the already segmented solder joints. This subset is determined by the Fisher information entropy calculated per pixel for a training set of labelled defective or functioning samples. In this example the ensemble of ANN’s performs best.

Instead of using raw pixel values, Mar et al. (2010) use a Gabor filter bank to extract features from solder joints. The type of solder joint (good, excess, less, no solder or bridge) is then identified using supervised learning.

A different type of quality inspection is that of semiconducting wafer surfaces. One method that is particularly interesting is discussed by Shankar and Zhong (2005), where test wafers are compared with a golden reference wafer. The pixel-wise absolute or squared difference between the test wafer and the reference wafer is calculated. This difference is then multiplied with a mask. This mask marks which regions are important (close to one) and which are not. The mask could mark the edges of the template, to indicate that differences around the edges are more important. Some examples are shown in Figure 2.1.

Pharmaceutical tablets are marked with a numeric or alphabetic code. These marks need to adhere to high standards in order to make sure they are readable and prevent mix-ups. Considering images of round tablets, a system was developed to register various correctly imprinted tablets to have the same angular orientation and create an eigenspace (Moˇzina et al., 2013). Three registration methods are discussed by Bukovec et al. (2007). Next the features are extracted from test samples by projecting them on this eigenspace. The advantage of such a system is that there is no need to know exactly what types

(17)

2.1. Compact survey 7

(a) reference (b) test sample 01 (c) test sample 02 (d) test sample 03

(e) mask (f) result 01 (g) result 02 (h) result 03

Figure 2.1:Reference, test and mask wafers for semiconductor quality inspection. Each sample is compared pixel-wise with the reference, these difference maps are multiplied with the mask, resulting in result 01-03. The mask has higher values near edges, indicating that differences at edges are more important. Image courtesy: Shankar and Zhong (2005)

of defects can occur, yet the projection can be seen as a measure of how compliant a test tablet is with what is considered correct.

Another interesting note is that this system uses a template model, and hence needs to register the image in order to correctly overlay geometric locations of test tablets with the model. This method of quality control is useful when the tested product does not have a homogeneous pattern to verify, but has a global structure which needs to be verified. In this case a local filter based approach is not adequate.

Agricultural products, colour based Agricultural products are typically not flat. On the other hand, photographic inspection of the surface of agricultural products is a non-intrusive way to determine their quality. Photographic inspection is effective and relevant, since consumers prefer products with a good appearance (Radman, 2005; Dransfield et al., 2005) but also consider other factors like costs and labels. Surface defects are of great influence to the price of fruits. Moreover, apples sold in Europe get official labels, like class I and class II as defined by Dransfield et al. (1989), depending on their quality.

In general farmers can benefit from agricultural visual quality inspection systems.

Contrary to many other industrial products, agricultural products vary a lot in size, shape and colour, even though they were harvested in a similar way (Cubero et al., 2011). Also the conditions during the processing of the fruit influences its characteristics; like humidity, temperature and presence of diseases. This needs to be taken into consideration when creating models to identify healthy and defect agricultural products.

Example defects to detect are bruises, russet, scab, fungi and wounds (Patel et al., 2012). Leemans et al. (1998) describe a method based on colour information, which quantifies the quality of Golden Delicious apples. They use a training set of 80 healthy apples. Based on these apples, the mean RGB

(18)

8 2. Related work

Figure 2.2:Schematics of the MIA system as discussed by L ´opez-Garc´ıa et al. (2010) and some example results, the middle row shows the ground truths and the bottom row shows the results of the MIA method. Image courtesy: L ´opez-Garc´ıa et al. (2010)

value of a pixel µ and the covariance matrix Σ is calculated. Then to classify whether a pixel belongs to healthy tissue or defect tissue, the Mahalanobis distance between the pixel RGB value and the Gaussian approximation using µ and Σ is calculated. Using this distance, the pixels are classified using a threshold, based on another 32 defective training apples. This method had trouble detecting defects with less contrast. At worst 10% of defects are, however, not detected.

Another similar method by Leemans et al. (1999) uses Bayesian statistics to classify pixels of Jon- agold apples as healthy or defective. The reason being that Jonagold apples have both greenish and reddish areas and thus colour distribution is not modelled appropriately by a Gaussian approximation, as was the case for Golden Delicious apples, which are uniformly greenish.

More recently Blasco et al. (2009) introduced a method that sorts pomegranate arils based on the average colour. The arils, together with peels from the pomegranate, passed over a conveyor belt. The conveyor belt was segmented from objects on it using a threshold based on the R channel of the RGB colour space. Then each object was classified via Bayesian statistics using the average colour and a labelled training set of 550 samples. Different classes of arils have different colours, as they vary colour between white-pink and red-brown. This method achieved an accuracy of 90%.

Agricultural products, texture based Although colour information may be sufficient, some defects are more prominently marked by their texture. Defining what texture features are, however, is not trivial.

This section discusses several techniques that applied texture-based features to detect defects.

The study by Zhu et al. (2007) applies several Gabor filters for different orientations and sizes to an apple image. Each image results in a set containing a two-dimensional signals for each filter. These sets are then classified to healthy or defect using either kernel PCA or kernel SVM. The Gabor-kernel based PCA method performs best with an accuracy of 90.5%

Another method (L ´opez-Garc´ıa et al., 2010) unfolds all the pixels of an orange, in order to create texture features. The unfolding process simply concatenates the RGB values of a pixel and its neighbours in a 3 ˆ 3 window or other odd square windows. This unfolding results in a feature-vector for each pixel. Based on a set of training images, PCA is used to create an eigenspace of such feature vectors representing texture. Test feature vectors are then projected upon this eigenspace to classify them as defective or healthy surface pixels. A schematic and some example results are shown in Figure 2.2. The method has an overall accuracy of 91.5%.

Du and Sun (2006) discuss a method which tries to estimate how much of the surface of ham consists of pores. Existence of pores is correlated to the quality of the meat. This research used a watershed method to segment the image. It is well-known that the watershed method over-segments images,

(19)

2.1. Compact survey 9

Table 2.1:Features applied for machine vision quality inspection.

Method Represents Usage

RGB colour colour Leemans et al. (1999, 1998)

Not-RGB colour colour equivalent to perception Fathi et al. (2011); Kang et al. (2008); Blasco et al. (2007) Unfolded neighbourhood adjacent colour correlation, texture L ´opez-Garc´ıa et al. (2010)

Gabor filter bank texture Tolba (2011); Mar et al. (2010); Zhu et al. (2007) Statistical aggregation mean, variance, covariance Wang and Bovik (2009)

Segments homogeneous regions Shafarenko et al. (1997); Du and Sun (2006)

due to local minima introduced by noise. This was tackled by merging adjacent segments which have similar texture or colour properties. Finally the porosity was estimated as the percentage of pore area with respect to the total ham area. Afterwards these results were used to correlate porosity with the actual quality of the ham.

2.1.2 Example overview

Surface defect detection often tries to segment the surface of an agricultural product in a healthy and a defect part and may try to classify the type of defect. Although colour may allow for pixel-wise detection of unhealthy skin (Leemans et al., 1998, 1999), some problems are harder since colour does not clearly separate healthy and defect surfaces. In such cases texture features can be used, popular ones are pixel-value correlations (L ´opez-Garc´ıa et al., 2010) and Gabor filter banks (Zhu et al., 2007).

These texture features are advantageous since they can be computed relatively fast, allowing for a fast industrial throughput. A summary of the encountered features is shown in Table 2.1. All the features are local, and represent colour, texture or homogeneous regions.

Table 2.2 summarizes the encountered classification techniques, they can have very different strategies. The main difference is that some methods need defective samples during training, whereas others can detect abnormalities as they are presented.

Most of the methods discussed in this chapter only used local features to inspect the quality of certain products. Many products however contain complicated structures, such that local features say little about the quality. In such cases it may be useful to use template based methods, as was done for semi-conducting wafers by Shankar and Zhong (2005).

Table 2.2:Classification methods applied for machine vision quality inspection.

Method Benefits Usage

PCA / Eigenspace No need to define what defects are, new abnormalities are detected when they are introduced

Moˇzina et al.

(2013); Zhu et al. (2007) Mahalanobis distance Results in a measure of how different a product is from a set of golden refer-

ences.

Leemans et al.

(1998) Bayesian classification Does not assume a Gaussian distribution or any distribution but also needs

defective training samples.

Leemans et al.

(1999) Thresholding Simple implementation, typically fast to compute, depending on the threshold

criterion

Mart´ınez et al.

(2013) SVM Allows for complicated and accurate decision boundaries, but needs defective

samples

Zhu et al. (2007) Probabilistic neural

network

Results in probabilities that a sample belongs to a certain correct or defect class.

Tolba (2011) Artificial neural

network

Automatically implements feature extraction and reduction. Bhatt and Pant (2013) Template correlation Allows for a definition of a products structural appearance to be tested. Shankar and

Zhong (2005) Image quality measures Measure human perception, especially useful for inspection of aesthetic prod-

ucts.

(20)

10 2. Related work

2.2 Perceptual quality measures

The inspection methods discussed in the previous chapter extract features locally from a neighbourhood of pixels. Two exceptions mentioned were the verification of pharmaceutical tablets (Moˇzina et al., 2013;

Bukovec et al., 2007) and semiconducting wafers (Shankar and Zhong, 2005). These researches applied some form of template matching, since a visual golden reference exists. The reason for this is simple:

in both cases we are not dealing with a homogeneous texture, but with a global structure. Hence local features say little about the quality of that region. For example, a mirrored homogeneous texture would still look similar, but a semiconducting wafer would look completely different. Similarly reordering letters on tablets would be a serious quality flaw. The same goes for the aforementioned Playmobil problem: the global pattern matters.

This chapter discusses several methods which are used to compare a template image with another, possibly distorted image. These are called perceptual image quality measures and typically result in a number between 0 and 1. In general they work as shown in Figure 2.3: both images are compared using a local distortion measure, e.g. the Euclidean distance between pixels. The resulting error map gives a pixel-wise estimate of how different the region around that pixel is. Then pooling is applied to merge this map into one number. An example could be simply summing the error map, or taking the average of the 5% worst values. Using Euclidean distance and averaging would be equivalent to the mean squared error.

The schema also shows registration as a preprocessing step of the local distortion measurements.

Not all image quality assessment methods apply registration, but it is used to tackle small geometric distortions. Since these methods are usually applied to assess image compression techniques or other digital image manipulations, global geometric distortions are not to be expected. The Playmobil parts in the test images are also very well aligned. Hence registration is not necessary for our measure.

Instead of naively taking Euclidean distances and summing, more complicated measures have been introduced to mimic human perception. The next section discusses some problems with the mean squared error.

2.2.1 Mean squared error

In image processing, fidelity measures are used to compare two images with each other, typically to assess how similar one test image is to a template. Such measures are often used in visual processing (Ghanbari, 2003; Netravali and Haskell, 1988), where for example the similarity between a compressed image and its original is determined (Wang et al., 2004).

A popular way to compare signals is by the use of a fidelity measure like the mean squared error (MSE). For any two signals x “ txi|i “ 1, 2, . . . , N uand y “ tyi|i “ 1, 2, . . . , N uit is defined as

Figure 2.3:General schema of a perceptional image quality assessment.

(21)

2.2. Perceptual quality measures 11

M SEpx, yq “ 1 N

N

ÿ

i“1

pxi´ yiq².

For example, x and y could be two images of the same size such that i iterates over all pixels and all colour channels. Then M SEpx, yq would measure how distant different pixels are on average. The MSE measure is popular for several reasons (Wang and Bovik, 2009):

• simplicity: the MSE has no parameters and allows for fast and online computation,

• valid distance metric: the MSE adheres to the identity, nonnegativity, symmetry and triangular inequality properties and can thus be interpreted as a distance metric,

• physical meaning: the MSE defines the energy of the error between both signals and this energy is preserved after orthogonal linear transformations like the Fourier transform,

• simple optimization: the MSE has the convexity, symmetry and differentiability properties, and closed form analytical solutions for MSE optimization often exist,

• models additive noise: for example if xi “ modelris ` noiseriswhere noiseris is Gaussian, and finally

• convention: historically most signal processing literature has used the MSE.

Unfortunately, the MSE does not match well with how the human visual system perceives difference in images (Girod, 1993; Wang et al., 2004; Wang and Bovik, 2009). Figure 2.4 shows some examples of distortions which impact human perception severely, yet have a relatively low MSE and vice-versa.

Especially spatial distortions like rotation and shift are barely noticeable by the human eye, yet have a relatively high MSE. The contrast shift barely changes human perception, since all pixels were altered equally. When these changes are not equal, but independently distributed in a Gaussian fashion, the visual perception changes drastically but the MSE not at all.

Reproducibility: illustrations/mse problems.m

original contr. stretch, 12 contrast shift, 21 additive noise, 21 salt & pepper, 24 blur, 42

downsample, 44 cw rotate, 46 ccw rotate, 46 zoomed, 213 shirt up, 200 shift down, 197 Figure 2.4:Distortions of the Lenna image and the mean squared error (ˆ10^´4) with the original.

(22)

12 2. Related work

Reproducibility: illustrations/mse problems.m

(a) original 1 (b) original 2 (c) original 1 + noise (d) original 2 + noise

Figure 2.5:Two images are shown in (a) and (b), 30% of image (b) is added to both (a) and (b), the results are respectively shown in (c) and (d). This simulates correlated noise. The Euclidean distance between (a) and (c) is about the same as between (b) and (d) (around 23 ˆ 10^´4). Image (d) is, however, brighter but appears similar to image (b). On the other hand, image (c) appears to be quite different than image (a). This shows that the Euclidean distance between images is not an effective perceptual measure.

The MSE also does not measure noise that is correlated to the image differently than noise that is not. Correlated noise may appear less intrusive than uncorrelated noise. This is illustrated in Figure 2.5. Secondly, certain regions may be more salient than others due to the existence of human faces or other cues of human perception. For example, removing the eyes of the woman in Figure 2.4 will have a heavier impact on perception than removing a significantly larger background object.

Although some applications in machine vision measure the quality of a printed or painted docu- ments with respect to a template (Vans et al., 2011; Torres et al., 1998), to the best of our knowledge, no research assesses the use of perceptual visual quality metrics for this purpose. Their use is especially useful for the assessment of the aforementioned Playmobil parts, where perception of quality is relevant.

2.2.2 Pooling methods

This section reviews some pooling strategies. Pooling combines a map which locally quantifies how defective a product is, into one response value, which quantifies the defectiveness of the product globally.

Wang and Li (2011) lists four, we assume that qimarks the error for pixel i “ 1 . . . N and that Q is the quality measure.

• Minkowski pooling: Qp“ _N¹ ř

iq_i^p,

• Local distortion based pooling: Q “

ř

iwpqiqqi

ř

iwpqiq , and

• Saliency based pooling: Q “ ^ř^řⁱ^wⁱ^qⁱ

iw_i

• Object based pooling: Local distortions are treated differently depending on salient object present in the image.

The first measure is a very common one, for p “ 1 it is the average absolute error, for p “ 2 the MSE and for p “ 8 is the maximum local distortion. In general a higher p-value means more attention for outliers. A more flexible approach is introducing a monotonically increasing weight function, which allows the user to indicate how much certain local distortion values distort the global image. The third method suggests the use of a saliency map, which indicates for each pixel how noticeable a distortion in that region would be. One could use edge, line or structure detection methods to indicate that such regions are more salient. As an example, Shankar and Zhong (2005) use edge detection to create such a

(23)

2.2. Perceptual quality measures 13

saliency map. Our philosophy is not to use saliency to compensate for allowed differences, but to select an adequate reference which has the same acceptable differences. This decreases the risk to ignore important differences, due to low saliency values. This is discussed in Section 3.3.

(24)

(25)

Chapter 3

Methods

Abstract

To increase the amount of training and test data, this chapter discusses two methods which deform acceptable samples into artificial samples with or without a defect. One deformation slightly moves regions using bilinear interpolation and a Gaussian random field to define relative local pixel movement, to simulate industrial production variance which is still acceptable. The other locates a point on an edge and deforms that edge more severely in the (counter)direction of the gradient. Subsequently, this chapter discusses the technical details of the structural similarity. Then, a vector space to approximate the set of golden references is provided. A projection of a test sample on this space should remove potential defects, but keep the acceptable deviations in tact. Comparing the sample with its projection instead of one golden reference is less sensitive to acceptable production variance.

Finally the SSIM values are pooled to be sensitive to regions with low similarity. These methods form a basis for the implementation and testing of the Playmobil quality inspection system.

3.1 Data generation

The Playmobil data set provided for the research of this thesis contains 64 sample images of the two toy figure types shown in Figure 1.3. Table 3.1 shows the distribution of images among defectiveness and toy figure type (see Figure 1.3). Three of the provided images were marked as defective by Playmobil.

To our judgement, they hardly show any noticeable defect and the defects are similar to the variance of the acceptable samples. Six samples were removed from the data set, three of which were labelled as acceptable but are also exactly the same as the other three which were labelled as defective. These six samples are shown in Figure 3.1b.

These numbers are quite low, and especially determining any statistics about the behaviour of the defective samples is not significant. Hence two deformation techniques were developed to simulate the

(a) falsely labelled (b) doubly labelled

Figure 3.1:(a) Three samples which are marked defective, however they seem acceptable. (b) Three samples which were labelled both as acceptable and defective. The samples in (a) are tested in the next chapter and included in Tab. 3.1 as hardly defective, but the samples in (b) are not tested and not included.

(26)

16 3. Methods

Table 3.1:Number of images in the original Playmobil set for two different types (see Fig. 1.3) and different defectiveness classes.

The labels F and G refer to the toy figure types of figure 1.3.

acceptable defective hardly defective Total

F 23 3 0 26

G 31 4 3 38

Total 54 7 3 64

effect of acceptable variance during production and variance which causes the sample to be defective.

Both are discussed in the next two sections.

3.1.1 Acceptable deformation

The original data set gives an idea of what variance is to be expected of acceptable examples. Figure 3.2 shows the largest standard deviation of the RGB-pixels among samples, this illustrates that high- variance areas coincide with edges. This suggests that edges may be slightly deformed among the samples. On the other hand, the displacement of one location on an edge is not independent of the displacement of a nearby location. That is, the edges are not getting whimsical or jagged due to acceptable production variance. A probable explanation for moved edges is that different colours are printed independently and hence their relative placement may differ slightly across samples.

A way to accomplish such a deformation computationally is by using two Gaussian random fields, which respectively define the displacement in pixels for the x and y directions. Even though the pixels in these random fields are correlated, they are normally distributed. For this reason, the iso-lines of the distribution of a pixel displacement p∆x, ∆yq are circular. This prevents blocking or other artefacts, since there is no bias for higher displacement in certain directions.

Normally distributed values are not necessarily integer. We use bilinear interpolation to cope with non-integer coordinates without introducing rounding artefacts other than those caused by the preci- sion of floating point numbers. Two such random fields are shown in Figure 3.3a.

Deformation process

The process of creating such Gaussian fields, the accomplished deformation and the deformations shown as a grid are shown in Figures 3.3, 3.4 and 3.5, respectively. This section elaborates on how these Gaussian fields are created and then used to deform.

Reproducibility: illustrations/illustrate deviation.m

(a) sample

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2

(b) st. dev. (c) sample

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16

(d) st. dev.

Figure 3.2:Per-pixel standard deviation of the RGB colours among (a) 23 samples of toy figure type F and and (b) 31 samples of toy figure type G. The maximum of the standard deviations of the three channels is shown. All colour values are normalized to be in r0, 1s instead of r0 . . . 255s, but are not constrast stretched or shifted.

(27)

3.1. Data generation 17

Defining pixel movement via such a random field also changes the boundary between the sample and its background, Figure 3.3a confirms this with non-zero values at this boundary. This is the equivalent of a shape change of the toy figure. Examples of such deformation are shown in Figures 3.4bc.

To tackle this effect, the field is modified to have value 0 outside the sample’s boundary. Never- theless, this introduces an abrupt transition between the non-zero and zero values at each side of the boundary. Therefore we expanded the region of zero values by 15 pixels inside the sample after which we applied a Gaussian blur with σ “ 5. The border effect is smoothly spread over 15 pixels, effectively removing it. The blur removes the abrupt transition and the expansion ensures that this blur does not move significantly large non-zero values outside of the sample’s boundary. These two operations result in the random fields of Figures 3.3bc.

Considering an input image the process of creating such a random field is summarized as:

1. Start: Create a field with independent identically distributed values according to a Gaussian dis- tribution with standard deviation 1 of the same dimensions of the image.

2. Introduce geometric correlation: Blur the field with a Gaussian kernel of large standard deviation σc, for example 150. Example result: Figure 3.3a.

3. Stabilize boundary: Set all field values outside or 15 pixels near the boundary to 0. Example result: Figure 3.3b.

4. Remove boundary artefact: Blur the field with a Gaussian kernel of standard deviation 5. Exam- ple result: Figure 3.3c.

5. Adjust to expected pixel displacement: Rescale the field values to standard deviation σd. The above steps need to be executed twice, to create fields for displacement of both directions. The parameter σd determines how far pixels are moved. Furthermore σc was introduced in the process.

It represents the radius in which pixels displacements are still correlated. Theoretically this radius is infinite due to the convolution of the second step. If the Gaussian kernel had infinite size, each field value before the convolution would be embedded in each field value after the convolution. Because the Gaussian kernel is already relatively weak after 1.5 standard deviation, it is fair to say correlation is barely noticeable between pixels 1.5σc pixels apart. A large correlation radius like 150 pixels creates smooth movements, however it also requires an expensive convolution with a large kernel. Therefore it is calculated relatively fast via the Fourier trick, that is F ˚ K “ F^´1pF pF q ˆ F pKqq, where F denotes the Fourier transform, ˆ element wise multiplication and ˚ convolution.

Finally, the deformed image I¹ is calculated via bilinear interpolation, that is I¹px, yq “ Ipx `

∆xpx, yq, y ` ∆ypx, yqq, where I¹ is the deformed image, I the original image and ∆x and ∆y are the two fields. Some results of such deformations are shown in Figure 3.6. In order to compare this result with the real Playmobil data, the per-pixel standard deviation among 50 artificial samples is shown in Figure 3.7. The figure shows similar variance patterns as those of Figure 3.2.

3.1.2 Defective deformation

The deformation of the previous section is hardly visible, as can be observed from Figure 3.6. In order to also create samples which have perceptually visual deformations in them, this section discusses a deformation on top of the previous one. The deformation of the previous section is applied first. This is important, since the defects created in this section are local and hence not doing so introduces a bias for the defective sample as they will not have variation elsewhere.

(28)

18 3. Methods

Reproducibility: illustrations/illustrate acceptable deformation.m

−50

−40

−30

−20

−10 0 10 20 30 40 50

−50

−40

−30

−20

−10 0 10 20 30 40 50

(a) zwo fixed boundary

−50

−40

−30

−20

−10 0 10 20 30 40 50

−50

−40

−30

−20

−10 0 10 20 30 40 50

(b) zw fixed boundary

−50

−40

−30

−20

−10 0 10 20 30 40 50

−50

−40

−30

−20

−10 0 10 20 30 40 50

(c) zw fixed, smoothed boundary

Figure 3.3:Several random fields that denote the amount a pixel at a specific location is moved in the x- and y-direction, respectively for the left and the right image. In (a) a random field is shown, with non-zero values on the boundary. In (b) a region of the field is set to 0 to remove movements around the boundary, in (c) the field is also blurred to remove boundary effects.

(a) original (b) x-direction (c) both directions (d) fixed boundary (e) fixed, smooth boundary

Figure 3.4:Deformations according to Gaussian fields: (a) illustrates the original. In (b) only the x-direction is deformed, while in (c) both directions are. The red outline indicates the original location of the sample’s boundary. In (d) the deformation keeps the boundary fixed via the fields of Fig. 3.3b. This introduces a boundary artefact, in this case part of the brown, bottom-right pocket outline is removed. In (e) the fields of Fig. 3.4c are used and the boundary artefact is gone due to the smooth transition of the fields near the boundary.

(a) none (b) x-direction (c) both directions (d) zw fixed

boundary

(e) zw smoothed, fixed boundary

Figure 3.5:The deformations of Fig. 3.4 shown as a grid.

Reproducibility: illustrations/example acceptable deformations.m

Figure 3.6:Examples of deformations via a Gaussian random field with σd“ 2and σc“ 150. The forelast sample has σd“ 20 and illustrates an exaggeration of the effect. The last sample has σc “ 2, to illustrate the whimsical edges produced by low pixel-deplacement correlation. Notice that the difference between the first 3 samples is barely visible or not even noticeable at all.

(29)

3.1. Data generation 19

Reproducibility: illustrations/illustrate deviation.m

(a) sample

0 0.05 0.1 0.15

(b) st. dev. (c) sample

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14

(d) st. dev.

Figure 3.7:Per-pixel standard deviation of the RGB colours among 50 artificial samples without a defect. The maximum of the standard deviation of the three channels is shown. All colour values are normalized to be in r0, 1s instead of r0 . . . 255s, but are not constrast stretched or shifted. The artificial samples were created with σd“ 2and σc“ 150.

The local deformation is created on and near an edge, where the color of one side of the edge is wiped to the other side of the edge. This way, no new colours are introduced locally, yet the effect is perceptually visible. This is in line with the defects shown in Figure 1.4, except for the third one, where white is introduced. Especially the second defect is of this type. We presume that defects which introduce new colours can be detected easier, and hence are less relevant for research. This use case is not created by this deformation. Summarized the aforementioned wipes are created by these steps.

Illustrations of the steps are shown in Figure 3.8.

1. Start: Given an input image I, the wipe length l and the wipe width w:

2. Locate edges: Apply the Sobel filters on a blurred I for both directions to I to get Ixand Iy, and also calculate the gradient magnitude G “ Ix²` I_y².

3. Delete weak edges: Set G’s values which are below fµ¨ Gto 0, where G is the mean of G and fµ

is a factor around 1.1.

4. Defect location: Pick a pixel px, yq from G with probability^Gpx,yq_ΣG . (Fig. 3.8a)

5. Defect direction: Determine the gradient p∆x, ∆yq at px, yq and with probability 0.5 rotate it to have opposite direction and normalize it to length 1 in any case. (Fig. 3.8a)

6. Defect colour: Determine the colour c at px, yq ´ 10p∆x, ∆yq. That is, select the colour at one side of the edge. (Fig. 3.8b)

edge

+gradient -

(a)

colour opposite side

(b)

width length

(c) (d)

Figure 3.8:Schematic illustrations of how a perceptually visible deformation is created by wiping out an edge. First a point on an edge is selected (a), then using the colour on the opposite side of the edge (b) a wipe is created in the (counter)direction of the gradient (c). This results in a deformed edge (d).

(30)

20 3. Methods

Reproducibility: illustrations/example defective deformations.m

Figure 3.9:Examples of deformations by wiping an edge with uniformly distributed l P r80, 120s and w P r10, 30s. The red circles are not part of the deformation but mark the deformation’s location. The fourth defect shows the interruption of a line.

7. Draw defect: Change the colour to c at locations px, yq ` l ¨ l¹¨ p∆x, ∆yq ` w ¨ w¹¨ l¹¨ p´∆y, ∆xqfor l¹P r0, 1sand w¹P r´1, 1s. That is, draw a line of length l starting at px, yq in the direction p∆x, ∆yq which starts with width w and linearly decreases to width 0. (Fig. 3.8c)

Several examples of such deformations are shown in Figure 3.9. Sometimes the edge chosen to deform, actually is part of a line. Perceptually this interrupts the line as a result of deforming the line’s edge at one side. Line interruptions are perceptually salient. This is shown in the fourth sample.

It is fairly arguable that the first sample’s defect is barely visible and hence this deformation does not necessarily create perceptual defects. If the deformation mostly overlaps a region where the structure contains the colour of the wipe, then the wipe might not be very salient.

A possible way to tackle this might be to measure whether the wipe changes the colours sufficiently, for example using the Euclidean distance. This would, however, bias the data set such that defects need to have some minimal Euclidean distance or other measure, which may not be in line with the human perceptual system. Ultimately this may result in an unjust advantage for certain defect detection systems and hence such filtering is avoided.

Subsequently, one may consider introducing multiple of such detects. Multiple defects increase the opportunity of a detection algorithm to find at least one and thereupon will increase the performance.

This way defects that are hard to detect may hide behind other occurrences of easier defects. Therefore our artificially defective samples contain one wipe.

3.2 Structural Similarity

This section discusses a popular image quality measure: structural similarity (SSIM), introduced by Wang et al. (2004). Contrary to Euclidean distance, this measure takes into account that structural changes are more important than just changes in luminance or contrast. SSIM assumes a greyscale image and is based on a sliding window of which statistical aggregations are combined in one metric between ´1 and 1. When proposed, the window size was suggested to be 11 ˆ 11 in size. The statistics computed for two windows ~p “ pp1, . . . , pNqand ~q “ pq1, . . . , qNq, where N “ 11 ¨ 11 “ 121, are mean, variance and covariance:

(31)

3.2. Structural Similarity 21

µp“

N

ÿ

i“1

wi¨ pi, (3.1)

σ_p“

˜_N ÿ

i“1

w_i¨ ppi´ µpq²

¸1{2

and (3.2)

σ_pq“

N

ÿ

i“1

w_i¨ ppi´ µpq ¨ pqi´ µqq. (3.3)

Note that µq and σq were not mentioned, but are calculated similarly. Moreover, a weighting ~w “ pw1, . . . , wNqis used to prevent blocking artefacts due to rectangular shape of the window. It defines a 2D Gaussian distribution, Wang et al. (2004) suggest its standard deviation to be 1.5. The Gaussian shape ensures that each pixel in each direction in the window is (approximately) weighed the same.

The luminance and contrast of two images can be compared using the means and variances respectively. Structure is compared using the covariance. They are transformed into a number in r0, 1s or r´1, 1sdenoting their similarity.

lp~p, ~q|C1q “ 2µ_pµ_q` C1

µ²_p` µ²_q` C1

P r0, 1s, (3.4)

cp~p, ~q|C2q “ 2σ_pσ_q` C2

σ_p²` σ_q²` C2

P r0, 1s, (3.5)

sp~p, ~q|C3q “ σpq` C3

σpσq` C3 P r´1, 1s. (3.6)

The constants 0 ă C1,2,3 ! 1stabilize the measures for denominators otherwise near or equal to 0. There are two additional notes: in theory it is also possible that lp~p, ~q|C1q P r´1, 0q, however this only occurs when ~por ~q contain negative values. This is not the case for images in the RGB colour space, but may happen for Lab images. In that case it is important to scale l to r0, 1s to ensure that high SSIM values indicate high perceptual similarity. Subsequently, via the Cauchy- Schwarz inequality,

|σpq| ď σpσq, which ensures that sp~p, ~q|C3q P r´1, 1s. Finally the three values are merged into one SSIM value:

SSIM p~p, ~q|α, β, γq “ lp~p, ~qq^αcp~p, ~qq^βsp~p, ~qq^γ (3.7) The parameters α, β and γ adjust the relative importance of the three components. There are some practical arguments to set them equal to 1, one being that non-integer values can not deal with negative values of s. Another is that when also C3“ C2{2, SSIM simplifies to:

SSIM p~p, ~q|1, 1, 1q “ p2µpµ_q` C1qp2σpq` C2q

pµ_p²` µ²_q` C1qpσ_p²` σ_q²` C2q (3.8) In order to combine the SSIM measures of three RGB channels, I chose to select the minimum value.

This corresponds to stating that a pixel is as defective as the least similar channel or that a pixel is defective if it is defective in at least one channel. A similar argument holds for the Euclidean distance, where the maximum distance of the channels is chosen.

(32)

22 3. Methods

Ultimately, the SSIM measure can be compared with a high school exam. When a student fails the first question, he or she can still answer the second or third question correctly, regardless of the first answer, and get a decent score. If the similarity between luminance is low, the sample can still score high on contrast, since contrast is corrected for with luminance. Similarly, if the contrast does not score high, the structure can still score high, as it is corrected for by both luminance and contrast. In conclusion, two regions can be similar, even though the Euclidean distance is high.

3.2.1 Examples

Figure 3.11 shows several example Euclidean and SSIM maps, they indicate the Euclidean distance or SSIM value for each pixel. During this section, a 15 ˆ 15 kernel is used with σssim “ 5. If the SSIM measure (Euclidean distance) performs well, the values at defective regions should be as low (high) as possible while maintaining high similarity (low distances) at the acceptable regions. The next section discusses methods to improve these maps.

The distribution of SSIM values and Euclidean distances for both defective and acceptable pixels are shown in Figure 3.13. For the distribution, each acceptable sample from Table 3.1 was compared with each defective sample of the same type from that table. Notice that the Euclidean distance measures distance, whereas the SSIM measures similarity, hence low and high values have opposite meanings for both.

Both distributions show that defective pixels can still produce high similarities. One possible reason is that not all manually marked pixels may be defective. Another possible reason is that the measures may not always be right. The former can be tested with some slack for the measures by setting each SSIM value (Euclidean distance) to the minimum (maximum) of a 3-pixel radius neighbourhood. The distributions for this particular case are shown in Figure 3.14.

This slack causes some SSIM values (Euclidean distances) with values near 1 (0) to decrease (increase), without significantly changing the distribution of the acceptable pixels. This implies that defective pixels with high (low) SSIM values (Euclidean distances) are often near pixels with low (high) values (distances).

Nevertheless, the SSIM distribution shows that the defective distribution contains more relatively low values than the Euclidean distribution. This suggests that the SSIM has better discriminating power between defective and acceptable samples. The distributions of the separate luminance, contrast and structure components are shown in Figure 3.15. They are also shown in order of discriminative power:

luminance, contrast and variance. The separate components have, however, less discriminative power than combines as SSIM.

The examples of Figure 3.11 are based on a golden reference which align well with the defective sample. Not each golden reference sample matches, however, the non-defective regions of a test sample well, especially near the edges where a lot of acceptable production variance is expected. Figure 3.12 shows example Euclidean and SSIM maps where the golden reference is chosen to illustrate this.

Note that all golden references, including the one used here, are included in the distributions shown in

defects1-3 defects4-6

Figure 3.10:Six defective samples used to illustrate proposed defect detection methods.

(33)

3.2. Structural Similarity 23

Reproducibility: illustrations/example ssims.m

defect1

0 0.05 0.1 0.15 0.2

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

defect4

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35

0.4 0.5 0.6 0.7 0.8 0.9 1

defect2

0 0.05 0.1 0.15 0.2 0.25 0.3

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

defect5

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2

0.4 0.5 0.6 0.7 0.8 0.9 1

defect3

0 0.05 0.1 0.15 0.2 0.25 0.3

Euclidean distance

0.4 0.5 0.6 0.7 0.8 0.9 1

SSIM

defect6

0 0.05 0.1 0.15

Euclidean distance

0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1

SSIM

Figure 3.11:Structural similarity and Euclidean distance between an acceptable sample and the defective samples of Fig. 3.10.

For each pixel the minimum (maximum) value of the SSIM (Euclidean distance) of the RGB channels is shown. Golden references are selected manually to align well with the defective sample.

defect1

0 0.05 0.1 0.15 0.2 0.25 0.3

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

defect4

0 0.05 0.1 0.15 0.2 0.25 0.3

0.4 0.5 0.6 0.7 0.8 0.9 1

defect2

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35

−0.2 0 0.2 0.4 0.6 0.8

defect5

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35

0.4 0.5 0.6 0.7 0.8 0.9 1

defect3

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

Euclidean distance

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

SSIM

defect6

0 0.05 0.1 0.15 0.2 0.25

Euclidean distance

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

SSIM

Figure 3.12: Similar to Fig. 3.11, however the test samples are compared with a golden reference manually selected to align poorly. This is expressed by the low (high) SSIM values (distances) near the edges.

PERCEPTUAL INDUSTRIAL SURFACE QUALITY INSPECTION