This m anuscript h as been reproduced from the microfilm master. UMI films the text directly from the original or copy submitted. Thus, som e thesis and dissertation copies are in typewriter face, while others may be from any type of com puter printer.
The quality o f th is reproduction is d e p en d e n t u p o n th e quality of th e c o p y su b m itte d . Broken or indistinct print, colored or poor quality illustrations and photographs, print bleedthrough, substandard margins, and improper alignment can adversely affect reproduction.
In the unlikely event that the author did not send UMI a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to t>e removed, a note will indicate the deletion.
Oversize materials (e.g., maps, drawings, charts) are reproduced by sectioning the original, t>eginning at the upper left-hand com er and continuing from left to right in equal sections with small overlaps.
Photographs included in the original manuscript have t)een reproduced xerographically In this copy. Higher quality 6” x 9 ' black and white photographic prints are available for any photographs or illustrations appearing in this copy for an additional charge. Contact UMI directly to order.
Bell & Howell Information and Learning
300 North Zeeb Road, Ann Arbor, Ml 48106-1346 USA 800-521-0600
by
Jason Leslie Szabo
B.Sc. Engineering. Université o f Calgaiy. 1979 M.Eng.. University o f Calgaiy, 1989
A Dissertation Submitted in Partial Fulfillment o f the Requirements for the Degree o f
DOCTOR OF PHILOSOPHY
in the Department o f Mechanical Engineering
We accept this dissertation as conforming to the required standard
D r^^^ F. McLean. Sbper\ isor (Department o f Mechanical Engineering)
Dr. A. G. Doige. Departmental Member ( Department o f Mechanical Engineering)
Dr. 1. Sharf. Depanmental M ember (Department o f Mechanical Engineering)
Dr. W. D. Little.
DiYR. D. Donv. E;
em ber (Department o f Electrical and Com puter Engineering)
al Examiner (School o f Engineering. University o f Guelph) © Jason Leslie Szabo. 2000
University o f Victoria
All rights reserved. This dissertation may not be reproduced in whole or in part, by photocopy or other means, without the permission o f the author.
ABSTR^\CT
This dissertation presents original computer vision algorithms to automate the
identification o f piping and photogrammetric piping features in individual digital images o f industrial installations.
Automatic identification o f the pixel regions associated with piping is the core original element o f this w ork and is accomplished through a re-representation o f image
information (light intensity versus position) in a light intensity versus gradient orientation data space. This work is based on the physics o f scene illumination/reflectance and e\ aluates pixel regions in a hierarchy o f data abstractions to identify pipe regions without needing specific information about pipe edges, illumination, or reflectance characteristics. The synthesis o f correlated information used in this image segmentation algorithm
pro\ ides a robust technique to identify potential pipe pixel regions in real images.
.An additional unique element o f this work is a pipe edge identification methodology, w hich uses the information from this light intensity versus gradient orientation data space to localize the pipe edge search space (in both a pixel position and gradient orientation sense). This localization provides a very specific, perspective independent, self-adaptive pipe edge filter. Pipe edges identified in this m anner are then incorporated into a robust region-joining algorithm to address the issue o f region fragmentation (caused by
A utom ated photogrammetric feature identification is also demonstrated through
algorithmically recognizing the intersection o f orthogonal pipe sections (with piping code acceptable diameter ratios) as potential T-junctions or 90-degree elbows. As pipe
intersections, these image points are located with sub pixel resolution even though the\ cannot be identified by simple inspection.
The computer vision algorithms o f this dissertation are robust physics based methods, applicable to the identification o f piping and photogrammetric pipe features in real w orld images o f industrial installations, being: perspective independent, albedo independent, and unaffected by inter-reflections. Automating these operator driven input tasks will improve the accuracy, ease-of-use. and cost effectiveness o f implementing existing photogrammetric programs to the significant industrial problem o f generating as-built piping drawings.
Examiners:
Dr. G ,J^/\1cLean. S u p ei^ so r (Department o f Mechanical Engineering)
Dr. A. G. Di ental Member (Department o f Mechanical Engineering)
Dr. I. Sharf. D e p a rtm e n t Member (Department o f Mechanical Engineering)
Dr. W. D. Little. Qtitside Member (Department o f Electrical and Com puter Engineering)
A b stra c t...ii Table o f C o n te n ts...v List o f T a b le s ... ix List o f F ig u re s... x List o f S y m b o ls...xv 1. Introduction... 1
1.1. Limitation o f Existing Photogrammetric M ethods... 2
1.2. Examples o f Existing Photogrammetric M ethods... 4
1.3. Scope o f Research...6
1.4. Presentation Block Diagram ...8
1.5. Dissertation O verview ...9
1.6. On 'Proving' Computer Vision .Algorithm... 10
2. Automated Pipe Finding - Literature R eview ... 15
2.1. 3D Range Data... 15
2.2. Multiple C am eras...17
2.3. Single Camera with Controlled lighting... 18
2.4. Single Camera with Unconstrained Lighting...19
2.4.1. Pipe Edges... 19
2.4.2. Pipe Ellipses... 23
2.4.3. Surface Reflectance M easurem ents... 24
2.5. Why is this difficu lt?...26
3. Imaging o f Industrial P ip in g ... 30
3.1. Simple Image Form ation... 30
3.1.1. Lambertian Shading Model...32
3.1.2. Planar Surfaces in Cross-section... 33
3.1.3. Cylindrical Surfaces in Cross-section... 33
3 .1.4. Edges in Cross-section... 35
3.2. Digital Image Formation - Quantization...36
3.3. Pseudo Im ages... 38
3.3.1. 3D Piping Model - Coordinate T ransform s... 39
3.3.2. Perspective... 42
3.3.3. S h a d in g ...44
3.4. Cam era Calibration and Resolution... 47
3.5. O bservations... 50
4. Pipe Region Segmentation - Physical B asis... 52
4.1. Light Intensity Ridge - Point / Line Light Source... 56
4.2. Gradient Magnitude and O rientation...58
4.3. n Pipe T e st... 63
5. Pipe Region Segmentation - Implementation... 65
5.1. Image Preprocessing - Color... 66
5.2. Image Preprocessing - Sm oothing... 67
5.3. Light Intensity R id g e...68
5.4. Intensit) Ridge Pixel G roups... 71
5.5. Linear Extent o f Similar Intensity... 73
5.6. Light Ridge R egions...77
5.7. n Pipe T e s t... 78
5.8. Sensitivity A nalysis...87
5.8.1. Sm oothing Radius...88
5.8.3. Delta Gradient Orientation... 90
5.8.4. Weighted Standard Deviation M ultiplier...92
5.8.5. Resolution...93
5.8.6. Rotation... 94
5.8.7. N oise...95
5.9. Conclusion...96
6. Pipe Region Edge Identification... 98
6.1. Description o f Edges... 99
6.1.1. Two Possible Edge Regions... 99
6 .1.2. Two Possible Gradient O rien tatio n s...102
6.1.3. Multiple Possible Edge L in e s ... 104
6.1.4. Extending Pipe Bounding Edges...108
6.1.5. Pipe Segment Identification...112
6.2. Joining Pipe Segm ents...113
6.2.1. Extended Region M ask...115
6.2.2. Merged Region Hypothesis T estin g... 116
6.3. Results and C onclusion ...120
7. .Automatic Feature Extraction... 129
7.1. Vanishing Point M erging...129
7.2. T-Junctions and Elbows... 137
7.3. Test Results...141
7.4. Conclusion...148
8. Single Image 3D Piping M odel... 149
8.1. Developing a World Coordinate F ram e w o rk ... 150
8.1.1. T-Junctions...153
8.1.2. Parallel Pipes (Pipe R acks)...157
8.2.1. Pseudo Image T-junctions... 158
8.2.2. Real Image T-junctions... 161
8.3. Sensitivity...163
8.4. Discussion...164
9. Recommendations for Future Work and C onclusions... 166
9.1. Future W ork... 166
9.1.1. Pipe Region Identification... 166
9.1.2. Edge Detection and Region Joining...167
9.1.3. Photogrammetric Pipe Features... 167
9.1.4. Make it fa ste r...168
9.2. C onclusions... 169
R eferences...170
.Appendix A - Canny Edge D etector... 177
List o f Tables
Table 3-1 Camera C alibrations... 48
Table 3-2 Horizontal Camera Resolutions...49
Table 3-3 Radial Cam era R esolutions...50
Table 6-1 Results o f Pipe F in d in g ... 127
Table 7-1 Dimensions o f Reducing T-Junctions. T-Junctions, and Elbow s...138
Table 7-2 Results o f Photogrammetric Pipe Feature Identification...146
Table 7-3 Pipe Feature Identification Measurement Error (160 x 120 p ix e ls)... 147
Table 7-4 Pipe Feature Identification Measurement Error (640 x 480 p ix e ls)... 147
Table 8-1 T-Junction 3D Location - Pseudo Im age...160
L ist o f Figures
Figure 1-1 Two Photograph Viewpoints o f a Box and Pipe (Plan V iew)... 3
Figure 1-2 Image as Pixel Based Light Intensity M easurem ents...7
Figure 1-3 Image Processing Accuracy Check Block D iagram ... 8
Figure 1-4 Sample Pseudo Im ages...11
Figure 1-5 Sample Uncalibrated Camera Image A ...12
Figure 1-6 Sample Uncalibrated Camera Image B ...13
Figure 1-7 Sam ple Calibrated Camera Image A ... 13
Figure 1-8 Sample Calibrated Camera Image B ... 14
Figure 2-1 Pipe Images (pseudo and real)... 20
Figure 3-1 Image F orm ation... 31
Figure 3-2 Lambert's L aw ... 32
Figure 3-3 Lambertian Box G ird e r... 33
Figure 3-4 Lambertian P ip e ...34
Figure 3-5 Lambertian Edges - Pipe and B o x ... 35
Figure 3-6 Lambertian Edges - Two P ip e s ...36
Figure 3-7 Idealized Pipe Im a g e ...37
Figure 3-8 Pipe with Rotation and Perspective... 37
Figure 3-9 Location o f Cam era in Pipe Coordinate S y stem ... 40
Figure 3-10 View from Cam era Centered Coordinate S y stem ...42
Figure 3-12 Pseudo Image T-Junction (facet, resolution, and lighting variations)... 47
Figure 3-13 Single Pixel - Horizontal A ccuracy... 48
Figure 3-14 Single Pixel - Radial R esolution... 49
Figure 4-1 Sam ple Images (pseudo and real)...53
Figure 4-2 Sam ple Images as 3D Light Intensity Surfaces... 54
Figure 4-3 Contours o f Uniform Light Intensity... 55
Figure 4-4 Light Intensity Contours (vertical cross-sections)...57
Figure 4-5 G aussian Smoothing Kernel...59
Figure 4-6 Smoothed Im ages... 60
Figure 4-7 G radient Magnitude Im ages...61
Figure 4-8 Gradient Orientation Im ages...62
Figure 4-9 Gradient Magnitude and Orientation (vertical cross-sections)... 63
Figure 5-1 C olor to Grayscale M ethodologies... 66
Figure 5-2 Collection o f Vertical Section Local M aximums and M inim um s...70
Figure 5-3 Pixel Grouping Algorithm... 72
Figure 5-4 Pixel Grouping at Various Line A n g les... 72
Figure 5-5 Ridge Pixel R egions... 73
Figure 5-6 Linear Ridge Pixel Regions... 77
Figure 5-7 Intensity Ridge Regions Mapped onto Im ages... 78
Figure 5-8 G radient Magnitude and Orientation (vertical cross-sections)... 79
Figure 5-9 Light Intensity (vertical cross-sections)... 80
Figure 5-11 Smoothed Intensity vs. Gradient Orientation (accum ulation)... 85
Figure 5-12 Pipe Hypothesis Regions Mapped onto Im ages... 87
Figure 5-13 Smoothing Radius Variations... 88
Figure 5-14 Minimum Region Length V ariations... 90
Figure 5-15 Acceptable Gradient Orientation V ariations... 91
Figure 5-16 Weighted STD Multiplier Variations... 92
Figure 5-17 Image Resolution Variation...93
Figure 5-18 Rotation o f P ipes... 94
Figure 5-19 Image Noise - Median Filter... 95
Figure 6-1 Edge Pixels... 98
Figure 6-2 Pipe Edge Search Area (vertical cross-section)...100
Figure 6-3 Pipe Edge Search A re a s...102
Figure 6-4 Gradient Orientations o f Edge Pixels...103
Figure 6-5 Potential Edge Lines... 104
Figure 6-6 Edge Extending M ask ...109
Figure 6-7 Identified Extensions o f Edges...111
Figure 6-8 Pipe Segment E dges... 113
Figure 6-9 Extended Region M ask ...116
Figure 6-10 Hypothetical Merged R eg io n ...117
Figure 6-11 Hypothetical Merged Region - E dges... 118
Figure 6-12 Hypothetical Merged Region - Extended Edge M ask... 118
Figure 6-13 Hypothetical Merged Region - Extended E dge...119
Figure 6-15 Sample Pseudo Image - Found P ip es... 121
Figure 6-16 Sample Images - Uncalibrated A - Found P ip e s...122
Figure 6-17 Sample Images - Uncalibrated B - Found Pipes...123
Figure 6-18 Sample Images - Calibrated A - Found P ip es... 125
Figure 6-19 Sample Images - Calibrated B - Found Pipes... 126
Figure 7-1 Found Pipes (pseudo and re a l)...129
Figure 7-2 Vanishing Point Iterations (shifted edges pseudo im ag e)... 135
Figure 7-3 Vanishing Point Iterations (shifted edges real im age)... 136
Figure 7-4 Pseudo Image at Two R esolutions... 142
Figure 7-5 Photogrammetric Pipe Features - Calibrated Camera Image A ...143
Figure 7-6 Photogrammetric Pipe Features - Calibrated Camera Image B ... 145
Figure 8-1 Location o f Cam era in Pipe Coordinate System ... 151
Figure 8-2 Pipe Section Along Viewing A x is ... 152
Figure 8-3 Pipe R acks...157
Figure 8-4 Ellipse Center vs. Pipe C enter...163
Figure A-1 Edge as Step Change o f Light Intensity... 177
Figure A-2 Second Derivative o f E dge... 178
Figure A-3 Circular Directional Second Derivative M ask ... 179
Figure A-4 Five Lobed Directional Second Derivative M a sk ...180
Figure A-5 Five Lobed M agnitude o f Circular First Derivative M ask ...180
Figure A-7 Test Image and Smoothed Test Image... 183 Figure A-8 Directional Second Derivative Results... 1 83 Figure A-9 M agnitude o f Directional First Derivatives... 184 Figure A -10 Canny E dges... 184
List o f Symbols
Coordinate systems
ix. iy image plane (ix positive right, iy positive up)
n, m Gaussian sphere (unit sphere positioned at camera focal point)
ix COS0
m = iy n = s in ^
f P
m a normalized vector describing a point on the image plane
n a normalized vector describing a line ( p = ix c o sû + n sin 0 ) on the image plane
Xc.Yc. Zc camera centered world coordinate system (Xc II ix. Yc II iy. Zc by right hand rule)
Xp. Yp. Zp pipe centered world coordinate system
(Xp positive right. Yp positive away from camera. Zp by right hand rule)
Variable values and thresholds
n, radius o f Gaussian smoothing kernel { 3 .5 .7 .9 .1 1 .1 3 .... J C7(i Standard deviation o f Gaussian smoothing kernel
[constant ratio in this work CTq = ru/2 { T\, minimum acceptable normalized magnitude
Tp minimum intensity difference between local peaks and valleys Tr minimum number o f pixels in peak region
Ti minimum length o f ridgeline and H region ô([) plus and minus width o f legs o f FT test n„ weighted standard deviation multiplier
Tps minimum edge strength ratio for picking best top and bottom pipe edges Tmrp minimum mid region pixel ratio for merging FI regions
To maximum acceptable deviation o f edge line angles for vanishing point calculation
Td minimum acceptable deviation o f edge lines from vanishing point Trpo maximum accepted difference from ideal ratio o f pipe diameters To maximum accepted orthogonality threshold
Tcra maximum accepted difference o f cam era rotation angle
Two-dimensional convolution cell
ii - I-re- - . - ro} cell width jj = J-tG-. - - ro* cell height g( ii. jj ) normalized Gaussian sm oothing operator
Two-dimensional data arrays
i = {1.2.3---ncoium ns j image width j = {1.2.3--- nro«s! image height I( i. j ) input grayscale intensity image
{0.1.2.3.. . . 255} light intensity measurements S(i. j ) smoothed grayscale intensity image
(0.0 255.0) light intensity measurements
m( i. j ) magnitude o f gradient o f smoothed grayscale intensity image above threshold T\,
(0.0 —> 1.0) normalized gradient magnitude (|)(i. j) orientation o f gradient o f smoothed grayscale
intensity image (magnitude > T\, )
(0.0 —► 1.0) normalized gradient orientation (1.0 = 360°) pv(i. j) local peaks and valleys o f smoothed grayscale intensity image
{+1.0.-1} +1 p e a k s.-1 valleys r(i. j) connected pixels o f ridge regions pv(i. j) = +1
Region specific two-dimensional data arrays
i = {1.2.3 r i c o i u m n s ! image width
j = {1.2.3.. . . nro«sl image height 7tr( i-j ) pixels describing potential pipe o f ridge region r
{0.1.2.3} background, top. middle, and bottom
E R M r ( i . j ) pixels describing edge region masks o f potential pipe o f ridge region r
10.1.2.3.4} background, top outer and inner, bottom inner and outer
E O M r ( i . j ) pixels describing edge orientation masks o f potential pipe o f ridge region r
{0.1.3 Î background, high and low orientations
E E M r ( i . j ) pixels describing extended edge masks o f potential pipe o f ridge region r
(0.1.3} background, top and bottom
E E O M r ( i . j ) extended edge orientation masks o f potential pipe o f ridge region r
{0.1.3} background, top and bottom o f high and low orientation
M R M r ( i . j ) pixels describing the merge region mask o f potential pipe o f ridge region r
{0.11 background, image wide extended pipe region
Region specific pixel groups
R S R r ridge support pixels o f region r
T E P H r top edge pixels o f high orientation o f region r
T E P L r top edge pixels o f low orientation o f region r
B E P H r bottom edge pixels o f high orientation o f region r
B E P L r bottom edge pixels o f low orientation o f region r
T E E P R r top extended edge pixels o f region r
Region specific ridgelines and edge lines RLr ridgeline o f region r
T E L H r top edge line o f high orientation o f region r
T E L L r top edge line o f low orientation o f region r
B E L H r bottom edge line o f high orientation o f region r
B E L L r bottom edge line o f low orientation o f region r
T E L r top edge line o f region r
BEL r bottom edge line o f region r
T E E L r top extended edge line o f region r
This dissertation presents original computer vision algorithms that automatically ideniif\ piping and photogrammetric piping features in digital images.
Photogrammetric features are those points in an image, corresponding to points in the three-dimensional world, that can be readily identified from various viewing angles (see Figure l-I ). Orthogonal intersections o f piping at 90-degree elbows and T-junctions are such pipe features (i.e. they retain there 3D relationship to each other independent of viewing angle) but they are not generally used in photogrammetiy because their precise position in an image cannot be readily identified by inspection. A T-junction intersection point is inside the pipes and therefore not visible in photographs. Similarly, since 90- degree elbow pipefittings have a significant radius o f curvature, their pipe intersection point is in open space and is therefore not visible in photographs. The ability to automate these feature point identifications is a critical precursor for the cost effective
implementation o f existing photogrammetric computer m odeling programs to the significant industrial problem o f generating as-built piping drawings.
There are many instances where engineering drawings o f existing facilities have been lost or are out o f date because o f unrecorded modifications. Yet current drawings are often required: for a change o f ownership or insurance, a change o f government regulations, decommissioning o f a facility, or to engineer a change to a facility . In "Reversed Engineering o f a Product Model". Yvan J. Beliveau et al [1] point out that
facilities. These existing facilities typically had drawings and
specifications at the time o f construction but as-built data o f the facility^ is incomplete and hard to find. Over time, constant changes have occurred and little seems to match the facility today.”
They go on to point out that "currently in the United States, approxim ately 40% o f all construction deals with retrofitting existing facilities" and that "the m easurem ent and cataloging o f these existing facilities generally costs approximately 7% - 15% o f the cost o f the facility."
There are two existing means o f developing as-built drawings. The first, and by far the most common, consists o f surveying the existing facility and redrafting the drawings [2. 3]. The second means available to industry makes use o f close range photogrammetry (making measurements from photographs) [2. 3. 4. 5. 6].
Generally', developing as-built drawings falls on the critical path for any given project. This suggested that the opportunity to automate the photogrammetric as-built drawing process should be investigated since traditional survey methods do not lend them selves to computerization.
1.1. Limitation o f Existing Photogrammetric Methods
Focusing on the application o f photogrammetry to industrial piping structures, (natural gas com pression and metering facilities, petrochemical plants, pulp and paper mills, breweries, etc.) imposes certain environmental constraints that suggest any m easurement instrumentation must be both small and light weight [7. 8]. Some o f the existing close
generate a 3D model from an assembly o f multiple view point redundant images o f the lacilit) without the need to modify the facility (by adding imaging targets [2. 6. 9]). the environment (by adding structured lighting [10. 11. 12. 13]). or by using additional measurement information [3. 14].
Tools for industrial photogrammetry. using a single roving camera. [15. 16. 17. 18] are available and can efficiently generate "as-built" drawings o f rectangular structures. To generate 3D models o f a facility these tools rely on being able to identify
photogrammetric features (such as the com ers o f rectangular elements) within an image and then unambiguously m atching these features between various images o f the same elements. Computer vision techniques to identify the edges o f rectangular structures [7.
19. 20] have been developed and can be integrated into close range photogrammetiy [21 ].
Box E dge: P o sitio n 1 Right P o sitio n 2 Right B ox E d g e: P o sitio n 1 Left P o sitio n 2 Left C y lin d er E dge P o sitio n 1 Right C y lin d er E d g e P o s itio n 2 Left C y lin d er E d g e P o s itio n 2 Right i C ylinder E d g e Position 1 Left
%
C a m e r a P o sitio n 1 C a m e r a P o sition 2Figure I - I Two Pholograph Viewpoints o f a B ox and Pipe (Plan View)
Unfortunately cylindrical features do not present such readily identifiable 3D features. Though theoretically pipe edges can be identified autom atically the actual 3D position o f
pipe edges cannot be used to develop a photogrammetric measurement framework.
The non-existence o f effective computer vision methodologies to automatically identifv piping elements makes the development o f 3D piping models a slow labor-intensive process [3]. Typically the manual task o f picking the rectangular features in the
photographs o f a facility takes approximately "five to six times the number o f man-hours required for the photo survey" [6]. Then the position o f piping w ithin that rectangular defined coordinate framework is determined by rationalizing the apparent piping edge information from the various viewpoints.
Automating the identification o f photogrammetric features specific to piping elements will augment existing photogrammetry technologies by significantly improving both their ease-of-use and potential accuracy. The precisely defined standard sizes o f piping
elements makes the identification o f components more accurate than the pipe edge rationalization method used by current photogrammetric measurements.
1.2. Examples o f Existing Photogrammetric Methods
An example case study posted on the World Wide Web by Vexcel [16] Corporation o f Boulder Colorado. USA. (suppliers o f industrial photogrammetry software and "as-built" drawing services) shows a portion o f a chemical facility (approximately 25m. x 15m. x
15m) after an industrial accident. An explosion had rendered the structure unstable, so physical access was limited. They created a 3D piping model from images acquired by a three-man team during one day at site from a man-lift positioned around the perimeter o f
dem olition planning, and to identify (and measure) the piping tie-in points. This allowed replacem ent components to be designed and constructed during the dem olition phase, shortening the overall project schedule. The limitations o f current photogramm etiy tools are apparent when one notes that it required 30 days to prepare the 3D model for this project and the piping tie-in points were defined to approximately ± 6 mm o f the true positions (accuracy’s in the order o f * 1 mm would elim inate the need for field welds).
Pow ers ( 1996) [2. 6] presented a photogrammetric study for the Nexgen paper machine conversion in Port Albemi. B.C. They used surveyed num bered stick-on targets and required ten days to take the required photographs. These images were digitized using a custom negative scanner (providing 6144 x 5000 pixel resolution) and from these
digitized images they developed drawings with an accuracy o f = 5 mm. This
photogramm etry model was used for; dem olition drawings, pipe routing, interference detection, and design coordination.
Andrew Bailey (1999) [3] presents a case study using a portable laser scanner for "Reverse Engineering a Process Plant into a 3D CAD Model" and states that
"Photogrammetry has obvious advantages over m anual measuring techniques. However it still does not address the needs o f a nuclear plant well. Photogrammetiy suffers the following drawbacks:
- locating and "fixing" targets is time consuming
- revisits to site are often required because it is not easy to check for com plete data collection whilst on site
He concludes by identifying the existence o f "a gap between the as-built CAD generation needs o f the nuclear industry and the ability o f commercially available systems to deliver the required CAD models to the desired quality at acceptable cost".
1.3. Scope o f Research
The research presented here applies com puter vision techniques to support the automated identification and accurate positioning o f photogrammetric pipe features in im ages and makes use o f the constraining information that the type and dimensions o f allow able piping elements is strictly and precisely defined by the applicable piping code for a given installation.
As stated above the identification o f salient points within photographs (feature extraction) is currently a manual task. Though it is straightforward to get photographs into a
com puter (digital camera, slide scanner, etc.). the computer cannot know which o f the thousands or millions o f pixels, edges, and/or lines, are significant. To a com puter, a gray scale image is expressed as a table o f numbers, one quantized m easure o f light intensity for each pixel (example intensity measures for a small image are shown below).
248 248 244 224 187 147 120 110 100 251 251 244 218 171 127 097 087 080 248 248 241 208 161 117 090 083 080 224 231 228 I 9 | 150 110 090 083 083 177 187 191 167 137 107 093 087 083 127 137 140 130 100 100 090 083 080 097 104 104 104 100 097 093 083 077
Figure I -2 Image as Pixel Based Light Intensity S^Ieasurements
As can be observed in the above image and sub-image, even the basic task o f identify ing edges is not straightforward in real world photographs (indeed new edge detectors are still being introduced into com puter vision literature [22. 23]).
The work presented here is specifically focused on the automatic identification o f pipe segments and the intersection o f those segments at T-junctions and 90-degree elbows. These pipe intersections are features that have a fi.xed 3D position that is identifiable from various viewpoints.
To construct the 3D model o f a structure one must be able to identify when a feature in one photograph corresponds to the same feature in other photographs o f the structure (feature correspondence). This identification o f feature correspondence between
photographs is currently a manual task but research to automate this task is ongoing [7. 8. 24]. This aspect o f the automated photogrammetry problem is outside the scope o f this dissertation.
As a first step a computer vision program to identify- piping and make basic image measurements o f piping is needed (“Image to Pipes”). A processing step incorporating scale information from real world piping elements and calibration o f the camera can then be used to transform image measurements to a three dimensional model o f the piping ("Pipes to Model"). Rendering an image o f that piping model from the calculated cam era position ("M odel to Image"), and comparing that pseudo image to the input image, gives one a qualitative evaluation o f the accuracy o f the piping model.
Model of Piping im age identification of Piping f Qualitative Quantitative im age of Model Piping Model of Piping
Figure 1-3 Image Processing Accuracy Check Block Diagram
Using these same program modules in a slightly different order (Figure 1-3) lets a known piping model be input to the computer vision algorithms: providing a way to
quantitatively measure the accuracy o f the pipe identification and pipe modeling program modules.
The original work o f this dissertation is contained in the "Images to Pipes" arrow o f Figure I -3. The remainder o f the work is included to provide a proof mechanism for the computer vision algorithms incorporated in "Images to Pipes".
A fter this introduction o f the dissertation topic, a literature review into the state-of-the-art o f finding cylinders and pipes in com puter vision research is presented in C hapter 2.
Chapter 3 examines how images are formed and how the pinhole camera model can be used to make representative perspective pseudo images.
Chapter 4 presents the physical basis o f the pipe specific image segmentation algorithm that is the core original contribution o f this work. The issues involved with im plem enting this methodology are discussed in chapter 5.
The methods used to describe the bounds o f pipe regions with appropriate edge
information and the rationale used to examine w hether pipe segments should be merged are examined in chapter 6.
The automatic identification o f T-junction and 90-degree elbow intersections as photogrammetric features is described in chapter 7. A quantitative measure o f this methodology is determined by comparing the identified photogrammetric feature points with those defined for a given piping model in the pseudo image generation program (Figure 1-3).
Chapter 8 examines how the information from a single image can be used to determ ine the position o f the camera and piping within a coordinate system defined by the imaged piping. This modeling step completes the quantitative evaluation o f these com puter
vision m ethods by providing the "M odel o f Piping" to "Model o f Piping" comparison shown in Figure 1-3.
discussion o f the conclusions and recommendations for future work, are presented in Chapter 9.
Two concepts used extensively in this dissertation are summarized in separate
appendices: the Canny edge detector [25] and computational projective geometr} [26. 27].
1.6. On Proving' Computer Vision Algorithm
The existence o f optical illusions, which fool the human visual system, shows that there is no "oracle" which can be used to absolutely identify correct answers for a computer vision system. Kanatani [27] states the problem well with regard to his algorithm "if it does not group two edges which belong to the same house wall, then that image is as such".
The two typical methods o f "proving" computer vision algorithms will be used in this dissertation. The use o f pseudo images, where the required results are known from the input, allows a quantitative check o f some elements o f the algorithms. However the simplicity o f pseudo images does not provide a realistic demonstration o f how robust the algorithm s must be to cope effectively with real world images. To address those elements o f the algorithm s a number o f sample images will be used and the reader must Judge the efficacy o f the algorithms.
A single pipe model has been used to generate an identical pseudo image at two different resolutions (Figure 1-4). In this piping model all pipe elements are o f the same diam eter and located on multiple orthogonal planes (where each plane contains at least 2 pipes). The image also contains two box girder pipe supports to demonstrate that the algorithms being developed can differentiate between cylindrical and planar surfaces.
g r a y sc a le im age g ra y sc a le im a g e
Figure I -4 Sample Pseudo Images
Two example images o f outdoor piping structures are included to demonstrate the effectiveness o f the developed algorithms with changes in light intensity (Figure 1-5 diffuse lighting. Figure 1-6 bright lighting), shadows, background clutter, and weathered piping surfaces.
Figure 1-7 and Figure 1-8 show sample images o f indoor piping structures and are included to illustrate the effectiveness o f the developed algorithms with color coded piping, glossy pipe finishes, and flash photography.
Though the methods developed here are based on the physics o f illuminated cylindrical surfaces there will always be cases that confuse the algorithm. The ideal for such an algorithm is to know the physical limitations o f the method and to be able to predict when and why it will fail in a given instance.
too 200 300 400 500
Figure I -5 Sample Uncalibrated Camera Image A
100 200 300 400 500
Figure 1-6 Sample Uncalibrated Cam era Image B
600
100 200 300 400 500 600
100 200 300 400 500 600
2. Automated Pipe Finding - Literature Review
The core o f this work is the search for pipe specific photogrammetric features that can be automatically and unambiguously identified from various viewing angles in images o f piping structures. This requires the segmentation o f an image into pipe and non-pipe regions.
Various researchers have attempted to autom ate the identification o f piping (or the more basic cylindrical elements) in images but with very limited success. Representative samples o f various methods, not ju st the single free-to-roam camera proposed for this work, are presented below.
2.1. 3D Range Data
M easurements produced by a three-dimensional laser range finder are markedly sim pler than images from a camera. In essence range data is a map o f the actual surfaces being measured whereas a photograph provides a map o f the interaction o f the illum ination and reflectance o f the surfaces involved. Still, identifying and measuring cylindrical features in range data is an ongoing area o f research.
Many Printz (1987) [28] discusses how most previous work on cylinder identification has been based upon the analysis o f an object's occluding contours but that "occluding
contours are difficult to compute with accuracy". He then presents w ork on finding the orientation o f a cylinder by interpreting laser range finder data as an EGI (Extended Gaussian Image) that maps the surface normals o f the objects in an image on to a unit
sphere positioned at the image viewpoint. The EGI representation has these advantages: it "incorporates information from all o f the objects visible surface", "is invariant under translation", and "rotates with the object". Though an intriguing approach their "principal problem is that it is hard to get accurate estim ates o f surface normals, except for those portions o f the surface with normals that lie within a few degrees o f the viewing direction" m aking the overall system very in-accurate.
Ponce et al ( 1993) [ I I ] have "attacked automatic model construction by developing novel m ethods for fitting algebraic surfaces to range data and calibrated sequences o f video images." They present an "efficient global method for estimating an object's pose from the m inimum number o f point features and a local method for refining this estimate using the entire image contour" but as with many researchers they "have not yet addressed the critical issue o f segmentation ".
Burgiss et al (1998) [12] use a laser-based system that gives them range and reflected intensity at each pixel in w hat they call a registered image pair. They then fuse these two sorts o f data to evaluate structure in the 3D world and include in their paper a good exam ple image o f the industrial piping problem. They segment a smoothed version o f these registered image pairs by "dividing an image into areas that are relatively uniform in som e value (e.g. intensity, range, or curvature) ". Though their process shows great
promise, the technology is currently limited to generating 256 x 256 pixel images; far too coarse for accurate photogrammetric measurements, and the bulk o f the current
Andrew D. Bailey (1999) [3] from UK Robotics has presented a commercial system for as-built drawing generation in the nuclear industry using a new scanning laser camera technology (8000 horizontal x 1200 vertical lines scanned in 2 minutes). This system was implemented for installations where man access was limited to 10 minutes per day (to limit radiation dose uptake), but again the bulk o f the equipm ent does not lend itself to free-to-roam use.
2.2. M ultiple Cameras
Some researchers use the additional constraint o f the known geometiy o f multiple fixed cam era locations to identify cylindrical elements.
Schneider (1993) [29] presents one o f the few industrial systems for finding pipes. They use a fixed 6 cam era framework on a temperature stabilized, marked reference plate background, to measure the dimension o f bent tubing assemblies. Their "requirements for a tube m easurement system are a short measuring time, contact-less measuring, and a m easurement volume that covers all possible tube shapes.” They have developed a system based on photogrammetric and image processing m ethods and have achieved an accuracy o f about ± 0.5 mm on their first practical experiences.
Quoging Zhou (1997) [13] presents a system that uses structured light and 6 cameras - 2 fixed in place and two robot arms each with a stereoscopic cam era pair. Through being able to control the lighting and image position they assumes all edges are found and use a graph matching methodology to find an object in an image versus a CAD database o f objects. They conclude their paper by referring to their system as "a good start for
photogrammetric application in industr>'. Through these experiments, we find that photogrammetry can build a bridge between CAD and CV."
Though the multiple camera approach is not addressed in this dissertation it is apparent from the above examples that the extra information provided by the known geometiy o f multiple cameras does not make the pipe segmentation problem trivial.
2.3. Single Camera with Controlled lighting
Some researchers use the additional constraint o f known illumination to help identify cylindrical elements.
Thrift and Lee ( 1982) [30] "show how highlights on three types o f objects - cylinders, spheres, and generalized cylinders can be used to provide constraints on their size and location". But their technique requires the "observations o f highlights from another light source or knowledge o f the occluding boundary o f the object" to provide accurate results.
W olff and Fan (1994) [10] present a method that uses a single camera location with three lighting locations to determine the type o f surface (elliptic, hyperbolic, and parabolic) at each pixel in a photo. They point out that intensity images "are formed from the
reflectance o f light from objects and derivation o f shapes from reflectance can provide a rich description o f objects in a scene. Unfortunately the unique quantitative derivation o f local surface orientation and curvature using the information available from a single intensity image is a highly under-constrained problem, even when the reflectance properties o f objects are precisely known." By using image ratios o f their three lighting
locations "the technique is invariant to the relative incident radiance o f the light sources and surface albedo, and the sign o f Gaussian curvature is directly and accurately
computed from image intensity gradients without the need for lower level knowledge about the surface such as surface orientation."
.Again, though the controlled lighting approach is not addressed in this dissertation it is informative to realize the extra information provided by this extra level o f image control also does not make the pipe segmentation problem trivial.
2.4. Single Camera with Unconstrained Lighting
For the case o f interest here, using a single intensity image with no control o f lighting, the literature breaks down into three possible pipe (cylinder) identification methods.
2.4.1. Pipe Edges
The work o f others [31. 32. 33. 34. 35. 36] has shown that pipe identification based on the detection o f the occluding edges across the pipe diam eter is unreliable in complex real w orld photos. Edges are generally assumed to be step changes in light intensity and in com plex images; edges can be obscured since the background to an occluding edge can be o f any intensity. Note that the background under the pipe at the center o f the image (Figure 2-1 a sub image o f Figure 1-5) changes from light to dark throughout the length o f the pipe. This edge demonstrates one o f the difficulties o f using step intensity
to darker support face, (b) dark pipe bottom to light concrete foundation, and (c) dark pipe bottom to sim ilar intensitv' gravel surface.
Figure 2-1 Pipe Images (pseudo a n d real)
Still most com puter vision researchers who attem pt to identify cylindrical o r piping components rely on being able to accurately define the piping edges.
Ponce et al (1987) [37] state their "goal is to support reasoning about three - dimensional objects, which is crucial for com puter v ision, while providing the modeling power and flexibility o f a CAD system, which is crucial for representing objects o f realistic
complexity" but in essence they decide if a surface is planar or curved by the shape o f the bounding edges (i.e. they need to see the end o f a region to differentiate between a
cylinder and a planar surface).
Hallset (1991) [14] presents a "Simple Visual Tracking o f Pipelines for an Autonomous Underwater Vehicle" but the limitations o f their rectangular region growing approach means that their method would be severely limited. It finds planar as well as curved surfaces identically and would be confused by surface highlights. Their m ethod also
requires other data (compass and sonar range finder information), a uniform flat seabed background, and would not work with perspective views. They acknowledge that at this stage o f their research they expect to have difficulty with real inspection situations.
Zerroug and Nevatia (1993) [38] obser\e that
"most previous work on inferring 3D shape from 2D contours assumes that the problem o f object (surface) segmentation has been solved, whereas this is a key and difficult step in monocular scene analysis. Real images
produce contours with many imperfections such as distortions, breaks and occlusion. Further, not Just 'real' image contours are present in an image. Surface markings, shadows and noise also produce contours. . . . The difficulty in dealing with such imperfections is that it is impossible, by just looking at the contours individually, to tell which constitute real contours and o f what objects and which do not. or simply how many objects there are in the scene. . . . To address this problem, it is necessarv' to use a grouping process to collect relevant features together and discard the irrelevant ones."
Since their method relies on grouping only edge information it does appear to be robust enough to properly identify pipes in an unconstrained environment.
Ulupinar and Nevatia (1995) [39] state that
"inferring 3D shape o f the objects in a scene from a single image remains an important and difficult problem in computer vision. The difficulty arises, o f course, from the fact that an image is a 2D projection o f the scene and the process is not invertible without making some assumptions. A num ber o f approaches for inferring 3D shape have been suggested, such as shape from shading, shape from texture, and shape from contour. We believe that o f all the monocular cues, shape o f the 2D contour itself is the m ost important one for the shape o f the 3D surfaces."
"To apply our method to real images we first need to find the boundaries o f the objects and then the symmetries, if any. contained in them. In general we can expect object boundaries to be fragmented, and several intensity' boundaries that correspond to surface markings, shadows, and noise, to be present. To separate the object boundaries from these other boundaries, and to fill in the gaps in object boundaries as appropriate, is a difficult problem in monocular image analysis and this paper is not about such analysis. The results shown hear assume that labeled curv es (cross section and limbs) are given to the algorithm as connected points with exact coordinates."
Zerrough and Nevatia (1996) [40] present work on free form shapes as well as straight cylinders and are pursuing "methods that are able to detect objects and recover their shapes in the presence o f noise, surface markings, shadows and partial occlusion." But thev" rely on having high contrast intensity images to get good edge images, from which the\- infer local surface patches and look at how these would need to act for given (predefined object) orientations which might match the edge images.
Huang et al (1996) [36] present a "method capable o f determining the 3D orientation and position o f a right cylinder with known radius from a single image" but they rely on having an almost complete edge determination o f the cylinder to work from. Their test images are high contrast non-specular situations with a solid black background.
Seyda-Mahmood (1997) [35] present "a method o f identifying groups o f closely spaced parallel lines in images that generates a linear num ber o f small-sized and reliable groups thus meeting several o f the desirable requirements o f a grouping scheme for recognition." In essence they look for parallel edge lines in an image and present a heuristic grouping method to identify' potential cylindrical elements. One constraint o f his approach is that it ignores perspective effects and would not find piping at an angle to the camera.
Though defining the edges o f pipes autom atically would provide a great deal o f
information for photogrammetric modeling no robust method to do this currently exists.
2.4.2. Pipe Ellipses
An ellipse can be interpreted as an oblique view o f a circular shape and m ethods to identify' pipes through identifying ellipses directly are of interest to com puter vision researchers.
Ellis et al (1992) [41] present the results o f their techniques on a photo o f industrial piping and dem onstrate m any o f the problems associated with ellipse finding techniques. Essentially, ellipse finding: is computationally expensive (processing many edges to determine if they are part o f ellipses where as the vast majority o f edges in the image are actually straight lines), contains an inherent curvature bias toward highly eccentric ellipses, and typically exhibits many false positives (image pixels erroneously associated as portions o f an elliptical edge).
W erghi and Doignon (1996) [42] present the application o f wavelet techniques to the segmentation o f elliptical edge data but the real image results they presented do highlight som e difficulties. Though ellipses identified on the near end o f cylinders (where the full circumference is available for the ellipse fitting) are reasonable approximations, ellipses identified on far cylinder ends are dramatically oversized (i.e. the parallel pipe sides seem to offset their ellipse estimator).
Though ellipses would provide a reliable estimate o f camera viewpoint, autom ated ellipse detection is not robust enough to provide unambiguous feature extraction from
photographs o f real piping structures.
2.4.3. Surface Reflectance Measurements
Researchers have sought to make use o f the additional information available in image regions (rather than ju st edges) to identify cylindrical components.
Cemuschi-Frias and Cooper (1984) [34] developed an image sub sampling methodology which they used to find uniformly lit Lambertian cylinders by looking for parallel lines o f equal light intensity. This would have limited applicability to the perspective views and non-Lambertian surfaces in photographs o f real piping structures.
Gross and Boult (1990) [43] present a method that "assumes that the generalized cylinder being recovered has purely diffuse reflectance and that the diffuse reflectance coefficient is constant". They recognize that such "a restriction is highly undesirable since, except for highly controlled research environments, such information is generally unavailable" and have tried to keep assumptions o f image modeling as simple as possible "scaled orthographic projection. Lambertian reflectance, and constant albedo". Each o f these assumptions is inappropriate for images o f typical piping structures.
Wink. Smeulders and Koelma (1994) [33]. expanded the work o f Cem uschi-Frias and Cooper to consider both diffuse and specular cylindrical surfaces with a Torrance- Sparrow reflectance model. But since their work required the inclusion o f a target o f
known reflectance in each image their work would have limited application to photogrammetrv- o f piping structures.
R\ u et ai ( 1996) [32] proposed a "method for analyzing the properties o f surface rellectance and reconstructing the shape o f an object using estim ated reflectance
parameters". They also insert a calibration object with known reflectance parameters into the scene and use a Torrance-Sparrow reflectance model to estim ate the reflectance properties o f the other elem ents in the image.
The work o f Stewart and Langer ( 1997) [44] is a shape-from-shading algorithm, limited to diffuse lighting on a m at surface but including the effects o f inter-reflection. They observe that their "algorithms underestimate the depth . . . due to a subtle ill-conditioning property o f the shape from shading under diffuse lighting problem. Small differences in image intensity o f the brightest points . . . can correspond to relatively large differences in depth." Unfortunately their teclmique is very computationally com plex - applying their approach to a simple uncluttered 100x100 pixel image required between "20 and 36 minutes to process".
In "Physics - Based Segmentation o f Complex Objects Using M ultiple Hypotheses o f Image Formation" M axwell and Shafer (1997) [45] present a comprehensive image segmentation method. They "divide an image o f a scene into regions that are meaningful in terms o f the objects constituting that scene. This means that the computer must
generate and reason about one or more descriptions o f the scene elements that formed the image - the illumination, material optics, and geometry - in order to form and
interpretation.” On intermediate images o f normalized color, gradient orientation, and reflectance ratio (measures the local change in intensity as a ratio that is invariant to illum ination and shape) they use region growing techniques to group pixels o f similar properties and use a sum squared difference measure at the borders to find the local sample variance (w ith a threshold chi-square test to define the regions edges). The\ also point out that "the intensity profiles contain a significant am ount o f information" and approxim ate their intensity scan-line with polynomials (up to order 5) to look for the minim um s / m axim um s. Unfortunately they have not im plem ented all o f the hypothesis tests they have proposed and for their sample images they have informed the com puter o f the appropriate results for some regions. They conclude by pointing out that "this is a work in progress. However, even with only two hypotheses im plem ented we are able to segm ent images containing more complex objects than previous physics-based
algorithms."
So. though shape-from-shading methods do make use o f more image information than edge based segmentation algorithms, no robust method o f identifying piping currently exists.
2.5. Why is this difficult?
Autom ated identification o f pipe edges and ellipses in real world images is such an
unconstrained problem that none o f the existing methods appear to be robust enough to be a useful adjunct to generating as-built piping drawings. Equally problematic is that the reflectance characteristics o f real surfaces are generally unknown. Therefore
com putational methods that rely on being able to model the illumination/surface
reflectance interaction to interpret the shape o f the underlying surface are not generally applicable to real world images. Another complicating factor in interpreting shape from real world images is the existence o f inter-reflections. Even in scenes where there is only one source o f illumination we can see sections o f the image that are not directly lit
because o f the light reflected o ff o f other elements in the scene (e.g. light bouncing o ff the ground illuminates the underside o f pipes Figure 1-5. Figure 1-6. Figure 1-7. and Figure
1-8). Inter-reflections [44. 45. 46] are not generally examined in image segmentation problem s since the complexity o f examining all o f the possible illumination sources for each pixel in the image is intractable. Finally, perspective is an additional area o f
com plexity in real world photographs. It is common for researchers to ignore perspective foreshortening and use the much sim pler mathematics o f orthographic projections [33. 34. 40. 47]. This orthographic simplification is not acceptable in this application since limited access to photograph industrial installations often results in significant
perspective effects in the images.
The additional step o f identifying T-Junction and 90-degree elbow pipe intersections, to identify photogrammetric pipe features, is also hampered by observing that these points can not be readily identified by inspection. A T-junction intersection is inside the pipes and is therefore not visible in photographs. Similarly, since 90-degree elbow pipefittings have a significant radius o f curv ature, the photogrammetric feature corresponding to the intersection point o f the two pipes is generally positioned in open space and is not directly visible in photographs.
These five issues: unconstrained problem, unknown reflectance characteristics, inter reflections. perspective, and non-visible photogrammetric pipe features, are common in images o f real world piping and combine to make pipe identification a formidable computer vision challenge.
2.6. Conclusion
Computer vision research has not. as yet. determined a method with sufficient
computational robustness to automatically identify pipes for industrial photogrammetiy.
Carsten Steger et al [48] have pointed out a number o f object recognition system (ORS) constraints in both biological vision (BV) and machine vision (MV) systems that have helped formulate the m ethods described in this dissertation.
"Though most recent techniques for ORS in MV use range data, there is still a need for intensity-based ORS not only because o f its relevance to BV but also because o f the robustness o f passive sensing for industrial and other applications." . . . "However, for passive sensing there is one major limitation: no depth or shape can be inferred from pixels whose
neighborhood variances are zero: where there is no variation in light or 'features'. This shows that the inference o f full range from intensity information cannot be obtained without prior or 'top-down' knowledge or constraints."
"It is widely accepted in both the psychological and in the machine vision literature, that som e form o f image segmentation must occur before recognition can take place.". . . "Segmentation reduces geometric
information about an object into discrete, manageable chunks. In this way. parts may be recognized in isolation, making it unnecessary for all object parts to be visible for object recognition to take place. It is necessary, however, to include information not only about the segmented parts themselves, but also about the spatial relationship between these parts."
"A final point should be made about the underlying physiology o f object recognition. First, it should be noted that vision, as a sense, is passive and so any theory o f biological object recognition that does not address the problem o f inferring depth or shape from intensity - as an integral part o f the processing system - is not complete."
The work presented in the following chapters addresses these issues and illustrates a pipe segmentation algorithm that has the computational robustness to be useful for real world imaaes o f industrial installations.
3. Imaging o f Industrial Piping
Industrial piping has some distinctive visual information that can be used to help in its identification. This information constrains the task o f image segmentation to a search for light intensity measurements, which would be generated by:
• smooth, straight, cylindrical surfaces. • o f a solid color.
• which exist only in precisely defined diameters.
This chapter is a discussion o f simple image formation using basic Lambertian reflectance modeling and perspective geometric transformations. Though far more complex and realistic reflectance models exist [49] the simple Lambertian model clearly demonstrates the physical foundation for the data feature extraction methods presented in the chapters 4. 5. and 6. The perspective geometric transformations presented here
demonstrate the need for cam era specific calibration information in the generation o f pseudo images and is the basis o f the pipe model reconstruction algorithms presented in chapters 7 and 8.
3.1. Simple Image Formation
The images under discussion in this dissertation are collections o f light intensity
measurements on an image plane, a two dimensional surface (e.g. the CCD array inside o f a digital cam era) used to capture information about how light is reflected o ff the surfaces o f three dimensional structures in the real world.
Æ Ligm Source
Image Plane
Pipe
Pipe Image Bounds Cross-Section '
Figure 3-1 Image Formation
Independent o f cam era parameters the image o f any structure is formed through the interaction o f a num ber o f physical radiometric factors:
• position and characteristics o f the illumination source(s) • position o f the three dimensional surfaces included in the view • reflectance properties o f the surfaces in the image
• the orientation and properties o f other surfaces reflecting light onto the surfaces contained in the image (illumination by inter-reflection) and
• the position and orientation o f the camera
So an image can be considered a combined geometric and radiometric transform ation o f simple three-dimensional surfaces (e.g. planes and cylinders) into a two dimensional matrix o f light intensity measurements.
In the following exam ples the light reflected from the background to the cam era image plane has been ignored as if the elements being photographed exist alone on a light absorbing background.
3.1.!. Lambertian Shading Model
.A key component o f this radiometric transformation is the complex material surface property o f reflectance which determines how light illuminating a surface will reflect to a given cam era location.
There are various reflectance models in computer vision literature [32. 44. 49. 50, 51. 52. 53] but the oldest (1760) and simplest o f these. Lambert's Law. is the model m ost widely used. It assumes that the amount o f light reflected from a surface can be defined from knowledge o f only three factors; the illumination intensity, a diffuse reflectance
coefficient for the surface, and the geometry o f the surface normal to illumination angle (Figure 3-2).
cos^ = / )
where;
I aiffuse = intensity of diffuse reflectance I source = intensity o f point light so u rce ko = diffuse reflection coefficien t (0=>1 )
0 = lighting to su rface normal an g le n = surface normal unit vector f = lighting direction unit v ecto r
Figure 3-2 Lambert's Law
Under the constraints o f Lambert's Law a surface appears equally bright from all viewing angles.
V
Though the Lambertian shading model has very limited applicability to real-world images it does provide a simple physical basis to describe image formation and helps to explain the origin o f the pipe specific light intensity data features developed in the next chapter.
3.1.2. Planar Surfaces in Cross-section
If the light source for a photograph is ver\ distant (e.g. the sun) the light beams
illuminating a subject can be assumed to be parallel and the lighting to surface normal angle (shown as 0 below) becomes a simple relationship o f the difference o f angles.
_ I la, = IlK, cos e, 4ÔV Lighi Source
Box Girder Cross-Seclion
Surface N o r m a ls ;
l«r = liKaCOS I): Image Plane Position x (pixels) Modeled Ligm Intensity Measure
Image : Plane
Box image Bounds
#
Figure 3-3 Lam hertian B ox Girder
Under Lambert's Law. for a given light intensity (U) and diffuse reflection coefficient (Kd). the intensity measured at the image plane is constant for each given surface normal. In the cross section o f a box girder image (show n above center) each visible surface has a different constant intensity measurement. Resulting in the classic step change o f light intensity measurements associated with edges.
3.1.3. Cylindrical Surfaces in Cross-section
The continuous change in surface normal involved w ith the cross section o f a cylinder produces a continuous light intensity distribution (as show n below).
Image Plane Positioo x (pixels) jugm I Source _ L», = LK, cos 0, Surface I Normals i Image Plane Modeled Ugnt intensity Measure P'P* Image Bounds
Figure 3-4 Lambertian Pipe
With the light source above and behind the photographer there is a point on the pipe cross-section where the angle between the surface normal and the incident light ray is zero: at this point the light reflected to the camera is maximum. For the top o f the shown pipe the viewing angle eclipses the illuminated pipe section. In such cases, the local light to surface normal angle determines the am ount o f light reflected to the camera. If the viewing angle includes the case where the light to surface normal angle is 90 degrees (near bottom o f shown pipe) the Lambertian model directs that the light reflected to the camera is zero and any further portion o f the pipe cross-section is not illuminated (bottom o f shown pipe). The bottom o f such structures are visible in real world images since the ground plane and other objects in the scene reflect light back onto those areas o f a structure not directly illuminated (illum ination by inter-reflection).
The observation that the surface o f a pipe produces a light intensity "hum p" is one o f the key data features used in this dissertation to identify the position o f pipes within images.
3.1.4. Edges in Cross-section
Many researchers have looked for edges alone to develop physical models from image data [19. 20. 25. 31. 42. 54. 55. 56].
Looking at the modeled light intensity measure for the box girder cross-section (Figure 3- 3 center) one can see three step changes corresponding to: the top occluding edge, the edge between planar surfaces o f the box. and the bottom occluding edge.
Looking at the modeled light intensity measure for the pipe cross-section (Figure 3-4 center) one can see two. step-changes corresponding to: the top occluding pipe edge and the bottom occluding pipe edge.
Shown below is a slightly more realistic case, which superimposes the above pipe and box girder examples. This example demonstrates that, even with such geometrically distinct elements and an extremely simple reflectance model, discerning edges can be difficult. COS O ie Ljo p = li_K@ l«3P = 0 Ugm Source Box Image . Bounds
Pipe & Box Girder Cross-Section
Image Plane Position x (pixels) Modeled Light Intensity Measure
Pipe Image Bounds
-0:p =0
0.
# #
A more realistic example o f this problem is demonstrated with the t\p ical pipe rack geometr) . where multiple pipes share a com m on support; the resulting difficulty o f discerning the edges between the pipes in this cross section is illustrated below.
Ut p; - 2 COS 0.P7 l-ac Pt “ kK, ; : XS: Light Scarce Ptpe 2 •mage Bounds Two Pipe Cross-Section l<ao p* — LKg 1 Image Plane Position x (pixels) Modeled Light intensity Measure
Pipe 1 Image Bounds
iy,. = o 0;p. =90^*
Figure 3-6 Lamhertian Edges - Two Pipes
Generally, edge detectors are designed with the assumption that edges can be modeled as step changes in local light intensity measurements. The above exam ples illustrate how difficult it can be to identify these step changes in even the sim plest o f pipe structures.
3.2. Digital Image Formation - Quantization
Though a pseudo image has been presented for each o f the above geometric exam ples (Figure 3-3 left. Figure 3-4 left. Figure 3-5 left, and Figure 3-6 left), only a single vertical slice o f the light intensity measurement (Figure 3-3 center. Figure 3-4 center. Figure 3-5 center, and Figure 3-6 center) has been examined, to demonstrate how objects would reflect light at some unique cross-section in the image.
In the case o f idealized image geometry shown below the image plane o f the cam era is parallel to the center line o f the pipe, the vertical direction o f the image plane is
