Specialized depth extraction for live soccer

(1)

Specialized depth extraction for live soccer

Citation for published version (APA):

Vosters, L. P. J., Haan, de, G., & Peset, R. (2010). Specialized depth extraction for live soccer. In Proceedings

of 21st ProRISC Workshop of the STW.ICT Conference, 18-19 November 2010, Veldhoven, The Netherlands

(pp. 1-31). STW Technology Foundation.

Document status and date:

Published: 01/01/2010

Document Version:

Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers)

Please check the document version of this publication:

• A submitted manuscript is the version of the article upon submission and before peer-review. There can be

important differences between the submitted version and the official published version of record. People

interested in the research are advised to contact the author for the final version of the publication, or visit the

DOI to the publisher's website.

• The final author version and the galley proof are versions of the publication after peer review.

• The final published version features the final layout of the paper including the volume, issue and page

numbers.

Link to publication

General rights

Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners

and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

• Users may download and print one copy of any publication from the public portal for the purpose of private study or research.

• You may not further distribute the material or use it for any profit-making activity or commercial gain

• You may freely distribute the URL identifying the publication in the public portal.

If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please

follow below link for the End User Agreement:

www.tue.nl/taverne

Take down policy

If you believe that this document breaches copyright please contact us at:

openaccess@tue.nl

(2)

Specialized Depth Extraction for Live Soccer

Video

Luc Vosters

Axon Digital Design

Eindhoven, University of Technology

(3)

Introduction

Related Work

Proposed Approach

Results

Conclusion

Questions

(4)

2D-To-3D Conversion

3D

I

3D Cinema.

I

3D TV sets.

I

3D broadcast.

I

3D Live Events.

(5)

3D Productions

I

3D cameras.

I

2D-to-3D conversion.

I

_{Offline (semi-) automatic.}

(6)

3D Productions

I

3D cameras.

I

2D-to-3D conversion.

I

_{Offline (semi-) automatic.}

(7)

Why 2D-To-3D conversion?

2D-to-3D Conversion

I

10,000$

I

Widely available.

I

No camera rig.

3D Recording

I

80,000$

I

Investment.

(8)

Stereoscopic 3D

(9)

Stereoscopic 3D

From Depth to Stereo 3D

I

Depth ↔ disparity.

I

2D-To-3D:

1. Extract depth.

2. Calculate disparity.

3. Render Left/Right

image.

I

Occlusion handling.

Right Eye

Left Eye

Screen

Left View Right View

(10)

Stereoscopic 3D

From Depth to Stereo 3D

I

Depth ↔ disparity.

I

2D-To-3D:

1. Extract depth.

2. Calculate disparity.

3. Render Left/Right

image.

I

Occlusion handling.

Right Eye

Left Eye

Screen

Left View Right View

(11)

2D-To-3D Conversion of Live Soccer Video

(12)

Jung et al.

(13)

Jung et al.

Jung et al. cont.

Advantages

I

No occlusions.

I

Few computations.

Disadvantages

I

Pan, tilt, zoom not modeled.

I

Audience depth constant.

(14)

Proposed Approach

I

Field and audience depth model.

I

Exploit Camera + Scene information.

I

Focal length

I

Tilt/Pan angle

I

_{Image sensor size}

(15)

Image Model Field + Player Detection Field Boundary Extraction Depth Offset Calculation Depth Calculation 2D Input Feed User Input: AR f, βt, βp, mx, my Input Camera Feed Output Proposed Method L Stereoscopic View Rendering 3D output

(16)

Field and Player Detection

Field Detector (Seo et al. [2])

I

Train Hue, Sat, Val histograms.

I

Extract

PeakValueIndex

,

SaturationMean

.

Field , if











G > 0.95 · R

R > 0.95 · B

V < 1.25 ·

PeakValueIndex

S > 0.8 ·

SaturationMean

(1)

(17)

Field Boundary Extraction

(18)

Player Detection

Input image

Connected Components

Player

if







0.4 <

AspectRatio

<

3 #Pixels

>

0.05MN

BBwidth·BBheight

#Pixels

>

0.25

(19)

Player Detection Cont.

I

Field Boundary Extraction aides Player Segmentation.

(20)

Camera Depth Model

Camera depth model

I

Long shot camera

I

_{Small lens aperture.}

I

_{High depth of field.}

(21)

Field Depth

Depth in YZ-plane:

f y z βt Do D αV βt-αV Lp V

D = D

o

_sin(|β

sin |β

_t

|−α

t

|

V

)

Depth in XZ-plane:

x z Do D βp αH βp-αH f Lp H

D = D

o

_cos(|β

cos |β

_p

_|−α

p

|

H

)

(22)

Field Depth cont.

Depth in XYZ-Plane.

Lin e pa ralle l to field bo un dary lin e Image Sensor

D

field

(α

V

, α

H

) =

D

o

_sin(|β

sin |β

t

|

t

|−α

V

)

cos |β

p

|

cos(|β

p

|−α

H

)

(23)

Field Depth Calculation

f Lp H Lp D Lp V αH αV A B C

α

_V

=

arctan

L

PV

f

α

_H

=

arccos

2f

2

+L

PV 2

+L

PD 2

−L

P H 2

2 q

(f

2

_+L

P V 2

)(f

2

_+L

P D 2

₎

!

( xo, yo) ( xl, yl(xl) ) ( xo, yl(xo) ) LHp LVp

Field boundary Parallel Field Lines Field Vertical center line y x Line l LDp

(24)

Depth Offset

I

Player length ≈ 1.80m

I

L

P

↔ Tilt, f , Depth, L.

I

D

o,i

depth offset player i.

I

D

o

=

median{D

o,i

|∀i}

Player Length

y z βt βt αV * * D L Lp f

D = L ·

sin β

t

[

L

P

·m

y

+f tan α

V

]

+f cos β

t

(25)

Audience Depth

D

aud

(y ) =

D

FB

_0.1N

·AR

(y − y

FB

) +

D

FB

,

Depth

( Y

FB

, D

FB

)

(Y

_FB

- 0.1N , D

_FB

AR)

(26)

Player Depth

Player Length

y z βt βt αV * * D L Lp f

I

Player approximately constant.

(27)

Complete Camera Depth Model

I

Depth Map Quantization.

DEPTH(x , y ) = 255 ·

D(x ,y )−D

min

D

max

−D

min

(28)

Depth Offset Calculation

Depth Offset Calculation

I

Zooming

I

Keep focal length constant in depth offset calculation.

22 24 26 28 30 32 34 36 D e p th O ff s e t

(29)

Qualitative Comparison

Qualitative Comparison

(30)

Conclusion

I

Jung et al. [1] proposed specialized 2D-3D conversion of

long shot images.

I

We propose more advanced model:

1. zooming, panning and tilting modeled.

2. Calculates depth offset from player length.

3. Improved player segmentation by field boundary extraction.

I

Few computations.

I

Hardware attractive.

(31)

Questions

(32)