s yst e m t h at ai ms t o pr o vi d e a n i m m ersi ve t el e pr es e n c e e x p eri e n c e w hil e b ei n g e as y t o s et u p a n d a c c essi bl e f or ot h er r es e ar c h ers t o us e as well.
T h e s yst e m is d esi g n e d ar o u n d t hr e e f e at ur es t h at ar e i nt e n d e d t o i n cr e as e i m m ersi o n a n d us a bilit y f or t h e us ers; m o bilit y at t h e r e m ot e l o c ati o n, a n i m m ersi ve vi e w of t h e r e m ot e e n vir o n m e nt a n d a virt u al e m b o di m e nt.
T h e s yst e m is d esi g n e d i n a n as y m m etri c al w a y, w h er e t h e visit or is virt u all y tr a ns p ort e d t o t h e visit e e’s e n vir o n m e nt usi n g a virt u al r e alit y s yst e m t h at dis pl a ys a virt u al r e pr es e nt ati o n of t h e visit e e’s s urr o u n di n gs. A virt u al e m b o di m e nt of t h e visit or is pr oj e ct e d i nt o t h e visit e e’s e n vir o n m e nt usi n g Mi cr os oft’s H ol o L e ns a u g m e nt e d r e alit y gl ass es. T h e H ol o L e ns’ l o w r es ol uti o n g e o m etri c al s c a n of t h e e n vir o n m e nt is us e d as a b asis f or t h e s c e n e t h at is s h o w n t o t h e visit or, w hi c h is e xt e n d e d wit h m or e d et ail e d p oi nt cl o u ds t h at ar e c a pt ur e d wit h c o m m o dit y d e pt h c a m er as.
A us er t est w as c o n d u ct e d i n w hi c h w as e v al u at e d h o w t h e t hr e e d esi g n as p e cts c o ntri b ut e t o t h e p erf or m a n c e a n d e x p eri e n c e of a t el e pr es e n c e s ys- t e m i n a c oll a b or ati ve c o nt e xt. Fo ur s yst e m c o n fi g ur ati o ns wer e pr e p ar e d;
o n e “ b as eli n e ”, i n w hi c h all d esi g n f e at ur es h a ve b e e n i m pl e m e nt e d a n d t hr e e
“t est ” c o n fi g ur ati o ns fr o m w hi c h o n e of t h e t hr e e d esi g n f e at ur e’s i m pl e m e n- t ati o ns w as r e m o ve d r es p e cti vel y. T hirt y p airs of p e o pl e p arti ci p at e d, w hi c h h a d t o p erf or m a s eri es of t as ks i n a n es c a p e r o o m-li ke s etti n g, o n c e wit h t h e
“ b as eli n e ” c o n fi g ur ati o n a n d o n c e wit h o n e of t h e “t est ” c o n fi g ur ati o ns. We s a w t h at es p e ci all y vi e w i n d e p e n d e n c e a n d i m m ersi ve dis pl a y h a d p ositi ve e ff e cts o n t h e p erf or m a n c e a n d t h e visit or’s f e eli n g of s p ati al pr es e n c e. All as p e cts i n cr e as e d t h e s yst e m’s us a bilit y f or t h e visit or.
T h e d esi g n e d s yst e m h as b e e n i m pl e m e nt e d a n d h as b e e n f o u n d t o b e us ef ul b y ot h er r es e ar c h ers alr e a d y as well. W h at’s m or e, we h a ve s h o w n t h at t h e i m pl e m e nt e d d esi g n f e at ur es d o i n d e e d i m pr o ve t h e e x p eri e n c e a n d us a bilit y of t h e s yst e m w hi c h c a n b e us ef ul f or ot h er r es e ar c h i n t h e ar e a of t el e pr es e n c e as well.
A c k n o wl e d g e m e nt s
I w a nt t o t h a n k m y s u p er vis ors f or g ui di n g m e t hr o u g h t his pr o c ess; M ari ¨et
f or gi vi n g m e t h e fr e e d o m t o e x pl or e di ff er e nt s u bj e cts, D e n nis f or his cr e-
ati ve i d e as a n d J a n f or ti n keri n g t o g et h er o n t h e s yst e m. I t h a n k m y p ar e nts
f or t h eir s u p p ort t hr o u g h o ut t h e ye ars. I w o ul d als o li ke t o t h a n k e ver y b o d y
w h o t o o k t h e ti m e t o p arti ci p at e i n t h e us er t ests a n d m y D esi g n L a b f a mil y
f or l etti n g m e us e t h e s p a c e.
1 I nt r o d u c ti o n 1 1. 1 Tel e pr es e n c e S yst e m D esi g n As p e cts . . . 2 1. 2 R es e ar c h Q u esti o ns . . . 6
2 B a c k g r o u n d: Fr e e Vi e w p oi nt Vi d e o 8
2. 1 A c q uisiti o n . . . 9 2. 2 Pr o c essi n g . . . 1 0 2. 3 Dis pl a y . . . 1 2 2. 4 C o n cl usi o n . . . 1 3
3 A r c hi t e c t u r e a n d Gl o b al D e si g n 1 4
3. 1 Gl o b al D esi g n C o nsi d er ati o ns . . . 1 4 3. 1. 1 A R vs V R . . . 1 5 3. 1. 2 O n-sit e E n vir o n m e nt C a pt uri n g . . . 1 6 3. 2 S yst e m C o n fi g ur ati o ns . . . 1 8 3. 3 Ar c hit e ct ur e . . . 1 9
4 I m pl e m e nt a ti o n 2 0
4. 1 O n- Sit e E n vir o n m e nt C a pt ur e a n d Vis u ali z ati o n S yst e m . . . 2 0
4. 1. 1 D e pt h C a m er a Str e a m er . . . 2 0
4. 1. 2 P oi nt Cl o u d Vi e wer . . . 2 1
4. 1. 3 D e pt h C a m er a Tr a c ker . . . 2 2
4. 1. 4 S p ati al M es h Str e a m er . . . 2 3
4. 1. 5 S p ati al M es h Vi e wer . . . 2 4
4. 2 Tr a nsf or m M a n a g e m e nt S yst e m . . . 2 5
4. 2. 1 Tr a nsf or m M a n a g er . . . 2 5
4. 2. 2 H ol o L e ns Tr a nsf or m . . . 2 6
4. 2. 3 H M D a n d C o ntr oll er Tr a nsf or ms . . . 2 6
4. 3 A n n ot ati o n S yst e m . . . 2 6
4. 3. 1 Li n e M a ker . . . 2 7
4. 3. 2 Li n e Vi e wer . . . 2 8
4. 4 N et w or ki n g S yst e m . . . 2 8
4. 4. 1 U D P C o n n e ct or . . . 2 8
4. 4. 2 M at c h m a ki n g S er ver . . . 2 9 4. 5 V R C o m p o n e nts . . . 3 1 4. 5. 1 V R S yst e m . . . 3 1 4. 5. 2 N a vi g ati o n . . . 3 1 4. 5. 3 S etti n gs M e n u . . . 3 2 4. 6 H a n d G est ur e S yst e m . . . 3 3 4. 6. 1 L e a p H a n d S e ns or . . . 3 4 4. 6. 2 L e a p H a n d Vi e wer . . . 3 4 4. 7 Ver b al C o m m u ni c ati o n S yst e m . . . 3 5
5 P r eli mi n a r y E v al u a ti o n 3 6
5. 1 S yst e m Di ff er e n c es . . . 3 6 5. 1. 1 M et h o d . . . 3 7 5. 2 R es ults . . . 3 8 5. 3 Dis c ussi o n . . . 3 8
6 M ai n E v al u a ti o n 4 1
6. 1 Perf or m a n c e . . . 4 2 6. 2 E x p eri e n c e . . . 4 3 6. 3 H y p ot h es es . . . 4 5 6. 4 Tas k . . . 4 7 6. 5 S yst e m S et- U p . . . 5 0 6. 5. 1 C o n fi g ur ati o ns . . . 5 0 6. 5. 2 M o di fi c ati o ns . . . 5 1 6. 5. 3 D at a L o g g er . . . 5 5 6. 5. 4 L a y o ut . . . 5 5 6. 6 St u d y D esi g n . . . 5 7 6. 7 M e as ur es . . . 5 7 6. 8 E x p eri m e nt Pr o c e d ur e . . . 5 9 6. 9 D at a Pr o c essi n g . . . 6 1 6. 9. 1 D ur ati o n E xtr a cti o n . . . 6 1 6. 9. 2 L e ar ni n g E ff e ct . . . 6 1 6. 9. 3 D at a A n al ysis . . . 6 2 6. 1 0 R es ults . . . 6 3 6. 1 0. 1 Vi e w I n d e p e n d e n c e . . . 6 3 6. 1 0. 2 I m m ersi o n . . . 6 5 6. 1 0. 3 Virt u al E m b o di m e nt . . . 6 6 6. 1 1 E x pl or at or y St u d y . . . 7 0 6. 1 2 Dis c ussi o n . . . 7 3
7 C o n cl u si o n 7 8
7. 1 R es e ar c h Q u esti o ns . . . 7 8
7. 2 A d diti o n al Fi n di n gs . . . 8 1
7. 3 F ut ur e Wor k . . . 8 4
7. 4 Fi n al T h o u g hts . . . 8 6
B i bli o g r a p h y 8 7
A p p e n di c e s 9 3
A M ai n E v al u a ti o n A p p e n di c e s 9 4
A. 1 E x p eri m e nt S e q u e n c e Ta bl es . . . 9 5
A. 2 E x p eri m e nt C h e c klist . . . 9 6
A. 3 O S O I nstr u cti o ns . . . 1 0 1
A. 4 O S O I nstr u cti o ns . . . 1 0 2
A. 5 C o ns e nt f or m . . . 1 0 3
Int r o d u c ti o n
C o n n e cti n g dir e ctl y wit h ot h ers o ver dist a n c e h as b e c o m e a n i m p ort a nt p art of p e o pl es e ver y d a y lif e. W h er e p e o pl e pr e vi o usl y h a d t o w ait t o m e et i n p ers o n, t e xti n g a n d c alli n g n o w a d a ys all o ws p e o pl e t o dir e ctl y c o m m u ni- c at e wit h e a c h ot h er fr o m a dist a n c e, all o wi n g t h e m t o p erf or m t as ks t h at pr e vi o usl y r e q uir e d t o tr a vel.
Alt h o u g h t e xti n g a n d c alli n g h as b e c o m e p art of o ur d a y t o d a y pri v at e a n d w or k lif e, c o m p ar e d t o b ei n g p h ysi c all y pr es e nt, t his t e c h n ol o g y is still li mit e d i n t h e w a y t h at c all ers ar e p er c ei ve d a n d c a n a ct i n e a c h ot h er’s e n vir o n m e nt. Tel e pr es e n c e s yst e ms tr y t o cl os e t his g a p. T h e ulti m at e g o al is t o m a ke a us er f e el li ke h e or s h e is a ct u all y pr es e nt wit h t h e ot h er p ers o n i n t h eir e n vir o n m e nt.
C urr e nt r es e ar c h t o w ar ds t el e pr es e n c e s yst e ms c a n b e di vi d e d i nt o t w o c at e g ori es; c o nf er e n ci n g t el e pr es e n c e s yst e ms, w h er e t h e f o c us li es o n m a ki n g r e m ot e p e o pl e f e el pr es e nt wit h e a c h ot h er, a n d c oll a b or ati ve t el e pr es e n c e s yst e ms t h at b esi d es m a ki n g p e o pl e f e el pr es e nt t o g et h er als o f o c us o n m a k- i n g t h e m f e el pr es e nt i n t h e r e m ot e e n vir o n m e nt.
I n c o nf er e n ci n g t el e pr es e n c e s yst e ms, t h e f o c us li es o n s u p p orti n g c o n- vers ati o ns b et we e n p e o pl e b y pr o vi di n g t o ols t o b ett er e x pr ess t h e ms el ves c o m p ar e d t o a si m pl e p h o n e c all, f or e x a m pl e b y o ff eri n g hi g h- q u alit y a u di o a n d hi g h- d e fi niti o n lif e-si z e vi d e o ( Fel d m a n n et al., 2 0 1 0) or st er e os c o pi c dis pl a ys ( M ai m o n e a n d F u c hs, 2 0 1 1). A t y pi c al a p pli c ati o n of t h os e s yst e ms is oft e n f o u n d i n c o nf er e n ci n g o ver dist a n c e f or c o m p a ni es b ut als o i n s o ci al c o nt e xts li ke ( vi d e o-) c alli n g wit h fri e n ds a n d f a mil y. C o nf er e n ci n g t el e pr es- e n c e is m ostl y i m pl e m e nt e d i n a s y m m etri c al f as hi o n, w hi c h m e a ns t h at at b ot h r e m ot e l o c ati o ns p e o pl e ar e c a pt ur e d a n d vis u ali z e d i n t h e s a m e w a y, as t h er e is n or m all y n o r e as o n t o bi as t h e s yst e m i n f a v or f or s o m e b o d y b y f or e x a m pl e gi ve n o n e us er a c c ess t o t o ols t h e ot h er d o es n’t h a ve; all p arti ci p a nts ar e r e g ar d e d e q u al.
I n t his t h esis we f o c us o n c oll a b or ati ve t el e pr es e n c e s yst e ms. T h os e
s yst e ms s u p p ort c oll a b or ati o n b et we e n p e o pl e b y virt u all y tr a ns p orti n g a
r e m ot e visit or i nt o t h e visit e e’s e n vir o n m e nt. H er e b y t h e f o c us i s l ess o n c o n vers ati o n a n d m or e o n a cti o n; t h e r e m ot e visit or s h o ul d b e a bl e t o o p er at e i n t h e visit e e’s e n vir o n m e nt as if t h e y wer e r e all y t h er e. T y pi c al a p pli c ati o ns c a n b e f o u n d i n us e c as es w h er e a l o c al p ers o n r e q uir es h el p fr o m a r e m ot e p ers o n, f or e x a m pl e c ar r e p airs, tr ai ni n g a n d i n vesti g ati o n w or k. I n t h es e s c e n ari os, t h e r e m ot e visit or n e e ds a c c ess t o t h e e n vir o n m e nt of t h e visit e e, b ut n ot t h e ot h er w a y ar o u n d. T h er ef or e, c oll a b or ati ve t el e pr es e n c e s yst e ms ar e i d e all y as y m m etri c al ( St e e d et al., 2 0 1 2). T h e visit e e f or e x a m pl e n e e ds h el p wit h t h eir c ar or wit h t h e cri m e s c e n e t h e y ar e at, w hil e t h e e n vir o n m e nt of t h e visit or is of l ess i nt er est t o t h e visit e e. I. e., t h e visit or n e e ds t o f e el pr es e nt i n t h e visit e e’s e n vir o n m e nt.
T h e m ai n as p e ct t h at c o ntri b ut es t o t his f e eli n g of pr es e n c e is i m m ersi o n ( Sl at er, 2 0 0 3). A s yst e m t h at is m or e i m m ersi ve will m a ke a r e m ot e visit or f e el m or e pr es e nt t h a n a l ess i m m ersi ve s yst e m. T h e i m m ersi ve n ess of a s yst e m c a n n ot b e e x pr ess e d o n a o n e- di m e nsi o n al s c al e, as it c o nsists of m ulti pl e as p e cts t h at t o g et h er c o ntri b ut e t o t h e o ver all i m m ersi ve n ess. I n t h e n e xt s e cti o n we will dis c uss t h os e as p e cts i n m or e d et ail.
1. 1 Tel e p r e s e n c e S y s t e m D e si g n A s p e c t s
We r e c o g ni z e t hr e e m ai n as p e cts t h at c o ntri b ut e t o a n i m m ersi ve t el e pr es- e n c e s yst e m; m o bilit y at t h e r e m ot e l o c ati o n, a n i m m ersi ve vi e w o n t h e r e m ot e e n vir o n m e nt a n d t h e a bilit y t o h a ve a n e ff e ct o n t h e r e m ot e e n vi- r o n m e nt. T h os e as p e cts ar e us e d t hr o u g h o ut t his t h esis t o dis c uss ot h er t el e pr es e n c e s yst e ms a n d m a ke d esi g n d e cisi o ns f or o ur o w n s yst e m. We will n o w dis c uss e a c h as p e ct i n m or e d et ail b y usi n g e x a m pl es fr o m pr e vi o us w or k.
M o bili t y a t t h e r e m o t e l o c a ti o n All o wi n g t h e visit or t o vi e w t h e s c e n e i n d e p e n d e ntl y i nst e a d of b ei n g fi x e d t o a st ati c vi e w or t o a l o c al us er, is a n i m p ort a nt as p e ct i n t el e pr es e n c e s yst e ms t h at f o c us o n r e m ot e c oll a b or ati o n.
It all o ws us ers t o si m ult a n e o usl y w or k i n di ff er e nt p arts of t h e s c e n e a n d gi ves t h e r e m ot e visit or m or e sit u ati o n al a w ar e n ess ( F uss ell, S etl o c k, a n d Kr a ut, 2 0 0 3).
Di ff er e nt t y p es of vi e w i n d e p e n d e n c e h a ve b e e n e x pl or e d o ver t h e ye ars.
O n e s ol uti o n is t o p h ysi c all y m a ni p ul at e t h e r e m ot e c a m er a, a p o p ul ar i m pl e-
m e nt ati o n of t his ar e t el e pr es e n c e r o b ots. T h e y us u all y c o nsist of a c a m er a
a n d s cr e e n o n w h e els t h at c a n b e c o ntr oll e d b e t h e r e m ot e visit or o ver t h e
i nt er n et ( Fi g ur e 1. 1). A vi d e o str e a m of t h e visit or’s f a c e is dis pl a ye d o n
t h e r o b ot’s s cr e e n w hil e t h e r o b ot c a pt ur es i m a g es of t h e e n vir o n m e nt a n d
s e n ds t h e m b a c k t o t h e visit or. T his w a y a r e m ot e visit or h as t h e a bilit y
t o n a vi g at e a s p a c e a n d g et a s e ns e of t h e e n vir o n m e nt. A l ot of di ff er e nt
t el e pr es e n c e r o b ots h a ve b e e n d e vel o p e d, m ost of t h e m ar e als o c o m m er ci all y
Fi g ur e 1. 1: A D o u bl e t el e pr es e n c e r o b ot.
a v ail a bl e ( Krist o ff erss o n, C or a d es c hi, a n d L o ut fi, 2 0 1 3). Alt h o u g h t el e pr es- e n c e r o b ots gi ve a r e m ot e visit or a p h ysi c al b o d y a n d c o m pl et e fr e e d o m t o n a vi g at e t h e e n vir o n m e nt, it als o h as its dr a w b a c ks. A t el e pr es e n c e r o b ot c a n b e c o nsi d er e d cl u n k y, as t h e u n a w ar e n ess of t h e visit or a b o ut t h e r o b ot’s b o d y m e a ns it c o ul d e asil y b u m p i nt o t hi n gs or g et st u c k s o m e w h er e. Als o t h e s p e e d a n d t h e a m o u nt of p ers p e cti ves t h e r o b ot all o ws f or ar e oft e n li mit e d.
First- p ers o n o m ni dir e cti o n al vi d e o
V oi c e c o m m u ni c ati o n
B o d y G h o st
Fir
C a pt ur e o m ni dir e cti o n al i m a g e
I m a g e st a bili z ati o n R e c o n str u ct virt u al vi s u al s p a c e Vi e wi n g i nt er a cti o n
Fi g ur e 1. 2: O ver vi e w of J a c kI n H e a d ( K as a h ar a a n d R e ki m ot o, 2 0 1 5).
O t h er s ol uti o ns m a ke us e of a s y nt h esi z e d vi e w b as e d o n i m a g er y c a p-
t ur e d fr o m t h e visit e e’s p ers p e cti ve. I n s u c h a s c e n ari o, a l o c al p ers o n we ars
a c a m er a t h at str e a ms a first- p ers o n p ers p e cti ve t o t h e visit or. T his li mits
t h e n a vi g ati o n al fr e e d o m of t h e visit or, alt h o u g h s ol uti o ns h a ve b e e n m a d e th at still gi ve t h e visit or s o m e d e gr e e of i n d e p e n d e n c e fr o m t h e l o c al p er- s o n. For e x a m pl e, J a c kI n H e a d ( K as a h ar a a n d R e ki m ot o, 2 0 1 5) us es a 3 6 0
◦d e gr e e c a m er a m o u nt e d o n a p ers o n’s h e a d t o all o w a r e m ot e visit or t o l o o k ar o u n d i n t h e p ers o n’s e n vir o n m e nt ( Fi g ur e 1. 2). Si mil arl y, G a o et al. ( 2 0 1 6) m a d e a s yst e m t h at us es a tr a c ke d h e a d m o u nt e d d e pt h c a m er a i nst e a d of a 3 6 0
◦c a m er a. T h e i m a g es ar e ori e nt e d t o t h e visit or i n s u c h a w a y t h at w h e n t h e w or ker t ur ns his / h er h e a d, t h e pr oj e ct e d i m a g e at t h e visit or w o ul d t ur n as well a n d t h er ef or e r e q uir es t h e visit or t o t ur n his / h er h e a d t o f oll o w t h e i m a g e. T his d o es n’t gi ve t h e visit or a n y e xt e n d e d n a vi g ati o n al fr e e d o m b ut h el ps t h e visit or t o ke e p tr a c k of t h e w or ker’s c urr e nt ori e nt ati o n.
A n ot h er s ol uti o n is d e m o nstr at e d b y Tait a n d Billi n g h urst ( 2 0 1 5), w hi c h us e d a st ati c all y m o u nt e d d e pt h c a m er a a b o ve a t a bl e t o cr e at e a li ve 3 D s c a n of t h e w or k pl a c e. T h e visit or w as t h e n fr e e t o n a vi g at e t his li ve 3 D s c a n wit h a virt u al c a m er a usi n g a d es kt o p us er i nt erf a c e. T h e y r es e ar c h e d t h e e ff e ct of t his i n a c oll a b or ati ve s etti n g a n d s h o we d t h at it r es ults i n
“f ast er t as k c o m pl eti o n ti m e, m or e c o n fi d e n c e fr o m us ers, a n d a d e cr e as e i n t h e a m o u nt of ti m e s p e nt c o m m u ni c ati n g ver b all y d uri n g t h e t as k ”.
A n i m m e r si v e vi e w o n t h e r e m o t e e n vi r o n m e nt T h e w a y t h e r e- m ot e e n vir o n m e nt is pr es e nt e d t o t h e visit or h as e ff e ct o n t h e t el e pr es e n c e e x p eri e n c e as well. A m or e i m m ersi ve dis pl a y m a kes t h e visit or f e el m or e pr es e nt i n a virt u al e n vir o n m e nt ( B a ˜n os et al., 2 0 0 4). A dis pl a y wit h a wi d er fi el d of vi e w will f or e x a m pl e fill m or e of t h e visit or’s vi e w wit h t h e r e m ot e e n vir o n m e nt w hi c h dr a ws t h eir att e nti o n a w a y fr o m t h e l o c al e n vi- r o n m e nt. Tel e pr es e n c e s yst e ms h a ve us e d di ff er e nt dis pl a y s yst e ms t h at c a n b e disti n g uis h e d b y t h e d e gr e e of i m m ersi o n t h e y pr o vi d e t h e visit or. E ar- li er s yst e ms us e d 2 D m o nit ors o n w hi c h a vi d e o str e a m of t h e l o c al us er is dis pl a ye d, li ke t h e H a n ds O n Vi d e o s yst e m ( H u a n g a n d Al e m, 2 0 1 1). N e wer s yst e ms tr y t o i m m ers e t h e visit or m or e i n t h e r e m ot e e n vir o n m e nt, t his is oft e n d o n e wit h h e a d m o u nt e d dis pl a ys as s h o w n i n b y Te c c hi a, Al e m, a n d H u a n g ( 2 0 1 2). Ot h er dis pl a y s yst e ms ar e als o us e d, li ke C A V E pr oj e cti o n s yst e ms w h er e m ulti pl e w alls of a r o o m ar e pr oj e ct e d wit h li ve i m a g es fr o m a r e m ot e e n vir o n m e nt as s h o w n b y K o mi y a m a, Mi y a ki, a n d R e ki m ot o ( 2 0 1 7).
T h os e s yst e ms ar e oft e n us e d i n c o m bi n ati o n wit h h e a d tr a c ki n g t o m a ke t h e m o ve m e nt of t h e i m a g es m at c h t h e visit or’s h e a d m o ve m e nts.
H a v e a n e ff e c t o n t h e r e m o t e e n vi r o n m e nt B esi d es all o wi n g a r e m ot e visit or t o o bs er ve a n e n vir o n m e nt, t el e pr es e n c e s yst e ms us u all y als o all o w visit ors t o h a ve a c ert ai n d e gr e e of e ff e ct o n t his e n vir o n m e nt. T his c a n b e d o n e b y a d di n g virt u al el e m e nts t o t h e e n vir o n m e nt or b y p h ysi c all y m a ni p ul ati n g t h e e n vir o n m e nt.
Virt u al el e m e nts c a n b e us e d f or s u p p orti n g i nt er a cti o n b et we e n p e o pl e
( a) ( b) ( c)
F i g ur e 1. 3: Vari o us st yl es of virt u al assist a n c e o verl a ys. a) S p ati al r e m ot e p oi nti n g ( K as a h ar a a n d R e ki m ot o, 2 0 1 4); b) R e m ot e h el p er’s h a n ds dis- pl a ye d o n a m o nit or ( Te c c hi a, Al e m, a n d H u a n g, 2 0 1 2); c) R e m ot e h el p er’s h a n ds dis pl a ye d i n a H M D ( G a o et al., 2 0 1 6).
b y s h o wi n g t hi n gs t o t h e l o c al us ers t hr o u g h a dis pl a y. T his c a n b e d o n e b y o verl a yi n g i nstr u cti o ns fr o m t h e r e m ot e visit or o nt o t h e l o c al us er’s vi e w b y f or e x a m pl e usi n g a m o nit or i n fr o nt of t h e l o c al us er or usi n g a h e a d m o u nt e d dis pl a y ( H M D). T h e i nstr u cti o ns c a n c o nsist of a n n ot ati o n-st yl e el e m e nts, li ke arr o ws a n d m ar kers as s h o w n b y K as a h ar a a n d R e ki m ot o ( 2 0 1 4) ( Fi g ur e 1. 3 a). T h os e ar e m ai nl y us e d t o dir e ct t h e l o c al us er t o c ert ai n el e m e nts i n t h e e n vir o n m e nt.
To assist t h e l o c al us er i n t as ks t h at r e q uir e m or e fi n e c o ntr ol, a n o verl a y of t h e r e m ot e us er’s h a n ds ar e oft e n us e d, as s h o w n b y A m or es, B e n a vi d es, a n d M a es ( 2 0 1 5), Te c c hi a, Al e m, a n d H u a n g ( 2 0 1 2), G a o et al. ( 2 0 1 6), a n d H u a n g a n d Al e m ( 2 0 1 1) ( Fi g ur e 1. 3 b a n d Fi g ur e 1. 3 c). T his all o ws t h e r e m ot e us ers t o g ui d e t h e l o c al us er i n a m or e n at ur al w a y w h e n d e ali n g wit h o bj e ct m a ni p ul ati o n t as ks, li ke r e p air or c o nstr u cti o n t as ks b y usi n g g est ur es o n e w o ul d i nt uiti vel y us e.
P h ysi c al m a ni p ul ati o n is a n ot h er w a y t o m a ke visit ors h a ve a n e ff e ct o n t h e r e m ot e e n vir o n m e nt, f or e x a m pl e b y all o wi n g t h e m t o m o ve o bj e cts.
B y t h e ms el ves t el e pr es e n c e r o b ots alr e a d y m a ni p ul at e t h e e n vir o n m e nt, as t h e y ar e a p h ysi c al e ntit y t h at m a kes p art of a n e n vir o n m e nt. B y dri vi n g ar o u n d ( a n d m a y b e e ve n b y b u m pi n g i nt o t hi n gs) t h e y all o w a r e m ot e visit or t o h a ve a n e ff e ct o n t his e n vir o n m e nt. B y e q ui p pi n g a t el e pr es e n c e r o b ot wit h a d e di c at e d c o m p o n e nt f or o bj e ct m a ni p ul ati o n, li ke a r o b oti c ar m, t h e m e a ni n gf ul n ess of t h os e m a ni p ul ati o ns c a n b e i n cr e as e d ( P a ul os a n d C a n n y, 2 0 0 1).
B e n a vi d es, A m or es, a n d M a es ( 2 0 1 5) m a d e a s yst e m c all e d Re m ot-I O t h at all o ws a r e m ot e visit or t o m a ni p ul at e d e di c at e d i nt er n et c o n n e ct e d o bj e cts b y usi n g virt u al g est ur es, wit h o ut r e q uiri n g a p h ysi c al e m b o di m e nt.
T h e y d e m o nstr at e t h e s yst e m wit h a r a di o t h at is c o n n e ct e d t o t h e s yst e m.
T h e r e m ot e us er c a n t ur n t h e k n o bs b y usi n g h a n d g est ur es t h at ar e c a pt ur e d
a n d tr a nsl at e d i nt o p h ysi c al r ot ati o ns o n t h e r a di o k n o bs.
T h e c urr e nt st at e of t h e art i n t h e fi el d of mi x e d r e alit y t el e pr es e n c e is H ol o p ort ati o n ( Orts- Es c ol a n o et al., 2 0 1 6). T his s yst e m us es c o ns u m er gr a d e A R a n d V R dis pl a y t e c h n ol o gi es i n c o m bi n ati o n wit h a n e w 3 D c a pt ur e s yst e m t o c a pt ur e f ull 3 6 0
◦3 D m o d els of p e o pl e a n d o bj e cts a n d vis u ali z e t h e m at a r e m ot e l o c ati o n i n r e al-ti m e. All of t his is d o n e wit h t h e g o al of cr e ati n g a n e x p eri e n c e t h at r es e m bl es p h ysi c al pr es e n c e as cl os e as p ossi bl e.
Us ers at b ot h l o c ati o ns us e a H ol o L e ns H M D t o g et a n i n d e p e n d e nt vi e w o n e a c h ot h er’s e n vir o n m e nt. Alt h o u g h n o p h ysi c al m a ni p ul ati o n of e a c h ot h er’s e n vir o n m e nt is p ossi bl e, t h e hi g h r es ol uti o n s c a ns of t h e us ers all o w f or ver y pr e cis e g est ur al i nstr u cti o ns t h at ar e pr o p erl y ali g n e d wit h t h eir c o nt e xt. Orts- Es c ol a n o et al. ( 2 0 1 6) c o n d u ct e d a pr eli mi n ar y q u alit ati ve us er st u d y wit h 1 0 p arti ci p a nts, usi n g a s o ci al i nt er a cti o n a n d a p h ysi c al o bj e ct m a ni p ul ati o n t as k. T h e p arti ci p a nts n ot e d h o w m or e i nt uiti ve it is c o m p ar e d t o n or m al p h o n e c alls or usi n g S k y p e.
B esi d es pr o vi di n g a n i m m ersi ve e x p eri e n c e, a s u c c essf ul t el e pr es e n c e s ys- t e m will als o h a ve t o b e a c c essi bl e. We ai m t o m a ke a s yst e m t h at c a n b e us e d b y ot h er r es e ar c h ers as well, t h er ef or e it s h o ul d b e e as y f or p e o pl e t o s et u p a n e n vir o n m e nt f or a virt u al visit, as well as f or p e o pl e t o g o o n a vir- t u al visit. N e xt t o b ei n g e as y t o s et- u p, t h e s yst e m s h o ul d als o b e e as y a n d i nt uiti ve t o us e, f or w hi c h we c a n l o o k at f a ct ors li ke t h e us ers’ p erf or m a n c e, us a bilit y a n d pr es e n c e.
1. 2 R e s e a r c h Q u e s ti o n s
T his t h esis is a b o ut t h e d esi g n, i m pl e m e nt ati o n a n d e v al u ati o n of a mi x e d r e alit y t el e pr es e n c e s yst e m c all e d O pe nI M P R E S S , w hi c h st a n ds f or o pe n s o urce i m m ersi v e pres e n ce s yst e m. T h e g o al is t o cr e at e a t el e pr es e n c e s yst e m t h at is b ot h i m m ersi ve a n d a c c essi bl e. T his p uts O p e nI M P R E S S i n li n e wit h t h e w or k t h at is d o n e o n H ol o p ort ati o n b ut w h er e H ol o p ort ati o n ai ms t o b e a n e x er cis e i n t h e fi el d of c o m p ut er visi o n a n d c o m p ut er gr a p hi cs, f o c usi n g o n r e alis m b y p us hi n g t e c h n ol o gi es t o t h eir li mit, o ur w or k d o es n ot tr y t o i m pr o ve o n t h at. O p e nI M P R E S S f o c us es i nst e a d o n t h e (s o ci al) i nt er a cti o n a n d c oll a b or ati o n as p e cts t h at s yst e ms i n t his li n e of r es e ar c h m a y b e us e d f or. T h e g o al is t o m a ke a n o p e n s yst e m t h at c a n b e us e d as a b asis b y ot h er r es e ar c h ers as well. We tr y t o d o t h at b y m a ki n g a s yst e m t h at o nl y us es o ff t h e s h elf c o ns u m er h ar d w ar e, is fl e xi bl e a n d e asil y r e c o n fi g ur a bl e a n d m a kes us e of o p e n s o ur c e c o d e or fr e el y a v ail a bl e s oft w ar e w hi c h m a kes it a c c essi bl e f or a n y b o d y t h at w a nts t o us e it.
T his l e a ds t o t h e f oll o wi n g first m ai n r es e ar c h q u esti o n a n d s u b q u esti o ns:
R Q 1. H o w t o cre at e a st at e of t h e art e n d-t o- e n d mi x e d re alit y t el e pres e n ce s yst e m t h at is i m m ersi v e a n d a ccessi bl e ?
• R Q 1. 1. H o w t o cr e at e a n i m m ersi ve t el e pr es e n c e e x p eri e n c e ?
• R Q 1. 2. W h at c o m p o n e nts c a n b e us e d, w hil e ke e pi n g it a c c essi bl e ?
• R Q 1. 3. H o w t o c o ver a l ar g e w or ki n g ar e a ?
• R Q 1. 4. H o w t o m a ke t h e s yst e m e as y t o s et- u p a n d us e ?
C o m p ar e d t o pr e vi o us t el e pr es e n c e s yst e ms a n d r es e ar c h, O p e nI M P R E S S us es p uts pr e vi o usl y t est e d c o m p o n e nts i n a di ff er e nt c o nt e xt t h a n t h e o n es t h e y h a ve ori gi n all y b e e n t est e d i n. T his m e a ns t h at a l ot of k n o wl e d g e a b o ut h o w t h os e as p e cts h el p or hi n d er c oll a b or ati o n m a y n ot a p pl y a n y- m or e. For e x a m pl e; V R h as b e e n us e d i n t el e pr es e n c e s yst e ms b ef or e, b ut i n m ost c as es o nl y i n c o m bi n ati o n wit h a first- p ers o n vi e w o n t h e r e m ot e e n vir o n m e nt w hi c h, at t h e m ost o nl y all o ws t h e r e m ot e visit or t o c h a n g e t h e r ot ati o n of t h e vi e w ( G a o et al., 2 0 1 6). F ull vi e w i n d e p e n d e n c e h as b e e n t est e d o nl y i n c o m bi n ati o n wit h a 2 D m o nit or s et- u p, n ot i n c o m bi n ati o n wit h V R ( Tait a n d Billi n g h urst, 2 0 1 5). O verl a yi n g g est ur es fr o m t h e visit or o nt o t h e visit e e’s vi e w h as b e e n t est e d as well, b ot h i n V R a n d wit h 2 D i n- t erf a c es b ut n ot wit h (f ull) vi e w i n d e p e n d e n c e ( Te c c hi a, Al e m, a n d H u a n g, 2 0 1 2; A m or es, B e n a vi d es, a n d M a es, 2 0 1 5; F uss ell, S etl o c k, Ya n g, et al., 2 0 0 4). A b ett er u n d erst a n di n g o n h o w t h os e as p e cts i n fl u e n c e c oll a b or ati o n w o ul d b e n e c ess ar y t o pi n p oi nt w h at as p e cts ar e w ort h f urt h er r es e ar c h a n d d e vel o p m e nt e ff orts.
Als o, t h os e c o m p o n e nts h a ve al w a ys b e e n t est e d i n sli g htl y di ff er e nt s c e- n ari os, wit h di ff er e nt t y p es of s yst e ms a n d t as ks. We n o w h a ve t h e o p p ort u- nit y t o b ett er c o m p ar e t h e e ff e cts of t h os e c o m p o n e nts, b y e v al u ati n g t h e m w hil e ke e pi n g v ari a bl es li ke t h e s yst e m a n d t as ks t h e s a m e.
T his l e a ds us t o t h e s e c o n d m ai n r es e ar c h q u esti o n, w hi c h is: R Q 2. H o w d o t h e t hree t el e pres e n ce s yst e m d esi g n as pects i n fl u e n ce t h e perf or m a n ce a n d e x peri e n ce of t h e us ers i n a c oll a b or ati v e pr o bl e m s ol vi n g t as k ?
To a ns wer t his q u esti o n we f o c us o n t h e as p e cts pr es e nt e d i n s e cti o n 1. 1.
B e c a us e O p e nI M P R E S S d o es n ot ( yet) h a ve t h e a bilit y t o l et t h e visit or h a ve a p h ysi c al e ff e ct i n t h e o n-sit e l o c ati o n dir e ctl y, we li mit t h e as p e ct of a ff ect t h e re m ot e e n vir o n m e nt t o vis u al e ff e cts o nl y. W hi c h m e a ns i n t his c as e t h e vis u ali z ati o n of t h e visit or i n t h e visit e e’s e n vir o n m e nt. T his r es ults i n t h e f oll o wi n g t hr e e s u b q u esti o ns:
• R Q 2. 1. H o w d o es vi e w i n d e p e n d e n c e i n fl u e n c e t h e p erf or m a n c e a n d e x p eri e n c e of t h e us ers i n a c oll a b or ati ve s etti n g ?
• R Q 2. 2. H o w d o es usi n g a n i m m ersi ve dis pl a y i n fl u e n c e t h e p erf or- m a n c e a n d e x p eri e n c e of t h e us ers i n a c oll a b or ati ve s etti n g ?
• R Q 2. 3. H o w d o es gi vi n g t h e visit or a virt u al e m b o di m e nt i n fl u e n c e
t h e p erf or m a n c e a n d e x p eri e n c e of t h e us ers i n a c oll a b or ati ve s etti n g ?
T h os e q u esti o ns ar e a ns wer e d wit h a q u a ntit ati ve us er st u d y d et ail e d i n
c h a pt er 6.
B a c k g r o u n d: Fr e e Vi e w p oi nt Vi d e o
M a n y r e m ot e c oll a b or ati o n t o ols all o w t h e visit or t o s e e t h e l o c al e n vir o n- m e nt t hr o u g h a fi x e d p ers p e cti ve. T his c a n b e fr o m a fi x e d c a m er a i n t h e r o o m or fr o m a us er- w or n c a m er a c a pt uri n g t h e visit e e’s vi e w. For O p e n- I M P R E S S o n t h e ot h er h a n d, we tr y t o cr e at e a n i m m ersi ve t el e pr es e n c e e x p eri e n c e, w hi c h i n cl u d es gi vi n g t h e visit or m o bilit y at t h e l o c al e n vir o n- m e nt. T h e gr o u p of t e c h n ol o gi es t h at f o c us o n t h e cr e ati o n a n d pl a y b a c k of vi d e o c o nt e nt t h at e n a bl es vi e wers t o c h o os e t h eir o w n p ers p e cti ve d uri n g pl a y b a c k is c all e d Free Vi e w p oi nt Vi d e o ( F V V).
T his as p e ct is i d e nti fi e d as a m aj or c h all e n g e f or t h e i m pl e m e nt ati o n of t his s yst e m a n d t h er ef or e dis c uss e d i n m or e d et ail
1. S m oli c ( 2 0 1 1) di vi d es t h e F V V pi p eli n e i nt o s e ve n st e ps ( Fi g ur e 2. 1) of w hi c h t h e m ost i m p ort a nt o n es ar e dis c uss e d i n di vi d u all y b el o w.
Fi g ur e 2. 1: T h e F V V pi p eli n e ( S m oli c, 2 0 1 1).
In t his c h a pt er, we l o o k at t h e m ost r el e v a nt as p e cts of fr e e vi e w p oi nt vi d e o f or o ur t el e pr es e n c e s yst e m. E a c h as p e ct is e x pl ai n e d a n d e x a m pl es
1M ost of t his s e cti o n h as b e e n t a ke n fr o m H ar ms e n ( 2 0 1 7).
ar e pr es e nt e d of t h eir i m pl e m e nt ati o ns i n ot h er s yst e ms. S e cti o n 2. 1 A cq ui- siti o n f o c us es o n h o w F V V is c a pt ur e d b y e x pl ai ni n g t h e di ff er e nt ki n d of c a m er a’s a n d s et- u ps. S e cti o n 2. 2 Pr ocessi n g pr es e nt di ff er e nt t e c h ni q u es of h o w t h e a c q uir e d d at a c a n b e pr o c ess e d a n d r e pr es e nt e d. S e cti o n 2. 3 Dis pl a y pr es e nts di ff er e nt o pti o ns of h o w t o dis pl a y t h e F V V d at a. S e cti o n 2. 4 c o n- cl u d es t h e c h a pt er b y s u m m ari zi n g t h e pr es e nt e d t e c h ni q u es a n d p oi nti n g o ut w hi c h ar e t h e m ost r el e v a nt f or O p e nI M P R E S S.
2. 1 A c q ui si ti o n
To c a pt ur e F V V o n e c a n us e a n o ut w ar d or a n i n w ar d a c q uisiti o n t o p ol o g y ( Fi g ur e 2. 2). A n o ut w ar d t o p ol o g y m e a ns t h at t h e c a pt uri n g d e vi c es ar e f a c e d o ut w ar ds fr o m a c e nt er p oi nt (li ke 3 6 0
◦c a m er as). I n t his c as e t h e vi e wer is li mit e d t o r ot ati o n al p ers p e cti ve c h a n g es o nl y a n d o n e c o ul d ar- g u e w h et h er t his c a n b e c o nsi d er e d Fr e e Vi e w p oi nt Vi d e o as t h e vi e w p oi nt r e m ai ns t h e s a m e. Wit h a n i n w ar d t o p ol o g y t h e c a pt uri n g d e vi c es ar e all p oi nt e d at t h e s a m e s u bj e ct fr o m di ff er e nt l o c ati o ns ar o u n d t h at s u bj e ct, s o i nst e a d of all o wi n g t h e vi e wer t o l o o k ar o u n d fr o m t h e s a m e p oi nt i n s p a c e, t h e vi e wer c a n “tr a vel ” t o di ff er e nt p oi nts i n s p a c e a n d o bs er ve t h e s u bj e ct fr o m t h er e.
Fi g ur e 2. 2: L eft: O ut w ar d F V V; Ri g ht: I n w ar d F V V ( L e e, Ta b at a b ai, a n d T as hir o, 2 0 1 5).
A p o p ul ar w a y t o c a pt ur e F F V c o nt e nt is t o us e o n e or m or e d esi g n at e d d e pt h c a m er as t h at n e xt t o c a pt uri n g a r e g ul ar 2 D i m a g e als o c a pt ur e a s e p ar at e d e pt h m a p t h at gi ves e a c h pi x el i n t h e 2 D i m a g e a d e pt h c o or di n at e.
T his pr o p ert y m a kes it es p e ci all y i nt er esti n g f or F V V as t his m e a ns o n e c a n cr e at e a v ol u m etri c c a pt ur e of a s c e n e i n r e al-ti m e. T h er e ar e v ari o us d e pt h c a m er as o n t h e m ar ket t h at us e di ff er e nt t e c h n ol o gi es t o d et er mi n e t h e d e pt h i nf or m ati o n. L a n g m a n n ( 2 0 1 4) c o n d u ct e d a c o m p ar ati ve st u d y, a n al y zi n g t h e di ff er e n c es b et we e n v ari o us m o d els of d e pt h c a m er as.
A d e pt h c a m er a t h at is us e d e xt e nsi vel y is t h e Mi cr os oft Ki n e ct ( K ust er
et al., 2 0 1 1) ( H a us wi es n er, Str a k a, a n d R eit m a yr, 2 0 1 1) ( C a z a mi as a n d R aj,
2 0 1 6). I z a di et al. ( 2 0 1 1) pr es e nt a s yst e m t h at g e n er at es a 3 D m es h r e pr e-
s e nt ati o n of a s c e n e b as e d o n t h e d e pt h d at a fr o m t h e Ki n e ct s e ns or. T his
t e c h ni q u e r e q uir es t h e us er t o m o ve t h e s e ns or ar o u n d t h e s c e n e s o it c a n b e c a pt ur e d fr o m m ulti pl e a n gl es. B y f usi n g t h os e di ff er e nt vi e ws t o g et h er, a hi g h- q u alit y m o d el of t h e s c e n e is g e n er at e d. B e c a us e t h e s c e n e is c a p- t ur e d fr o m m ulti pl e a n gl es o ver a n e xt e n d e d p eri o d of ti m e, t his t e c h ni q u e is c a p a bl e of pr o d u ci n g hi g h q u alit y m o d els b ut t h er ef or e is als o l ess i d e al f or r e al-ti m e str e a mi n g as it ass u m es n o o bj e cts ar e m o vi n g w hil e c a pt ur- i n g. T his t e c h ni q u e is us u all y a p pli e d w h e n cr e ati n g a m a p of a n i n d o or e n vir o n m e nt ( K h os h el h a m a n d El b eri n k, 2 0 1 2) ( H e nr y et al., 2 0 1 2).
S yst e ms m a d e t o c a pt ur e m o vi n g o bj e cts als o e xist; t h e y us u all y c o nsist of m ulti pl e Ki n e ct s e ns ors a n d h a ve b e c o m e p o p ul ar a m o n g r es e ar c h ers t o cr e at e F V V’s of o bj e cts a n d p ers o ns ( K ai n z et al., 2 0 1 2) ( D o c- O k, 2 0 1 4).
G o o gl e p ut t his i nt o pr a cti c e i n t h eir e x p eri m e nt c all e d “ Virt u al Art S es- si o ns ”
2, i n t his e x p eri m e nt a gr o u p of artists wer e as ke d t o cr e at e a s c ul p- t ur e i n virt u al r e alit y. N or m all y a s p e ct at or w o ul d n’t b e a bl e t o s e e b ot h t h e artist a n d t h e art w or k at t h e s a m e ti m e b e c a us e t h e y e xist i n di ff er e nt
“r e aliti es ”. To s ol ve t his, G o o gl e ( 2 0 1 6 b) r e c or d e d t h e artists v ol u m etri- c all y w hil e t h e y w or ke d o n t h eir pi e c e. T h os e r e c or di n gs wer e t h e n mi x e d t o g et h er wit h t h e virt u al e n vir o n m e nt a n d distri b ut e d o nli n e s o s p e ct at ors c a n w at c h F V V of t h e artists s c ul pti n g t h eir virt u al art w or ks i n mi d- air.
M ai m o n e a n d F u c hs ( 2 0 1 2 a) d e vel o p e d a s yst e m usi n g m ulti pl e Ki n e ct s e n- s ors f or r e al-ti m e v ol u m etri c 3 D c a pt uri n g of e n vir o n m e nts s p e ci fi c all y f or t el e pr es e n c e a p pli c ati o ns. Ki n e ct s e ns ors pr oj e ct i nfr ar e d li g ht p att er ns o nt o t h e e n vir o n m e nt i n or d er t o d et er mi n e its dist a n c e fr o m o bj e cts. T his c a n c a us e pr o bl e ms w h e n o n e us es m ulti pl e Ki n e cts i n t h e s a m e e n vir o n m e nt b e- c a us e a Ki n e ct t h e n c a n’t disti n g uis h its o w n p att er ns fr o m ot h er Ki n e ct’s p att er ns. To s ol ve t his pr o bl e m, t h e s yst e m us es a n I R i nt erf er e n c e r e d u ci n g m et h o d w hi c h is b as e d o n s h a ki n g t h e s e ns or ( M ai m o n e a n d F u c hs, 2 0 1 2 b) ( B utl er et al., 2 0 1 2). T h e s h a ki n g c a us es t h e p att er ns e mitt e d b y t h e ot h er Ki n e cts t o b e bl urr e d o ut, t h us i m pr o vi n g t h e d et e cti o n of t h e s e ns or’s o w n p att er n a n d t h er ef or e r e d u ci n g t h e i nt erf er e n c e.
2. 2 P r o c e s si n g
T h e c a pt ur e d s c e n e c a n b e r e pr es e nt e d i n a w a y s o m e w h er e b et we e n a n i m- a g e b as e d r e pr es e nt ati o n or a g e o m etr y b as e d r e pr es e nt ati o n ( S m oli c, 2 0 1 1), as s h o w n i n Fi g ur e 2. 3.
O n o n e e n d of t h e s p e ctr u m is t h e i m a g e- b as e d a p pr o a c h. I n t his a p- pr o a c h t h er e is n’t a n y k n o wl e d g e a b o ut t h e g e o m etr y of t h e s c e n e; i m a g es fr o m e a c h c a m er a ar e pr o c ess e d a n d tr a ns mitt e d as s e p ar at e vi d e os. I n- t er p ol ati o n c a n g e n er at e i nt er m e di at e vi e ws, b ut a hi g h c a pt ur e d e nsit y is r e q uir e d t o g et a n a c c e pt a bl e q u alit y.
2h t t p s : / / d e v e l o p e r s . g o o g l e . c o m / w e b / s h o w c a s e / 2 0 1 6 / a r t - s e s s i o n s
Fi g ur e 2. 3: 3 D s c e n e r e pr es e nt ati o ns f or 3 D V a n d F V V ( S m oli c, 2 0 1 1).
O n t h e ot h er e n d, wit h a g e o m etr y- b as e d a p pr o a c h t h e s c e n e is r e pr e- s e nt e d b y t hr e e di m e nsi o n al m es h es o n w hi c h i m a g e t e xt ur es ar e a p pli e d.
D e p e n di n g o n t h e m et h o d of a c q uisiti o n, t his r e q uir es a n a c c ur at e a n d r o- b ust 3 D r e c o nstr u cti o n of t h e s c e n e w hi c h c a n b e c h all e n gi n g b ut all o ws m or e fr e e d o m of n a vi g ati o n w h e n vi e wi n g. C arr a n z a et al. ( 2 0 0 3) m a d e o n e of t h e first att e m pts a n d d e vel o p e d a s yst e m t o cr e at e a g e o m etr y- b as e d r e pr es e nt ati o n of h u m a n a ct ors usi n g a s e ve n- c a m er a s et u p. A virt u al h u- m a n b o d y m o d el w as p ut i n t h e ri g ht p os e b y c o m p ari n g its sil h o u ett e t o t h e sil h o u ett es of t h e a ct or fr o m t h e di ff er e nt c a m er a a n gl es. Si n c e t h e n, t h os e c a pt ur e s yst e ms h a ve b e c o m e m or e a d v a n c e d, as s h o w n b y C oll et et al. ( 2 0 1 5). T h e y us e a n i n w ar d a c q uisiti o n t o p ol o g y wit h 1 0 6 c a m er as fr o m w hi c h hi g h- q u alit y 3 D tr a c ke d m es h es ar e g e n er at e d.
Alt er n ati ve a p pr o a c h es t h at li e i n b et we e n i m a g e a n d g e o m etr y- b as e d a p pr o a c h es ar e als o p ossi bl e; i nst e a d of usi n g m es h es, p oi nt cl o u ds c a n b e us e d t o r e pr es e nt g e o m etr y. P oi nt cl o u ds c a n b e g e n er at e d b y assi g ni n g e a c h pi x el of a 2 D c ol or i m a g e a d e pt h v al u e ( a n d t h us cr e ati n g a 3 D c o or di n at e) a n d pl a ci n g t h at pi x el i n 3 D virt u al s p a c e. S e ns ors li ke t h e Mi cr os oft Ki n e ct e x a ctl y pr o vi d e t his ki n d of d at a, t h us m a ki n g t his a p pr o a c h r el ati vel y e as y t o i m pl e m e nt. Usi n g p oi nt cl o u ds r e d u c es t h e c o m pl e xit y of t h e pr o c essi n g st e p dr asti c all y, as n o g e o m etr y h as t o b e g e n er at e d fr o m t his d at a. Als o, it is m or e f or gi vi n g w h e n c a pt uri n g c o m pl e x o bj e cts; “ Usi n g p oi nt cl o u ds e n a bl es t h e ali g n m e nt f or s c e n es f ull of o bj e cts t h at ar e di ffi c ult t o m o d el;
f or e x a m pl e, tr e es. ” ( Z h a o, Nist er, a n d Hs u, 2 0 0 5). T his is b e c a us e f e wer ass u m pti o ns ar e m a d e d uri n g t h e pr o c essi n g of t h e dis pl a ye d d at a a n d t h us r el ati vel y f e wer mist a kes ar e m a d e. T h e dis a d v a nt a g e of s ki p pi n g t h e 3 D g e- o m etr y r e c o nstr u cti o n st e p is t h at o bj e cts l o o k l ess s oli d b e c a us e t h e s p a c es i n b et we e n t h e p oi nts of t h e p oi nt cl o u d ar e n ot g etti n g fill e d i n.
I n or d er t o s e n d t h e d at a t h at is c a pt ur e d, it h as first t o b e e n c o d e d a n d
c o m pr ess e d. C o m pr essi o n is n e c ess ar y t o ke e p t h e d at a str e a m wit hi n t h e
b a n d wi dt h li mits. T h er e ar e di ff er e nt e n c o di n g a n d c o m pr essi o n m et h o ds a v ail a bl e f or di ff er e nt t y p es of Fr e e Vi e w p oi nt Vi d e o. For e n c o di n g g e o m- etr y b as e d F F V, t h e M P E G- 4 st a n d ar d pr o vi d es a n e xt e nsi o n c all e d t h e A ni m ati o n Fr a m e w or k e Xt e nsi o n ( A F X) w hi c h all o ws t o d e fi n e hi g h-l e vel g e o m etr y, t e xt ur e, v ol u m e, a n d a ni m ati o n c o m p o n e nts f or e n h a n c e d i nt er- a cti ve m ulti m e di a a p pli c ati o ns ( B o ur g es- S ´e ve ni er a n d J a n g, 2 0 0 4) ( S m oli c et al., 2 0 0 6). A l ot of r es e ar c h h as als o b e e n d o n e alr e a d y i nt o c o m pr essi o n al g orit h ms f or str e a mi n g v ol u m etri c p oi nt cl o u d vi d e o d at a: ( M or e n o, C h e n, a n d M. Li, 2 0 1 7) ( M or e n o a n d M. Li, 2 0 1 6) ( R us u a n d C o usi ns, 2 0 1 1) ( K a m- m erl et al., 2 0 1 2) ( M e hr otr a et al., 2 0 1 1) ( S o h n, B aj aj, a n d Si d d a v a n a h alli, 2 0 0 4).
2. 3 Di s pl a y
B ef or e 3 D d at a c a n b e dis pl a ye d it h as t o b e r e n d er e d. T his m e a ns t h at t h e 3 D d at a is t ur n e d i nt o i m a g e d at a t h at c a n b e dis pl a ye d o n f or e x a m pl e a m o nit or. 3 D vi d e o- g a m e e n gi n es li ke U nit y
Ror U nre al E n gi n e
Rar e ver y s uit a bl e f or t his, as t h eir m ai n t as k is c o n verti n g 3 D d at a i nt o 2 D i m a g e d at a t h at c a n b e dis pl a ye d o n s cr e e ns, b ut b esi d es t h at t h e y als o all o w f or i nt e gr ati o n of c ust o m f e at ur es, o ff er a l ot of s u p p ort vi a o nli n e f or u ms, ar e fr e e t o us e f or t h es e p ur p os es a n d o ff er e as y i nt e gr ati o n wit h virt u al r e alit y h e a ds ets.
Virt u al r e alit y ( V R) h e a ds ets or h e a d m o u nt e d dis pl a ys ( H M D) h a ve r e c e ntl y b e c o m e a p o p ul ar w a y of dis pl a yi n g virt u al e n vir o n m e nts. H M Ds all o w f or t h e us er t o b e c o m pl et el y i m m ers e d i n a virt u al w orl d b y m e a ns of a st er e os c o pi c dis pl a y t h at c o vers a bi g p art of t h e us er’s fi el d of vi e w a n d s e ns ors t h at r e gist er t h e us er’s m o ve m e nts a n d tr a nsl at e t h e m t o virt u al m o ve m e nts. T h er e ar e m ulti pl e H M Ds c urr e ntl y o n t h e m ar ket i n cl u di n g m o d els t h at r e q uir e a p h o n e t o b e i nst all e d t o a ct as a s cr e e n, s e ns or a n d pl a y b a c k d e vi c e a n d m o d els t h at h a ve t h e s cr e e n a n d s e ns ors alr e a d y b uilt- i n b ut n e e d t o b e wir e d t o a P C t o f u n cti o n. Of t his l ast gr o u p t h e O c ul us Rift a n d t h e H T C Vi ve ar e t h e m ost p o p ul ar m o d els a n d ar e als o oft e n us e d i n r es e ar c h s etti n gs. B ot h n o w s u p p ort f ull r o o m-s c al e p ositi o n al tr a c ki n g w hi c h gi ves a c c ess t o t h e h e a d’s c urr e nt p ositi o n a n d r ot ati o n i n t h e p h ysi c al s p a c e.
T h e dis a d v a nt a g e of usi n g a virt u al r e alit y H M D li ke t h e H T C Vi ve is t h at t h e us er is c o m pl et el y s h ut o ut of t h e r e al e n vir o n m e nt a n d b yst a n d ers c a n’t g et a pr o p er i d e a of w h at t h e o p er at or is d oi n g. Usi n g a n a u g m e nt e d r e alit y ( A R) h e a ds et i nst e a d of a V R h e a ds et s ol ves t his pr o bl e m p arti all y.
A R h e a ds ets li ke t h e Mi cr os oft H ol o L e ns ( Mi cr os oft, 2 0 1 6) us e tr a ns p ar e nt gl ass es t h at all o w f or virt u al o bj e cts t o b e dis pl a ye d o n t o p of t h e us er’s vi e w of t h e r e al w orl d.
A n ot h er o pti o n w o ul d b e t o us e a st er e os c o pi c pr oj e cti o n s cr e e n i n c o m bi-
n ati o n wit h h e a d-tr a c ki n g t o cr e at e a n i m m ersi ve e x p eri e n c e li ke P o u li q u e n- L ar d y et al. ( 2 0 1 6) us e d f or t h eir r es e ar c h e x p eri m e nt. T h e a d v a nt a g e of t his m et h o d is t h at b yst a n d ers c a n l o o k at t h e s a m e dis pl a y as t h e o p er a- t or is l o o ki n g at a n d c a n t h er ef or e b ett er pl a c e t h e o p er at or’s a cti o ns i nt o c o nt e xt.
2. 4 C o n cl u si o n
T his c h a pt er pr es e nt e d di ff er e nt as p e cts of fr e e vi e w p oi nt vi d e o s yst e ms. We dis c uss e d di ff er e nt a c q uisiti o n, pr o c essi n g a n d dis pl a y t e c h ni q u es, t o g et h er wit h e x a m pl es of v ari o us i m pl e m e nt ati o ns. We s a w t h at t e c h ni q u es t h at ar e us e d i n F V V s yst e ms c a n b e a g o o d b asis f or a t el e pr es e n c e s yst e m. I n f a ct, t h e li n e b et we e n a F V V s yst e m a n d a t el e pr es e n c e s yst e m c a n oft e n b e di ffi c ult t o dr a w, es p e ci all y wit h a li ve str e a mi n g F V V s yst e m.
W h e n a c ert ai n F V V t e c h ni q u e is c h os e n, it d et er mi n es m a n y of t h e c h ar a ct eristi cs of t h e t el e pr es e n c e s yst e m t h at es p e ci all y t h e visit or will e x- p eri e n c e. For e x a m pl e: t h e a c q uisiti o n t e c h ni q u e d et er mi n es w h at t h e visit or will b e a bl e t o s e e a n d t h e l e vel of d et ail, t h e pr o c essi n g t e c h ni q u e d et er- mi n es h o w m u c h fr e e d o m t h e visit or will h a ve i n n a vi g ati n g t h e c a pt ur e d d at a a n d t h e dis pl a y t e c h ni q u e d et er mi n es f or a bi g p art h o w i m m ers e d t h e visit or will b e.
I n t h e n e xt s e cti o n we dis c uss t h e d esi g n c h oi c es t h at ar e m a d e f or t h e
O p e nI M P R E S S s yst e m, i n cl u di n g p arts i ns pir e d b y F V V.
A r c hi t e c t u r e a n d Gl o b al D e si g n
O p e nI M P R E S S is d esi g n e d as a mi x e d r e alit y t el e pr es e n c e s yst e m t h at us es t h e l at est c o ns u m er el e ctr o ni cs t o c a pt ur e, str e a m a n d vis u ali z e a n e n vi- r o n m e nt a n d t h e p e o pl e i n it wit h t h e g o al t o m a ke p e o pl e f e el as pr es e nt as p ossi bl e fr o m a p h ysi c all y r e m ot e l o c ati o n. T his c h a pt er d es cri b es t h e c h oi c es t h at wer e m a d e w h e n d esi g ni n g t his s yst e m a n d t h e ar c hit e ct ur e t h at t h os e c h oi c es e ve nt u all y l e a d t o.
3. 1 Gl o b al D e si g n C o n si d e r a ti o n s
T h e d esi g n of O p e nI M P R E S S is b as e d o n t h e i d e a of m a ki n g t h e visit or f e el as if b ei n g pr es e nt at t h e o n-sit e l o c ati o n a n d m a ki n g t h e visit e e f e el as if t h e visit or is pr es e nt wit h hi m or h er t h er e as well. T h e s yst e m s p a ns t w o di ff er e nt l o c ati o ns: t h e o n-sit e l o c ati o n a n d t h e r e m ot e l o c ati o n, w h er e r es p e cti vel y t h e o n-sit e o p er at or ( O S O) a n d t h e r e m ot e o p er at or ( R O) ar e l o c at e d. C o m p ar e d t o H ol o p ort ati o n ( Orts- Es c ol a n o et al., 2 0 1 6), t his s ys- t e m is as y m m etri c al. T his m e a ns t h at t h er e is a cl e ar disti n cti o n b et we e n t h e R O, w h o visits t h e o n-sit e l o c ati o n a n d t h e O S O, w h o is bei n g visit e d.
T h e b asi cs of t h e s yst e m ar e s h o w n i n Fi g ur e 3. 1. At t h e o n-sit e l o c ati o n
t h e e n vir o n m e nt is c a pt ur e d a n d s e nt t o t h e r e m ot e l o c ati o n. H er e, t h e
R O c a n fr e el y n a vi g at e a virt u al r e pr es e nt ati o n of t his e n vir o n m e nt. T h e
R O’s h e a d a n d h a n d m o ve m e nts ar e c a pt ur e d a n d s e nt b a c k t o t h e O S O
w h er e t h e y ar e vis u ali z e d t o t h e O S O. T h e O S O is we ari n g a Mi cr os oft
H ol o L e ns H M D, w hi c h ar e a u g m e nt e d r e alit y ( A R) gl ass es. T h e y c a n dis pl a y
c o nt e nt o n t o p of t h e r e al w orl d usi n g tr a ns p ar e nt dis pl a ys t h at ar e pl a c e d
i n fr o nt of t h e us er’s e yes. B esi d es vis u ali zi n g el e m e nts i n a n e n vir o n m e nt,
t h e H ol o L e ns c a n als o b e us e d t o c a pt ur e a n e n vir o n m e nt. T h e H ol o L e ns is
e q ui p p e d wit h m ulti pl e d e pt h s e ns ors t h at ar e us e d t o cr e at e a s p ati al m o d el
of its c urr e nt e n vir o n m e nt i n r e al-ti m e a n d tr a c k its o w n l o c ati o n wit hi n t his
H ol ol e ns
O S O T o ol
D e pt h C a m er a Str e a m er D e pt h C a m er a
wit h A R M ar k er
R O Cli e nt
R O r e pr es e nt ati o n h e a d / h a n ds
O n-sit e l o c ati o n R e m ot e l o c ati o n L o w-r esS p ati al m es h
Hi g h-r es P oi nt cl o u d
Fi g ur e 3. 1: O p e nI M P R E S S b asi c s yst e m o ver vi e w.
e n vir o n m e nt. A g e o m etri c al r e pr es e nt ati o n of t his s p ati al m o d el is s e n d t o t h e R O, w h er e it s er ves as a l o w q u alit y b ut wi d e ar e a r e pr es e nt ati o n of t h e e n vir o n m e nt. T his m o d el is e xt e n d e d wit h s c a ns a c q uir e d b y o n e or m or e d e pt h c a m er a’s t h at ar e als o i nst all e d at t h e o n-sit e l o c ati o n. T h os e s c a ns ar e vis u ali z e d as p oi nt cl o u ds a n d s er ve as a m or e d et ail e d b ut n arr o w r e pr es e nt ati o n of t h e e n vir o n m e nt. I n or d er t o pr o p erl y ali g n t h e p oi nt cl o u ds fr o m t h e d e pt h c a m er a’s wit h t h e l o w q u alit y s p ati al m o d el, t h e d e pt h c a m er a’s ar e e q ui p p e d wit h A R m ar kers. T h os e m ar kers c a n b e s c a n n e d b y t h e H ol o L e ns t o d et er mi n e t h eir p ositi o n r el ati ve t o t h e H ol o L e ns’s s p ati al m o d el. T his p ositi o n is t h e n s e nt t o t h e r e m ot e l o c ati o n w h er e it is us e d t o ali g n t h e p oi nt cl o u d d at a. Fi g ur e 3. 2 s h o ws t h e e n visi o n e d s et u p pr o c e d ur e e x p eri e n c e st e p b y st e p pl us a b asi c i nt er a cti o n.
3. 1. 1 A R v s V R
I n o ur d esi g n we w a nt t o m a ke t h e R O f e el tr a ns p ort e d i nt o t h e O S O’s s p a c e a n d m a ke t h e O S O f e el li ke t h e R O is t h er e wit h t h e m. Orts- Es c ol a n o et al. ( 2 0 1 6) m e nti o n i n t h eir fi n di n gs t h at p arti ci p a nts t h at us e d t h eir A R c o n diti o n f elt as if t h eir r e m ot e p art n er w as t h er e wit h t h e m i n t h ei r s p a c e, w hil e p arti ci p a nts t h at us e d t h e V R c o n diti o n h a d t h e i m pr essi o n of b ei n g i n t h e p art n er’s s p a c e. T his s u g g est t h at f or m a ki n g p e o pl e f e el tr a ns port e d t o a n ot h er l o c ati o n V R m a y b e a b ett er o pti o n, w h er e A R is m or e i d e al f or m a ki n g p e o pl e f e el li ke t h e y ar e b ei n g visit e d.
O ur d esi g n is i nf or m e d b y t his a n d t h er ef or e t h e R O is usi n g a V R
s et u p a n d t h e O S O is usi n g A R. T h e R O’s o w n s urr o u n di n gs ar e c o m pl et el y
bl o c ke d fr o m t h e R O, l e a vi n g o nl y t h e vis u ali z ati o n of t h e o n-sit e l o c ati o n’s
e n vir o n m e nt. T h e O S O o n t h e ot h er h a n d, c a n s e e t h e o n-sit e l o c ati o n’s
e n vir o n m e nt as h e or s h e w o ul d n or m all y b e a bl e t o, as well as t h e s u p eri m-
p os e d r e pr es e nt ati o n of t h e R O. T his is ass u m e d t o b e m or e i d e al f or r e m ot e
assist a n c e s c e n ari os, w h er e o nl y t h e O S O’s e n vir o n m e nt is of i m p ort a n c e a n d
a d di n g i nf or m ati o n a b o ut t h e R O’s e n vir o n m e nt w o ul d cr e at e c o nf usi o n.
( a) ( b) ( c)
( d) ( e) (f )
F i g ur e 3. 2: a) O S O i nst alls a d e pt h c a m er a. b) O S O c ali br at es t h e d e pt h c a m er a. c) R O r e c ei ves s p ati al m es h fr o m H ol o L e ns. d) R O r e c ei ves p oi nt cl o u ds, w hi c h ar e ali g n e d wit h s p ati al m es h. e) R O p oi nts at o bj e cts usi n g c o ntr oll ers ( gr e e n c o n es). f ) O S O s e es R O r e pr es e nt ati o n t hr o u g h H ol o L e ns.
3. 1. 2 O n- si t e E n vi r o n m e nt C a p t u ri n g
R es e ar c h q u esti o n R Q 1. 3. f o c us es o n h o w t o c o ver a l ar g e w or ki n g ar e a.
C urr e nt t el e pr es e n c e s yst e ms t h at m a ke us e of 3 D s c a ns ar e oft e n li mit e d b y t h e ar e a i n w hi c h t h e us ers c a n o p er at e. 3 D d e pt h c a m er a’s ar e i nst all e d at pr e- pl a n n e d str at e gi c pl a c es a n d t h eir e xtri nsi cs ( p ositi o n a n d ori e nt ati o n) ar e d et er mi n e d d uri n g a c ali br ati o n st e p w hi c h is n e c ess ar y t o pr o p erl y ali g n t h eir c a pt ur e d vi e ws t o e a c h ot h er a n d t o t h e r est of t h e virt u al r e pr es e nt a- ti o n. T his fi x es t h e si z e of t h e s c a n ni n g ar e a a n d t h us als o fi x es t h e w or ki n g ar e a t h at c a n b e us e d d uri n g a t el e pr es e n c e s essi o n.
For O p e nI M P R E S S we l o o ke d at s ol uti o ns t h at all o w f or m or e d y n a mi c r e c o n fi g ur ati o n of t h e s c a n ni n g ar e a. S o t h at w h e n t h e w or k s hifts t o a di ff er e nt l o c ati o n, t h e s c a n ni n g s p a c e a d a pts o n t h e fl y a n d t h e R O is ke pt i n v ol ve d i n t h e t as k at h a n d.
T h e di ff er e nt s ol uti o ns t h at h a ve b e e n c o nsi d er e d all b uil d o n t o p of t h e
H ol o L e ns’ b uilt-i n tr a c ki n g s yst e m. T h e H ol o L e ns a ut o m ati c all y cr e at es a
g e o m etri c al s p ati al m o d el of e a c h e n vir o n m e nt it visits b y c o nti n u o usl y s c a n-
ni n g its s urr o u n di n gs a n d u p d ati n g t h e s p ati al m o d el w h e n n e c ess ar y. T his
m o d el c a n t h e n b e us e d b y d e vel o p ers t o m a ke t h eir a p pli c ati o ns i nt er a ct
wit h t h e e n vir o n m e nt. B e c a us e it is n ot i nt e n d e d t o b e vi e we d b y its elf,
t h e s p ati al m o d el is a r el ati vel y l o w r es ol uti o n m o d el a n d c o nt ai ns n o c ol or
i nf or m ati o n, alt h o u g h t e c h ni q u es e xist f or a d di n g c ol or l at er o n
1 2. D es pit e t his li mit e d l e vel of d et ail, it still c o nt ai ns a l ot of us ef ul g e n er al i nf or m ati o n a b o ut t h e we ar er’s e n vir o n m e nt; t h e si z e a n d s h a p e of t h e r o o m, l o c ati o ns of f ur nit ur e, d o ors a n d wi n d o ws a n d t h e s h a p e of t h e m. B e c a us e t h e s p ati al m o d el is e xt e n d e d a n d u p d at e d wit h t h e c urr e nt w or ki n g e n vir o n m e nt i n r e al-ti m e a n d wit h o ut r e q uiri n g a n y us er i nt er ve nti o n, it is a n e as y st arti n g p oi nt f or c o veri n g a l ar g e w or ki n g s p a c e. Als o, t h e a bs e n c e of t h e n e e d f or t h e us ers t o d o a n y s et u p a cti o ns or c ali br ati o n st e ps h el ps wit h ke e pi n g t h e s yst e m e as y t o s et- u p a n d us e. T his bri n gs us t o t h e first s ol uti o n t h at is i m- pl e m e nt e d f or c o veri n g a l ar g e w or ki n g ar e a. T h e s p ati al m o d el is str e a m e d fr o m t h e O S O’s H ol o L e ns t o t h e R O w h er e it is us e d as a b as e l a yer of t h e virt u al r e pr es e nt ati o n of t h e O S O’s e n vir o n m e nt.
T his b as e l a yer is e xt e n d e d wit h hi g h r es ol uti o n p oi nt cl o u ds. T h e y a d d m or e d et ail t o t h e o ver all virt u al r e pr es e nt ati o n of t h e O S O’s e n vir o n m e nt.
T h e y ar e c a pt ur e d b y 3 D d e pt h c a m er as, li ke t h e Mi cr os oft Ki n e ct, t h at ar e pl a c e d i n t h e w or ki n g ar e a. C o m p ar e d t o t h e s p ati al m o d el, a p oi nt cl o u d us u all y c o nt ai ns s u bst a nti all y m or e d et ail, as it h as a hi g h er r es ol uti o n, h as a hi g h er r efr es h r at e, is m or e a c c ur at e a n d c o nt ai ns c ol or. W h e n a v ail a bl e, p oi nt cl o u ds ar e cl e arl y t h e pr ef err e d w a y of vis u ali zi n g t h e O S O’s e n vir o n- m e nt. T h e pr o bl e m is t h at it is di ffi c ult t o c o ver O S O’s c o m pl et e e n vir o n- m e nt wit h 3 D c a m er as t o all o w t h e p oi nt cl o u d t o b e a v ail a bl e e ver y w h er e, t h at’s w h y t h e s p ati al m o d el is us e d t o fill t h e “ e m pt y g a ps ”.
T h e p oi nt cl o u ds will h a ve t o b e p ositi o n e d c orr e ctl y r el ati ve t o e a c h ot h er a n d t o t h e r est of t h e virt u al r e pr es e nt ati o n. To d o t his, a c ali br ati o n st e p is n e c ess ar y i n w hi c h t h e e xtri nsi cs of t h e c a m er as is d et er mi n e d. T h er e ar e s e ver al m et h o ds t h at c a n b e us e d f or c ali br ati o n.
A d e di c at e d p ositi o n al tr a c ki n g s yst e m c a n b e us e d, li ke O pti Tr a c k or t h e H T C Vi ve tr a c ker, t o tr a c k e a c h d e pt h c a m er a’s l o c ati o n i n t h e e n vir o n m e nt.
T his m et h o d is t h e m ost r eli a bl e b ut r e q uir es a n e xtr a c o m p o n e nt t o b e s et u p a n d t h er ef or e c o m pr o mis es t h e e as e of d e pl o y m e nt.
A n ot h er o pti o n is t o att a c h t h e d e pt h c a m er a o n t o p of t h e H ol o L e ns.
T h e l o c ati o n of t h e H ol o L e ns is alr e a d y k n o w n a n d t h er ef or e, b y att a c hi n g t h e d e pt h c a m er a t o t h e H ol o L e ns, t h e d e pt h c a m er a’s l o c ati o n will als o b e k n o w n. G ar o n et al. ( 2 0 1 6) h a ve us e d a si mil ar s et- u p f or di ff er e nt p ur p os es.
To ke e p t h e w h ol e s yst e m m o bil e, t h e y als o att a c h e d a sti c k- P C a n d a b at- t er y p a c k t o t h e H ol o L e ns. T his m et h o d r et ai ns t h e e as e of d e pl o y m e nt as it d o es n’t i ntr o d u c e e xtr a c o m p o n e nts (it m a kes us e of t h e tr a c ki n g fr o m t h e H ol o L e ns) b ut it is li mit e d t o tr a c ki n g t h e l o c ati o n of o nl y o n e d e pt h c a m er a.
T h e r el ati ve p ositi o ns b et we e n t h e d e pt h c a m er as a n d t h e s p ati al m es h
1h t t p : / / w w w . m a t r i x i n c e p t i o n . c o m / w p / H o l o L e n s - r o o m s c a n - r e l o a d e d /
2h t t p s : / / g i t h u b . c o m / y w j 7 9 3 1 / H o l o L e n s _ I m a g e _ B a s e d _ T e x t u r e /
c a n b e c al c ul at e d b y cr e ati n g a n al g orit h m t h at s e ar c h es f or si mil ar g e o m et- ri c al p att er ns. T his m et h o d is t h e m ost vers atil e as it d o es n’t n e e d a n y e xtr a c o m p o n e nts t o b e a d d e d t o t h e s yst e m a n d s c al es w h e n m or e d e pt h c a m er as ar e a d d e d. T h e d o w nsi d es ar e t h at t h er e h as t o b e o verl a p pi n g d at a a n d it r e q uir es r e c o g ni z a bl e f e at ur es i n t h e g e o m etr y a n d c o m pl e x al g orit h ms t h at m a ke it pr o n e t o m a ki n g mist a kes.
We c h os e t o att a c h pri nt e d m ar kers o n e a c h d e pt h c a m er a a n d s ol vi n g t h eir l o c ati o ns usi n g t h e i m a g e c a m er a o n t h e H ol o L e ns. T his m et h o d is s c al a bl e as a n u ni q u e m ar ker c a n b e a d d e d f or e a c h d e pt h c a m er a, b ut is li mit e d b y t h e c a m er a o n t h e H ol o L e ns, w hi c h h as t o b e a bl e t o s e e t h e m ar kers t o b e a bl e t o s ol ve t h eir l o c ati o ns. B e c a us e t h e c a m er as ar e n ot s u p p os e d t o b e m o vi n g t his is n ot a bi g pr o bl e m. If c a m er as ar e m o ve d fr o m t h eir i niti al l o c ati o n a n y w a y, t h eir l o c ati o ns will h a ve t o b e s ol ve d a g ai n b y
“ o bs er vi n g t h e m ” wit h t h e c a m er a of t h e H ol o L e ns.
3. 2 S y s t e m C o n fi g u r a ti o n s
B esi d es t el e pr es e n c e, a s yst e m li ke O p e nI M P R E S S c a n b e us e d f or a v ari et y of di ff er e nt a p pli c ati o ns b y m o dif yi n g h o w t h e s yst e m is c o n fi g ur e d. Fo ur m ai n s yst e m c o n fi g ur ati o ns h a ve b e e n i d e nti fi e d t h at di ff er e nti at e t h e ms el ves b y eit h er dis pl a yi n g t h e c a pt ur e d d at a i n r e al-ti m e or at a di ff er e nt p oi nt i n ti m e a n d h a vi n g t h e c a pt ur e d d at a b ei n g pl a ye d b a c k at eit h er t h e s a m e or at a di ff er e nt l o c ati o n fr o m w h er e it w as c a pt ur e d.
p h y si c all y
c o-l o c at e d di sl o c at e d t e m p o r all y c o-l o c at e d m o di fi e d r e alit y r e m ot e assist a n c e
di sl o c at e d i n- pl a c e tr ai ni n g r eli ve m e m or y Ta bl e 3. 1: T h e b asi c s yst e m c o n fi g ur ati o ns wit h c orr es p o n di n g a p pli c ati o n e x a m pl es
Ta bl e 3. 1 s h o ws t h e f o ur b asi c p ossi bl e c o n fi g ur ati o ns i n r el ati o n t o e a c h ot h er. Te m p or al c o-l o c ati o n m e a ns t h at t h e vi e wer r e c ei ves t h e r e c or d e d d at a as it’s b ei n g r e c or d e d. Te m p or al disl o c ati o n m e a ns t h at t h e vi e wer is l o o ki n g at d at a t h at is pr e-r e c or d e d. P h ysi c al c o-l o c ati o n m e a ns t h at t h e vi e wer is st a n di n g p h ysi c all y at t h e s a m e l o c ati o n fr o m w h er e t h e d at a w as or is b ei n g r e c or d e d; t h e virt u al e n vir o n m e nt is ali g n e d wit h t h e p h ysi c al ali g n m e nt. P h ysi c al disl o c ati o n m e a ns t h at t h e vi e wer is l o o ki n g at t h e virt u al e n vir o n m e nt w hil e b ei n g at a p h ysi c all y di ff er e nt l o c ati o n fr o m w h er e t h e d at a w as or is b ei n g r e c or d e d.
D e p e n di n g o n h o w t h e s yst e m is c o n fi g ur e d, it c a n b e a p pli e d i n di ff er e nt
a p pli c ati o n s c e n ari os. Of c o urs e, t h os e b asi c c o n fi g ur ati o ns c a n b e e xt e n d e d
t o s uit di ff er e nt s c e n ari os b y f or e x a m pl e s u p p orti n g m ulti pl e p e o pl e vi e wi n g
or c a pt uri n g at t h e s a m e ti m e, usi n g di ff er e nt pl a y b a c k or r e c or di n g d e vi c es a n d usi n g di ff er e nt w a ys of distri b uti n g t h e r e c or d e d c o nt e nt. For t his t h esis h o we ver, we will f o c us o n t h e r e m ot e assist a n c e a p pli c ati o n, w hi c h m e a ns we us e a t e m p or all y c o-l o c at e d a n d p h ysi c all y disl o c at e d c o n fi g ur ati o n.
3. 3 A r c hi t e c t u r e
T h e ar c hit e ct ur e of t h e s yst e m is b as e d o n t h e d esi g n c o nsi d er ati o ns fr o m s e cti o n 3. 1. A ar c hit e ct ur e di a gr a m is s h o w n i n Fi g ur e 3. 3. T h e s yst e m is s plit u p i nt o s e ve n m ai n c o m p o n e nts. T h e s yst e m c a n b e r e c o n fi g ur e d b y dis a bli n g or r e arr a n gi n g c ert ai n c o m p o n e nts t o m a ke it m or e s uit a bl e f or di ff er e nt s c e n ari os. E a c h c o m p o n e nt c a n als o b e us e d s e p ar at el y a n d b e i nt e gr at e d i nt o ot h er s yst e ms if n e c ess ar y.
Fi g ur e 3. 3: T h e s yst e m ar c hit e ct ur e di a gr a m of O p e nI M P R E S S.
T h e O n- Sit e E n vir o n m e nt C a pt ure a n d Vis u aliz ati o n S yst e m f o c us es o n c a pt uri n g t h e o n-sit e e n vir o n m e nt wit h as m u c h d et ail as p ossi bl e a n d vi- s u ali zi n g it at t h e r e m ot e l o c ati o n. T h e Tr a nsf or m M a n a g e m e nt S yst e m m a n a g es t h e s h ari n g of m ulti pl e tr a nsf or m b et we e n t h e R O a n d O S O. T h e A n n ot ati o n S yst e m e n a bl es t h e R O t o cr e at e a n n ot ati o ns i n t h e O S O’s e n vi- r o n m e nt. T h e N et w or ki n g S yst e m h a n dl es all t h e d at a c o n n e cti o n b et we e n c o m p o n e nts t h at h a ve t o c o m m u ni c at e o ver a n et w or k. T h e V R C o m p o- n e nts m a n a g e t h e e x p eri e n c e t h e us er h as i n V R. T h e H a n d G est ure S yst e m c a pt ur es t h e R O’s h a n ds a n d dis pl a ys t h e m at t h e O S O. T h e Ver b al C o m- m u ni c ati o n S yst e m all o ws t h e R O a n d O S O t o t al k wit h o n e a n ot h er b y usi n g t h eir v oi c e.
E a c h c o m p o n e nt is d es cri b e d i n m or e d et ail i n c h a pt er 4 t o g et h er wit h
i m pl e m e nt ati o n d et ails.
Im pl e m e nt a ti o n
I n t his c h a pt er we pr es e nt h o w t h e di ff er e nt c o m p o n e nts t h at ar e d es cri b e d i n s e cti o n 3. 3 ar e i m pl e m e nt e d.
4. 1 O n- Si t e E n vi r o n m e nt C a p t u r e a n d Vi s u ali z a- ti o n S y s t e m
T h e O n- Sit e E n vir o n m e nt C a pt ur e a n d Vis u ali z ati o n S yst e m c o nt ai ns all t h e c o m p o n e nts t h at ar e us e d t o c a pt ur e t h e O S O’s e n vir o n m e nt a n d vis u ali z e it at t h e R O. It c a n b e di vi d e d i nt o t w o s u bs yst e ms; o n e f or p oi nt- cl o u d d at a a n d o n e f or m es h d at a.
4. 1. 1 D e p t h C a m e r a S t r e a m e r
T h e D e pt h c a m er a stre a m er
1is a st a n d al o n e c o m p o n e nt writt e n i n C + + t h at c a pt ur es d e pt h a n d c ol or i m a g e d at a fr o m a d e pt h s e nsi n g c a m er a a n d tr a ns mits t h e m o ver a n et w or k usi n g a n U D P C o n n e ct or. Di ff er e nt versi o ns of t his c o m p o n e nt e xist t h at ar e m a d e t o w or k wit h di ff er e nt br a n ds of d e pt h c a m er as or wit h di ff er e nt li br ari es f or t h e s a m e c a m er a. Versi o ns f or t h e Mi cr os oft Ki n e ct V 2 usi n g t h e d ef a ult Mi cr os oft li br ar y
2a n d t h e li bfr e e n e ct 2 li br ar y
3h a ve b e e n m a d e, as well as f or t h e I nt el R e al S e ns e c a m er a
4.
T h e c urr e nt versi o n of t h e str e a m er us es J P E G c o m pr essi o n o n t h e c ol or i m a g e d at a, e n c o di n g e a c h fr a m e as a s e p ar at e J P E G i m a g e, a n d n o c o m pr es- si o n o n t h e d e pt h d at a. A n e xt st e p i n t h e d e vel o p m e nt of t his c o m p o n e nt w o ul d b e t o fi n d a n d i m pl e m e nt a n a p pr o pri at e c o m pr essi o n al g orit h m f or t h e d e pt h d at a as well.
1M ost of t his c o m p o n e nt is d e vel o p e d b y J a n K ol k m ei er
2h t t p s : / / d e v e l o p e r . m i c r o s o f t . c o m / e n - u s / w i n d o w s / k i n e c t
3h t t p s : / / g i t h u b . c o m / O p e n K i n e c t / l i b f r e e n e c t 2
4h t t p s : / / g i t h u b . c o m / I n t e l R e a l S e n s e / l i b r e a l s e n s e
