Measuring Engagement: Affective and Social Cues in Interactive Media

(1)

63(&,$/6(66,21

0HDVXULQJ(QJDJHPHQW$IIHFWLYHDQG6RFLDO&XHVLQ,QWHUDFWLYH0HGLD

&KDLU$QWRQ1LMKROW_{$OHVVDQGUR9LQFLDUHOOL}

1

Human Media Interaction, University of Twente, Enschede, The Netherlands. anijholt@cs.utwente.nl

2_{Department of Computing Science, University of Glasgow, Scotland, UK. vincia@dcs.gla.ac.uk}

,QWURGXFWLRQ

,QJDPHRUHQWHUWDLQPHQWHQYLURQPHQWVWKHµXVHU¶PD\WDNHSDUWLQHYHQWVWKDWUHTXLUHERGLO\LQWHUDFWLRQZLWK VHQVRUHTXLSSHGHQYLURQPHQWV(PEHGGHGPRWLRQFDSWXUHDQGJ\URVFRSLFGHYLFHVFDSWXUHPRYHPHQWV7KDQNV WR 1LQWHQGRV UHPRWH :,, PRWLRQFRQWUROOHG JDPHV DUH QRZ ZLGHVSUHDG &DPHUDV PLFURSKRQHV SUHVVXUH VHQVRUV DQG SUR[LPLW\ VHQVRUV KDYH EHHQ DGGHG 7KDQNV WR 0LFURVRIWV ;ER[ .LQHFW ZKROH ERG\ LQWHUDFWLRQ JDPHVKDYHEHFRPHSRSXODU$SDUWIURPHQWHUWDLQPHQWVXFKVHQVRUHTXLSSHGDQGJDPHRULHQWHGHQYLURQPHQWV FDQEHGHVLJQHGWRLPSURYHKHDOWKFRQGLWLRQVVSRUWVSHUIRUPDQFHRUWKHUDSHXWLFSK\VLFDOUHKDELOLWDWLRQ 0RUHVHQVRUVDUHEHFRPLQJDYDLODEOHDOORZLQJDJDPHWRREWDLQPRUHLQIRUPDWLRQDERXWDSOD\HULQSDUWLFXODU KLVRUKHUERGLO\DQGHPRWLRQDOFRQGLWLRQV,QDGGLWLRQWKHVHVHQVRUVDOORZPRUHLQSXWPRGDOLWLHVIRULQWHUDFWLRQ ZLWK DQ HQYLURQPHQW >@ 7KHUH DUH PDQ\ H[DPSOHV RI DGYDQFHG JDPHV ZKHUH SRVWXUH JHVWXUHV ERG\ PRYHPHQWVIDFLDOH[SUHVVLRQVDQGEUDLQDFWLYLW\DUHDPRQJWKHPRGDOLWLHVWKDWDUHXVHGIRUFRQWURO&RQWUROFDQ EH GLUHFW EXW LW FDQ DOVR EH PHGLDWHG IRU H[DPSOH WKURXJK D EDODQFH ERDUG D WDQJLEOH RU D ZHDUDEOH 2EVHUYDWLRQVRIWKHIDFHDQGERG\FDQEHXVHGLQGLIIHUHQWIRUPVGHSHQGLQJRQZKHWKHUWKHXVHUPD\WDNHWKH LQLWLDWLYH WR FRQWURO WKH LQWHUDFWLRQ RU ZKHWKHU WKH DSSOLFDWLRQ WDNHV WKH LQLWLDWLYH WR DGDSW LWVHOI WR WKH XVHU 7KHUHIRUHZHFDQGLVWLQJXLVK

· &RQWUROWKHXVHUFRQVFLRXVO\SURGXFHVIDFLDOH[SUHVVLRQVKHDGPRYHPHQWVRUERG\JHVWXUHVWRFRQWURO DJDPH)RUH[DPSOHWRQDYLJDWHPRYHJDPHFKDUDFWHUVRUFKDQJHDSSHDUDQFHV

· $GDSWDWLRQ WKH JDPHUV VSRQWDQHRXV IDFLDO H[SUHVVLRQV DQG ERG\ SRVHV DUH LQWHUSUHWHG DQG XVHG WR DGDSW WKH JDPH )RU H[DPSOH WKLV LQIRUPDWLRQ PD\ OHDG WR FKDQJHV LQ WKH GLIILFXOW\ OHYHO WKH DSSHDUDQFHRIWKHJDPHHQYLURQPHQWWKHLQWHUDFWLRQPRGDOLWLHVRUWKHQDUUDWLYH

7KHJDPHHQYLURQPHQWLVQRWQHFHVVDULO\DWKRPH,WFDQEHDWUDLQLQJDQGVLPXODWLRQILHOGRUDOHVVFRQWUROOHG VHWWLQJ VXFK DV DQ XUEDQ DUHD ZKHUH ERGLO\ PRYHPHQWV DQG DFWLYLWLHV DUH GHWHFWHG E\ PRELOH GHYLFHV DQG ZHEFDPQHWZRUNV

(QJDJHPHQW

2QH RI WKH NH\ LVVXHV LQ YLGHR JDPHV DQG LQ LQWHUDFWLYH HQWHUWDLQPHQW DQG DUW LV HQJDJHPHQW 5HVHDUFK TXHVWLRQVDUHKRZFDQZHGHVLJQDQGSUHGLFWHQJDJHPHQW"+RZFDQZHDGDSWDJDPHWRLWVXVHUVDQGDXGLHQFH WRLQFUHDVHRUGHFUHDVHHQJDJHPHQW"$JDLQZKHQZHWDONDERXWDµJDPH¶ZHDUHWDONLQJDERXWHQYLURQPHQWV WKDWSURYLGHHQWHUWDLQLQJLQWHUDFWLRQRSSRUWXQLWLHVWRDµXVHU¶$XWRPDWLFDOO\REWDLQHGHQJDJHPHQWLQIRUPDWLRQ DOORZVDGDSWDWLRQRIDJDPHRUJDPHHQYLURQPHQWE\WKHJDPHFRQWUROOHU7KDWPD\LQFOXGHSK\VLFDODQGYLUWXDO DSSHDUDQFHDYDLODELOLW\DQGFKDUDFWHULVWLFVRILQWHUDFWLRQPRGDOLWLHVIHHGEDFNE\DFWXDWRUVJDPHQDUUDWLYHDQG JDPHVWUDWHJ\,WVKRXOGEHPHQWLRQHGWKDWDERYHZHXVHGWKHWHUPµHQJDJHPHQW¶LQDQLQIRUPDOZD\1XPHURXV WHUPV DQG GHILQLWLRQV KDYH EHHQ LQWURGXFHG WR GHVFULEH DVSHFWV RU IHDWXUHV RI HQJDJHPHQW RU WKH RWKHU ZD\ DURXQGWRGHILQHHQJDJHPHQWDVDQDVSHFWRIXVHUH[SHULHQFHFRQFHSWVVXFKDVSUHVHQFHOHYHOVRILPPHUVLRQRU IORZ>@

7KHTXHVWLRQVPHQWLRQHGDERYHQHHGWREHLQYHVWLJDWHG.QRZOHGJHDERXWIDFWRUVWKDWDIIHFWHQJDJHPHQWFDQEH XVHGLQJDPHGHVLJQDQGJDPHHYDOXDWLRQ$OWKRXJKWKHUHLVWKHRUL]LQJDURXQGFRQFHSWVVXFKDVIORZSUHVHQFH

Proceedings of Measuring Behavior 2012 (Utrecht, The Netherlands, August 28-31, 2012)

(2)

DQG LPPHUVLRQ WKHUH LV D FRQVLGHUDEOH JDS ZLWK DSSOLHG NQRZOHGJH FRPLQJ IURP GHVLJQ SHUIRUPLQJ DUWV QDUUDWLYHDQGGUDPDWKHRU\/HWDORQHWKHµJXWIHHOLQJ¶RIDJDPHGHVLJQHUZKRKDVSOD\HGJDPHVIRUWKRXVDQGV RIKRXUVKLPVHOIDQGKDVDFFHVVWRWKHH[SHULHQFHVRIKXQGUHGVRIJDPHWHVWHUVWKDWKDYHVSHQWKXQGUHGVRIKRXUV SOD\LQJDSDUWLFXODUJDPHEHIRUHLWJHWVLWVILQDOIRUPDQGLVPDGHSXEOLFO\DQGFRPPHUFLDOO\DYDLODEOH

1HYHUWKHOHVV GHWHFWLQJ GLIIHUHQW OHYHOV RI HQJDJHPHQW GXULQJ JDPH SOD\LQJ FDQ PDNH LW SRVVLEOH WR PDNH GHFLVLRQV DERXW LQFUHDVLQJ RU GHFUHDVLQJ D OHYHO RI GLIILFXOW\ WR SURYLGH LQJDPH H[SODQDWLRQV WR DGDSW WKH VWRU\OLQHRUWRDGDSWWKHEHKDYLRURIJDPHFKDUDFWHUV,WVKRXOGEHQRWHGWKDWUHVXOWVRQHQJDJHPHQWPRGHOLQJLQ VHQVRUHTXLSSHG JDPH HQWHUWDLQPHQW DQG LQWHUDFWLYH DUW HQYLURQPHQWV GR QRW QHFHVVDULO\ WUDQVODWH WR RWKHU DSSOLFDWLRQVZKHUHIRULQVWDQFHWDVNHIILFLHQF\LVDQLPSRUWDQWLVVXH,QWKHJDPHDQGHQWHUWDLQPHQWVLWXDWLRQV ZH DGGUHVV LPPHUVLRQ SUHVHQFH DQG IORZ QHHG WR EH FRQVLGHUHG ZKHQ LQYHVWLJDWLQJ XVHU H[SHULHQFH XVHU VDWLVIDFWLRQDQGXVHUHQMR\PHQW$VPHQWLRQHGE\%HUWKRX]HHWDO>@WKHUHDUHORWVRIWKHRULHVWKDWDWWHPSWWR GHILQHDQGPRGHODVSHFWVRIHQJDJHPHQWEXW³PRVWWKHRULHVRIHQJDJHPHQWKDYHIRFXVHGSXUHO\RQLWVPHQWDO DVSHFWV´

6SHFLDO6HVVLRQ³0HDVXULQJ(QJDJHPHQW´

7KH DLP RI WKLV VSHFLDO VHVVLRQ DW 0HDVXULQJ %HKDYLRU LV WR ORRN DW HQJDJHPHQW DQG ZD\V WR PHDVXUH HQJDJHPHQW LQ VLWXDWLRQV ZKHUH XVHUV DUH QRW JOXHG WR WKHLU FKDLU DQG NH\ERDUG WKDW LV LQ VHQVRUHTXLSSHG HQYLURQPHQWVWKDWDUHDEOHWRSHUFHLYHQRQYHUEDOLQWHUDFWLRQEHKDYLRU$QGPRUHRYHUZHIRFXVRQDFWLYLWLHVWKDW DUH QRW QHFHVVDULO\ DLPHG DW SHUIRUPLQJ D SDUWLFXODU WDVN LQ WKH PRVW HIILFLHQW ZD\ 5DWKHU WKH IRFXV LV RQ HQMR\PHQWGXULQJSHUIRUPDQFHDQGVDWLVIDFWLRQGXULQJDQGDIWHUILQLVKLQJWKHLQWHUDFWLRQ

5HDOWLPHLQIRUPDWLRQSURYLGLQJVHQVRUVDOORZXVWRGHWHFWFXHVIURPWKHEHKDYLRURIRQHRUPRUHSDUWLFLSDQWVLQ WKHHQYLURQPHQWIURPZKLFKGLIIHUHQWOHYHOVRIHQJDJHPHQWGXULQJSHUIRUPDQFHFDQEHFRQFOXGHG$QGDOVR GLIIHUHQW NLQGV RI IHHGEDFN DQG DGDSWDWLRQ RI WKH JDPH HQYLURQPHQW FDQ EH GHFLGHG 6RFLDO DQG DIIHFWLYH EHKDYLRUDOFXHV>@WREHGHWHFWHGIURPQRQYHUEDOEHKDYLRUQHHGWREHUHFRJQL]HGDQGLQWHUSUHWHGLQRUGHUWREH XVHGDVLQSXWWRRXUDOJRULWKPVWKDWSURYLGHXVZLWKLQIRUPDWLRQDERXWHQJDJHPHQWLPPHUVLRQDQGIORZ $OO WKHVHLVVXHVZLOOEHFRYHUHGE\WKHVL[FRQWULEXWLRQVIRUWKLVVSHFLDOVHVVLRQDW0HDVXULQJ%HKDYLRU

$FNQRZOHGJHPHQWV

7KLVUHVHDUFKKDVEHHQVXSSRUWHGE\WKH(XURSHDQ&RPPXQLW\¶VWK)UDPHZRUN3URJUDPPH)3± XQGHUJUDQWDJUHHPHQWQR6631HWDQGWKH&D6$&RPSXWHUVDV6RFLDO$FWRUVSURMHFWRIWKH(,7,&7 /DEV(,7,&7 63(&,$/6(66,21&217(176VRUWHGE\SDSHU,' 9LGHR%DVHG0XOWLSHUVRQ+XPDQ0RWLRQ&DSWXULQJ

1LFR YDQ GHU $D /XFDV 1ROGXV 1ROGXV ,QIRUPDWLRQ 7HFKQRORJ\ 7KH 1HWKHUODQGV DQG 5HPFR 9HOWNDPS 8WUHFKW8QLYHUVLW\7KH1HWKHUODQGV

0DNLQJ$PELHQW6SDFHVLQWR3OD\JURXQGV

'HQQLV 5HLGVPD 'DQLHO 7HWWHURR DQG $QWRQ 1LMKROW +XPDQ 0HGLD ,QWHUDFWLRQ &UHDWLYH 7HFKQRORJ\ 8QLYHUVLW\RI7ZHQWH7KH1HWKHUODQGV

%XLOGLQJ&RUSRUDRI%RGLO\([SUHVVLRQVRI$IIHFW

0DUFR3DVFKDQG0RQLFD/DQGRQL8QLYHUVLW\RI/XJDQR6ZLW]HUODQG

0HDVXULQJ)XQDQG(QMR\PHQWRI&KLOGUHQLQD0XVHXP(YDOXDWLQJWKH6PLOH\RPHWHU )UDQVYDQGHU6OXLV%HWV\YDQ'LMNDQG%HUW3HUOR\8QLYHUVLW\RI7ZHQWH7KH1HWKHUODQGV

(3)

2Q0DNLQJ(QJDJHPHQW7DQJLEOH

(JRQ / YDQ GHQ %URHN 712 8QLYHUVLW\ RI 7ZHQWH 5DGERXG 8QLYHUVLW\ 0HGLFDO &HQWHU 1LMPHJHQ 7KH 1HWKHUODQGV :KDW&DQ%RG\0RYHPHQW7HOO8V$ERXW3OD\HUV¶(QJDJHPHQW" 1DGLD%LDQFKL%HUWKRX]H8QLYHUVLW\&ROOHJH/RQGRQ8.

5HIHUHQFHV

3DQWLF01LMKROW$3HQWODQG$+XDQJ7+XPDQ&HQWUHG,QWHOOLJHQW+XPDQ&RPSXWHU ,QWHUDFWLRQ +&,ð +RZ IDU DUH ZH IURP DWWDLQLQJ LW" J. Autonomous and Adaptive Communications

Systems

1LMKROW $ 3DVFK 0 YDQ 'LMN (0$* 5HLGVPD ' +H\OHQ '.- 2EVHUYDWLRQV RQ ([SHULHQFH DQG )ORZ LQ 0RYHPHQW%DVHG ,QWHUDFWLRQ &KDSWHU LQ Whole Body Interaction ' (QJODQG(G+XPDQ±&RPSXWHU,QWHUDFWLRQ6HULHV6SULQJHU9HUODJ/RQGRQ

%LDQFKL%HUWKRX]H1.LP::'DUVKDN3,'RHVERG\PRYHPHQWHQJDJH\RXPRUHLQ GLJLWDO JDPH SOD\" $QG :K\" Affective Computing and Intelligent Interaction (ACII 2007) /1&6 /LVERQ

9LQFLDUHOOL $ 3DQWLF 0 %RXUODUG + 6RFLDO VLJQDO SURFHVVLQJ 6XUYH\ RI DQ HPHUJLQJ GRPDLQImage and Vision Computing