54
International Journal of Bioelectromagnetism www.ijbem.org Vol. 13, No. 1, pp. 54- 55, 2011
User Experience Evaluation in BCI: Filling the Gap
Hayrettin Gürkök, Danny Plass-Oude Bos, Bram van de Laar, Femke Nijboer, Anton Nijholt
Human Media Interaction, Faculty of EEMCS, University of Twente, Enschede, The Netherlands
Correspondence: H Gürkök, Human Media Interaction, Faculty of EEMCS, University of Twente, P.O. Box 217, 7500 AE, Enschede, The Netherlands. E-mail: h.gurkok@cs.utwente.nl, phone +31 53 4894647, fax +31 53 4893503
Abstract. Brain-computer interface (BCI) systems can improve the user experience (UX) when used in
entertainment technologies. Improved UX can enhance user acceptance, improve quality of life and also increase the system performance of a BCI system. Therefore, the evaluation of UX is essential in BCI research. However, BCI systems are generally evaluated according to the system aspect only so there is no methodology to evaluate UX in BCI systems. This paper gives an overview of such methods from the human-computer interaction field and discusses their possible uses in BCI research.
Keywords: Brain-computer interface; human-computer interaction; user experience evaluation; entertainment
1. Introduction
The most up-to-date definition of user experience (UX) according to the ISO 9241-210:2010 standard is: “A
person's perceptions and responses that result from the use or anticipated use of a product, system or service.”. In
human-computer interaction (HCI), especially for entertainment technologies which simply aim to improve the well-being of users, UX is a major concern. Therefore, the HCI community designs for UX and develops methods to evaluate it. On the other hand, brain-computer interface (BCI) systems are generally evaluated according to the system aspect only: based on the classification accuracy or the communication speed in terms of bit rates [Plass-Oude Bos et al., 2010a]. This evaluation is an incomplete one as it ignores the human aspect. Especially with the emerging virtual reality applications [Leeb et al., 2007] and games [Plass-Oude Bos et al., 2010b] for BCIs, UX evaluation is indispensable. So far, no methodology has been suggested to evaluate UX in BCI systems. This gap can be filled by learning from the HCI methods to evaluate UX. This paper gives an overview of such methods applied to entertainment technologies and discusses their possible uses in BCI research.
2. User Experience Evaluation Methods in Entertainment Technologies
We can classify the current methods for evaluating user experience in entertainment technologies as objective versus subjective or as qualitative versus quantitative [Mandryk et al., 2006a]. The objective methods rely on overt and covert user responses during interaction. The classical way of observing overt user behaviour is through audiovisual recorders which provide qualitative data for gestures, facial expressions and verbalisations. There are difficulties associated with annotating and analysing such rich data though. Firstly, while analysing the data, the researchers should acknowledge their biases, address inter-rater reliability and not to read inferences where none are present. Secondly, there is an enormous time commitment associated with observational analysis [Mandryk et al., 2006b].
Task performance metrics have been suggested as quantitative-objective measures of UX but these are not necessarily the indicators of UX. Especially in entertainment applications, there might not be a clear task or users might prefer navigating in the virtual environment without any urge to complete tasks. Use of neurophysiological sensor technologies was proposed for modelling user emotional state in play technologies [Mandryk et al., 2006b]. Modelled emotions are powerful as they capture usability and playability through metrics relevant to ludic experience; account for user emotion; are quantitative and objective; and are represented continuously over a session. There are difficulties attached to measurements using neurophysiological sensors. Firstly, the sensors attached to the user might induce discomfort to the user, restrict movements or influence the experience. Secondly, care must be taken to avoid stimuli that affect neurophysiological responses, such as the caffeine intake. Due to the challenges with observational analysis, many researchers usually opt for subjective methods.
55
Subjective methods include interviews and questionnaires. Unlike objective methods, they are unintrusive as well as generalisable, convenient, amendable to rapid statistics and easy to administer. Interviews yield qualitative data while questionnaires are designed to provide structured, categorised and quantitative data. Nevertheless, for all subjective methods, there is the issue of privacy and the possibility that the subject responses may not correspond to the actual experience. For instance, children’s responses are strongly influenced by what the adults want to hear [Jones et al., 2003].
Development of UX questionnaires for entertainment applications has received a lot attention from researchers, especially those who are interested in games. The recently developed game engagement questionnaire [Brockmyer et al., 2009] includes items related to absorption, flow, presence and immersion. There are also questionnaires focusing exclusively on the UX components such as presence [van Baren and IJsselstein, 2004] and immersion [Jannett et al., 2008]. Another concept related to UX is the usability. Heuristics have been proposed for evaluating the usability of video games [Omar and Jaafar, 2010]. However heuristic evaluation does not involve actual users, but is administered by usability specialists. Thus, experts can only guess how the technologies will impact users.
3. Filling the Gap
As described in the previous section, HCI research has developed a variety of methods that BCI research can adopt, adapt or be inspired by. There is no factor that would prevent adopting the observation of overt behaviour through audiovisual recorders and covert responses through neurophysiological sensors. BCI is even advantageous since the drawbacks associated with these techniques, such as ensuring a controlled environment and good recording quality, would have already been taken care of. On the other hand, overt behaviour is not much encouraged in BCI systems so its effectiveness in assessing UX is arguable. Interviews and questionnaires may require adaptation taking into account that state-of-the-art BCI applications are relatively simple thus modest in providing rich UX [van de Laar et al., 2010]. One should also assess the BCI recognition performance, as a relatively low performance might influence the UX.
Acknowledgements
The authors acknowledge the support of the BrainGain Smart Mix Programme of the Netherlands Ministry of Economic Affairs and the Netherlands Ministry of Education, Culture and Science.
References
van Baren J, IJsselsteijn W. Measuring Presence: A Guide to Current Measurement Approaches. Deliverable 5 for OmniPres project IST-2001-39237, 2004.
Brockmyer JH, Fox CM, Curtiss KA, McBroom E, Burkhart KM, Pidruzny JN. The development of the Game Engagement Questionnaire: A measure of engagement in video game-playing. J. Exp. Soc. Psychol., 45(4): 624-634, 2009.
Jennett C, Cox AL, Cairns P, Dhoparee S, Epps A, Tijs T, Walton A. Measuring and defining the experience of immersion in games. Int. J. Hum.-Comput. St., 66(9): 641-661, 2008.
Jones C, McIver L, Gibson L, Gregor P. Experiences obtained from designing with children. In proceedings of IDC 2003, 2003, 69-74. van de Laar B, Nijboer F, Gürkök H, Plass-Oude Bos D, Nijholt A. User Experience Evaluation in BCI: Bridge the Gap. IJBEM, 2010. (to appear)
Leeb R, Friedman D, Müller-Putz GR, Scherer R, Slater M, Pfurtscheller G. Self-paced (asynchronous) BCI control of a wheelchair in virtual environments: a case study with a tetraplegic. Intell. Neuroscience, 2007, 1-12.
Mandryk RL, Atkins MS, Inkpen KM. A continuous and objective evaluation of emotional experience with interactive play environments. In proceedings of CHI 2006, 2006a, 1027-1036.
Mandryk RL, Inkpen KM, Calvert TW. Using psychophysiological techniques to measure user experience with entertainment technologies. Behav. Inform. Technol., 25(2): 141-158, 2006b.
Omar H, Jaafar A. Heuristics evaluation in computer games. In proceedings of CAMP 2010, 2010, 188-193.
Plass-Oude Bos D, Gürkök H, van de Laar B, Nijboer F, Nijholt A. User Experience Evaluation in BCI: Mind the Gap!. IJBEM, 2010a. (to appear)
Plass-Oude Bos D, Reuderink B, van de Laar B, Gürkök H, Mühl C, Poel M, Nijholt A, Heylen D. Brain-Computer Interfacing and Games, in Brain-Computer Interfaces: Applying our Minds to Human-Computer Interaction. Tan D, Nijholt A, Editors. Springer, London, UK, 2010b, 149-178.
