Increasing engagement in a MACT game through feedback and adaptability

(1)

Master’s Thesis

Increasing engagement in a MACT game through feedback and adaptability Luuk Schlette (5970709)

Graduate School of Natural Sciences Master Human-Computer Interaction

Supervisor: Anja Volk Second Supervisor: H anna Hauptmann

(2)

1 Acknowledgement

Firstly I want to thank Anja Volk and Hanna Hauptmann for the continuous feedback their support and overall motivational conversations to aid me through the process. A lot of effort can be attributed to my supervisors. Than I would also want to thank Laurien Hakvoort for her valuable insights into the music therapy topic. Also providing great examples and think- ing throughout the project. Also I would like to express my thanks to Ermis Chalkiadakis, whose previous work lead to this thesis project as an extension on the LastMinuteGig game.

Last but not least I want to thank other master-student Evangelos Potamianos, with whom I shared many meetings discussing the project.

(3)

Master’s thesis

December 23, 2022

Abstract

This thesis investigates improving attention control for Parkinson Disease patients in the context of Musical Attention Control Training (MACT). This is done by implementing a form of performance feedback within a pre-existing serious game. By applying concepts from providing feedback in serious games and introducing higher levels of musical diversity. More specifically applying Dynamic Difficulty Adjustment (DDA) using rhythmic difficulty levels. These levels are subdivided by difficulty based upon measures of syncopation. An experiment was conducted with 14 participants testing the proposed improvements. The data being gathered consisted of game-scores and game experience using a pre-and post game survey. These results were extended using qualitative interviews with participants and a music therapist. Different approaches for providing feedback and enhancing the gaming experience are discussed. Recommenda- tions are given for future implementations and practical limitations are acknowledged.

Results indicate added value for providing feedback, although more focus can be on the personalisation of the provided messages. Further suggestions are provided on how to improve engagement within a MACT-game.

Keywords - MACT, Serious Games, Music Therapy, Rhythm game

(6)

2 Introduction

For this thesis music therapy in the form of a serious game in a home environment is investigated, particularly focusing on assessing in-game feedback to the player, while using Dynamic Difficulty Adjustment (DDA) and taking motivational factors such as musical variation into account. This continues on already established concepts for Musical Attention Control Training (MACT) in games by Vriezenga (2021) and utilizes the established game structure by Chalkiadakis (2021)¹. MACT is a specific form of Neurologic Music Therapy (NMT) which focuses on training attention control. Music therapy is generally assumed to be an effective method to treat attentional problems, within the wide range of children to elderly people [Gregory, 2002, Kim et al., 2008, Rickson, 2006]. For Parkinson’s Disease (PD) these treatments can be useful, as PD patients also suffer from attentional deficits [Zhou et al., 2012].

Due to the complex nature of music it stimulates the brain in fascinating ways, it affects most areas, indicating it is the key to linking those and maintaining the functioning connections [V¨olker, 2021, de Oliveira et al., 2021, de Santana et al., 2021, Kuˇcikien˙e and Praninskien˙e, 2018]. Music therapy applies this concept of complex interaction with music. There are different approaches of this type of therapy and application areas are wide spread. Usage of music in a therapy setting can be beneficial, in the case of PD patients MACT can lead to increased Quality of Life (QoL) [Lawson et al., 2016]. QOL contributes to the general life satisfaction (LS), especially in the begin/mid-stages of the disease. PD patients score lower on their assessed LS in comparison to other older people [Rosqvist et al., 2017].

Therapists are shorthanded when it comes to their time, not all patients can get their needed amount of therapy sessions, an issue that became more prevalent during the Covid-19 crisis as face-to-face contact was decreased. Translating a MACT-protocol into a serious game is novel solution. Previous work by Chalkiadakis created a serious game intervention to simulate a MACT-protocol, such that the therapy session could be extended to the home situation. Key components for the efficiency of the serious game to properly create similar benefits as a therapy session are among others the way it simulates the interaction between therapist and patient. Moreover, the effectiveness of treatment matters. as it should aid in improving attentional skills. In a sense the game emulates functioning of a therapist in a therapy session and should enable smooth interaction and adaptability to the patient, as this mimics a real therapy session and creates a more appropriate treatment.

In this thesis we focus on aiding patients dealing with Parkinson Disease (PD), by creating a mobile serious game which mimicks a MACT session.

1https://github.com/ErmisChalkiadakis/LastMinuteGig

(7)

2.1 Problem statement

Parkinson’s disease is being a increasingly common disease, it is the second most common neurodegenerative disease [Wang et al., 2022]. This is where MACT can play a role. At- tention can be trained to improve and maintain cognitive functioning using music applied within MACT protocols. These utilize the musical elements to train the attentional abilities of patients. Translating such protocols into a game can be beneficial for multiple reasons.

Earlier efforts to create such a game can be found within LastMinuteGig. Chalkiadakis (2021) created a set of requirements on the related topics to create a gamified version of a MACT-protocol. Within MACT there are several protocols to be found which can be implemented by the therapist [Thaut and Hoemberg, 2014]. These protocols are shaped around the practice of improving attention control, specifically adapted to what type of attention should be improved. The used musical exercises are based on the protocol.

The translation from a music therapy session to a serious game and the design of the actions done by the therapist in a natural setting are important. How this is done wildly influences the efficiency of the application. Within a real-life session the therapist can adapt and adjust according to the needs of a patient, which leads to small nuances within each session for patients. Capturing these nuances in a game to give the player a sense of realism are key to providing a good therapy session. In order to provide correct feedback to the patient assumptions of their performance within the game are made. Measuring performance thus is key to gain sufficient insight into player performance and impact of the feedback. Both performance of the player in the game and their attentional progress. To see whether the translation of the applied MACT-protocol provides effective training.

Besides measuring performance for the use of analysis as a researcher these metrics do also provide useful information to the practitioner and learner/player. However providing a correct form of feedback within the gaming environment can be difficult, as it should be interpretable for the patient to use and improve. As time is limited for the therapist, the game provides an additional form of practicing attention control. Usage of game design mechanics that are evaluated can be found within section 6.2. The aims for such therapy sessions can be to improve on various forms of attention (see section 4.3) With the usage of a serious game the practitioner could be aided in providing music therapy, especially in the case of Covid-19 measures forcing to use distanced online-sessions. The usage of serious games has been widely researched in many areas, serious games are, if designed correctly, a useful approach to enable game-based learning. [Krath et al., 2021, Sudarmilah et al.]. For the particular case of Parkinson’s Disease (PD) different fields are being widely researched [Dias et al., 2021]. Most serious games targeting PD patients are Exergames (52,63%) [de Oliveira et al., 2021]. While the motor symptoms are most obvious to the eye, cognitive deficiencies are an additional issue.

The LastMinuteGig game as created by Chalkiadakis (2021) consisted of the application of a MACT-protocol to improve on sustained attention: A Therapeutic music exercise for attention improvement [Thaut and Hoemberg, 2014, Chalkiadakis, 2021]. Within this type of therapy the client would be able to play along however they want or feel like. By choice there is no right or wrong way. In order to see whether the client pays attention to changes in the music are made. In accordance to a therapy session where the therapist and client are playing on instruments and the therapist changes the played music, within the game the music changes. The following musical elements are used to create a musical diverse experience: Rhythm pool, Chord progression pool, tempo pool, key pool. Changes are applied to these parameters, for instance tempo variance per play sessions and rhythmic differenti-

(8)

ation. These are done within the background track, not as part of the interaction with the player.

The game provides musical cues to be interacted with. The changes within the background track are such cues that are used as a mechanism to see whether a player adjusts playing after a music change. These changes consist of rhythmic changes and tempo variation or a pause. Only these three are used as cue. However different forms of musical variation are present in the game. Another musical element used are key-changes (scale) to enable better repeated playing, to make a more diverse playing experience throughout the experiment by changing the key for each play session. Furthermore, the chord orders were dependent on the algorithm, for which a set of common progressions has been used. This to ensure that repeated playing did not cause predictability. Which would impact the results by being able to recognize the music, thus memorizing, instead of paying attention to the musical changes.

Lastly the game offers changes to the player only using auditory cues. Offering more visual elements should thus be done with care as this should not distract from the task to focus on audio cues.

The previous implementation did not include a form of Dynamic Difficulty Adjustment (DDA), which would further approach a therapy session where the therapist adapts to the interactions with the player. This was a problem in the application, as feedback to the player about their performance is non-existent. Although the previous approach does adhere to the chosen MACT-protocol allowing free improvisation, it creates a non-uniform playstyle for the players, making comparison between players difficult. Results from the previous research Chalkiadakis (2021) indicates a distinction in playstyles. This can be attributed to the given freedom of the chosen MACT-protocol.

Furthermore, based upon the previous results the liking of the game was somewhat (5.39 on 1-10 scale) impacted by how much the players liked the music, while their game experience did no t(2.58 on a 1-10 scale) significantly influence liking the game. In particular, users’

musical experience had negative correlation with enjoyment, motivation, challenge and skill increase, in general the enjoyment of the music and game slightly increase. These results highlight the importance for a more personalised approach (using DDA) once more.

Moreover, other improvements are discussed that could be added to the previous implementation. For example the harmonic founding, as chord progressions are limited. For such an application assumptions on expected chord progressions are made. The used chords could be further expanded. However these can be interpreted differently based on personal pref- erence (cultural dependencies, music taste, or background in music education).

Secondly in the old implementation the player/patient does not get any feedback on their performance. This can be both confusing and frustrating as there is no indication on their efforts. Furthermore, the previous music generation algorithm does provide room for improvement. As within instrumentation no variety is found and the player is presented with a similar setting each game session.

Lastly improving the game from a entertainment perspective can be useful. The previous implementation was a novel way of providing MACT, thus leaves room for improvement.

The impact of providing an experience which does not evoke feelings of boredom while being also useful as a therapy could work as a motivational factor to pursue the therapy.

This research aims to explore these improvements for a MACT-game. This can lead to valuable insights for further development of music therapy applications. Thus we investigate improving upon the previous game intervention by making the game intervention solution mobile and addition of DDA and feedback.

The following parts can be found within this thesis: section 3 provides a background into the

(9)

literature research protocol, while section 4 provides the related works to MACT and background on argumentation towards design-choices for PD patients and following research questions in section 5. Additionally section 6 gives insight into the methods and experi- mental setup. Results are found within 7, followed by the discussion in section 8. Lastly conclusions for the project can be found within section 9.

3 Background

For the literature research the following protocol has been used. Most sources of information are found via the Google Scholar database, also Worldcat (UU library system) and PubMed have been consulted. To find the correct topics a design method for sketching ideas has been used. This topic grouping resulted in the visualisation of the following concept map, in which all used search terms can be found, see figure 1. As shown briefly with this sketch some topics are connected. For instance within PD and different forms of therapy treatment are connected to MACT as useful treatment, in which all subtopics from PD (Parkinson’s Disease) could than be linked to training/learning as well, however for clarity purposes not all topics are as connected as could be. The figure further also exposes the relation between self-efficacy and feedback within a game.

Diving deeper into the material the snowball method has also been used, by looking at refer- enced papers. Moreover, to improve the game for repeated playing the rhythmic complexity, musical structure and sound design are investigated and used as starting point for improved music variety.

Figure 1: Concept map resulting from brainstorm sessions, topics used as search terms for the literature review

(10)

4 Related work

Within the related work the background on the topics related for this thesis is given. Re- sulting from this framework we continue with the research questions in section 5.

4.1 Parkinson’s Disease

The target audience for this research are patients suffering from Parkinson’s Disease (PD).

For this reason we begin with a description of PD and current therapy approaches.

4.1.1 What is Parkinson’s Disease

Parkinson Disease is a neurological disorder, leading to problems with the human motor system. The disease occurs more often within the older population.

It is being diagnosed in 5 different stages of severity. As Parkinson’s Disease is a Nervous System condition, no general treatment exists. Based upon the diagnosis of the patient different specific forms of therapy can be used target the deficits of the patient [Elpidoforou et al., 2022].

4.1.2 Treatments

Current treatments consist of multiple of approaches. This due to the multiple variants of PD with different prognoses [Armstrong and Okun, 2020]. Moreover as no cure is available for this progressive disease, different issues arise next to the motoric problems ranging from psycho-social problems to limitations of social and economic activity, stigma, frustration, fear for the future; all of these factors can impact the Quality of Life (QOL) [Choi et al., 2020].

Looking at serious games for the multitude of issues occurring in the different stages of PD the Personalised Game Suite (PGS) can be consulted in which different symptoms are connected to the respective type of game (ExerGames, DietaryGames, EmoGames, Hand- writing/Voicegames) [Dias et al., 2021]. As can be seen in the aforementioned categories of games it could be argued that not all type of games fall into these pre-defined categories.

Exergaming is identified as the most explored category (52,63%) regarding serious games in relation to PD [de Oliveira et al., 2021]. These type of games employ different types of movement sensors, capturing body movements, which can be acknowledged to nature of the physical problems encountered by PD patients. However Music Therapy in a gaming context is not mentioned within this systematic overview. This highlights the need for such a gaming intervention within this area.

4.2 Music Therapy

The effects of music on people are different for everyone. Music therapy is an continuously extending interdisciplinary field that has evolved over the past few decades to create a better well-being for patients. Technological advancement provides the tools to increase accessibility [Agres et al., 2021]. Using music in a clinical setting is not uncommon. The way music is used within therapy can differ however. Music interventions in therapy can consist of different forms, either active (singing, playing an instrument), or passive (listening to music) to aid the patient with their problems [Zanders, 2018].

(11)

4.2.1 Neurologic Music Therapy

Music Therapy consists of a broad area of research and applicable areas. We can divide these areas into different subdivisions. The five pillars as used by Gattino (2021 (biological, cognitive, developmental, social and personality, mental and physical health) provide an insight into different type of needs within the research area. For each of these pillars forms of therapy can be found and their associated assessments. The broad impact music therapy has does also apply to PD patients, as previously mentioned the disease consists of a wide range of symptoms. Neurologic Music therapy (NMT), as Thaut and Gardiner 2014 describe, specifically looks into the monitoring of functioning of a patient, by creating a quantifiable assessment. Assessment is focused on the monitoring of the patients functioning during sessions, where the patients differences in capacities and behaviour is observed. This approach provides the therapist direction for further treatment decisions. Using the transformational design model (TDM), an analysis for the patient is given to focus on the occurred events during the session, this shows the effects of a session and the difference that can be made.

TDM foundations lie within Rational Scientific Mediating Model (RSSM) and adheres to the following three principles: Scientific validity (congruence between the scientific information stemming from RSMM is important), Musical logic (music used should be aesthetically well composed and performed, using optimal musical patterns, regardless of complexity) and Structural Equivalence (the therapeutic music exercise has to be isomorphic in therapeutic structure and function to the non-musical functional design). Meaning that the structure of the therapeutic exercise entails the same functional motions as the musical playing, as clarified by Thaut and Hoemberg (? ”e.g. exercises designed to improve the range of arm motion by playing musical instruments must entail the functional motions of the nonmusical therapy goals in order to accomplish useful training for the patient”. Furthermore, Thaut and Hoemberg(2014) provide 6 basic steps for therapeutic purposes: [Thaut and Hoemberg, 2014]

1. Diagnostic and functional/clinical assessment of the patient 2. Development of therapeutic goals/objectives

3. Design of functional, non-musical therapeutic exercise structures and stimuli 4. Translation of Step 3 into functional therapeutic music exercises

5. Outcome reassessment

6. Transfer of therapeutic learning to functional applications for ”activities of daily liv- ing” (ADL)

As the above steps suggest a serious game can function as an extension on Step 6 by integrating a activity within daily life as part of the therapy.

The effects of Covid-19 on daily life, caused several new challenges, thus also on music therapy, [Gattino, 2021] Furthermore the pandemic has forcibly impacted the delivery of practices regarding the appliance of therapy. Herein there are cultural differences which can also influence the adoption of new ways of delivering such therapy, as stated by Agres et al.(2021) ”the adoptation rates differ North America (80.6%) and Asia/Oceania (85.2%), Europe (48.2%) to use the (necessarily used) technologies after the pandemic.” in relation to the forced use of virtual meetings throughout the Covid-19 pandemic. Thus adaptation to a gaming solution can be different between cultures.

(12)

Potential issues within the virtual sessions (TeleHealth) can be of technical nature, with connection issues and sub-optimal audio quality. However the Telehealth approach can benefit from shared best-practices. Telehealth has the benefit of increased accessibility, it is however not optimal as overall clinical hours decreased from in-person to TeleHealth. [Cole et al., 2021] One of the most mentioned reasons of the increased degree of difficulty in Telehealth relates to earlier mentioned technical problems. As uncovered by Cole et al.(2021) 59,4%

reported internet connectivity issues to be an issue. Would a gaming intervention be used as an extension on such treatment this could resolve such problems. As the MACT-game can aid in filling this gap by functioning offline. By acknowledging the disadvantages of lack of interpersonal experiences and technological difficulties we can thus also discover potential benefits for TeleHealth. [Vinciguerra and Federico, 2022]

In healthcare music has been increasingly used for different purposes. Laksmidewi and Dewi(2021) defined the impact of classic and traditional music on cognitive function in crit- ically ill patients. It was emphasized that music has affect on the encoding process from short-term to long -term memory. Furthermore relating this to attention. They report that the stimulation of music on the reward center and further underlying principles result in the benefits such as lower anxiety, less stress and reduced pain. They also report the emotions, positive emotions in their examples, obtained when people listen to music, as a reason to improve cognitive functions such as memory, learning and attention.

Furthermore looking into the broad area of music for therapeutic use a look into different characteristics and affordances is provided by Agres et al.(2021). Music in health care touches upon multiple areas such as emotion regulation, motivation and adherence, Percep- tual Entrainment and Motor Coordination, Social Interaction and Group Therapy. Music in therapy provides a way of improvement in a home situation. Positive effects on life satisfaction in elderly can be observed [Ziv and Lidor, 2011]. The MACT-application will be used at home, in which music can thus act as beneficial way to increase the patients motivation and adherence.

4.2.2 Musical Attention Control Therapy

Musical Attention control Therapy (MACT) as proposed by Thaut(2005) specifically ad- dresses retention and training of attentional skills. This can be achieved by improvisational practice upon a pre-composed musical piece. Different musical elements create cues to the patient on which should be interacted, in order to test different kinds of attention [Thaut and Hoemberg, 2014]. Thaut and Gardiner(2014) describe a set of new dimensions that music brings to the process of recovery of attention.

• Rhytmic patterns drive attention focus by interacting with attention oscillators via coupling mechanisms [Thaut, 2005, p. 74]

• Music can facilitate divided (alternating) attention by providing multidimensional stimuli, such as melody and rhythm.

• Music brings timing, grouping, and organization, so that attention can be sustained.

• Music recruits shared or parallel brain systems that assist the frontal lobes with alternating attention.

• Music provides the additional dimensions of emotion and motivation to help to facilitate concentration and keep the person on task Thaut [2005].

(13)

For each of the different suggested exercises a way of measurement is proposed, consisting of standardized attention tests, but also self-reported confidentiality by the patients immediately after playing.

Although most studies applying MACT have low number of participants, previous research shows promising results. For instance within adolescents in secured residential youth-care (n=6), the application of such practices have proven to show positive trends in attentional skills (selective, focused, sustained and alternating) [Abrahams and van Dooren, 2018]. This randomized controlled pilot study compared the standardized MACT-protocol with a non- standardised music therapy intervention and a control group. However due to the low sample size the results should be interpreted with caution.

4.3 Attention

Attentional problems are troublesome as attentional skills are used to function within daily tasks. For describing the cognitive function of attention there are several approaches, multiple models and measurements exist for which no single explanation suffices.

4.3.1 Types of attention

Within research attention is described in different ways, we continue with the descriptions by Thaut and Gardiner(2014). One inhibits different types of attention:

• Focused attention

• Sustained attention

• Selective attention

• Alternating attention

• Divided attention

These forms of attention are of a different nature and can be seen as more advanced from top to bottom. It adheres to the hierarchical clinical model of attention [Thaut, 2005].

The way our attention can be focused is caused by different stimuli. Focused attention can be described as being able to react to a specific stimuli. While selective attention is being able to actively make a selection of one stimuli over another, which builds upon the concept of focused attention. Having the ability to not being distracted by other stimuli can be described as a bottom-up process. Sustained attention is the ”ability to maintain a consistent behavioral response during continuous and repetitive activity.” [Vriezenga, 2021]. Having mental flexibility to allow the shift of focused attention can be described as Alternating attention, specifically related to the shifting between stimuli requiring a different cognitive demand. Lastly Divided attention is being able to respond to different stimuli concurrently.

This however is difficult to do as one can not actually ”multitask”. This can be seen as being able to quickly switching between tasks [Salo et al., 2017]. Improving these types of attention in therapy require different stimuli.

Based upon having control of attention within a task we can describe two types of effects occurring: [Berger et al., 2005, Theeuwes, 1991]

• Exogenous: Pop-out effect (breaking expectations, which than could be related to musical expectancy)

(14)

• Endogenous: Internalized, based upon instructions/previous knowledge, can be described as a spotlight

The way one focuses their attention can lead to inattentional blindness (IB), which means you miss something obvious right in front of you, for instance a big red fire extinguisher might not be consciously made aware of. Another type of attention flaw that could appear is change blindness, change in previous and current state is ignored by the brain. Interestingly auditory attention can lead to a decrease in such IB. Beanland et al.(2011) reported that

”Listening to music was associated with significantly lower IB, but only when observers actively attended to the music.” Within their research they let participants do an IB task that varied both visual and auditory demands. The active part of listening was of importance.

Although counter-intuitive they argued that the distracting qualities of the audio facilitate detection of unexpected objects by weakening the observer’s attentional set. A continuous auditory stream could be the reason why this is the case, contrary to if there is a clear

”onset” which causes disruption [Beanland et al., 2011].

The impact of music on attention can also be found within relation of memory and functions such as speech[Shagan et al., 2018, Besson et al., 2011]. These type of effects can also occur whilst listening to music, as ones working memory is limited and switching attention can be a challenge. Being in control of your attention is useful for most daily tasks, meaning training can be beneficial if cognitive functioning deteriorates. As discussed by Vuilleumier and Trost(2015), music activates the brain in a widespread combination, relating to both motor and cognitive systems. Which than can be associated with basic dimensions of emotion and reward system within the brain. Also reporting: ”Furthermore by entraining neural activity in particular brain systems according to its rhythmic structure, music may have a direct influence on behavioral and cognitive processes mediated by these brain system.”.

Meaning that entrainment, by activation of the particular frontoparietal areas (responsible for directing attention [Scolari et al., 2015]) motivates attention performance. The effect of music on attention can thus be tested on a broad spectrum, this makes comparing the effects of certain treatments non-standardized.

Understanding auditory attention is complex and could be different from visual attention tasks [Ahuja and Setia, 2022]. When you are occupied with a task both visually and auditory, the two can affect eachother. Having an increase in perceptual load within a dual task situation increases the difficulty to sustain attention [Ersin et al., 2021]. Having a lot of visual information to process thus can severely impact performance, as less attention resources are available. Modelling auditory attention can be done with both a top-down and bottom-up approach, as reported by Kaya and Elhilali(2017). Our perception approaches listening to music with a top-down predictive model, where listening to music provides a bottom-up sensory input [Vander Elst et al., 2021]. In other words our brain makes active predictions about the continuous auditory input, which is observed and processed within the brain to detect prediction errors and update our mental model about expectations. When providing a cue to the player these predictions are interrupted. Breaking those expectations form the essential mechanism for providing a cue.

4.3.2 Auditory feedback

Cues in the gaming setting can be given in different forms, think of visual cues to let the player engage in interaction. In the MACT setting the focus lies on the auditory cues, this however can be ambiguous if the audio does signal something not congruent to the players expectations. As argued by Vriezenga (2021) a random timing of musical cues in combination

(15)

with a non-random timing of the game events would miss the mark. The random timings of the game events should force the player to focus on the musical cues, instead of other cues, such as vibration of the phone or visual scenery. Furthermore, the different type of interactions a player has can be incorporated in a wide variety of forms, however is limited to the capacity of the player. The 4 response types (start, stop, change, react ) as seen in table 8 should be mapped to concepts relative to the players game experience, i.e. ingame the player acts as a musician hitting a different key.

4.3.3 Importance of memory

Executive attention lies within the core of our functioning as human beings. It can be seen as core of human thought, emotion, and behavior [Engle, 2018]. The connection between attention control and indicators of intelligence can be made as argued by Ellingsen et al.(2020):the existence of the relationship indicates that something that affects performance on attention control tasks also affects performance on much more complex tasks.

4.3.4 Tests

Measurement of attention control can be done in multiple ways. Using score-based systems for instance. There is debate on which type of measurements are the most suiting for the tasks on-hand [Draheim et al., 2021]. For the separate types of attention different tests can be used. For selective and sustained attention the D2 test (a cancellation test) [Brickenkamp, 1962, Bates and Lemay, 2004] can be used, which also assesses stressing working memory. It presents the participant with the letters d or p, which have up to two straight lines above or below them. Then the participant is tasked to mark up all target characters (the “d” having two dashes placed above and/or below), these are alternated with the other characters (the

“d” having other than two dashes, or any number of dashes “p”.

Furthermore, for testing the working memory the digit span task can be used. It consists of a series of numbers shown to the participants which should be remembered in order, this test can be done both backwards and forwards. It stresses the participants working memory and attention span [Hilbert et al., 2014].

4.4 Music in games

The term serious games, or applied game, describes a game in which the main purpose is not necessarily entertainment. Music within a serious game does not has to be different than in a ’normal’ game. When designing audio in games one must be aware of the effects that audio can ambiguously function on the player. The different functions audio can have within a game can be more concretely described within the IEZA framework [Huiberts and Van Tol, 2008]. These concepts can than be mapped within two spaces of diegetic/non-diegetic, and setting/activity. This framework describes audio within games by splitting functionality between them. Within these dimensions 4 types of audio can be distinguished. Interface (non-diegetic, activity): audio from outside of the game world (i.e. menu button sound), Effect (diegetic, activity): Audio originating from within the game world, Zone(diegetic, setting): Audio originating from the diegetic part of game (thus part of the environment;

i.e. ambient noises), Affect (non-diegetic, setting): audio that expresses the non-diegetic setting (i.e. orchestral music). This framework is used to support the decisions for mapping sounds to specific meaning in within a game. Congruence of music in accordance to the visual aspect impacts how the portrayed information is perceived. This concerns a broader range

(16)

of subjects than games, and can be traced back to the film industry. Millet et al.(2021) for instance looked into how visual attention and affective response is impacted by the played music in films. Within a music intervention the variety in musical elements, such as instrumentation can be adapted to the situation. However in comparison to film-composition there is no pre-existing score tied to a scenery. Meaning that within a game the player can continuously change the scenery. Making the audio in games by definition dynamic.

Applying these concepts of musical variety within games can be challenging from a composer view. As keeping dynamically changing music interesting to listen to is challenging without it becoming either complex or repetitive.

To create an enjoyable experience from the audio perspective different musical properties can be tuned. Efforts to solve this problem have been made by adjusting musical properties and implementing these using a dedicated music engine [Gungormusler et al., 2015]. Whereas within the previous experiment trying to overcome that challenge lead to changes to the chord progression and key selection between sessions as provided by Chalkiadakis(2021).

Furthermore Plut and Pasquier (2020) provide an overview of the current challenges within music generation. Providing insights into the wide range of musical dimensions regarding musical composition within games. They make the distinction between vertical arrangement and horizontal arrangement within compositional practice and describe current approaches of generative music within games. One of the challenges they describe relates closely to the music generation algorithm within the previous game intervention: ”Generative music can also display the “10,000 bowls of oatmeal” problem, where the music is acceptable, but monotonous”. The oatmeal problem refers to perceptual uniqueness of music in games, which originates from the ”No Mans Sky” game, which has a lot of completely unique (from a mathematical point of view) variations in planets, but these planets do not provide a perceived uniqueness to the player as they feel similar [Challies, 2016]. However within LastMinuteGig this was not necessarily reported, as no decrease in enjoyment of the music was measured throughout the experiment. On the other hand when a game does trigger you to keep repeatedly playing it stimulates the motivation to keep playing, as a results of endo- centric persuasion over intrinsic motivation [de la Hera Conde-Pumpido, 2017]. Increasing the variety and quality of the music can thus be used as a potential motivational factor.

4.4.1 Rhythm

Besides instrumentation in order to achieve more rhythmic variety syncopation can be applied. Syncopation as described by G´omez et al. (2005) means ”a contradiction between strong and weak beats against other parts of the musical texture whose metrical context is fixed.”. When people experience syncopation they try to fill in the missing part (i.e. tapping our feet, moving their body), within our body we are sampling the stimulus in a different way, experiencing what is called ”groove” [Vuust and Witek, 2014, Vuust, 2022]². More specifically in the case of music therapy syncopation could act as a measure of rhythmic complexity. The act of playing syncopated patterns or rhythms can be described as more challenging [Kogutek et al., 2021].

Other forms of rhythmic variation can consist of change of measure. Arguments can be given that forms of extended use syncopation and a change of measure are relatively similar in complexity. However the way a player reacts to such rhythmic elements differs.

Rhythmic complexity could have different implications on PD patients as compared to healthy people. [Pando-Naude et al., 2022, Matthews et al., 2019] The preferred type of

2https://youtu.be/hoW0exfFZoM?t=485

(17)

complexity in syncopation in addition to harmonic complexity has a sweet spot, which for PD patients is different. Looking into low-medium-high rates of complexity a pattern can be seen as an inverted U-shape. For healthy people the sweet spot lies somewhere within the medium regions, while for PD patients this sweet spot of pleasure, wanting to move along with music is lower, and dips severely on high complexity. Meaning including both high harmonic and rhythmic complexity can be problematic for such deficiencies. Pando-Naude et al.(2022) thus conclude with the following: ”PD patients may benefit from low complexity in music, which appears to represent their musical “sweet spot”. This further establishes the need to adapt to the player in a fitting way. Finding the sweet spot for each individual player that should be adapted to.

4.5 DDA

’Serious games for PD patients must be customizable, simple, and smart.’, as stated by de Oliveira et al.(2021). One way to achieve a more custom, simple and smart experience is Dynamic Difficulty Adjustment (DDA). DDA has been proven to be a useful concept in games to retain attention. By adjusting specific parameters that impact difficulty in a game, for instance changing amount of enemies or increasing odds of an event happening the player is being persuaded to remain within the flow [Zohaib, 2018].

DDA thus relates to the feeling of flow within a game, which describes the matching of player knowledge and challenges within a game. Flow as as proposed by Csikszentmihalyi(1997) describes the optimal state for a player. It is about finding the balance between enough challenge to not be bored, but also not be too challenge to get frustrated.

Applying DDA to games in relation to PD rehabilitation has been done in the context of Exergames [de Oliveira et al., 2021]. Where it potentially makes the game more efficient and useful, as there are varying needs for PD patients which can change harshly related to the severity of the disease [Siegel and Smeddinck, 2012]. Increasing personalising can be done using objective indicators (i.e. points, score, amount of errors) or a different or complementary approach could include physiological parameters and affective states. These are however extremely intrusive, even within a experiment context [Ahuja and Setia, 2022].

Perceived difficulty is closely related to self-efficacy. Being convinced that you are able to fulfill a task effectively can lead to higher motivation. Furthermore using a more personalised training is suggested to have great performance improvements [Bauer et al., 2012]. As described by Power et al.(2020) these insights have the following implications:

• The designers of serious games need to be cognisant of the influences of game properties on participants’ perceptions of competence.

• Excessive difficulty can result in early disengagement and ultimately poor performance.

• Optimised game difficulty can support self-efficacy development 4.5.1 Self-efficacy

As Heslin and Klehe(2006) describes self-efficacy as the ”most powerful motivational predic- tors of how well a person will perform at almost any endeavour”. We use the definition by Bandura et al.(1999) who defines self-efficacy as ”a person’s belief in her capability to successfully perform a particular task.”[Heslin and Klehe, 2006] Furthermore, Bandura et al.(1999) define a set of factors impacting the perceived self-efficacy. These consist of performance outcomes, vicarious experiences, verbal persuasion and physiological feedback.

(18)

But what does self-efficacy mean in the context of serious games and PD? In the context of PD it is clear that the awareness of the disease can contribute to the belief in being not capable of functioning anymore.

As Rosqvist et al.(2017) report on Life Satisfaction(LS) in relation to PD ”to maintain or improve LS, depressive symptoms and general self-efficacy should be prioritized in the clinical management of PD.

For an increase in LS boosting self-efficacy is important. As acknowledged by Fujii et al.(1997) who report: ”In order to increase self-efficacy among Parkinson’s disease patients, this study suggests that support, both social and psychological, and providing health education, are important.”

These provide insight into the social aspect where verbal communication can play a signifi- cant role in convincing a PD patient in persisting a therapy and improving or maintaining their attentional skills. As Klimmt and Hartmann(2006) state: ”People who believe in their competences will invest more effort to overcome opposing forces in a game and to master the game, whereas players who doubt their skills will be more reluctant to devote as much energy to playing.” Personal mastery (performance outcome) is deemed to be crucial in the motivation to keep playing a game. Thus to effectively use efficacy beliefs and increase one’s self-efficacy they should be able to successfully full-fill the challenge a game offers, while also reinstating their beliefs through positive feedback.

When providing specific individual feedback, one should consider how it is interpreted by the player. Schunk(1995) describe the effects of specific feedback on both self-efficacy and related motivation to be positive, but also mention the importance of genuineness: ”Feed- back also will not be beneficial for self-efficacy and motivation if learners do not view it as credible”.

4.6 Feedback

Integrating feedback in a gaming environment should be done using the right audience in mind and adjusted accordingly. As pointed out by Chalkiadakis(2021) no negative feedback should be integrated as PD symptoms worsen over time. Thus feedback stating a decrease in performance over time is undesired. This however does not mean no feedback at all should be given, as this leads to an insufficient gaming experience. When players do not have clear feedback whether they understand the game, the measurements of accuracy on acting on cues within the therapy protocol cannot be done with certainty. In order to gain insight into player performance a clear feedback mechanism should be implemented to the player, to provide just enough guidance without interfering. Feedback mechanisms suitable for the case of PD patients in the specific case of MACT should be included to give the player a sense of achievement and control.

Audio cues to receive feedback on performance are unsuitable, due to the musical cues within the training protocol. However visual cues can distract the player from the audio, finding the balance within this to not overload the player with information remains the challenge.

Furthermore, balancing the information given should be included into the overall balancing of difficulty. As more visual aspects can draw more attention from the player. As mentioned by Thaut and Hoemberg(2014) applying visual cues within a real-life setting of therapy can be also be done. For instance by colour coding the instruments when these are the musical instruments to play with. For the LastMinuteGig-game this is currently not implemented due to limitations of the chosen protocol.

(19)

4.7 Usability

In the case of a mobile application some elements of variance can be added by introducing other interaction types. Think of the metaphor for holding a rhythm instrument such as a tambourine. The way that instrument is being played can be also done using a phone, or external device. This is implemented to the game as a way of diversifying the interaction. A big factor within this is usability. Creating a game which deviates too much from the MACT protocol can however lead to incorrect application of the protocol. For instance when giving the player/patient too much customisation options regarding the speed/time of a game.

[Vriezenga, 2021]. This creates the difficult situation of balancing between the fun a game creates, but also being structured as a training protocol, see figure 8. Each and every user of a mobile application differs. In the case of PD this is really important to acknowledge, as motoric skills deviate from standard users. Not only in between patients, but also over time for one patient. The usage of the application thus should be made approachable and ease of use is a highly valued factor.

Interaction with a mobile device can be done using both hands or one hand. Within these two groups there is typically also variety in the exact holding methods (i.e. one finger vs multiple at the bottom of the device for improved grip) Due to possible impairments holding the phone in a similar position every time might not be a commonality. Thus the to be interacted objects should be reachable both with one-handed and two-handed usage. In the case of one-handed usage the reach of the thumb should be taken into account, meaning elements of interaction can be hit less accurately if be placed too high up, or to close to the edges [Lee et al., 2016, Mayer et al., 2019], see figure 2 for illustrative purposes.

Figure 2: Reachability of thumb in one-handed (right-hand) usage of a mobile device, where green area displays the best to reach area, while outside of the ellipse are harder to reach areas, as suggested by Mayer et al.(2019)

These findings also partially would be applicable to PD patients. However more care for User Interface(UI) design should be taken. As certain elements of interaction, for instance dragging gestures, can be difficult. Nunes et al. (2016) defined a set of 12 requirements that should be taken into account when designing mobile UI design in the area of PD.

• DG1: Use tap targets with 14 mm of side

• DG2: Use the swipe gesture, preferably without activation speed

• DG3: Employ controls that use multiple-taps

• DG4: Use drag gesture with parsimony

(20)

• DG5: Prefer multiple-tap over drag

• DG6: Adapt interfaces to the momentary characteristics of the user;

• DG7: Use high contrast coloured elements

• DG8: Select the information to display carefully

• DG9: Provide clear information of current location at all times

• DG10: Avoid time-dependent controls

• DG11: Prefer multi-modality over a single interaction medium

• DG12: Consider smartphone design guidelines for older adults

Furthermore, L´opez-Blanco et al.(2021) put such design guidelines into perspective, and examines the needs for specific use cases. While supporting the usage of touchscreen devices within PD care research for mild to moderate PD, but mentioning the importance of nuance in usability: ”it should be examined whether the slower performance of the task found in patients with moderate severity really affects the usability of a touchscreen display significantly.” [Blaˇzica et al., 2019].

Even more guidelines for older adults can be assumed to be useful for PD patients as well ([Nunes et al., 2016] DG12). Nurgalieva et al. (2019) extend the findings by Nunes et al.

(2016) by creating a more general heuristic overview not only expanding the user group (older adults, ages 55 and older) but also discussing shortcomings of existing guidelines.

Further heuristics that can be taken into account in relation to the previous research by Chalkiadakis (2021) can be seen within 3.

Touchscreen interaction can also be used as an skill training for PD patients. Difference in healthy people and PD patients can be seen in the retention of these touch screen skills, where there is a clear cap on the performance for PD patients, while healthy people can improve overtime. Meaning the touchscreen skill is mostly related to the severity of the disease [Nackaerts et al., 2020]. ³

3Heuristics and Recommendations for the Design of Mobile Serious Games for Older Adults

(21)

Table 1: Critical Heuristics for mobile games for older adults, combined with previous Chalkiadakis [2021] requirements³

DESCRIPTION Updated Requirements

CH1 The game should provide clear rules, present the main objectives at the beginning, as well as short-term goals throughout the game.

Similar to previous requirement 6 (Clear instructions and goals)

CH2 The game must teach the basic skills needed at the beginning, to be used later by the players.

No concrete mention of skills, but existing tutorial complies

CH3 The game should provide an interesting, engaging, and absorbing tutorial, that also simulates gameplay, allowing easy understanding.

Tutorial could use improvements as pointed out by Ermis ”there is no way of telling whether the tutorial was successful in teaching the users what they had to do when the music changed”

CH4 The game should use visual and audio effects to arouse interest and interaction, considering the restrictions due to ageing

Due to specific focus on music less relevant, in this case even redundant

CH5 The player should find the game fun and enjoyable, without boring tasks.

Repetitive tasks are acceptable as the advanced age can cause memory deficit.

Similar to requirement 2, as protocol that gets remembered hinders the therapeutic mechanisms used

CH6 The game should provide rewards which should be meaningful, enabling

immersion in the game as a result of encouraging older people to play more.

Currently no rewards, besides the positive effects of the therapy (potential for increased attention, or fun game experience)

CH7 The older player should not be penalized for the same error, since he/she may have attention and memory deficits.

Besides that, the player should not lose any earned rewards.

Similar to requirement 4, no rewards, but DDA should be included

CH8 Challenges should be consistent with the age restrictions of older people, providing positive experiences of the game rather than negative ones, making the player want to play more, rather than giving up.

Similar to requirement 8 (no negative feedback), but positive experiences more relate to the difficulty, making the case for DDA even stronger

CH9 The level of difficulty should vary so that the player experiences more challenges as the experience with the game is

developed, but at a pace that does not generate frustration.

Similar to requirement 4, DDA should be incorporated

(22)

CH10 Tiredness or boredom should be minimized by varying activities and rhythm during play.

Similar to requirement 4, DDA should be incorporated, but also more variation in other musical dimensions could be beneficial

CH11 The game should be fair, engaging, easy to learn and should not be difficult to master, because the older player may have physical or cognitive limitations.

Similar to requirement 4, to be evaluated with the post-game questionnaire

CH12 The player must feel in control. Similar to requirement 4, to be evaluated with the ingame question and post-game questionnaire

CH13 The gameplay should be long and enduring to keep the players’ interest.

Similar to requirement 4, to be evaluated with the post-game questionnaire

CH14 The first ten minutes of the game and the actions of the players should be obvious and should result in immediate and positive feedback for all types of players, encouraging them from the beginning of the game.

Similar to requirement 4 and 8, the tutorial adds to this aspect with a positive beginning of the game experience

CH15 The player must be able to visualize their progress in the game and also to compare the results.

Not included currently within the game. Using the self-perceived measure the player is pointed to awareness of progress

CH16 The game should be fun first for the player, then for the designer.

Not yet in the requirements, but does strengthen the case for a minimalistic approach (Requirement 5, minimizing cognitive demand- ing tasks)

(23)

5 Research questions

Following the abovementioned foundations the main research questions for this thesis are given. The focus lies on feedback and musical variety within a MACT-game. The first question we want to answer is similar to the previous approach by Chalkiadakis (2021). With the established game structure being adapted, but changed mechanics we should again look into the the efficacy of the game.

RQ1:

Does the game intervention have a positive effect on attention control?

H0The game intervention has a positive effect on attention control.

Based on previous results, although having a low amount of participants, we argue that when correctly applied the MACT-game intervention can have a positive effect on attention. The goal of MACT is to train the attention control, so this is an essential part of the game intervention. Testing attention associated with the chosen protocol is used to disclose whether the game actually contributes to the therapy and sufficiently applies the concepts of MACT. Following we look into addition of performance feedback elements.

RQ2

How can feedback be given in this MACT protocol which is meaningful and stimulating to the player?

With subquestion:

What is the effect of the feedback on the gamer?

H0: Feedback increases positive attitude of the player towards the game.

The positive reinforcement from the feedback can increase the motivation to keep playing the game. Adjusting the visual look of the game can negatively influence the effectiveness of the audio cues. Using the continuous background track (see 4.3) we argue that the motivational benefits outweigh the possible distraction of visual cues. Moreover, related to self-efficacy we look into the effectiveness of motivational factors in relation to the performance within the game. Lastly by having different ways of integrating measurements, adaptive performance is being investigated:

RQ3

Can DDA be achieved by means of changing tempo and rhythm?

With subquestions:

What is the effect of the DDA on the game experience?

What is the effect of the DDA on player engagement?

The purpose of the MACT game, should be focused on improvement of attention. DDA is used to increase a tailored experience to facilitate a higher retention of the players motivation. Providing more complex rhythms could lead to unnecessary challenge. The difficulty thus is adapted to the player. However higher levels of complexity do not necessarily mean a higher level of perceived pleasure from the music to the player. Thus measurement of player performance is necessary, to determine which amount of complexity to provide to the player. Moreover, a method of comparison between players enables to compare the adapted difficulty in order to validate whether this impacts retaining attention control. While motivational factors are taken into account. These factors contribute to the engagement to the game .

(24)

6 Methodology

Following the research questions we continue with the methods used to improve upon the existing game by Chalkiadakis (2021). A mixed-method approach is used. Beginning with qualitative method of interviews. Afterwards an updated design of the game is tested on its effectiveness and engagement. This is done using quantitative data from the game, complemented with interview explanations.

6.1 Experiment design

The research design is structured as following:

• Cognitive walk-through on usability throughout the game

• Semi-structured interviews and with music therapist for game design and MACT- insights

• A experiment with the game intervention accompanied with a pre-and post game survey

• Post-game interviews

Interviews with the music-therapist were used to gain feedback on initial design ideas for the game, while also to reflect on the final prototype. For this semi-structured interviews were conducted, by presenting design ideas and doing a linear structured walk-through. After these meetings the collected suggestions have been used to improve the gaming intervention.

The complete experiment overview can be seen in figure 3

Figure 3: Schematic overview of the experiment. Where R are actions for the Researcher, while P are actions for Participants.

6.2 Game design

First goal for the intervention as established by Chalkiadakis (2021) would be to gamify the MACT-protocol in the optimal way possible.

The previous implementation focused on the Clinical Protocol for Sustained Attention(4), this protocol does not allow for a strict form of performance measurement, due to the free improvisation. This mimicked protocol is used to see whether the player acts upon musical

(25)

changes. The player is free to play whenever, whatever, thus instructions are given before in a tutorial. This sets boundaries to the freedom of own interpretation but is necessary to ensure that consistent playing is done and measure if the player actually pays attention to the musical changes. At the end of the tutorial example playstyles, see figure 17, are given. More than one is given to not influence and force one play-style to the player. Firstly starting of with a clear four-quarternotes (four on the floor) pattern. Continuing this in double time, thus in 8th notes, and ending with a rhythm where on the offbeat a note plays.

Also a short change into a less straightforward rhythm is played, by addition of a 8th note pause on the beat. Within that part the first note is shifted an 8th note. This change of rhythm adds to the guiding role of the tutorial stating that the player does not necessarily have to keep playing the same rhythm throughout and is encouraged to change their playing on musical cues.

(a) Begin of the tutorial (b) End of the tutorial, start of practice playing

(c) In-game with feedback message

Figure 4: Screenshots of the game, for more screenshots see Appendix B

The music therapy setting is being simulated within the game, this provides a need to replace the instrument. The playing of a guitar is simplified, in this case with a simple button-tap on the touchscreen. Placement of the button to interact with has not been changed from the previous iteration. The button should be easy to reach for all players, thus is placed at the bottom of the screen where reachability is optimal when holding the phone in vertical mode.

Cues within the game consist of different types of changes: a change in rhythm, a change in tempo, or a brief pause. Within the tutorial hints are given using text bubbles to guide the players visually, see 4b. After the tutorial popup messages display feedback on the played

(26)

song. These are regularly timed events to ensure sufficient feedback throughout the session.

For screenshots of the game see figures 4. The updated game design applies the previously mentioned guidelines 4.7, for instance using higher contrast in colour. In comparison to earlier version the background has changed to a warmer colour. (DG7) Additionally, the fonts used are easily readable and do not change throughout the game to provide consistent information. (DG12)

Furthermore, by remaining a simple interaction with one big button at the bottom of the screen we adhere to guidelines for elderly (DG1, DG10, DG3).

To increase player understanding of the game playstyle differentiation is highlighted within the tutorial. This guides the player through the game mechanics. One additional change within this tutorial is the next button, this one is enlarged and the hit-box increased as well. Early testing on different devices resulted in this change, as on some devices it would be hard to press continue accurately.

Improvement in challenge is created by using DDA. This is done using two different elements of the used background rhythms. Firstly by adjusting the tempo. Tempo can influence the perceived attention as previously discussed in section 4.3. Secondly by implementing more rhythmic variety. The rhythmic patterns start of easy. By increasing difficulty a form of measurement can be done to see whether the player adapts their playstyle to the rhythm.

For this we use the levels (divided in low-medium-high) of rhythmic complexity, see 6.2.

Within the game Dynamic Difficulty Adjustment (DDA) is integrated by adjusting musical elements. For the implementation of DDA we adjust the rhythm and tempo of the used background tracks. These parameters influence the perceived complexity of the musical background track. The rhythmic complexity levels (syncopation index S = N − Ndi + I, where N is the metric weight, in relation to the weight of the following note Ndi and I is the instrumental weight) are divided using a three-level categorization as follows:

• Low: 0 < S < 21

• Medium: 22 < S < 49

• High: 49 < S < 81

The instrumental weight is used due to the polyphonic nature of the used rhythm. This results in a different measurement from monophonic rhythm.

Beside rhythmic adaptation adjustment is made to the tempo. Higher tempo leads to higher difficulty to follow along. For this a similar starting point as the previous implemented music generation algorithm (96-120 BPM) is used. Difficulty thus consists of a combination of syncopation level and tempo, divided within low/medium/high. For each level of difficulty there are eight bars with different rhythms, as an extension on the 4 basic rhythms from the previous implementation, for the complete set of rhythms see Appendix figure 13.

Player performance is captured using tapping behaviour. The distance metric as provided by Chalkiadakis(2021) is used. This metric can be used to identify if a musical cue is being acted upon and uses the distances between buttons presses compared between bars. If the distance between measured bars is low, it means the player adheres to a steady rhyth- m/tempo. While high peaks in distance display a change in playing. If these alterations occur when a musical change is happening we can argue that the player performs according to the expected and is actively paying attention to the given cues. However if many cues are not acted upon, this means a lower difficulty could be necessary. See Appendix figure 25 for a visual representation of how the distances can differ throughout a game-session based upon the provided cues, where the pause at bar 56 and 126 are apparent in the distance

(27)

metric.

Using this knowledge about player performance a player score can be obtained. This profile will be updated real-time within each session and connected to the username saved for future sessions. It is used to determine the level of complexity for the upcoming rhythm and tempo. This makes different possible playing difficulties by combining: Low/Medium/High Rhythmic complexity with Lower/Higher tempo.

The musical changes consist of the following: change of Rhythm, Tempo or Pause. These are applied using percussion audio clips. This has been kept from the previous implementation. After 8 musical bars a change happens. This change is randomized, making it less predictable. In order to keep a clear musical structure these changes will not be too randomized, thus the 8 bar-structure is used. For the first play session an arbitrarily 5 times 8 bars is used for initial performance measuring. This is evaluated and rhythmic difficulty is adjusted accordingly. To not immediately judge the player on the changed difficulty the 3 bars after a change are treated with a lower weight, compared to the last 5 bars. These results are used to further evaluate after the songs progress.

The intermediate scores add up to the player score, in the background this value is is a combined measure of earlier performance scores. When this value becomes lower we adjust to a lower difficulty, if it rises we adjust to higher difficulty. The music generation algorithm has the following steps added in order to tailor the difficulty:

1. Let first 5 times 8 bars play to obtain initial player score determine player level 2. If rhythm change: select difficulty pool, player score progression; Randomly select

percussion clip from corresponding difficulty pool (see 6.2, and appendix C) 3. If tempo change: higher/lower bpm based on player score progression 4. Update player score difficulty (first 3 bars with lower weight)

5. Loop from step 2

6. For the next play-session repeat the process, but skip initial step and continue from step 2 using saved player performance score

As previously stated there are different approaches how to achieve a more individualized experience. Due to privacy and complexity reasons only ingame performance is used as input-parameter for the difficulty adjustment. Thus no physiological measurements are utilized.

Furthermore feedback is provided based upon the performance measure gathered from the distance metric compared throughout the game intervention. The feedback always shows positive information about their performance. It is focused on the process, thus not the result of a good/bad performance. For example positive like: “Great playing so far, keep going”, “Good job, your listening skill is great”, ”Wow, have you been practising?” or more negative: ”Next one might be yours”, ”Maybe next song fits you”, ”Keep going, you can do it”.

One method of applying this feedback is to have these as timed events. Since the comments should be truthful, a specific timing for these events is required. Meaning if a player does not perform great, they could still use the feedback as it might be motivating. For this reasoning the feedback is displayed after a specific amount of time, and displays different text based upon the performance score threshold. The game displays the messages once through every song and when transitioning towards the next song. At the end of the session another message is displayed with a different text such as: ”Good job see you tomorrow”.

(28)

6.3 Game experience

The gaming solution functions as an intervention in which the player acts as a musician. In this case the patient interacts with the music being played, similarly as within the MACT Clinical Protocol 4. The game acts as an accessible extension to the current music therapy sessions within the home environment. It should be fun and challenging enough to retain interest. This benefits the patient, because therapy sessions during the Covid-19 pandemic were sparse.

The gaming experience is measured via a questionnaire. This questionnaire has been developed by Leiden University and has been used for the previous study as well as this one, to make a fair comparison. The player performance will be captured via game score. Further details about the game experience is gathered using interviews, which touch upon topics that are roughly taken from the Game Experience Questionnaire (GEQ) and The Player Experience of Need Satisfaction (PENS).

6.4 Participants

Convenience sampling has been used to gather participants for this experiment. Under the condition that the participant is over the age of 18 and can play the game using an Android phone, no further constraints are added. Furthermore due to ethical reasoning no PD-patients participated.

6.5 Materials

Before receiving the files to play the game participants received an information sheet and addition consent form. The participants are presented the form of consent, see appendix 18, which also clearly states the possibility to always withdraw from the study if desired. Par- ticipants received the game as downloadable APK to install on their Android smartphone.

Due to Apple app installation policy we cannot provide the Ios version. Furthermore the participants are instructed to use headphones for the experiment for optimal sound quality.

The participants are asked complete the following parts:

• Ingame pre- and post-test

• Pre-game questionnaire about the gaming experience and music experience, as well as a post-game questionnaire on their experience

• Scale of confidentiality about score throughout the experiment

• Post-game interviews

The questionnaire includes general details (age, gender), but also musical background, attentional deficit background and cultural background. Moreover gaming experience is also measured. These are categorized into game-interest, boredom, enjoyment, motivation, challenge and liking of the music. All these aspects could influence the results within the gaming performance, but also the attitude towards the game. Only relevant information of the participants gets stored, which is connected to gameplaydata by entering the username with their assigned Unique ID. After playing the game and hitting submit the gameplaydata, consisting of tapping data, score and rating, is sent to a private server stored within JSON

Increasing engagement in a MACT game through feedback and adaptability

Master’s Thesis

1 Acknowledgement

Contents

Master’s thesis

December 23, 2022

2 Introduction

2.1 Problem statement

3 Background

4 Related work

4.1 Parkinson’s Disease

4.2 Music Therapy

4.3 Attention

4.4 Music in games

4.5 DDA

4.6 Feedback

4.7 Usability

5 Research questions

6 Methodology

6.1 Experiment design

6.2 Game design

6.3 Game experience

6.4 Participants

6.5 Materials