Emotional Arousal Detection

(1)

Emotional Arousal Detection

Master's Thesis

November 2013

Student: Ing. H.J. Zuidhof

Primary supervisor: Prof.Dr.Ir. M.Aiello Secondary supervisor: Dr. H. Riese

(2)

(3)

Emotional Arousal Detection

Master’s thesis Written for the University of Groningen

Faculty of mathematics and natural science and UMCG for the department of psychiatry

Supervised and coordinated by

Prof. Dr. Ir. M. Aiello (University of Groningen), Dr. H. Riese (University Medical Center Groningen)

By

Ing. H.J. Zuidhof

born the 11th of January 1988 Groningen, The Netherlands

(4)

During the day everyone experiences all kinds of emotions. Knowing a person’s current emotion is very useful in medical and psychology research. Detecting these emotions is not a trivial problem. Even the human brain has trouble detecting emotional states. Technology can help gaining more information about the emotional state of a human subject. As the emotional state of a human subject changes an increase in heart rate can occur. Previous studies show that a combination of heart rate and physical movement can be combined to form an index for additional heart rate caused by emotional arousal. In this study, an ambulatory real-time system is developed based on the additional heart rate index to detect emotional arousal of a human subject. The system is evaluated by exposing ten female participants to emotional stimuli. Subjective rating methods performed on the data acquired by these experiments show that the moments of additional heart rate are indeed caused by emotional arousal during laboratory experiments.

Ambulatory experiments show a less significant distinction between additional heart rate and emotional arousal.

Keywords: Additional heart rate, emotional arousal, emotions

iv

(5)

This report is the result of the master thesis project carried out by H.J.

Zuidhof at Department of Psychiatry University Medical Center of Groningen (UMCG) and is the final product of the master Comput- ing Science - Software Engineering & Distributed Systems at the Uni- versity of Groningen (RUG). I thank all people from the UMCG who helped me achieve this result. Especially I would like to thank Harri- ette Riese for the coordination and motivation. I want to thank Marco Aiello from the University of Groningen for the help and feedback during the project. Without their help I would not have succeeded in finishing this thesis.

v

(6)

(7)

1 i n t r o d u c t i o n 1

1.1 Emotional arousal detection . . . 1

1.2 Research hypothesis . . . 2

1.3 Contribution . . . 3

2 b a c k g r o u n d i n f o r m at i o n a n d r e l at e d w o r k 5 2.1 Background information . . . 5

2.2 Related work . . . 6

2.2.1 Heart rate analysis . . . 8

2.2.2 Feedback . . . 14

3 h a r d wa r e 21 3.1 Sensor . . . 21

3.2 Architecture . . . 23

3.3 Future hardware . . . 23

3.4 Discussion . . . 24

4 d e v e l o p m e n t 27 4.1 Algorithms . . . 27

4.1.1 QRS complex . . . 27

4.1.2 R top triggering . . . 28

4.1.3 Filtering . . . 28

4.1.4 Emotional arousal detection . . . 29

4.2 Software . . . 32

4.2.1 Class diagram . . . 33

4.2.2 EmozionActivity . . . 33

4.2.3 MeasurementService . . . 33

4.2.4 Parcels . . . 33

4.3 Questionnaires . . . 34

5 e va l uat i o n o f a r e a l-time ambulatory system for t h e d e t e c t i o n o f e m o t i o na l a r o u s a l 37 5.1 Validity of additional heart rate as an indicator of emotional arousal . . . 38

5.2 Laboratory and field study . . . 39

5.3 Aims of the present study . . . 39

5.4 Methods . . . 40

5.4.1 Participants . . . 40

5.5 Hardware . . . 41

5.5.1 Questionnaires . . . 42

5.6 Procedure . . . 43

5.6.1 Intake . . . 43

5.6.2 Experiment . . . 44

5.6.3 Finish . . . 44

5.7 Data assessment . . . 44

5.7.1 Subjective rating by testleader . . . 45

vii

(8)

5.7.2 Subjective rating by participants . . . 45 5.8 Statistical analysis . . . 46

5.8.1 During the laboratory experiments I expect true feedbacks at predefined emotionally intense moments . . . 47 5.8.2 During the laboratory experiments higher sub-

jective affective scores and heart rate will be found at the true feedback moments compared to random feedback moments . . . 47 5.8.3 In contrast to the laboratory experiments, dur-

ing ambulatory experiments the differences in heart rate and subjective affective score between true and random feedback moments are less/not distinguishable . . . 47 5.9 Results . . . 47 5.9.1 Descriptives . . . 47 6 d i s c u s s i o n a n d c o n c l u s i o n 55 6.1 Discussion . . . 55 6.2 Conclusion . . . 56 a e m o t i o na l a r o u s a l a l g o r i t h m (myrtek and foer-

s t e r, 2001) 59

b s e n s o r s 61

(9)

Figure 1 Heart system . . . 9

Figure 2 Heart rate variability . . . 12

Figure 3 ANI . . . 14

Figure 4 Example of an android implementation of a slidebar . . . 15

Figure 5 Simplified Geneva Emotion Wheel . . . 16

Figure 6 Affect grid . . . 17

Figure 7 Manikin . . . 18

Figure 8 Visualization of the system . . . 23

Figure 9 Flexible hardware attached to the skin . . . 24

Figure 10 System during use . . . 25

Figure 11 QRS Complex . . . 27

Figure 12 Bandpass filtering . . . 29

Figure 13 Disturbed ECG . . . 29

Figure 14 Killing 1 . . . 31

Figure 15 Killing 2 . . . 32

Figure 16 Class Diagram . . . 34

Figure 17 Mood scale . . . 35

Figure 18 Illustration of position of the BioHarness during use . . . 41

Figure 19 Affect grid . . . 43

Figure 20 Trainspotting intensity . . . 45

Figure 21 Questionnaire mood . . . 46

Figure 22 Trainspotting emotion intensity . . . 52

L I S T O F TA B L E S Table 1 Physiological symptoms and motor expressions 7 Table 2 Sensor Summary . . . 22

Table 3 Information regarding the participants of the experiment . . . 41

Table 4 Specifications of the HTC One S . . . 42

Table 5 Specification of the Huawei Ascend Y300 . . . 42

Table 6 participant data . . . 49

Table 7 Feedback data . . . 53

ix

(10)

(11)

1

I N T R O D U C T I O N

Computers and mobile devices have become everyday products which are carried around by a vast majority of people. Where in the early days computers were mainly focused on just brute force operations.

Nowadays computers have become part of our daily life. For example while traveling or just to keep up to date with everything that’s new. Besides their use in comfort they can help to learn more about ourselves, they can assist us in making decisions in certain situations.

For the computer to make decisions it needs information, this information can be provided by numerous amounts of sensors. Improve- ments in technology causes sensors to keep getting smaller and more advanced. This allows whole systems to become smaller, more comfortable and smarter by combining data provided by different sensors.

During this project research is done in the field of medical and techni- cal practices. By combining these fields the aim is to find a solution for medical and psychology researchers to help in the emotional arousal detection of a person. Emotional arousal can be detected in many different ways, as reviewed in the first part of this thesis. This chapter mainly focusses on introducing the problems that needs to be solved during this study and to give a global outline of this thesis.

1.1 e m o t i o na l a r o u s a l d e t e c t i o n

Detecting emotional arousal of a human subject is not a trivial problem. Emotions are influenced by many external stimuli like other per- sons or stress and happens everywhere: at home, at school/work or when traveling from one location to the other. Knowing these states at any moment of the day can help researchers with understanding human subjects or to assist doctors and psychiatrists to help their patients. Numerous studies [30, 31, 25, 38, 11, 15, 5, 8, 42] tried to describe, analyze and evaluate various aspects of affective states and some proposed technological solutions for detecting the emotional arousal of a human subject. Instruments used to measure emotions depend on the emotional theory involved and include, among others, self-reports of feelings, methods of behavioral rating, physiological parameters, and examination of facial and vocal expressions [38, 42].

This study focusses on the detection of emotional arousal by analyzing heart rate and physical movement and self-reports of feelings.

The main goal of this project is to develop a system with the ability to detect the amount of emotional arousal of a person.

1

(12)

e m o t i o na l a r o u s a l Every human subject experiences emotions, where one emotion can have more impact on your feelings then the other, this impact is the emotional arousal. Before detecting the emotional arousal of a person, a definition of emotions is made to help understand what this project is about. This is not as easy as it may seem. The dictionary states a large number of definitions for emotion, for example:

1. the affective aspect of consciousness 2. a state of feeling

3. a conscious mental reaction (as anger or fear) subjectively experienced as strong feeling usually directed toward a specific object and typically accompanied by physiological and behavioral changes in the body.

These definitions still leave things open for interpretation and is not strict enough for this study. Previous studies have tried to come up with answers for this problem. As Scherer et al (2005) put it, the question ”What is an emotion?” rarely returns the same answer from different individuals, scientists or non-professional [39]. Numerous scientists proposed definitions but they do not seem to generate one final answer. Kleinginna et al (1981) listed 92 of the proposed scientific definitions and merged them to one final definition which is used in this paper: [18]

Emotion is a complex set of interactions among subjective and objective factors, mediated by neural/hormonal systems, which can

1. give rise to affective experiences such as feelings of arousal, pleasure/displeasure;

2. generate cognitive processes such as emotionally relevant per- ceptual effects, appraisals, labeling processes;

3. activate widespread physiological adjustments to the arousing conditions; and

4. lead to behavior that is often, but not always, expressive, goal- directed, and adaptive

These changes in physiological properties, stated in point 3, can be used to measure emotional arousal. Examples of measurable properties are breathing rate, speech recognition or heart rate. In chapter2 is elaborated on the properties and the techniques to analyze them.

1.2 r e s e a r c h h y p o t h e s i s

Although numerous systems are already available to detect emotional arousal [31,38,15,5,42], most systems require carrying around multiple pieces of hardware or hardware interrupting with normal daily

(13)

activities. For this study I hypothesize that, based on previous studies, an ambulatory real-time analyzing tool can be developed that is able to detect the increase of emotional arousal of a human subject by analyzing changes in heart rate. The goal is to achieve the same outcome as previous studies but with more comfortable and ambulatory hardware. To test this hypothesis a system is develop and an experiment is set up where human subjects are exposed to a video to induce emotional arousal, followed by ambulatory monitoring in everyday life. At moments of emotional arousal detection feedback from the human subject is needed for evaluation and future purposes. In this document the current state of the art and the steps and methods used to develop a system are discussed.

1.3 c o n t r i b u t i o n

With this system researchers can monitor and evaluate patients by retrieving the emotional arousal throughout the day. With the technology of today a system can be developed which is much more comfortable to use for longer periods of time. Data from patients can be much easier extracted from the hardware by simply connecting a smartphone to a computer.

o u t l i n e

c h a p t e r 2: In this chapter the current state of the art of physiological emotional arousal detection is reviewed and discussed.

Research in both medical and technological studies and their different techniques and methods are evaluated.

c h a p t e r 3: Background information of the heart itself is provided in this chapter. After this brief introduction to the heart the main requirements for the system are discussed.

c h a p t e r 4: In this chapter the architecture of the system developed during this study is discussed. All parts of the system are described and put together into one overview.

c h a p t e r 5: During the implementation and testing of the system different challenges were encountered. In this chapter these challenges are discussed and elaborates on the solutions and choices made are given.

c h a p t e r 6: Validation of the system is done in three experiments.

The set up and evaluation of these experiments is described in this chapter. For the final experiment ten women participated by watching a movie and by wearing the system in daily life situations.

(14)

c h a p t e r 7: A look back is taken on the work conducted during this research project and draw conclusions of the evaluation and findings of the whole project. Whether or not the system and/

or method is feasible for detecting emotional arousal. Finally in this chapter suggestions for future research directions and further implementation of the system are discussed.

(15)

2

B A C K G R O U N D I N F O R M AT I O N A N D R E L AT E D W O R K

Before diving into the development of the system an overview of the current literature and methods for the detection of emotional arousal is provided. Besides heart rate there are other bodily symptoms that can be measured for the detection of emotional arousal. This chapter discusses the heart rate to be the best physiological symptom to detect the changes in emotional states in ambulatory situations. Starting with discussing background information about the bodily symptoms which may change during emotional arousal. Previous work is discussed in emotional arousal detection by physiological properties after which will be elaborated on the work performed in heart rate analysis. Besides the physiological and technological methods and techniques research is done in subjective emotional scoring. On the event of detection of emotional arousal, feedback needs to be acquired from the human subject. Section 2.2.2elaborates on these methods.

2.1 b a c k g r o u n d i n f o r m at i o n

Scherer et al (2005) proposed the distinguishment between emotions and other affective phenomena such as feelings, moods or attitudes.

In his study he discussed ways of measuring emotions. Because emotions have such a high impact on physiological properties the per- fect solution for detecting emotional arousal is to analyze all of these properties. In Scherer’s words; ’in an ideal world of science, we would need to measure (1) the continuous changes in appraisal processes at all levels of central nervous system processing, (2) the response patterns generated in the neuroendocrine, autonomic, and somatic nervous systems, (3) the motivational changes produced by the appraisal results, in particular action tendencies, (4) the patterns of facial and vocal expression as well as body movements, and (5) the nature of the subjectively experienced feeling state that reflects all of these component changes [39]. Such measurements have never been performed together and is unlikely to become standard procedure in the near future. However, there have been major advances in recent years with respect to measuring individual components. In this chapter the different individual components which can be measured during emotional arousal are investigated. K.R.Scherer set up the list provided in table1 which includes the physiological symptoms and motor expressions which can change or occur during an emotion. Using this information a system can be developed which analyzes one or more

5

(16)

of these properties to make a suggestion of the emotional arousal of a human object. Chapter2.2.1discusses previous studies where such a system is build and elaborates on the choses made for the system build during my research project.

2.2 r e l at e d w o r k

The research for this work is focused on the crossroad of medical and technological studies. Previous studies show that emotional arousal detection by analyzing physiological properties or motor expressions is feasible. E. Leon et al (year) [11] discussed and developed a speech recognition implementation to detect the emotional arousal of a human subject by classifying different features of the voice like pitch contour, intonation and speech style. Other motor expressive analyzing is performed by Sun et al [42]. By detecting facial expressions of a human subject an estimation of the emotional state of the person is possible. Studies about the correlation of physiological properties and emotional arousal are just as popular, if not more popular.

Brain wave analyzes is one example. Technology has come very far in using the electroencephalogram (EEG) to assist people. Analysis of brain waves are mainly used for humans who are physically disabled by, such as paralysis, brain stroke and other kind of brain disor- ders to communicate with the real world application like controlling of house hold equipment, playing games, making or receiving calls from phone, and using the computer [5]. A.S. AlMejrad et al (2010) discussed the use of EEG signals for use in emotion recognition. An- other possibility of emotional arousal detection is with the use of bio gases emanated from the human body: e.g., breath, oral cavity, skin gas and body odor. Breath gas is easy to collect in bio gases and it includes more than 200 kinds of biochemical feature. Breath gas information can be considered to have a relationship with mental states such as emotions and stress, since feeling emotions affect the autonomic nervous system and the autonomic nervous system controls physiological conditions [21]. Besides facial expression, brain wave analyzes and breathing (gas) analyzes the heart has shown to be a good indicator of emotional arousal. Previous studies show systems detecting the changes in heart rate of a human subject and couple them to emotional arousal [30,31,25,40,15]. The studies have shown to successively detect emotional arousal. For this study the chose is made to continue with this method, but with new hardware and techniques to make a system which is comfortable and easy to use in daily life.

c a r d i ova s c u l a r s y s t e m Before diving into previous studies we take a step back to elaborate on the heart itself, what it is and why it is good for the detection of emotional arousal. The primary func-

(17)

p h y s i o l o g i c a l s y m p t o m s m o t o r e x p r e s s i o n s Feeling cold shivers (neck,chest) Smiling

Weak limbs Mouth opening

Getting pale Mouth closing

Lump in throat Mouth tensing

Stomach troubles Frown

Heart beat slowing down Eyes closing Heart beat getting faster Eyes opening Muscles relaxing, restful (whole body) Tears

Muscles tensing, trembling (whole body) Other changes in face Breathing slowing down Voice volume increasing Breathing getting faster Voice volume decreasing Feeling warm, pleasant (whole body) Voice trembling

Perspiring, moist hands Voice being assertive Sweating (whole body) Other changes in voice Feeling hot, puff of heat (cheeks, chest) Abrupt bodily movements

Blushing Moving towards people or things

Sweating Withdrawing from people or things

Moving against people or things Other changes in gesture

Silence

Short utterance Long utterance

Speech melody change Speech disturbance Speech tempo changes Table 1: Physiological symptoms and motor expressions [39].

(18)

tion of the heart, a muscular pump, is to supply the energy required for the circulation of blood in the cardiovascular system. The cardiovascular system (figure1 is comprised of the heart and vasculature.

The vasculature serves as the distribution, exchange and collection channels by which the blood, pumped by the heart, travels to every living tissues in the body and is subsequently returned to the heart [17].

The heart is comprised of four chambers that function as two pumps in series. The right-side pump (right atrium and right ventricle) provides the energy necessary to move blood through the pulmonary circulation. The left-side pump (left atrium and left ventricle) provides the flow through the systemic circulation. As a result of a common pacemaker and the nature of cardiac muscle fibers [17], both atria contract rhythmically and in synchrony, thereby causing these chambers to periodically empty their contents and to fill the ventricular chambers. A fraction of a second later the blood-filled ventricles also contract in synchrony into the vasculature. This process is one heartbeat. This beating of the heart is an electromechanical event. Electrical impulses generated by specialized cells within the heart, referred to as pacemaker cells, initiate the mechanical contraction of heart muscle [24, 17]. Using these techniques an electrocardiogram (ECG) can be extracted tracking the muscle contractions of the human heart.

2.2.1 Heart rate analysis

Emotional arousal detection by heart rate analyzes can be executed with multiple methods. The most popular methods are additional heart rate (AHR) and heart rate variability (HRV) analysis. Additional heart rate is a method of detecting increases in heart rate that are not correlated to changes in physical activity. This method was proposed by Myrtek in 1996, in this chapter this method is introduced and discussed.

2.2.1.1 Myrtek

In 1996 Myrtek et al [30] proposed an AHR algorithm to detect changes in emotional arousal. In 2005 Myrtek published a new study with an improved algorithm and new research material [31]. The complete algorithm is available in appendix A. This study and algorithm is the bases of numerous subsequent studies performed by other scientists, discussed later on in this chapter. Myrtek developed a system to analyze human subject’s heart rate changes and their physical activity in ambulatory settings [31]. Emotional arousal is suspected by linking increases in heart rate to the physical activity of a human subject.

When an increases in heart rate is detected without an increase in physical movement a feedback signal is given to the user to fill out an (external) questionnaire.

(19)

Figure 1: Human ciculatory system. Top figure presents detailed schematic of circulation while bottom figure highlights functional divisions of circulatory system [17].

a l g o r i t h m The AHR signal is a continuous estimation of arousal from heart rate and physical activity. The default process of additional heart rate computation is explained below. (See appendixAfor details of the algorithm) For the current emotional arousal state the

(20)

current time window is compared to the mean of the previous 3 time windows. The length of the time window is 60 seconds on default, more detailed information about the time window and the length is found in chapter4.

HRC[i] = HR[i] − ((HR[i − 3] + HR[i − 2] + HR[i − 1])/3) (1) HRC is the current change in heart rate compared to the previous 3 time windows (1). When this value reaches HRPLUS (threshold, 3 heart beats on default), an AHR signal is generated. HRPLUS is in- creased by adding the mean acceleration of the current time window.

For additional heart rate a score is formed defined as the factor, by which the current heart rate change (HRC) exceeds the minimal heart rate change (HRPLUS)

AHR = HRC/HRPLUS (2)

Requirements for a feedback moment (additional heart rate detected) are as follows:

1. HRC >= HRPLUS 2. AHR >= 1

3. Acceleration must not exceed the mean of the previous 3 time windows of acceleration

4. The current heart rate must be higher than its predecessor r a n d o m f e e d b a c k For reasons of comparison, but also to avoid possible conditioning, random feedbacks are given with no indication of an emotional event after three event-related feedbacks. In chapter5 the random and true feedbacks are compared to evaluate the software and help prove the hypothesis discussed in chapter 1. Random feedbacks are given randomly after 3 true feedbacks have been generated.

For every generated true feedback the change of a random feedback increases. The change of random feedback is reset after a random feedback is generated. If in the minute for random feedback the criteria for a true feedback are fulfilled, then the next following minute is used for random feedback. If the ratio between event-related and random feedbacks is smaller than 5:2, then no further random feedbacks are given with the exception that within the interval of 20 minutes no event-related feedback is possible. In this case a random feedback is given in minute 20 [31].

(21)

m y r t e k c o n t i n u e d Kusserow [27] is one of the researchers con- tinuing Myrtek’s study to do stress arousal detection. In this study multiple scenarios are analyzed; monitoring a public speaker, monitoring a musician on stage, monitoring an Olympic champion and monitoring in free-living daily activities. Two methods were used to measure the emotional states of the participants, AHR and HRV.

The focus in this chapter is on the AHR method, in the next chapter the HRV method is discussed. The experiments during free-living daily activities were performed with the AHR method. In his study Kusserow improved the algorithm proposed by Myrtek. Myrtek’s algorithm does not address the physiological changes caused by body motion transition, e.g., transitions from sitting to standing. Kusserow addressed this limitation by using multiple accelerometers to indicate the changes in body transitions. A four participant’s data set of 182 h during natural daily activities was used to test the stress arousal detection and estimation procedure. The wearable system according to the AHR algorithm sensor setup comprised two 3-axis wireless accelerometers (BodyANT) placed at the chest and right thigh, and a wireless heart rate monitor chest belt. Data samples were stored by the Q-belt integrated computer. To facilitate interpretation of esti- mated stress arousal phases, participants kept an activity diary and completed mood state questionnaires throughout the recordings.

2.2.1.2 Heart rate variability

Besides analyzing increases in heart rate as used with the AHR method, the changes between every heartbeat can be used to make an indication of the state of a human subject. The studies discussed in this chapter use this concept for emotional state and stress detection. The central nervous system (CNS) is involved in the pathways linking emotions to ANS reactivity. The activation of the autonomic nervous system (ANS) can be evaluated by HRV and can therefore help in measuring the emotional arousal of a human subject. The changes between heartbeats are computed by taking the R-R intervals. An R-R interval is the time between two subsequent heartbeats, figure2illus- trates an ECG where the heartbeats are marked. HRV is computed by summing the difference in time between two subsequent R-R intervals.

An example HRV index is the RMSSD; Root Mean Square of the Successive Differences. Higher levels of HRV indicate appropriate fluctuations in ANS activation. The higher the HRV, the more capable the heart is of adapting to stressors, such as increases in physical or mental activity, and reductions in activity. Lower HRV may indicate a decreased ability for the heart to compensate or react appropriately during stressful or otherwise demanding situations [23]. Appelhans et al (2006) reviewed the use of HRV as an index of emotional re- sponding. Their research and other theory support the use of HRV

(22)

Figure 2: Heart rate variability. R-tops (heart beats) are marked in red, the time between two R-tops is represented by the numbers in between (in milliseconds).

as an objective index of emotional responses [8]. Examples of other studies and their methods and findings are discussed in this chapter.

k u s s e r o w In the same study Kusserow discussed the AHR method he worked on stress arousal detection by analyzing HRV. Experiments are performed with a public speaker, an activity with little physical movement. Kusserow investigated seven features of heart activity from time- and frequency-domain that were reported to indicate body stress in laboratory investigations. The investigations showed that time-domain features, in particular heart period, can provide ro- bust information for the talk situation.

a na l g e s i a n o c i c e p t i o n i n d e x De Jonckheere et al (2012) hy- pothesized that the Analgesia nociception Index (ANI), an index based on HRV analysis, can be used as a tool to investigate the processes of emotional regulation of a human subject. The ECG signal is acquired using classical electrodes. The algorithm used for this study is explained below. They concluded that the ANI is indeed capable of measuring changes in emotional state [15]. Tests were performed by inducing a negative emotional stimulus through the projection of an 80 seconds video sample of the movie ’American History X’. This sample has been characterized by experts as an outlier in negative emotions [6].

ANI is computed from the ECG signal as described below (summa- rized). The ECG signal is acquired using classical electrodes and dig- itized at a sampling rate of 250 Hz. The RR series is then built as the time evolution of the time intervals between two successive R waves of the ECG (RR value). R tops are detected using a R wave detection algorithm [35]. However, RR series is usually disturbed by different kinds of interferences (ectopic beats, electrode motion...) involving an erroneous analysis of HRV. Therefore, RR intervals series is filtered in real time using an original non linear filtering algorithm [36]. This filtering algorithm, which is based on a morphological analysis of the

(23)

RR series, is able to detect each disturbed area and to replace the erroneous samples with the most probable ones.

Filtered RR series are then re-sampled at 8 Hz using a linear inter- polation algorithm. RR_isamples are isolated into a 64 seconds moving window for analysis and the series is mean centered and normalized for inter patients comparability.

In a first step, the mean value (M) is computed as:

M = 1 N

XN i=1

(RR_i); (3)

Where RR_i represents the RR samples values and N the number of samples in the window. Then M is subtracted from each sample of the window as: RR_i = (RR_i - M). The norm values (S) is then computed as:

S = v u u t

XN i=1

(RR_i)²; (4)

and each RR_i is divided by S: RR’_i = RR_i / S.

Since the method is based on the analysis of HF changes, the RR’_i series is then band pass filtered between [0.15-0.5 Hz], which leads to RR’_HF[37].

HF changes are mainly modulated by respiration. Indeed, breathing influences the way the ANS regulates heart rate. Inhalation tem- porarily inhibits the influence of the parasympathetic nervous system and increases heart rate, while exhalation stimulates the parasympathetic nervous system and decreases heart rate. These rhythmic oscil- lations, which are caused by breathing, are called respiratory sinus arrhythmia (RSA).

We observed that, in case of well-being, RSA modulation on HR is important and causes large magnitude on the RR’HF series (figure 3, upper panel), while in case of pain, stress or anxiety, the influence of each respiratory cycle is more chaotic and the series magnitude decreases (figure3, lower panel).

In order to transform this qualitative observation into quantitative information, we designed a graphical index by computing the area under the RR’HF series curve as shown in figure3. Local minima and maxima are detected and the upper and lower envelopes are plotted by connecting the local maxima together and the local minima as well. This envelope estimation allows to obtain an index which not depends on respiratory frequency changes [26].

In order to improve the time sensitivity of the method, we then divide the 64 second moving window into four subwindows of 16 second. The areas between the lower and upper envelopes are then

(24)

measured in the four sub-windows. We defined AUCmin as the small- est of these sub-areas.

ANI is then computed in order to express a fraction of the total window surface, leading to a measure between 0 and 100:

ANI = 100∗ [a ∗ AUCmin + b]/12.8; (5)

Where a = 5.1 and b= 1.2 have been empirically determined in a general anesthesia data set of more than 100 patients in order to keep the coherence between the visual effect of respiratory influence on RR series and the quantitative measurement of ANI.

Figure 3: normalised, mean centered and band pass filtered RR series in two different states of analgesia / nocicetion during general anaesthe- sia with controlled ventilation: adequate analgesia (upper window) and in the case of surgical stimulus, few minutes before haemody- namic reactivity (lower window).

2.2.1.3 Discussion

Studies show that both methods, HRV and AHR, can be used for emotional arousal detection. Previous studies showed succesive results with both methods. The AHR method takes physical activity into account. Therefor this method is chosen for the system developed during this research study.

2.2.2 Feedback

’Emotions are what people say they are’ [34]. Although different studies doubt this phrase, for our study it is used to get subjective feedback from the human subjects. At the moment of emotional arousal detection, feedback is needed for subjective evaluation of the system.

Numerous methods of feedback extraction are proposed in previous studies [22,41,16]. In this chapter these methods are introduced and discussed.

(25)

2.2.2.1 Scale

One of the simplest solutions is to ask for an emotion and let the user choose a value on a certain scale. Figure 9shows an example of the implementation on an android phone. Although simple, this does not provide flexible measures. More flexible methods are discussed in the sections below.

Figure 4: Example of an android implementation of a slidebar. This particular example shows a scale from positive to negative where the middle is a neutral state.

2.2.2.2 Geneva emotion wheel

The Geneva Emotion Wheel (GEW) is an example of a self-report measure. The design of the GEW has elements of a free response format, a discrete emotion response format, and a dimensional approach to emotions. The GEW consists of discrete emotion terms corresponding to emotion families that are systematically aligned in a circle. Under- lying the alignment of the emotion terms are the two dimensions valence (negative to positive) and control (low to high), separating the emotions in four quadrants: Negative/low control, negative/high control, positive/low control, and positive/high control. An example is visualized in figure5.

(26)

Figure 5: Geneva Emotion Wheel [41], divided in two axes: control and valence. Users select one place of the wheel of which they think reflects their current emotional state the best.

2.2.2.3 Affect grid

The affect grid (Figure 6) is designed as a quick means of assess- ing affect along the dimensions of pleasure-displeasure and arousal- sleepiness. Previous work showed that the scale is suitable for any study that requires judgments about affect of either a descriptive or a subjective kind [16].

Before a participant fills out this grid the testleader explains what the different boxes mean by explaining the means of the X and Y axes. The description below is an example of how this affect grid can be explained.

(27)

Figure 6: Affect grid [16]

’Let’s practice the grid by describing how you were feeling this morning when you woke up. There are basically two axes to think about. For the time being, you can ignore the words in the corners of the grid. We’ll talk about these two main directions first. The horizontal direction ranges from unpleasant feelings on the left to pleasant feelings on the right. So the more unpleasant or pleasant you were feeling this morning, the more to the left or right you’d mark your answer. If you felt more neutral, you would place a mark somewhere near the center of the grid. Once you’ve decided how far to the left or right your mark needs to be, you need to decide how high or low to put it. The vertical direction of the grid ranges from high arousal at the top of the grid to sleepiness at the bottom. So now place your mark higher or lower on the grid depending on how aroused or alert / energetic, versus sleepy you were. Please note that people’s emotions can be described by any combination of these four basic states. Now look at the words in the corners of the grid. These adjectives describe feelings that fall into that section of the grid. For example, you can see that feeling very pleasant and highly aroused may be called excitement. This would probably occur for example if you’d win a large sum of money in the lottery. However, there are other examples of pleasant experiences that don’t necessarily involve high levels of arousal.

Pleasant feelings may also be associated with low arousal or sleepiness. Think for example of taking a long bath or receiving a massage. In the bottom right- hand corner of the grid, you can see that pleasant sleepy feelings may be described as a state of relaxation. Let’s look at the left side of the grid now. The top left-hand corner indicates that unpleasant emotions coupled with high arousal may be described as stress. The bottom left-hand corner describes a state of unpleasant feelings and low arousal. This may be described as depression. Finally, if you feel very neutral you’d describe this with a mark placed in the center of the grid. So for this grid, we ask you to place an X

(28)

in just one of the boxes to describe how you were feeling. Try thinking first about how pleasant versus unpleasant you felt, and select the column which best describes this. Then think about how aroused versus sleepy you felt, and move either up or down the column accordingly. Finally, take a look at the descriptive word nearest the square you have chosen, and see if your feelings were closer to that word than to any of the other descriptive words. Give it a try, and let me know if you have any questions. Let me know when you’re finished. I’d like to see your response and then I’ll try to describe the emotion based on your answer, just to check if I’ve explained the grid well enough.’

2.2.2.4 Manikin

The Self-Assessment Manikin (SAM) is a non-verbal pictorial assessment technique that directly measures the pleasure, arousal, and dominance associated with a person’s affective reaction to a wide vari- ety of stimuli. SAM ranges from a smiling, happy figure to a frown- ing, unhappy figure when representing the pleasure dimension, and ranges from an excited, wide-eyed figure to a relaxed, sleepy figure for the arousal dimension. The dominance dimension represents changes in control with changes in the size of SAM: a large figure indicates maximum control in the situation. In this version of SAM, the subject can place an ’x’ over any of the five figures in each scale, or between any two figures, which results in a 9-point rating scale for each dimension [22].

Figure 7: The Self-Assessment Manikin (SAM) used to rate the affective dimensions of valence (top panel), arousal (middle panel), and dominance (bottom panel) [22]

(29)

2.2.2.5 Discussion

Means of retrieving affective states can be performed in several ways, this chapter discussed a few of them. Depending on the experiments/hypotheses a chose needs to be made for the one fitting the required results. In later chapters the choices for different questionnaires are explained.

(30)

(31)

3

H A R D WA R E

A device for the extraction of the heart rate is needed which does not only send average heart rates every second, but a device that sends the complete ECG of a person. Having access to the ECG afterwards allows researchers to analyze the data more closely when needed. An- other reason to analyze the ECG instead of using the average heart rate provided by the device is to have more control over the situation. Future implementations may use different sensors which may provide different heart rate data. An accelerometer is used to measure the physical movement. The processing of both the ECG and the acceleration data is done on an external processing unit. Because the system will be used in ambulatory situations/in everyday life, the processing is done on a smartphone, this allows the users to use their own phone instead of carrying extra hardware with them. The software for this study is developed for Android phones, future implementations can be developed for iPhones or Windows phones.

3.1 s e n s o r

In this section the various heartbeat sensors found are discussed, each with their pros and cons. Note that this section contains sensors specifically designed to work on mobile (Android) phones via Bluetooth, as well as other, less specific sensors. For this research project a system must be developed to give a comfortable feeling to the people wearing it. Therefor sensors are required which do not in- terfere with daily life activities of a human subject. Although none of the suggested sensors are eliminated because of the need of comfort, some provide more comfort then the other. Another requirement is the availability of wireless communication to other devices for real- time analyzing of the gathered data. Table 2lists possible candidate sensors and their details.

21

(32)

devicecomfort(1-10)wirelessandroidsdkecgdatanumberofelectrodesaccelerometer PolarWearlink6XX2 ZephyrHxMBT6X2X ChronosEZ430/EZ430-RF25605XX SmartHRM6X ZephyrBioHarnessBT6XXX2X VUAMS4X5 Smartshirt7 Table2:Listanddetailsofcandidatesensors

(33)

3.2 a r c h i t e c t u r e

Architecture can be defined as the framework supporting each and every component used in, but also used by the appliance. The architecture is therefore a vital part of every software and hardware project.

One of the main aspects of this research is ambulatory monitoring in real-life situations. This means that the research is not performed on stationary, but on people freely moving around. To enhance this in the research our idea is to fit the test subjects with their own ’computer’

to perform measurements. This computer found its implementation as an Android smartphone. This device connects to the monitoring sensor and processes the data. For this project all data is stored on the Android device and the heart rate monitor itself. Figure 8 visualizes the architecture of the system. More detailed information is available in appendixB

Figure 8: Visualization of the system.

3.3 f u t u r e h a r d wa r e

As mentioned in the introduction of this document sensors are be- coming better and smaller every year. This means that future implementations of the system proposed during this study can be easily improved. During the current study, after the decision for the BioHar- ness was made, several other devices came to our attention. Some of

(34)

these devices fit the profile for acceptable sensors that can be used for this project. Some of these devices have advantages compared to the BioHarness. Future studies can take these devices into account for their own research projects.

v ua m s At the moment of writing this document the sensor does not include all the features needed to be integrated into our system, namely a Bluetooth connection. Contact with the Vrije Universiteit van Amsterdam taught us that they are currently working on inte- grating a Bluetooth connection. The VUAMS uses a larger number of electrodes to extract the ECG and is therefore much more reliable during physical activity compared to the BioHarness which only uses two electrodes.

e q u i v i ta l e q 0 2 This sensor seems to have the same functional- ities as the BioHarness, except that the hardware itself can be worn more comfortable without much sliding and is therefore maybe a good alternative to the BioHarness used during this study.

tat t o o s e n s o r s In today’s research flexible sensors are developed that can be attached to the skin, like a tattoo. This gives them the advantage of not being able to slide, compared to the BioHarness.

Although not yet available for today’s research this can be a good alternative for future implementations [4].

Figure 9: Flexible hardware attached to the skin [4].

3.4 d i s c u s s i o n

None of the sensors stand out from the others. Based on the provided information we have chosen to use the Zephyr BioHarness BT.

Its comfort is sufficient, it provides wireless communication via Blue- tooth and an SDK is available. Besides this the raw ECG data is available. The final system will exists of two devices, the BioHarness and a smartphone, figure10illustrates what this will look like in real life.

(35)

Data from the BioHarness is retrieved by a smartphone. During the tests and experiments during this study the android devices used were the android emulator provided by the SDK the HTC One S and the Huawei Ascend Y300 phone.

Figure 10: The system during use. On the left the BioHarness and on the right an Android phone receiving ECG data via Bluetooth.

b at t e r y l i f e Battery life is a problem for most smartphones nowadays. Most smartphones will last a day or maybe two. Running an extra application does not contribute to this problem. The battery life during one hour of running the application is compared to an hour of idle state. During these tests both Wi-Fi and mobile network are disabled and Bluetooth is enabled. During a period of 5.5 hours the battery consumption of the smartphone is evaluated. Tests show that during this period the battery power of the Huawei Ascend Y300 is lowered to 86% when the app is not running. When the app is running during the 5.5 hours the battery power is lowered to 80%.

(36)

(37)

4

D E V E L O P M E N T

During the implementation of the architecture described in chapter3 challenges are encountered. This chapter describes these challenges and their solutions.

4.1 a l g o r i t h m s

The software on the smartphone uses several algorithms, for example the detection of heart beats. In this chapter the most important algorithms are discussed.

4.1.1 QRS complex

One of the biggest challenge is to retrieve the R tops from the ECG data. The R tops are used to calculate the heartbeat per minute (bpm).

The QRS complex is a name for the combination of three moments during one heartbeat on an electrocardiogram. Figure 11 shows an example of how such a QRS complex looks like.

Figure 11: QRS complex visualizing the activity during one heart beat [3] The image is a representation of the muscle contractions of the heart during one heartbeat. The one we are most interested in is the R top. This is the peak with the highest amplitude and therefore the

27

(38)

best signal to detect. Once we have two subsequent R tops the bpm can be calculated by formula 6.

bpm = 60000/R − Rinterval (6)

Because it is not reliable to only use the last known R-R interval the bpm is calculated by taking the mean of the R-R intervals during the last known 10 seconds of the ECG data, which is the length of standard electrocardiograms [40].

4.1.2 R top triggering

The RR series can be detected in multiple ways. The software used at the psychiatry department of the UMCG (PreCar) uses a threshold algorithm. Every peak above a certain threshold is signed as a R top. This software is used for analyzing data after all data is received.

The user can change the threshold value to check for better results.

Because our system analyses the data in real-time this is not a possible solution. As discussed in a previous study of Blaauw & Zuidhof (2012) an algorithm is developed. Here we compare this algorithm with BandPass filtering.

b a n d pa s s f i lt e r i n g BandPass filtering uses the frequencies in the data signal to filter the signal to only let certain frequencies pass.

By using a combination of low- and high pass filtering the R tops can be triggered. Figure12shows an ECG with the R tops triggered using this filter.

When the signal gets disturbed the band pass algorithm has trouble finding R tops. This is the main disadvantage of this method and will be taken into account when deciding the R top triggering method.

p e a k d e t e c t i o n Because the ECG data exists only of peaks (positive/negative), a peak detection algorithm is developed. The algorithm detects peaks by building a peak while the continuous data values are going constantly in the same direction (up or down). When the next point is going in the other direction a new peak is started and the current peak has reached its top. When the peak has a max width of 20 (default) and a minimum height of 20 (default), it is signed as an R top. The two values max width and minimum height can be changed in the settings menu. Figure 13 visualizes the results retrieved with this method.

4.1.3 Filtering

The data retrieved from the BioHarness are not always reliable. Due to physical movement the heart rate monitor band can slide which

(39)

Figure 12: Bandpass filtering example of four heart beats. A decision is made according to the frequencies passing the filters. The orange line illustrates the decision threshold which is used to mark the signal as an R-top.

Figure 13: Disturbed ECG with R tops marked in blue.

causes a very disturbed signal. Other reasons of signal disturbance can be caused by other physical movements, the pulses generated when contracting other muscles can be intercepted by the sensor.

Although the algorithm does still detect R tops when there is such disturbance it is not always safe to use them. Therefore the data is filtered. One of the options is to ignore the R tops and, while there is disturbance in the time window, the last known heart rate is taken.

Because this solution ignores all of the data, including correct data, another solution is chosen. When disturbance is detected by the ECG analyzer, the disturbed parts are ignored and heart rate is calculated over the correct data.

4.1.4 Emotional arousal detection

For emotional arousal detection the algorithm from Myrtek et al (2005) is used. In chapter 2this algorithm is discussed. This algorithm is re- peated below.

(40)

a l g o r i t h m The AHR signal is a continuous estimation of arousal from heart rate and physical activity. The default process of additional heart rate computation is explained below. (See appendixAfor details of the algorithm) For the current emotional arousal state the current time window is compared to the mean of the previous 3 time windows. The length of the time window is 60 seconds on default, more detailed information about the time window and the length is found in chapter4.

HRC[i] = HR[i] − ((HR[i − 3] + HR[i − 2] + HR[i − 1])/3) (7) HRC is the current change in heart rate compared to the previous 3 time windows (7). When this value reaches HRPLUS (threshold, 3 heart beats on default), an AHR signal is generated. HRPLUS is in- creased by adding the mean acceleration of the current time window.

For additional heart rate a score is formed defined as the factor, by which the current heart rate change (HRC) exceeds the minimal heart rate change (HRPLUS)

AHR = HRC/HRPLUS (8)

Requirements for a feedback moment (additional heart rate detected) are as follows:

1. HRC >= HRPLUS 2. AHR >= 1

3. Acceleration must not exceed the mean of the previous 3 time windows of acceleration

4. The current heart rate must be higher than its predecessor r a n d o m f e e d b a c k For reasons of comparison, but also to avoid possible conditioning, random feedbacks are given with no indication of an emotional event after three event-related feedbacks. In chapter5 the random and true feedbacks are compared to evaluate the software and help prove the hypothesis discussed in chapter 1. Random feedbacks are given randomly after 3 true feedbacks have been generated.

For every generated true feedback the change of a random feedback increases. The change of random feedback is reset after a random feedback is generated. If in the minute for random feedback the criteria for a true feedback are fulfilled, then the next following minute is used for random feedback. If the ratio between event-related and random feedbacks is smaller than 5:2, then no further random feedbacks are given with the exception that within the interval of 20 minutes no event-related feedback is possible. In this case a random feedback is given in minute 20 [31].

(41)

t i m e w i n d o w The algorithm uses time windows to compare heart rate means with each other. In his papers Myrtek uses time windows of 60 seconds without any clear reasoning why this length has been chosen. This section elaborates on the best length to use for the time windows by storing ECG data retrieved from the system during different episodes from the TV show ’The killing’ watched by the testleader.

Afterwards the algorithm is run over the data with each time different time windows. A time window is needed which detects increases in heart rate that are not caused by reflexes or physical activity. To test this two different time windows are used, 30 and 60 seconds. Data is logged from the BioHarness while the testleader watches episodes from two episodes of the TV show ’The Killing’. During the episodes the testleader is sitting still in a stationary position.

Figures14and15visualize the heart rate of the testleader and the moments of feedback.

Figure 14: Heart rate during an episode of the Killing. The red dots show the feedback moments when a time window of 60 seconds is used.

The green dots show the feedbacks when a time window of 30 seconds is used.

The figures show no significant differences in moments of feedback. The 30 second time windows show a few extra feedbacks at moments where the heart rate increases for just a small time. These short peaks in heart rate is not what needs to be detected with this system, therefor the time window of 60 seconds used in Myrtek’s experiments stays. The time window can be changed depending on what kind of behavior has to be detected. As the results show a low time window (10-30 seconds) can be used to detect short, high increases in heart rate. These increases can be caused by reflexes and emotions. As emotions imply massive response mobilization and syn- chronization as part of specific action tendencies, their duration must be relatively short in order not to tax the resources of the organism and to allow behavioral flexibility [39]. Therefore a time window of

(42)

Figure 15: Visualization of episode X from the Killing. The red dots show the feedback moments when a time window of 60 seconds is used.

The green dots show the feedbacks when a time window of 30 seconds is used.

30-90 seconds is best when trying to detect emotional arousal. Time windows higher than 90 seconds can be used to detect moods.

a c c e l e r at i o n Physical activities of the human body cause an increase in heart rate. Because an increase in heart rate is exactly what the algorithm is supposed to detect the thresholds for heart rate are dynamically changed according to the acceleration data retrieved from the accelerometer of the BioHarness. The acceleration is computed by using the standard deviation over the same time windows defined for the heart rate computation.

r m s s d The Root Mean Square of the Successive Differences (RMSSD) is an index of heart rate variability, discussed in chapter2. Although not used in this study the computation of the RMSSD is implemented to be logged at moments of additional heart rate for future purposes.

4.2 s o f t wa r e

The software developed in this research project is based on the software developed during an earlier study by Blaauw & Zuidhof [12].

The software running on Android consists of multiple components. In this section only the main flow of information, and the main components are described. This section was designed for future developers and can therefore contain some jargon.

(43)

4.2.1 Class diagram

In software engineering, a class diagram in the Unified Modeling Lan- guage (UML) is a type of static structure diagram that describes the structure of a system by showing the system’s classes and the rela- tionships among objects. Figure16 shows a simplified class diagram of the software developed during this research project to give an im- pression of the system.

4.2.2 EmozionActivity

When developing an Android application, one will most certainly use Android’s Activities at some point in the code. An activity is a single, focused thing that the user can do. Almost all activities interact with the user, so the Activity class takes care of creating a window for you in which the user interface can be placed. In this application everything starts at the EmozionActivity. This Activity creates a simple GUI for the user to interact with. This allows the user to connect to, e.g., the BioHarness or the simulator. More importantly, this Activity starts the MeasurementService, which keeps the application running when the GUI does not have focus, or is closed.

4.2.3 MeasurementService

The measurement service is the heart of the application. In contrary to the EmozionActivity, this is an Android service. A service is an application component representing either an application’s desire to perform a longer-running operation while not interacting with the user or to supply functionality for other applications to use. As said, this service is the heart of the application. From this service the different detecting components are started and connections to the data listeners are made (such as the BioHarness or a simulator). When the GUI of the application closes, this service will continue capturing measurements.

4.2.4 Parcels

Communication between the data listeners and the emotion detection components flows using parcels and handlers, provided by the An- droid’s SDK. Parcels in this case are containers which can be used to marshall and unmarshall data, for sending it between classes. Using parcels can be compared with using Java’s serializable.

(44)

Figure 16: Simplified class diagram of the software developed during this study.

4.3 q u e s t i o n na i r e s

Feedback from the user is needed to measure the amount of emotional arousal at the moment of additional heart rate. The software is built to propose single or multiple emotional states. Figure 17shows

(45)

two example. Figure A shows an example of just one mood, the user will not have to select a state since there only is one. Another example is to ask for one out of a list of emotions, for example angry, happy, disgust, anxious, scared, and other (Figure17B). The user is asked to select the emotional state he or she feels is most present at that moment. The slide bar is used to give a weight to the selected emotional state. For the final experiment only one question is asked: Mood. The scale is set from negative to positive where the middle is neutral.

Figure 17: Example of the implementation for the extraction of the affective state. In this case only one state can be selected, mood. The slidebar gives the possibility to select a value between a negative and positive mood, where the middle is a neutral state.

(46)

(47)

5

E VA L U AT I O N O F A R E A L - T I M E A M B U L AT O R Y S Y S T E M F O R T H E D E T E C T I O N O F E M O T I O N A L A R O U S A L

Emotional arousal studies in the past have showed that emotional arousal detection is not a trivial task. Several issues arise when detecting, eliciting and questioning the emotional arousal from participants during experiments in laboratory and real-life environments. In previous laboratory experiments different emotional states are elicited by the testleader. Among others, dependent variables are activities of the autonomic nervous system (heart rate or respiration; [7, 13]) and self-reports of the emotional state. As opposed to field studies, subjects in their laboratory experiments know that they will perceive special situations which may enhance the correlations between physiological activation and self-reported emotional states [28]. In his previous work Myrtek et al performed experiments on emotional arousal.

Emotional events were defined by an increase in heart rate without an accompanying increase in physical activity (additional heart rate).

The conclusion of these ambulatory monitoring studies on the percep- tion of emotions in everyday life [30] was that the detection of additional heart rate indicating emotional arousal was quite different from the results suggested by laboratory experiments. Obviously, in everyday life the identification of emotional arousal is much more difficult than suggested by laboratory studies because we have to account for subjective hypotheses and schemata as proposed by several authors [19,9]. In these studies with nearly 500 subjects, physiological parameters (heart rate, physical activity, additional heart rate) and psycho- logical parameters (excitement, enjoyment) were assessed simultane- ously throughout the day. Based on additional heart rate a feedback signal was given which requested subjects to answer predefined questions (excitement, enjoyment). Besides these true feedbacks, random feedbacks were generated with no indication of additional heart rate.

Subjects in these studies were unaware whether the feedback signal was triggered by their own heart rate or not. Despite great mean differences in heart rate (about 7 bpm) with no differences in physical activity, true and random feedbacks did not show significant differences in excitement and enjoyment ratings. The above ambulatory studies did not ask the subjects for their current emotional state during the experiments. Myrtek continued with these studies but added direct feedback for both feedback moments (true and random feedbacks).

In case of a feedback moment participants were asked to disclose the emotion. The following emotions were listed on a display of a de-

37

(48)

vice the participants had with them: happiness, anger, anxiety/fear, sadness, surprise, and disgust [31].

5.1 va l i d i t y o f a d d i t i o na l h e a r t r at e a s a n i n d i c at o r o f e m o t i o na l a r o u s a l

The method used in the present study is based on the decomposi- tion of heart rate, which reflects both the physical activity and the emotional activation. Emotional arousal was operationalized in the present study by the simultaneous comparison of heart rate and physical activity. An accelerometer (motion detector) is used in order to differentiate metabolic and non-metabolic (additional) heart rate. Be- cause heart rate is dominated by the physical activity, we may expect strong correlations between physical activity measured by motion detectors and heart rate, regardless of whether emotional events are present or not. The strong correlation between physical activity and heart rate is only a prerequisite for the detection of emotional arousal. Other proofs are necessary to endorse the assumption that additional heart rate taps emotional arousal. Myrtek performed two laboratory experiments with male students, where he was able to show that the frequency of additional heart rates was significantly higher when watching an erotic film as compared to a comedy. This finding holds for the student sample watching the films under rest- ing conditions as well as for the second sample watching the films during light but varied physical activity [30]. An ambulatory monitoring study with train drivers showed that the frequency of additional heart rate was higher for the driving modes "start" and "braking to stop" as compared to train speeds between 0 and 200 km/h [32]. The heightened risk of accidents while driving into and leaving the sta- tion may be the reason for the increase in emotional arousal. A third proof stems from a study with schoolboys [30,29]. In this study, boys with light TV consumption were compared to boys with heavy TV consumption. Boys with heavy TV consumption displayed a significantly lower frequency of additional heart rate while watching TV than boys with light consumption. In a study with preschool children Myrtek showed two cartoons. This study showed that a cartoon with many action scenes caused a higher frequency of additional heart rate compared to more neutral cartoons. A movie with more predefined emotionally intense moments causes more additional heart rates. Fur- ther arguments supporting the view that additional heart rate is an indicator of emotional arousal derive from special social situations.

With student samples, Myrtek showed that social contacts with peers revealed a lower frequency of additional heart rate than social contacts with strangers [30]. Other ambulatory monitoring studies with samples of cardiac or rheumatic patients showed that the frequency of additional heart rate was higher during contacts with doctors or

(49)

nurses as compared to contacts with other patients or being alone. In a study with students, it was shown that additional heart rate was much higher during driving a car than during usual daily activities [25,31].

5.2 l a b o r at o r y a n d f i e l d s t u d y

In the present study ten female participants use a system capable to detect additional heart rate for 4 hours. The experiment consists of two parts of both 2 hours starting with a controlled, static environment followed by an ambulatory, open environment. The first part is the same for every participant, the participant is asked to watch the movie ’Trainspotting’. This decision is made because an activity is required that causes minimal physical activity and a way to provoke emotional arousal in a human subject. To provoke these emotions research is done on emotional arousing movies. According to Myrtek’s paper negative emotions (anger and anxiety/fear) result in higher increases in heart rate compared to a positive emotion (happiness). In a paper of Schaefer et al [6] research is done in movies that excel in 10 emotional categories. The movie that excelled in negative affect was the movie ’Trainspotting’. The second part of the experiment is focused on the reliability of the BioHarness signal. Because the Bio- Harness is a band and does not include electrodes placed on the body, the band can slide during physical activity which may result in an un- reliable signal. This part is different for everyone, when the movie is finished the participant is asked to continue with their normal daily activities. Because every person is different, a personality questionnaire is filled out by the participant. This can help in cases of deviated results during analyses of individual participant. For this study only female participants were selected since female subjects have a higher emotional responsiveness compared to male subjects [10].

5.3 a i m s o f t h e p r e s e n t s t u d y

In the current study a replication of Myrtek’s experiments are performed, both in a laboratory and ambulatory setting while using state-of-the-art hard- and software. Recent technological developments give the possibilities for conducting these measurements in a way that may be more convenient for participants. Therefore, in the present study is decided to use the same method of feedback moments as used in Myrtek’s experiment. Participants are asked to indicate their emotional state on both true and random feedback moments on an app running on their smartphone. The experiment is set up to evaluate the system on accuracy and reliability by analyzing data from multiple participants in two different settings inspired by previous studies described earlier in this introduction [30, 31]. In his paper

(50)

Myrtek has shown that lower mean heart rates were found during random feedback moments compared to true feedback moments, as is expected since true feedback moments are triggered when an increase in heart rate is detected. This is what is expected in this study [30]. In the same paper Myrtek indicates that the expected differences between excitement and enjoyment during true and random feedback moments were not found during ambulatory measurements.

This was in contradiction with his previous work in laboratory experiments. When human subjects had minimal physical activity the additional heart rate could be addressed to emotional arousal [30,31].

Based on this information I hypothesize that in the present study:

1. During the laboratory experiments I expect to detect additional heart rates triggering true feedback moments at predefined emotionally intense moments in the movie

2. During the laboratory experiments higher subjective affective scores and heart rate will be found at the true feedback moments compared to random feedback moments

3. In contrast to the laboratory experiments, during ambulatory experiments the differences in heart rate and subjective affective score between true and random feedback moments are less/not distinguishable

5.4 m e t h o d s

This chapter elaborates on the participants, methods, procedures and results used during this experiment and will conclude with answers to the hypotheses defined above.

5.4.1 Participants

Ten female Dutch students (mean age 22.7, range 19-26) were re- cruited by posters at and around the university. Women were eligi- ble for participations if they fulfilled the following criteria; aged between 18 and 26 years, good general health, no known cardiovascular problems, Dutch as the native language, and preferably possessing an Android phone. The complete experiment was set up in accordance with the Ethical Committee. Before the participants started with the experiment they gave written informed consent. Participants were re- warded with a chocolate heart of chocolate and had a chance to win a gift certificate of 50 Euros.