Applications of wavelet transform and recurrence quantification analysis for embodied music interaction: Analysis of the SoundBike and the Pendulum experiment

(1)

recurrence quantification analysis

for embodied music interaction

Analysis of the SoundBike and the Pendulum experiment

Word count: 26809

Kristel Crombé

Student number: 19962326

Supervisor: Prof. dr. Pieter-Jan Maes

A dissertation submitted to Ghent University in partial fulfilment of the requirements for the degree of Master of Arts in Art History, Musicology and Theatre Studies

(2)

Declaration of Authorship

The author and the supervisor give permission to make this study entirely available for consultation for personal use. In all cases of other use, the copyright terms have to be respected, in particular with regard to the obligation to state explicitly the source when quoting results from this study.

(3)

Abstract

The thesis applies two advanced mathematical techniques, wavelet transform and Recurrence Quantifi-cation Analysis (RQA), to music interaction experiments and presents guidelines that may be helpful for the use of the methods in future studies. The broad research field is embodied music cognition and the experiments that are analysed in the thesis, question synchronisation and entrainment. Many people spontaneously synchronise their movements to music. In the SoundBike experiment the motor rhythm of cyclists is made audible (i.e. sonified) in order to increase their tendency to synchronise to an external musical rhythm. Different sonification methods are investigated and compared to control conditions. Results from a joint RQA analysis suggest that sonification of the cycling rhythm indeed improves the spontaneous synchronisation and the stability of the pedalling, when people have a musical background. With an explicit instruction to synchronise, a significant improvement in syn-chronisation strength and pedal cadence stability is observed in general, also with experienced cyclists without musical background. The Pendulum experiment is inspired by the finding of Christiaan Huy-gens that two pendulum clocks spontaneously synchronise when hanging from a common wooden beam. We explore the possible use of a physical (mechanical) coupling between people to strengthen (spontaneous and instructed) musical synchronisation. The effects of performing an additional col-laborative task and of moving at non-equal starting tempi, are investigated. The analysis is based on cross wavelet and synchrosqueezed transforms, as well as joint RQA. The overall conclusion is that a vestibular exchange of information is indeed established, which influences the social coordination and the stability of the interpersonal interaction, in particular when instructed to sync. The additional collaborative task and the initial tempi have no strong effect on the interpersonal interaction. Both experiments have possible applications in the fields of sports and motor rehabilitation, music eduction and performances, music therapy, virtual (and augmented) reality and well-being.

(4)

Quotations

“ In the melodies themselves there are imitations of character, and this is clear. For the nature of the harmoniai is simply different, and as a result listeners are put into different dispositions and do not have the same way of reacting to each of them. [...] this applies in the same way to rhythms. For some of them have a steadier character, others a dynamic one, and among them some have more constrained movements, others have movements more fit for the free. ”

Aristotle, Politics, Book VIII 1340 a40 - b10, as translated in Fiecconi.1

“ It occurred to me by intuition, and music was the driving force behind that intuition. My discovery was the result of musical perception. ”

Albert Einstein, being asked about his discovery of the theory of relativity.

“ It turns out that an eerie type of chaos can lurk just behind a facade of order - and yet, deep inside the chaos lurks an even eerier type of order. ”

Douglas R. Hofstadter, Metamagical Themas: Questing for the Essence of Mind and Pattern.

1_{Elena C. Fiecconi, "Harmonia, melos and rhythmos: Aristotle on musical education," Ancient Philosophy 36 (2016):}

(5)

Acknowledgements

My study of art history and musicology started as an adventure a few years ago, purely driven by my passion for art and music and my strong curiosity and eagerness to learn. The courses have made me discover original approaches and have forced me, more than once, to change perspective on history, society and life in general. It is impossible to say how much I have learnt in the lectures and practical sessions and how the new insights will continue to form me as a person. I can hardly believe this exciting and interesting period is coming to an end.

With a background in physics, it is no surprise that my master’s thesis goes in a scientific direc-tion. The interdisciplinary domain of systematic musicology comprises many crossovers between art and science. The lectures about acoustics, music perception and music psychology, as well as the hands-on experiences during the research seminar on music interaction, all combine physics and en-gineering with music and social interaction. Here I encountered prof. Pieter-Jan Maes, to whom I would like to pay my special regards. It is thanks to him that I entered this research field. He listened carefully to my interests and tailored topics for a bachelor’s and a master’s thesis that would captivate and challenge me. I have learned valuable novel analysis techniques that are broadly applicable. I have high respect for his thorough scientific way of tackling the complex studies of human behaviour, as well as for his kind guidance and focus during the analysis process. I also wish to express my gratitude to prof. Marc Leman for opening up the Art-Science-Interaction Lab (ASIL) of IPEM to the students and for stimulating us to design a new experiment. His broad knowledge and creativity in applying advanced mathematical methods to the data are inspirational; he always seeks to try out something new, to change the point of view, in order to understand the underlying mechanisms.

Although we have never met in person, prof. Alexander Demos’s help with the RQA and statisti-cal analysis of the SoundBike experiment is much appreciated.

Special thanks to the members of the IPEM team, for their valuable practical assistance, scien-tific advice, positive criticism and especially their patience during the research seminar. The variety of personal backgrounds and skills are precisely the strength of this internationally renowned group. Finally, I have not gone through this adventure alone. I am grateful for the support and appreci-ation from friends and family. I feel lucky to have been surrounded by a warm group of students, future musicologists and art historians, young and not so young, with strong characters and nice personalities. The last line is for my parents, who have taught me to question ’normality’.

(6)

List of Figures

1.1 The assessment model as proposed by Leman [Leman 2017]. The observer interacts with a pattern (sound) and generates an estimate about the state of being that might have produced the observed pattern. An enactive assessment would imply that the assumed state is an action-state, which the observer connects with the own repertoire of action experiences. . . 4 1.2 The SoundBikes used in the experiment [Maes 2018a]. . . 6 1.3 The set-up of the Pendulum experiment [Denys 2019]. . . 7 1.4 Recurrence plots of (A) a periodic motion with one frequency, (B) the chaotic Rössler

system and (C) of uniformly distributed noise [Marwan 2007]. The periodic signal is deterministic with large diagonal lines. The noise is uncorrelated with many single recurrence points. The chaotic Rössler system falls in between. . . 10 1.5 Morlet wavelet with centre frequency fc=1.27 Hz (ω0= 8 rad) and positive bandwidth

parameter fb = 0.75 Hz. . . 14

1.6 Signals from the accelerometers in the shakers for one of the trials of the Pendulum experiment (duo 12, signals from the shakers, coupled platforms, identical starting tempi, no additional task). . . 16 1.7 Wavelet information for the same trials as fig. 1.6. Left: WPS for participant 1; middle:

WPS for participant 2; right: wavelet coherence and relative phase, indicated by arrows. 17 1.8 Left: a harmonic signal x(t) =sin(8t); middle: the continuous wavelet transform of

x; right: the synchrosqueezed transform of x [Daubechies 2011]. . . 18 1.9 Synchrosqueezed wavelet transforms of the participants’ shaker signals for the same

trial as fig.1.6. . . 18 2.1 Music and pedalling are modelled as two coupled oscillators. (a) The BPM of the music

determines the angular frequency of the music oscillator ωm. The vector rotates and

triggers sound samples (beat, hi-hat, or snare) at specific angular positions. One rota-tion is defined by 250 discrete angular values. (b) An optical rotary encoder mounted in the bike, detects the angular position of the pedal. Similar as to the external music oscillator, one rotation is defined by 250 discrete values. The phase θp is determined

relative to the point of maximal pedal pressure of the left foot (Pmax index). . . 22

2.2 Schematic overview of the experimental procedure, consisting of four distinct parts, but performed uninterruptedly. . . 23 2.3 Joint Recurrence Plot for the musical rhythm and pedal cadence for participant 9 in

three conditions. From left to right: spontaneous synchronisation without sonification (Condition 2), spontaneous synchronisation with sonification (Condition 3), instructed synchronisation (Condition 5). . . 26

(9)

2.4 Means and standard errors for the Group x Sonification Condition interaction effect for RR. Statistically significant differences between the control conditions, chance (Con-dition 1) and spontaneous synchronisation (Con(Con-dition 2), and the con(Con-ditions with sonification (but without instructed sync) are indicated with stars. For the musicians (Group=0) statistically significant differences are found between the reference (Con-dition 1) and the con(Con-ditions with sonification (Con(Con-ditions 3 and 4). Moreover the sonification with reward (Condition 4) yield a higher RR than the spontaneous synchro-nisation without sonification (Condition 2). For the cyclists no statistically significant effect of the sonification on spontaneous synchronisation is found. . . 27 2.5 Mean values and standard errors for the Sonification Condition x Group interaction

effect for RR. Only for the instructed synchronisation (Condition 5) a statistically sig-nificant difference is found between the two groups, with a higher RR for the musicians (Group 0). . . 28 2.6 Overview of the RR (mean values with standard error as error bars) for the Sonification

Condition x Group interaction effect. . . 29 2.7 Means and standard errors for the JRQA stability value for the different conditions.

Statistically significant differences are indicated by stars. . . 30 2.8 ADL for Sonification Condition x Group. Overall the pedal cadence of the cyclists

(Group 1) is more stable in all conditions. . . 31 2.9 Post-hoc analysis for the main effect of Sonification Condition: statistical results for

the differences in recurrence rate values. . . 36 2.10 Post-hoc analysis for the main effect of Sonification Condition: statistical results for

the differences in stability values. . . 37 3.1 Schematic representation of the set-up of the experiment. . . 41 3.2 Scalogram of a continuous wavelet (top row) and a synchrosqueezed transform

(bot-tom row) for the two participants of duo 16 in the condition: coupled Platforms, spontaneous sync, with collaborative Task and at equal starting Tempo. . . 43 3.3 Resulting dominant frequencies of the two participants and the mean frequency of the

duo, obtained by the synchrosqueezed transform for the same trial as in figure 3.2 (top). Root-mean-square deviation (RMSD) (bottom). . . 44 3.4 Coherence and relative phase (indicated by arrows) resulting from the cross wavelet

transform (same trial as in figure 3.2). The red line is the mean dominant frequency of the duo as derived from the synchrosqueezed transforms. . . 45 3.5 Coherence and relative phase, determined by the cross wavelet transform, at the mean

dominant frequency of the duo. The mean dominant frequency is derived from the synchrosqueezed transforms. . . 46 3.6 Joint recurrence plot with the selected parameter settings (same trial as in figure 3.2). 47

(10)

3.7 Resultant vector lengths and RMS differences in dominant movement frequencies for the three different Platform conditions. The data from the accelerometers connected to the knees of the participants is used in (a) and (b) and the data from the shakers is used in (c) and (d). Significant differences are indicated by stars. . . 49 3.8 The diagonal parameters from the JRQA all yield a main effect for Platform. . . 51 3.9 JRQA parameters for the knees and the shakers. Significant main effects are found

for Platform, in particular for the condition with coupled platforms and instructed synchronisation. . . 52 4.1 Comparison of Morlet and symmetric Morse wavelets in time domain. . . 59 4.2 Time domain representation of symmetric Morse wavelets. Each subfigure shows the

wavelet with the lowest and highest central frequency (CF). Time-bandwidths are changing: left: P2 _{= 9, middle: P}2_{= 49, right P}2 _{= 100. The lowest P}2 _{has the most}

narrow time response. . . 59 4.3 Frequency domain representation of the symmetric Morse wavelet with three different

time-bandwidths (left: P2 _{= 9, middle: P}2 _{= 49, right P}2 _{= 100) and two different}

values for the number of voices per octave (top: 10, bottom: 40). The bottom right figure (highest P2 _{and highest number of voices per octave) has the most narrow}

frequency response. . . 60 4.4 Scalograms of the shaker data for participant 2 of duo 11 for one of the conditions

(coupled platforms, no collaborative task, equal starting tempi) for different parameters of the CWT using Morse wavelets. The time (frequency) resolution decreases (increas-ing) with P2_{value. The larger the number of voices per octave the more accurate the}

frequencies are defined. . . 60 4.5 Left: synthetic signal with different frequencies and amplitudes, middle:

convolution-based CWT scalogram, right: FFT-convolution-based CWT scalogram. . . 62 4.6 Example of significant regions for the wavelet coherence parameter using the adapted

Grinsted routines. Left: number of scales to smooth = 48 (recommended), middle: number of scales to smooth = 24 (still appropriate), right: number of scales to smooth = 4 (too little). . . 63 4.7 (a) Shaker’s acceleration for participant 2 of duo 10 in the condition: coupled platforms,

with collaborative task, equal starting tempi; (b) FNN test to find the embedding dimension, minimum for D = 4; (c) AMI test to find the delay parameter, first local minimum for τ = 11 - 13; (d) resulting phase-space plot for D = 4 and τ = 12. . . 65 4.8 RP for the shaker data of participant 2 of duo 10 in the condition: coupled platforms,

with collaborative task, equal starting tempi. Left: low threshold parameter T = 0.7, which corresponds to %REC = 5%. Right: high threshold parameter T = 1.7, which corresponds to %REC = 31%. . . 68

(11)

4.9 Effect of increasing the time delay parameter τ. Left: τ = 10, right: τ=18. Top: complete time window, bottom: zoom on the first 15 seconds. Increasing the time delay makes it more difficult to find long pieces of connected diagonals, DET decreases. The same time series was taken as in fig. 4.7(a), D=4 and T=0.7. . . 69 4.10 Effect of increasing the embedding parameter D. Left: D = 3, right: D=5. The same

time series was taken as in fig. 4.7(a) and the other parameters were set to τ=12 and T=0.7. Increasing the embedding results in a reduction of the number of recurrences. 69

(12)

List of Abbreviations

General

ASIL Art-Science-Interaction Lab

IPEM Institute for Psychoacoustics and Electronic Music

Recurrence quantification analysis

ADL Average Diagonal line Length

AMI Average Mutual Information

CRP Cross Recurrence Plot

CRQA Cross Recurrence Quantification Analysis

DET DETerminism

FNN False Nearest Neighbour

JRP Joint Recurrence Plot

JRQA Joint Recurrence Quantification Analysis

MDL Maximum Diagonal line Length

MdRQA Multidimensional Recurrence Quantification Analysis

%REC percent RECurrence

RP Recurrence Plot

RR Recurrence Rate

RQA Recurrence Quantification Analysis

Wavelet transform

COI Cone Of Influence

CWT Continuous Wavelet Transform

FFT Fast Fourier Transform

FT Fourier Transform

RPh Relative Phase

WCO Wavelet COherence

WCS Wavelet Cross Spectrum

WPS Wavelet Power Spectrum

WT Wavelet Transform

(13)

1 Research context and selected methodology

With an earlier degree in engineering physics, the field of systematic musicology caught my partic-ular attention during my study of art history and musicology at Ghent University. The Art-Science-Interaction Lab (ASIL) of IPEM is an ideal environment to combine physics and engineering with music and social interaction.1 _{The focus is on embodied music cognition and expressive interaction}

with music, such as listening, performing, conducting, dancing, sporting, and physical rehabilitation. The research is challenging because neither human behaviour nor music can be exactly described. Pro-cesses are complex and dynamic and therefore intrinsically difficult to model and interpret correctly. Many different parameters play a role, which are non-linearly coupled to each other. At the same time, the multiple layers and constant changes might also explain the strong impact of music on humans, since it challenges the predictive coding capabilities of the brain.2 _{Interdisciplinary methods are}

de-veloped based on music theory, performer-inspired analysis, advanced behavioural and neuroscience empirical experimentation, statistics, and computer modelling.

This thesis applies two advanced mathematical techniques, Wavelet Transform (WT) and Recurrence Quantification Analysis (RQA), to actual experimental data obtained at ASIL. Many of the activities performed at IPEM lead to scientific publications, therefore it was decided to follow a similar approach and to build the thesis around two research papers. First the background, context and analysis methods are introduced (chapter 1). For more details on the chosen mathematical tools and their use in music interaction studies, I also refer to my bachelor’s thesis.3 _{The datasets originate from the SoundBike}

and the Pendulum experiments.4,5_{The results are presented in the papers in chapter 2 and chapter 3.}

The main conclusions and guidelines for the use of the methods in future experiments are summarised in chapter 4.

1_{IPEM (Institute for Psychoacoustics and Electronic Music) started in 1963 as a studio for electro-acoustic music}

production. At present it is the Ghent University research institute for systematic musicology. It is situated in De Krook building in Ghent, Belgium. Information on the research can be found on the IPEM website:https://www.ugent.be/ lw/kunstwetenschappen/ipem/en, last consulted July 2020 [IPEM].

2_{The brain model based on predictive coding theory is explained in Karl Friston, "Hierarchical models in the brain,"}

PLoS Computational Biology 4 (2008): e1000211 [Friston 2008]. Some examples of their applications to music can be found in Ole A. Heggli et al., "Musical interaction is influenced by underlying predictive models and musical exper-tise," Nature Scientific Reports 9 (2019): 11048 [Heggli 2019] and in Peter Vuust and Maria A.G. Witek, "Rhythmic complexity and predictive coding: a novel approach to modeling rhythm and meter perception in music," Frontiers in Psychology 5 (2014): 1111 [Vuust 2014].

3_{Kristel Crombé, "Wavelet transform and Recurrence Quantification Analysis for embodied music interaction.}

Ex-ploration of the methods," Bachelor’s thesis (Ghent University, 2019) [Crombé 2019].

4_{The SoundBike experiment is described in Pieter-Jan Maes et al., "Embodied, participatory sense-making in}

digitally-augmented music practices: theoretical principles and the artistic case ’SoundBikes’," Critical Arts 32 (2018): 77 - 94 [Maes 2018a] and in Pieter-Jan Maes, Valerio Lorenzoni and Joren Six, "The SoundBike: musical sonification strategies to enhance cyclists’ spontaneous synchronization to external music," Journal on Multimodal User Interfaces (2018): 1 - 12 [Maes 2018b].

5_{The Pendulum experiment is described in Marlies Denys, "De rol van vestibulaire informatie in het tot stand komen}

(14)

1.1 Embedded in an interdisciplinary background

Systematic musicology is a broad and highly interdisciplinary research domain. Since ancient times the study of music has been connected to various other fields, ranging from exact sciences as mathe-matics and astronomy, over psychology and well-being to pedagogy, politics and social science. Early western writings on the subject include the viewpoints of Greek philosophers and of scholars from the Middle Ages to the Renaissance.6 _{Less well-known are for instance ancient Indian texts as the}

Natyashastra from around 500 BC on music and arts, or the complex tonal system theories embedded in the Cilappatikaram, a Tamil epic of today’s Sri Lanka, comparable in importance to the Iliad and Odyssey. Also the Chinese Yueji, the ’Record of Music’, from the 2nd century BC, includes discussions on nature and the social role of music.7

Despite the long tradition, it takes until the 19th century for musicology to enter European universi-ties as an academic discipline. In the beginning research focuses strongly on historical and philological aspects as well as on music theory and analysis. Belgian composer and musicologist François-Joseph Fétis (1784-1871) somewhat broadens the view by his interest in folk genres and non-Western music traditions.8 _{For many years the musical score is considered as the most important and autonomous}

centrepiece of the work. Positivism and empiricism dominate the way of thinking. Empirical proof, facts and knowledge are considered as the main driving forces for progress. It is believed that the original ideas of the composer and the real meaning of the work are confined in the score waiting to be revealed by an accurate musical analysis. In the course of the 19th century more emphasis is put on the experience of the listener, moreover influenced by Hermann von Helmholtz’s (1821-1894) work about the relationship between vibration and sound, hearing as a sensory and perceptual pro-cess, and music as an art based on patterns of sound that are meant to communicate.9 _{Concepts as}

embodied cognition and perception enter musicological research. The change is driven by scientific developments in the 19th and early 20th century in modern physics (quantum mechanics, relativity theory) and medicine (psychoacoustics, psychology, neuroscience) and by postmodern philosophy. A hermeneutic analysis method accepts elements as socio-cultural context and personal interpretation. Nevertheless it took until the end of the 20th century for systematic musicology to develop as a full-fledge discipline. Schneider clarifies the distinction:

A significant difference between historic and systematic orientations in musicology in fact was and is that the former worked (and continues to work) predominantly with music as text (conventional notations or similar symbolic representation), while the latter from the very beginning had a focus on sound and perception. Further, systematic musicology,

6_{Lelouda Stamou, "Plato and Aristotle on music and music education: lessons from Ancient Greece," International}

Journal of Music Education 39 (2002): 3 - 16 [Stamou 2002].

7_{preface to Springer handbook of systematic musicology by Rolf Bader (Berlin, Heidelberg: Springer, 2018), v - vi}

[Bader 2018].

8_{Albrecht Schneider, "Systematic musicology: a historical interdisciplinary perspective," in Springer handbook of}

systematic musicology, ed. Rolf Bader. (Berlin, Heidelberg: Springer, 2018), 17 [Bader 2018].

9_{Hermann von Helmholtz, "Ueber die Wahrnehmung der Klangfarben," in Die Lehre von den Tonempfindungen als}

physiologische Grundlage fuer die Theorie der Musik (Braunschweig: Verlag von Fr. Vieweg und Sohn, 1863), 182 -223 [Helmholtz 1863].

(15)

like acoustics, psychoacoustics, and psychology, made use of experimental methodology including statistics.10

A short introduction to embodied music cognition is useful to contextualise the empirical research of this thesis. Human interaction with music, individually or in group, can be analysed in the context of embodied cognition.11,12_{Musical meaning is explained as an active process in which the body interacts}

with the music. This is the case both for the performer and for the listener. Central in the approach is the bodily comprehension of sounds and of sound-producing actions. The human body is seen as the mediator for meaning formation by connecting sensorimotor inputs with cognitive and emotional systems. Arnie Cox starts from the principle of imitation, in the broad sense, a key element in social psychology. He develops a theoretical background for embodied music cognition based on mimetic comprehension and mimetic motor imagery.13 _{In the case of performers it can be understood in the}

sense that performing, planning, and thinking about musical performance are tied to bodily actions. However the situation is less straightforward in the case of listeners: How and why would listening to or thinking about music, apart from planning or recalling one’s own performance, have anything to do with embodiment beyond the operations of the auditory system? The answer offered by Cox is that listening to, recalling, or otherwise thinking about music involves one or more kinds of vicarious performance, or imitation (or simulation), and that the role of this imitation in music is a special case of its general role in human perception.14 _{The assessment model by Leman follows a similar idea.}15

It assumes that listeners assess music in terms of action-states that could have caused the particular expressive character of the music; it forms the basis for musical meaning formation. In the process listeners connect to their own repertoire of experiences. A schematic representation of the model is depicted in figure 1.1. The observer interacts, through action and perception, with a sound pattern and infers an estimate about the state of being that might have produced the observed pattern. The inferred action-state refines a predictive model for future assessments of the sonic patterns (i.e. the music as sound).

Furthermore, interaction with music is a time dependent process, often relying on socio-motor im-provisation. The assessment of music and meaning formation is not constant, it may change during the actual interaction process.16 _{Important temporal parameters are rhythm, synchronisation and}

en-trainment. They are also the central parameters in the experiments for this thesis. Rhythmic structure is encoded in our bodily movements as we listen to music, and movement enhances our rhythmic

10_{Schneider, "Systematic musicology: a historical interdisciplinary perspective," 17 [Bader 2018].}

11_{Embodied music interaction is explained in Marc Leman, Micheline Lesaffre and Pieter-Jan Maes, "What is}

embod-ied music interaction?," in The Routledge companion to embodembod-ied music interaction, ed. Micheline Lesaffre, Pieter-Jan Maes, Marc Leman. (New York: Routledge, 2017), 4 - 5 [Leman 2017].

12_{The principles of embodied music cognition are summarised in Marc Leman et al., "What is embodied music}

cognition?," in Springer handbook of systematic musicology, ed. Rolf Bader. (Berlin, Heidelberg: Springer, 2018) 747 - 60 [Leman 2018].

13_{Arnie Cox, Music and embodied cognition. Listening, moving, feeling, and thinking. (Bloomington, Indianapolis:}

Indiana University Press, 2016), 11 - 57 [Cox 2016].

14_{Cox, Music and embodied cognition. Listening, moving, feeling, and thinking, 11 [Cox 2016].} 15_{Leman,"The interactive dialectics of musical meaning formation", 13 [Leman 2017].} 16_{Leman,"The interactive dialectics of musical meaning formation", 13 - 18 [Leman 2017].}

(16)

Figure 1.1: The assessment model as proposed by Leman [Leman 2017]. The ob-server interacts with a pattern (sound) and generates an estimate about the state of being that might have produced the observed pattern. An enactive assessment would imply that the assumed state is an action-state, which the observer connects with the

own repertoire of action experiences.

awareness.17 _{Resonances of the human body may explain beat-synchronised movements.}18_Yet

musi-cal melodies and rhythms are more complex than metronomes and typimusi-cally involve multiple, related periodicities.19 _{In view of embodied cognition it is thus assumed that by observing patterns in bodily}

motions, information can be retrieved about the musical experience. Techniques generally used for the analysis of dynamical systems are useful tools in a holistic study of social music coordination. The time varying and self-organising aspects of interaction are stressed by Leman:

Form and regularity, change and evolution, emerge from dynamical interaction effects and constraints at lower levels rather than being externally imposed, as for example by a central human mind. This focus on the processes of interaction, being self-organizing and emer-gent, entails focusing on time-varying aspects within music-based interactions, including temporal patterns of change (instabilities, transitions) and recurrence (stability).20

The holistic approach yields a better comprehension of a performance than looking at partial isolated aspects. In this philosophy the inclusion of subjective experiences (qualitative information) to comple-ment the quantitative data is essential. The mixed-method approach integrates the two forms of data in one single framework. The core assumption is that the combination of qualitative and quantitative approaches provides a more complete understanding of the research problem than either approach alone.21 _{It is possible to apply mixed-methods to human music interaction.}22 _{The information on}

17_{Fiona Manning and Michael Schutz,"Trained to keep a beat: movement related enhancements to timing perception}

in percussionists and non-percussionists", Psychological Research, 26 (2019): 532 - 42 [Manning 2016].

18_{Frederik Styns et al.,"Walking on music", Human Movement Science, 26 (2007): 769 - 85 [Styns 2007].}

19_{Justin London et al.,"Tapping doesn’t help: synchronized self-motion and judgments of musical tempo", Attention,}

Perception, & Psychophysics, 81 (2019): 2461 - 72 [London 2019].

20_{Leman, "What is embodied music interaction?," 4 - 5 [Leman 2017].}

21_{John W. Creswell, "The three approaches to research", in Research design: Qualitative, quantitative, and mixed}

methods approaches, (4th ed., international student edition, Los Angeles: SAGE, 2014) 32 [Creswell 2014].

22_{Bavo Van Kerrebroeck, "Linking embodied coordination dynamics and subjective experiences in musical}

(17)

emotional and other personal experiences can be obtained by questionnaires, ratings and physiological measurements including skin conductance, eye movements and heartbeat. The thesis however focuses on quantitative information, but results can be complemented later by an analysis of subjective data, obtained from the participants before and after the experiments. A further connection to dynamical systems is made by the identification of states, actions and predictions:

Another way to approach this is to see it [embodied music cognition] in terms of patterns and states, and the predictive models surrounding the connections between patterns and states [. . .] Patterns are the elements that can be observed, while states are the conditions that lead to those patterns. Predictive models of the human brain are specialized in making assumptions about those conditions, given the observed patterns. The mechanism works in fact in dual direction; namely, action and perception. First of all, humans learn that actions cause particular changes in the environment. Actions are supported by predictive models that associate motor commands for actions that have perceived sensations of the outcomes of those actions. Secondly [. . .] Perceptions are supported by predictive models that associate observed patterns as action outcomes in dual direction; namely, from actions to action outcomes and observed patterns, and, from observed patterns to assumed conditions and actions that cause these patterns.23

A stable state that combines an optimal equilibrium of behavioural, physiological and subjective pa-rameters within a system is defined as ’homeostasis’.24

In order to get a good idea of the actual processes, a variety of data is collected during experi-ments, often using innovative technology. Movements and gestures are captured with MoCap suits, pressure sensors or accelerometers; physiological parameters as skin conductance, heartbeat or breath-ing tempo, provide information on arousal, emotions and attention; neural activity is measured by brain imaging techniques as EEG, ERP and fMRI. The tools to analyse the data are taken from different domains: signal processing, physics, engineering, geology, computer science, . . . Advanced methods, both linear and non-linear, are applied to interpret observed features in the data, including Fourier and wavelet transforms, Bayesian statistics, Recurrence Quantification Analysis and neural network analysis. An overview of quantitative techniques to analyse rhythm and timing is given in the tutorial paper by Ravignani and Norton.25_{The use of non-linear methods in the broad domain of behavioural}

science is introduced in the book by Riley and Van Orden, with emphasis on recurrence quantification analysis, social group coordination, and human development.26

23_{Leman, "What is embodied music cognition?," 748 - 9 [Bader 2018].}

24_{Marc Leman, "Chapter 7: Expressive alignment", in The expressive moment: How interaction(with music) shapes}

human empowerment, (Cambridge, MA and London: MIT press, 2016) 178 - 9 [Leman 2016].

25_{Andrea Ravignani and Philipp Norton, "Measuring rhythmic complexity: A primer to quantify and compare temporal}

structure in speech, movement, and animal vocalizations," Journal of Language Evolution 2 (2017): 4 - 19 [Ravignani 2017].

26_{Charles L. Webber Jr. and Joseph P. Zbilut, "Recurrence quantification analysis of nonlinear dynamical systems",}

in Tutorials in contemporary nonlinear methods for the behavioral sciences, ed. Michael A. Riley and Guy C. Van Orden. (Arlington, VA: National Science Foundation, 2005), 26 [Webber 2005].

(18)

1.2 Focus of the empirical research for the thesis

The starting point for studies in systematic and cognitive musicology is the performance itself, as opposed to the musical score or historical context. The research questions in the two selected studies for the thesis, the SoundBike and the Pendulum experiment, all focus on synchronisation and entrainment. Synchronisation to a rhythmic sound seems to be a rather natural capacity of human beings. With this can come a sense of agency, a rewarding feeling of control. Sound and movement can even stimulate a coordinated action of a group of individuals.27 _{The synergy between performers can create a higher}

level of consciousness that is transcending the individual players. The term entrainment refers to the process by which independent rhythmical systems interact with each other.28 _{Interpersonal musical}

entrainment is what happens when people get together for activities involving music (concerts, dances, religious rituals, even sporting events), and synchronise what they are doing.

1.2.1 The SoundBike experiment

Figure 1.2: The SoundBikes used in the experiment [Maes 2018a].

Originally, the so-called SoundBikes have been designed as part of a music installation, which fits the concept of embodied music cognition and is based on the idea that music performance is an embodied and a participatory activity, in which sensorimotor control and social interaction are fundamental. The main components of the installation are stationary bikes equipped with sensors for sound control. The performers/cyclists can dynamically control playback parameters of precomposed songs, such as the musical tempo, the number of musical layers, the spatialisation of the sound. The primary goal is to create embodied and participatory human interactions using sound synthesis and movement sensing

27_{Studies on group interaction are reported in Pauline M. Hilt et al., "Multi-layer adaptation of group coordination}

in musical ensembles," Nature Scientific Reports 9 (2019): 5854 [Hilt 2019] and in Donald Glowinski et al., "Music ensemble as a resilient system. Managing the unexpected through group interaction," Frontiers in Psychology 7 (2016): 1548 [Glowinski 2016].

28_{Martin Clayton, "What is entrainment? Definition and applications in musical research," Empirical Musicology}

(19)

technology.29 _{An illustration of the original set-up is shown in figure 1.2. In a later implementation}

the SoundBikes are used to make the motor rhythm of cyclists audible and to have them interact with beat-annotated music pieces.30_{The beat can be adapted and aligned to the pedal cadence.}

Par-ticipants are monitored individually. The experiment focuses on the synchronisation with music. The interpersonal aspect is not considered here. The main research question is which kind of sonification strategies enhances cyclists’ tendency to synchronise the pedalling cycle to the rhythm of the music. Different sonification methods are investigated and compared to a spontaneous synchronisation con-dition, during which external music is played without an audible pedal frequency. The results of the analysis using RQA are discussed in chapter 2. Possible application domains are sports and physical rehabilitation. Three functions of sonification for human performance can be identified: motivation, monitoring and modification.31 _{An interactive sonification device, as the SoundBike, can assist motor}

learning activities and monitor and correct repetitive movements.

1.2.2 The Pendulum experiment

Figure 1.3: The set-up of the Pendulum experiment [Denys 2019].

The Pendulum experiments looks at interpersonal synchronisation and entrainment. Joint musical in-teraction typically relies on the mutual exchange of auditory and visual information between musicians. In that way the musical actions are modified to improve the synergy and coordination between per-formers. Inspired by the finding of Christiaan Huygens in 1665 that two pendulum clocks spontaneously synchronise when hanging from a common wooden beam, an experiment has been designed to ex-plore the possible use of a physical (mechanical) coupling between people to strengthen (spontaneous and instructed) musical synchronisation.32 _{The general research question is whether an exchange of}

vestibular information induces social coordination and entrainment. For that purpose duos perform a synchronisation-continuation task, while standing on a movable platform, which mimics the beam in Huygens’s experiment. A schematic representation of the experimental set-up is shown in figure 1.3.

29_{Maes, "Embodied, participatory sense-making in digitally-augmented music practices: theoretical principles and the}

artistic case ’SoundBikes’," [Maes 2018a].

30_{Maes, "The SoundBike: musical sonification strategies to enhance cyclists’ spontaneous synchronization to external}

music" [Maes 2018b].

31_{Pieter-Jan Maes, Jeska Buhmann and Marc Leman, "3Mo?: a model for music-based biofeedback," Frontiers in}

Neuroscience 10 (2016): 548 [Maes 2016].

(20)

Participants stand on a wooden platform (2.5 m x 1 m). A fabric screen is placed between them, to avoid visual contact. On a computer screen different tasks may appear. Rubber bands are placed under the wooden board (with a pressure of 0.150 bar), which creates the effect of standing on springs. The movements of each of the participants are measured by accelerometers (MMA7260Q low cost capac-itive micro-machined accelerometers) at two different locations. One accelerometer is placed inside a silent shaker, which participants hold in the hand, the other one is attached to the knee. Headphones are used (type Sennheiser) to play music and to avoid any other auditory stimulus. The rhythmic movements are compared during different conditions: coupled and decoupled platforms, spontaneous and instructed synchronisation. The effects of performing an additional collaborative task and of start-ing the movements at non-equal tempi, are investigated. The data are analysed usstart-ing cross wavelet and synchrosqueezed transforms and joint RQA. Results are presented in chapter 3. Applications are possible in the field of music education and performances. Learning tools for rhythmic activities in music or dance could accommodate vestibular communication.33_{The use of weakly coupled platforms}

for musicians or dancers could help coordination and synchronisation. Other domains are sports, re-habilitation, music therapy, virtual or augmented reality and well-being.34 _{Vestibular stimuli can be}

deployed in exercises involving gait, balance, speed and repetitive movements in general.

33_{Yian Zhang et al., "Adaptive multimodal music learning via interactive haptic instrument," in 19th international}

conference on new interfaces for musical expression, NIME 2019, Porto Alegre, Brazil, June 3-6, 2019 : 140 - 5 [Zhang 2019].

34_{Maes, "3Mo?: a model for music-based biofeedback," [Maes 2016] and Shashank Ghai and Ishan Ghai, "Effects of}

(music-based) rhythmic auditory cueing training on gait and posture poststroke: a systematic review & dose-response meta-analysis," Scientific Reports 9 (2019): 2183 [Ghai 2019].

(21)

1.3 Selected analysis methods

1.3.1 Recurrence quantification analysis

Dynamical systems theory is an appropriate way to look at social interaction with music. It is an inter-disciplinary approach found in fields as diverse as physics, architecture, biology, chemistry, psychology and sociology. Walton et al. apply non-linear methods of dynamical systems to quantify the coordi-nation that emerges between improvising musicians.35 _{There are infinite ways musicians can respond}

to each other in the pitches, chords, and timbre afforded by their instrument, still these responses are situated within patterns of synchronisation and coordination. We will apply the same method to the data from the Soundbike and the Pendulum experiments to deduce information about synchronisation and stability.

One of the aims of dynamical modelling is to effectively capture the attractors of a system. An attractor is a state or subset of states toward which the dynamical system moves over time. It thus corresponds to a final future state or set of states. Also important is to find changes in the state of a system, as for example transitions between good or bad interaction of the performers. A pow-erful theorem developed by the Dutch mathematician Floris Takens makes attractor reconstruction possible, even when not all underlying parameters are known and/or measurable. It allows to recover the dynamics of a (chaotic) system from the observation of just one variable.36 _{Takens’s theorem is}

valid for so-called ’strange attractors’, characteristic of chaotic (although not exclusively) systems. A strange attractor is a complex subset of states within a systems phase-space that the state variable(s) of the dynamical system evolve toward over time. The term ’complex’ refers to the fact that strange attractors have a non-integer or fractal dimension.37

An interesting way to visualise the data from dynamical systems are recurrence plots (RP). A quan-titative analysis of the recurrence maps makes it possible to determine the behaviour in an objective and accurate way. The principles of Recurrence Quantification Analysis (RQA) are summarised by Richardson.38 _{The set of all possible states of a dynamical system is called the phase-space. Each}

state corresponds to a unique point in phase-space. Essentially, recurrence analysis identifies the dy-namics of a system by discerning (i) whether the states of the system’s behaviour recur over time and, (ii) if states are recurrent over time, the degree to which the patterns of recurrences are highly regular or repetitive (i.e. deterministic). Conceptually, performing recurrence analysis on behavioural data is relatively easy to understand; one simply plots whether the recorded points in a time series or behavioural trajectory, are revisited over time on a two-dimensional plot. This recurrence plot provides

35_{Ashley E. Walton et al., "Creating time: social collaboration in music improvisation," Topics in Cognitive Science}

10 (2018): 95 - 119 [Walton 2018].

36_{Floris Takens, "Detecting strange attractors in turbulence," in Dynamical systems and turbulence, Warwick 1980.}

Lecture notes in mathematics, Vol 898, eds. David Rand and Lai-Sang Young. (Berlin, Heidelberg: Springer, 1981) [Takens 1981].

37_{Richardson, "Complex dynamical systems in social and personality psychology: theory, modeling and analysis,"}

[Richardson 2014].

38_{Michael J. Richardson, Rick Dale and Kerry L. Marsh, "Complex dynamical systems in social and personality}

psychology: theory, modeling and analysis," in Handbook of research methods in social and personality psychology, eds. Harry T. Reis and Charles M. Judd. (Cambridge: Cambridge University Press, 2014), 251 - 80 [Richardson 2014].

(22)

Figure 1.4: Recurrence plots of (A) a periodic motion with one frequency, (B) the chaotic Rössler system and (C) of uniformly distributed noise [Marwan 2007]. The periodic signal is deterministic with large diagonal lines. The noise is uncorrelated

with many single recurrence points. The chaotic Rössler system falls in between.

a visualisation of the patterns of revisitations in a system’s behavioural state-space and can be quan-tified in various ways. The mathematical background is clearly described in the papers by Marwan and Wallot.39_{A summary of the interesting output parameters for human behaviour to music can be}

found in my bachelor’s thesis and in a tutorial by Goswami.40_{The quantities that are relevant for our}

two experiments, in particular measures that characterise synchronisation and determinism, will be repeated below.41 _{Their physical meaning and interpretation in the context of the specific research}

questions is further explained in the papers.

Figure 1.4 illustrates the general characteristic of RPs. It is reproduced from the tutorial paper on RQA by Marwan.42_{It shows the RP of three prototypical systems: a periodic motion on a circle (Fig. 1.4 A),}

a chaotic Rössler system (Fig. 1.4 B), and uniformly distributed, independent noise (Fig. 1.4 C). In all systems recurrences can be observed, but the patterns of the plots are rather different. The periodic motion is reflected by long and non-interrupted diagonals. The vertical distance between these lines corresponds to the period of the oscillation. The chaotic Rössler system also leads to diagonals which are seemingly shorter. There are also certain vertical distances, which are not as regular as in the case of the periodic motion. However, on the upper right, there is a small rectangular patch which rather looks like the RP of the periodic motion. The RP of the uncorrelated stochastic noise signal consists of many single black points. The distribution of the points in this RP looks rather erratic. Reconsidering all three cases, we might conjecture that the shorter the diagonals in the RP, the less predictable the system. The diagonal lines in the RP are related to the predictability of the system.

39_{Norbert Marwan et al., "Recurrence plots for the analysis of complex systems," Physics Reports 438 (2007): 237}

-329 [Marwan 2007] and Sebastian Wallot, "Recurrence Quantification Analysis of processes and products of discourse: a tutorial in R," Discourse Processes 54 (2017): 382 - 405 [Wallot 2017].

40_{Bedartha Goswami, "A brief introduction to nonlinear time series analysis and recurrence plots," Vibration 2 (2019):}

332 - 68 [Goswami 2019].

41_{Crombé, "Wavelet transform and Recurrence Quantification Analysis for embodied music interaction. Exploration}

of the methods," 24 - 5 [Crombé 2019].

(23)

This very first visual inspection indicates that the structures found in RPs are closely linked to the dynamics of the underlying system.

Joint Recurrence Quantification Analysis (JRQA) is a bivariate extension of RQA. A Joint Recur-rence Plot (JRP) reveals the relationship between variables of two systems. First, the recurRecur-rences of the trajectories in the respective phase-space of the two different systems are calculated separately. Then, both recurrence plots are joint together: the times when both systems have simultaneous re-currences are indicated, while the instances of recurrence that are different between the two plots are discarded. In that way the individual phase-spaces of both systems are preserved.43 _{Another bivariate}

extension of RQA is the Cross RQA (CRQA). The difference with JRQA is that in the case of a CRQA the different time series are forced onto one single phase-space. The CRQA measures are thus based on the distances between profiles in the same phase-space. The choice for JRQA over CRQA in our experiments is motivated by the consideration that the two participants in the Pendulum experiment and the pedalling and music time series in the SoundBike experiment, all have their own charac-teristics. There is no reason to assume one single phase-space can be used. Moreover the research questions are about synchronisation, for which joint recurrences are recommended by Marwan et al.:

We can state that CRPs are more appropriate to investigate relationships between the parts of the same system which have been subjected to different physical or mechanical processes [. . .] On the other hand, JRPs are more appropriate for the investigation of two interacting systems which influence each other, and hence, adapt to each other, e.g., in the framework of phase and generalised synchronisation.44

The following quantities are used for the analyses: Recurrence rate

The simplest measure of the RQA is the recurrence rate (RR). The recurrence rate for a JRP with a certain threshold T, is defined as:

RR(T) = 1 N2 N

∑

i,j=1 JRP_ij(T), (1.1)

which is a measure of the density of recurrence points in the JRP. N is the number of states in the JRP. It corresponds to the number of points in the phase-space, which was reconstructed from the embedding of the given time series. The maximum number for N is thus n− (D−1)τ, with n the number of data points, D the embedding parameter and τ the delay time. N also corresponds to the length of the axes of the JRP. The choices made for the embedding, delay and threshold parameters for our experiments are given in the papers. The effect of changing these parameters is discussed in chapter 4. Overall it has been found that the results are quite robust across a large range of parameter selections.

43_{Marwan, "Recurrence plots for the analysis of complex systems" [Marwan 2007].} 44_{Marwan, "Recurrence plots for the analysis of complex systems", 263 [Marwan 2007].}

(24)

Connectedness of recurrence points: DET, ADL, MDL

The next three measures capture how much the recurrence points are connected. They carry infor-mation on the stability of the system and the duration of a joint state.

• Determinism (DET)

The parameter determinism (DET) quantifies how many of the individual repetitions occur in connected trajectories. Low values are associated to stochasticity, high values suggest deter-ministic behaviour. In practice a minimal length lmin needs to be set to find the diagonal line

structures. DET= ∑ N l=lminlP(l) ∑N l=1lP(l) (1.2) In other words, DET measures the proportion of recurrent points forming diagonal line structures with minimal length lmin.

P(l)is the histogram of diagonal lines of length l:

P(l) = N

∑

i,j=1 1−JRP_i−1,j−1 1−JRP_i+1,j+1 l−1

∏

k=0 JRP_i+k,j+k. (1.3)

Periodic signals (e.g. sine waves) will give very long diagonal lines, chaotic signals will give very short diagonal lines, and stochastic signals will give no diagonal lines at all.

• Average Diagonal line Length (ADL) or MEANLINE

The Average Diagonal line Length (ADL) is the average length of sequences of recurrent points that form lines parallel to the main diagonal. It is the average time that two segments of the trajectory are close to each other and quantifies how long sequences of joint recurrence persist. It can be interpreted as the mean prediction time. An alternative name is MEANLINE.

ADL= ∑ N l=lminlP(l) ∑N l=1P(l) (1.4) • Maximum Diagonal line Length (MDL) or MAXLINE or Lmax

The Maximum Diagonal line Length (MDL) is the longest line segment measured parallel to the main diagonal. The shorter the MDL the more chaotic, and thus less stable, the joint signal is. Alternative names are MAXLINE or LINEMAX (LMAX, Lmax).

MDL= Lmax=max({li}_iN=l1) (1.5)

where Nl is the total number of diagonal lines with length l larger than lmin:

Nl =

∑

l≥lmin

P(l). (1.6)

The differences between the three quantities are rather subtle. A high DET, but low ADL and MDL, indicates a high, and relatively homogenous, correlation between the time series. A high ADL, but

(25)

comparatively low MDL, could be the sign of bursting behaviour, with certain periods of stable interaction. When MDL is high as well, performance might exhibit a long lasting settling down instead of multiple, equally distributed patterns of joint activity.45

1.3.2 Wavelet transform

As already mentioned above, signals of human motor behaviour are often complex and non-stationary. Algorithms in our brain continuously predict aspects of the world around us and steer our actions and even our perceptions.46 _{Improvisation and adaptation are key skills of humans to act in the}

environment. It is therefore obvious that the frequency content of measured signals will evolve over time. Anticipating the behaviour of others is also crucial during musical interaction and especially for improvisation sessions. It is challenging to study the time and frequency evolution of such signals, even more so if interpersonal interactions are taken into account, as for example the coordination between performers. Walton et al. look at spontaneous coordination of multiple musical bodies by measuring spatiotemporal patterns at various levels.47 _{Self-organisation of a group of performers is defined as}

the moment that their actions become so intertwined that the group behaves as a single synergistic system. Wavelet transforms of the measured signals are used to find common periodicities in nested time scales and local micro-scale structures (a note or a bar) within global macro-scale patterns (a musical sentence or a piece). Hand or finger movements can be compared to oscillatory motions of other limbs or even the full body sway. The strength of the interaction and the relative phase are derived from a cross wavelet analysis. The same techniques have been applied to the Pendulum experiment in chapter 3. For clarity it is useful to repeat the equations for the most relevant cross wavelet parameters, coherence and phase, from my bachelor’s thesis.48

Continuous wavelet transform

The technique of wavelet transform (WT) originates from geology and is fundamentally an extension of a Fourier transform with adaptable time resolution.49 _{Effectively predicting earthquakes requires}

the combination of many non-stationary phenomena at wide frequency ranges. The system is highly non-linear, parameters are (weakly to strongly) coupled. Changes in the state of this higher order system need to be detected. Since the 1980s the application domain of wavelets has broadened enormously including climatology, biology, neuroscience and psychology. Recent studies about human behaviour also touch upon the field of systematic musicology, investigating moreover body movement,

45_{Wallot, "Recurrence Quantification Analysis of processes and products of discourse: a tutorial in R," 401 [Wallot}

2017].

46_{Pieter-Jan Maes et al., "Action-based effects on music perception," Frontiers in Psychology 4 (2014): 1 - 14 [Maes}

2014].

47_{Ashley E. Walton et al., "Improvisation and the self-organization of multiple musical bodies," Frontiers in}

Psychol-ogy 6 (2015): 313 [Walton 2015].

48_{Crombé, "Wavelet transform and Recurrence Quantification Analysis for embodied music interaction. Exploration}

of the methods," 9 - 11 [Crombé 2019].

49_{Jean Morlet, "Sampling theory and wave propagation," in Issues in acoustic signal - Image processing and}

recog-nition. NATO ASI Series (Series F: Computer and System Sciences), ed. Chi Hau Chen. (Berlin, Heidelberg: Springer, 1983)) [Morlet 1983].

(26)

synchronisation and interaction in the context of musical performances and improvisation.50 _In

gen-eral the wavelet decomposition allows for an accurate time localisation of short-time (high-frequency) events and an accurate frequency localisation of long-term (low-frequency) components, which is an advantage over a Fourier transform. A Continuous Wavelet Transform (CWT) decomposes a signal (typically a time series), x(t), into a number of translated and dilated wavelets. First a mother wavelet

has to be chosen that resembles roughly the behaviour of the signal. It is then translated (in time) and contracted and dilated (in scale s, related to frequency) to fit the original signal. An example of a complex Morlet wavelet, suited for musical performance data, is shown in figure 1.5. It can be

Figure 1.5: Morlet wavelet with centre frequency fc=1.27 Hz (ω0 = 8 rad) and

positive bandwidth parameter fb = 0.75 Hz.

seen from the real and imaginary parts that the Morlet wavelet is a combination of a sine wave and a Gaussian function. It is therefore appropriate for complex dynamic time signals containing different frequencies, typical for human behaviour. Other wavelet families have been developed, and can be used as well. The Morlet wavelet however is widely applied, it is a relatively simple function and well capable of reproducing periodic behaviour. The periodicity will show up as a pattern spanning all translations at a given dilation (it will be seen as horizontal lines on a time-frequency plot). The wavelet transform preserves temporal locality, which is an advantage over Fourier analysis. A com-pact and comprehensible description of the mathematical background is given in Maraun et al. and Torrence et al.51

50_{Walton,"Improvisation and the self- organization of multiple musical bodies," [Walton 2015] and Johann Issartel,}

Mathieu Gueugnon and Ludovic Marin, "Understanding the Impact of Expertise in Joint and Solo-Improvisation," Frontiers in Psychology 8 (2017):1078 [Issartel 2017].

51_{Douglas Maraun and Juergen Kurths, "Cross wavelet analysis: significance testing and pitfalls," Nonlinear Processes}

in Geophysics 11 (2004): 505 - 14 [Maraun 2004], and Christopher Torrence and Gilbert P. Compo, "A practical guide to wavelet analysis," Bulletin of the American Meteorological Society 79 (1998): 61 - 78 [Torrence 1998].

(27)

For a complex wavelet function the wavelet transform of a time series, x(t), will also be complex. The

wavelet component at a given time ti and scale s can be divided in a real part,<(Wi(s)), and an

imag-inary part,=(Wi(s)). The amplitude is given by|Wi(s)|and the phase is tan−1(=(Wi(s)/<(Wi(s)).

The Wavelet Power Spectrum (WPS) is |Wi(s)|2, also called the wavelet spectrum. It describes how

much power of the signal goes to the wavelet with scale s at time ti and is directly related to the

temporal evolution of the power distribution over the different frequencies. A plot of the WPS is called a scalogram. It is the wavelet equivalent of the spectrogram of a windowed Fourier transform. Cross wavelet transform

For the Pendulum experiment the comparison of the participants’ movements reveal underlying cou-plings and interactions of the duos. When looking at two time signals a bivariate extension of wavelet analysis is required, which is called cross wavelet transform (XWT). It can be seen as the wavelet version of a windowed cross-correlation function.52

Using the formulation of Maraun et al. and Torrence et al. the univariate Wavelet Power Spectrum (WPS) can be extended to compare two time series x(t) and y(t).53 The Wavelet Cross Spectrum

WCSi(s) at time ti is defined as the product of the corresponding wavelet transforms W_ix(s) and

W_iy(s): WCSi(s) =Wix(s)W y i (s) ? , (1.7)

where s is the scale and Wy

i(s)? is the complex conjugate of the WT of y(ti)at scale s. The WCS is a

complex value (while the WPS is real), analogue to the Fourier cross spectrum. It can be decomposed into amplitude and phase:

WCSi(s) = |WCSi(s)|exp(iΦi(s)). (1.8)

The amplitude |WCSi(s)| is the cross wavelet power, it reveals the local covariance between the

time series and indicates the areas where the time series have a high common power. The phase Φi(s) = tan−1(=(WCSi(s)/<(WCSi(s)) describes the delay between the two signals at time ti

on a scale s. The phase is indicated by arrows on the wavelet coherence plots. Zero phase difference means that the examined time series move together at a particular scale s. Arrows are pointing to the right (left) when the time series are in-phase (anti-phase), i.e. positively (negatively) correlated. An arrow pointing up means that the first time series leads the second one by 90◦_{, an arrow pointing} down indicates that the second time series leads the first one by 90◦_.

As a warning for the interpretation Maraun notes that peaks in the WCS can even appear when x(ti) and y(ti) are independent, because WCS is not normalised.54 It can thus produce misleading

results, for instance, if one of the spectra is locally flat and the other exhibits strong peaks, then also

52_{Steven M. Boker et al., "Windowed cross-correlation and peak picking for the analysis of variability in the association}

between behavioral time series," Psychological Methods 7 (2002): 338 - 55 [Boker 2002].

53_{Maraun, "Cross wavelet analysis: significance testing and pitfalls" [Maraun 2004], and Torrence, "A practical guide}

to wavelet analysis" [Torrence 1998].

(28)

large variations in the cross spectrum will be produced, which may have nothing to do with any corre-lation of the two time series. Therefore WCS is not suitable for significance testing of the recorre-lationship between two time series. The problem can be avoided by looking at the Wavelet COherence (WCO):

WCOi(s) =

|WCSi(s)|

q

WPSx_i(s)WPS_iy(s)

, (1.9)

which is in fact the cross power normalised to both power spectra. 0 ≤ WCOi(s) ≤ 1: a value of

1 means a linear relationship between x(t)and y(t) around time t_i on a scale s. A value of zero is

obtained for vanishing correlation. If a significant interrelation is detected using WCO, then WCS can be used to estimate the phase spectrum.

Figure 1.6: Signals from the accelerometers in the shakers for one of the trials of the Pendulum experiment (duo 12, signals from the shakers, coupled platforms, identical

starting tempi, no additional task).

For one of the trials in the Pendulum experiment (duo 12, signals from the shakers, coupled platforms, identical starting tempi, no additional task) the time series of the accelerometers in the shakers are given in figure 1.6. Continuous and cross wavelet transforms are performed. The individual wavelet power spectra and the coherence and phase between the two signals are presented in figure 1.7. The WPS show the frequencies that are present in the movements of the individual participants. The

(29)

interaction is characterised by the cross wavelet parameters. The strongest interaction is seen for frequencies around 3 - 3.5 Hz, which corresponds to the frequency of the music (see also chapter 3).

Figure 1.7: Wavelet information for the same trials as fig. 1.6. Left: WPS for par-ticipant 1; middle: WPS for parpar-ticipant 2; right: wavelet coherence and relative phase,

indicated by arrows.

Synchrosqueezed wavelet transform

In addition, in the paper, the dominant movement frequency of both participants is calculated using a so-called synchrosqueezed wavelet transform, which is a combination of a wavelet transform and a reallocation method. The method aims to sharpen the time-frequency representation by allocating its value to a different point in the time-frequency plane, determined by the local behaviour. Mathe-matical details are rather complex and elaborated in Daubechies et al.55 _{First a continuous wavelet}

transform is performed, which decomposes the signal into its components, using translated and scaled versions of a mother wavelet as its ’time-frequency atoms’. However, some time-frequency spread-ing is always associated with these time-frequency atoms, which affects the sharpness of the signal analysis. Moreover the family of chosen template functions unavoidably ’colours’ the representation, and can influence the interpretation. In addition, Heisenberg uncertainty principle limits the resolution that can be attained in the time-frequency plane; different trade-offs can be achieved by the choice of wavelets, but none is ideal. The time-frequency limitations of a wavelet transform are further discussed in chapter 4.

Yet the wavelet synchrosqueezed transform is a time-frequency method that reassigns the signal energy in frequency. This reassignment compensates for the spreading effects caused by the mother wavelet. Unlike other time-frequency reassignment methods, synchrosqueezing reassigns the energy only in the frequency direction, which preserves the time resolution of the signal. By preserving the time, the inverse synchrosqueezing algorithm can reconstruct an accurate representation of the original signal. To use synchrosqueezing, each term in the summed components signal expression must be an intrinsic mode type (IMT) function. Daubechies et al. prove that the method can be applied to arbitrary functions. Each IMF is an amplitude modulated - frequency modulated (AM - FM) signal. After the decomposition of the signal into its IMF components, an Empirical Mode Decomposition (EMD) algorithm proceeds to the computation of the instantaneous frequency of each component. The resulting transformed signal has reduced energy smearing compared to a continuous wavelet

55_{Ingrid Daubechies, Jianfeng Lu and Hau-Tieng Wu, "Synchrosqueezed wavelet transforms: an empirical mode}

(30)

transform. The energy is ’squeezed’ into dominant frequencies. The time resolution remains the same (hence the prefix: ’synchro’). A comparison between the continuous wavelet and the synchrosqueezed transforms for a harmonic signal is given in figure 1.8.

Figure 1.8: Left: a harmonic signal x(t) =sin(8t); middle: the continuous wavelet transform of x; right: the synchrosqueezed transform of x [Daubechies 2011].

The corresponding synchrosqueezed transforms of the two participants in the example of figure 1.7 are depicted in figure 1.9.

Figure 1.9: Synchrosqueezed wavelet transforms of the participants’ shaker signals for the same trial as fig.1.6.

(31)

2 Analysis of the SoundBike experiment

The role of musical and cycling background on spontaneous and

instructed synchronisation of the pedal cadence to external music

using sonification

Kristel Crombé1_{, Alexander P. Demos}2 _{and Pieter-Jan Maes}1

1_{IPEM, Department of Art, Music and Theatre Sciences, Ghent University,}

Miriam Makebaplein 1, 9000 Ghent, Belgium

2_{Department of Psychology, University of Illinois at Chicago,}

1007W, Harrison St., Chicago, IL 60607-7137, USA

Abstract

Many people spontaneously synchronise their movements to music. Applications for tempo adaptation of simple repetitive movements have been successfully developed based on this principle. The present study focuses on one specific activity, cycling. Cyclists’ motor rhythm is made audible (i.e. sonified) in order to increase participants’ tendency to synchronise to an external musical rhythm. For that purpose a dedicated set-up has been developed, the SoundBike. It consists of a stationary bike, equipped with sensors, that permits an interactive sonification of the cyclists’ motor rhythm. Different sonification methods are investigated and compared to two control conditions: a reference (’chance’) condition, where synchronisation happens by coincidence, and a spontaneous synchronisation condition, where external music is played without an audible pedal frequency. The data are analysed using joint recur-rence quantification analysis. Results of the experiment suggest that especially people with a musical background benefit from a sonification of the cycling rhythm. It enhances their tendency to synchro-nise to the external music and it also helps to keep a more stable pedal cadence. Moreover adding an additional musical layer as reward to indicate when synchronisation is established, is advantageous to them. When explicit attention is given to the music, a significant improvement in synchronisation strength and pedal cadence stability is observed in general, also for experienced cyclists without mu-sical background. The SoundBike seems therefore a promising interactive sonification device to assist motor learning and motor adaptation in the field of sports and physical rehabilitation.