The function of head nods
Concerning positive and negative feedback
Bas Gerding Master thesis Information Science Bas Gerding S3140555 September 15, 2020
A B S T R A C T
Head gestures are known to play an important role in face-to-face communication. Mapping the form and function of head gestures is useful for the understanding of human communication. This research examines the functions of these differ-ent head gestures by modifying them during a collaborative task and comparing them before and after this modification. The head movements of the participants were either amplified vertically or horizontally and these manipulations were pos-sible with the use of virtual reality. The head movements hold their functions in the grounding of communication and this was investigated in the thesis. A total of 74 people participated, resulting in 37 experiments in which head movements were simulated with the use of a computer mouse. Two research questions were asked: "What effect does the amplification of head nods and head shakes have on game performance?" and "What is the impact of removing the amplifications dur-ing the task?". Several hypotheses were tested and it emerged that the participants adapted quickly to the amplified motions. The game performance was better in those conditions than in the baseline. Within the conditions it showed that after the amplifications were removed, the participants performed worse although this was not significantly confirmed.
C O N T E N T S
Abstract i Preface iii 1 introduction 1 2 background 2 2.1 Coordination in dialogue . . . 2 2.1.1 Grounding . . . 22.1.2 Emergence of referring conventions . . . 2
2.2 The form and function of head movements . . . 4
2.2.1 Head nods and head shakes . . . 4
2.3 Nonverbal studies with virtual reality . . . 4
3 research questions 5 4 method 6 4.1 Materials . . . 6
4.2 Task . . . 7
4.3 Agent Behavior Manipulations . . . 7
4.4 Participants . . . 8 4.5 Behavioral Measures . . . 8 4.5.1 Game score . . . 8 4.5.2 Sentiment scores . . . 9 4.5.3 Character length . . . 9 4.6 Questionnaire . . . 9 5 hypotheses 10 6 results 11 6.1 Hypothesis 1 . . . 11 6.2 Hypothesis 2 . . . 12 6.3 Hypothesis 3 . . . 12 6.4 Hypothesis 4 . . . 13 6.5 Hypothesis 5 . . . 13 6.6 Hypothesis 6 . . . 14 6.7 Hypothesis 7 . . . 15 6.8 Hypothesis 8 . . . 16 6.9 Questionnaire . . . 17 7 conclusion 18 Appendices 22 a formulas 23 b informed consent 24 ii
P R E F A C E
In the context of my Master Information Science, I have looked into the function of head nods as a non-verbal communication act in human conversation. Part of this thesis was the creation of a game inside the virtual reality world in which the participants had to detect if they were looking at the same image or not. With the use of virtual reality, I was able to register and modify the head movements of a participant and use this data in the analysis. With this topic and the use of virtual reality, I was trying to diverge from the subjects that we got accustomed to in the Bachelor and Master of Information Science (e.g. text classification with the use of machine learning) and dive into a new growing area of research.
This thesis was written during the 2019–20 coronavirus pandemic which made it difficult to meet with my supervisor and impossible to implement these experi-ments with the participants in a physical room. With the use of different means of online communication (e.g. Blackboard Collaborate and WhatsApp), I was still able to carry out a number of these experiments to conclude the thesis.
I want to thank Gregory Mills for being a supportive supervisor in these dif-ficult, and weird times. He was always available for a brainstorming session and could look at the issues at hand from a different angle. I also wish to thank all of the people that participated in the experiments, without whose cooperation I would not have been able to conduct this analysis.
I hope you enjoy reading it. Bas Gerding
Groningen, September 15, 2020
1
I N T R O D U C T I O N
Head gestures are known to play an important role in face-to-face communication (Hadar et al.,1983;McClave,2000). Mapping the form and function of head gestures
is useful for the understanding of human communication. In the past, head gesture annotation has shown to be too expensive because it required frame-by-frame anal-ysis of video data (Allwood and Cerrato,2003;Poggi et al.,2010). Nowadays, this
limitation can be overcome with the use of virtual reality. Virtual reality enables us to measure and manipulate different aspects of the human head movements and with high accuracy.
Several researchers have looked at the function of head nods and head shakes, and if or when they differ in the application during a conversation (Hadar et al.,
1985; McClave, 2000; Kousidis et al., 2013). The goal of this thesis is to include
the grounding model into the analysis. The grounding model developed byClark and Brennan (1991) states that to accomplish a successful conversation, the two
participants must keep track of their common ground. In communication, common ground cannot be properly updated without a process they call grounding (Clark and Schaefer,1987,1989;Isaacs and Clark,1987;Clark and Wilkes-Gibbs,1986). In
conversation, participants try to establish that was has been said, has also been understood. In the terminology of Clark and Brennan(1991), they try to ground
what has been said and make it part of their common ground.
To accept utterances from the conversational partner and show that they have understood the utterance, positive evidence of understanding needs to be given by the participant. There are many different types of positive evidence, but one of the most important ones are head nods. Head shakes are often used as a component of a negative expression (Kendon,2002).
With the use of virtual reality in the web browser, a game is created in which the head movements of the participants are either amplified or attenuated. The effects of these conditions are then analyzed by looking at the game performance from the participants along with several other variables.
Two research questions have been created to answer the research goals of this thesis. These research questions are stated here but will be repeated in Chapter
3. The first question goes as follows: “What effect does the amplification of head
nods and head shakes have on game performance?”. The second question denotes: “What is the impact of removing the amplifications during the task?”
Chapter 2 contains a review of the relevant literature. As mentioned before,
the research questions are noted in Chapter 3 and the methods used in the study
are then described in Chapter4. The hypotheses corresponding with the research
questions are written down in Chapter 5 and the results from these hypotheses
are presented and discussed in Chapter 6. Finally, Chapter 7 outlines the main
conclusions and identifies both limitations to the study and recommendations for further research.
2
B A C K G R O U N D
2.1
coordination in dialogue
2.1.1 Grounding
Clark and Brennan (1991) explain that most contributions to a conversation begin
with the contributor presenting an utterance to his or her partner. The contributor needs to present this utterance because he or she wants his or her conversation part-ner to hear, register, and understand it. The contributor cannot know whether he or she has succeeded unless the partner provides evidence of positive understand-ing. One might assume that all we need to look for in a conversation is negative evidence, the evidence that we might have been misheard or misunderstood. If we could discover this problem we could make adjustments, but we don’t. As humans, we by default assume that we have been understood by our partner. If we would only seek negative evidence, we would often accept information that we had little justification for accepting. As the contribution model says: people ultimately seek positive evidence of understanding.
The grounding model includes grounding states with five levels of understand-ing in a conversation. In state 0, the conversation partner did not notice anythunderstand-ing. In state 1, the partner noticed that the contributor uttered something. In state 2, the partner correctly heard the utterance but did not fully understand it. In state 3, the partner understands the utterance, and in state 4 responds to the utterance. The conversation shifts through these stages, in which the contributor strives for stage 4.
Acknowledgments are the most obvious form of positive evidence. With ac-knowledgments, we mean most of what has been called back-channel responses. These include continuers such as uh-huh, yeah, and the British m (Schegloff,1982).
These types of continuers are used by the partners to signal that they think they have understood the turn so far. Head nods have much the same force as these con-tinuers (Goodwin, 1981) and in many cultures resemble positive evidence. Head
shakes can be seen as negative evidence since they are each other’s opposites. 2.1.2 Emergence of referring conventions
In conversation, also as part of the grounding model, speakers and addressees work together in the making of a definite reference. Clark and Wilkes-Gibbs (1986)
pro-posed a model in which they suggest that the speaker initiates the process by pre-senting or inviting a noun phrase. Before going on to the next contribution, the participants, if necessary, repair, expand on, or replace the noun phrase in an itera-tive process until they reach a version they mutually accept. In doing so they try to minimize their joint effort, striving for least collaborative effort (Clark and Brennan,
1991).
Schober and Clark (1989) questioned two different views on how people
un-derstand each other in conversations. The traditional view, which they called au-tonomous view, argued that the partners listened to the words uttered, decoded them, and interpreted them against what they take to be the common ground. A different view, called the collaborative view, explains that speakers and their ad-dresses go beyond these autonomous actions and collaborate moment by moment to try to ensure that what has been said is also understood.
2.1 coordination in dialogue 3
They looked at triples of people in which one person told another person in conversation how to arrange 12 complex tangram figures while an overhearer tried to arrange them too. Each of the participants began as strangers with the same background information and thus no common ground. The first time that figures appeared, the directors described them, but from then on they referred to them with definite descriptions, which got shorter and shorter as the experiment progressed. For example, in the first trial where one pair saw a figure for the first time, this was their exchange:
• D: Then number 12 . is (laughs) looks like a, a dancer or something really weird. Urn . and, has a square head . and urn, there’s like, there’s uh- the kinda this urn .
• M: Which way is the head tilted?
• D: The head is . eh- towards the left, and then th- an arm could be like up towards the right?
• M: Mm-hm.
• D: *And . It’s- *
• M: *an- . a big* fat leg? *You know that one?* • D: *Yeah, a big* fat leg.
• M: and a little leg. • D: Right.
• M: Okay.
• D: Okay?
• M: Yeah.
This exchange resulted in a high frequency of negative evidence of understand-ing. By the last trial, their exchange got significantly shorter:
• D: Urn, 12 . the dancer with the big fat leg? • M: Okay.
One of the figures was variously referred to as “the rice bag,” “the whale,” “the complacent one, ” “the stretched-out stop sign,” and “the baby in a straitjacket”, showing the emergence of referring conventions.
In their second experiment, the task was the same, except this time an overhearer was present in the same room separated by visual barriers. The research concluded that it did not help that the overhearers could listen to the conversations live. They still did not perform as well as the matchers, providing more evidence against the autonomous view of understanding in conversation.
One could say that coordination in dialogue is a difficult process to explain. Some questions arise from these papers. The collaborative view of communication showed that people modify the meanings of their words to their partner via inter-action but what about the meanings of non-verbal signals such as head nods and head shakes? Do participants adapt the meaning of these signals too?
2.2 the form and function of head movements 4
2.2
the form and function of head movements
2.2.1 Head nods and head shakes
As previously mentioned, head movements, primarily nodding and shaking are a major mode of communication in the back-channel during listening turns ( Rosen-feld,1978). The most obvious meaning is signaling ‘yes’ and ‘no’ (Jakobson,1972),
but they can also signal interest or impatience (Birdwhistell,1970), enhance
com-municative attention (Condon and Ogston, 1967; Kendon, 1970), or anticipate an
attempt to capture the floor (Duncan,1972). Their uses could differ from specific
properties of movement such as tempo, intensity, and configuration. It could be that when signaling ‘yes’, a cyclic and of ordinary tempo nod is used, or when signal-ing impatience it would result in a rapid and sharp nod. In their research, Hadar et al.(1985) monitored head movements during conversations, specifically the head
nods and head shakes to examine these different types of functions. The horizontal distance in mm between two extremums e1, and e2 was denoted by H(e1, e2), and their vertical separation by V(e1,e2). Each triad of successive extremums e1, e2, e3, created a cycle in the chart. A cycle was ascribed to a linear movement if V(e1, e2) > 3V(e2, e3).
Several properties of the listeners’ head movements, such as amplitude and fre-quency differentiated in various conversational functions. Symmetrical and cyclic movements were employed to signal ‘yes’ or ‘no’. Linear and wide movements an-ticipated claims for speaking, for instance, turn-taking. Narrow linear movements occurred concurrently with stressed syllables in the other’s speech. Wide, linear movements occurred during pauses in the other’s speech.
McClave (2000) concluded that head shakes correlate with verbalizations
ex-pressing inclusivity, e.g. co-occurring with words such as ’everyone’ and ’every-thing’, intensification, e.g. co-occurring with lexical choices such as ’very’, ’a lot’, and ’great’, and uncertainty, e.g. co-occurring with sentences that resemble ’I guess’, ’I think’ or ’whatever’. Contrary to these findings, Boholm and Allwood (2010)
found that repeated head movements mainly function as feedback but there is no strong tendency for repetition in head movements to co-occur with repetition in speech or vice versa.
Looking at the different views on the function and forms of head nods and head shakes, we could ask ourselves whether participants adapt the meaning of their head nods and head shakes with each other. Virtual reality will be used to experimentally manipulate the behavior in order to see what effect it has on the interaction.
2.3
nonverbal studies with virtual reality
Virtual reality has made it possible to research aspects of different subjects that we would not be able to measure or change in the real world. Various studies can be found who have researched and experimented with nonverbal interaction in virtual reality (Blascovich et al., 2002; Bailenson et al., 2001;Healey et al., 2009). Virtual
reality has made it possible to experiment with controlled manipulations, which was challenging to introduce in face-to-face interaction studies. Immersive virtual environments enable control of all aspects of a participant’s non-verbal behavior (Bailenson et al.,2001), and provide researchers with access to all participant’s
mo-tion data, including visible movements, gaze, and gestures (Blascovich et al.,2002).
This allows the researchers to create artificial movements and actions that can’t be detected by either one of the entities located in the virtual environment. A partici-pant could stare to the sky on their screen, but on the screen of their partner to the ground.
3
R E S E A R C H Q U E S T I O N S
The proposed research will look at different aspects of communication, verbal as well as non-verbal during a game. Amplifying positive or negative feedback will break communication. If positive feedback is more important than negative feed-back, then the participants who get their head nods amplified more will perform worse. If negative feedback is more important than positive feedback then the par-ticipants will perform worse once their head shakes are amplified.
RQ1: What effect does the amplification of head nods and head shakes have on game performance?
If participants negotiate the meaning of head nods and head shakes and create conventions as mentioned by Schober and Clark(1989), then they should quickly
adapt to their partner in the amplification conditions. We would expect there to be no difference between the two. However, there should be an effect before and after the swap, as it will effectively be changing the meaning.
RQ2: What is the impact of removing the amplifications during the task?
The hypotheses corresponding with these research questions will be given in Chapter5.
4
M E T H O D
A quantitative method was used to experimentally investigate what factors in con-versation can influence human interaction. To address this research question and the hypotheses, a practical approach was used with the help of virtual reality to conduct an experiment in which participants performed a game while interacting with each other in a 3D space in which they had to work out if they saw the same image or not. The analysis was performed with the use of the tools Python and R. Python and its popular library pandas were used for the pre-processing of and calculations with the data before it was inserted into R for analysis. In R, the lme4 package was used to import and make use of the statistical tests.
4.1
materials
A total of 37 experiments were taken in a virtual environment in which two par-ticipants participated in a game. Both parpar-ticipants were invited to a group call with the instructor. The participants were instructed to open up the virtual envi-ronment in their web browser and open an additional tab with the online service from WhatsApp: WhatsApp Web. This way the participants could see the virtual environment and their group chat at the same time. After the instructions were given and both the participants agreed that everything was understood, the exper-iment started. The game was a referential task similar to the Tangram task used bySchober and Clark(1989), but where one participant had to communicate using
head movements.
The environment in which the participants played the game was created us-ing A-frame, a web framework for buildus-ing virtual reality experiences. A-frame is a Javascript library that is based on top of HTML that uses a powerful entity-component framework that provides a declarative, extensible, and composable struc-ture to three.js. This allowed for the creation of different components that could be applied to the entities in the virtual world. The tool Glitch was used for creating and hosting an online web application. The code of the webpage can be found through the following URL:https://glitch.com/~basgerding.
In this environment, each participant’s virtual head was positioned at a fixed location. The two participants were beforehand randomly selected as either player 1 or player 2. Player 1 used his or her mouse to simulate the rotational movement of their head. The mouse allowed for the 2 degrees of freedom (DoF), which was used for the generation, manipulation, and registration of the rotational movement around the x, and y axes. Player 2 was able to chat with player 1 through the online web-service of the chatting application WhatsApp.
A-frame does not support multi-user experiences out-of-the-box, so a template called networked a-frame created by Hayden Lee was selected and adapted to be used in the experiments. This template already took care of most of the commu-nication between the pages that had to run through a server.js-page. Through this server, the rotational data of player 1 was captured and stored in a .txt file.
The experiment was initially intended to be conducted in a physical room with the use of Oculus GO headsets, but due to the coronavirus outbreak, this was no longer a possibility.
4.2 task 7
4.2
task
The goal of the game was to determine if the two participants were looking at the same or a different image. The way the game worked was that, on each trial:
1. The players would see an image in front of them.
2. After 5 seconds the image would disappear and participants would see each other’s avatar.
3. Their goal was to work out together, as a team, whether they saw the same face as each other, or whether they saw a different face.
4. This meant that player 2 would have to describe the face to player 1 through text chat. Player 1 could not reply in WhatsApp, he or she could only use the head movements to communicate and answer the questions.
5. If the participants thought that they both had seen the same face, player 2 could type and send “SAME” in WhatsApp. If they thought they had seen different faces, player 2 could type and send “DIFFERENT” in WhatsApp. 6. After the choice, the game would load the next set of faces.
The perspective from the participants can be seen in Figure 1. It shows the
virtual environment, as well as the WhatsApp service: WhatsApp Web.
Figure 1: View from a participants perspective
4.3
agent behavior manipulations
Two experimental manipulations were performed: vertical amplification and hori-zontal amplification. These manipulations were created with the use of JavaScript. The formulas were generated and applied to the simulated head movements to create the two conditions. The attenuation formulas were applied to the baseline condition after 20 minutes and were used in one of the analyses. All of the formulas that were applied are listed in Table1. An additional function was implemented to
limit the x rotation from a minimum of 110 to a maximum of 250 degrees and the y rotation from a minimum of -70 to a maximum of 70 degrees. This limitation had to be implemented because otherwise player 1 was able to infinitely rotate 360 degrees around his or her axes, messing up the application of the formulas and allowing impossible head motions for humans. The graphs of these formulas can be found in the Appendix.
4.4 participants 8
Condition Formula
Vertical positive amplification y = 70x
1 2· 70 Vertical negative amplification y = -|x|70
1 2
· 70 Horizontal positive amplification y = x−18070 12· 70 + 180 Horizontal negative amplification y = -|x−180|70
1 2
· 70 + 180
Vertical positive attenuation y =x702
Vertical negative attenuation y =−x702
Horizontal positive attenuation y =(x−180)70 2+ 180 Horizontal negative attenuation y = -(|x−180|)70 2+ 180 Table 1: List of applied formulas to create the three different conditions
4.4
participants
The participants were recruited through Gregory Mills, a lecturer, and researcher at the University of Groningen. The experiments were conducted in May till the middle of June of the year 2020, at the same time that Mr. Mills taught the courses Social Media and Caleidoscoop. He could oblige his students to participate in the experiment for course credit, which resulted in a total of 74 participants and thus 37 experiments. All of the participants were students of either one of the courses. There were 13 male duos, 11 female duos, and 11 mixed duos resulting in a frequency of 39 males, and 35 females. A total of 5 groups chatted in English, the remaining 32 duos in Dutch. Before the experiments began the participants were asked to sign an informed consent, which they all signed.
4.5
behavioral measures
4.5.1 Game score
Each round of the experiment was registered with an output to the webpage con-sole which was saved after each experiment. For each game, the system recorded whether the selection was correct (e.g. correctly selecting same or correctly select-ing different) or incorrect (e.g. incorrectly selectselect-ing same or incorrectly selectselect-ing different). The script also registered when the swap in the conditions occurred. Timestamps were included in the output to provide clarity and support the anal-ysis. This output produced two numbers: the number of correct choices and the number of incorrect choices. Both of these numbers were used in the statistical analysis.
4.6 questionnaire 9
4.5.2 Sentiment scores
The chatting application WhatsApp was used by player 2 to communicate questions and statements about the game to player 1. WhatsApp has a built-in function that lets the user save and export the WhatsApp conversations with additional times-tamps, which were used to register the textual data. This data was used to provide exploratory statistics and sentiment analysis on the conversations. The positive and negative sentiment scores were generated using a text classification process created byBoot et al.(2017). This classification process returned sentiment scores for each
sentence in the group chat. Two different numbers were used for the statistical anal-ysis: the total amount of sentiment in the conversation, and the average number of sentiment in the conversation. Both of these numbers resulted in different tests that are shown in Chapter6.
Before the analysis, the gathered data was prepared. Information that could lead back to the users, e.g. names, from the WhatsApp conversation was anonymized and any additional non-textual data, except for emojis, were removed. The words similar to "SAME" or "DIFFERENT" were removed for the sentiment analysis. 4.5.3 Character length
The same textual chat data that were used to calculate sentiment scores were again used to calculate character length per round in the experiment. Python was used to split the conversation into rounds and then calculate the number of characters inside each round. The data was used as a response variable in one of the hypotheses tests that can be found in Chapter5.
4.6
questionnaire
In addition to the behavioral measures, a questionnaire was held amongst the par-ticipants to determine the effect of the manipulations on their perception of the interaction and to see if they detected the nature of the experimental interventions. The survey consisted of 2 Likert scale questions, 1 question that was measured on a scale of 0-10, and two open questions. A total of 84 responses were recorded and used for the analysis. The additional 10 responses to the questionnaire resulted from five experiments that were not used in the analysis due to technical difficulties but did manage to successfully finish the game. The following 5 questions were asked:
• How well do you think you understood your partner? • How well do you think your partner understood you?
• How would you grade the communication between you and your partner?
• Can you guess what the experiment is investigating? • What was the most difficult thing to communicate?
The last question was added on a later date while the experiments had already started, resulting in only 62 responses. The questionnaire was only used for ex-ploratory analysis, not for explanatory analysis.
5
H Y P O T H E S E S
As researched byClark and Brennan(1991), people seek for positive evidence in a
conversation. The thought behind the hypothesis was that if positive evidence is of great importance in conversation, the game performance would decrease if it was modified. The same thought applies to negative feedback. Which one of these types of feedback would be more important during the game? This resulted in the first two hypotheses:
• Hypothesis 1 (H1): Amplified vertical head nods will result in a worse total game performance compared to the baseline condition.
• Hypothesis 2 (H2): Amplified horizontal head nods will result in a worse total game performance compared to the baseline condition.
If it is the case that participants negotiate the meaning of head nods or head shakes, then they should quickly adapt to their partner in the amplified conditions. The first two hypotheses examine this, but there should be no difference between the two if they adapt to the different meanings. However, there should be an effect of the swap, as it will effectively be changing the meaning of the head nods and head shakes during the game. This resulted in the following three hypotheses:
• Hypothesis 3 (H3): The game performance after the vertical amplification is removed will be better than before the removal.
• Hypothesis 4 (H4): The game performance after the horizontal amplification is removed will be better than before the removal.
• Hypothesis 5 (H5): The game performance after the vertical and horizontal attenuation is added will be worse than before the addition.
Schober and Clark(1989) explain that speakers and their addresses collaborate
moment by moment to try to ensure that what has been said is also understood. They do this by creating definite descriptions, which get shorter and shorter as the experiment progresses. This discovery is tested in the last hypothesis. The first two hypotheses look at the sentiment in the text. Since head nods co-occur with positive sentimental words like ’yes’, and head shakes with negative sentimental words like ’no’, exaggerating the head movements of the participants may result in an increase
in these types of sentiment in the chat.
• Hypothesis 6 (H6): The amplification of vertical head movements will increase positive sentiment in the communication between participants compared to the baseline.
• Hypothesis 7 (H7): The amplification of horizontal head movements will in-crease negative sentiment in the communication between participants.
• Hypothesis 8 (H8): The character length per round inside the game will de-crease as the game continues.
6
R E S U L T S
This chapter will divide the previously mentioned hypotheses into different sec-tions. To test these hypotheses, the statistical language R was used in combination with different statistical tests such as (generalized) linear mixed-effects models and linear regression models.
6.1
hypothesis 1
Hypothesis 1 (H1): Amplified vertical head nods will result in a worse total game performance compared to the baseline condition.
A generalized linear mixed-effects model was used to test this hypothesis. The model incorporates both fixed-effects parameters and random effects in a linear predictor, via maximum likelihood. As the fixed effect, the type of condition was used. As a random effect, the experiment number was used to distinguish each of the pairs.
The response variable, game performance, is shown in Table2. Binomial logistic
regression was used to investigate the relationship between the categorical response variable and one two-level explanatory variable. The baseline condition in each test is always chosen as the reference category and the result of the first test can be seen in Table3. The formula used in R: glmer(cbind(correctScore, incorrectScore) ∼
Condition + (1|ExpNo), family = binomial, control = glmerControl())
Condition Correct Incorrect
Baseline 123 87
Vertical amplification 127 85
Horizontal amplification 134 80
Table 2: The total scores in each of the conditions from the experiments
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Probabilities Game performance Intercept 0.3463 0.1401 2.47 0.013 0.586 Game performance Condition type: Vertical amplification 0.0553 0.1981 0.85 0.394 0.599
Table 3: The output of the logistic regression model comparing the vertical amplification condition to game performance with the experiment number as a random effect.
In this case, because logistic regression was used, the estimates are given in log odds. Table3shows that the slope of the vertical amplification condition is positive,
which indicates that amplification of the vertical head movements corresponds to an increase in the log odds of the experiment being more correct. Furthermore, it shows that the sign of the intercept is positive, which indicates that for the baseline condition, it is the case that participants make the correct choice instead of the in-correct choice more than 50% of the time. To transform the log odds, and create probabilities, a logistic function was applied and the values were added as an extra column to Table 3. Given this model, it can be expected that the experiment ends
with more correct choices than incorrect choices in the baseline on average about
6.2 hypothesis 2 12
58.6% of the time. For the vertical amplification, this is 59.9%. The output gave dif-ferent results than expected. The vertical amplification increases game performance compared to the baseline condition, but not significant. The hypothesis can not be confirmed.
6.2
hypothesis 2
Hypothesis 2 (H2): Amplified horizontal head nods will result in a worse total game performance compared to the baseline condition.
Similar to the first one, a generalized linear mixed-effects model was used to test this hypothesis. The same response variable and random effect were used in this test, except this result shows the horizontal amplification condition compared to the baseline. The output of the test can be seen in Table4. The formula used in R:
glmer(cbind(correctScore, incorrectScore) ∼ Condition + (1|ExpNo), family = binomial, control = glmerControl())
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Probabilities Game performance Intercept 0.3463 0.1401 2.47 0.013 0.586 Game performance Condition type: Horizontal amplification 0.1695 0.1990 0.28 0.780 0.626
Table 4: The output of the logistic regression model comparing the horizontal amplification condition to game performance with the experiment number as a random effect
Table 4 shows that the slope of the horizontal amplification is also positive,
which indicates that amplification of the horizontal head movements corresponds to an increase in the log odds of the experiment being more correct. Given this model, it can be expected that the experiment ends with more correct choices than incorrect choices in the amplification condition on average about 62.6% of the time. This also was not the result that was expected. The participants performed better in the amplified conditions, but not significantly. This hypothesis is not confirmed.
6.3
hypothesis 3
Hypothesis 3 (H3): The game performance after the vertical amplification is removed will be better than before the removal.
Although the vertical amplification did not reach significance with game per-formance as the response variable, it could still be the case that the vertical ampli-fication does in some way affect the game performance during the experiments. The results of the analysis of the vertical amplification pre and post swap can be found in Table 5. The formula used in R: glmer(cbind(Correct, Incorrect) ∼
PreOrPost + (1|ExpNo), family = binomial, control = glmerControl())
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Probabilities Game performance Intercept 0.4015 0.1401 2.87 0.0042 0.599 Game performance Post swap -0.0702 0.2846 -0.25 0.8052 0.582
Table 5: The output of the logistic regression model comparing pre or post swap in the vertical condition to game performance
Table5shows that the slope of the post swap condition is negative, which
indi-cates that the removal of the amplification of vertical head movements corresponds to a decrease in the log odds of the experiment being more correct instead of incor-rect. Furthermore, it shows that the sign of the intercept is positive, which indicates
6.4 hypothesis 4 13
that for the pre swap condition, it is the case that participants make the correct choice instead of the incorrect choice more than 50% of the time. Given this model, it can be expected that the experiment ends with more correct choices than incorrect choices in the vertical amplification condition on average about 59.9% of the time. After the amplification is removed, this is 58.2%. The results show a slight increase in in-game performance for the vertical amplification condition, but not enough to reach any significant change. The hypothesis can not be confirmed.
6.4
hypothesis 4
Hypothesis 4 (H4): The game performance after the horizontal amplification is re-moved will be better than before the removal.
As explained above, analysis within the condition could still result in some inter-esting outcomes. The results of the analysis of the horizontal amplification pre and
post swap can be found in Table6. The formula used in R: glmer(cbind(Correct, Incorrect)∼
PreOrPost + (1|ExpNo), family = binomial, control = glmerControl())
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Probabilities Game performance Intercept 0.516 0.141 3.65 0.00026 0.626 Game performance Post swap -0.382 0.295 -1.30 0.19477 0.533
Table 6: The output of the logistic regression model comparing pre or post swap in the horizontal condition to game performance
Broadly speaking, just as in the first hypothesis, the horizontal amplification condition shows more relation to the game performance. Table 6 shows that the
slope of the post swap condition is negative, which indicates that the removal of the amplification of horizontal head movements corresponds to a decrease in the log odds of the experiment being more correct instead of incorrect. Also, it shows that the sign of the intercept is positive, which indicates that for the pre swap condition, it is the case that participants make the correct choice instead of the incorrect choice more than 50% of the time. The model reveals that it can be assumed that the experiment ends with more correct choices than incorrect choices in the horizontal amplification condition on average about 62.6% of the time. After this amplification is removed, this is 53.3%. The results show an increase in in-game performance for the horizontal amplification condition, but not enough to reach significance. This hypothesis can not be confirmed.
6.5
hypothesis 5
Hypothesis 5 (H5): The game performance after the vertical and horizontal attenua-tion is added will be worse than before the addiattenua-tion.
This condition differed from the other two because it acted as the baseline condi-tion. The other two conditions would return to the baseline condition after 20 min-utes, but in this condition, the horizontal and vertical head movements were attenu-ated. Although this was not explained by the background, examining if it has any ef-fect would still be compulsive. The formula used in R: glmer(cbind(Correct, Incorrect)∼ PreOrPost + (1|ExpNo), family = binomial, control = glmerControl())
Table 7 shows that the slope of the post swap condition is negative, which
in-dicates that the application of attenuated horizontal and vertical head movements corresponds to a decrease in the log odds of the experiment being more correct in-stead of incorrect. The sign of the intercept is positive, which indicates that for the
6.6 hypothesis 6 14
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Probabilities Game performance Intercept 0.346 0.140 2.47 0.013 0.586 Game performance Post swap -0.148 0.277 -0.54 0.591 0.549
Table 7: The output of the logistic regression model comparing pre or post swap in the baseline condition to game performance
baseline condition, it is the case that participants make the correct choice instead of the incorrect choice more than 50% of the time. The model reveals that it can be assumed that the experiment ends with more correct choices than incorrect choices in the baseline condition on average about 58.6% of the time. For the attenuated movements, this is 54.9%. The results do show an increase in in-game performance if no attenuation is applied which supports the hypothesis, but not significantly.
6.6
hypothesis 6
Hypothesis 6 (H6): The amplification of vertical head movements will increase posi-tive sentiment in the communication between participants compared to the baseline. This analysis was done to examine if there is a relation between positive sen-timent in the chat and positive evidence through the head nods. Four different linear regression models were distinguished in this hypothesis. In the first two models, the sums of all of the positive and negative sentiment scores, extracted with a program by Boot et al. (2017), are used as the response variable and the
type of condition as the predictor variable. In the last two models, the averages of all of these positive and negative sentiment scores are used as the response vari-able and the condition remains the predictor varivari-able. The baseline condition is always chosen as the reference category. The total positive sentiment scores can be found in Table8, and the negative scores in Table9. The average positive sentiment
scores can be found in Table 10, and the average negative scores in Table11.
For-mulas used in R: lm(posemoT otal ∼ Condition), lm(posemoAvg ∼ Condition),
lm(negemoT otal∼ Condition), and lm(negemoAvg ∼ Condition)
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Total positive sentiment Intercept 149.513 31.886 4.689 4.33e-05 Total positive sentiment Condition type: Vertical amplification -9.797 45.094 -0.217 0.829
Table 8: The output of the linear regression model comparing the total positive sentiment between the conditions
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Total negative sentiment Intercept 66.76 21.67 3.081 0.00407 Total negative sentiment Condition type: Vertical amplification 22.16 30.64 0.723 0.47458
Table 9: The output of the linear regression model comparing the total negative sentiment between the conditions
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Average positive sentiment Intercept 3.322 1.321 2.515 0.0168 Average positive sentiment Condition type: Vertical amplification 0.213 1.868 0.114 0.9099
Table 10:The output of the linear regression model comparing the average positive senti-ment between the conditions
6.7 hypothesis 7 15
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Average negative sentiment Intercept 0.9977 0.4567 2.185 0.0359 Average negative sentiment Condition type: Vertical amplification 0.8432 0.6459 1.306 0.2005
Table 11:The output of the linear regression model comparing the average negative senti-ment between the conditions
Given the models from Tables 8and 9, it predicted that the baseline condition
would result in a total positive sentiment score of 149.513. With the vertical ampli-fication applied a total score of 139.716. Table9 does show a decrease in negative
sentiment comparing 22.16 in the vertical amplification condition to 66.76 in the baseline condition. The hypothesis can not be confirmed. It shows a decrease in total negative sentiment, but not an increase in positive sentiment.
A slightly different effect can be seen in Tables10and11in which the prediction
of average total positive sentiment in the vertical amplification condition shows a slight increase compared to the baseline condition, but not significantly.
6.7
hypothesis 7
Hypothesis 7 (H7): The amplification of horizontal head movements will increase negative sentiment in the communication between participants compared to the baseline.
This hypothesis follows the belief that negative expressions can be generated by head shakes, by stating that the amplification of horizontal head movements will result in an increase in negative sentiment in a conversation. This hypothesis is again tested with four separate linear regression models. In the first two models, the sums of all of the negative and positive sentiment scores are used as the response variable and the type of condition as a predictor variable. In the last two models, the average negative and positive sentiment scores are used as the response variable and the condition is still the predictor variable. Formulas used in R: lm(posemoT otal∼
Condition), lm(posemoAvg ∼ Condition), lm(negemoTotal ∼ Condition), and
lm(negemoAvg∼ Condition)
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Total negative sentiment Intercept 66.76 21.67 3.081 0.00407 Total negative sentiment Condition type: Horizontal amplification 11.92 30.05 0.397 0.69409
Table 12:The output of the linear regression model comparing the total negative sentiment between the conditions
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Total positive sentiment Intercept 149.513 31.886 4.689 4.33e-05 Total positive sentiment Condition type: Horizontal amplification 10.381 44.218 0.235 0.816
Table 13:The output of the linear regression model comparing the total positive sentiment between the conditions
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Average negative sentiment Intercept 0.9977 0.4567 2.185 0.0359 Average negative sentiment Condition type: Horizontal amplification 0.4106 0.6333 0.648 0.5211
Table 14:The output of the linear regression model comparing the average negative senti-ment between the conditions
6.8 hypothesis 8 16
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Average positive sentiment Intercept 3.322 1.321 2.515 0.0168 Average positive sentiment Condition type: Horizontal amplification 3.291 1.832 1.796 0.0813
Table 15:The output of the linear regression model comparing the average positive senti-ment between the conditions
The first two models predicted that the baseline condition would result in a to-tal negative sentiment score of 66.76. With the horizonto-tal amplification applied, it predicted a total negative sentiment score of 78.68. The total negative sentiment increased in the horizontal amplification condition in comparison to the baseline condition, but not significant. The averages of both the numbers show a slight increase in negative sentiment when the horizontal movements are amplified com-pared to the baseline, but almost double the amount of positive sentiment. This hypothesis could not be confirmed.
6.8
hypothesis 8
Hypothesis 8 (H8): The character length per round inside the game will decrease as the game continues.
As mentioned earlier, the theory that participants create definite descriptions, which get shorter and shorter as the experiment progresses researched bySchober and Clark (1989) is tested with this hypothesis. Linear regression is used with
character length as the response variable and round as the explanatory variable. The results are shown in Table16. Formula used in R: lm(characterLength∼ Round)
Response variable Explanatory variable Estimate Std. Error t value Pr(>|t|) Character length Intercept 128.5600 4.4778 28.711 2e-16
Character length Round -2.5675 0.2945 -8.717 2e-16
Table 16:The output of the linear regression model showing the change in character length per round
The table shows that on average for each experiment, the character length is 128.5600. It also shows a significant decrease in character length once the experi-ment progresses. This hypothesis can be confirmed, the character length per round inside the game does decrease as the game continues.
6.9 questionnaire 17
6.9
questionnaire
The questionnaire resulted in a total of 84 responses and was only filled in for exploratory analysis. The first question asked the participants to classify their part-ner’s communication by a Likert scale. In Figure2is shown that the majority, a total
of 68 participants, classified their partner’s ability as moderately well or better. The answers to the second question, in which they classified their communication from the partner’s perspective, showed a similar distribution as can be seen in Figure
3. In general, the participants were pleased with the communication between each
other. This can also be seen by the grades that were given, the mean grade resulted in a 6.36 for the 84 responses, from which only 19 participants gave an insufficient grade.
The fourth question was added to detect if the participants had any idea that their non-verbal communication was being modified during the experiment. The majority of the participants said that the experiment was focused on how communi-cation works in a virtual reality setting. Some participants mentioned the grounding model byClark and Brennan(1991) and how they had to ground with a new form of
communication. The identification of all of the different faces was also mentioned, and especially the differences between the ways of describing several identifiable facial features.
In the last question they got asked, one answer stood out from the rest. They were asked what the most difficult thing to communicate was. They explained that they had a lot of difficulties with explaining doubt in the conversation. Some participants stopped moving their head, nodded slowly, or instead really fast to explain that they were not certain about the questions. Most of them also mentioned that describing the faces, especially all of the different features was difficult.
Figure 2: Distribution of the first question
7
C O N C L U S I O N
This research aimed at answering the questions: "What effect does the amplifica-tion of head nods and head shakes have on game performance?" and "What is the impact of removing the amplifications during the task?". Based on quantitative anal-ysis of game performance in response to different conditions, it can be concluded that the game performance did differ per condition, but that the differences were not significant. The initial belief that the amplification of vertical and horizontal head movements would result in worse game performance was not shown in the results. The hypotheses concerning the amount of sentiment in the communication between the participants even showed the contrary of what was expected because the horizontal condition resulted in more positive, and the vertical condition in more negative sentiment.
Interestingly the results from within the conditions show that removing the am-plification of the head movements resulted in a worse game performance than with the amplification applied. This result shows that participants do adapt fairly quickly to the newly made conventions, although this was not significantly supported.
The expression of doubt was one of the most interesting outcomes of these ex-periments. Participants had a difficult time expressing doubt, as well as reading if the partner was in doubt or not. The following chat logs with additional times-tamps are from two different experiments, translated from Dutch to English, and show parts of the conversation in which player 2 does not know what to choose due to doubt from player 1:
Experiment 20
• [14:49:07] Okay, so not the same? Shake yes if you agree
• [14:51:25] Okay, I now understand that I read everything wrong. Could you shake yes if that’s the case and you mean same?
• [14:51:48] You’re just making circles now Experiment 37
• [15:35:13] Was that a nod?
• [15:35:38] Hahaha your head is just going in circles • [15:46:34] If yours is smirking pls move head • [15:46:51] Nothing happens
• [15:53:52] Was that a no? • [15:53:53] If so pls move head
conclusion 19
These examples show the different ways that the participants who were selected as player 1 tried to express doubt. In the first example, player 1 used circular motions to signal doubt to player 2. In the second example, player 1 initially started with this tactic but changed it during the conversation to not showing any motion, throwing off player 2. In almost all of these cases, player 2 had no idea that player 1was doubting the choice and was constantly trying to guess what player 1 meant. Doubt can be seen as negative feedback since it is not instantly accepted. Instead of accepting this feedback, player 2 kept asking questions to obtain positive feedback. Due to the coronavirus outbreak during these experiments, a diversion had to be made from experiments with headsets in physical rooms to online experiments with a mouse that simulated head movements and non-verbal communication methods. The simulated head movements may not have been the best representations of non-verbal feedback in a conversation. In addition, as mentioned in Chapter4, a lot of
experiments were lost due to online technical difficulties which possibly could have resulted in more significant results. Further research and additional experiments are needed to determine the relationship between these head movements and game performance.
B I B L I O G R A P H Y
Allwood, J. and L. Cerrato (2003). A study of gestural feedback expressions. In First nordic symposium on multimodal communication, pp. 7–22. Copenhagen.
Bailenson, J. N., J. Blascovich, A. C. Beall, and J. M. Loomis (2001). Equilibrium the-ory revisited: Mutual gaze and personal space in virtual environments. Presence: Teleoperators & Virtual Environments 10(6), 583–598.
Birdwhistell, R. (1970). Sequence and tempo. In Kinesics and context. University of Pennsylvania Press Philadelphia, PA.
Blascovich, J., J. Loomis, A. C. Beall, K. R. Swinth, C. L. Hoyt, and J. N. Bailenson (2002). Immersive virtual environment technology as a methodological tool for social psychology. Psychological inquiry 13(2), 103–124.
Boholm, M. and J. Allwood (2010). Repeated head movements, their function and relation to speech. In Proceedings of LREC workshop on multimodal corpora advances in capturing coding and analysing multimodality, pp. 6–10. Citeseer.
Boot, P., H. Zijlstra, and R. Geenen (2017). The dutch translation of the linguistic inquiry and word count (liwc) 2007 dictionary. Dutch Journal of Applied Linguis-tics 6(1), 65–76.
Clark, H. H. and S. E. Brennan (1991). Grounding in communication.
Clark, H. H. and E. F. Schaefer (1987). Collaborating on contributions to conversa-tions. Language and cognitive processes 2(1), 19–41.
Clark, H. H. and E. F. Schaefer (1989). Contributing to discourse. Cognitive sci-ence 13(2), 259–294.
Clark, H. H. and D. Wilkes-Gibbs (1986). Referring as a collaborative process. Cog-nition 22(1), 1–39.
Condon, W. S. and W. D. Ogston (1967). A segmentation of behavior. Journal of psychiatric research.
Duncan, S. (1972). Some signals and rules for taking speaking turns in conversations. Journal of personality and social psychology 23(2), 283.
Goodwin, C. (1981). Conversational organization. Interaction between speakers and hearers.
Hadar, U., T. J. Steiner, E. C. Grant, and F. Clifford Rose (1983). Head movement correlates of juncture and stress at sentence level. Language and speech 26(2), 117– 129.
Hadar, U., T. J. Steiner, and F. C. Rose (1985). Head movement during listening turns in conversation. Journal of Nonverbal Behavior 9(4), 214–228.
Healey, P. G., C. Frauenberger, M. Gillies, and S. Battersby (2009). Experimenting with non-verbal interaction. In 8th international gesture workshop.
Isaacs, E. A. and H. H. Clark (1987). References in conversation between experts and novices. Journal of experimental psychology: general 116(1), 26.
Jakobson, R. (1972). Motor signs for’yes’ and’no’. Language in Society, 91–96.
BIBLIOGRAPHY 21
Kendon, A. (1970). Movement coordination in social interaction: Some examples described. Acta psychologica 32, 101–125.
Kendon, A. (2002). Some uses of the head shake. Gesture 2(2), 147–182.
Kousidis, S., Z. Malisz, P. Wagner, and D. Schlangen (2013). Exploring annotation of head gesture forms in spontaneous human interaction. In Proceedings of the Tilburg Gesture Meeting (TiGeR 2013).
McClave, E. Z. (2000). Linguistic functions of head movements in the context of speech. Journal of pragmatics 32(7), 855–878.
Poggi, I., F. D’Errico, and L. Vincze (2010). Types of nods. the polysemy of a social signal. In LREC.
Rosenfeld, N. (1978). Conversational control function of nonverbal behavior. Non-verbal behavior and communication.
Schegloff, E. A. (1982). Discourse as an interactional achievement: Some uses of ‘uh huh’and other things that come between sentences. Analyzing discourse: Text and talk 71, 93.
Schober, M. F. and H. H. Clark (1989). Understanding by addressees and overhear-ers. Cognitive psychology 21(2), 211–232.
Appendices
A
F O R M U L A S
Figure 4: All of the applied formulas for the conditions
B
I N F O R M E D C O N S E N T
Figure 5:The Informed Consent form
