Conversation with Virtual Assistants; Searching for Deeper Meaning in Voice

(1)

Conversation with Virtual Assistants;

Searching for Deeper Meaning in Voice

Olga Pegiou Student number: 11416130

University of Amsterdam Faculty of Science

Thesis Master Information Studies: Business Information Systems Final version: 05/07/2018

Supervisor: Vanessa Dirksen Examiner: Toon Abcouwer

Abstract. Rapid advances in natural language processing (NLP) empowered voice

to become the novel form of interaction with technology. The field of conversational Human-Computer interaction calls for more research around the speaking style of the new Voice User Interfaces; as the current voices of virtual characters are rated inferior by users. The primary objective of this study is to expand the current knowledge, reveal new and conclusive information about the optimal Virtual Assistants’ speaking behavior towards human users from the end-user point of view, looking into the possible personality features that would serve best the human needs. By combining an experimental design with qualitative research strategy, a short experiment and following interviews were conducted. The outcomes of this analysis provide valuable guidelines for the design of relevant software, and for further research in the field.

Keywords. Virtual assistant, virtual agent, voice user interface, Human-Computer

interaction, social interface theory, social agency theory, character, personality, emotions.

(2)

In the past few years, along with the vast developments in ubiquitous computing and Artificial Intelligence (AI), Voice User Interfaces (VUI) have been immensely advanced. Research and development in the speech and language fields empowered us to interact with machines in a profound natural manner, through conversation. Natural Language Processing (NLP) augmented with Deep Learning algorithms and Neural Network Architectures, has introduced a new era in Text-to-Speech (TTS) Synthesis; and thus, in Human-Computer Interaction (HCI), where the VUI has become mainstream, as it aims at an eventually frictionless and excessively faster interaction.

VUI’s usually use a Virtual Assistant - VA as the representative of the technology, following after chatbots, which are usually used as customer service solutions and interact through text. A VA is a computer software that uses machine learning algorithms to perform tasks on behalf of the user, and simulates human behavior by using voice and conversation as input and output. The most widely used commercial VAs at the moment are Apple’s Siri, Google’s Google Assistant, Amazon’s Alexa, and Microsoft’s Cortana.

The appearance of VAs enabled a variety of digital experiences for consumers and enterprise applications across many industry sectors (automotive, retail, healthcare, education, telecommunications), providing unique opportunities for companies to compete in digital transformation. The trend is verified, according to market research (Richter, 2016), by a large number of already frequent users of VAs and the significant increase of the number of worldwide users. In 2018, it is predicted to be a little more than one billion users worldwide, while in 2021 the number is almost doubled (1,83 billion). Additionally, the market for speech recognition technology is expected to exceed a global market size of USD 7.5 billion by 2024 (Hebbalkar, 2017).

1.1. Problem statement

Although highly advanced, speech technology and the VUIs have not yet reached the state of enabling effortless conversations with machines. The human actors, on the one side of the interaction, are still expected to constantly adjust their behavior and expectations from such conversation, which makes the VUI not as efficient as its intended potential. This demand for adjustment plays an important role to the user satisfaction and undoubtedly influences the quality of the customer journey and thus, the whole quality of the HCI. Moreover, the voice of the state-of-the-art VAs and their style of speech seems to be unnatural or non-realistic; in current studies where human voice and synthesized voice are compared, the human is still preferred in terms of naturalness, appeal, intelligibility (Cabral, Cowan, Zibrek, & McDonnell, 2017). That component constitutes the interaction through conversation to feel inadequate and non-efficient, leading to lesser usage of the technology, unable to get users to engage.

The literature and research on the field so far, strongly suggests that these insufficiencies can be addressed by designing the VA in a way that is more natural to humans. Thus, by attempting to implement various kinds of emotional intonations into

(4)

the conversation, the necessary effort from the human side of the interaction would subside and the communication would become more pleasant and comprehensible (Mayer, Sobko, & Mautone, 2003). Ultimately, the goal is to further improve the translation of the computer language to human language, so that it is even easier for users to interact with the technology, in the way they effortlessly communicate with other humans; hence, further improve the interface.

In this paper the main focus is to optimize the user experience of interacting with a voice enabled VA. It will be presented that the intend of such user interfaces should take into consideration the specific nature of the synthesized speech and voice, that can influence positively the user experience; by presenting it as more natural, human-like and effective for serving the human needs. Any speech component that is shown to influence the end-user experience, and accordingly the HCI, should be incorporated into the design. The competence in engaging users to the interface finally defines the success of the technology.

The VAs’ aspect that will be investigated here is limited to the type of the voice; where type is the expressive voice prosody and the prosodic speech patterns embedded in the VA (prosody is defined as the ranges of rhythm, pitch, stress, intonation etc. in a language). It will be argued that a voice that consists of a specific and consistent set of characteristics, which imply a certain personality or character for the speaker, is able to make a difference in the HCI. More specifically, this paper aims at providing answers to the following research questions:

• What are the human characteristics of a VA’s voice that will optimize the HCI? • Which personality attributes are desired by the users for a VA?

• How does a perceived character/persona in a VA’s voice influences HCI? Other aspects of a VA that have power over the user experience; for example, the plausible appearance of the VA, which can be closer to anthropomorphism or not; or the content of the synthesized speech and conversation, are out of the scope of this research paper. Additionally, the process of developing an actual VA with integrated social skills and personality characteristics is out of the scope of this paper as well.

In this paper, firstly the previous research in the field is explored and the outcomes and suggestions from related work are taken into consideration, in order for the main hypothesis to be formed. Next, the proposed methodology is described and the selection of a qualitative research strategy, in light of the research topic, is justified. Afterwards, follows the execution of the experiment and interviews. Subsequently, the primary and most relevant results of the research, as well as their analysis are presented; and their interpretation and relation to previous research and theories is evaluated. Finally, the limitations of this research paper are presented, as well as suggestions for future work, and the final conclusions are drawn.

(5)

“Words mean more than what is set down on paper. It takes the human voice to infuse them with deeper meaning.”

- Maya Angelou

2. Literature Review

2.1. The momentum of the VUI

The use of voice as means of communication with machines has always been desired, simply due to the factor of time. Services that deliver results sooner seem to be of high importance in the modern age. The Voice User Interface (VUI) is constantly improving, to allow only the use of voice for completing tasks and finding information, by removing typing, clicking, tapping, and reading from the human-computer interaction (HCI) equation. A virtual assistant – VA (also referred to as Voice Assistant, Virtual Agent, Intelligent Virtual Agent or Assistant, or Virtual Personal Assistant) is a common form of a VUI, which implies a conversational agent serving as a human representative, capable of providing various services.

The brisk advancements in pervasive and ambient computing witnessed nowadays, constantly demand and deliver corresponding developments in speech technology. With the recent innovations in Natural Language Processing (NLP), and specifically the use of artificial neural networks for deep-learning, a very high-quality expressive speech can now be synthesized (Van den Oord, et al., 2016), constituting the traditional, “robotic” voice obsolete. By manipulating speech prosody (stress or pitch accent) among words or utterances, different emotional states and speaking styles can be expressed (Nagy & Nemeth, 2016); enabling new applications and solutions to emerge.

A variety of current surveys on voice technologies and virtual assistance show the tremendous potential across industries. Promising affordances for personalization, high quality customer service, and low operating costs pushes major investments in research and development, and further influence the industry trends (Hebbalkar, 2017). Products, machines, and applications integrated with speech technology can now understand different languages, accents, dialects and even distinguish multiple speakers at once.

Virtual assistants are most commonly used as personal virtual agents embedded in mobile phones, computers and other smart devices, destined to serve the consumer’s needs. According to Forrester, VA use increased 30%, from 2012 to 2015 (Witcher, 2015); and Google claims that 20% of mobile queries in 2016 were voice searches. Consumers keep disconnecting physically to reconnect digitally with companies and use the VAs more and more for customer self-service (Witcher, 2015) and for a variety of daily tasks, as the convenience of their use is getting expanded. Some of the most common uses of mobile VAs are searching something online, get directions, call or text someone, while VAs in smart speakers and homes usually play music, read the news, control smart lights and shop online (Bajarin, 2016). This notable rise in adoption rates of VAs over the last few years indicates the importance of immediate, smart, self-service support for the modern consumer.

VAs are widely used as digital customer service agents, attempting to reduce the high labor costs of support agents (Witcher, 2015). At the same time, these company

(6)

VAs introduce a new era in branding; the voice of a company can serve as the new logo. The brand identity, company profile and culture can be designed in the gender, accent, articulation etc. of the VA’s voice. According to Gartner, customers will manage 85% of their enterprise relationship without interacting with humans by 2020 (Moore, 2018). Thus, digital customer experience is fundamental for the retail industry as well, in constantly advancing and benefiting from e-commerce (Johnson, 2018). Additionally, automotive, healthcare, banking-financial-services-and -insurance (BFSI), IT & telecom, and education are some of the other key application areas for the VA market. The global VA market size is estimated to grow at an annual growth rate of 34.9% over the coming six years, as seen in Figure 1 (Hebbalkar, 2017).

Figure 1. U.S. Intelligent Virtual Assistant Market size by application, 2013-2024 (USD Million).

2.2. The research so far

“Designing virtual personal assistants that are able to engage users in an interaction has been a challenge for HCI researchers for the past 20 years.” (Shamekhi, Czerwinski, & Mark, 2016)

Several experimental studies over the years have shown that people do not respond to interactive software as a simple device or tool. Instead, humans appear to “mindlessly apply social rules and expectations to computers” (Nass & Moon, 2000), while they recognize the fundamental truth that a computer system is not a real person. This paradigm is known as “Computers As Social Actors or CASA” and was introduced by Reeves & Nass (1996) when they observed polite interactions with computers through experiments. The paradigm was later refined as Social Interface Theory (SIT) (Dryer, 1999) and it was further explored and extended to Social Agent Theory (SAT). SAT proposes that by enhancing the humanness of appearance and behavior of an interactive agent, human users will exhibit social responses found in human–human interactions; thus, making the agent not only more appealing but also more effective (Mayer, Sobko, & Mautone, 2003). By recognizing social cues, users are motivated to make sense of the information being communicated and to apply the norms and attitudes of human interaction (Mitchell, Ho, Patel, & MacDorman, 2011).

Based on these fundamental theories, the primary goal of studies that look into improving HCI with a conversational interactive system, up to the present day, is to make synthesized voice as natural as possible. There are numerous attempts in increasing vocal

(7)

realism found in literature. A set of those studies are found within an annual event named “The Blizzard Challenge”, started in 2005. Usually, 10 to 20 groups independently build synthetic voices from a common speech corpus and the speech samples are evaluated, using a large pool of listeners. More recently, the 2016 and 2017 Blizzard Challenges aimed to deliver expressive speech synthesis related to a specific context (children’s audiobooks) (King, 2014).

A lot of research explored the preferences of users for the gender of a VA’s voice. The results, although often contradictory, show eventually that people have explicit preferences on the gender factor and most of the times, they differ due to context, user personality or stereotypical expectations, presenting a general preference towards female voices, probably due to positive attitudes towards maternal attributes (Payne, Szymkowiak, Robertson, & Johnson, 2013), (Beale & Creed, 2009). Moreover, findings over an assistant’s accent present conclusive and show no preference for a foreign accent (Mayer, Sobko, & Mautone, 2003), (Mitchell, Ho, Patel, & MacDorman, 2011).

The expressivity is frequently approached through simulation and expression of emotions in the vocal responses of the VAs. Firstly, Picard claimed that “computers that will interact naturally and intelligently with humans need the ability to at least recognize and express affect; to best leverage users’ life-time of experience with social interaction” (Picard R. W., 1997). In addition, humans can benefit from emotional interactions, even if the emotion is just an idealized simulation of a real emotion, in the same way they have long benefited from pet interactions; since animals have a very superficial understanding of the human emotions (Picard & Klein, 2002). Empathy is one of the main emotions integrated in VAs, and seems to play an important role in HCI, as it has been described to provide more enjoyable interactions (Leite, Pereira, Mascarenhas, Martinho, & Prada, 2013). Also, empathic VAs were seen as more trustworthy, likeable, supportive, caring (Fan, Scheutz, Lohani, McCoy, & Stokes, 2017) (Brave, Nass, & Hutchinson, 2005); as well as friendlier, encouraging and sensible (Leite, Pereira, Mascarenhas, Martinho, & Prada, 2013). Empathetic responses in general, have the potential of enhancing the perceptions of VAs (Beale & Creed, 2009). Improving the emotional perceptions of a software has been addressed in a variety of other ways as well, i.e. incorporating speech breathing behaviors to convey information about the speaker’s mood, age and health (Bernardet, Kang, Feng, DiPaola, & Shapiro, 2017). Synchronous haptic experience has been proven useful in increasing the users’ feelings of interpersonal closeness to a VA (Kelly, 2015).

In the same direction, efforts to improve the realism of HCI rendered essential the design and implementation of personalities into VAs. Many authors that examined the effects of a personality of VAs in HCI agree that a personality, presented in a consistent way indeed increases the perceived realism of a VA (Becker, Kopp, & Wachsmuth, 2007), (Dryer, 1999), (Isbister & Nass, 2000). Furthermore, important outcomes describe some of the preferred personalities to include cooperative, outgoing, calm, organized, or curious attributes, against competitive, withdrawn, anxious, lax, or close-minded traits (Dryer, 1999). Some authors choose to focus more on the human user of the interactive agent, to clarify the appropriate personality for VAs. The similarity-attraction theory “suggests that people are attracted to similar others, who tend to be perceived as more credible, resulting in increased liking, and in technology, reduced discomfort with the interface” (Payne, Szymkowiak, Robertson, & Johnson, 2013). Authors that tested that theory in their research do not always seem to draw similar conclusions. For example, Shamekhi, Czerwinski, & Mark claim that users strongly prefer VAs with similar conversational style (Shamekhi, Czerwinski, & Mark, 2016), while Isbister & Nass

(8)

advocate personalities that are complementary, rather than similar, to the personality of the user (Isbister & Nass, 2000). Finally, Dryer observes that even though a similar personality is generally better than a dissimilar, sometimes complementary personalities can be a better option (Dryer, 1999), which entails that there are other factors that also influence and define the preferences of the users.

2.3. Impediments in research

However extensive, the research in improving HCI with conversational appears to have some noticable drawbacks. Firstly, in the majority of studies, the diverse and often competitive manners that the developers try to include additional prosodic tags to the text to be synthesized are presented (Boros, Stam, Watts, & Dumitrescu, 2014), providing technical details on building various emotional, expressive and realistic voices. They however, fail to capture the validity of the intended emotions and personalities, from the end-users’ point of view; and in general disregard to test the influence of those expressive agents on humans (Nunnari & Heloir, 2017) (Doce, Dias, Prada, & Paiva, 2010).

Another inconvenience of the research in the field is that the studies are frequently conducted under a specific domain, or for an explicit use; like in educational scenarios (Leite, Pereira, Mascarenhas, Martinho, & Prada, 2013) (Mayer, Sobko, & Mautone, 2003), for supporting vulnerable users (Yaghoubzadeh & Kopp, 2012), or for interrogative scenarios (Nagy & Nemeth, 2016). This variability of approach makes it hard to draw conclusive results and build on previous work, highlighting however that the end-use of a VA also serve as an essential guideline for the its appropriate vocal, human-like characteristics.

Authors keep underlining the necessity of more comprehensive research (Lee, 2010), often of qualitative nature, in identifying the perceptions of VAs and all the possible factors that affect the quality of a conversational HCI (Payne, Szymkowiak, Robertson, & Johnson, 2013). In current studies, where human voice and synthesized voice are compared, the human is still clearly preferred in terms of naturalness, appeal, intelligibility (Cabral, Cowan, Zibrek, & McDonnell, 2017). Consequently, the fact that VUIs can be very helpful but they can also fail to engage users is self-evident. Within a survey, 43% of the participants strongly agreed to the statement “I would use my devices voice capabilities more if I could speak to it more naturally” (Bajarin, 2016).

2.4. Theoretical Framework

“To design intelligent, adaptive systems, we need a deep comprehension not only of the computer system, but of the human system as well.” (Kelly, 2015)

This study aims to confront the aforementioned issues, by using personality as an exploratory tool, to identify the set of human attributes and the level of realism that makes appealing and helpful digital interaction partners. Given the previous work in the field, research needs more innovative approaches to gain deeper understanding of this relatively new nature of HCI with conversational VAs. To approach this issue, a more user-centered design is deemed to be fitting, to get insight into the problem from user’s point of view. It is worth mentioning that almost every relevant study uses experimental designs and quantitative strategies to give answers to similar questions, which often seems to lead to contradictory and inconclusive results (Beale & Creed, 2009).

(9)

To begin with, the fact that conversational computer agents are indeed social actors, capable of forming relationships with users comparable to those found in the world of human–human interactions (Nass & Moon, 2000), according to social interface and agency theories, is accepted and sets the foundations of this study’s methodology. After subjecting participants to a hypothetical interaction with VAs that are enhanced with basic personality features, a further discussion with the end-users through qualitative interviews is imperative for a meaningful interpretation of users’ perceptions and favoritism of VAs. Hence, new and explicit information regarding the social nature of the HCI and the people’s desires is presented; providing valuable instructions for further research and design. (fig. 2)

3. Methodology

3.1. Design of Experiment and Interviews

For this research, an experimental crossover design is chosen, in which every subject is being exposed to every treatment and all the subjects have an equal distribution of characteristics among them. More specifically, the subjects share similar educational level (tertiary) and age (20-30); same time and place where the experiment and interview took place (office); and an equal distribution between female and male participants. This ensures that other factors that might affect the results are minimized, increasing the validity of the design. In addition, those characteristics and conditions feature a potential large target group and/or potential circumstances under which the product might be used, increasing external validity. The treatments consist of four recorded voices that simulate voices of VAs, and include two conditions, VAs with and without personality. The problem of carryover effects, where the first treatment adversely influences the other, is

(10)

a disadvantage of this research design. As counterbalance, the order of the audio stimuli is chosen randomly and presented for each subject. At the end, each subject is interviewed in a semi-structured way. The subjects have the opportunity to still listen to the samples when they find it necessary to reinforce their opinion. The exposure of the subjects to all of the voices is preferable. This reduces the amount of error arising from natural variance between individuals.

The recorded voices have certain characteristics that demonstrate and imply specific personalities, based on the Big Five personality traits or the Five Factor Model (FFM) (McCrae & John, 1992). Dryer (1999) describes personality as: “A personality is defined as the collection of individual differences, dispositions, and temperaments that are observed to have some consistency across situations and time.” The FFM is a categorization of personality traits in terms of five basic dimensions: Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness to Experience. For this study, the personality of conscientiousness is simulated, as it is described as an asset for any type of job (Furnham & Fudge, 2008); as well as agreeableness, which suggests sympathy, unselfishness, cooperation, and fairness, traits appropriate for the position of an assistant (Furnham & Fudge, 2008). Moreover, neuroticism, a negative type of personality in relation to conscientiousness, is also included. A negative set of characteristics is considered necessary within the experiment, for a proper and balanced stimulation of the participants opinions; a component that is missing notably from previous research (Becker, Kopp, & Wachsmuth, 2007) (Beale & Creed, 2009). The final voice is characterized as neutral in emotion or perceived personality, compared to the rest; and it resembles the current broadly used VAs’ voices. It can be described as slightly expressive and prosodically enhanced, but not in a consistent manner that implies any personality attributes. That type of voice is required for more valid results, to be ensured that the comparison between conditions is fair (Beale & Creed, 2009). Although this may seem reasonable, only a minority of studies has taken this into consideration (Beale & Creed, 2009).

The content of each audio stimuli consists of two parts. This study’s primary focus is the most common and broad use of VAs, i.e. assisting users in everyday tasks. Therefore, the first part is an instruction-giving script for a simple task (simulating assistance) and the second simulates a common response from the VA (simulating interaction).

The questions of the interview were designed with the advice of a professional psychologist (Marina Vamvaka), to further refine the interview protocol (Rabionet, 2011), This encourages the responses to be as unbiased and intuitive as possible; and compatible to the purpose of this paper. The nature of the interview wassemi-structured (Bryman, 2015), and the few fixed open-ended interview questions were (Castillo-Montoya, 2016) (Robinson & Mendelson, 2012):

• Which of the voices do you prefer intuitively and why?

• Describe all the voices and point out what you like and don’t like • How would the voice of your VA would be if it was up to you?

• Would you prefer an adjustment of the voice based on your mood or time of day?

• Would you like different voices under different circumstances?

The interviews were recorded and transcribed. The interview transcripts were openly coded, and codes and categories were recognized and organized in a comparative and

(11)

iterative matter, until saturation was reached (Bryman, 2015) (Jabareen, 2009). (See Appendix I for the coding overview)

3.2. Qualitative data collection and analysis rationale

The majority of previous studies that explored the nature of the conversational HCI with a VA, repeatedly follows a quantitative research paradigm, and always includes experimental designs. Such strategies are most often limited to close-ended data collection techniques, such as self-reported questionnaires, surveys, and/or computer software. Even when open-ended questions are used, the responses are usually quantitatively analyzed. Although they provided useful information, it seems unclear whether these approaches can capture the full opinions that people form in response to such experiences, i.e. interacting with a VA (Beale & Creed, 2009). Quantitative research designs sometimes tend to disregard the interpretation of the world around them, while the limited and structured form of inquiries and analysis tools might hinder valuable information and lack a desired reflexivity, significant for social research (Bryman, 2015). In this research paper, even though an experimental design is adopted as well, the data is collected qualitatively via interviews with the participants and is analyzed in a qualitative manner, through the grounded theory approach.

An experiment was considered appropriate, to introduce participants to the topic and trigger their thoughts on the matter, by presenting examples of VAs with different voices due to different personalities. The concept of personality in the experimental voices is used as a tool, to introduce and investigate the variety of character components. After being exposed to the voices, the respondents were then encouraged to “think-aloud” (Robinson & Mendelson, 2012), as they are subjected to open-ended questions designed to elicit their ideas and opinions. Extended responses and follow-up clarifications allowed the participants “the opportunity to struggle to find the right words to express their own interpretations” (Rabionet, 2011).

The purpose of this work is to explore user’s perceptions and preferences, for a more efficient design of products, and for a deeper understanding of the nature of this new form of social interaction. Literature states that the basic purpose of a qualitative study is “to seek, discover and understand a phenomenon, a process, the perspectives and worldviews of the people involved or a combination of these” (Merriam, 2002). Hence, the interpretative nature of qualitative research in this setting is necessary, to support the construction of context that is grounded in the meanings that the actors themselves place on their statements (Maxwell & Reybold, 2015). Guided by grounded theory, the data were analyzed through coding, constant comparison and theoretical saturation (Bryman, 2015).

Nevertheless, qualitative analysis can bear a few shortcomings worth mentioning. Personal bias and judgment can easily affect the interpretation of the data, and so the researcher should be careful in analysing and presenting the end-results appropriately. Moreover, a qualitative research design is usually unique and cannot be exactly replicated, meaning a lack in external reliability; which is often an important measure of the quality of the research (Bryman, 2015).

3.3. Procedure

The experiment was a randomized, counterbalanced, within-subjects design with two conditions, VAs with and without personality. The study objective was to have the

(12)

participant interact with an agent, via a scripted dialog. After shortly interacting with each conversational style (listening to four VAs giving instructions and a response) for one and a half minutes, users were asked questions about their enjoyment and satisfaction with the agents, as well as about their personal preferences about the voice of a VA. Their answers were further clarified and discussed in depth with follow-up questions. In total, 22 subjects participated in the experiment and interviews, and the interviews lasted 10-15 minutes.

4. Results and Analysis

In this section, the most predominant and relevant to the present study results, that emerged from the analysis of the interview data are presented. Furthermore, a discussion on the most practical and theoretical implications of the findings, as well as pertinent answers to the research questions are introduced.

4.1. Humor

Every subject of the study reacted very positively when the concept of humor and its integration into a VA came up during the discussion. The inclusion of humorous behavior in the interaction was either suggested by the participants as an element they miss; or it was mentioned by the interviewer as an example, e.g. “Would you like to see some personality characteristics in your VA, maybe humor or compassion?”. In addition, participants stated that the exhibition of humor should be subtle, meaning that it should be displayed under proper conditions, i.e. when the user is alone and not in a public/professional setting. More interestingly, even participants who explicitly stated that they do not wish VAs to become more human-like and realistic than they currently are, seemed to entertain the idea of their VA expressing some sense of humor; trusting that it will make the conversation more pleasant. This is quite intriguing, because even though realistic perceptions of VAs were openly and consciously rejected, humor is still undoubtedly a social interactive behavior shared and enjoyed among humans.

Psychological studies around the effects of humor in humans’ everyday life underline the importance of humor in personal well-being as well as in social interactions. More specifically, literature claims that a humorous behavior “produces a cognitive shift in perspective that provides individuals with a sense of control and empowerment over their environment, even for more serious situations” (McGhee, 1999) Humor has the effect of improving emotional well-being and optimism; increases the perception of control and consequently strengthens coping mechanisms; and decreases negative affect, stress, depression and anxiety (Crawford & Caltabiano, 2011). Furthermore, the impact of humor in social interactions seems to be extensive as well. The use of humor in an interaction between individuals can help establish closeness, signal liking and form bonds with others (Treger, Sprecher, & Erber, 2013).

The strong favoritism of this study’s subjects over displayed and perceived humor by a VA is not a surprise, given its strong influence in social dynamics. The omission of this characteristic however, in the behavior of common modern VAs appears to be a critical oversight by designers and developers. Listeners that perceive their partners as humorous and funny, they report that they like them more, they feel closer to them, and that the interaction is more enjoyable; even in online conversations among individuals

(13)

(Treger, Sprecher, & Erber, 2013). Therefore, the inclusion of humor in VAs suggests that end-users will possibly form a more positive impression of the VA; the HCI will be considered more pleasant and satisfying; while at the same time, improving the emotional well-being of the user and lowering negative sentiments.

4.2. Individuality

Almost half of the participants, in their efforts to describe and explain the kind of speaking style that would be optimal for them, came up with a specific popular character that has the role of a VA or PA in science-fiction. The mentioned fictional characters were Jarvis (Ironman), Alfred (Batman), HAL 9000 (2001: A Space Odyssey), and Samantha (Her). These characters share some common traits in their behavior. The purpose of their existence is to serve their human “master”, and they can be characterized by their assistive tone, trustworthy and confident disposition, clear devotion and loyalty. They also appear to acquire some unique traits that apparently distinguish them and individualize them among others, i.e. a certain tinge in their voice or a sum of qualities that indicate a distinctive character; as Dryer (1999) claims that personalities with a foible are more attractive. The rest of the participants, although they did not explicitly refer to a character, they provided desired characteristics that are in sync with the aforementioned traits of well-known assistants. Indicatively, the VA has been requested predominantly to be clear, calm, assistive, trustworthy, polite, confident, supportive, friendly, distinctive and consistent.

To sum up, it appears that the appropriate design of a VA should include a collection of traits that form a consistent character, one who ‘makes sense’ for the context; who is unique in some engaging way; who respects and willingly obeys the user; who can be seen as the ultimate ally; resembling an impersonation of the “traditional butler”. 4.3. Adjustment

The participants often suggested the idea, that the VA should have the ability to change and adjust its speaking style. This was triggered by the realization of having different preferences for different conditions and uses of the VA. Specifically, some examples by the participants, when adjustment was deemed quite useful are when the user is in a rush, where the VA should provide direct, fast and short responses; when tired after a long day, with a less active and more soothing tone in the VA’s voice; or when used in public places, where discretion is vital, and a formal voice would be more appropriate. An adjustable VA, which can adapt by itself or enable the user to manually change its voice (both options were equally desired), was described as flexible, more pleasant and easier to use. Interestingly, subjects that showed favoritism over neutral and stable voices, they further justified that this preference is the “safer” choice. Neutral and superficial voices are preferred in some cases; in order to avoid the risk of a realistic VA displaying an unwanted behavior, that might irritate the user.

Literature seems to be in agreement with these results. Horzyk et al. (2009) observed a significant increase in customer satisfaction, when a chatbot recognized the type of personality of the human-user and adapted its reactions accordingly, by changes in the choice of phrases in conversations. Ultimately, this favoritism of adjustment demonstrates that people are indeed strongly influenced by the speaking style of a VA.

(14)

4.4. Realism vs. Closeness

Participants seem to draw a line to the level of a realistic relationship with a VA. The responses of the participants suggest that the VA is indeed seen and treated as a social actor. Even more, the further integration of consistent and realistic features, that would imply a specific character or personality, is believed to improve the interaction. By making it more natural and human-like, the agent is perceived as more attractive, which would lead to an increase in usage. However, subjects objected to the notion of developing any kind of personal relationship with their VA. The VA is not seen as a possible friend, who would provide comfort or personal advice. Instead, it should be as real as a human personal assistant, who respects boundaries and acts in a common professional manner. This demarcation is possibly related to the uncanny valley paradigm, which explains feelings of revulsion and eeriness caused by highly anthropomorphized technology, usually related to its physical appearance (Mori, MacDorman, & Kageki, 2012).

Nevertheless, the present study limits the use of VAs in the context of assisting users in completing tasks and going through their everyday lives. Applications of VAs in different domains, like healthcare and education; and for specific operations, like consulting, might require deeper connection with the human user. Thus, a diverse set of exhibited skills that aim more to emotionally engage the user should be investigated. 4.5. Similarity and Complementarity

The similarity-attraction theory, relevant to research on VAs and their desired characteristics, suggests that “people are attracted to similar others, who tend to be perceived as more credible, resulting in increased liking; and in technology, reduced discomfort with the interface.” (Payne, Szymkowiak, Robertson, & Johnson, 2013). In the same line, the complementarity principle has been introduced to describe contradictory findings, where individuals seem to favor a personality that complements their own (Isbister & Nass, 2000).

The results of this research seem to reflect on both principles. Many of the suggested desired features for a VA by the participants were in alignment with their own character and way of communicating, e.g. humor, accent, speed, volume. At the same time, and often by the same participants, other attributes seemed to be preferred in order to assist in skills that the end-user is lacking, e.g. motivation, encouragement, confidence, organization. The occurrence of both principles suggests the important role of user characteristics and context of use in determining the speaking style of a conversational software, as it has also been proposed by Dryer (1999).

4.6. Accent

An accent, close to British, has been recognized in one of the voices, by half of the participants. The reaction to it was negative and wished to be removed every time, claiming that the conversation is more confusing, distracting and uncomfortable. This corresponds to results from previous related research, where a foreign accent is not favored and is described as unpleasant (Mayer, Sobko, & Mautone, 2003) (Mitchell, Ho, Patel, & MacDorman, 2011). In addition, there has been a hesitation from the side of the

(15)

participants in expressing this preference, in the fear of appearing as non-tolerant of other cultures/nations.

4.7. Negative Qualities

A few negative characteristics were recognized in the experimental voices. The participants fairly reacted and clearly stated their disapproval. Some of the provoking traits were the incorrect levels of speed and volume in the speaking style, that made the interaction distracting and difficult to follow. Speed and volume should always be in line with the purpose and context of use. Other basic negative features that were recognized and should be avoided in the design of VAs were aggressiveness, hesitation, condescension, arrogance, laziness. Although these reactions from the subjects are logical, future design should be cautious in avoiding implications of such undesired elements.

5. Limitations and future research

One basic limitation of the present study is that the VAs that the participants were exposed to during the experiment were represented by recordings of a human actor, and not a real VA. The experience of the participants and the setting of the experiment would possibly be more accurate and thorough, if the VA was an actual interactive software that would respond to the users’ voice commands on the spot. However, the fact that the voice was not synthesized but human did not affect the results of this research, as it is not the quality of the synthesized speech that is being studied or evaluated.

Moreover, the experienced interaction of the participants with the VAs was quite short and one-time only. However, the perceptions and assessment of a VA might not remain consistent after some time; and multiple interactions under various conditions might produce more informed and detailed descriptions of the end-users’ preferences. Longitudinal research designs seem to be a more appropriate choice for future studies, where participants can have multiple and more extended interactions over time, given the opportunity to fully form an opinion on the matter.

Furthermore, the simulated personalities and the characteristics and emotions that they expressed were just the basic ones, meaning that other more cognitive emotions were not taken into consideration. Human users with high levels of emotional intelligence would possibly perceive the communication of only basic expressions simplistic and fabricated; interfering with a more immersive experience. Future research should concentrate more on cognitive and complicated emotions, like pride, guilt, sarcasm etc., that humans employ in their social interactions.

The relatively small sample of subjects, as well as the structured setting of the experiment decreases the external and ecological validity, prominent evaluation criteria of social research (Bryman, 2015).

It is worth mentioning that the simulated personalities in the experimental voices were not perceived by the subjects exactly as intended. This did not have any negative influences in the furtherance of the present research. However, it is an imperative issue for related future work, to always validate the expressions of the agents prior to the experiment. Depending on the individual setting of every study, an incorrect perception

(16)

of the VA’s intended emotion can route to misleading measurements and outcomes (Beale & Creed, 2009).

Finally, the emergence of AI infused technologies that engage in conversation with humans is a powerful development for the social sciences. The perpetual emergence of new social actors among humans, which are treated socially but are not human, is a unique phenomenon. It is imperative that future research will keep exploring how the human behavior around this technology will evolve; and the possible impact of this new type of social actor in human-human interaction, in a world where such devices are regularly used. (Picard & Klein, 2002)

6. Conclusion

Our ability to design and build virtual agents that will naturally interact with us, stimulates the exploration of the optimal VA, from an end-users’ point of view. The underlying goal of the present study was to explore user preferences and provide deeper insight for the design of the voice of VAs, to create a more pleasant and efficient interaction among humans and machines. Qualitative data collection and analysis in combination with experimental design obtained new insights on the users’ perceptions and opinions about the modern VAs.

In essence, the outcomes of this work suggest that the speaking style of conversational VAs has indeed the power to influence the perceived quality of the HCI. The participants of this study showed intense favoritism over the idea of an assistant enhanced with humor. Additionally, in terms of personality, a distinctive and confident VA, who behaves in a professional manner, and inspires trust and calmness seems to be the most attractive. Also, users seem to appreciate the option of the VA adjusting the way it talks, depending on the context of use.

Overall, in this era of artificial revolution, and as the use of virtual characters in software increases, it is essential that designers work more with people's natural interpretation strategies. A deeper understanding of the role that these characters play in individuals’ reality, as they constantly evolve and improve, is the key to build the right affordances into VAs.

7. References

Bajarin, B. (2016, June 6). The Voice UI has Gone Mainstream. Retrieved from Tech.pinions: https://techpinions.com/the-voice-ui-has-gone-mainstream/46148

Beale, R., & Creed, C. (2009). Affective interaction: How emotional agents affecr users. International Journal of Human-Computer Studies, 755-776.

Becker, C., Kopp, S., & Wachsmuth, I. (2007). Why emotions should be integrated into conversational agents. Conversational Informatics.

Bernardet, U., Kang, S.-h., Feng, A., DiPaola, S., & Shapiro, A. (2017). A Dynamic Speech Breathing System for Virtual Characters. Intelligent Virtual Agents 17th International Conference, IVA 2017, Proceedings, 43-52.

(17)

Boros, T., Stam, A., Watts, O., & Dumitrescu, D. S. (2014). RSS-TOBI- A Prospdically Enchanced Romanian Speech Corpus. Proc. The 9th edition of the Language Resources and Evaluation Conference.

Brave, S., Nass, C., & Hutchinson, K. (2005). Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent. International Journal of Human-Computer Studies, 161-178.

Bryman, A. (2015). Social Research Methods. Oxford University Press.

Cabral, J. P., Cowan, R. B., Zibrek, K., & McDonnell, R. (2017). The Influence of Synthetic Voice on the Evaluation of a Virtual Character. Interspeech, 20-24. Castillo-Montoya, M. (2016). Preparing for Interview Research: The Interview Protocol

Refinement Framework. The Qualitative Rport, 811-831.

Crawford, S. A., & Caltabiano, N. J. (2011). Promoting emotional well-being through the use of humour. The Journal of Positive Psychology, 237-252.

Doce, T., Dias, J., Prada, R., & Paiva, A. (2010). Creating Individual Agents through Personality Traits. Intelligent Virtual Agents, 10th International Conference Proceedings, 257-264.

Dryer, C. D. (1999). Getting Personal With Computers: How to Design Personalities For Agents. Applied Arti® cial Intelligence, 13 : 273 295.

Fan, L., Scheutz, M., Lohani, M., McCoy, M., & Stokes, C. (2017). Do We Need Emotionally Intelligent Artificial Agents? First Results of Human Perceptions of Emotional Intelligence in Humans Compared to Robots. Intelligent Virtual Agents, 17th International Conference, IVA 2017, Stockholm, Sweden, August 27–30, 2017, Proceedings, 129-141.

Furnham, A., & Fudge, C. (2008). The Five Factor Model of Personality and Sales Performance. Journal od Individual Differences, 11-16.

Hebbalkar, S. (2017, January 30). Virtual Assistant Industry Statistics. Retrieved from Global Market Insights: https://gminsights.wordpress.com/tag/virtual-assistant-industry-statistics/

Horzyk, A., Magierski, S., & Miklaszewski, G. (2009). An Intelligent Internet Shop-Assistant Recognizing a Customer Personality for Improving Man-Machine Interactions. Recent Advances in Intelligent Informtion Systems, 13-26. Isbister, K., & Nass, C. (2000). Consistency of Personality in Interactive Characters:

verbal cues, non-verbal cues, and user characteristics. International Journal of Human-Computer Studies, Volume 53, Issue 2, p. 251-267.

Jabareen, Y. (2009). Building a Conceptual Framework: Philosophy, Definitions, and Procedure. International Journal of Qualitative Methods, 49-62.

Johnson, L. (2018, April 16). Digital Transformation; The Retail Industry Is Focusing on Customer Experience and Convenience to Win in Ecommerce. Retrieved from ADWEEK: https://www.adweek.com/digital/the-retail-industry-is-focusing-on-customer-experience-and-convenience-to-win-in-ecommerce/

(18)

Kelly, N. (2015). Interactions in affective computing: Sharing a haptic experience increases calmness and closeness. Kurosu M. (ed.): Human-Computer Interaction: Design and Evaluation, HCII 2015, 477-488.

King, S. (2014). Measuring a decade of progress in Text-to-Speech. Loquens 1(1). Lee, E.-J. (2010). The more humanlike, the better? How speech type and users’ cognitive

style affect social responses to computers. Computers in Human Behavior, 665-672.

Leite, I., Pereira, A., Mascarenhas, S., Martinho, C., & Prada, R. P. (2013). The influence of empathy in human–robot relations. International Journal of Human-Computer Studies, 250-260.

Maxwell, J. A., & Reybold, L. E. (2015). Qualitative Research. International Encyclopedia of the Social & Behavioral Sciences (Second Edition), 685-689. Mayer, E. R., Sobko, K., & Mautone, D. P. (2003). Social Cues im Multimedia Learning:

Role of Speaker's Voice. Journal of Educational Psychology, 419-425. McCrae, R. R., & John, P. O. (1992). An Introduction to the Five-Factor Model and Its

Applications. Journal of Personality, 175-215.

McGhee, P. E. (1999). Health, healing and the amuse system: Humor as survival training. Iowa: Kendal/Hunt Publishing.

Merriam, S. B. (2002). Introduction to qualitative research. Qualitative research in practice: Examples for discussion and analysis, 1-17.

Mitchell, W. J., Ho, C.-C., Patel, H., & MacDorman, K. F. (2011). Does social desirability bias favor humans? Explicit–implicit evaluations of synthesized speech support a new HCI model of impression management. Elsevier, 402-412. Moore, S. (2018, February 19). Gartner Says 25 Percent of Customer Service Operations

Will Use Virtual Customer Assistants by 2020. Retrieved from Gartner : https://www.gartner.com/newsroom/id/3858564

Mori, M., MacDorman, K. F., & Kageki, N. (2012). The Uncanny Valley [From the Field]. IEEE Robotics & Automation Magazine, Volume: 19 Issue: 2, p. 98 - 100.

Nagy, P., & Nemeth, G. (2016). Improving HMM speech syntheis of interrogative sentences by pitch track transformations. Speech Communication, 97-112. Nass, C., & Moon, Y. (2000). Machines and Mindlessness: Social Responses to

Computers. Journal of Social Issues, 81-103.

Nunnari, F., & Heloir, A. (2017). Generation of Virtual Characters from Personality Traits. Intelligent Virtual Agents, 17th International Conference, IVA 2017, Stockholm, Sweden, August 27–30, 2017, Proceedings, 301-314.

Payne, J., Szymkowiak, A., Robertson, P., & Johnson, G. (2013). Gendering the Machine: Preferred Virtual Assistant Gender and Realism in Self-Service. Springer, 106-115.

(19)

Picard, R. W. (1997). Affective Computing. M.I.T Media Laboratory Perceptual Computing Section Technical Report.

Picard, W. R., & Klein, J. (2002). Computers that recognise and respond to user emotion: theoretical and practical implications. Interacting with computers, 141-169. Rabionet, S. E. (2011). How I Learned to Design and Conduct Semi-Structured

Interviews: An Ongoing and Continuous Journey. The Qualitative Report, 563-566.

Reeves, B., & Nass, C. (1996). The Media Equation: How people treat computers, Television, and New Media Like Real People and Pleces. Bibliovault OAI Repository, the University of Chicago Press.

Richter, F. (2016, August 26). Statista. Retrieved from Statista: https://www.statista.com/chart/5621/users-of-virtual-digital-assistants/ Robinson, S., & Mendelson, L. A. (2012). A Quaitative Experiment: Research on

Mediated Meaning Construction Using a Hybrid Approach. Journal of Mixed Methods Research, 332-347.

Shamekhi, A., Czerwinski, M., & Mark, G. (2016). An Exploratory Study Toward the Preferred Conversational Style for Compatible Virtual Agents. Springer International Publishing, 40-50.

Treger, S., Sprecher, S., & Erber, R. (2013). Laughing and liking: Exploring the interpersonal effects of humor use in initial social interactions. European Journal of Social Psychology, 43, 532-543.

Van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., . . . Kavukcuoglu, K. (2016). WAVENET: A GENERATIVE MODEL FOR RAW AUDIO. Arxiv.

Witcher, B. (2015). The Business Case For Virtual Agents. Cambridge: Forrester Research.

Yaghoubzadeh, R., & Kopp, S. (2012). Toward a Virtual Assistant for Vulnerable Users: Designing Careful Interaction. Proceedings of the 1st Workshop on Speech and Multimodal Interaction in Assistive Environments, 13–17.

8. Appendix I 8.1. Coding Scheme

Code Grounded Code Groups

w.clear 15 All positive/Character alignment/Wanted traits

w.humor 14 All positive/Character alignment/Wanted traits

(20)

w.normal 11 SIT/All positive/Character alignment/Wanted traits

speed balance 10 Character alignment

accent 10 All negative/Voice traits-Negative

w.personality 10 SIT/All positive/Character alignment/Wanted traits

w.adaptability 9 Adjustment/All positive/Character

alignment/Wanted traits

w.natural 9 SIT/All positive/Character alignment/Wanted traits

v.pauses 8 All negative/Voice traits-Negative

human treatment 8 SIT/Character alignment

adjust-mood 7 Adjustment/Character alignment

creepy (too human) 7

human perception 7 SIT/Character alignment

w.personality-less 7 SIT/All positive/Character alignment/Wanted traits

w.helpful 7 All positive/Character alignment/Wanted traits

w.direct 7 All positive/Character alignment/Wanted traits

w.consistent 7 All positive/Character alignment/Wanted traits

relation/closeness 6 All negative

volume balance 6 Character alignment/Wanted traits

emotional response 6 SIT/Character alignment

w.human-like 6 SIT/All positive/Character alignment/\Wanted

traits

v.confident 6 All positive/Character alignment/Voice traits-Positive

robotic 5 SIT/All negative/Character alignment

v.hesitation 5 All negative/Character alignment/Voice traits-Negative

v.monotonous 5 All negative/Voice traits-Negative

w.relation-not much 5 SIT/All positive/Character alignment/Wanted traits

adjust-activity 5 Adjustment/Character alignment

similarity 5 SIT/Similarity/Complementarity/Character

alignment

w.trust 5 All positive/Character alignment/Wanted traits

samantha 5 Fictional Character/SIT/Character alignment

v.mumble/not clear 5 All negative/Voice traits-Negative w.real (like a real

person)

5 SIT/All positive/Character alignment/Wanted traits

w.support 5 All positive/Character alignmentWanted traits

w.confident 5 All positive/Character alignment/Wanted traits

v.annoying 4 All negativeVoice traits-Negative

(21)

v.respectful 4 All positive/Character alignment/Voice traits-Positive

w.warm 4 All positive/Character alignment/Wanted traits

w.polite 4 All positive/Character alignment/Wanted traits

w.personalized 4 All positive/Character alignment/Wanted traits

adjust-rush 4 Adjustment/Character alignment

w.instructive 4 All positive/Character alignment/Wanted traits w.subtle (depending) 4 Character alignment/Wanted traits

human=attractive 3 SIT/Character alignment

v.distracting (accent/intonation)

3 All negative/Voice traits-Negative w.neutral like current 3 Wanted traits

w.soothing 3 All positive/Character alignment/Wanted traits

will get used to it 3 Character alignment

v.controlling 3 All negative/Voice traits-Negative

v.irritating 3 All negative/Voice traits-Negative

v.unnatural 3 All negative/Wanted traits/Voice traits-Negative

w.efficient 3 All positive/Character alignment/Wanted traits

complementarity 2 SIT/ Similarity/Complementarity/Character alignment

comfortable 2 All positive/Character alignment

v.condescending 2 All negative/Character alignment/Voice traits-Negative

v.enthusiastic 2 All negative/Voice traits-Positive

v.flow 2 All positive/Character alignment/Voice

traits-Positive

alfred 2 Fictional Character/SIT/Character alignment

jarvis 2 Fictional Character/Character alignment

w.character 2 SIT/All positive/Character alignment/Wanted traits too human

=uncomfortable

2 All negative

v.aggressive 2 All negative/Voice traits-Negative

v.lazy 2 All negative/Voice traits-Negative

funny helps connection 1 SIT/All positive/Character alignment

Hal 9000 1 Fictional Character/Character alignment

v.angry 1 All negative/Voice traits-Negative

v.arrogant 1 All negative/Character alignment/Voice

traits-Negative

v.basic 1 Voice traits-Negative

(22)

v.controlled 1 All positive/Character alignment/Voice traits-Positive

v.dominant 1 All positive/Character alignment/Voice

traits-Positive/Voice traits-Negative

v.emphasis 1 Voice traits-Positive

v.slow 1 All negative/Voice traits-Negative

v.uncomfortable 1 All negative/Wanted traits/Voice traits-Negative w.approachable 1 All positive/Character alignment/Wanted traits

w.cheerful 1 All positive/Character alignment/Wanted traits

w.fluent 1 All positive/Character alignment/Wanted traits

w.friendly 1 All positive/Character alignment/Wanted traits

Conversation with Virtual Assistants; Searching for Deeper Meaning in Voice

Conversation with Virtual Assistants;

Searching for Deeper Meaning in Voice

Table of Contents