Beyond the stereotypes
Gender of conversational agents and the effects of
product type on trust, attitude
and patronage intention
Name: Ani Kusmenoglu Tuna Student Number: 12358835
Master’s Thesis Graduate School of Communication Master’s programme Communication Science
Supervisor: Hilde Voorveld Date: 31-01-2020 Word count: 6500
1 Abstract
Conversational agents and similar technological entities are becoming part of our daily lives. As their technological capabilities develop, they become more humanlike and we treat them as such. At the same time, humanness of these are entities are enhanced by certain gender assignments. These maybe in the form of audial or visual cues. There is evidence that the gender assignments are pre-dominantly female. This research tries to investigate whether the use of a specific gender has any persuasive implications. The study experiments with two conversational agents (male vs. female) in a virtual e-commerce environment. Moreover, the experiment is conducted within the context of either a highly masculine or a highly feminine product. Results showed that following the interaction with the conversational agents, the participants showed no difference in their evaluations of trust, attitude towards the agent, attitudes towards the website and patronage intention. Whether the agents were providing information about a gender congruent product or not, did not influence the user's perceptions. Implications are that in commercial usage, it might be possible to use either gender as
conversational agents, without any negative outcomes. A move towards 'gender-diversification' in conversational tools seems possible.
2 Introduction
"I'd blush if I could" is a recent report by UNESCO (2019) which draws attention to the way conversational agents are designed. The report reveals that majority of voice-based conversational agents are designed to be either exclusively female or are female in their default settings. The organization cautions about the negative implications of the gendered design of commercial voice-based conversational agents (e.g. Amazon's Alexa), in that such practice positions women as order taking, "docile helpers"; and when subjected to insults, their submissive replies may reinforce the gender bias in the society.
Following up on the aforementioned report, Feine , Gnewuch , Morana and Maedche (2019) have investigated whether the use of the female gender is also predominant in text-based conversational agents. Conducting an analysis on 1,375 such agents, they have
discovered that among the 64% percent that provide gender cues, most gender-specific names (76.94%), avatars (77.56%) and descriptions (67.40%) are female. These results confirm that, just as with voice-based conversational agents, a bias towards choosing the female gender in text-based conversational agents exists. Further analysis shows that this bias is most common in three domains: customer service, branded conversation, and sales ( Feine et al., 2019). It is possible that the gender choice in these domains is not biased or habitual, but purposeful, with valid persuasive implications for the companies. Whether there are justifiable reasons in the widespread use of a specific gender, from a commercial and practical point of view, begs investigation.
E-commerce sites are one of the most common implementers of text-based conversational agents. These sites aim to increase customer experience by providing
interactive tools. Presence of conversational agents has been shown to evoke social presence and increase customer trust (Gefen & Straub, 2004). Conversational agents are often used as product recommendation agents by such sites (Qiu & Benbasat, 2010). The embodiment of
3
certain cues help to humanize these agents and enhance the shopping experience of the customers. In the process of "humanizing" the conversational agents, a gender is assigned to the agent, almost by default. Whether the assigned gender of the conversational agent has persuasive implications in the frame of online shopping will be investigated in this study.
In doing so, the effect of the type of products will be examined. Humans have a tendency to assign masculine and feminine connotations to products (Golden, Allison & Clee, 1979). During our shopping experiences, gender of the salesperson becomes important, especially during purchase of products perceived as highly masculine or feminine (Foster & Resnick, 2013). Such customer expectations can also be relevant in online shopping.
Customers may prefer to interact with a specific type of agent when seeking a certain product. In turn, conversing with the right agent might evoke positive attitudes towards the agent and the website, as well as behavioral intentions.
The study will also look at the concept of trust as an underlying determinant of the relationship between interaction with an agent and the resulting persuasive outcomes. In the context of this study, trust is conceptualized as trust towards the conversational agent. Since the focus is on humanlike perception of the agents, trust is studied in two relevant
dimensions; affective (i.e. warmth) ) and cognitive (i.e. expertise) ( Koh&Sundar, 2010). It is argued that because of expectancies of certain attributes like knowledge and empathy,
receiving product recommendations from the matched gender could lead to greater trust which will lead to more positive attitudinal and intentional outcomes. These issues are formulated as follows:
RQ: In online shopping situations, to what extent does the assigned gender of a
conversational agent influence attitudes and behavioral intentions of the users towards the agent or the website? Does the product type in discussion influence the effect, and is this relationship mediated by trust?
4 Theoretical Background
Gender assignment to conversational agents
Conversational agents (CAs) can be described as "software that accepts natural language as input and generates natural language as output, engaging in a conversation with the user " (Griol, Carbo & Molina, 2013, p.760). Today, they are being used across a
spectrum of fields such as retail, education, healthcare as well as for various purposes such as answering questions, providing recommendations, training. Conversational agents can be categorised as embodied and disembodied. Embodied conversational agents engage in dialogue and use dynamic cues (i.e. movements) in real-time whereas disembodied conversational agents communicate with the user primarily via a messaging interface,
containing only static cues such as a profile picture (Araujo, 2018). In this context, a cue can be defined as signal that presents a source of information to the receiver. In the scope of this study, conversational agents (CAs) will refer specifically to text-based, disembodied
conversational agents (also often referred to in literature and practice as 'chatbots').
When communicating with conversational agents, people rely on certain cues just as they do with other humans. This is explained by the media-equation hypothesis which posits that computers are social actors and that people apply social rules and expectations to them (Reeves & Nass, 1996). In the presence of salient cues, people are more likely to treat conversational agents as they would a similar human. Among the possible cues, gender cue carries an essential importance since it is one of the first information people exchange with one another (Niculescu, Hofs, Van Dijk & Nijholt, 2010). In line with the media- equation hypothesis, placing gender cues (i.e. a pictorial representation or human photo) has been demonstrated to invoke gender stereotypes towards the conversational agents (Forlizzi, Zimmerman, Mancuso, & Kwak, 2007; Louwerse, Graesser, Lu & Mitchell,2005; Niculescu et al., 2010, Brahnam& De Angeli 2012).
5 Gender of products and gender stereotypes
Cognitive representations and normative beliefs determine how people perceive others in terms of gender and what is appropriate for others to do given their gender (Welle and Heilman, 2007). This is what we refer to as stereotyping. Stereotypes about gender are particularly powerful beliefs and difficult to change. (Brahnam & Angeli, 2012). In fact, our tendency to stereotype is so inherent that, we develop gender stereotypes for non-human entities as well. Assigning the female gender, for example, to transportation vehicles, such as ships or planes, is a common practice. Research has shown that people actually assign
genders to a whole variety of products, and even services (Fugate & Phillips, 2010; Golden, et. al, 1979).
The gendering of products and services has certain implications such that, it might have a carry-over effect on the preferred gender of the service giver, especially in retail. This is why certain sectors are “gendered”, meaning male employees work in shops that sell products with male connotations, i.e. car sales; and females are likely to be employed in selling products that are stereotypically female, i.e. cosmetics (Foster, 2004). Of course, this tendency of retailers' to recruit certain gendered salespersons is justified by the customers’ actions. In certain situations, customers show preference towards a specific gender when seeking help. For example, a study has shown that when shopping for cosmetic products, female customers do tend towards a female salesperson when seeking information about cosmetic products (Foster &Resnick, 2013).
As in actual shopping situations, people employ gender and product stereotypes to online interactions with conversational agents as well. Beldad, Hegner and Hoppen (2016) has shown that male and female agents were more favorably regarded when giving advice about gender congruent products and McDonell and Baxter (2019) have demonstrated that a male conversational agent was perceived more competent as a mechanic, a "masculine"
6
service, compared to a female agent. These findings support the Computers are Social Agents (CASA) paradigm, in that, people form a relationship with computers as they do with other humans and apply inter-personal social rules and scripts to them. (Nass, Steuer & Siminoff, 1994).
Trust and agent - product gender match
An underlying mechanisms that could explain consumers’ preference for a certain agent gender is trust. Trust is a multi-dimensional construct that plays an important role in customer interaction. In real life shopping situations, trust in the salesperson is an important factor in our purchase decisions and our perceptions of the brand or company. In the context of e-commerce settings, this need for a sense of trust is even higher, due to the uncertainty involved in online transactions (Chattaraman, 2012) and the lack of possibility to
immediately verify products and services (Gefen &Straub, 2004).
Under this uncertainty, the use of conversational agents, helps to induce social presence and thereby increase trust towards the website and the vendor (Almutairi &Rigas, 2014). However, the presence of the conversational tool is not likely to invoke positive attitudes, unless the tool (CA) itself is perceived as trustable. Trust in the agent can be categorised in two dimension: Knowledge‐driven, cognitive trust and emotion‐driven affective trust (Koh & Sundar, 2010). Cognitive trust is based on attributes such as expertise, credibility and reliability. It arises from an accumulated knowledge that allows one to make predictions about the likelihood that the person they are interacting with will live up to his/her obligations. (Koh & Sundar, 2010). These attributes are key factors during the customer - salesperson encounter. Expertise of the source is a cue that triggers heuristics in relation to the quality and credibility of information. (Sundar, 2007). The more the perceived credibility and expertise of the source, the more likely the customer will internalize the message and accept it without counter-arguments (Belch&Belch, 2009).
7
When online shopping, in the absence of any other cues, the expert source would be the conversational agent with the correct gender cues. As discussed in the previous sections, customers prefer the assistance of sales people with the correct gender, especially for highly gender stereotypical products. A similar inclination could be expected in an e-commerce setting when receiving a product recommendation.
H1a: Cognitive trust towards the agent is higher when the agent and product genders are matched than when the agent and product genders are not matched.
The second underlying mechanism, affective trust, is the emotional dimension of trust. It is based on how one feels about attributes such as empathy, warmth and openness during an interaction (Edell & Burke, 1987). It is also characterized by feelings of security and level of care and concern the partner demonstrates (Johnson & Grayson, 2005). Affective trust is closely related to the perception that a partner’s actions are intrinsically motivated (Rempel, Holmes & Zanna, 1985).
In an online recommendation setting, customers can be expected to feel that the agent shows more concern and is empathetic when the conversation is about a product that is closely related to the agent. Also the level of care and sense of security are likely to be enhanced during this interaction.
Different from the cognitive trust that measures perceptions of 'expertise' of the agent, affective trust is based more on heuristics; our conditioning about the gender scripts. Actual conversational quality of the online chat agent might not matter in evaluating the agents performance if the identity assigned to the agent already elicits stereotypical judgements. (Go&Sundar, 2019). Therefore when the agent is in the "correct setting", meaning when he/she is involved with a product that meets the gender expectations, affective responses in the receiving party will be activated through heuristics.
8
H1b: Affective trust is higher when the agent and product genders are matched than when the agent and product gender are not matched.
Elaboration Likelihood Model (ELM) proposes that people form and change their attitudes by processing information via a central route or a peripheral route (Petty,Cacioppo & Goldman,1981). The route is determined by a number of factors, such as ability,
motivation or involvement. When these factors are lacking, people rely on salient cues in the persuasion context, such as the attractiveness, credibility or expertise of the source (Petty, 1984). These attributes of the source can increase or decrease elaboration and ultimately influence attitudes.
If the message recipient is confident that a source will provide accurate information because of his or her high trustworthiness, they may forgo the effortful task of scrutinizing the message and, instead, accept the conclusion as valid (Priester and Petty,2003). Trust and credibility are the key elements in persuasive technology (Fogg, 2003). Perceptions of trustworthiness (cognitive and affective) of the source (agent or website) will result in more persuasive outcomes, such as positive attitudes and behavioral intentions Therefore, when making a product recommendation, an agent is perceived as trustable both in terms of
knowledge and characteristics, this will provide a salient cue and result in positive attitudes. Furthermore, a conversational agent can evoke certain heuristics that could influence the evaluation of the medium itself, as well as the organisation behind it. (Voorveld & Araujo, 2019). In the context of e-commerce, the experience of interacting with a CA could result in opinion formation towards the CA itself as well as the website it is used on.
H2a: Matched agent gender – product gender will result in higher positive attitude towards the agent mediated by cognitive trust
9
H2b: Matched agent gender – product gender will result in higher positive attitude towards the agent mediated by affective trust
H3a: Matched agent gender – product gender will result in higher positive attitude towards the website mediated by affective trust
H3b: Matched agent gender – product gender will result in higher positive attitude towards the website mediated by cognitive trust
Patronage intention can be defined the likelihood of a customer visiting and doing business with a vendor in the future.. In the marketing literature, general trust has been linked to willingness to make business transactions (Harrison McKnight, Choudhury& Kacmar, 2002). Trust with a vendor is closely linked to the trust in the mediating agent, the
salesperson. Literature shows that future sales opportunities depend mostly on trust in and satisfaction with the salesperson. (Crosby, Evans & Cowles,1990)
Research has shown that high levels of consumer trust encourage also increase online purchase intentions while the lack of it is the main reason individuals do not shoponline . (Gefen &Straub,2014). Furthermore, interactive features of a Web site motivate shoppers or browsers to revisit the Web site (Kolesar and Galbraith, 2000). Therefore, if the interaction with a mediating (conversational) agent, that the customer finds trustworthy will result in future likelihood of visiting and shopping in the website.
H4a: Matched agent gender – product gender will result in higher patronage intention towards the website mediated by affective trust
H4b: Matched agent gender – product gender will result in higher patronage intention towards the website mediated by cognitive trust
10 Figure 1: Conceptual Model
Method
Design and Sample
The hypotheses were tested by conducting an experimental study with a 2 (gender of agent: female vs. male) x 2 (product type: skincare set vs. whisky) between-subjects design. A total of 176 respondents participated in the study. They were recruited using Amazon Mechanical Turk, as well as convenience and snowball sampling through social media networks, e.g. Facebook, Whatsapp. After reviewing the data, participants who had not effectively utilized the chatbot and failed to get a recommendation or a conversation code (n= 14), quit the survey too early (n= 20), or did not give consent (n= 1) were excluded. The final sample consisted of 141 participants of which 51.8% were male, with age ranging from 18 to 68 (M = 36.59, SD = 8.47). 122 of the respondents had completed a higher education (college
11
level or above) while the remaining 19 were high school graduates. The nationalities of the participants varied, with highest numbers of participation from USA (n=54) and Turkey (n=34).
Stimuli Pre-test
The stimuli to be used for the experiment were selected by conducting a pre-test. A survey was conducted in order to determine which products and which agent visual cues were the most suitable. The participants (N= 27) were recruited by way of convenience sampling and using Mechanical Turk.
Product type: The respondents were asked to rate the masculinity and femininity of 25 products. All products were items that could be purchasable from a major e-commerce
website. A past study by Iver and Debevec (1984) served as a guide in the selection of certain products. Also inspired by the findings of past studies (Eyssel&Hegel,2012; Naas& Moon, 2005) products such as coding book and romantic comedy DVD were added. Participants were asked to rate how masculine or feminine they perceived each product on a 7-point Likert scale ( 1- extremely masculine and 7- extremely feminine). The results (Appendix A) showed that scotch whisky was perceived as the most masculine (M = 2.59, SD = 1.52) and skincare products were perceived as the most feminine (M= 5.93, SD = .96), paired sample t-test showing significant difference t (26) = 8.20, p < .001 in the scores. These two products were also in similar price categories, making them suitable for the experiment.
Gender of the agent: The pre-test was also used to select the avatars which would serve as agents' gender stimuli. Since attractiveness of the agents in the visual cues of chatbots could have an effect on the outcome variables (Yuksel, Collisson & Czerwinski, 2017;
Khan&Sutcliffe, 2014), the two conditions (male vs. female) needed to be controlled for perceived attractiveness. In order to achieve this, photos of 3 males and 3 females with
12
similar ages (30s), ethnicities (white), outfits (business-like) and facial expressions (smiling) were selected. Respondents were asked to rate how attractive they found each person on a 7-point Likert scale. The results of a series of paired sample t-tests, comparing the females against the males showed that the perceived attractiveness of "Female 2" (M=4.74, SD=1.58) and "Male 1" (M=4.93, SD=1.24) had the least mean difference among the other parings and were not significantly different from each other in terms of attractiveness, t(26)= -.50, p= .624, making them the most suitable stimuli for the two agent conditions.
Configuration of the chatbots
Using the Conversational Agent Research Toolkit (CART) framework (Araujo, 2018), two chatbots were created (Appendix F). Photos of the male and female personages determined by the pre-test were placed as avatars. Since names also evoke stereotypes (Sundar, 2019), the agents were assigned names with high female ('Kimberly') and male ('Thomas') connotations. Dialog management was done using DialogFlow, a service provided by Google. Within this tool, conversation flows were created for the two products that were selected after the pre-test. The flow design involved determining the options that the agent would present to the participants, and the actions to be taken in accordance with the user responses. Each conversation flow was designed to fit the characteristics of the specific products, but the scripts were kept as similar as possible in order to ensure internal validity. Procedure
The study started with the participants clicking on the invitation link and entering the survey platform, Qualtrics. In the opening page, they were asked to read and approve the informed consent in order to continue. Next, the participants answered a set of questions regarding their familiarity with chatbots and the specific product type they would be exposed to in the next step. Afterwards, the participants were presented a scenario in which they
13
would be getting a gift (either whisky or skincare set) with a budget of €50 and they needed advice to make a decision.
After the scenario text, they were exposed to a page in which they saw the homepage of an imaginary website (myvirtualstore.com) created for the purpose of this study. Under the webpage image, a chatbot frame was available to converse with the agent. Randomization of the agent gender condition (chatbot to be displayed) and the product type condition (scenario) were randomized in the survey platform.
Each assistant greeted the participant by introducing themselves and prompting the participant to start a conversation. After answering the agent’s questions, the participants were able to receive two product recommendations based on their criteria. After rounding up the chat, the participants were given a unique conversation code which they had to entered in a box provided. In addition, they were asked to confirm whether they had been able to get a recommendation. Following this, the respondents completed a survey about their opinion of the conversational agent and the website. Finally, a set of questions regarding their
demographic information were asked. Measures
Items with validated reliability from prior research were used in measuring the mediators and the dependent variables. In order to reflect the current experimental context, modifications were made, where needed. All variables except those related to attitudes were measured on a 7-point Likert scale (1 = “strongly disagree,” 7 = “strongly agree”). Attitude variables were measured using semantic scales, e.g. Good / Bad. Fill list of questions can be found in the survey (Appendix E).
Mediators
Two dimensions of trust were measured to be used as mediators. In order to measure affective trust towards the virtual assistant, seven items were adapted from Koh and Sundar
14
(2010). The participants rated statements such as: “The agent was empathetic”, “The agent was personal” (M = 4.61, SD = 1.33, α = .94). The second dimension, cognitive trust towards the agent was measured using a seven item scale again based on Koh and Sundar (2010), e.g,. “The virtual assistant was credible”, “The virtual assistant was professional”. All trust items were measured by a 7-point Liker scale. (M = 5.40, SD =1.12, α =.94).
Outcome variables
Attitudes: A semantic differential scale adapted from Becker-Olsen (2003) was used to measure attitudes towards the agent. A total of six items were included, e.g. “I found the virtual assistant positive/negative”, “I found the virtual assistant appealing/unappealing”. The scale was reliable (M = 5.46, SD = 1.15, α = .91). Attitude towards the website was also measured using the semantic differential scale from Becker-Olson (2003) with 5 items e.g. “I found the website good / bad”, “satisfactory / unsatisfactory”. Both measurements were done on a 7-point scale, where negative associations scored the highest point. The scores were reverse coded, before the analysis stage. (M = 5.43, SD = 1.36, α =.97)
Patronage intention: Future usage intention of the website and inclination to recommend it to others was measured by a four item scale adapted from Kim, Fiore & Lee (2007) and Baker, Parasuraman & Voss (2002), e.g. “I would visit this website again”, “I would recommend this website to others”. The measurement was done on a 7-point Likert scale. (M = 4.95, SD = 1.44, α =.96)
Control variables
In order to control for the effect of previous knowledge, 3 variables were measured. Product familiarity (skincare or whiskey), familiarity with chatbots, familiarity with using recommendation chatbots, were measured. Demographic information e.g. age, education were also asked.
15 Results
Randomization check
In order to test whether demographics of the participants were successfully distributed across the conditions, a randomization check was conducted. Results of Chi-squared tests for gender χ2 (3) = 1.13, p =.771 and education χ2 (3) = 0.91, p =.993, as well as a one-way ANOVA for age F (3,137) =.137, p= .938 showed that distribution across the conditions was successful.
Hypothesis testing
To test the hypotheses Hayes' PROCESS macro v.3.2 was used. Model 8 was selected to analyse the complete moderated parallel mediation model. The independent variable agent gender and moderator variable were dummy coded. Cognitive and affective trust were defined as mediators. Results of bivariate analysis showed that familiarity with
conversational agents and familiarity with recommendation agents correlated with all the mediators and dependent variables, therefore were entered as covariates. Bootstrapping was set at 5000. The test was repeated for each of the three dependent variables: attitude towards the agent, attitude towards the website and patronage intention.
Cognitive trust and affective trust: Results of the analysis showed that the model explaining cognitive trust was significant, R = .30, F(5, 135) = 2.65, p = .026, with 8.9% (R-squared = .089) of the variance explained. There was a no significant direct effect of the agent gender on cognitive trust (b = -.16, SE = 0.27, p = 0.561). The interaction between the agent gender and product type did not have a significant effect on cognitive trust (b = 0.26, SE = 0.37, p = 0.968).
The model for affective trust was significant, R =.35, F(5, 135) = 3.84, p = .003, with 12.4% (R-squares = .124) of the variance explained. There was no significant direct effect of agent gender on affective trust (b = 0.06, SE = 0.43, p = 0.892) and no significant interaction
16
effect of agent gender and product type on affective trust (b = 0.06, SE = 0.43, p = 0.892). H1a and H1b were rejected.
Table 1
Direct effects – Mediators
Cognitive Trust Affective Trust
Coefficient (SE) P t Coefficient (SE) p t Agent Gender -0.16 (0.27) 0.561 -0.58 -0.06 (0.31) 0.847 -0.19 Product type -0.14 (0.26) 0.611 -0.51 -0.20 (0.31) 0.519 -0.65 Interaction 0.26 (0.37) 0.492 0.69 0.06 (0.43) 0.892 0.14 Cognitive Trust - - - - Affective Trust - - - - Chatbot familiarity 0.16 (0.07)* 0.029 2.20 0.00 (0.08) 0.993 -0.01 Rec. bot familiarity 0.04 (0.07) 0.542 0.61 0.24 (0.08)* 0.002 3.12 * p < 0.05
Attitude towards the agent: The model measuring attitude towards the agent was significant, R =.57, F(7, 133) = 3.84, p < .001, with 32.5% (R-squared = .325) of the variance explained. The gender of the agent did not significantly influence attitudes towards it, (b = -0.08, SE = 0.24, p = 0.744). Interaction of the agent and product type did not have a direct effect on attitude towards the agent (b = 0.22, SE = 0.33, p = 0.516). A significant direct effect was found between cognitive trust and attitude towards the agent , The indirect effect results, b= .76, SE= .14, 95%CI [-.11, .43] showed there was no mediation effect however. Similarly, although affective trust had a significant effect on attitude towards the agent, indirect effect results confirmed there was no mediation effect of cognitive trust or affective trust on attitudes towards the agent (b= .01, SE= .11, 95%CI [-.19, .26]) . Therefore, H2a and H2b were both rejected.
Attitude towards the website: The model measuring attitude towards the website was significant, R =.72, F(7, 133) = 20.23, p < .001, with 51.6% (R-squared = .516) of the variance explained. Tests for this second outcome variable showed that interaction of agent and product type did not have a direct effect on attitude towards the website (b = 0.23, SE =
17
0.33, p = 0.485). Indirect effect results, b= .12, SE= .20, 95%CI [-.22, .58] showed there was no mediation effect. Significant direct effects were found for both cognitive and affective trust variables towards the dependent variable (Table 2). There were no significant indirect effect to show mediation, b= .02, SE= .16, 95%CI [-.30, .33]. Therefore, H3a and H3b were both rejected.
Patronage intention: The model measuring patronage intention was significant, R =.69, F(7, 133) = 16.89, p < .001, with 47.1% (R-squared = .471) of the variance explained. The gender of the agent did not have direct effect on patronage intention (b = 0.39, SE = 0.24, p = 0.744). Results showed that interaction of agent and product type did not have a direct effect (b = 0.39, SE = 0.37, p = 0.295). The indirect effect results, b= .09, SE= .16, 95%CI [-.17, .47] confirmed there was no mediation effects; a direct significant effect between cognitive trust and patronage intention existed (Table 2). Similarly, a significant effect of affective trust on attitude towards the website was found, but there was no indirect mediation effect, b= .03, SE= .19, 95%CI [-.34, .41]. Therefore, H4a and H4b were rejected.
Table 2
Direct effects – Dependent variables
Attitude towards Agent Attitude towards Website Patronage Intention Coefficient (SE) p t Coefficient (SE) p t Coefficient (SE) p t Agent Gender -0.08 (0.24) 0.744 -0.33 0.06 (0.24) 0.800 0.25 -0.08 (0.27) 0.767 -0.30 Product type 0.01 (0.17) 0.945 0.07 -0.03 (0.24) 0.909 -0.11 0.14 (0.26) 0.597 0.53 Interaction 0.22 (0.33) 0.516 0.65 0.23 (0.33) 0.485 0.70 0.39 (0.37) 0.295 1.05 Cognitive Trust 0.30 (0.09)* 0.001 3.25 0.47 (0.09)* 0.000 5.21 0.37 (0.10)* 0.000 3.62 Affective Trust 0.24 (0.08)* 0.002 3.09 0.36 (0.08)* 0.000 4.63 0.43 (0.09)* 0.000 4.91 Chatbot fam. 0.06 (0.07) 0.357 0.93 0.20 (0.07)* 0.003 3.01 0.13 (0.07) 0.082 1.76 Rec. bot fam. 0.04 (0.06) 0.495 0.69 -0.09 (0.06) 0.147 -1.46 0.03 (0.07) 0.666 0.43
Discussion
The aim of this research was to understand the effects of gender use in conversational agents in an e-commerce setting. It set out to investigate whether the agent gender has any
18
persuasive implications for the website it was operated on. In order to test these effects, the condition of product type was introduced together with the agent. Building upon the CASA paradigm (Reeves & Nass, 1994), the expectation was for the offline shopping experiences to be reflected to the online store as well. It was also assumed that the match between the gender of the product and the gender of the agent would elicit, cognitive and affective responses. In line with the Elaboration Likelihood Model (Petty & Ciappo, 1981) the heuristics of expertose, empathy and stereotyping would decrease counterarguing and positively influence trust and attitudes.
The results of the study, first of all, demonstrated that in the setting of a multi-product online store, the gender of the conversational agent has no overall effect on attitude towards the agent and the website. The gender of the agent also has no effect on patronage intention towards the website, which is a key measure for an online business.
Furthermore, the results of the study showed that the indifference to gender of conversational agents applied even in situations where product type was highly gender-stereotypical. Agent gender and product gender matches did not significantly differ on any of the trust, attitude or behavioral outcomes. These results were contrary to the expectations of the CASA paradigm (Reeves & Nass, 1996). Even with the knowledge of interacting with a machine, the prediction was that the users would apply their gender stereotypical attitudes towards the agents.
The results were contrary also to past studies on conversational agents (or similar entities) which demonstrated stereotypical behavior towards anthropomorphised agents (Nass, Moon and Green ,1997; Brahnam and De Angeli;2012; Eyssel &Hegel, 2012; Forlizzi, Zimmerman, Mancuso & Kwak, 2007). It is important to note that these studies did not specifically focus conversing with an agent in an e-commerce shopping situation. Still, a more relevant study (Beldad et al., 2016) which did experiment with an online store setting,
19
also found strong interaction effects between the agent and product gender congruence on outcomes such as trust and purchase intention.
There maybe several explanations for this disparity. First of all, the aforementioned study has used high involvement goods, such as motorcyles and solarium machines, as stimuli. In this current study, products were chosen from low involvement goods and price was further limited to €50. Enhanced trust and reduced perceived product risk are associated with attitudes and patronage intentions (Chattaraman, 2012). In this specific case, the low risk seems to have influenced the factors leading to the necessity of trust building, and eventually have no effect on behavioural intentions. Future studies could look at the gender- product matching effect in the framework of high vs. low involvement goods and the effect of perceived risk as an underlying determinant.
Another reason for the non-significant results in comparison with past studies may be due to the specific scenario: In order to increase the believability of the experiment, certain factors were taken into account: The participants were told that they had collected money with others, in order to but a gift for a friend. This was done for several reasons: Since the products were highly gender-stereotypical, the participants would be less convinced to seek a product associated for the opposite gender. Buying a gift for someone else controlled for the gender and other biases (i.e. not drinking alcohol) towards the products were told to buy the gift as a representative of a group to further distance them from the previous (‘I would never buy whisky for a friend’). These specific conditions may have influenced the perceived risk.
The results were also surprising in another aspect: During the debriefing process of participants who requested more information about the survey, many respondents stated that in an offline setting they would indeed seek help from assistants with the same gender as the product. These statements were in line with the qualitative study by Foster and Resnick (2014) who demonstrated that women would rely more on the knowledge of female store
20
assistants while buying beauty items. Although, the results showed that on average, female agent was indeed more positively evaluated when skincare product was concerned, the difference with other conditions was not significant. Hence, the results of this study, showed that gender expectations from offline sales experiences may not be parallel to online
experiences. Future studies could investigate what the underlying discrepancy between the explicit and implicit attitudes might be.
Implications
This study has contributed empirical evidence to the discussion of gender use in conversational agents. From a commercial point of view, there seems to be no benefit of using one gender over the other in terms of persuasive outcomes; a male agent is perceived as trustworthy as a female agent when giving recommendations. More importantly, the
customers are likely to have similar attitudes and behavioral intentions towards the website. Therefore, designers and managers of websites can consider diversifying the genders that they assign to their conversational tools. At the very least, similar experiments or tests could be conducted in order to decide what the most effective choice will be. In fact, it could be left to the customers to decide which conversational agent they would like to interact with. Such customisation functions can be integrated in the conversational tools.
From the societal point of view, the study provides support for the argument that the widespread use of female conversational agents may be more intuitive rather than rational. Results have shown that there is no bias on the receiver (customer) side which implicates that the bias could be inherent in the source. The decision makers may consciously or
21 Limitations
The design of the experiment was set up in a way that participants could converse with the chatbot in a very structured framework. This has prevented the participants from deviating from the script, but at the same time restricted them from freely interacting with the agent. A more flexible formatted interaction could have helped in more nuanced attitude formation. As efficient as the dialog format was, it may have been perceived as too pre-scripted and resulted in similar evaluations across the different conditions.
The sample was balanced between snowballing and online survey platform however this did not ensure a diversified group in terms of education. 87% of the participants had completed university education or above. Since level of education is a factor in technology adoption (Czaja,et al.,2006) highly educated people may have relied on their knowledge about the algorithmic system behind the chatbots, suppressing the effects of artificially assigned gender cues. This may have caused the non-significant results between the male and female agents.
Finally, the product stimuli of the study may have caused certain issues. Certain stimuli may not work for specific groups of participants; for example, cosmetics may not work for male participants due to lack of familiarity or interest (Meyvis et al.,2018 ). Such limitations are of course unavoidable in an experiment of this context. Similarly, some participants may have preconceptions about alcoholic drinks. This could have effected their evaluations. In the future, a similar study could be conducted with different types of products.
Conclusion
The discussions around, gender and technology seem to be growing. These discussions are very positive in that they make us aware of our inherent biases. As the
technology develops at a rapid state and the technological entities become more humanlike, it is important that we make the right design decisions. During this process, it is important that
22
the arguments are backed with empirical evidence. Although this study has provided some evidence, it is by no means sufficient to make a judgement. It would be beneficial for future studies to continue to focus on this important issue and provide guidance to businesses and public in general. These evidences can be used to draw ethical guidelines for more diversified and conscientious design decisions. help designers ensure they don’t knowingly or
unknowingly perpetuate gender inequality when building chatbots towards a more diverse and inclusive with integrated technology.
23 References
Alves, António & Soares, Ana. (2013). Evaluating the use of avatars in e-commerce. Revista Portuguesa de Marketing. 16. 37-52.
Angeli, A. D. and Brahnam, S. (2006), Sex Stereotypes and Conversational Agents. Gender and Interaction: Real and Virtual Women in a Male World Workshop paper, 1-4. Araujo, T.: Conversational agent research toolkit: an alternative for creating and managing
chatbots for experimental research. (2019). https://doi.org/10.31235/osf.io/9ukyf Araujo, T.: Living up to the chatbot hype: the influence of anthropomorphic design cues and
communicative agency framing on conversational agent and company perceptions. Comput. Hum. Behav. 85, 183–189
(2018). https://doi.org/10.1016/j.chb.2018.03.051CrossRefGoogle Scholar
Araujo, T. (2018). Conversational Agent Research Toolkit (CART) (Revision 79e8e634) [Software]
Bagozzi, R. P., Gürhan-Canli, Z., & Priester, J. R. (2002). Applying social psychology. The social psychology of consumer behaviour. Open University Press. Retrieved from http://books.google.com
Beldad, Ardion & Hegner, Sabrina & Hoppen, Jip. (2016). The effect of virtual sales agent (VSA) gender - Product gender congruence on product advice credibility, trust in VSA and online vendor, and purchase intention. Computers in Human Behavior. 60. 62-72. 10.1016/j.chb.2016.02.046.
Belch, G. & Belch, M. (2009). Advertising and Promotion. An Integrated Marketing Communication Perspective.
Chattaraman, V., Kwon, W.-S., & Gilbert, J. E. (2012). Virtual agents in retail web sites: Benefits of simulated social interaction for older users. Computers in Human Behavior, 28(6), 2055–2066. https://doi.org/10.1016/j.chb.2012.06.009
24
Crosby, L., Evans, K., & Cowles, D. (1990). Relationship Quality in Services Selling: An Interpersonal Influence Perspective. Journal of Marketing, 54(3), 68–81.
https://doi.org/10.2307/1251817
Czaja, Sara & Charness, Neil & Fisk, Arthur & Hertzog, Christopher & Nair, Sankaran & Rogers, Wendy & Sharit, Joseph. (2006). Factors Predicting the Use of Technology: Findings From the Center for Research and Education on Aging and Technology Enhancement (CREATE). Psychology and aging. 21. 333-52. 10.1037/0882-7974.21.2.333.
Eagly, Alice & Wood, Wendy. (2012). Social role theory. Handbook of theories in social psychology. 2. 458-476. 10.4135/9781446249222.n49.
EYSSEL, F. and HEGEL, F. (2012), (S)he's Got the Look: Gender Stereotyping of Robots1. Journal of Applied Social Psychology, 42: 2213-2230.
doi:10.1111/j.1559-1816.2012.00937.x
Feine, J., Gnewuch, U., Morana, S., & Maedche, A. (2019). A Taxonomy of Social Cues for Conversational Agents. International Journal of Human - Computer Studies, 132, 138– 161. https://doi.org/10.1016/j.ijhcs.2019.07.009
Feine J., Gnewuch U., Morana S., Maedche A. (2020) Gender Bias in Chatbot Design. In: Følstad A. et al. (eds) Chatbot Research and Design. CONVERSATIONS 2019. Lecture Notes in Computer Science, vol 11970. Springer, Cham
Forlizzi, J., Zimmerman, J., Mancuso, V., & Kwak, S. (2007). How Interface Agents Affect Interaction Between Humans and Computers. In : DPPI ’07, Proceedings of the 2007 Conference on Designing Pleasurable Products and Interfaces (pp. 209-221). New York, NY, USA: ACM.
25
Foster, C. & Resnick, S.(2013). “Service Worker Appearance and the Retail Service
Encounter: The Influence of Gender and Age.” The Service Industries Journal 33, no. 2 (February 1, 2013): 236–247.
Foster, C. (2004). “Gendered Retailing: a Study of Customer Perceptions of Front-Line Staff in the DIY Sector.” International Journal of Retail & Distribution Management 32, no. 9 :442–447.
Fugate, Douglas & Phillips Melancon, Joanna. (2010). Product gender perceptions and antecedents of product gender congruence. Journal of Consumer Marketing. 27. 251-261. 10.1108/07363761011038329.
Gefen, D. and Straub, D.W. (2004) Consumer Trust in B2C E-Commerce and The Importance of Social Presence: Experiments in E-Products and E-Services. The International Journal of Management Science, 32, 407-424.
Go, Eun, and Sundar, S. Shyam. “Humanizing Chatbots: The Effects of Visual, Identity and Conversational Cues on Humanness Perceptions.” Computers in Human Behavior 97 (2019): 304–316. Web.
Golden,L.L., Allison, N., & Clee,M. (1979). The Role Of Sex Role on Self-Concept in Masculine and Feminine Product Perceptions. Advances in Consumer Research. 6(1)599-605
Griol, David, Carbó, Javier, and Molina, José M. “An Automatic Dialod Simuation Technique to Develop and Evaluate Interactive Conversational Agents” Applied Artificial
Intelligence 27.9 (2013): 759–780. Web.
Gulz, A., Ahlnér, F., Haake, M., Paiva, A., Prada, R., & Picard, R. (2007). Visual femininity and masculinity in synthetic characters and patterns of affect. Lecture Notes In Computer Science (Affective Computing And Intelligent Interaction), 4738/2007, 654–665.
26
Harrison Mcknight, D., Choudhury, V., & Kacmar, C. (2002). The impact of initial consumer trust on intentions to transact with a web site: a trust building model. Journal of
Strategic Information Systems, 11(3-4), 297–323. https://doi.org/10.1016/S0963-8687(02)00020-3
Johnson, D., & Grayson, K. (2005). Cognitive and affective trust in service relationships. Journal of Business Research, 58(4), 500–507.
https://doi.org/10.1016/S0148-2963(03)00140-1
Kim, J., Fiore, A., & Lee, H. (2007). Influences of online store perception, shopping enjoyment, and shopping involvement on consumer patronage behavior towards an online retailer. Journal of Retailing and Consumer Services, 14(2), 95–107.
https://doi.org/10.1016/j.jretconser.2006.05.001
Koh, Yoon & Sundar, S. Shyam. (2010). Effects of specialization in computers, web sites, and web agents on e-commerce trust. Int. J. Hum.-Comput. Stud.. 68. 899-912. 10.1016/j.ijhcs.2010.08.002.
Kolesar, M. and Wayne Galbraith, R. (2000), "A services‐marketing perspective on e‐ retailing: implications for e‐retailers and directions for further research", Internet Research, Vol. 10 No. 5, pp. 424-438. https://doi.org/10.1108/10662240010349444 Louwerse, M. M., Graesser, A. C., Lu, S. L., & Mitchell, H. H. (2005). Social cues in
animated conversational agents. APPLIED COGNITIVE PSYCHOLOGY (19:6), pp. 693-704.
Marian McDonnell, David Baxter, Chatbots and Gender Stereotyping, Interacting with Computers, Volume 31, Issue 2, March 2019, Pages 116–
27
Marian McDonnell, David Baxter, Chatbots and Gender Stereotyping, Interacting with Computers, Volume 31, Issue 2, March 2019, Pages 116–
121, https://doi.org/10.1093/iwc/iwz007
Mcknight, D.,Choudhury, V. & Kacmar, C. (2002). The Impact of Initial Consumer Trust on Intentions to Transact with a Web Site: A Trust Building Model. The Journal of
Strategic Information Systems. 11. 297-323. 10.1016/S0963-8687(02)00020-3. Meyvis, T., Van Osselaer, S., Darren, D., Fischer, E., Johar, G., & Morwitz, V. (2018).
Increasing the Power of Your Study by Increasing the Effect Size. Journal of Consumer Research, 44(5), 1157–1173. https://doi.org/10.1093/jcr/ucx110
Nass, C.,Moon, Y., & Green, N. (1997). Are Machines Gender Neutral? Gender‐Stereotypic Responses to Computers With Voices. Journal of Applied Social Psychology, 27(10), 864–876. https://doi.org/10.1111/j.1559-1816.1997.tb00275.x
Nass, Clifford & Steuer, Jonathan & Siminoff, Ellen. (1994). Computer are social actors. Conference on Human Factors in Computing Systems - Proceedings. 204.
10.1145/259963.260288.
Niculescu, A. I., Hofs, D. H. W., van Dijk, E. M. A. G., & Nijholt, A. (2010). How the agent’s gender influence users’ evaluation of a QA system. In F. Mohd Saman, N. L. Md Noor, H. Haron, A. Yeo, & A. Edwards (Eds.), Proceedings of the International
Conference on User Science and Engineering 2010 (pp. 16-20). USA: IEEE. https://doi.org/10.1109/IUSER.2010.5716715
Payne J., Szymkowiak A., Robertson P., Johnson G. (2013) Gendering the Machine:
Preferred Virtual Assistant Gender and Realism in Self-Service. In: Aylett R., Krenn B., Pelachaud C., Shimodaira H. (eds) Intelligent Virtual Agents. IVA 2013. Lecture Notes in Computer Science, vol 8108. Springer, Berlin, Heidelberg
28
Petty, R. E., Cacioppo, J. T., & Goldman, R. (1981). Personal involvement as a determinant of argument-based persuasion. Journal of Personality and Social Psychology, 41(5), 847–855. https://doi.org/10.1037/0022-3514.41.5.847
Priester, J., & Petty, R. (2003). The Influence of Spokesperson Trustworthiness on Message Elaboration, Attitude Strength, and Advertising Effectiveness. Journal of Consumer Psychology, 13(4), 408–421. https://doi.org/10.1207/S15327663JCP1304_08 Qiu, L., & Benbasat, I. (2010). A study of demographic embodiments of product
recommendation agents in electronic commerce. International Journal of Human - Computer Studies, 68(10), 669–688. https://doi.org/10.1016/j.ijhcs.2010.05.005 Rabia Fatima Khan & Alistair Sutcliffe (2014) Attractive Agents Are More
Persuasive, International Journal of Human–Computer Interaction, 30:2, 142-150, DOI: 10.1080/10447318.2013.839904
Reeves, Byron & Nass, Clifford. (1996). The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Pla. Bibliovault OAI Repository, the University of Chicago Press.
Rempel, John K. ; Holmes, John G. ; Zanna, Mark P. Journal of Personality and Social Psychology, 1985, Vol. 49(1), pp. 95-112
Sundar, S. (2007). The MAIN Model : A Heuristic Approach to Understanding Technology Effects on Credibility.
UNESCO (2019) “I’d Blush if I Could” unesco.org
Voorveld, H.A.M. & Araujo, T. (2019) Making Conversations?. Manuscript submitted for publication.
Welle, B., & Heilman, M. (2007). The exclusion of women from full engagement in the workplace: The role of gender stereotypes. In D. Steiner, S. Gilliland, & D. Starlicki (Eds.), Research in social issues in management (Vol. 5)
29
Yuksel, B. F., Collisson, P., & Czerwinski, M. (2017). Brains or Beauty: How to Engender Trust in User-Agent Interactions. ACM Trans. Internet Technol. (17:1), pp. 2:1-2:20, from http://doi.acm.org/10.1145/2998572.
30 Appendix A
Gender evaluations of the product types (1=Extremely masculine, 7 = Extremely feminine)
Product M SD Skincare products 5.93 0.96 Flower bouquet 5.70 1.24 Romantic comedy DVD 5.59 1.25 Jewellery 5.59 1.42 Perfume 5.33 1.21 Cookbook 4.93 1.14 Curtains 4.89 1.31 Food processor 4.70 1.33 Baby stroller 4.70 1.38 Vacuum cleaner 4.67 1.18 Clothes dryer 4.67 1.00 Espresso machine 4.33 1.21 Wine 4.30 1.30 Smartphone 4.22 1.25 Coat 4.15 1.32 Furniture 4.00 0.96 Camera 3.85 1.10 Bicycle 3.81 1.08 Sneakers(sports shoe) 3.74 1.35
Book about coding 3.52 1.58
Watch 3.48 1.45
Drone 3.00 1.54
Game console (Playstation) 2.63 1.36
DVD about sports 2.63 1.64
31 Appendix B
Mean and standard deviation values for agent genders
N Cognitive Trust-Mediator Affective Trust-Mediator Attitude towards Agent Attitude towards Website Patronage Intention M SD M SD M SD M SD M SD Female 71 5.42 1.10 4.61 1.25 5.45 1.18 5.37 1.37 4.91 1.58 Male 70 5.38 1.14 4.62 1.41 5.47 1.12 5.48 1.35 4.99 1.30 Total 141 5.40 1.12 4.61 1.33 5.46 1.15 5.42 1.36 4.95 1.44
32 Appendix C
Mean and standard deviation values for product gender – agent gender pairings (mediators)
Cognitive Trust Affective Trust
N M SD M SD
Feminine product -Female agent 31 5.50 1.09 4.81 1.20 Masculine product -Male agent 36 5.42 1.06 4.55 1.41 Masculine product -Female agent 40 5.36 1.11 4.45 1.28 Feminine product -Male agent 34 5.34 1.24 4.69 1.43
33 Appendix D
Mean and standard deviation values for product gender – agent gender pairings (dependent variables)
Attitude towards Agent
Attitude towards
Website Patronage Intention N M SD M SD M SD Feminine product-Female agent 31 5.59 1.17 5.45 1.40 4.95 1.58 Masculine product-Male agent 36 5.52 1.03 5.56 1.39 5.22 1.16 Masculine product-Female Agent 40 5.34 1.20 5.31 1.36 4.88 1.59 Feminine product-Male agent 34 5.42 1.22 5.41 1.32 4.75 1.42
34 Appendix E
Survey
Dear Participant,
I would like to invite you to participate in a research study to be conducted under the auspices of the Graduate School of Communication, a part of the University of Amsterdam. This online survey is a study about communication with virtual assistants. You will be asked to chat with a virtual
assistant after reading a short scenario about an online shopping situation. Afterwards, you will be asked a set of questions about your chat experience. Finally, we will ask you to provide some information about your demographics such as your age and gender.
As this research is being carried out under the responsibility of the ASCoR, University of Amsterdam, we can guarantee that:
1) Your anonymity will be safeguarded, and that your personal information will not be passed on to third parties under any conditions, unless you first give your express permission for this.
2) You can refuse to participate in the research or cut short your participation without having to give a reason for doing so. You also have up to 7 days after participating to withdraw your permission to allow your answers or data to be used in the research.
3) Participating in the research will not entail your being subjected to any appreciable risk or discomfort, the researchers will not deliberately mislead you, and you will not be exposed to any explicitly offensive material.
For more information about the research and the invitation to participate, you are welcome to contact Ani Kusmenoglu Tuna at any time. Should you have any complaints or comments about the course of the research and the procedures it involves as a consequence of your participation in this research, you can contact the designated member of the Ethics Committee representing ASCoR, at the following address: ASCoR Secretariat, Ethics Committee, University of Amsterdam, Postbus 15793, 1001 NG Amsterdam; 020‐525 3680; ascor‐secr‐fmg@uva.nl. Any complaints or comments will be treated in the strictest confidence.
I hope that we have provided you with sufficient information. I would like to take this opportunity to thank you in advance for your assistance with this research, which we greatly appreciate.
Kind regards,
35
Informed consent
I hereby declare that I have been informed in a clear manner about the nature and method of the research, as described in the email invitation for this study.
I agree, fully and voluntarily, to participate in this research study. With this, I retain the right to withdraw my consent, without having to give a reason for doing so. I am aware that I may halt my participation in the experiment at any time.
If my research results are used in scientific publications or are made public in another way, this will be done such a way that my anonymity is completely safeguarded. My personal data will not be passed on to third parties without my express permission.
If I wish to receive more information about the research, either now or in future, I can contact ani.kusmenoglutuna@student.uva.nl. Should I have any complaints about this research, I can contact the designated member of the Ethics Committee representing the ASCoR, at the following address: ASCoR secretariat, Ethics Committee, University of Amsterdam, Postbus 15793, 1001 NG
Amsterdam; 020‐ 525 3680; ascor‐secr‐fmg@uva.nl.
o I agree to participate and I acknowledge that I have read and understood the text above and that I am at least 18 years of age.
36
Please read the text carefully and start a conversation on the chat window on the next page.
A friend's birthday is coming up. After some discussion, you and a couple of other friends have decided to buy a skincare set as a present. Everybody pitched in and a total of 50 euros was collected.
You have been given the task of purchasing the gift. You think there might be more options online, so you are visiting an e-commerce website. You need some help with choosing the right skincare set and decide to chat with the website's virtual assistant.
40
You are almost done! Only a few sets of questions to go.
Please answer the following questions as though you had just visited the website providing the chatbot service.
43
Thank you very much for taking part in my survey. This study focuses on how different gendered virtual assistants influence our attitudes and intentions during our online shopping experiences. If you are curious about the results of this research, you can contact me in a few weeks to find out. Thank you once again and a happy new year.
44 Appendix F
