On-Road and Online Studies to Investigate Beliefs and
Behaviors of Netherlands, US and Mexico Pedestrians
Encountering Hidden-Driver Vehicles
Jamy Li
j.j.li@utwente.nl University of TwenteRebecca Currano
bcurrano@stanford.edu Stanford UniversityDavid Sirkin
sirkin@stanford.edu Stanford UniversityDavid Goedicke
dg536@cornell.edu Cornell TechHamish Tennent
ht@mynameishamish.com Cornell UniversityAaron Levine
aaronclevine@gmail.com Stanford UniversityVanessa Evers
v.evers@utwente.nl University of TwenteWendy Ju
wendyju@cornell.edu Cornell TechFigure 1: From left: Original ghostdriver study vehicle; Vehicle used in this work (Study 1); Driver in car seat costume in vehicle; Driver in car seat costume outside vehicle; Still from video stimuli (Study 2).
ABSTRACT
A growing number of studies use a “ghost-driver” vehicle driven by a person in a car seat costume to simulate an autonomous vehicle. Using a hidden-driver vehicle in a field study in the Netherlands, Study 1 (N = 130) confirmed that the ghostdriver methodology is valid in Europe and confirmed that European pedestrians change their behavior when encountering a hidden-driver vehicle. As an important extension to past research, we find pedestrian group size is associated with their behavior: groups look longer than single-tons when encountering an autonomous vehicle, but look for less time than singletons when encountering a normal vehicle. Study 2 (N = 101) adapted and extended the hidden-driver method to test whether it is believable as online video stimuli and whether car characteristics and participant feelings are related to the be-liefs and behavior of pedestrians who see hidden-driver vehicles. As expected, belief rates were lower for hidden-driver vehicles seen in videos compared to in a field study. Importantly, we found
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.
HRI ’20, March 23–26, 2020, Cambridge, United Kingdom © 2020 Association for Computing Machinery. ACM ISBN 978-1-4503-6746-2/20/03. . . $15.00 https://doi.org/10.1145/3319502.3374790
noticing no driver was the only significant predictor of belief in car autonomy, which reinforces prior justification for the use of the ghostdriver method. Our contributions are a replication of the hidden-driver method in Europe and comparisons with past US and Mexico data; an extension and evaluation of the ghostdriver method in video form; evidence of the necessity of the hidden driver in creating the illusion of vehicle autonomy; and an extended anal-ysis of how pedestrian group size and feelings relate to pedestrian behavior when encountering a hidden-driver vehicle.
CCS CONCEPTS
• Human-centered computing → Field studies; Empirical stud-ies in HCI; Empirical studstud-ies in interaction design.
KEYWORDS
Hidden-Driver Vehicles; Autonomous Vehicles; Self-Driving Vehi-cles; Pedestrians; Cars; Wizard-of-Oz; Human-Robot Interaction.
ACM Reference Format:
Jamy Li, Rebecca Currano, David Sirkin, David Goedicke, Hamish Tennent, Aaron Levine, Vanessa Evers, and Wendy Ju. 2020. On-Road and Online Studies to Investigate Beliefs and Behaviors of Netherlands, US and Mexico Pedestrians Encountering Hidden-Driver Vehicles. In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’20), March 23–26, 2020, Cambridge, United Kingdom. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3319502.3374790
1
INTRODUCTION
A growing number of researchers are studying how pedestrians interact with autonomous vehicles (AVs), given that pedestrians represent 22% of all traffic fatalities [21], that confusion about right-of-way (also called “crossing priority”) can cause pedestrian acci-dents [14, 22], that acciacci-dents are more likely as pedestrian crossing time increases [15], and that pedestrians do not get an opportunity to opt-in or out of autonomous vehicles the way that a human passenger might. As acquiring a real autonomous vehicle is diffi-cult, Rothenbücher et al. [25] proposed a method to gather data on pedestrian-autonomous vehicle interaction in the wild using a normal vehicle driven by a human “ghost driver” hidden in a car seat costume. This method has subsequently been used in Mexico [7]. However, there has not yet been work that confirms the method extends to Europe by comparing how successfully the ghost-driver methodology can create a “false belief” of an autonomous vehicle across regions, which could add evidence to its effectiveness.
A possible extension of the ghostdriver method is to use videos of hidden-driver vehicles in online studies. An initial exploration and validation of video-based ghostdriver studies could enable additional research avenues that utilize advantages of online-based research, such as access to different populations.
Although HRI has a long history of Wizard-of-Oz’d robots [24], few studies have explored parameters that influence belief rates in a wizarded system’s autonomy. What characteristics of a hidden-driver vehicle (i.e., its sensors, decals, motion or seemingly absent driver) lead people to believe it is autonomous? A new analysis of whether a hidden driver or other car characteristics convince viewers of a vehicle’s autonomy would reinforce prior justification for using the ghostdriver method.
Moreover, past field studies have noted atypical behavior such as staring and stopping when pedestrians encounter hidden-driver vehicles [25] (similar behavior has been observed with non-car-shaped robots [16, 28, 29]). However, these past works typically don’t assess potential explanations for people’s behavior, such as their feelings about the vehicle or mimicking others in a group. An analysis of how pedestrians’ feelings and group size relate to atypical behavior when encountering autonomous vehicles can advance the overall understanding of pedestrian-AV interaction.
Our contributions are: (1) a replication to extend on-road ghost-driver studies from US and Mexico to the Netherlands and com-parisons across those regions; (2) an adaptation of the ghostdriver method to an online video study and a comparative assessment with an on-road study; (3) an extended analysis of groups versus singletons; (4) empirical evidence that noticing a hidden driver in-creases people’s belief that a vehicle is autonomous; and (5) new findings about how feelings influence perceptions of and behavior with an autonomous vehicle.
2
BACKGROUND
2.1
Field Studies about Pedestrian Interaction
with Autonomous Vehicles
Field studies on pedestrian-AV interaction have primarily used vis-ible driver vehicles and focused on intent communication. [19] conducted a field study in which participants crossed the road in
front of an autonomous vehicle that had a visible driver. Partici-pants looked longer and crossed more confidently when an intent communication system was used compared to not used. [6] con-ducted a field study in which participants waited at a pre-defined crossing location facing away from a non-autonomous vehicle that drove toward them. At 7 seconds until the vehicle’s arrival at the pedestrian’s position, an experimenter told them to turn to face the vehicle and indicate when they think it would be safe to cross. They found no differences in the time it took pedestrians to make a decision to cross when a display was used versus not used. We note that most pedestrian-AV lab studies also focus on intent com-munication: for example, [20] and [11] explore how AVs could use gestures projected onto windshields, robots, special headlights or shape-changing chassis to communicate intent. These studies do not assess whether participants believe a vehicle is autonomous.
Field studies with hidden-driver vehicles have explored natural-istic pedestrian responses to a seemingly autonomous vehicle. [25] used a hidden-driver vehicle in the US and found naïve pedestrians believed the vehicle was autonomous and crossed the intersection without difficulty. [7] conducted a similar study with a hidden-driver vehicle in two cities in Mexico and found people in the coastal city stopped more often in front of the car than people in the large metropolitan city. However, past studies were not con-ducted in Europe and did not compare belief rates across studies.
Research Question 1: Will pedestrians in the Nether-lands have similar belief rates as pedestrians in the US California Bay Area or in Mexico?
2.2
Video Studies with Autonomous Vehicles
Video studies are an increasingly popular method to evaluate au-tonomous vehicle design. [31] used short videos of an auau-tonomous vehicle to assess different intent communication systems in a large online study. [4] similarly used videos of autonomous vehicles in their lab study on intent communication systems. Some benefits of video studies are reduced cost and access to larger groups of participants [30].
Prior research has suggested that results obtained using video studies are comparable to in-person studies. [30] compared video and in-person lab studies of a robot approaching a person and found high agreement in results between video and online. Past work in autonomous vehicles found that a car simulator is a valid method to study people’s driving [2]. These works, however, did not assess whether people believe an autonomous vehicle is autonomous when viewing it in a video compared to live in-person.
Research Question 2: Will people who view videos of a hidden-driver vehicle online have similar belief rates as pedestrians who see the vehicle in person?
2.3
Pedestrian Groups and Their Behavior
Pedestrian group size has a range of effects on behavior. Pedestrians talking in groups cross roads more slowly than individuals [28]. [12] found that pedestrians are more likely to stop and abort their crossing due to uncertainty about the timing of their crossing when they are in groups compared to in singletons. [27] found that pedes-trians who encountered a mobile robot make deviations around the
robot more often when they are in groups than alone, perhaps be-cause cautiousness spreads in the group [1]. However, these works did not study pedestrians encountering an autonomous vehicle. We would anticipate that conversation and uncertainty (and con-sequently crossing time) would be likely to occur when groups of pedestrians encounter an autonomous vehicle.
Hypothesis 1: Groups encountering an autonomous vehicle will take longer to cross than singletons.
People in groups could look longer at an autonomous vehicle for two reasons. People automatically mimick the emotions and nonverbal behavior of other group members [1], so one person’s increased attention toward an autonomous vehicle could lead to propagation of visual attention. [13] observed this mimicry in non-acquainted crowds where confederates suddenly stare at a distant building and others follow suit. Another reason may be that ac-quainted members of a group sometimes talk to each other when seeing a novel robot in the field [16]. Although past work did not use an autonomous vehicle, we anticipate propagation of visual at-tention or conversation to increase looking time when pedestrians encounter a novel vehicle as well.
Hypothesis 2: Groups will look longer at an autonomous vehicle than singletons.
2.4
Cues of Intent and Autonomy in
Autonomous Vehicles
Past work that has looked at autonomous vehicle characteristics has primarily focused on how those characteristics can communicate intent rather than autonomy. [18] investigated visual cues such as a waving hand on the roof of a car to communicate vehicle intent and found that pedestrians prefer having explicit visual cues of what the car will do rather than relying on motion cues. Several authors have investigated how other car features can help communicate with children [5] and adults [31] using field studies [19] and video studies [4]. As an exception, [10] recently looked at factors affecting vehicle autonomy. They found that futuristic appearance in a car could make it easier to believe it is an autonomous vehicle. However, these past works did not explore what visual features of a car (for example, the presence of cameras or the absence of a driver) can affect pedestrians’ perceptions of the car’s autonomy.
Past field studies using hidden-driver vehicles [25] manipulated several features of the vehicle to make the car seem autonomous (i.e., hidden driver, Lidar on the roof, cameras by the headlights, a decal stating “Autonomous Vehicle”) without evaluating which features are noticed by pedestrians. Assessing whether each feature is important for creating the impression of an autonomous vehicle among pedestrians can motivate the additional effort of using a hidden driver in the first place.
Research Question 3: What visual or motion cues make a vehicle seem autonomous?
2.5
Pedestrian Feelings and Behavior with
Autonomous versus Normal Vehicles
Past public opinion surveys have explored people’s feelings toward autonomous vehicles. [26] found US, UK and Australian adults felt positive about and desired autonomous vehicles, but were very
concerned about safety and the absence of driver controls. Similarly, [8] found MTurk participants’ self-reported likelihood of crossing in front of an autonomous vehicle was positively correlated with willingness to interact with the vehicle and perceived safety of the vehicle. A qualitative study [23] asked pedestrians their impressions of the Uber autonomous vehicle; while some said they would trust it until they heard of accidents, many said they did not trust the vehicle to drive without a human operator. These studies show that concern and trust are key feelings toward autonomous vehicles but do not link the feelings to pedestrian behavior.
Recent research has started to explore how pedestrians’ feel-ings toward an autonomous vehicle could influence their belief and behavior with the vehicle. Recent work by [9, 10] showed people videos of hidden-driver vehicles and found a vehicle that people judged to look futuristic was perceived as more autonomous than a vehicle people said was not as futuristic. The futuristic appearance of the vehicle was not related to people’s decision to cross in front of it. However, that work did not explore feelings of concern, which could be more important than impressions of a futuristic appear-ance. Further, the authors look only at crossing decision, which is surprising because [25] found AVs don’t affect crossing decision (but do affect looking time/crossing path).
Research Question 4: Are pedestrians’ atypical cross-ing and lookcross-ing behavior with autonomous vehicles related to feelings of concern?
3
STUDY 1: ON-ROAD STUDY IN THE
NETHERLANDS
Study 1 was a reproduction of the hidden-driver on-road study methodology [25] in the Netherlands, including a methodological addition of a normal driver control.
3.1
Materials and Method
3.1.1 Participants. 130 pedestrians (57 female, 73 male; age 18-64, M = 32, SD = 13) on a Netherlands university campus encountered either an autonomous vehicle (N = 72) or a normal vehicle (N = 58) at a video-recorded pedestrian crossing. 58 of 72 pedestrians in the autonomous vehicle condition completed the belief rate portion of the interview; all gave consent.
3.1.2 Study Design. A two-condition (vehicle autonomy: driver-less vs. normal) between-participants experiment was conducted to assess belief rates that a hidden-driver vehicle is autonomous and to evaluate how vehicle autonomy and pedestrian group size influence pedestrian behavior in the Netherlands. The breaching condition (“Autonomous” or “NL-f”) consisted of a seemingly fully autonomous vehicle without a visible human driver (Fig. 1). The control condition (“Normal” or “NL-f-no-AV”) consisted of the same vehicle without modifications driven by a visible human driver. 3.1.3 Vehicle and Driver. We used a black 2015 Audi A4 sedan, fitted with a custom-built model LiDAR, roof- and side-mounted GoPro/V360 cameras and decals with the text “Autonomous Vehicle” and the university name on the side and front. English text (rather than Dutch) was used based on high English fluency at the uni-versity and English being its official language. In the autonomous vehicle condition, the driver wore a seat costume constructed of
a pedestrian autonomous vehicle b pedestrian autonomous vehicle approach return approach return
Figure 2: Study 1 setting.
plastic canvas, yarn, tapestry needles, steel wire and vinyl tubing. In the normal vehicle condition, the driver drove without a cos-tume. The driver was an experienced driver and received one day of training on location prior to the start of the study.
3.1.4 Procedure. Our procedure was based on [25]. We conducted the study over six days in parking lot crossings where cars travel at low speeds and where there is moderate pedestrian traffic. Each trial began with a researcher radioing the driver that a pedestrian was approaching. The driver was asked to drive the car to arrive at the crosswalk at the same time as the pedestrian. The car passed right-of-way to the pedestrian unless it was clearly given to the vehicle, then drove a set path to return to a waiting spot (Fig. 2). An inter-viewer who was previously hidden walked to the pedestrian and presented an information sheet about the study. If the pedestrian agreed to an interview, they were asked their consent to be audio-recorded, consent to analyze and show their video-recordings, their age, nationality and a series of increasingly-specific open-ended questions about their interaction (“Can you please describe the experience you had at the crosswalk with the car?”; “What did you observe about the car?”; “Was there anything special about the car that you observed?”; “How did you think the car was moving?”; “Did you think the car was moving on its own?”; “Could you tell that there was no driver in the car?”; “Did the fact that the car was autonomous influence your behavior?”; “Did the car do what you ex-pected it to do?”; “How did you decide whether to continue to cross the intersection or not?”). A total of six to eight researchers were present for each session (i.e., two pedestrian spotters, three-five interviewers and one videographer).
We concurrently coded our Netherlands dataset (NL-f, NL-f-noAV) with videos from our original California dataset [25] (US-CA-f). Videos were cut in Final Cut Pro X to isolate interactions between pedestrians and the vehicle, producing a corpus consisting of a street-perspective clip and a car-perspective clip per interaction (no car-perspective clip was used for the normal vehicle). Survey data were matched to video clips using descriptions of the pedestrians (“man with green shirt”) and encounter times. Only pedestrians who were interviewed and who granted data consent were included in the video corpus. The procedure was approved by the ethics board at the University of Twente.
3.1.5 Analysis. We analyzed two measures: belief rates and pedes-trian behavior. Analysis of belief rates needed pedespedes-trians to com-plete a full interview and their responses coded for belief in vehi-cle autonomy (hidden-driver conditions only). We collected code counts from all four field studies: Netherlands (NL-f) (this work, Study 1, N = 58 interviews), California (US-CA-f) [25], Mexico City (MX-MC-f) and Mexico Colima (MX-CO-f) [7]. We also col-lected belief rate data from two online samples (Study 2) conducted with California (US-CA-v) and Netherlands (NL-v) participants. All datasets included “Yes”, “Maybe” and “No” categories for pedestrian belief in car autonomy. The Mexico dataset had an additional cate-gory for participants who thought the car was remotely driven (i.e., teleoperated by a person), which we excluded from our analysis.
Analysis of pedestrian behavior required videos of the pedestri-ans crossing the road. We obtained such videos only for the Califor-nia hidden driver (US-CA-f), Netherlands hidden driver (NL-f, N = 72) and Netherlands normal car (NL-f-no-AV, N = 58) samples (video data was not collected online and not acquired from the Mexico study at time of coding). Therefore, statistical analysis of pedestrian behavior was performed separately on: (1) the US-f data; and (2) the combined NL-f and NL-f-no-AV data (to identify interaction effects). We first viewed the videos to design a codebook with explanatory text and images and with 5% of the sample coded by an expert as a baseline. Two coders (one living in the Netherlands, one living in California) annotated all videos from their region, while a third coder (who lived recently in both regions) annotated all videos. Coders annotated each individual in a group separately. Measures had inter-rater reliability of Krippendorf’s alpha between 0.74 and 0.951. We asked coders to estimate crossing time using start and end times and looking duration with a mental “running total” of glance periods. For straight path deviation, we provided coders with a graphical image (similar to [7]) and asked them to judge whether the deviation was because of the car’s approach or for other rea-sons (e.g., to get to a car, to avoid pedestrians walking toward them; these cases were excluded from path deviation analysis). Decision to cross was approximated by coders’ judgment of who crossed the intersection first (i.e., who took priority). Coder disagreement in nominal variables was resolved by an additional coder and if still unresolved (four cases in total), by random selection [17].
Statistical tests were done in R version 3.4.2 and RStudio version 1.1.383. We performed Pearson chi-squared tests of independence to assess the relationship between region and belief, summing together the “Maybe” and “No” counts to ensure each cell was at least five [3]. T-tests were used for the US-CA-f data to assess the effect of group size on looking and crossing times, while ANOVA was used to assess the effect of group size and car autonomy on pedestrian looking and crossing times. The “mass” package was used to perform ordinal logistic regression to get odd ratios for straight path deviation (ordinal measure).
3.2
Study 1 Results
3.2.1 Effect of Region. We performed a Pearson chi-squared test of independence to examine the relation between the four regions and belief in the hidden-driver vehicle’s autonomy (Fig. 3, left).
1who took priorityα = 0.95; crossing duration α = 0.80; looking duration α = 0.85; path deviationα = 0.74; gesture use α = 0.74
8 16 28 5 23 24 4 6 48 4 7 10 2 2 26 30 14 36 1 2 2 2 1 2 3 0 1 3 0 2 1 2 2 3 0 1 2 1 2 2 2 2 1 2 3 2 0 3 1 3 1 1 0 5 1 1 3 3 2 1 1 1 4 1 1 3 1 0 4 2 1 1 2 1 2 1 1 2 0.00 0.25 0.50 0.75 1.00
NL−f US−CA−fMX−MC−fMX−CO−f NL−v US−CA−v video2 video9 video11 video12 video14 video20 video25 video27 video28 video31 video34 video38 video44 video45 video47 video49 video50 video51 video100 video104
Region / Video Propor tion ThoughtAutonomous No Maybe Yes
Figure 3: Proportions of participants who believed the car was autonomous across different studies. Bars from left to right: be-lief rates for pedestrians in the Netherlands (Study 1), California, Mexico City and Colima (past work); for online participants in the Netherlands and California (Study 2); and per video (Study 2).
Degrees of belief that a hidden-driver vehicle was autonomous was significantly related to study region,χ2(3, N = 189) = 30, p < 0.001. We performed follow-up Pearson chi-squared tests with Bonfer-roni correction to assess the effect of specific regions on degree of belief in the vehicle’s autonomy. We did not find a difference in degree of belief for US-CA-f versus NL-f field trials,χ2(1, N = 88) = 0.03, p = 0.9, but degree of belief was significantly different between NL-f and both MX-MC-f and MX-CO-f regions,χ2(1, N = 110) = 15, p < 0.001 andχ2(1, N = 110) = 9.4, p = 0.002. We also found differences between the US and both MX regions,χ2(1, N = 82) = 11, p < 0.001 andχ2(1, N = 82) = 7.7, p = 0.005, but not between the two MX regions. US and NL field trial participants had higher degrees of belief than the MX field trial participants but comparable belief rates with each other, answering Research Question 1. 3.2.2 Effect of Pedestrian Group Size. A 2x2 analysis of variance (ANOVA) for crossing time with both group size and condition as independent variables found a main effect for group size, F (1,124) = 9.6, p = 0.002, but did not find a main effect for condition, F (1,124) = 2.6, p = 0.11, or interaction effect, F (1,124) = 5.0, p = 0.36. As predicted in Hypothesis 1, groups took longer to cross than single-tons, M = 9.2, SD = 2.7 vs. M = 7.8, SD = 2.2 seconds. A similar 2x2 ANOVA for looking time found an interaction effect, F (1,124) = 5.6, p = 0.02. Groups looked for less time at a normal vehicle compared to singletons, M = 1.2, SD = 1.1 vs. M = 2.7, SD = 1.9 seconds, but as predicted in Hypothesis 2, groups looked for a longer time at an autonomous vehicle compared to singletons, M = 11.7, SD = 7.4 vs. M = 8.6, SD = 3.9 seconds. Hypotheses 1 and 2 were confirmed.
We also conducted a new analysis on the effect of group size in the US-CA-f dataset videos, which we concurrently coded during analysis. Welch’s t-test comparing single versus group crossing time in the US-CA-f dataset was significant, t(12.2) = 2.2, p = 0.045, 95% CI [0.03, 2.35] seconds. Participants crossed more slowly when part of a group than when alone, M = 8.4, SD = 1.2 vs. M = 7.2, SD = 2.1 seconds (Fig. 4). Welch’s t-test on looking time was also significant, t(6.5) = 3.5, p = 0.01, 95% CI [3.05, 16.9] seconds. Participants looked longer at the autonomous vehicle when they were in a group than when alone, M = 16.1, SD = 7.5 vs. M = 6.1, SD = 4.1 seconds. Hypotheses 1 and 2 were confirmed for the US-CA-f dataset.
0 5 10 15 20 Group Single Dur
ation (seconds), 95% CI error bars
Cross Look
Figure 4: Vertical bar graph showing duration of crossing and looking by group. Groups took more time crossing and looking at the vehicle than did singleton pedestrians.
3.2.3 Effect of Autonomy. A t-test comparing crossing time with the hidden-driver autonomous vehicle versus with a normal vehicle (i.e., the NL-f vs. NL-f-noAV dataset) was significant, t(126) = 2.17, p = 0.032, 95% CI [0.085, 1.84] seconds. Pedestrians took longer to cross with an autonomous vehicle compared to a normal vehicle, M = 9.0, SD = 3.1 vs M = 8.0, SD = 1.4 seconds. A similar t-test comparing looking time with a hidden-driver autonomous vehicle vs. normal vehicle was significant, t(128) = 9.96, p <0.001, 95% CI [7.07, 10.6] seconds. Pedestrians spent more time looking at an
0.0 2.5 5.0 7.5 10.0 12.5 Autonomous Normal Dur
ation (seconds), 95% CI error bars
Cross Look
Figure 5: Vertical bar graph showing duration of crossing and looking by condition. Pedestrians took more time cross-ing and lookcross-ing when encountercross-ing an autonomous vehicle compared to a normal vehicle.
autonomous vehicle compared to a normal one, M = 10.6, SD = 6.6 vs M = 1.8, SD = 1.7 seconds (Fig. 5).
A binomial logistic regression with who took priority as the response variable and condition as the predictor was also significant, X2(1) = 4.3, p = 0.037. Pedestrians were more likely to cross in front of a normal car than an autonomous car, z = 2.1, p = 0.038, 52 cross, 6 wait vs. 54 cross, 18 wait. These results align with qualitative findings in past research [25].
Pearson’s chi-squared test comparing straight path deviation with an autonomous vs. normal vehicle was significant, X2(3) = 15.3, p = 0.002. An ordinal logistic regression on autonomous vs. normal vehicle with path deviation as the response variable gave an odds ratio of OR = 1.1 x 108, 95% CI [8.4 x 107, 1.5 x 108]. Participants who saw an autonomous vehicle were much more likely to deviate compared to those who saw a normal vehicle, 6 small, 5 medium, 4 large deviations and 48 no deviations (9 excluded) vs. 0 deviations and 56 no deviations (2 excluded).
Therefore, we confirm that NL pedestrians have atypical crossing and looking behavior in agreement with past US data.
3.2.4 Qualitative Results on Pedestrian Feelings. Thirty-six of the NL-f interviews were audio-recorded and transcribed for analysis2.
2Not all participants completed full interviews
One researcher first reviewed the full set of interview data to iden-tify themes (particularly in understanding reasons for behavior). The researcher then coded the full dataset of interview responses for phrases that referred to each of these specific themes.
In twelve interviews, participants mentioned that their feelings influenced their behavior. One participant mentioned fear affecting path deviation: “I was a bit anxious...I’m going to be, like, maybe a bit more distant from it.” Similarly, another said, “We were trying to run away from it...It’s creepy.” Some pedestrians specified the reason for fear: “We don’t know how advanced it is or how it works...we’re afraid, because we don’t know.”
Pedestrians in thirteen interviews mentioned feelings of con-fusion and concern about safety. One participant said, “I was like ‘Oh, it’s going to drive me over,’ ‘Oh it’s not going to drive me over.”’ Another said, “I’m not sure if it’s going to drive over me or not.” Some pedestrians phrased concern as lack of trust: “If I see a driver I can always make eye contact... With an autonomous car with no driver, of course, this doesn’t work, so of course you stay away from a car you don’t trust.”
We were surprised to find that some pedestrians tested the car to see if it would let them pass. One participant said, “I made the easiest experiment. I just stopped in front of the car.” Another had a similar experience, “I wanted to stand in front of [it], but it just stopped there, and so I can really walk by.” These pedestrians seemed to feel more curiosity than concern.
4
STUDY 2: ONLINE VIDEO STUDY
Study 1 confirmed the ghostdriver method was successful with European pedestrians and investigated their behavior. In Study 2, we extended the method to evaluate its success in video form and explored potential reasons for pedestrian behavior we observed in Study 1.
4.1
Materials and Method
4.1.1 Study Design. A 2 (participant region: US-California or Nether-lands) by 20 (videos) between-participants experiment was con-ducted to assess how video form and vehicle characteristics influ-ence the degree of belief in a hidden-driver vehicle’s autonomy, as well as potential reasons for atypical behavior with autonomous vehicles. Participants were either Mturk workers residing in Cali-fornia or residents (MTurk workers or university affiliates) in the Netherlands.
4.1.2 Participants. 101 participants were recruited online from California or the Netherlands. California participants (N = 80, 36 female, 42 male, 1 fey/non-binary, 1 undisclosed; age 18-63, M = 35, SD = 10) were MTurk workers in California. Netherlands par-ticipants (N = 21, 9 female, 12 male, age 18-55, M = 31, SD = 11) were either MTurk workers in the Netherlands or affiliated with University of Twente (added due to low numbers of Netherlands MTurk workers). MTurk workers were paid 0.40 USD (California) or 0.35 USD (Netherlands) for the two-minute study. Participants affiliated with University of Twente were not compensated. 4.1.3 Video Stimuli. We selected twenty “street-view” videos from [25] where the vehicle can be seen approaching the intersection
(Fig. 1). Netherlands videos were not used to avoid violating General Data Protection Regulation (GDPR) policies on EU citizen data. 4.1.4 Procedure. After reading consent information, participants were asked to “Please imagine you are on a university campus in your area and notice the scene shown in the following short video” and consider only “the main car that is approaching the intersection”. They subsequently viewed one of twenty videos of the US-CA-f vehicle approaching the intersection. On the next page they were asked open-ended text-entry questions: “What did you notice about the car? Anything about its appearance that caught your attention” and “How would you feel if you encountered the car in real life and why would you feel that way?” On the following page, they were asked “Did you think the car was driving by itself? Please explain briefly” and “Did you think there was a human driver in the car? Please explain briefly.”
4.1.5 Analysis. One researcher who lived recently in both the Netherlands and California coded all text-entry responses. What the pedestrian noticed (i.e., car characteristics) were coded into roof, decal, car make-model, motion and no-driver. Participant feeling was coded into feeling normal, annoyed, concerned about privacy, concerned about being hit and interested (categories were based on the researcher first reviewing a third of the responses). Belief rates in vehicle autonomy and lack of human driver were coded into “Yes”, “Maybe” and “No” by the same researcher. Statistical tests were done in R version 3.4.2 and RStudio 1.1.383.
4.2
Study 2 Results
4.2.1 Effect of Participant Region. We performed chi-squared tests to assess the relationship between participant region and degree of belief in the videos (Fig. 3). We did not find a significant difference in the degrees of belief between the US-CA-v and NL-v participants, χ2(1, N = 101) < 0.1, p = 1.
4.2.2 Effect of Video. We performed chi-squared tests to assess the effect of video on belief in vehicle autonomy. A chi-squared test comparing the belief rates of the California video sample vs California field sample was significant,χ2(1, N = 110) = 13.8, p = 0.0002. Fewer video participants believed the car was autonomous than the field study participants (36 out of 80 versus 26 out of 30) (Fig. 1). The Netherlands video sample vs field sample was also significant,χ2(1, N = 79) = 8.0, p = 0.004. Fewer video participants believed the car was autonomous than the field study participants (10 out of 21 versus 48 out of 58). Some participants who did not believe the vehicle was autonomous said, “jettas cant do that”, “no, looked like it was driving normally” or “no, someone is inside and moving the car”. Videos of hidden-driver vehicles have lower belief rates than on-road vehicles, answering Research Question 2. 4.2.3 Relationship between Car Characteristics and Perception of Autonomy. Descriptive counts of people who noticed each of the car characteristics were: roof (58), motion (15), make-model (11), no driver (9) and decal (2). A chi-squared test found no significant dif-ference in the distribution of these counts between regions,χ2(4, N = 95) = 1.5, p = 0.8. Some participants were unclear about what was on the roof (“Looked like it had something on top” or “There was a billboard-like thing on top”) but most referred to it as a camera or
detectors. Comments about car motion included “car was not totally stopped” or “it was very appropriate and cautious”. Comments on make-model included “jetta white” or “volkwagon golf”. No-driver comments included “I did not see a driver” and “believe they [sic] was no driver in the car”. Only two people mentioned the decal, so we removed this from further analyses.
We conducted an ordinal regression (ordered logit) to predict the likelihood that participants thought the car was autonomous based on whether they noticed the roof device, car make-model, car motion and no car driver. No car driver was a significant predictor, t(96) = 2.1, p = 0.04, parameter estimate = 1.58 (95% CI of estimate 0.18 to 3.25). People who mentioned seeing no driver were 4.9 times more likely to think the car was autonomous. Roof, make-model and motion were not found as significant predictors. Research Question 3 was affirmed for the visual cue of no driver.
Qualitative responses showed a reason why people who noticed the roof device did not think the car was autonomous: they said the roof camera could be used for road image collection rather than autonomous driving. Participants said “It looked like the type of car Google used for Google Maps because of the rotating camera on top” and “the sensor could be for any purpose, not just self-driving”; these participants reported “no” or “maybe” the car was autonomous and “yes” to believing there was a driver in the car. 4.2.4 Relationship between Feeling and Belief Rates. Descriptive counts of people’s imagined feelings if they were in the video were: concern about safety (34), normal (31), interest (18), concern about privacy (9) and annoyance (8). Sample responses included “scared because it is automated and dangerous” (safety concern), “I probably wouldn’t feel any different about this car than any other car” (nor-mal), “Curious about it, because I’d like to know more” (interest), “Looks highly suspicious and i would hope it was not recording me”
(concerned about privacy) and “I would be irritated” (annoyed). We conducted an ordinal regression to predict the likelihood that participants thought the car was autonomous based on whether they felt normal, annoyed, concerned about privacy, concerned about being hit or interested toward the car. Significant effects were found for feeling concerned about safety and interested, t(95) = 3.3, p = 0.001, estimate = 2.30 (95% CI of estimate, 1.01 to 3.79) and t(95) = 2.3, p = 0.02, estimate = 1.73 (95% CI of estimate, 0.33 to 3.3). Participants who reported feeling concerned about safety in the vehicle were ten times more likely to have thought the car was autonomous than those who did not report feeling that way, while those who reported feeling interested were 5.6 times more likely to have believed the car was autonomous.
4.2.5 Relationship between Feeling on Looking and Crossing Time. We assessed whether the feeling elicited by the video predicted the looking time of the pedestrian in the video using a linear re-gression with looking time as a response variable and participants’ feelings (normal, annoyed, privacy concern, safety concern and interest) toward the vehicle as predictors. We used average looking time in cases of videos with pedestrian groups. Feeling concerned about safety was a significant predictor, t(95) = 1.96, p = 0.049, esti-mate 3.57 (95% CI of estiesti-mate, -0.05 to 7.2). Videos where viewers reported feeling concerned about safety if they were approached by the vehicle in the video featured 3.57-second-longer looking times by pedestrian(s) compared to videos where viewers did not
report feeling concerned, thereby affirming Research Question 4 for looking time. Feeling concerned about privacy was borderline significant, t(95) = 1.96, p = 0.05, estimate = 4.58 (95% CI, -0.06 to 9.2). Feelings of normalcy, annoyance and interest were not significant. We assessed whether viewer feeling predicted crossing time using the same regression as above but with crossing time as the measure. No significant effects were found.
4.2.6 Effect of Video Stimuli on Belief Rates. We calculated belief rates for each of the videos shown to participants to see which were most convincing as featuring autonomous vehicles (Fig. 1). Based on simple belief rate counts, videos 9, 28, 34, 45 and 49 were best at achieving the illusion (see Supplementary Material: Videos).
5
DISCUSSION
5.1
Summary of Results on Reproducibility
We confirmed the ghostdriver method works in Europe: pedestrians in the Netherlands had similar belief rates that a hidden-driver vehicle was autonomous as did pedestrians from a prior study. We also confirmed past findings that pedestrians look longer and cross differently with an autonomous vehicle compared to a normal vehicle in Europe.
5.2
Summary of Extension Results
In an on-road field study, pedestrian groups took longer to cross and looked longer than singletons when encountering an autonomous vehicle, but groups looked less than singletons when encountering a normal vehicle. We extended the ghostdriver methodology into an online study with a video of a hidden-driver vehicle and found belief rates were lower than in-person. Our online study revealed that noticing an absent driver in a video of a hidden-driver car (but not noticing a Lidar, cameras or car motion) was a significant predictor that adults believed the car was autonomous. Feelings of concern among viewers of videos of hidden-driver vehicles were related to longer looking times in the recorded responses of pedestrians who had actually experienced the situations in the videos.
5.3
Implications for Theory
The finding that feelings of concern about safety are related to how long a pedestrian looked at a hidden-driver vehicle provides some evidence that atypical behaviour around autonomous vehicles could be due to a feeling of concern. While past work has highlighted “interest” behaviors such as taking photos of a hidden-driver vehicle, video study participants reported more concern than interest.
Groups behave differently than singletons when encountering an autonomous vehicle, which provides evidence that groups are an important variable to consider in pedestrian-AV interaction. This could be because visual attention propagates in groups or because groups discuss the vehicle.
5.4
Implications for Methodology
We provide a replication and corresponding belief rate analyses that confirm the “ghost-driver” (i.e., hidden human driver) method to simulate an on-road autonomous vehicle can be effectively used in Europe.
Belief rate data from our video study were lower than correspond-ing field studies, suggestcorrespond-ing that videos of hidden-driver vehicles might not convince viewers that a vehicle is autonomous (although some videos used in this work had higher belief rates than others, so this method may merit future work). Belief rates were also lower in past Mexico field studies; interview data suggests this is because fewer people noticed the absent driver (although it is unclear why). Noticing an absent driver was the only significant predictor of belief in a car’s autonomy, which provides justification for the use of hidden-driver methods in the first place: people realize that cameras and car motion do not necessarily mean a car is autonomous, but seeing an absent driver does seem to convince them that a vehicle is autonomous.
5.5
Limitations and Future Work
In confirming the reproducibility of the hidden-driver method in Europe, we compared only the belief rate and not the behavior of pedestrians across regions because road infrastructure was not possible to control across regions (i.e., the road crossing length and presence of a stop sign were different).
We did not discuss in detail cross-cultural differences observed in the data. We encourage future research on the types of and reasons for cross-cultural differences in pedestrian belief and behavior.
In assessing the importance of car characteristics on a hidden-driver vehicle’s believability, we did not do a control study with a human driver visible inside a vehicle with a Lidar, decal and cameras. Such a field study could run into difficulty because the interaction timing between pedestrian and vehicle would be quite different with a visible human driver who can communicate with the pedestrian; we observed, for example, that pedestrians walk faster when a human driver was visible (in a car without decals or cameras). Future work could try to resolve this issue through a study that manipulates both the presence of a human driver and vehicle appearance. We chose to instead conduct an online study with existing videos of pedestrian field trials because it assessed what features are typically noticed in a hidden-driver vehicle (rather than manipulating those features).
6
CONCLUSION
In a field experiment where Netherlands pedestrians encountered a hidden-driver vehicle, belief rates that the vehicle was autonomous were similar to a past study in California, suggesting that the hidden-driver method is valid in Europe. Pedestrians in Europe looked longer and had more deviations in their path when encountering a hidden-driver vehicle compared to a normal one, replicating the finding that atypical responses to autonomous vehicles occur in Eu-rope as well. An online video study showed that viewers’ noticing of an absent driver (but not car cameras or decals) predicted their belief that the car was autonomous, justifying the use of hidden drivers. Feelings of concern about safety among viewers of videos featuring hidden-driver vehicles were positively related to longer looking times among pedestrians in those videos, suggesting pedes-trian behavior is related to concern. On-road and online studies featuring hidden-driver vehicles are emerging as key methods to obtain greater understanding of pedestrians’ beliefs and behaviors toward autonomous vehicles.
REFERENCES
[1] Sigal G Barsade. 2002. The ripple effect: Emotional contagion and its influence on group behavior. Administrative Science Quarterly 47, 4 (2002), 644–675. [2] Gerard J Blaauw. 1982. Driving experience and task demands in simulator and
instrumented car: a validation study. Human Factors 24, 4 (1982), 473–486. [3] Gregory Camilli and Kenneth D Hopkins. 1978. Applicability of chi-square to 2×
2 contingency tables with small expected cell frequencies. Psychological Bulletin 85, 1 (1978), 163.
[4] Chia-Ming Chang, Koki Toda, Takeo Igarashi, Masahiro Miyata, and Yasuhiro Kobayashi. 2018. A Video-based Study Comparing Communication Modalities between an Autonomous Car and a Pedestrian. In Adjunct Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 104–109.
[5] Vicky Charisi, Azra Habibovic, Jonas Andersson, Jamy Li, and Vanessa Evers. 2017. Children’s views on identification and intention communication of self-driving vehicles. In Proceedings of the 2017 Conference on Interaction Design and Children. ACM, 399–404.
[6] Michael Clamann, Miles Aubert, and Mary L Cummings. 2017. Evaluation of vehicle-to-pedestrian communication displays for autonomous vehicles. Technical Report.
[7] Rebecca Currano, So Yeon Park, Lawrence Domingo, Jesus Garcia-Mancilla, Pe-dro C Santana-Mancilla, Victor M Gonzalez, and Wendy Ju. 2018. ¡ Vamos!: Observations of Pedestrian Interactions with Driverless Cars in Mexico. In Pro-ceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 210–220.
[8] Shuchisnigdha Deb, Lesley Strawderman, Daniel W Carruth, Janice DuBien, Brian Smith, and Teena M Garrison. 2017. Development and validation of a questionnaire to assess pedestrian receptivity toward fully autonomous vehicles. Transportation Research Part C: Emerging Technologies 84 (2017), 178–195. [9] Debargha Dey, Marieke Martens, Berry Eggen, and Jacques Terken. 2017. The
impact of vehicle appearance and vehicle behavior on pedestrian interaction with autonomous vehicles. In Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications Adjunct. ACM, 158–162.
[10] Debargha Dey, Marieke Martens, Berry Eggen, and Jacques Terken. 2019. Pedes-trian road-crossing willingness as a function of vehicle automation, external appearance, and driving behaviour. Transportation Research Part F: Traffic Psy-chology and Behaviour 65 (2019), 191–205.
[11] Debargha Dey, Marieke Martens, Chao Wang, Felix Ros, and Jacques MB Terken. 2018. Interface Concepts for Intent Communication from Autonomous Vehicles to Vulnerable Road Users.. In AutomotiveUI ’18 Adjunct. 82–86.
[12] Jolyon J Faria, Stefan Krause, and Jens Krause. 2010. Collective behavior in road crossing pedestrians: the role of social information. Behavioral Ecology 21, 6 (2010), 1236–1242.
[13] Andrew C Gallup, Joseph J Hale, David JT Sumpter, Simon Garnier, Alex Kacelnik, John R Krebs, and Iain D Couzin. 2012. Visual attention and the acquisition of information in human crowds. Proceedings of the National Academy of Sciences 109, 19 (2012), 7245–7250.
[14] Julie Hatfield, Ralston Fernandes, RF Soames Job, and Ken Smith. 2007. Misunder-standing of right-of-way rules at various pedestrian crossing types: observational study and survey. Accident Analysis & Prevention 39, 4 (2007), 833–842. [15] Julie Hatfield and Susanne Murphy. 2007. The effects of mobile phone use
on pedestrian crossing behaviour at signalised and unsignalised intersections. Accident Analysis & Prevention 39, 1 (2007), 197–205.
[16] Kotaro Hayashi, Daisuke Sakamoto, Takayuki Kanda, Masahiro Shiomi, Satoshi Koizumi, Hiroshi Ishiguro, Tsukasa Ogasawara, and Norihiro Hagita. 2007. Hu-manoid robots as a passive-social medium-a field experiment at a train station. In 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 137–144.
[17] Matthew Lombard, Jennifer Snyder-Duch, and Cheryl Campanella Bracken. 2010. Practical resources for assessing and reporting intercoder reliability in content analysis research projects. (2010).
[18] Karthik Mahadevan, Sowmya Somanath, and Ehud Sharlin. 2018. Communi-cating awareness and intent in autonomous vehicle-pedestrian interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 429.
[19] Milecia Matthews, Girish Chowdhary, and Emily Kieson. 2017. Intent
com-munication between autonomous vehicles and pedestrians. arXiv preprint
arXiv:1708.07123 (2017).
[20] Nicole Mirnig, Nicole Perterer, Gerald Stollnberger, and Manfred Tscheligi. 2017. Three strategies for autonomous car-to-pedestrian communication: A survival guide. In Proceedings of the Companion of the 2017 ACM/IEEE International Con-ference on Human-Robot Interaction. ACM, 209–210.
[21] World Health Organization. 2015. Global status report on road safety 2015. World Health Organization.
[22] David R Ragland and Meghan Fehlig Mitman. 2007. Driver/pedestrian under-standing and behavior at marked and unmarked crosswalks. (2007). [23] Samantha Reig, Selena Norman, Cecilia G Morales, Samadrita Das, Aaron
Ste-infeld, and Jodi Forlizzi. 2018. A Field Study of Pedestrians and Autonomous Vehicles. In Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 198–209.
[24] Laurel D Riek. 2012. Wizard of oz studies in hri: a systematic review and new reporting guidelines. Journal of Human-Robot Interaction 1, 1 (2012), 119–136. [25] Dirk Rothenbücher, Jamy Li, David Sirkin, Brian Mok, and Wendy Ju. 2015.
Ghost driver: a platform for investigating interactions between pedestrians and driverless vehicles. In Adjunct Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 44–49. [26] Brandon Schoettle and Michael Sivak. 2014. A survey of public opinion about
autonomous and self-driving vehicles in the US, the UK, and Australia. Technical Report. University of Michigan, Ann Arbor, Transportation Research Institute. [27] Masahiro Shiomi, Francesco Zanlungo, Kotaro Hayashi, and Takayuki Kanda.
2014. Towards a socially acceptable collision avoidance for a mobile robot navigating among pedestrians using a pedestrian model. International Journal of Social Robotics 6, 3 (2014), 443–455.
[28] Leah L Thompson, Frederick P Rivara, Rajiv C Ayyagari, and Beth E Ebel. 2013. Impact of social and technological distraction on pedestrian crossing behaviour: an observational study. Injury Prevention 19, 4 (2013), 232–237.
[29] Astrid Weiss, Regina Bernhaupt, Manfred Tscheligi, Dirk Wollherr, Kolja Kuhn-lenz, and Martin Buss. 2008. A methodological variation for acceptance evaluation of human-robot interaction in public places. In RO-MAN 2008-The 17th IEEE In-ternational Symposium on Robot and Human Interactive Communication. IEEE, 713–718.
[30] Sarah N Woods, Michael L Walters, Kheng Lee Koay, and Kerstin Dautenhahn. 2006. Methodological issues in HRI: A comparison of live and video-based methods in robot to human approach direction trials. In RO-MAN 2006-The 15th IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 51–58.
[31] Jingyi Zhang, Erik Vinkhuyzen, and Melissa Cefkin. 2017. Evaluation of an autonomous vehicle external communication system concept: a survey study. In International Conference on Applied Human Factors and Ergonomics. Springer, 650–661.
