User Experience Feedback Dashboards: The effects of layered design on cognitive and behavioral performance

User Experience Feedback

Dashboards: The effects of layered

design on cognitive and behavioral

performance

by Ana Vojvodic

*

27 February 2015

Second-year Internship Project Report

Master Brain and Cognitive Sciences, Track Cognitive Science

In collaboration with the University of Amsterdam and Usabilla B.V.

Supervisor: Sabina Idler, MSc.

Co-Assessor & UvA representative: Dr. Marcel Worring

Student Number 10628916

Abstract

Over the past few decades, user experience (UX) has become an increasingly important factor to consider when designing an interactive technical product, such as a website. Building web-based systems upon the user’s expectations and needs and applying the principles of user-centered design allows the designer to make informed data-driven decisions in creating optimal experiences for the user. However, measuring and understanding UX is abstract and

challenging. UX data inherently covers a broad range of qualitative and quantitative data types, and is difficult to interpret quickly and efficiently. A dashboard displaying collected UX data can serve as a tool to facilitate the interpretation and monitoring of a website’s performance by illustrating key metrics in one display so the interested stakeholder can get an overview of their website’s performance at a glance. However, little is currently known about the optimal

formatting strategy to effectively display all the necessary information. In this experiment, we aim to characterize the cognitive and behavioral consequences of displaying UX data in two different dashboard formats. The formats differ in that one displays related groups of information in subsections on a single-page dashboard, and the other displays related groups of information in separate tabs on the dashboard. We found no significant differences in task performance or perceived usability ratings between the two dashboard formats, yet more users reported an overall preference for a single-page dashboard. From our qualitative and quantitative results, we draw insights about user experiences and formatting strategies by highlighting the costs and benefits of each format type. We then outline considerations for future UX data dashboard designs and propose further directions for research.

Introduction

As interactive technology systems have become an integral part of our daily lives over the past few decades, the field of human-computer interaction (HCI) has increased emphasis on

understanding and incorporating human factors and user experience (UX) research in constructing these systems. Many activities we engage in on a regular basis, such as

commerce, communication, education, and services, are rapidly shifting to an online presence and development, leading to an expansion of web-based systems. As a result, the importance for website stakeholders to understand UX and its relationship with website performance has increased. Web UX is composed of the cognitive, perceptual, sensory, and emotional processes that occur when an individual interacts with a given product or system (Carroll, 1997; Forlizzi & Battarbee, 2004; Hassenzahl & Tractinsky, 2006). Given the abstract nature of these factors, UX is difficult to measure and research. In applied psychology and industrial settings, UX research presents challenges not only because the sensory and psychological underpinnings of UX are complex, but also because UX data is unique and can encompass a broad range of data types to collectively depict the users’ experience. Further, the methodologies employed to conduct UX

research can vary greatly from in-person interviews to A/B testing to remote online user feedback surveys.

Despite the challenges of UX research, understanding UX helps identify the users’ expectations and needs of a website, as well as the website’s strengths and weaknesses. Insights from these results can then be incorporated into the development of solutions for improvement. The

stakeholder interested in UX data must be able to extract relevant information from the raw dataset and formulate these insights based on their users’ thoughts and behaviors. Insight, according to North (2006), must be:

• complex - synergistically involving a large amount of individual data points • deep - cumulate over time, bring up further questions

• qualitative - have the potential to be uncertain and subjective • unexpected - serendipitous and creative

• relevant - connect the dry data to create impact in the existing domain

Thus, selecting which aspects of UX to measure and analyze in order to develop these insights is a difficult process due to the unique nature of UX data. UX data is unique in that it reflects a website's performance beyond simply profiling basic statistics such as page views, bounce rates, and the demographics of the website’s user base. UX data encompasses a broad range of different variables that contribute to the user’s whole experience, resulting in many different data types. The data may seem fragmented at first glance, but when combined as a whole, constitute the complex and interrelated factors that form the user’s overall experience. For example, UX data can quantify behavioral patterns (e.g. task-completion times, click paths, gaze-determined attention points), measure subjective reports of affect (e.g. emotion, valence), cognition (e.g. ease of use, perception, attention, knowledge acquisition), and highlight technical consequences (e.g. encountered bugs, identifying problematic web browsers, operating

systems, or devices) of users’ interactions with the website. With such a broad range of

qualitative and quantitative data contributing to the overall UX dataset, this leads to the question of how to approach effective analysis of raw UX data and efficiently formulate insights from these analyses.

One solution is to create visualisations of the most important and relevant information from the UX dataset and display them in a dashboard to give a general overview of the website’s performance at a glance (Few, 2006). Applying visualizations makes a large dataset more compact and readable, allowing the dashboard user to draw insights by offloading the cognitive requirements from higher-level reasoning and analysis to simpler mechanisms of the visual perception system. As needed, the dashboard user can then drill down deeper and inspect the raw data for further analysis. The dashboard serves as a tool for extending human cognition by offloading working memory and creating a coupled system in aiding the user to complete a cognitive task (Clark & Chalmers, 1998). This coupled system, which can be considered a cognitive system of its own, is comprised of multiple components that contribute to the overall UX of the dashboard user. Ultimately, these elements dictate the user’s behaviors and thoughts

as they map the correct psychological models to executable action sequences with the

dashboard in order to complete a desired task. Dashboards employ the “use of interactive visual representations of abstract data to amplify cognition”, allowing the user to search, perceive, and acquire high volumes of information in a short amount of time (Card et al., 1999).

Another challenging characteristic of UX data is that the same UX dataset, or specific subset within this dataset, is relevant with various intended purposes to different dashboard users with differing roles and functional requirements. Although the dashboard users work towards the same general goal — to improve the overall user experience of their website — they may focus on different elements of the experience to meet this goal. For a UX designer, the relevant UX data may be regarding the visual appearance or interactivity of a website, for a developer, the technical source of a problem, or for a business manager, the user’s engagement with the website. Studies in HCI and cognitive psychology suggest that individual personality and cognitive differences may also affect user performance in visualization system interaction and evaluation (Ottley et al., 2013). Depending on the cognitive goals, situational contexts, and motivations of the user acquiring and applying the UX data, the same information a dashboard presents may be interacted with and perceived differently (Endsley, 2001; Hassenzahl & Tractinsky, 2006; Yigitbasioglu & Velcu, 2012). This could be attributed to the differences in users’ internally constructed mental models of a dashboard system and associated task-execution planning skills, despite the data presented and general end goal being the same (Gentner & Stevens (eds.), 2014; Norman, 1986). Thus, for such a broad range of users who may experience the dashboard differently, it is a challenge to design and implement optimal data visualisations to collectively constitute an effective dashboard to suit the users’ varying needs.

The principles of information visualization and dashboard design, built upon theoretical and experimental psychology and informatics frameworks, form the basis for researching and designing effective information-representation systems (Few, 2006; Koffka et al. 2013; Norman, 1986; Ware 2013; Wolfe, 1994; Yi et al. 2007). Users interact with an information-representation system, such as a dashboard, to extract information and gather insight. The users’ interactions influence their cognition, affect, and behavior, cumulating in an experience as the user

completes goal-directed tasks like visual queries and evaluation, information search and acquisition, and decision-making. Considering and capitalizing on cognitive capacities such as memory, visual perception, and attention when designing a dashboard system for such tasks enhances the system’s usability and effectiveness as a whole (Few, 2006; Norman, 1986; Ware, 2013; Wolfe, 1994).

Pike et al. (2009) suggest that current research lacks sufficient measurements of interaction costs and benefits for tasks such as the information search and acquisition process. Further, the authors suggest that “using the same visualization but varying the interaction techniques can illuminate the specific benefits and costs of those interactions.” Interaction techniques may be

type of cognition the task requires (Vessey, 1991). One possible means of facilitating the required cognitive processing for interaction, offloading working memory, and simplifying visual queries could be by formatting related information into groupings that minimize the cognitive load and are easier to identify and process (Ware, 2005). We aim to test this by comparing the effects of presenting UX data in a dashboard in two different formats -- one that groups

information into subsections on a single page, and one that breaks up the subsections onto multiple pages.

To investigate potential costs and benefits of grouping information and varying the dashboard interaction techniques, we propose to compare the cognitive and behavioral effects on simple visual search and information-acquisition tasks by displaying these groupings of UX data in two different design formats: either a single-page view (single-layered dashboard) with multiple sections, or a tabbed page view (multi-layered dashboard) with one tab corresponding to each analogous section of the single-page dashboard. The single-page dashboard view conforms to Few’s (2006) highly regarded and followed principle of dashboard design by displaying all information on one page. According to Few, exceeding the boundaries of a single page is a common pitfall in dashboard design. Thus, the tabbed dashboard design violates this principle by fitting on one screen, but not displaying all information at one time. We aim to test whether Few’s principle still holds when the information displayed on multiple layers is segregated by specific categories with the intention to decrease visual noise and minimize the cognitive load on the user. Our goal is to investigate what users think and say, how users behave when this design principle is not abided, and whether the users’ needs are still met when the dashboard information is grouped into smaller chunks compared to viewing all information at once. Little is currently known about the cognitive effects of page layering in dashboard formatting techniques when users extract information from the same system and apply it to various contexts (Pike et al., 2009; Sutcliffe et al. 2000; Yigitbasioglu & Velcu, 2012). The two design formats principally differ in that they present different interaction techniques with the dashboard itself and different framing of the same UX data to the dashboard user viewing the UX data.

Our specific aims are to investigate whether:

a) visual query and information-acquisition task performance differs in a single- vs. multi-layered dashboard design.

b) the single- and multi-layered dashboard designs are subjectively reported as experienced differently by dashboard users.

The cognitive, behavioral, and perceptual consequences of each design are used as indices of optimality for the dashboard designs. We aim to characterise how the different presentation formats influence these indices across expert UX data dashboard users.

Materials and Methods

UX is a cognitive process that can be modeled and measured by a number of different

approaches, and selecting the optimal methodology and data analysis approaches are areas of research that remain to be explored further (Vermeeren et al., 2010; Law, 2011). Although the methods and data applications for UX research remain under scrutiny and debate by the academic community, it is frequently noted that reliable UX research methodology should integrate both qualitative and quantitative approaches (Bargas-Avila & Hornbæk, 2011; Hassenzahl & Tractinsky, 2006; Law, 2011; Plaisant, 2004; Vermeeren et al., 2010). Further, current dashboard user research tends to focus on one user group (e.g. business executives) retrieving quantitative information from the dashboard interface (e.g. finances and accounting). Current research rarely addresses how information from a system containing both qualitative and quantitative data (as the UX data encompasses both) are retrieved and applied by users concerned with the dataset. We aim to expand on this topic in the case of UX data by including multiple relevant user groups: marketers, business developers, UX designers and researchers, product managers, and web analysts.

The UX Data

Usabilla is a start-up based in Amsterdam that provides SaaS (software as a service) solutions for websites to collect user feedback data. Usabilla’s service and tools allow users to place a feedback button on their website as a channel to ask their visitors for general feedback or specific questions about their website. A visitor can easily access this button to provide ratings, comments, and answer questions regarding their experience on the website. At the same time, automated data such as geolocation, browser, operating system, and device are also collected. Each individual report is called a “feedback item”. The data from the collected UX reports are then aggregated and displayed on a dashboard to provide a general overview to the user, with the option to access individual feedback items as needed. Thus, the user of the tool can then inspect and explore the individual as well as collective UX reports of the website’s visitors.

Creating the Dashboards

Taking the approach of user-centered design and development provides a means for dashboard designers to find solutions and improve the overall UX of the dashboard. Gould and Lewis (1985), suggest three principles of designing an effective and intuitive system:

• Understand the user group and the nature of the work they do with the system. • Measure user performance and reactions to simulations and prototypes of the system. • Follow a cycle of iterative design — test, measure, and redesign in multiple rounds.

In this experiment, we aim to follow the first principle based on the original dashboard design that is currently implemented on Usabilla’s website (Figure 1). The second principle is followed by using new dashboard design mock-ups to measure user performance with prototypes of the redesigned system. The intention is then to use the results from this experiment to meet the third principle, and integrate the gathered insights into the next iteration of the dashboard. The new dashboard designs (Figures 2 and 3-6) contain the same data types as the original dashboard, with the addition of several more UX data elements that are being considered as additions to the dashboard based on user needs.

In order to focus on the core functionality of the dashboard and understand users’ intentions and expectations, we did not implement interactive features, such as mouse-overs and tool tips on specific data points in the graphs, adjustable date ranges, and active hyperlinks to individual feedback items. Despite these limitations, results gathered from the user tests with the prototypes provide valuable insights highlighting which visualisations to include, remove, or modify in the following iteration of the dashboard. Further, the gathered user insights illustrate which interactions are necessary in successfully completing tasks and effectively interacting with the dashboard, and outline which elements of the dashboard contribute to either positive or negative user experiences.

Figure 2. Single-page format dashboard prototype.


—

Comment (preview)

—

Button doesn’t work. —

Link to homepage is… —

—

Great navigation, but…

Easy to use, clear, and…

Device Tablet Desktop Desktop Mobile Desktop Tablet Mobile Desktop Desktop 3.0 1.5 3.5 5.0 5.0 2.5 Rating 3.0 1.0 4.0 4 h 4 h 4 h 3 h 2 h 1 h 1 h 40 m 30 m Time

5 h 2.0 Tablet Would be better if this…

Last 30 Days 30

Feedback Items

811

204 488

Net Promoter Score

Detractors

15% 25% Passives 60%Promoters

NPS = % Promoters - % Detractors 45

-100 _-15 0 100

Net Promoter Score

Detractors 32% 13%Passives 55%Promoters NPS = % Promoters - % Detractors 23 -100 0 100 Latest Feedback

4.0

/ 3.0

17

/ 36

3.0

Top traffic sources

303 Newsletter 92 Facebook Entries 108 86 Direct 274 Blog Google Page User Engagement

Feedback Frequency Trendline

10 20 30 40 50 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15161718 19 20 21 22 23 24 25 26 27 28 Date Items 0

Feedback Items by Rating

1 2 3 4 5 Rating Items 50 100 150 200 250 0 1 2 3 4 5 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15161718 19 20 21 22 23 24 25 26 27 28 Date Rating Rating Trendline Feedback

Feedback Items by Label

Bug 162 Compliment 147 Suggestion 96 Question 89 Browsers Chrome Internet Explorer Firefox Platforms Linux Windows Mac OS X Devices Tech Stats 674

Comments Time before feedback

11 sec >101 76-100 25-50 1-25 0 51-75 Users Engagement by Country

Figure 3. Tabbed format dashboard prototype — open on Latest Feedback section. Tech Stats User Engagement Feedback Latest — Comment (preview) —

Button doesn’t work. —

Link to homepage is broken. —

—

Great navigation, but design…

Easy to use, clear, and looks…

Device Tablet Desktop Desktop Mobile Desktop Tablet Mobile Desktop Desktop 3.0 1.5 3.5 5.0 5.0 2.5 Rating 3.0 1.0 4.0 4 h 4 h 4 h 3 h 2 h 1 h 1 h 40 m 30 m Time

5 h 2.0 Tablet Would be better if this link was…

Latest Feedback

4.0

/ 3.0

17

/ 36

Today

Net Promoter Score

NPS = % Promoters - % Detractors -100 0 100

45

Figure 4. Tabbed format dashboard prototype — open on Feedback of Last 30 Days section. Tech Stats User Engagement Feedback Latest

Last 30 Days

Feedback Frequency Trendline

10 20 30 40 50 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15161718 19 20 21 22 23 24 25 26 27 28 Date Items 0

Feedback Items by Rating

1 2 3 4 5 Rating Items 50 100 150 200 250 0 1 2 3 4 5 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15161718 19 20 21 22 23 24 25 26 27 28 Date Rating Rating Trendline

Feedback Items by Label

Bug 162 Compliment 147 Suggestion 96 Question 89 Feedback Items

811

3.0

Figure 5. Tabbed format dashboard prototype — open on User Engagement section. Tech Stats User Engagement Feedback Latest

Last 30 Days

Top traffic sources

303 Newsletter 92 Facebook Entries 108 86 Direct 274 Blog Google Page

674

Comments Time before feedback

11 sec

811

Net Promoter Score

NPS = % Promoters - % Detractors

23

3.0

Figure 6. Tabbed format dashboard prototype — open on Tech Stats section. Tech Stats User Engagement Feedback Latest

Last 30 Days

65

%

15

%

20

%

811

3.0

Participants

The study consisted of ten participants, four males and six females, between the ages of 24 and 48. All participants had normal or corrected-to-normal vision and none were colorblind.

Participants were recruited through a database of organizations that use the Usabilla tool and have high-traffic websites (500,000 or more visitors per month), thus, in turn have high volumes of user feedback reports. All selected participants were considered expert users of the UX data dashboard to visualise their users’ feedback, meaning they regularly use the system to perform tasks required for their professional and organisational needs. Participants reported their role in their organization from a general list, with the option to specify if their role did not fit in any of the given categories. Some participants had multi-functional roles that fit into multiple categories, so they were encouraged to select their primary role. Of the ten participants, four were in marketing and business development, three were web analysts, two were product managers, and one was involved with UX research and design. Four participants reported that they used other tools and resources to collect supplementary UX reports and feedback from their website visitors.

Pre-testing questionnaire

Prior to participating in the study, participants completed a survey with questions designed to give an idea about their background and intentions with a user feedback dashboard (see Appendix A). The questions were used to categorize the participants into different user groups, to determine their level of knowledge and experience with a user feedback dashboard, and to outline their cognitive goals with the information extracted from the dashboard. These reports were used to compile a list of tasks that the users currently perform using the dashboard in practice, as well as tasks they would like to be able to complete using the dashboard. The information-seeking tasks in the test sessions were designed based on these reports. The rationale for using tasks that actual users performed is that this allowed us to observe how dashboard formatting influences changes in users’ behaviours and expectations, specifically with regards to performing visual queries, identifying the correct location and source for required information, and reporting the result.

Test Sessions

Each session lasted approximately 45 minutes and was held in a quiet meeting room or office. All participants conducted the session on a 13,3-inch (1440 x 900) Macbook laptop. The audio, video, and user’s mouse movements of each session were recorded using Screenflow software for later analysis. At the beginning of each session, the participant, or user, was instructed to follow the think aloud protocol (described below) as they interacted with the dashboards and performed the tasks.

The interactions with the two different dashboards were conducted using a repeated measures design. The user was first presented with either the single-page dashboard (referred to as Dashboard A) or the tabbed dashboard (referred to as Dashboard B) and asked to answer several questions pertaining to attention and perception. The user was then presented with the alternative dashboard design and asked to answer the same questions for this dashboard. To reduce the effects of presentation ordering, we counterbalanced by assigning half the users to view Dashboard A first and the other half to view Dashboard B first. During this time and in answering the questions about perception and attention, the user was able to explore the dashboard, to familiarise themselves with the format and grouping of information, learn the navigation, and ultimately, join these interactive experiences together to create an internal mental model of the dashboard.

The subjective questions pertaining to how the user perceives the dashboards, as well as the information and visualisations displayed in them, are listed below. Each question was followed up with “Why?” to encourage the user to provide more in-depth insights about their subjective reports:

• Which elements draw your attention the most? • Which elements to you like the least?

• Which elements do you like the most?

• Which elements do you consider most relevant for your needs? • Which elements do you consider irrelevant for your needs?

After going through these questions for each dashboard, the user then completed the

information-seeking task portion, as determined by the tasks reported by users in the pre-test questionnaire. The tasks were meant to be open-ended instead of multiple-choice so that the user could go through and describe a natural and unrestricted information-seeking process. The user performed each task on each dashboard in a randomized order. The assignment on which dashboard to perform the task was also randomised. Thus, the resulting sixteen possible combinations of dashboard and task were instructed to the participant in a randomized order. The eight tasks and their respective correct answers are listed below:

• Task 1: Which date in the past month had the highest average rating? Answer: 23rd.

• Task 2: Was there a comment given for the lowest-rated feedback today? If so, what was the comment?

Answer: Yes, “Button doesn’t work.”

• Task 3: Which countries had the highest user engagement in the past month? Answer: The Netherlands, the UK, and Denmark.

• Task 4: How many feedback items were labeled as bugs in the past month? Answer: 162.

• Task 5: How many users visited your site through the newsletter in the past month? Answer: 108.

• Task 6: From which platform did users most frequently provide feedback in the past month?

Answer: iOS.

• Task 7: How many feedback items were given on mobile devices? Answer: 204.

• Task 8: What is today’s Net Promoter Score, and how does it compare to yesterday’s score?

Answer: Today 45, Yesterday -15.

After finishing the last task, participants then completed a questionnaire adapted from the System Usability Scale (SUS, described below) for each dashboard. The presentation order for which dashboard the questionnaire was associated with was also counterbalanced in

accordance with the order the dashboard was viewed during the first portion of the session. The SUS questionnaire asked the following ten questions, each to be assessed on a five-point Likert scale ranging from “strongly disagree” (1) to “strongly agree” (5). Participants were encouraged to continue thinking aloud when assessing each statement and explain why they gave the ratings they selected.

1. I think that I would like to use this dashboard frequently. 2. I found the dashboard unnecessarily complex.

3. I thought the dashboard was easy to use.

4. I think that I would need the support of a technical person to be able to use this dashboard. 5. I found the various purposes of the dashboard were well integrated.

6. I thought there was too much inconsistency in the dashboard.

7. I would imagine that most people would learn to use this dashboard very quickly. 8. I found the dashboard very cumbersome to use.

9. I felt very confident using the dashboard.

Finally, to conclude the test session, participants were asked to provide a brief retrospective and subjective report of their experiences with each of the dashboards by answering the following questions:

• Is there anything you think should be added to the dashboard? • Is there anything you think should be removed from the dashboard? • Is there anything else you would change about the dashboard? • How did you find the navigation of the dashboard?

• How would you describe your overall experience in using the dashboard? • Is there anything else you would like to add?

Think Aloud Protocol

The think aloud protocol is a well-established means to capture qualitative insights from system users in the research phase of the user-centered design process (Ericsson & Simon, 1980; Holzinger, 2005; Van Someren et al.,1994; Yen & Bakken, 2009). Users are asked to verbalize their thought processes and actions as they perform a given task or interact with the dashboard. The think-aloud protocol is commonly applied in user testing in order to evaluate the salient positive points and obstacles in usability. Further, it is used to assess task performance while revealing why the user takes the actions they take in interacting with the system and completing the assigned task (Holzinger, 2005; North, 2006; Yen & Bakken, 2009). However, verbalizing actions and thoughts while completing a cognitive task, such as information-seeking, is not natural behavior for most users. Thus, one drawback of the think aloud protocol is that it takes more time to verbally detail the executed action sequences and driving thought processes. The think aloud protocol may then slow down the users’ task completion rate, however, does not generally interfere with performance (Ericsson & Simon, 1980). As this experiment was not concerned with speed as much as accuracy in completing information-seeking tasks, this factor was not a hindrance in using the think aloud protocol for gathering user experience reports from the dashboard users.

Data gathered from the think aloud protocol is primarily qualitative. The issues or positive points encountered in completing tasks provide the “what” of the user’s experience — identifying these key points, quantifying how frequently they are encountered, and at what time points they are encountered. Complementing this, the articulated comments from participants provide the “why” — the reasons a certain visualisation was confusing to the user, personal suggestions for functional improvement, or explanations why a sequential process went smoothly. Drawing from Ericsson & Simon's (1980) traditionally advised and followed analysis methodology, think aloud data is afterwards transcribed and reviewed in thorough detail. In the analytical review process, we detailed when participants are able to explain why they encountered an issue when they did, assess if they were aware they were encountering an issue, draw correlations between what participants said and what they did, and overall, evaluate the dashboard’s effects on the user’s behaviors. Flaws in the dashboard manifest themselves as apparent and obstructive issues

across analyses of individual participants. These issues may range from minor hindrances that cause delays or confusion in successfully completing the task to severe obstacles that

completely prohibit a user from completing the task efficiently and successfully. The types of issues encountered are then grouped into top-level categories to be focused on for

improvement in the next iteration of the dashboard.

System Usability Scale

The System Usability Scale (SUS) is a commonly applied usability questionnaire in UX research and industry, widely accepted for its robustness and reliability (Bangor et al., 2008; Brooke, 1996; Lewis & Sauro, 2009; Tullis & Stetson, 2004). The perceived usability of the system can be quantified using the SUS even with a relatively small sample size, as is the case in this study (Tullis & Stetson, 2004). The SUS is a quick and efficient way for researchers to gather

subjective reports about a system’s usability from their users and translate these subjective reports into a quantifiable value. The SUS questionnaire consists of ten questions on a five-point Likert scale ranging from “strongly disagree” (1) to “strongly agree” (5), with the key difference that the original version of the SUS uses the word “system” instead of “dashboard”. The SUS is versatile and easily adapted to the needs of the research project at hand to specify the type of system the questionnaire is referring to. The ten questions are divided into five positively worded and five negatively worded questions in order to avoid a positivity bias (Brooke, 1996). Initially, the SUS was meant to describe the usability of a system, but more recently, Lewis & Sauro (2009) illustrated how further information can be extracted from the responses in the SUS, as questions 4 and 10 describe the learnability of the system as well. Thus, the SUS can be considered a measurement of two interrelated dimensions of the user’s experience — usability and learnability.

The SUS score is calculated by first converting the user responses to a new value per user: by subtracting one from the user response for odd-numbered questions and subtracting the user response from five for the even-numbered questions. These individual values are summed and multiplied by 2.5, converting the possible values to range from 0 to 100. This conversion is repeated for each SUS questionnaire completed. From decades of SUS questionnaire results and data reflecting a broad range of interactive systems, the average SUS score is 68, marking this score at the reference of 50th percentile in terms of usability compared to other interactive systems (Bangor et al, 2008; Brooke, 2013).

Results

Average error rate

Dashboard A (Single-page): 0.866 (std = 0.841; range = 2) Dashboard B (Tabbed): 0.966 (std= 0.701; range = 2)

There was no significant difference between the average error rates between the two

dashboards (t(9) = 0.408, p = 0.693, std = 0.245). The average difference in error rate between Dashboard A and Dashboard B was 0.5 (std =0.577; range = 1). Individual error rates per participant per dashboard are illustrated in Figure 7.

Figure 7: Individual error rates per participant, compared against the two dashboard designs.

Error Rates per Participant

Err or Rate 0 0,5 1 1,5 2 2,5 Participant Number 1 2 3 4 5 6 7 8 9 10 Dashboard A Dashboard B

Click path on tabbed format

On Dashboard B, the total number of average clicks in path to correct tab containing the required information was 1.5 (std= 0.437, range 1.375).

In calculating the clicks in the path, the number of clicks was “0” if the dashboard was already open to the correct tab and the user understood this immediately. As the tasks were presented in a randomized order, the tab on which the dashboard was open at the beginning of the task was effectively randomized as well. Individual participants’ average clicks in path are shown in Figure 8.

Figure 8: Individual average clicks in click path per participant on Dashboard B.

Average Number of Tab Clicks in Path

Number of Clicks 0 0,5 1 1,5 2 2,5 Participant Number 1 2 3 4 5 6 7 8 9 10

Reported Preferences

At the end of each session, participants reported a preference to either dashboard format, distributed as follows and illustrated in Figure 9:

Dashboard A (Single-page): 6 participants Dashboard B (Tabbed): 3 participants Neither: 1 participant

Figure 9: Distribution of preferred dashboard format across ten participants.

Preferred Dashboard

System Usability Scale

Dashboard A (Single-page) SUS: 74.75 (std = 10.169; range = 27.5) Dashboard B (Tabbed) SUS: 75.50 (std= 9.113; range = 22.5)

There was no significant difference between the average error rates between the two dashboards (t(9) = 0.153, p = 0.882, std = 4.903). Individual SUS scores per participant per dashboard are displayed in Figure 10.

Figure 10: Individual SUS Scores per participant, compared against the two dashboard

designs.

SUS Scores per Participant

SUS Scor e 0 25 50 75 100 Participant Number 1 2 3 4 5 6 7 8 9 10 Dashboard A Dashboard B

II) Qualitative Results

Information-Seeking Tasks

In assessing the qualitative data from subjective participant reports, each issue encountered per information-seeking task was categorized into a top-level category to which the source of the issue could be attributed. In-depth detailed reports are in Appendix B, and an overview of the issue categories encountered per task are summarised in Table 1. Further explanations are found in Tables 2-4, summarising the positive findings from each task, the negative findings from each task, and the overall strengths and weaknesses of the two dashboard designs. The qualitative analysis followed the following operational structure to define key points per task and issue:

Task: The assignment posed to the participant during the test session.

Goal: The required information to be found in order to successfully complete this task. Positive points (if applicable): Elements that contributed to a positively perceived user

experience and in successfully and efficiently completing the task.

Dashboard: Specification whether an issue was encountered on only one or both of the

dashboards. The issue could be attributed to a design flaw in one or both dashboards, or, when there was only one instance of the issue, could be attributed to chance as the task and

dashboard appearance orders were randomized.

Issue: A top-level, categorical description of what the encountered problem was.

Explanation: Observations and details about what participants said and did while encountering

this issue.

Conclusion: A further detailed description of the issue, based on the observations. Severity: Determined by either or both of the following criteria:

• How many participants struggled with completing the task due to this issue. • 0-1 participant: Low severity. Struggle with task may be due to individual

differences in experience, knowledge, or expectations of the dashboard. However, participant was still able to complete the task successfully. • 2-4 participants: Medium severity. Most participants did not encounter this

problem and were successfully able to complete the task. However, a few participants still struggled with completing the task efficiently.

• 5 or more: High severity. Half or more of the participants that encountered this issue were critically impeded in completing the task and/or failed.

• High severity: One or more participants encountered an issue that created a significant hindrance in their experience, so they were unable to complete the task successfully.

Possible solutions: Ideas for changes to be made in the next iteration of the dashboard that

Table 1. Summary of main issue categories from user report analyses, split by task and

dashboard. (Note: A dash (—) signifies that there were no issues encountered specifically on this dashboard or no duplicate issues were encountered across both dashboards)

Task Dashboard A Dashboard B Both Dashboards

Which date in the past month had the highest average rating? • Visual presentation • Labelling • Limited details available on vizualisation • Visual presentation • Lack of confidence

Was there a comment given for the lowest-rated feedback today? If so, what was the comment?

— — • Use of numerical

figures in table

Which countries had the highest user

engagement in the past month? — — • Visual presentation • Limited details available on visualisation. • Lack of confidence. How many feedback

items were labeled as bugs in the past month?

• Visual presentation • Visual presentation —

How many users visited your site through the newsletter in the past month?

• Visual presentation • Visual presentation —

From which platform did users most frequently provide feedback in the past month?

• Data visualization — • Data visualisation

• Limited details available on visualisation. • Word choice. • Lack of confidence. How many feedback

items were given on mobile devices? — • Repeated data visualizations • Duplicate and conflicting information displayed. • Visual presentation. What is today’s Net

Promoter Score, and how does it compare to yesterday’s score?

— • Data visualization • Visual presentation.

• Lack of confidence. • Labelling.

Summary of Positive & Negative Findings, Issues, and Conclusions from User Reports Table 2. Positive findings from user reports, split by category and dashboard.

Positive Findings Dashboard A Dashboard B Both Visual Presentation • Easier to get an overview of general performance when everything is on one page.

• Large, bold numbers are clear and easy to read.

• Tabs allow more information to be presented with less clutter.

• Able to switch to desired information when needed.

• Easy to hide information user might not be interested in seeing.

• Participants understood and used the icons for mobile/tablet/desktop easily and effectively to identify breakdown of quantity of feedback items by device.

Labeling • Trendline subheaders were clear and made finding the correct visualisation easy.

None for this dashboard. • The tables had a clear subheader and column labels were easy to remember where to find correct information. • Bar charts had clear

subheaders and labels for the bars.

• Pie charts had clear subheaders. Use of choropleth map for proportions of user engagement • Map is unique visualization on the page, easy to locate.

• Easy to find map under “User Engagement” tab, possibly because the question contained this phrase.

• All participants intuitively knew this geographical question must be related to the only map on the dashboard, so they understood where to look.

• Color contrast between different value ranges was sufficiently distinguishable.

Word Choice None for this dashboard. • For some participants, the tab titles helped in searching for the correct tab with correct

information.

• Participants understood to look under “Tech Stats” for a figure describing technical details, thus the title of this section/tab is appropriate to group this

Table 3. Negative findings from user reports, split by category and dashboard.

Negative Findings Dashboard A Dashboard B Both Visual Presentation • Two trendlines

representing two different datasets are too similar in

appearance.

• Too much information/ too many

visualisations on one page made it difficult to scan and query for certain elements in the middle of the page (i.e. bar chart by labels). • Long scrolling

obscures information from part of the dashboard. • Difficult to share

conclusions form past month with

conclusions from today.

• Sequence of tabs is predefined and isn’t necessarily an intuitive order for the user. • Can’t visually compare

information across multiple tabs easily. • Having information spread across tabs does not give an instant overview of website performance.

• X and Y axis labels are too small on

trendlines.

• Map is too small to read in detail and recognize darkest blue regions.

• Dashboard lacks sufficient visual or textual indication where “Traffic

Sources” table would be found.

• Tablet and mobile icons similar in appearance and lack clear labels.

• Unclear mapping between the styling and color of “Today/ Yesterday” in the header and the colors in the NPS graph itself.

Labeling • “Last 30 days” header does not clearly illustrate its

relationship with the bottom three subsections -- “Feedback”, “User Engagement”, and “Tech Stats”.

• The visual relationship between “User

Engagement” as a header for this section and “Traffic Sources” as a piece of

information under this

• “Last 30 days” header does not clearly illustrate its

relationship with the top three tabs it is found under -- “Feedback”, “User Engagement”, and “Tech Stats”. • Unclear distinction between the NPS score of yesterday and the NPS score of the last 30 days.

• Unclear indication of timeframe for either NPS graph in either dashboard format. • Placement of NPS

score of yesterday below the line is a point of confusion.

Limited details available on visualisations

None for this dashboard.

None for this dashboard.

• Lack of interactivity to provide further detail on demand per data point.

• Mouseover, zoom functionalities would help alleviate this issue.

Lack of confidence None for this dashboard.

None for this dashboard.

• The interfaces lacked sufficiently distinct textual and visual cues as well as interactive drill-down features to provide sufficient information, enable participants to verify their answer, and gain confidence in their responses.

Use of numerical figures in the latest feedback table

None for this dashboard.

None for this dashboard.

• Numerical figures are difficult to scan and interpret quickly and intuitively.

Word Choice None for this dashboard.

• The semantic relationship between “User Engagement” as a header for this section and “Traffic Sources” as a piece of information under this group is unclear.

• The use of the word “platform” in both the question and the interface to refer to this particular dataset is ambiguous and unclear to some users as the word “platform” can have multiple meanings in technology.

Data Visualisation • Pie chart visualizations are too small to see clearly.

None for this dashboard.

• The largest and second-largest donut chart pieces were too close in size and the visualisation was too small, so it was difficult to discern the larger of the two.

• Incorrect response for NPS was perceived as a ratio of scores rather than a comparison. • There is no scale

provided for maximum rating (e.g. out of 5).

Repeated data visualisation

None for this dashboard.

• Showing the

necessary information multiple times on tabbed dashboard (on 3 of 4 tabs) can be a point of confusion in constructing a mental model of the

dashboard, searching, and retrieving this information.

None for both dashboards.

Duplicate and conflicting

information displayed

None for this dashboard.

None for this dashboard.

• Error in mockup data numbers and having two visualizations for a similar purpose (one showing raw numbers, one showing

percentages), leading to conflicting

User-reported Strengths & Weaknesses of the Dashboards

According to the aggregated subjective reports from all ten participants, each dashboard had its own advantages and disadvantages in terms of visual appeal, navigation, and formatting. Table 4 summarizes the reported advantages and disadvantages.

Table 4. User-reported strengths and differences for the two dashboard designs.

Strengths Weaknesses

Single-layered dashboard

• Scrolling navigation is easy.

• Easier to scan for correct visualisation rather than search for the correct section first to find this visualisation.

• Too much (irrelevant) information on one page to get a clear overview of website performance.

• Scrolling up and down makes it difficult to share conclusions from information found in top and bottom of the page. • Order of appearance for elements is

pre-defined and not customizable, thus the most relevant information is not necessarily at the top, the easiest location to access.

Multi-layered dashboard

• Tabs provide more structure.

• Easy to exclude (irrelevant) information you are not interested in.

• Easy navigation if user knows what information is under each tab.

• Clearer separation of related grouped information.

• Need to remember what information is grouped under which tab.

• Visually, the grey coloured tabs are not immediately noticeable.

• It is more work to click through tabs in searching for information rather than scanning a single page.

• Breaking information up across several tabs does not provide an overview of the overall website’s performance in one glance.

• Names of tabs are not always indicative of what information may be found there.

Discussion

Overall, more participants (6 total) reported a preference for the single-page dashboard format than the tabbed, multi-layered format (3 total). These individual participants qualitatively

reported a more positive experience in interacting with the single-page dashboard, with respect to navigation, performing visual queries, and ease of use than with the multi-layered dashboard. However, based on the quantitative subjective reports from the SUS, there was not a statistically significant difference in perceived usability between the two dashboard formats. The single-page dashboard rated slightly lower than the tabbed dashboard, yet more participants qualitatively reported a preference for the single-page dashboard. This preference was commonly attributed to being able to view all the information at once rather than breaking up the information. This discrepancy between the perceived usability (as measured by the SUS) and the verbally reported reflections on the dashboard experience suggests that usability — a complex factor in itself — is just one of several components that interact with each other and contribute to the overall user experience.

Usability describes the ease of use of a system as a result of the users’ visual, behavioral, cognitive, and emotional processes. However, usability does not describe these processes on their own. Asking users to verbally describe why they chose the SUS scores in the

questionnaire could possibly provide explanation as to why the quantitative usability metrics scored as they did. Thus, it is important to compare qualitative data to quantitative results in order to further understand the multiple processes involved in perceiving usability and developing an internal preference for the two dashboard options. For example, in the quantitative results we see that the average error rate was slightly lower for the single-page dashboard than the tabbed dashboard (although the difference is not statistically significant). Qualitative data allows us to understand these quantitative results — whether the error rates were increased or decreased due to the visual presentation in the dashboards, the choice of data visualizations, or simply not knowing where to look for information in the dashboard. Trends in errors and user-reported explanations of their thought processes in committing these errors illuminate the flaws and obstacles of the dashboard designs.

Another example is that the average number of clicks in the path to finding the correct information source in the tabbed dashboard was 1.5 clicks. This means that on average, the user did not go directly to the correct tab containing the desired information in one click,

indicating that they knew under which tab to find the information, or no clicks at all if the tab was already open on the correct tab and the user understood it was the correct place for the sought information. Linking the click path to the verbal reports illuminates whether the issue in

searching was due to how the information was grouped, the semantics of the tab titles, or the level of confidence the user had in their navigation, as participants sometimes clicked through

Further, a user may have reported equally perceived usability for the two dashboards in completing the SUS questionnaire, yet built their opinion on overall preference based on other influencing factors such as visual appeal, personal inclination towards interaction with or without tabs, and ease of accessibility to relevant information for that user. Questions about these factors are not included on the SUS questionnaire and are difficult to quantify, thus, we must rely on qualitative data to further explain the basis behind overall preference for either

dashboard. Another option to elucidate the users’ reasoning and perceptions would be to use a supplementary questionnaire specifically focused on these factors. However, relying only on questionnaire data without verbal subjective reports limits the depth and richness of the reported descriptions. This, in turn, would reduce the insights sought from the users to a confined

number of categories, whereas verbal reports provide the user with more freedom to reveal their perceptions, thought processes, and justifications of their actions.

Based on the combined results from the qualitative and quantitative data, we are able to specify which elements of the dashboard contributed to particular interaction costs and benefits to the user (see Results and Appendix for further detail). The accumulation of these elements

(formatting, data visualizations, visual presentation techniques, availability of details, word choices, etc.) adds up to the overall user experience. However, the results of this study may be limited in that we only tested eight different tasks across the dashboards, when in practice, users seek and apply information from the dashboard from a broader range of purposes.

To reiterate, the initial aims of this experiment were to investigate whether:

a) visual query and information-acquisition task performance differed in a single- vs. multi-layered dashboard designs.

b) the single- and multi-layered dashboard designs were subjectively reported as experienced differently by dashboard users.

Based on our results for (a), there was no significant difference in information-acquisition task performance measured by average error rate across participants. From these results, it remains unclear whether displaying all information in one view or splitting information by tabs facilitates task performance. There is currently little literature providing evidence for either format being optimal in performing information-seeking tasks or advising optimal strategies on how to group and present information in a multi-layered format. For more conclusive results, further evidence would be needed to understand the effects of dashboard formatting on cognitive processing and behaviors during information-seeking and acquisition. Possible directions for further research could include: varying and broadening the types of tasks users perform from the ones used in this study, creating several levels of task difficulty for comparison, using fully interactive dashboards instead of mock-ups, and testing with the same experimental framework using different types of datasets beyond UX to validate whether the same conclusions about formatting hold regardless of data types.

However, based on our results from (b), subjective reports regarding user experience did differ across the two dashboards. In terms of perceived usability, the multi-layered format scored

slightly, but not significantly, higher. However, based on verbal subjective preference reports, the single-layered format was generally preferred. The primary reason participants reported they preferred the single-page view was that it is easier to get a general overview on the website’s performance based on user feedback if everything is displayed on one page, rather than

comparing data and visualizations across multiple tabs. This insight is in accordance with Few’s (2006) recommended design principle that a dashboard should display all relevant information to fit on a single screen and provide a performance overview to the user in one glance.

However, the primary benefit users pointed out from the tabbed format was that it is easier to focus on a smaller subset of data and disregard grouped information they were not interested in. Grouping the information in smaller chunks generally minimizes the cognitive load on the user, but causes problems by increasing the cognitive load when the desired information is found across multiple tabs and the user must click through these tabs to access the information they wish to acquire. Thus, this format may be optimal for users who wish to only view data

presented in one of the tabs, but not for users who wish to compare or relate information across multiple tabs. Alternatively, allowing the user to customize the information grouping and

sequences as they see fit could make the tabbed format more optimal for viewing smaller chunks of information in an intuitive and logical manner for the user.

Our results were limited by a several possible confounds in the experimental design. The primary confound would be that the dashboards were reduced to mock-ups to focus on the dashboard’s core functionalities, and not fully functional with all planned features such as interactive graphs, drill-down options, and hyperlinks to individual feedback items. Some of these features, such as mouseovers for more detail and clickable hyperlinks are available in the previous version of the dashboard, so users were accustomed to the presence of these features and expected to see them available in the new versions of the dashboard. The expectations drawn from the users’ internal mental models of the dashboard may not have been met when performing the task, and the users had to adapt their behaviors accordingly. We intended to test this adaptive behavioral effect as a result of dashboard formatting, and not as a result of the individual visualization techniques. Thus, the lack of sufficient and expected drill-down and interactivity could be an influential factor as to why participants were not always able to find the desired information quickly or accurately.

Another possible confound in this experiment was that the values displayed on both dashboard formats were exactly the same, so it is possible some users correctly performed a task because they knew the correct answer from having already performed it on the alternative dashboard rather than performing the task in a natural sequence seeking novel information in their query. Several users reported that they learned how to read a reportedly problematic visualization (for example, the NPS gauge from Task 8 or the Labels bar chart from Task 4) from the first

dashboard on which they encountered this visualization, so they already knew how to answer the question without necessarily performing the task naturally and step-by-step on the second

Next Steps & Further Research

Based on the results of this study, it is unclear whether the general benefits of a single-layered, single-page dashboard design significantly outweigh those of the multi-layered, tabbed design or vice-versa. Although users reported a preference for the single-page view dashboard, it is necessary to take the next steps in the iterative design process for deeper user insights and more concrete results. This experiment is only a starting point to expand the current lack of sufficient literature on this topic in interaction design and UX research, so further experiments are needed to find stronger evidence to support conclusions about the effects of dashboard layer formatting on user cognition, perception, and behavior.

Following the principles of user-centered iterative design, the insights gathered from this study should be incorporated in creating the next version of the UX data dashboard. To summarize, this study elucidated the following key points as the features requiring the most attention on improvement. With the exception of the first two points, these suggestions for improvement apply to either dashboard design:

• Visual prominence of main section headers (“Today/Yesterday” and “Last 30 days”) on the single-layered dashboard and construct a clear visual hierarchy with their

respective subsections (“Feedback”,”User Engagement”,”Tech Stats”).

• Word choice, semantics, and clear representation of a relationship between tab titles (currently, “Feedback”,”User Engagement”,”Tech Stats”) and information found in these sections.

• Visual presentation of the individual graphs and data visualizations — include

guidelines, clearer and more readable labelling, more distinct differentiation between graphs reflecting different datasets.

• Reconsider the appropriate visualization strategy for NPS and comparison between today and yesterday’s score.

• Ensure clear word choice and semantics of all visualization headers, subheaders, and labels.

• Implement interactive functions to provide drill-down functionalities and detail on demand — mouseovers, tool tips, date range selection, hyperlinks to individual feedback items, etc.

• Implement dashboard customisation so that users can select what data they would like to have available and hide data they find irrelevant and clutters the dashboard for them.

On a broader scale in the fields of UX and HCI, further research is required to provide conclusive evidence about the optimality of single-page dashboard vs. tabbed dashboard designs. As the subjective reports varied across the ten participants of the study, there does not appear to be a generalized optimal design strategy for all users’ needs. Individual preferences towards the dashboards and the elements included are heavily built upon the users’

In a future study, the same experimental design could be repeated, but broken down further to investigate optimising dashboard designs for specific user groups. The sample size and uneven distribution of user groups (business/marketing, web analysts, UX designers, and product managers) within the sample set from our experiment was too small for this purpose. Different user groups are presumed to have different cognitive abilities based on their experience, education, and trainings, as well as differing goal-directed behaviours with the information presented in the dashboard (Endsley, 2001; Hassenzahl & Tractinsky, 2006; Ottley et al., 2013; Yigitbasioglu & Velcu, 2012). Thus, future research should aim to investigate whether the subjective reports and differences in task performance are different across specific user groups and dashboard formats. This can be accomplished by using the same experimental paradigm with the application of a crossed design and a much higher volume of study participants. Further, the differing performances in dashboard interaction could then possibly be categorised as correlates of these users’ cognitive differences across groups and cognitive similarities within groups. If there are significant differences across groups, then these insights could be used to construct an optimal dashboard design per user group. If there are no correlations among design formatting across or within user groups, then it may be feasible that an optimal design strategy does exist for broader user groups rather than specific user groups.

References

Bangor, A., Kortum, P.T., & Miller, J.T. (2008). An empirical evaluation of the system usability scale.International Journal of Human-Computer Interaction, 24 (6), 574-594.

Bargas-Avila, J. A., & Hornbæk, K. (2011, May). Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In Proceedings of the SIGCHI

Conference on Human Factors in Computing Systems (pp. 2689-2698). ACM.

Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189, 194.

Brooke, J. (2013). SUS: a retrospective. Journal of Usability Studies, 8(2), 29-40.

Card, S. K., Mackinlay, J. D., & Shneiderman, B. (Eds.). (1999). Readings in information visualization: using vision to think. Morgan Kaufmann.

Carroll, J. M. (1997). Human–computer interaction: Psychology as a science of design. International Journal of Human-Computer Studies, 46(4), 501-522.

Clark, A., & Chalmers, D. (1998). The extended mind. Analysis, 7-19.

Endsley, M. R. (2001, November). Designing for situation awareness in complex systems. In Proceedings of the Second International Workshop on symbiosis of humans, artifacts and environment.

Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological review, 87(3), 215.

Few, S. (2006). Information dashboard design (pp. 120-206). O'Reilly.

Forlizzi, J., & Battarbee, K. (2004, August). Understanding experience in interactive systems. In Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques (pp. 261-268). ACM.

Gentner, D., & Stevens, A. L. (Eds.). (2014). Mental models. Psychology Press.

Gould, J. D., & Lewis, C. (1985). Designing for usability: key principles and what designers think. Communications of the ACM, 28(3), 300-311.

Hassenzahl, M., & Tractinsky, N. (2006). User experience-a research agenda. Behaviour & Information Technology, 25(2), 91-97.

Holzinger, A. (2005). Usability engineering methods for software developers.Communications of the ACM, 48(1), 71-74.

Koffka, K. (2013). Principles of Gestalt psychology. Routledge.

Law, E. L. C. (2011, June). The measurability and predictability of user experience. In

Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systems (pp. 1-10). ACM.

Lewis, J R & Sauro, J. (2009) The Factor Structure Of The System Usability Scale. Proceedings

of the Human Computer Interaction International Conference (HCII 2009), San Diego CA, USA. Norman, D. A. (1986). Cognitive engineering. User centered system design, 31-61.

North, C. (2006). Toward measuring visualization insight. Computer Graphics and Applications,

IEEE, 26(3), 6-9.

Ottley, A., Crouser, R. J., Ziemkiewicz, C., & Chang, R. (2013). Manipulating and controlling for personality effects on visualization tasks. Information Visualization, 1473871613513227.

Plaisant, C. (2004, May). The challenge of information visualization evaluation. In Proceedings of the working conference on Advanced visual interfaces (pp. 109-116). ACM.

Pike, W. A., Stasko, J., Chang, R., & O'Connell, T. A. (2009). The science of interaction. Information Visualization, 8(4), 263-274.

Sutcliffe, A. G., Ennis, M., & Hu, J. (2000). Evaluating the effectiveness of visual user interfaces for information retrieval. International Journal of Human-Computer Studies, 53(5), 741-763. Tullis, T.S., & Stetson, J.N. (2004). A Comparison of Questionnaires for Assessing Website Usability. Proceedings of UPA Conference.

Van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. (1994). The think aloud method: A

practical guide to modelling cognitive processes (Vol. 2). London: Academic Press.

Vessey, I. (1991). Cognitive Fit: A Theory‐Based Analysis of the Graphs Versus Tables Literature*. Decision Sciences, 22(2), 219-240.

Vermeeren, A. P., Law, E. L. C., Roto, V., Obrist, M., Hoonhout, J., & Väänänen-Vainio-Mattila, K. (2010, October). User experience evaluation methods: current state and development needs. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending

Ware, C. (2013). Information visualization: perception for design. Elsevier.

Ware, C. (2005). Visual queries: The foundation of visual thinking. In Knowledge and information visualization (pp. 27-35). Springer Berlin Heidelberg.

Wolfe, J. M. (1994). Guided search 2.0 a revised model of visual search. Psychonomic bulletin & review, 1(2), 202-238.

Yen, P. Y., & Bakken, S. (2009). A comparison of usability evaluation methods: heuristic

evaluation versus end-user think-aloud protocol–an example from a web-based communication tool for nurse scheduling. In AMIA Annual Symposium Proceedings (Vol. 2009, p. 714).

American Medical Informatics Association.

Yi, J. S., ah Kang, Y., Stasko, J. T., & Jacko, J. A. (2007). Toward a deeper understanding of the role of interaction in information visualization. Visualization and Computer Graphics, IEEE Transactions on, 13(6), 1224-1231.

Yigitbasioglu, O. M., & Velcu, O. (2012). A review of dashboards in performance management: Implications for design and research. International Journal of Accounting Information Systems, 13(1), 41-59.

Appendix

Appendix A: Pre-testing questionnaire

■ Which category best describes your role in your organisation? ■ Marketing/Business Development

■ UX/Research/Design ■ Developer/IT

■ Other (specify)

■ Does your role involve feedback from your users? If "yes", for what purpose do you use this feedback? You can select multiple options.

■ Analytics ■ Design ■ Decision-making ■ Customer support ■ Bug reporting ■ Other (specify)

■ How long have you been using the feedback dashboard? ■ I have never used it.

■ Less than 1 month. ■ Between 1-6 months. ■ Between 6-12 months. ■ More than 1 year.

■ How often do you use the feedback dashboard? ■ Never.

■ Once a month.

■ Once every few days. ■ Once a day.

■ Multiple times a day.

■ What tasks do you perform using the feedback dashboard? ■ What are your goals in using the feedback dashboard?