1
DIVE+: User Interaction Design with Digital Humanities
Minyi Cheng
MSc Information Studies Assumburg 53 1081GB Amsterdam, The Netherlands
0031612299884
Minyicheng92@gmail.com
ABSTRACT
Previous user studies have shown that the User Experience (UX) of the DIVE+ User Interface (UI) is not optimal. There is a need to make the interface more logical and more intuitive for the user to use. In this paper, a few user studies were conducted in a user evaluation on the DIVE+ explorative search browser for digital humanities scholars. DIVE+ is a digital linked-data browser for cultural heritage institutions and focuses on supporting digital humanities scholars in their online explorations and research questions. Based on the results from previous user studies, a new DIVE+ UI/UX has been designed and implemented to address the main user needs for humanities researchers. In this research, we will examine how efficient and effective the DIVE+ browser is for digital humanities scholars with the following research question: “How efficient and effective is the exploration based on ‘events’-enriched collections and ‘narratives’ support for Digital Humanities scholars for their (1) understanding of objects, context and collections, and for their (2) discovery of new insights in the object of study?”. To find an answer to this question, a few user studies have been conducted to collect data from digital humanities scholars during their usage of DIVE+. A few of the main findings were that the tools events and narratives were not that effective or useful for the user to use due to shortcomings in the browser. Based on these findings, recommendations were made to improve the UI of DIVE+, and the UX for digital humanities scholars.
Keywords
Exploratory search; DIVE; User Interaction Design; User Interface; Usability testing; Digital Humanities
1.
INTRODUCTION
Many users search on the Internet today, use a traditional keyword-based search engine such as Google, Bing or Yahoo. Besides the traditional search engines, there are also exploratory search engines e.g. Europeana, Delpher and Euscreen.eu. Marchionini (2006) explains that exploratory search provides information retrieval to users that don’t have a clear information need. The goal of exploratory search is according to White et al., (2006) refining the not-so-clear information need by interacting with information objects and sources that leads to a better understanding of their problem or question. So, the difference between the users of traditional and exploratory search is that the users of traditional search have a clear information need and the users of exploratory search don’t. They are exploring a domain, to get insights and understanding of specific questions they are having. The exploratory search engine should then recommend interesting parts of the collection to the user in an interactively and iteratively way (Odijk et al., 2012).
White and Roth (2009) describe exploratory search as a process with an unclear information need at the beginning of the process. While researchers browse through an information stream, they discover relevant sources to their questions. Only there are some challenges for exploratory search, and that is to create an interface that supports all the needs of the user. This is difficult because for many users, there are different types of search strategies (Marchionini, 2006). Another challenge of exploratory search is to support the user by giving them an overview of the collection(s), and get familiar with this. Otherwise it will be too overwhelming and the user won’t be able to see the relevancy anymore.
To let users, make use of an exploratory search engine, a user interface should be created. To create a user interface for exploratory search, there are some main Human Computer Interaction (HCI) challenges. The first challenge is to reduce the insecurity about the unclear information need and increase the understanding with the user (White et al., 2006). This can be done by showing users relevant results that matches their query. The second one is to show the user if they are on the right path or
not, by showing them directly the results of the query
(Hutchinson, Bederson and Druin, 2006). And the third challenge is to show the relationships between the results and the query by giving feedback (Marchionini, 1997). All these challenges should be integrated in the exploratory interface to support the process of the user.
A use case of an interface for exploratory search, in the context of digital humanities scholars and cultural heritage, is DIVE+. DIVE+ is a digital exploratory browser with several cultural heritage collections that focuses on supporting digital humanities scholars in their development of research questions and their research explorations. To create interactive access between these digital heritage collections, linked data is used. The user can directly search through textual keyword queries, but is also able to search through structured links between cultural heritage objects and related entities. Digital humanities scholars rather use exploratory search than just regular keyword-based search because exploratory search gives them more links between relationships. In this way, they can explore more and stumble upon unexpected links (serendipity) between objects. Within DIVE+, there are additional research tools that could help the user search such as ‘narratives’ and ‘events’ (De Boer et al., 2015).
In previous research, Collijn (2016) did a user study on DIVE+ with digital humanities scholars. The goal of this research was to examine the possibility of DIVE+ supporting humanities scholars in their research. The conclusion was that the DIVE+ demonstrator didn’t completely support the researcher’s research. Research did also show that the browser DIVE+ itself does not in develop questions, there were upcoming questions with
2
researchers, but these questions were more about provenance (why is this related with this object). Later, in the research, Collijn (2016) also questioned several different user groups, such as students and (professional) researchers. They mentioned that the interface is not intuitive. Collijn suggested to ask interaction designers to criticize the designs and flows of the tool within a focus group.After the previous user study, the DIVE+ interface has been revived and a 2nd user evaluation is required to improve the
user experience of the browser. The main contribution of this user evaluation is to improve the DIVE+ suitability for digital humanities scholars, with the focus on the tools ‘events’ and ‘narratives’, by acquiring results and giving recommendations. This is done by:
● performing usability sessions within workshops; ● conducting user testing methodology e.g. think-aloud
protocol, simulated work task and online questionnaires; and
● collecting results will give insights on the needs and will be transformed into recommendations for the UI usage for digital humanities scholars
1.1
Problem Statement
After the findings of Collijn (2016), the development team has renewed the interface of DIVE, which led to a renewed browser and interface: DIVE+. This user evaluation will focus on the usability of the renewed interface of DIVE+. These are the problems that was raised by previous research that the exploratory browser DIVE+ has:
● the interface of DIVE+ is not logical or intuitive for usage
● digital humanities scholars are not able to efficiently use data and information in their research
● the overall user experience of the browser DIVE+ is not optimal
We will examine the new user interface of DIVE+ by focusing on the overall user experience of the DIVE+ interface and the perceived usefulness of the tools ‘narratives’ and ‘events’, for digital humanities scholars. The findings of this study will be converted into recommendations for the DIVE+ user interface to gain improvement in the user experience. Besides recommendations, we will also be looking at a new future user group: media professionals (journalists, documentalists and editors). The focus of this study will be on the digital humanities scholars, but we will also look at another user group: media professionals. Verhulst (2017) will be looking at the media professionals as a potential new user group that could be using DIVE+. We will be doing research on these two user groups and compare them with each other, to see if the media professionals have the same needs as the humanities scholars during their usage of DIVE+.
1.2
Research question
In line with the problem statement, the following research questions have been formed: “How efficient and effective is the exploration based on ‘events’-enriched collections and ‘narratives’ support for Digital Humanities scholars for their (1) understanding of objects, context and collections, and for their (2) discovery of new insights in the object of study?”
To properly answer this question, the following sub questions have been stated:
- what is the role of events and event types in understanding of objects and their context?
- what is the role of narratives for the discovery of new insights, objects, links and other information?
- how effective and efficient is the DIVE+ Browser UI/UX for the typical tasks that Digital Humanities scholars perform during their exploration search?
2.
RELATED WORK
This section briefly reviews literature relevant to this study. The next four topics will be presented: exploratory search, the rise of DIVE+, the research cycle of humanities scholars and we will scan other related studies and their used methods.
2.1
Exploratory Search
Searching on the Internet has become a daily information task for people. These information tasks can be divided into two groups; single-user and exploratory search. Web search engines, such as Google, are called the traditional information retrieval method or single-user search (Golovchinsky, Dirye & Dunnigan, 2012). Besides this traditional search, there is another variant, exploratory search. Users of exploratory search engines don’t have a clear information need. Instead, they are mostly exploring a topic and investigating other related topics (Odijk et al., 2012). Marchionini (2006) explains the different goals in information retrieval. According to him, the goal of single-user search is only to ‘lookup’, while exploratory search learns and investigates during every search query. See Figure 1 below for a visualization of the goals in both search variants.
Figure 1. Exploratory Search
Fox et al. (2006)has a different approach and describes exploratory search as making their users aware of the “big picture’’ by visualizing new insights, finding helpful connections, and discovering more relevant items. They believe in the importance of visualized interfaces as a natural and efficient way to support exploratory search through navigation and hypotheses generation. Since the user is mostly exploring and discovering new topics with exploratory searches, the results that comes from these exploratory searches will also be linked to other related items and objects.
2.2
The Rise of Dive+
However, linking items to each other is not new. It was already introduced with tagging. This was introduced by heritage institutions as one of the first ways to explore related items. Users
3
could add their tags to enrich the vocabulary of the institution so that other users could use these tags to explore the collections. After the introduction of social tagging by heritage institutions, the Agora project was created to facilitate new ways of exploring cultural heritage collections by utilizing metadata and other contextual knowledge (Van Erp et al., 2006). This project was a multi-disciplinary collaboration between the History and Computer Science departments at the VU University Amsterdam, the Rijksmuseum Amsterdam and the Dutch Institute for Sound and Vision.After the Agora project was created, DIVE followed-up (in 2015). DIVE is a digital linked-data browser for cultural heritage and focuses on supporting digital humanities scholars in their online explorations and research questions. It makes connections between media objects and other objects by using semantics from different vocabularies. To create connections and interactive access between the diverse cultural heritage collections, DIVE is using linked data. Linked Data is creating links between data and different sources through the World Wide Web. The data that are linked to the sources have a defined meaning, it’s also defined in a way that it makes it machine-readable (Bizer, Heath & Berners-Lee, 2009). After addressing a few shortcomings of DIVE, users can now search through keywords, structured links and related entities (De Boer et al., 2015).
In 2016 Kruijt conducted a research study with the previous DIVE browser. She examined how scholars could be supported during their exploratory stage of research, and how narratives can contribute in this. The conclusion was that there were interface design solutions for users that need support during their preliminary research phase, also there were technical developments that could contribute to this support. Meanwhile, Kruijt (2016) examined that, in theory, narratives could contribute in the exploratory stage of the DIVE browser, only this was not yet proven in practice.
In the previous research, Collijn (2016) performed a user study on DIVE+ with digital humanities scholars. The goal was to examine the possibility of DIVE+ supporting humanities scholars in their research. She defined four research stages (exploring, assembling, analysis and presenting) in humanities research and examined if the tool supported them during each of these stages. The conclusion was that the DIVE+ demonstrator didn’t completely support the researcher’s research stages. It seemed that the assembling stage was better supported than the exploration stage. Also, the goal of DIVE+ is primarily about developing new questions, secondly about answering questions. Research did show as DIVE+ does not in itself develop questions, there were upcoming questions with the researchers, but these were more about provenance (why is this related with this object). Collijn (2016) also questioned several user groups, such as students and (professional) researchers, they mentioned that the interface is not intuitive. The interface of DIVE+ has a lot of data and information which is not logical or intuitive for usage. Due to this shortcoming, the digital humanities scholars are not able to efficiently use those data and information in their research. Collijn suggested to let interaction designers criticize the designs and flows of the tool within a focus group.
2.3
Research Cycle of humanities scholars
According to Marshall & Rossmann (2014), the process of research in the humanities is a dynamic process in which the researcher and the research are constantly changing. The researchers vary in the humanities research cycle, between theory, practice, research questions, and personal experience. In figure 2 the research cycle is shown, in which a research project can start at any point, and the (research) question is formed by interacting with various cycle steps.Figure 2. Research Cycle of Marshall & Rossman (2010)
Stone (1982) recorded five steps that humanities scholars take during their research:
1. thinking and talking to people about the topic 2. reading what has already been written
3. studying original sources of information and making observations and notes
4. drafting a document on what is found 5. revising the draft into a final document
These stages are mostly similar to the stages that Collijn (2016) have stated. Collijn (2016) have mapped four research stages for digital humanities scholars, together with the DIVE+ team, literature by Melgar Estrada (2016) and Bron, Gorp & Rijke (2015). It’s valuable to know the research stages of the digital humanities scholars so that we gain information about their search behavior.
These four humanities scholars stages are: ● Exploring stage
● Assembling stage ● Analysis stage ● Presenting stage
The browser DIVE+ is an exploratory search engine, this means that it will be focusing on the first two stages of the humanities scholars. We will be holding on to these two stages of Collijn and use this information for this research. In table 1 the researcher’s purpose and the required support for these two stages will be viewed (Collijn, 2016).
Stage Researcher’s purpose Exploring - Idea generation
- Research question development - preliminary research
- decide if collection is suitable - decide sampling strategy
4
Assembling - information gathering- refine a research question/topic Table 1. Researcher’s purpose (Collijn, 2016)
2.4
Related (Usability) Studies
According to Dumas & Redish (1999) the primary goal of a usability test is to improve the usability of the product that is being tested. In this user evaluation, we will combine the methods of a few user studies with the focus on the usability between the system and the user. Now we will look at other studies who also conducted a research on exploratory research and which tools they have used compared to ours.
In the study of Manzari & Trinidad-Christensen (2006), a website was created for the library of the Long Island University (LIS). This website got examined for her usability by applying the methods heuristic evaluation and usability testing. Test participants were asked to perform real tasks while using the website, and their reactions and actions were recorded by the interviewer. Furthermore, the participants were asked to perform the think aloud technique while performing the task. The think aloud technique is a technique where the participants think aloud in order to capture their thoughts on the product (Rubin & Chisnell, 2008). Advantages of this technique are:
● it can help some participants focus and concentrate ● the interviewer receives early clues about
misconceptions and confusions of the product
● participants can reveal how they are thinking about performing a task and why elements work and don’t work for the user
However, there are also disadvantages of the think aloud: ● it can be unnatural and distracted for the user
● it can be exhausting to verbalize thoughts for very long ● the think aloud can prevent errors that otherwise might
have happened in the real workplace
Another method that was used in this study (Manzari & Trinidad-Christensen, 2006) was the heuristic evaluation. With this methodology, a small group of experts will individually review the user interface for design problems. After this review, the experts can discuss their findings with each other. Nielsen & Molich (1990) recommend to use three to five evaluators for this evaluation. They experience that the use of multiple experts will discover more bugs, but using more than five experts would not yield better results. Manzari & Trinidad-Christensen (2006) have combined these two methods with each other and discovered the biggest problem among users, which was navigating between the Web pages of the main library back to the LIS site.
Glowacka et al. (2013) conducted a task-based user study by comparing a traditional search with an interactive information retrieval system. The participants in the user study had to perform a task with a given system, where half of the participants were using a traditional search and the other half the information retrieval system. Chosen experts defined the context of the tasks so that it would represent an actual situation for the participant where the participant would be found within during work. By giving such a realistic task, the participant can accordingly interact with the system and give honest results and feedback (Borlund, 2016). Due to the given task that the participant had to perform, results have shown that the interactive information retrieval system exposed a higher number of relevant articles compared to the traditional search system. Another study conducted by Kules et al. (2009)
examined the interaction of searchers during exploratory searches. This was done in a special Human Computer Interaction lab, where the participants had to perform two search tasks with the same interface. In this study, they used tools, such as eye-tracking, while the participants had to perform the task. To gain a better understanding of the participants, e.g. how long and at what place they were looking at during an exploratory search, Kules et al. (2009) focused on gaze behavior. As a result of the performed task by the participants and the eye-tracking, results have shown that participants were spending more time looking at the facets then on the results.
3.
DIVE+
DIVE+ is a digital exploratory browser with diverse cultural heritage collections that focuses on supporting digital humanities scholars in their development of research questions and their research explorations. After the introduction of DIVE, DIVE+ was created in2016 as an extension of the DIVE project. This extension enlarges its collection size by adding several cultural heritage institutions e.g. Tropenmuseum, Amsterdam Museum and Dutch Institution of Sound and Vision. To create interactive access between these collections, linked data is used. Within DIVE+, there are also additional research tools that could help the user search such as ‘narratives’ and ‘events’. DIVE+ is now capable of making more connections between objects which leads to a broader narrative and a better understanding of the content (De Boer et al., 2015). As mentioned earlier, this research will examine the user interface of DIVE+ by focusing on the overall user experience of the DIVE+ interface and the usefulness of the tools ‘narratives’ and ‘events’, for digital humanities scholars. Screenshots of the browser DIVE+ can be found in appendix I. We will now shortly describe the tools ‘events’ and ‘narratives’ that are in DIVE+.
3.1
DIVE+ Exploration Search
Components
3.1.1 Events
According to Van den Akker et al. (2011) an event is a relationship between an object and an event that has a historical meaning. An event can be defined with the SEM model, which has four properties: actor, place, time and type (De Boer et al., 2015). These properties can be all historical objects e.g. a sword that is used in a war. These historical objects can be all linked to an event which leads to other linked events. In DIVE+, the user can explore through all objects, these objects are all described with metadata. When the user search for an object, related (media) objects will be shown to the user because e.g. they have the same location. This way the user can stumble upon unexpected related (media) objects which they didn’t expect. An example of an event could be a painting by Mohammed Toha on the attack of Yogyakarta, in which the metadata is that Mohammed Toha is the actor, Yogyakarta is the location and the time was 19 December 1948. The unexpected but related object could be that on the same time another historical event took place in Yogyakarta (Van den Akker et al., 2011). In the interface of DIVE+, there are filters with different event characteristics which the user can use to search for interesting objects, events or narratives. These different event characteristics can be seen in figure 2.
5
Figure 2. Filters of the event characteristics3.1.2 Narratives
As explained earlier, in DIVE+, historical objects can be linked to an event and this event linked to other events. This can lead to a network full of linked events which all have meaningful relationships with other events. Van den Akker et al. (2011) explains that narratives are two or more events that are somehow related with each other because they are involved with the same property (place, type or actor). So, in general a narrative is a sequence of events which can lead to an interesting angle/perspective. For example, the topic Beatrix, one user could find a narrative about her coronation and another user could find an interesting narrative about her relationship with her husband. These narratives can be saved in the tool: the exploration path (see figure 3). The user can add interesting queries to the path so that it won’t go lost and they can form a narrative. When a user search for a query, this query will be automatically added to the path. The user is also able to save the exploration path for later use or load existing ones.
Figure 3. The exploration path
4.
METHODOLOGY
In this research, we have performed extensive usability testing of the DIVE+ browser, with a specific focus on testing the usability of the tools events and narratives. We have conducted workshops where usability sessions were performed with the different user groups. The usability sessions consisted of a simulated work task situation where they had to interact with the exploratory browser DIVE+. Besides the usability sessions, the users have filled in an online questionnaire. After the questionnaire, a focus group gathered to discuss about their search behavior and their experience with DIVE+. For both workshops, there are protocols which can be found in appendix II and III.
4.1
Tool Development
To prepare for the workshops, data from DIVE+ co-creation sessions with Digital Humanities and Media Studies students from Hagedoorn & Sauer (2017) have been analyzed. These data include research diaries about the students’ general search behavior and use of DIVE+. Based on this analysis, a first draft of the tools, a questionnaire and a simulated work task, was created for each user group. After multiple iterations on the drafts, a first version was ready to be pilot tested on the 25th of April with 6 Human Centered Multimedia students. The primary goal of the pilot tests was to see if the tools were clear for the participant and to receive feedback. In figure 4 the process of the development of the tools have been visualized. The results from the pilot tests were that the students were overall positive about the two tools, only the simulated work task was a bit too broad which we refined by giving four topics to choose from in the task.
Figure 4. Process tool development
4.2
Workshops
In this section, the details of the workshops will be presented. In figure 5 a timeline of the workshops, DIVE+ usage and the methods have been visualized.
4.2.1 Workshop Digital Humanities Scholars
The first workshop (Workshop1_DH) was held on the 9th of May 2017 in the CreateSalon of the UvA with the humanities scholars. These researchers are the current target users of DIVE+ and have different backgrounds within the humanities. This could be students, academics, post-doc, researchers or lecturers. During the CreateSalon workshop there were around 11 digital humanities scholars that attended the usability session and filled in an online questionnaire. These current users will be taken as the main users for the UI/UX evaluation.4.2.2 Workshop Media Professionals
The second workshop (Workshop2_MP) took place at the Dutch Institute for Sound and Vision on the 24th of May 2017 with the media professionals. These professionals are a combination of journalist specialists, image researchers, cross media storytelling experts, and media coaches. During this workshop, there were 8 media professionals that performed a simulated work task during the usability session and filled in the online questionnaire. These users are potential future users which will be used as a comparison to the main users for the UI/UX evaluation.
6
Figure 5. Timeline workshops
4.3
Evaluation Tools
So, the following tools have been used during the workshops: ● DIVE+ browser
● Simulated work task situation ● Think-aloud protocols ● Online questionnaire ● Focus group
We’ll now go in detail about the tools that have been used.
4.3.1 DIVE+ Browser
The contextual framework of this research project will be DIVE+. The interface of DIVE+, which you can see in figure 6, will be used and the tools will be examined.
Figure 6. The DIVE+ Browser
4.3.2 Simulated Work Task Situation
For the workshops, we have created simulated work task situations. For the participants to be as genuine as intended, the work task has to be as realistic as possible to the participant. It has to be a situation where the participant can relate with and has to consist realistic information in order to properly interact with the system (Borlund, 2016). Looking at the working field of the different users, we have made two different tasks for each user group with each subsequent information. In the task, there is an introduction to the task and following steps with links to the DIVE+ browser and online questionnaire. In appendix IV and V the full text of both work tasks can be found.
● For the digital humanities, we created a contextual situation where they had to search for a research angle for a paper in an academic journal. We gave them four
topics (Jakarta, Islam, Beatrix and Watersnoodramp) which they could choose.
● The media professionals were given information about a potential program that is going to be produced with the same four topics. They also had to search for an interesting angle about one of the topics for the program.
4.3.3 Think-aloud protocols
Currently, a very popular and effective method during a usability session is the think-aloud protocols. Before the user had to perform a simulated work task with the information retrieval system, we asked them to think out loud during the task performance. It can provide useful qualitative data about the user having problems with the system (Krahmer & Ummelen, 2004). The think-aloud of the users have been recorded during the workshop and transcribed afterwards.
4.3.4 Questionnaire
After the participants finished their simulated work task situation, the participants were asked to fill in an online questionnaire about their experience with DIVE+. The questionnaire link was already provided in the task, they only had to click on the link. The questionnaire can also be found in the appendix VI. The questionnaire was divided in six parts:
● their task performance and which topic did they choose ● did they find an interesting angle
● personal experience with other exploratory tools ● the usefulness of events and narratives in DIVE ● general evaluation of DIVE+ and the UI ● demographics and background
4.3.5 Focus Group
During both workshops, there was a focus group where the participants could discuss their experience with each other. For the focus group, questions were prepared to start a discussion between the participants. These questions were about their search practices and their experience with the browser. Both focus groups sessions have been recorded and transcribed afterwards.
4.4
Limitations
During the execution of the user study there were a few limitations. The first limitation was the shortcoming of time. In the first workshop with the humanities scholars there was time reserved for an hour within the CreateSalon workshop. This was unfortunately not enough for us and the participants to fully commit to the task performance, questionnaire and focus group
7
session. Another limitation was the DIVE+ browser which gave some errors during the workshop. Some participants were unable to visit the browser and perform their task, a reason for this could be an overcapacity of visitors to the browser.5.
Results
In this section, we present the results of analyzing the data collected in this user evaluation for the digital humanities scholars. First, we explain how we analyzed the data. Second, the demographics of the user groups is shown. Then we present the results of the task performance that the participants had to perform. After that, we present the results of the participants experience with the tools narratives, events and their evaluation of the browser DIVE+. At last, an overview follows with the notable results.
5.1
Data Analysis
To analyze the data that was gathered, we developed an approach. The data that was gathered are sourced from:
● Online questionnaire responses
● The think-aloud recordings of the task performance ● Focus group recordings
These data were gathered from four different user groups. These user groups will now be referred to as Group_1DHS, Group_2MP, Group_3HS and Group_4CSS.
1. digital humanities scholars (Group_1DHS) 2. media professionals (Group_2MP) 3. (digital) humanities students (Group_3HS) 4. computer science students (Group_4CSS)
All the four user groups have filled in the online questionnaire, only the (digital) humanities students and computers science students did not have to perform a task because there was no physical workshop with them. They only received the online questionnaire without the task performance section.
First, we transcribed all the recordings from the think-aloud of the participants and the focus group. Second, the responses from the questionnaire were gathered and analyzed. From these transcriptions and responses, we collected the outstanding and corresponding results. As third and last step, we’ve made categories with corresponding results that we’re acquired from all the four groups. All these results can be found online in a Google document1. From all the results, the focus lies
on Group_1DHS since they are the focus of this research. Verhulst (2017) will be focusing on Group_2MP and these results will be included. To complete this analysis for a better scope, we also take the results of the other groups into account.
5.2
Demographics
• Digital humanities scholars
During the digital humanities workshop, 18 participants have attended the workshop. From the total participants, 11 have filled in the questionnaire. This was due to the fact participants had to leave early or multiple participants were performing the task and questionnaire
1
https://drive.google.com/open?id=1Uq4eGAmAD7oWa3zX6fQ PWLEI5yVkhj-1I-Ku3SRo8sk
together, which reduced the number of questionnaire respondents. Their backgrounds of are students (5) or scholars (6). For each of them had their own domain expertise e.g. art, history, literature or media studies. In the questionnaire, we also asked for their experience with linked data which 72.7% answered with some experience.
• Media professionals
From the 11 media professionals that attended the workshop, 8 participants filled in the online questionnaire. The reason for this was that some media professionals worked together on one laptop, which explains the number of questionnaire responses but reduces the reliability. The media professionals had varying backgrounds, 5 of them were from a broadcasting organization and 6 of them had a background in libraries/archives. 50% of this group answered with some experience with linked data. • (Digital) humanities students
These (digital) humanities students varied in digital humanities students and just humanities students. This is the reason why digital is put between brackets. This group did not have a workshop, but only filled in the online questionnaire. There were 16 students who responded on the questionnaire. The responses started in the beginning of the questionnaire with 16, but further on each question in the questionnaire the total responses became less. In the end, 81.3% of the students answered with some experience with linked data.
• Computer science students
The computer science students were in a number of 22 and as well as the previous group did not have a workshop. The same as the (digital) humanities students, the responses started with 22, but during the questionnaire it descended. 55% of these students had some linked data experience.
5.3
DIVE+ Task performance & research
angle
5.3.1 Task performance of digital humanities
scholars
For the simulated work task situation, the participants had to find a research angle on DIVE+. From the 11 participants, 5 participants found an interesting angle and the other 6 participants did not found an interesting angle. The 6 participants that didn’t find an angle, struggled with the interface. They needed time to figure out how the interface worked, which cost a lot of time. Group_1DHS participant 4: “It [the search results] also does not narrow down enough, which is why it feels like an information overload.” The participants also experienced problems with the data collection, they didn’t know what kind of collections existed in DIVE+, which made it hard to know what to search for. Group_1DHS participant 3 noted: “I didn’t know how to tailor my search to the collection (of objects & texts) because I didn’t know what the collection was”.
Besides the collection, the amount of data was also
experienced limited, as in information about the objects but also
the metadata. The participants tried to find help in the browser with the tool ‘related searches’ on top of the screen. After testing
8
the tool, the participant experienced that the given ‘related searches’ were not related to the given query. The participants eventually thought that it wasn’t even at all related to the query that they typed in or clicked on.5.3.2 Task performance of media professionals
The media professionals also had to perform a task, which consisted of finding an interesting angle for a potential program. From the 8 participants, there were only 2 media professionals that were able to come up with an angle. The remaining participants couldn’t come up with an interesting angle, they also experienced the amount of data as limited. The participants noted that they wanted more fine-grained selection and filters. In that way, they could filter between the many results. Also, they would like to see more information about the relations between the queries and results, and about why objects receive certain labels. Group_2MP participant 2: “It’s unclear why search results receive certain tags. Thereby it’s impossible to acknowledge any value to it.” Two media professionals noted that they couldn’t complete the task because of technical issues and limitations of the interface.5.4
DIVE+ Narratives
The results of the questionnaire regarding DIVE+ are shown in figure 7. The first chart represents the responses of the digital humanities scholars. As shown in the figure, a huge amount did not discover different narratives, when searching in DIVE+. Reason for this most likely is the limited time of testing therefore they could not fully discover the narrative tool, the exploration path, in DIVE+. Group_1DHS participant 7 state: “The search results showed a few different topics, but not narratives. And the exploration path is using in tracking search history, but did not in itself help me learn more about historical events.”
Group_1DHS also experienced that the results only showed topics, not potential narratives. They expected that DIVE+ would suggest or show different narratives, but the way they tried to link topics with each other didn’t result a narrative. The other groups were slightly agreeing with Group_1DHS, most of them also did not discover different narratives or were neutral. Group_1DHS would like to see suggested narratives to explore a topic in depth, this also counts for Group_3HS and Group_4CSC. These three groups agree with each other about suggested narratives and that they can be helpful to learn more about an event. Only Group_2MP, the media professionals, did not agree about the suggested narratives.
The tool exploration path is seen as a useful feature, by almost all the groups, only it did not help the groups learn more about the historical events. For Group_1DHS it was not clear how the tool was constructed. Some participants wanted to play with the tool, which was not possible and caused confusion. Group_DHS1 participant 11: “The navigational aspect confused me: I struggled with organising a cohesive research path, and found myself wondering why the things are organised in such an order, instead of looking for more material.” The tool was more seen as a list of the search results and not a tool to form a narrative. The participants discussed in the focus group that they would like to interact more with the path, so that they could build their own narrative with the founded results/queries. The interaction with the exploration path could be dragging and dropping queries, re-naming queries and categorizing with colors. Group_1DHS participant 11 recommended: “a more indented tree structure. Clearer exploration. Marking the narrative tree with various categories.” The other groups did not mention to interact more with the exploration path. Group_3HS, the (digital) humanities students, did mention that they would like to see the tool easier or clearer with instructions to use.
Figure 7. Results narratives
5.5
DIVE+ Events
During the use of DIVE+, the different event characteristics did not leave an (good) impression with the digital humanities scholars, or the other groups. Some participants of Group_1DHS did not notice the event characteristics and other participants did not understand the purpose of it. This was due to some of the
underlying function of some icons. They did not know what it represented, this confused them. Group_1DHS participant 8: “the icon for 'event' was confusing to me: I interpreted it for a while as denoting 12 events. The person category was not fully clear: there is no mention of who the person actually was, I needed to figure this out on the basis of the related events.” A small percentage of Group_1DHS, understood the meaning of the different
9
characteristics. In the other groups, the number of confused participants corresponded with Group_1DHS (see figure 8). Most of the humanities scholars argued, after discovering the different event characteristics, they wanted morefine-grained filtering. The current event characteristics are too
broad which did not narrow down the overload of results enough. Group_1DHS participant 3: “For me, the media object was way too general. I’m a media scholar, not a historian, so saying there are 400 media objects, includes from photos from a museum to.. It is not enough. I need a more granular selection. [the filters] didn’t eliminate the type of media what’s on offer for example, at least in a rough categorization.” This can be also seen in figure 8, where almost all groups responded that they
missed some characteristics of events. Only Group_4CSS was
very divided about this statement, some agreed, disagreed or were neutral.
The experience about the usefulness from the different characteristics was experienced as negative among the user groups, the tool did not help Group_1DHS to learn about the events. This also counts for Group_2MP and Group_4CSS. In Group_3HS, a small percentage did see the usefulness though. This negative outcome about the usefulness correlates with the discussed points above. The unclear purpose of some icons, uncertain meaning of the characteristics and the too general event characteristics led to a negative outcome. When these points are improved, the usefulness will also be clearer. Overall, the current meaning and usefulness of the different characteristics was not perceived positively among the digital humanities scholars.
Figure 8. Results Events
5.6
Evaluation of DIVE+ and DIVE user
interface
Group _1DHS noted that the current DIVE+ is not ideal yet, but it has potential. This is also shown in figure 9, the results of the questionnaire. These results show that the humanities scholars do see the usefulness and potential, but they would not use it in their daily work. Participants of Group_1DHS noted that they would like more information, such as more metadata, object description and from which source/collection this object originates. So, in general, more information would improve the experience with DIVE+.
Not only more information would help, also the meaning of buttons & icons should be intuitive. In the current interface, the meaning of some buttons is unclear which creates irritations and confusions. Group_1DHS participant 7: “I also had some trouble figuring out the meaning of some of the icons.”
Another lack in the UI was the feature ‘Related Searches’. They experienced that the feature does not show, to their opinion, related searches. This causes confusion, but also irritation with the participants. Group1_DHS participant 7: “The ‘related’ entities’ suggestions were not very helpful, and it was not clear how they were related.”
Compared to other groups, Group_1DHS was positive about DIVE+ and the UI. The other groups did not agree on almost any statements of the questionnaire. Group_2MP, Group_3HS and Group_4CSS did not find DIVE+ useful for the exploration of their topics. This can be because the collections in DIVE+, are not in their domains. DIVE+ did not inspire them for new ideas or had a satisfactory experience with the UI of DIVE+. Also, Group_3HS and Group_4CSS would not use DIVE+ in their daily work. Only Group_2MP, who had the highest percentage of all groups, would use DIVE+ in their daily work.
10
Figure 9. Evaluation of DIVE+
5.7
Findings
Given the results, we can make a few findings. The most notable findings are recorded, in table 2, as an overview. In appendix VII, the results of all the groups combined can be found. The full documents of findings with all supported quotes from all the user groups can be found online as a Google spreadsheet2.
General More information is needed about which collections are in DIVE+
Respondents needed more (meta)data and topic descriptions.
The amount of data was experienced limited
The ‘related searches’ was experienced not related to the given query
Events and event types
More fine-grained sorting of the media types is needed
Respondents had trouble with the event characteristics
Narratives Respondents find that the exploration path does not result in narratives For the respondents, it was not clear how the exploration path is constructed
2
https://drive.google.com/open?id=1YsR7L3Qe3da_D7TgspD8B MF71mWacXUl466czcA5x6Q
DIVE+ browser
UX/UI More interaction with the exploration path needed
For respondents, it was unclear what the meaning was of some buttons/icons Respondents needed time to figure out the interface
Table 2. Overview findings
6.
Discussion
The findings of the results point out several shortcomings of the exploratory search engine DIVE+. The findings were mostly drawbacks that needs to be improved to increase the UX of the browser. This leads to the creation of figure 10, the representation of the shortcomings of DIVE+ which leads to UI/UX problems.
One of these shortcomings are the event characteristics which are too broad. The DIVE+ browser can filter within event characteristics such as (person, location, media object and event), Only this was unsatisfactory to the participants. All the participants of Group_1DHS have their own special background e.g. one researcher is only interested in radio and the other in image. This counts for every user who is going to use DIVE+ in the future, this kind of fine-grained sorting filter is needed to sort between the results and to explore deeper into the collections. The participants of Group_2MP and Group_3HS agree with this finding, especially Group_2MP make use of these kind of fine-grained filters for in-depth exploration. Group_2MP respondent 2 state: “Eventually, how you turn it, the filters of person, event or date, are important. But you can never pass by the basic filters (image, sound, text) that should have been there from the beginning.” This was a significant finding between the Group_1DHS, Group_2MP and Group_3HS. The different event characteristics is needed to sort and filter between the different types of objects and understand what kind of entity it is.
A difference that was noticed was that the participants of Group_2MP needed to know why entities received certain labels. This group wanted to know what’s behind DIVE+, how
11
relationships were generated between entities. They needed more transparency in the DIVE+ browser, in order to understand the events and origin. This is due to the work background of the Group_2MP, they need to justify their articles or programs with reliable sources. This was a substantial distinction with the other groups. Group_2MP participant 6 state: “But what’s behind it. On the basis of which terms, which tags, is a relationship generated.” Another difference between the comments of the three groups was that Group_2MP also wanted to filter between collections, Group_1DHS and Group_3HS did not mention this. Group_2MP would like to, for example, exclude certain collections so that the focus lays on one collection. Group_2MP participant 4 said: “I want to see what’s in Delpher, what have been in the newspaper, because I don’t see that anywhere. I know that’s in Delpher.”Figure 10. Shortcomings of DIVE+
Another shortcoming was the purpose of the narratives tool. The groups, Group_1DHS, Group_2MP and Group_3HS, did not find that the exploration path results in narratives. This is due to the not-known purpose of this tool. The participants did not know what the purpose was of this tool and saw it more as a search result list. Group_1DHS participant 4 state: “I didn’t really see how this [exploration path] would help me with the narrative, or how I should see a narrative in there. I saw it more as a search results, or not even necessarily queries, that maybe combined gives a sense of my search history, but not necessarily a historical narrative. And the same actually with the results. I found topics in there that you might consider a narrative, but the way that I read them or try to link them wasn’t necessarily a narrative.” A lot of participants did not made use of the exploration path during the workshops because they thought it was a search result list. They were mostly busy analyzing the results, and searching for links between them. In my opinion, this is also due to the static visualization of the path, it does not attract people to form a narrative with it. The participants of Group_1DHS, mentioned that they would like to interact more with the tool. Group_1DHS participant 1 note: “What I would like is to change the search query. It places my search term, the imported search term as name of the path, only I would like to try Jakarta again, but then I would lay the focus different within the results. You must put the search term in a new exploration path, but it would be nice if you can edit the name of that exploration path. [..] Now I have two times Jakarta [as query in exploration path], that is confusing. Imagine that I would do that ten times, then I would have ten times Jakarta, then I would not know which is what.” If the users could interact more with the exploration path, the purpose of the
tool would be much clearer. The participants would be able to make their own narrative in the path and the tool would be much more efficient and effective. The exploration path would be much more user-friendly, and objects could be easier linked with each other which leads to the discovery of more new insights. There were a few suggestions of the participants for the interaction of the exploration path e.g. dragging-and dropping, re-naming queries and color categorizing.
The shortage of information and data is also seen as a shortcoming. Due to this lack, the UX is not optimal. When users want to use DIVE+, and they search for a query in the browser which gives zero results it will cause confusion with the user. When the user does receive results, but it gives not enough information to the user to learn from it will cause irritation with the user. The participants of Group_1DHS and Group_2MP, both indicate that they need more information and data about collections, topic descriptions and objects. Participant 3 of Group_2MP stated that the information given with a video was very limited. The participants need enough information to know what they are looking at in order to form an event or narrative. Group_1DHS participant 10 said: “I realized that I didn’t really know what I saw, because there was also little information to date the objects - I actually ended up going back to the original repositories objects came from (through the link provided) to find out more about what I saw. I think this sort of information in particular is a kind I miss here.“
What is interesting to see, is that both user groups, Group_1DHS and Group_2MP have their own way of doing research. Group_2MP are more focused on results, they would like to know the reason for the relationship between the objects and they want to know which source the object originates. Group_1DHS are more explorative and less critical about the results. This may be related to that Group_2MP have a different search behavior and purpose then Group_1DHS. The Group_2MP mentioned during the workshops that they are dealing with a time constraint, because they are mostly making an article item for a program such as a news program. So, if they are making a narrative, they need to know if the source is reliable and they need to substantiate the relationship.
The goal of the DIVE+ browser is to explore through the collections of cultural heritage institutions. Collijn (2016) have mapped the research stages of the digital humanities scholars, and in the first two stages, it’s more about exploring, deciding and gathering information. After analyzing the results and findings, the browser would not be that very effective and efficient for the humanities scholars. This is due to the shortcomings, which is discussed earlier. These shortcomings result in the lack of efficiency for scholars to perform typical tasks/queries during their exploration search. After implementing our recommendations, the DIVE+ browser would be more effective and efficient to use for the digital humanities scholars.
6.1
Recommendations
With the founding findings, recommendations were made to improve the browser DIVE+ and the UI. A few of these findings are visualized.
Finding 1: more information is needed about which collections are in DIVE+
For this finding there are two possibilities for the recommendation:
12
● provide details about the collection on the result objectpage, under the title ‘sources’
● include a section about the DIVE+ on the start page of DIVE+. In figure 11 an example of how it can be visualized.
Figure 11. recommendation collections
Finding 2: Respondents needed more (meta)data and topic descriptions, or the given information was not clear enough about the object
For some entities, there is very little information available. For this finding, more information and additional metadata should be added.
Finding 3: More fine-grained sorting of the media types is needed This finding has a few options for implementation:
● In the current interface, a ‘by collection/mediatype’ filter should be added for the user to have options to sort the results. The user can select multiple collections and media types from the filter.
● An extension of the current media object event characteristic. The extension should be when the user clicks on it, that 3 options should appear which the user can select to filter on. In figure 12 an example of this option is visualized.
Figure 12. recommendation filter
● Another option is to reduce the space on the left side of the screen. The filter ‘filter by keyword’ should be put above the exploration path and a new filter ‘filter by’, where the user can filter on collection/media object, should be also put on the left side. This will create less distraction on the results page. An example of the UI is shown in figure 13.
figure 13. recommendation UI Finding 4: related searches is not clear
The back-end algorithm should be adapted so that the related searches is only related to the current query.
Finding 5: respondents struggled with the interface
The interface and results page was experienced very cluttered, it would help if it was less cluttered. Besides that, the meaning of some buttons and icons were unclear to the users. This can be solved by putting the title of the button/icon in text together with the button.
Finding 6, 7 & 8: respondents noticed that the exploration path does not result in narratives, they have mixed attitudes about the current usefulness of the exploration path. More interaction with the exploration path could resolve these findings.
To let the user, interact more with the exploration path, users should have the possibility to drag-and-drop interesting queries in the path. Also, they would like to re-name queries/blocks in the path otherwise they will have the same queries a couple of times. It is recommended to give the user the possibility to make categorizations by colorization, the user can hereby make more narratives in one path.
7.
Conclusion
In this user evaluation, multiple user studies have been conducted on DIVE+ with digital humanities scholars, and other user groups. The focus of this research was to examine the usability of the renewed interface of DIVE+, in particular the tools ‘events’ and ‘narratives’. The problems that were raised by previous research was that the interface of DIVE+ was not logical or intuitive for usage. Due to these shortcomings, the digital humanities scholars were not able to efficiently use the data and information for in their research. To examine these flaws, and improve the overall user experience, workshops have been organized for different user groups. Usability sessions have been performed where the participants had to perform simulated work tasks with a think-aloud protocol. Additionally, they had to fill in a questionnaire and a focus group session was held after. The focus of the user groups lies on the digital humanities scholars, but for a better scope, we will also take the results of the other groups into account.
The main findings from these results were (1) the lack of information in DIVE+ e.g. participants needed more information about which collections there are, more (meta)data and more topic descriptions. Another finding was (2) that
13
participants of the user groups did not see the purpose of the exploration path. The digital humanities mentioned that they would like to interact more with the path to form a narrative. And (3) participants had trouble with the event characteristics. The characteristics did not catch their attention, or they did not understand what the purpose was of this. Also, they would like to have a more fine-grained sorting in the characteristics. The participants would like to filter in collections but also in the media objects.From these findings, recommendations have been made to improve the user interface for exploration. The current interface was not effective or efficient for exploration due to the shortcomings of DIVE+. The ‘events’-enriched collection was experienced as an overload of results. Due to multiple reasons, no relevancy could be seen with the objects. Also, the purpose of the ‘narratives’ support was not optimal yet because it did not support the researcher by forming a narrative. With the given recommendations, the exploration through the ‘events’-enriched collections of DIVE+ will be improved for the digital humanities scholars supported with the ‘narratives’ tool.
After the implementation of this user evaluation’s recommendations, more evaluations should be conducted, to test the performance of DIVE+. These future user evaluations can be focused on an improved feature of DIVE+ or the overall user experience.
8.
ACKNOWLEDGMENTS
This thesis has come together with much effort, help and not that much sleep. I would like to thank my thesis buddy, Justin Verhulst, for his equal commitment and much appreciated help in this process and my supervisor Lora Aroya, Sabrina Sauer and Berber Hagedoorn for their always critical, but valuable feedback.
9.
REFERENCES
Bizer, Christian, Tom Heath, and Tim Berners-Lee. "Linked data-the story so far." Semantic services, interoperability and web applications: emerging concepts (2009): 205-227.
Bron, Marc, et al. “A subjunctive exploratory search interface to support media studies researchers.” Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval. ACM, 2012.
Bron, Marc, Jasmijn Van Gorp, and Maarten Rijke. "Media studies research in the data-driven age: How research questions evolve." Journal of the Association for Information Science and Technology 67.7 (2016): 1535-1554.
Collijn, Annemarie,. “DIVE: an exploratory search system for humanities scholars”. (2016)
De Boer, Victor, et al. “DIVE into the event-based browsing of linked historical media.” Web Semantics: Science, Services and Agents on the World Wide Web35 (2015): 152-158
Dumas, Joseph S., and Janice Redish. A practical guide to usability testing. Intellect books, 1999.
van Erp, Marieke, et al. "Automatic heritage metadata enrichment with historic events." (2011).
Fox, Edward A., et al. "Exploring the computing literature with visualization and stepping stones & pathways." Communications of the ACM 49.4 (2006): 52-58.
Glowacka, Dorota, et al. "Directing exploratory search: Reinforcement learning from user interactions with keywords." Proceedings of the 2013 international conference on Intelligent user interfaces. ACM, 2013.
Golovchinsky, Gene, Abdigani Diriye, and Tony Dunnigan. "The future is in the past: designing for exploratory search." Proceedings of the 4th Information Interaction in Context Symposium. ACM, 2012.
Hagedoorn, Berber and Sabrina Sauer. “Getting the Bigger Picture: Evaluating Narrative Creation and Media Exploratory Search for AV Research” (in preparation, 2017).
Hutchinson, Hilary Browne, Benjamin B. Bederson, and Allison Druin. "The evolution of the international children's digital library searching and browsing interface." Proceedings of the 2006 conference on Interaction design and children. ACM, 2006. Krahmer, Emiel, and Nicole Ummelen. "Thinking about thinking aloud: A comparison of two verbal protocols for usability testing." IEEE Transactions on Professional Communication 47.2 (2004): 105-117.
Kruijt, M. (2016). Supporting exploratory search with features, visualizations, and interface design: a theoretical framework. (Master’s thesis, Universiteit van Amsterdam, Amsterdam). Kules, Bill, et al. "What do exploratory searchers look at in a faceted search interface?." Proceedings of the 9th ACM/IEEE-CS joint conference on Digital libraries. ACM, 2009.
Manzari, Laura, and Jeremiah Trinidad-Christensen. "User-centered design of a web site for library and information science students: Heuristic evaluation and usability testing." Information technology and libraries 25.3 (2006): 163.
Marchionini, Gary. "Exploratory search: from finding to understanding." Communications of the ACM 49.4 (2006): 41-46. Marchionini, Gary. Information seeking in electronic environments. No. 9. Cambridge university press, 1997.
Marshall, Catherine, and Gretchen B. Rossman. Designing qualitative research. Sage publications, 2014.
Melgar Estrada, L. (2015). From Social Tagging to Polyrepresentation: A Study of Expert Annotating Behavior of Moving Images.
Nielsen, Jakob, and Rolf Molich. "Heuristic evaluation of user interfaces." Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 1990.
Odijk, Daan, et al. "Time-aware exploratory search: Exploring word meaning through time." SIGIR 2012 Workshop on Time-aware Information Access, Portland, OR, USA. Vol. 8. 2012.
14
Papoutsaki, A., Laskey, J., & Huang, J. (2017). Searchgazeer: Webcam eye tracking for remote studies of web search. In Proceedings of the 2017 conference on conference human information interaction and retrieval (pp. 17-26). New York, NY,USA: ACM. Retrieved from
http://doi.acm.org/10.1145/3020165.3020170
doi:10.1145/3020165.3020170
Pia Borlund , (2016)," A study of the use of simulated work task situations in interactive information retrieval evaluations A meta-evaluation ", Journal of Documentation, Vol. 72 Iss 3 pp. 394 - 413
Rubin, Jeffrey, and Dana Chisnell. Handbook of usability testing: how to plan, design and conduct effective tests. John Wiley & Sons, 2008.
Stone, Sue. "Humanities scholars: information needs and uses." Journal of documentation 38.4 (1982): 292-313.
Van Den Akker, Chiel, et al. "Digital hermeneutics: Agora and the online understanding of cultural heritage." Proceedings of the 3rd International Web Science Conference. ACM, 2011.
Verhulst, J. (2017). Optimizing Explorative Search for the Needs of Media Professionals: The DIVE+ Use Case. (Master’s thesis, Universiteit van Amsterdam, Amsterdam).
White, Ryen W., Bill Kules, and Steven M. Drucker. "Supporting exploratory search, introduction, special issue, communications of the ACM." Communications of the ACM 49.4 (2006): 36-39. White, Ryen W., and Resa A. Roth. "Exploratory search: Beyond
the query-response paradigm." Synthesis Lectures on Information Concepts, Retrieval, and Services 1.1 (2009): 1-98.
15
Appendices
Appendix I: Screenshots DIVE+
Appendix II: Protocol workshop Digital Humanities Scholars
Online available via this online link.
Appendix III: Protocol workshop Media professionals
Online available via this online link.
Appendix IV: Simulated Work Task Description Digital Humanities Scholars
Online available via this online link.
Appendix V: Simulated Work Task Description Media Professionals
Online available via this online link.
Appendix VI: Questionnaire Digital Humanities Scholars
Online available via this online link.
Appendix VII: Results
Online available via this online link.
Appendix VIII: Main Insights
