2
Exploratory Political Search
Bachelors thesis Information Studies
Student: Laura Mul
Studentnumber: 10003287
Supervisor: Maarten Marx
3
1.
Abstract
The Political Mashup search engine (www.search.politicalmashup.nl) can be used to search through political debates. It is possible to search through data from the UK, Canada and The Netherlands. The Dutch database consists of 105.734 topics, 198.909 scenes and 2.616.865 speeches, which are all easily accessible through their xml metatags. This paper focuses on designing and testing the search interface of the Political Mashup search engine.
The search interface is intended to support exploratory search, meaning that the Political Mashup search engine enables the user to compare and aggregate (Marchionini, 2006). By looking at ways to support exploratory search and creating use cases, a design is proposed to the programmer.
The final interface design is tested and compared to various existing websites. The main results indicated that the Political Mashup search engine is the most optimal search engine to explore political debates when it comes to effectiveness. Although, this is the case, the users still complain about various malfunctions. Therefore it is suggested to test the Political Mashup search engine further when it is completely finished.
4
Inhoud
1. Introduction ... 6
2. Research method ... 7
3. Theoretical Framework ... 8
3.1 Information seeking behaviour ... 8
3.1.1 Defining information seeking behaviour ... 8
3.1.2. Goals of information seeking ... 10
3.2 Supporting exploratory search ... 12
3.3 Evaluation of search user interfaces ... 14
3.4 Implications for our study ... 15
4. The data ... 15
5. The baseline search engine... 17
6. Improving the search engine ... 19
6.1 Lookup searches ... 20 6.1.2 Question Answering ... 20 6.2 Learn searches ... 21 6.2.1 Comparison ... 22 6.2.2 Aggregation ... 22 6.3 Investigate ... 23 6.3.1 Transformation ... 23
7. Proposed interface design ... 23
8. Final interface design ... 26
9. Analysis of comparitive websites ... 27
9.1 Comparing other sites’ functionalities with the Political Mashup Search Engine. ... 27
9.1.1. Google Advanced ... 27
9.1.2 Staten Generaal Digitaal ... 28
9.1.3 Overheid.nl ... 29
10. User tests ... 29
10.1 User Feedback Sessions ... 29
10.2 Expert interview ... 30 10.3 Usability tests ... 30 11. Results ... 32 11.1 Usability tests ... 32 11.1.1 Effectiveness ... 32 11.1.2 Efficiency... 33
5
11.1.3 Satisfaction ... 33
11.2 Expert interview ... 36
11.3 Suggestions for improvement ... 37
12. Conclusion & Discussion ... 39
12.1 Hypotheses ... 39
12.2 Conclusion ... 40
12.3 Discussion ... 41
12.3.1 Unfinished website ... 41
12.3.2 Comparison of the two websites ... 41
12.3.3 One observer ... 41
12.3.4 Further research ... 41
12.3.5 Self reflection ... 41
13. Bibliography ... 43
Appendix 1 Tools to support exploratory search... 46
1. Sorting and filtering ... 46
1.2 Breadcrumbs ... 46
1.3 Clustering ... 46
1.4 Facets ... 47
2. Query support ... 47
3. Visualization of results ... 48
Appendix 2 Results of user feedback sessions ... 49
Appendix 3 Consent Form: Usability Test ... 51
Appendix 4 Queries carried out during the usability tests ... 52
Appendix 5 Questionnaire ... 53
6
1.
Introduction
Search is an essential part of people’s online lives; search engines on the internet are used to help with a great amount of needs and desires (Hearst, 2009). The Political Mashup website (http://search.politicalmashup.nl/) is one of those search engines that can be found on the web.
The Political Mashup project aims at searching through big amounts of digital political debate data, from 1946 onwards (Lacroix, 2008). One can, for example, search for the Dutch
politician Balkenende and get all the debates in which he appears or in which others talk about him. It is also possible to carry out more complex queries, like how often did someone interrupt Balkenende during his speech?
At the moment, there are search engines that make it possible to search through political data, like Staten Generaal-Digitaal (http://www.statengeneraaldigitaal.nl/), but not in such a way that complex queries can be carried out. The existing Dutch sites that can be used to search through political debates are made to find a specific debate or document, while the Political Mashup search engine allows for more exploratory search. While the existing websites focus on fact retrieval and known item search, the Political Mashup search engine enables the user to compare and aggregate (Marchionini, 2006).
The data that can be consulted has been organized in such a way that it is possible to easily search through it. All debates have an xml file that contains metatags which makes it easy to search through these files quickly.
All the data is available and neatly organized, but how will people consult the data? This is where the search user interface comes in. The database is available, but if users don’t have the right tools at their disposal to get to the data, there is no possibility to get to the
information they are looking for. Therefore this paper proposes a search user interface for the Political Mashup search engine and thereby aims to answer this main research question (MQ):
MQ: What is the optimal search interface for exploring political debates?
In order to answer this research question, several subquestions are introduced. The first three are based on Marchionini’s definitions of search (2006).
Q1. How can lookup-searches be supported?
Q2. How can learn-searches be supported when aiming for knowledge acquisition,
comparison and aggregation?
7
These three subquestions will be answered by looking at various use cases that aim to answer several information needs. The use cases will be supported by literature and by early user tests.
The fourth subquestion looks at how the user experiences the Political Mashup search engine after the final design has been proposed.
Q4. What is the difference in functionality between the Political Mashup Search Engine and the existing websites Staten Generaal Digitaal, Overheid.nl and Google Advanced Search?
Q5. How does the user experience the effectiveness, efficiency and satisfaction of the Political Mashup search engine and how does this differ from Overheid.nl? The final design of the search engine will be tested to answer subquestion 5.
After having answered all 5 questions it will become clear what is the most optimal search interface is to explore political debates.
To give an overview of the thesis, it will first be found out whether the Political Mashup search engine needs improvement by looking at existing literature, use cases and user feedback sessions. Secondly, the design is proposed to the programmer who will decide whether to implement it. Finally, the definite search engine is tested.
2.
Research method
At the moment, there is a Political Mashup baseline search engine. With the help of literature, use cases and user feedback sessions, it will be found out whether the baseline search engine can be improved. The final design of the Political Mashup search engine will then be tested again to find out if it is the most optimal search engine for exploring political debates.
This research will start by looking at existing literature. The literature will give insights into what tools can be used to design a search user interface.
After this, the data structure of the database will be explored. The structure of the XML behind the documents in the database gives insight into what queries can be executed. The website will be tested constantly to get an agile evaluation. This is a methodology that embraces high rates of change (Cockburn, 2003). Agile software development stresses the importance of iteration, early and repeated user feedback as to handle emergent
requirements (Rogers, 2011). In the beginning, the existing website will be tested by 5 participants which will reveal about 85% of the usability problems (Nielsen, 2000). This is what Nielsen calls formative evaluations which are conducted during design as a part of an iterative process (Nielsen, 1994). The outcome of these tests will feed the use cases that aim to answer Q1, Q2, and Q3.
8
The use cases exist of several information needs that require varying functionalities of the website. By looking at what the user is going through step by step, it will become clear what should be added to the website. In order to answer Q1, Q2, and Q3, each use case will focus on different kinds of information needs.
After these use cases have been set up, it will become clear whether the search engine lacks functionalities. This way, requirements are set up to design the final interface. This design is proposed to the programmer who decides whether to implement it or not. Next, the functionalities of Staaten-Generaal Digitaal, Overheid.nl and Google Advanced search are compared to the functionalities of the Political Mashup search engine.
Final summative evaluations are carried out, which are tests of completed products (Nielsen, 1993). A comparative study will be done by testing the Political Mashup search engine and an existing comparable website, which is Overheid.nl. Through these tests it will be found out if the Political Mashup search engine is the most optimal tool for exploratory search through political debates.
During these final usability tests the technique ‘direct observation in a controlled
environment’ will be used (Rogers, 2011). The participants will get various tasks to perform to see if the website works properly. Effectiveness will be measured by looking at whether the participant finds what he is asked to find. Efficiency will be measured by looking at how quickly the participant finds what he is meant to find. Finally, the satisfaction is measured through a questionnaire that is given to the participant afterwards and through the
participant’s comments. During the test the participant is asked to think aloud, so the observer can record what he or she is thinking during the test (Rogers, 2011).
The testing will be done by participants who are no expert on political debates. Therefore there will be one expert interview to find out what she uses the website for exactly and how her needs can be supported.
These final steps will answer the subquestions Q4 and Q5 which eventually answer the main research question.
3. Theoretical Framework
3.1 Information seeking behaviour
From 1981 and onwards, information seeking behavior has been defined by various people. Several of the most influential ones will be discussed. After this, the various goals of
information seeking will be considered.
3.1.1 Defining information seeking behaviour
9
Wilson is one of the pioneers when it comes to defining information seeking behaviour. According to him, the search starts with a need and is followed by an information-seeking behavior (Wilson, 1981). This can be seen in Figure 1 (Wilson, 1981). When the need is found, also known as a success, a new need is initiated and when there is failure, the information seek simply ends. After Wilson, others followed to define information behavior. Kuhlthau claims there to be 5 stages in the information seeking process which are initiation, selection, exploration, formulation and collection (Knight, 2008). Kuhlthau claims that, due to the cognitive and affective aspects of the process of information seeking, there is a gap between the user’s natural process of information use and the information system (Knight, 2008). Marchionini takes this idea further, but takes a more contextual approach by focusing mostly on how the searcher and the increasingly complex electronic information environment relate to each other (Knight, 2008).
Figure 1. Information seeking behavior defined by Wilson
The main assumption that Marchionini is making, is that information seeking is a relatively linear process as can be seen in figure 2 (Knight, 2008). It does state that a query can be adjusted or that the searcher formulates a whole new query, but it is Bates that steps away from the linear search model (1989). Bates comes up with the berry picking model, which states that in real-life searches people start with a broad topic, or just one relevant reference and move through a variety of resources (Bates, 1989). The searcher obtains new information or is send in a new direction which causes him to shift the query in part or in a whole (Bates, 1989).
10
Figure 2. Marchionini’s information seeking in electronic environments
Figure 3. Bates’ Berry Picking Model
In conclusion: Information seeking behaviour is not a linear process. It is an iterative process in which the user starts with a certain query, but changes it during the search. Now that it is clear how information search has been defined, various goals of
information search can be discussed.
3.1.2. Goals of information seeking
Broder comes up with a taxonomy of web searches, which are three classes of intent. There is the navigational, the informational and the transactional intent (Broder, 2002). The first one is concerned with reaching a particular site. The second one has the goal of reaching some information that is assumed to be present on one or more web pages. The latter one has the intent to perform some web-mediated activity (Broder, 2002). These definitions are, to a certain extent ambiguous. Rose presents more detailed information on this taxonomy (2004). By further defining the three classes of intent, it becomes clear what each class means exactly.
11
Class of web query Purpose
Navigational My goal is to go to specific known website
that I already have in mind.
Informational My goal is to learn something by reading
or viewing web pages ● Directed
○ Open ○ Closed ● Undirected
Resource My goal is to obtain a resource (not
information) available on web pages Table 1. Taxonomy of web searches defined by Rose (2004)
The Political Mashup search engine focuses on Informational queries. According to Rose, Informational queries can be directed or undirected. Directed questions can be open or closed. If the user wants to learn something particular about a topic, it is a direct query. If this direct query is closed, the query has a single, unambiguous answer. A directed open question is open-ended or has an unconstrained depth. A question is called undirected when the user wants to learn anything or everything about the topic. In order to explain informational queries further, it is useful to look at Marchionini’s categorization of web searches. He talks about lookup, learn and investigate as can be seen in Figure 4 (Marchionini, 2006). Lookup focuses on activities like fact retrieval, known item search and verifications. Lookups are related to questions like who, what, where.
Next to Lookup there are Learn and Investigate searches that belong to exploratory search. Exploratory search will be explained more in depth as this is an important part of the Political Mashup search engine.
12
3.1.2.1. Exploratory search
Exploratory search covers learning and investigating, as can be seen in Figure 4 (Marchionini, 2006). The three different ways of searching are depicted in clouds, because they are all related to each other and are almost never used independently (Marchionini, 2006). It is not a linear process: behaviours within ‘investigate’ might precede from ‘lookup’ or ‘learn’ (Nolan, 2008) .
Learn searches involve comparing, obtaining knowledge and interpreting. Investigation involves multiple iterations that take place over possibly long periods of time and the results are analyzed before being integrated into personal and professional knowledge bases (Marchionini, 2006). Learning and investigative research require, opposed to lookup searches, strong human participation.
To obtain the appropriate information, browsing services can help to support this human participation (Marchionini, 2006) .
3.2 Supporting exploratory search
The browsing services to support exploratory search will be discussed after the 8 design requirements for search user interfaces are examined.
Shneiderman et al. identify 8 design requirements for search user interfaces: go for
consistency; offer informative feedback; provide simple error handling; allow easy reversal of actions; support user control; minimize short term memory load; design for closure; and provide shortcuts for experts (1997).
To give a practical example, most of these 8 points can be found in the Google Search Engine. Table 2 sums up the characteristics of Google’s Search Engine that realize the 8 requirements and Figure 5 depicts this.
Requirement Realized in Google’s Search Engine
Go for consistency Results are shown in the same layout Offer informative feedback Query suggestions are made.
Related topics are suggested
Provide simple error handling When a mistake is made, Google suggests query
Allow easy reversal of actions
The user can easily change his or her query
Support user control The user has the control to choose his or her query
13
Minimize short term memory load The interface is really easy to use as the interface is very intuitive
Design for closure Eventually the aim of Google is to help the user find what he or she is looking for. Provide shortcuts for experts -
Table 2. Scheidermann’s 8 requirements explained through Google’s Search Engine
Figure 5. Google’s Search Egine
To take it to a more general level, these characteristics of the Google Search Engine can be divided into different tools. These tools help to realize the requirements, which can be seen in Table 3. For more information on these tools, Appendix 1 can be consulted.
Requirement Tool
Go for consistency Visualization of results
Offer informative feedback Query suggestions, breadcrumbs Provide simple error handling Query reformulation
Allow easy reversal of actions
Query reformulation
Support user control Sorting and filtering, clustering Minimize short term memory load Breadcrumbs
14
Design for closure -
Provide shortcuts for experts Facets Table 3. Tools realizing Schneidermann’s requirements
3.3 Evaluation of search user interfaces
In order to find out if a search engine works as it should and the users find what they are looking for, it has to be tested properly. Search user interfaces are generally evaluated in three main aspects of usability: effectiveness, efficiency and satisfaction (ISO, 1998).
1. Effectiveness Accuracy and completeness with which
users achieve specified goals
2. Efficiency Resources expended in relation to the
accuracy and completeness with which users achieve goals
3. Satisfaction Freedom from discomfort, and positive
attitudes towards the use of the product Table 4 : Aspects of usability evaluation
Table 4 shows the 3 aspects with a short description. Effectiveness is about quality of solution and error rates, thus the outcome of the user’s interaction with the system. Efficiency is about the resources used in relation to the accuracy and completeness with which the users achieve goals (ISO 1998). The last point, satisfaction, has to do with the feelings of the user towards the system (Frøkjær 2000).
All three aspects must be tested, because they have a low correlation in usability studies (Hornbaek 2007, Frøkjær 2000). This can be done through informal usability which focuses on the involvement of representative users (Hearst, 2009). It is important to involve the target users of the interface and therefore this process is often called user-centered design (Abras, 2004).
In the first rounds of evaluation, a few participants are needed to identify major problems (Nielsen & Landauer 1993). These quick informal usability tests are what Nielsen calls discount usability testing contrary to full formal laboratory studies. (1994) In the early stages of design, low fidelity prototypes are used for user testing, where paper prototypes are used (Hearst, 2009). In the study of the Political Mashup Search Engine we will immediately use high fidelity prototyping as the website is already built. High fidelity prototyping means that there is already an amount of interactivity built in (Hearst, 2009). The question is how many people are needed for this informal testing. Nielsen (2000) claims that only 5 participants are needed to find 85% of the usability problems. Others disagree to this (Spool & Schroeder, 2001).
15
When testing the high fidelity interface with participants, it is most effective if there are two evaluators present (Hearst, 2009). One evaluator acts as host and explains what the participant should do. The second sits farther away, doesn’t say anything and records the participant’s comments, impressions and suggestions. It is very useful if the participant thinks aloud (Nielsen, 1993).
Another form of informal usability testing is field study where one visits a user in their natural environment, which can be at home or at work (Hearst, 2009). Here, the assumption is that people work more realistically when they are in their own environment rather than working in an unfamiliar space (Hearst, 2009).
Next to informal studies, there is formal studies where a controlled experiment is done. This is mostly used to find out which specific feature in a website makes a difference to how the user uses the website. Due to time constraints, this will not be used in this study.
In order to test the user interface, scenarios will be used. The user will be told what he has to find and this will be carried out. The effectiveness will be measured by looking at whether the user has found what he is looking for. The efficiency will be measured by looking at what how much time is used to complete the goal. The satisfaction will be measured through a questionnaire and people’s comments that they share when thinking aloud.
3.4 Implications for our study
Based on the definitions of exploratory research from section 3.1, use cases will be made to depict possible information needs of a person using the Political Mashup search engine. Based on these use cases, improvements might be suggested which will be partly taken from section 3.2. Finally, the search user interface will be tested using the information from section 3.3.
4.
The data
The database behind the Political Mashup search engine exists of an enormous amount of debates. It is possible to search through data from the UK, Canada and The Netherlands. The Dutch database consists of 105.734 topics, 198.909 scenes and 2.616.865 speeches. Figure 7 shows what a part of a debate looks like from the Dutch database. This is one of the debates that show up when the search term is ‘Balkenende zorg’. On the left of the page information on the debate can be found. Also the speakers are listed.
It is useful to look at the xml code behind the database, so we know what can be used in the search engine. There are various tags in the xml code, of which one is ‘scene’. Each scene tag has the following information:
<scene xmlns="http://www.politicalmashup.nl" xmlns:pm="http://www.politicalmashup.nl"
pm:type="speaker" pm:speaker="Balkenende" pm:party="CDA" pm:function="De heer" pm:role="mp" pm:party-ref="nl.p.cda" pm:member-ref="nl.m.02207" pm:id="nl.proc.ob.d.h-tk-20012002-1779-1783.1.4">
16
First of all there is a speaker, a party, a function and a role in the scene tag. The speaker that is tagged here opens the scene which means that it is the first speaker and others interrupt him or her.
Within the scene tag, there can be several speech tags. These speech tags also have a speaker, a party, a function and a role as can be seen below.
<speech pm:speaker="Kok" pm:function="Minister" pm:role="government" pm:member-ref="nl.m.02804" pm:id="nl.proc.ob.d.h-tk-20012002-1779-1783.1.5.3">...</speech>
The tag that surrounds the whole debate is the topic tag as can be seen below.
<topic xmlns="http://www.politicalmashup.nl" xmlns:pm="http://www.politicalmashup.nl" pm:title="Verpleeghuiszorg" pm:source-start-page="3285"
pm:source-end-page="3324"pm:id="nl.proc.ob.d.h-tk-20042005-3285-3324.1">...</topic>
More information on the data can be found here: http://schema.politicalmashup.nl/.
17
5.
The baseline search engine
At the moment the baseline search engine looks like this:
Figure 8. The baseline search engine
There is a search bar on top of the page in which the user can type. If one types, for
example, the word ‘zorg’, this will appear in the search bar: ‘text: zorg’. It will inform the user of what he or she has typed. If a user wants to add a party to the query, he can do this by typing in the two bars designed for this. There will be query suggestions when one starts typing. There is the possibility to search through 3 collections, which is The Netherlands, Canada and the United Kingdom. There is a granularity option in which one can select Speech, Scene, Topic or Stage Direction. The features have been assigned names as can be seen in Table 5. This will make it easy to refer to them when needed.
Feature Names
Main search bar text_search
Collections selector collections_selector
Granularity granularty_selector
Party search bar party_search
Speaker search bar speaker_search
Table 5. The Political Mashup search engine’s features and their assigned names
The results page can be seen in figure 10. A word cloud is shown in which the search results are depicted. Below this word cloud, the results can be found with facets that can help to refine the results. It is possible to select a certain year range and only show the results from this period of time. The results can be sorted by date (descending and ascending),
18
relevance, number of speeches (descending and ascending) and number of paragraphs (descending and ascending). For each search result, the speaker, date and amount of paragraphs is shown in respectively green, blue and brown. When one searches on scene (the search shown in Figure 10 is on speech, not on scene), the number of speeches and stage directions are shown in brown as can be seen in Figure 9. If there is a party involved, the party is depicted in red.
Figure 9. Zoom in on one specific scene
19
6.
Improving the search engine
This section aims to answer sub questions Q1, Q2 and Q3:
Q1 How can lookup searches be supported?
Q2 How can learn searches be supported when
aiming for knowledge acquisition, comparison and aggregation?
Q3 How can investigate searches be supported
by looking at transformation? Table 6. Sub questions to be answered in this section
In order to answer subquestion 1, there will be a use case that deals with a look up search which can be found in section 6.1. Section 6.2 will show two use cases to answer
subquestion 2 which which looks at how learn-searches can be supported. Section 6.3 looks at one transformation query as to answer subquestion 3.
In order to possibly improve the design of the interface, use cases will be used. For the use cases, the definition of the designers of UML will be used. Use cases are the specification of sequences of actions, including variant sequences and error sequences, that a system, subsystem, or class can perform by interacting with outside actors (Booch, 1999). If the user, the actor, has a specific query, it will be looked at what can go wrong and what is needed to improve this. In order to see the user interface from different perspectives, several sorts of searches will be carried out.
As the Political Mashup search engine focuses on exploratory research, the queries will mostly focus on learn and investigate as defined by Marchionini (2006). Although this is the case, there will also be an example of a look-up search to show that simple queries can be executed. The queries that are used as use cases are aiming at showing the diverse possibilities of the Political Mashup search engine.
From the beginning on, the users will be involved. As stated by Nielsen (2000), only 5 participants are needed for this informal testing to find 85% of the usability problems. This amount will be used to find the major problems of the interface and to make sure that the use cases are realistic.
20
6.1 Lookup searches
Lookup-searches aim at answering questions that start with who, when and where (Marchionini, 2006). The parts of the website that this use case is focusing on are the text_search, speaker_search and party_search.
6.1.2 Question Answering
6.1.2.1. Query 1A question that fits in the ‘question answering’ category is:
When did Balkenende talk about Zorg for the last time? 1. Collection Netherlands is chosen
2. Balkenende is added as a speaker in speaker_search
3. Balkenende shows up in speaker_search
4. Zorg is added in text_search
5. Zorg shows up in text_search 6. Enter is pressed
7. Search results are shown
8. User can sort the search results on date with the newest one on top by using a sort button
In the baseline search engine, the Collections feature is placed below the text_search. This means that people will first enter a query after which they select the collection. It is more logical that people first select the collection and then put in a query.
The same goes for the speaker_search and the party_search; people do not look at these bars. They start typing in the search bar, but do not know they can add specific speakers or parties. Figure 10 shows a logical search bar where the ‘search’, or in this case Go! Button is placed at the end of the input for the query (Krug, 2000). In this example, people first select a category and then put in their query.
Figure 10. Example of a search bar (Krug, 2000)
The user tests support these observations. When the participants were asked ‘look up what Wilders has said about the Islam’, they all started typing into the text_search, but are not aware of the party_search and speaker_search. One of the participants typed in ‘Wilders Islam’ in the text_search. When he found out that there was a specific bar to put in the speaker, he typed in Wilders there and deleted it in the text_search. The text_search is
21
situated at the so called sweet spot. This is the spot that captures most attention of the users and thus the place where they start their search (Hearst, 2009).
Another explanation for why users don’t see the party_search and speaker_search is that people tend to focus one thing they are searching for, so for example they look for the text_search (Krug, 2000). Intuitively, they want to type the search term in a search box and do not look for extra supporting bars.
The granularity_selector causes confusion. People do not get the meaning of this. One of the participants started searching for the term on Google, but still did not understand it. The terms that can be chosen, which are speech, scene, topic and stage direction are also confusing. During the user studies, people did not understand the meaning of it and
randomly chose one. Showing unclear options to users is useless (Krug, 2000). The payoff for adding options to a search box is seldom worth the cost of making the user figure out what the options are and whether he has to use it (Krug, 2000).
6.1.2.2 Query 2
In order to show how it should be possible to search for a specific known debate, the following question is used:
Give me the debate in which Balkenende talks about ‘zorg’ of october 21st 1988. 1. Collection netherlands is chosen with collections_selector
2. Balkenende is chosen as a speaker with speaker_search 3. Balkenende shows up in speaker_search
4. Zorg is typed in in text_search 5. Zorg shows up in text_search 6. One specific day is chosen 7. Enter is pressed
At the moment, this is not possible. The user can select a year range, but he or she cannot select specific dates. Therefore a calendar is needed that enables the user to select specific periods of time, but also one specific day.
6.2 Learn searches
Learn searches involve multiple iterations and return a set of objects that require cognitive processing and interpretation (Marchionini, 2000). This is often done through various media (graphs or maps, text and videos) and the seeker often has to scan, compare and make qualitative judgements. Query’s that enable knowledge acquisition, comparison and aggregation will show how learn searches can be executed.
22
6.2.1 Comparison
The Political Mashup search engine allows for comparison of topics, parties and speakers. An example of a query is:
How often do CDA and PvdA talk about zorg over time?
1. Collection The Netherlands is chosen by using the collections_selector 2. CDA is entered as a speaker in a party_search
3. CDA shows up as suggestion
4. CDA is selected and shows up in party_searh 5. PvdA is entered as a speaker in party_search 6. PvdA shows up as a suggestion
7. PvdA is selected and shows up in party_search 8. PvdA shows up in general search bar
9. In the text_search ‘zorg’ is entered 10. Enter is pressed again
11. Results appear 12. User clicks on graphs 13. Graph shows up
14. A line for each party shows how often they talk about the subject
6.2.2
Aggregation
An aggregation can be: how often do Pechtold, Balkenende and Wilders appear together in a debate?
1. Collection Netherlands is chosen with the collections_selector 2. Pechtold is entered in the speaker_search
3. Drs. A. (Alexander) Pechtold shows up as a suggestion 4. Pechtold is selected and shows up in speaker_search 5. Wilders is entered in speaker_search
6. G. (Geert) Wilders shows up as a suggestion
7. Wilders is selected and shows up in speaker_search 8. Balkenende is entered in speaker_search
9. Prof. Mr. Drs. J.P. (Jan Peter) Balkenende shows up as a suggestion 10. Balkende is selected and shows up in speaker_search
11. Enter is pressed
12. All debates with the three politicians as a speaker appear
At the moment, this is not possible. When these three speakers are added without adding anything else in the general search bar, there are no results. It seems like the search bar needs some input to get the results.
Another way to search for it, is to search on a certain topic in the debate. One can search on a debate e.g. zorg (care) and then later select the speakers.
23
6.3 Investigate
Investigate-searches are the most challenging of the three (Nolan, 2008). How do we locate information beyond our area of expertise that might be important to us? Carrying out these kind of queries can be very time consuming. For this research paper, an example of transformation will be used.
6.3.1 Transformation
For this question, the aim is to test for transformation. This means finding out how something changes over time. The query to be answered is:
Over the years, does CDA discuss huurtoeslag (housing benefit) more often or less often?
1. CDA is entered in party_search
2. CDA shows up as a suggestion and is selected 3. CDA shows up in party_search
4. Huurtoeslag is entered in text_search 5. Huurtoeslag shows up in text_search 6. Enter is pressed
7. The user clicks on graphs
8. Clicks on hits per party over time
9. A graph shows up which shows how often a party discusses huurtoeslag over time 10. Hovers mouse over CDA and sees how it has changed over time
7. Proposed interface design
Based on the user tests and use cases, a final design is proposed.
The home page can be seen in Figure 11. The order of the bars has been changed. People will first type in the speaker in the speaker_search and then a party in the party_search. After this, the user types in a search term in the text_search.
Instead of the terms speaker and party, it now says ‘add speaker’. This is to make it clearer that the user should add a speaker.
In the baseline design, there was the granularity_selector. People did not understand what this meant and also did not understand the options. Therefore, this button is moved to advanced search. The default setting for the granularity is now set to topic.
24
Figure 11. Proposed homepage
It should be possible to only search on speakers without entering a text to search on. The database is to be used to search through Dutch debates and therefore the standard collection to search through is the Dutch collection. Therefore the website should be in Dutch as it will mostly be used by Dutch people. Although the site should be in Dutch, there should also be an option to change to English. In Figure 11, the website is in English as this
research paper is written in English.
There is a calendar on the front page. Also, there is the option to select 1 day. When this is selected, it is only possible to select one date instead of two.
There is a button which says ‘advanced search’ to go to advanced search. This page gives the user the possibility to search for specific information needs. Once the user has used the advanced page, he or she knows what the possibilities are and can use it on the home page as well. For example OR and AND can be used in the main search bar.
25
The ‘speaker appearing as’ requires explanation. When one searches on ‘scene’, the system can see who the main speaker is. When one selects main speaker, it will only be possible to search on scene.
26
Figure 13. Proposed results page
The results page almost stayed the same as the baseline design. The graphs are very useful. It is already possible to compare how often a party talks about something. It should also be possible to see how often a speaker talks about a subject over time. Therefore it would be useful to add the option ‘hits over time per speaker’ to the graph.
8. Final interface design
The interface discussed in section 7 was proposed to the programmer and partially integrated.
Integrated Rejected
Order of search bars Advanced search on a seperate page View speaker over time in graph Granularity deleted
27
and speaker entering search term
Default setting of granularity button is set to topic
Site must be in Dutch
Calendar is added Detailed advanced search with every option proposed
Advanced search is added on the home page
Table 7. Features integrated or rejected in the final interface design
9.
Analysis of comparative websites
9.1 Comparing other sites’ functionalities with the Political Mashup
Search Engine.
In order to find out whether the Political Mashup Search Engine has all the functionalities that are expected to be there, the website will be compared to Google Advanced Search and Staten Generaal Digitaal.
9.1.1. Google Advanced
The Google Advanced Search Engine allows for very specific and detailed searches. In order to find out if the relevant functions are also implemented in the Political Mashup Search Engine, the two websites will be compared.
Google advanced search Political Mashup Search Engine
Find pages with Possible?
All of these words (type rat terriër) Yes
This exact word of group of words (type “rat terrier”)
Yes
One or more of these words (type rat OR terrier)
Yes
None of these words No
Table 8. Comparing Google Advanced search functionalities to the Political Mashup Search Engine functionalities
28
9.1.2 Staten Generaal Digitaal
Staten Generaal Digitaal can be used to search through parliamentary documents from 1814 till 1995. Several functionalities overlap when comparing the Staten Generaal Digitaal and the Political Mashup Search Engine, but in some ways the latter beats the former. Below a comparison is made between the two websites.
Figure 14. Staten Generaal search engine
Political Mashup Search Engine Staten Generaal Digitaal Search for speakers and have query
suggestions
Yes
Speaker facets No
Party facets No
Period selection through calender No Compare speakers and parties through a
graph
29
Parlementary documents from 1946 till 2009
No
Table 9. Comparing the Political Mashup search engine to the Staten Generaal Digitaal search engine
In addition, there are certain features that the Staten Generaal has, but the Political Mashup search engine does not have. It is not possible to search on document types like
‘kamerstukken’ and ‘kamervragen’.
9.1.3 Overheid.nl
Overheid.nl continues where Staten Generaal Digitaal stops. Staten Generaal Digitaal enables users to search through parliamentary documents till 1995. Overheid.nl makes it possible to search through parliamentary documents from 1995 till now.
Political Mashup Search engine Overheid.nl Search for speakers and have query
suggestions
No, not possible to search on speaker
Speaker facets No
Party facets No
Period selection through calender Yes Compare speakers and parties through a
graph
No
Parliamentary documents from 1814 till 2014
No, Overheid.nl has more recent parliamentary data
Table 10. Comparing the Political Mashup search engine to the Overheid search engine
10. User tests
10.1 User Feedback Sessions
As discussed before, the use cases were created with the help of 5 testers. They were all asked the same, which was ‘look up what Wilders says about the Islam’. Before this instruction, they were told what the search engine could be used for. One observer took notes on what the user was clicking on. The user was asked to think out loud as to share his or her thoughts. These thoughts were written down by the observer. After the test, several questions were asked like ‘Why didn’t you use the speaker search bar?’ This was a very informal test to find the bigger mistakes in the user interface. The results can be found in Appendix 2.
30
10.2 Expert interview
In order to find out what the experts are searching for on the website, one of them was interviewed. This was Sandra Boersma who works for ProDemos which is a house of democracy and rule of law. Also, she is part of the team creating the Dutch political block-calendar. She was asked to use the website for her own purposes and to report on what she thought of the website.
10.3 Usability tests
In order to evaluate the final design of the search interface, usability tests are carried out.
10.3.1 Problem statement
The aim of the experiment is to find out to which extent the Political Mashup Search Interface is considered effective, efficient and provide the user satisfaction. This was done by letting 6 participants carry out 6 use cases. Subquestion 4 (How does the user
experience the effectiveness, efficiency and satisfaction of the user search interface and how does this differ from Overheid.nl?) is to be answered through these tests.
The hypotheses for this experiment were:
1. The Political Mashup Search Engine will score well on effectiveness, efficiency and satisfaction.
2. The Political Mashup Search Engine will have a higher appreciation of effectiveness, efficiency and satisfaction compared to Overheid.nl. This means that it will be most suitable for both look-up search and exploratory search.
10.3.2 Setting
The 6 participants consisted of 4 females and 2 males. The mean age was 34. The median age was 22,5.
It is be a same-participant design. This means that each person tested both the Political Mashup Search Engine and Overheid.nl (Rogers, 2011). The same tasks were carried out by one participant on both websites.
The experiment took place in a room with a computer. During each test, there was one observer present.
10.3.3 The interfaces
For this study two different interfaces were used. One of them was Overheid.nl
(https://zoek.officielebekendmakingen.nl/zoeken/parlementaire_documenten) while the other
is The Political Mashup Search Engine (http://search.politicalmashup.nl/). They can both be used to search through political documents, but differ in functionalities.
31
10.3.4 Procedure
The participant was given a consent form which can be found in Appendix 3 (usability.gov). There is one observer present who told the participant that he or she would have to carry out queries on two different websites that can be used to search through parliamentary
documents. For each website, the different functionalities are explained to the participant, so it can be found out whether the participant uses the functionalities optimally. The participant was asked to think out loud. When the participant started to carry out the tasks, the steps recorder was turned on which takes screenshots of the screen each time the user clicks. The amount of time it took for the user to reach the goal was also recorded.
One half of the participants started with The Political Mashup Search Engine, while the other half started with Overheid.nl. The 6 queries are carried out by the participant on one of the websites. The queries can be found in Appendix 4. The first three queries are simple queries that are qualified as look-up queries, whilst the last three are exploratory queries. After these tasks have been carried out, the participant was given a questionnaire based on SUS (Brooke, 1996). After the questionnaire was filled in the user was asked to carry out the same queries on the different website. When this was done, the participant filled out the questionnaire again.
10.3.4 Analysis
The results of the tests will be categorized in three sections. First of all there is effectiveness. This section focuses on whether the information need has been found. Secondly, there is efficiency. This section focuses on how long it took to reach a goal. Finally, there is
satisfaction which focuses on the questionnaire results and summarizes people’s opinions that are based on what they said during the usability tests.
10.3.5 Instruments
A questionnaire was used to evaluate the satisfaction of the users. This questionnaire can be found in Appendix 5.
32
11. Results
11.1 Usability tests
11.1.1 Effectiveness
As stated before, the effectiveness is the accuracy and completeness with which users achieve specific goals. Therefore we look at whether the specific goals are achieved. The information searches to be carried out by the participants can be seen in table 11.
Information need
1 Question answering : When did Balkenende talk about zorg (care) for the last time?
2 Question answering: Find kamerstuk (parliamentary paper) with filenumber 25604, number 8 about zorg (care). This was published on 10-11-1997.
3 Question answering:
Find the debate named ‘regelingen van werkzaamheden’ that took place on 6-10-1998. 4 Comparison: How often do CDA and PvdA talk about zorg (care) over time?
5 Aggregation: How often do Pechtold, Balkenende and Wilders appear together in a debate talking about zorg (care)?
6 Transformation: Over the years, does CDA discuss huurtoeslag (rent allowance) more often or less often?
Table 11. Information searches carried out by the participants
Look-up searches Exploratory searches
Query 1 2 3 4 5 6 Goals achieved on Political Mashup( %) 100 40 60 100 100 100 Goals achieved on Overheid.nl (%) 0 100 100 0 0 40
Table 12. Percentage of goals achieved on the two websites
Query 1, 2 and 3 were look-up searches, which means that the participants were asked to find a specific document (Marchionini, 2006).
Query 4 involved comparison of parties and query 5 involved aggregation of speakers. Query 6 was about how often a topic is being discussed over time.
Three times there was a 0% success at the Overheid search engine. This happened to query 1, 4 and 5. Query 1 was ‘When did Balkenede talk about care for the last time?’. It was very difficult to successfully complete this query, because there was no possibility to search on speaker. Query 4 was ‘How often do CDA and PvdA talk about care over time?’ On the
33
Overheid.nl website, there is no possibility to visualize the search results, which made comparing difficult. The 5th query was ‘How often do Pechtold, Balkenende and Wilders appear together in a debate talking about care?’. Again, there was no easy way to search on speaker, so every document had to be checked manually to find out if they really speak together in a debate.
11.1.2 Efficiency
The efficiency is defined as the resources expended in relation to the accuracy and
completeness with which users achieve goal (ISO, 1998). Therefore it will be looked at what features of the site are used in relation to the completed goals. Before doing that, it will be looked at how long it took to reach a specific goal.
Website PoliticalMashup Overheid.nl
Query Mean Time Taken in minutes Mean Time Taken in minutes
1 2:18 x 2 4:33 1 3 4:35 2:30 4 3:20 x 5 2:37 x 6 1:19 x
Table 13. Mean time taken to reach the specified goal. x means that more than half of the people couldn’t reach the goal.
It is clear that the look-up searches (question 2, 3) take less time to complete on Overheid.nl than on the Political Mashup website.
Considering only the Political Mashup search engine, information search 6 takes the least time to complete.
11.1.3 Satisfaction
The satisfaction is measured in two ways. First of all, the participants were asked to think aloud, so they gave comments about how they experience the website. Secondly, they were given questionnaires after having used each website, so they could give their opinion on several statements on a scale from 1 to 5.
11.1.3.1 Questionnaire
The questionnaire measures the satisfaction of the user. All were measured through a Likert scale from 1 to 5. A higher score means a higher satisfaction. The statements in the
questionnaire can be found in table 14. After the results were gathered, some statements were changed in order to be able to compare the results. To give an example, during the usability tests, statement 2 was ‘I found the system unnecessarily complex’. In order to compare the results, statement 2, 4, 6, 8 and 10 were changed. Statement 2 was changed to ‘I did not find the system unnecessarily complex’. The results were also adjusted to this, which means a 2 out of 5 become a 4 out of 5.
34
Statement
1 I think I would like to use this system frequently 2 I did not find the system unnecessarily complex 3 I thought the system was easy to use
4 I do not think I would need the support of a technical person to be able to use this system
5 I found the various functions in this system were well integrated 6 I did not think there was too much inconsistency in this system
7 I would imagine that most people would learn to use this system very quickly 8 I did not find the system very cumbersome to use
9 I felt very confident using the system
10 I did not need to learn a lot of things before I could get going with this system Table 14. The statements that could be found in the questionnaire
Question 1 2 3 4 5 6 7 8 9 10
Political mashup
3 3,17 2,666 4 3,166 3 3,833 3 3,166 3,33
Overheid 2,8 3,5 3,833 4,66 2,666 3,66 3,666 3,5 3,166 4,17 Table 15. Mean user’s satisfaction. The higher the score, the higher the satisfaction
Both websites score highest on statement 4 which is: ‘I think I would not need the support of a technical person to be able to use the system’
Political Mashup scores lowest on statement 3. Overheid.nl scores lowest on statement 5.
The Overheid search engine has the highest score for 6 times. The Political Mashup search engine has the highest score 3 times. For one question they have the same score. This means that the Overheid search engine gets the highest satisfaction.
11.1.3.2 Think aloud
35
11.1.3.2.1 Political Mashup Search Engine
Feature Positive feedback Negative feedback
Sort by button - 2 Graph 1 7 Granularity - 2 Calender - 6 Party_search - 1 Text_searh - 5 Results page - 2 Speaker_search 2 Language - 2
Table 16. Participants’ feedback about the Political Mashup search engine
First of all, the graph has most negative feedback. These are some of the quotes about the graph function:
‘Why do I see all those parties and get all this crap?’
‘Why are all those parties there? Now I have to turn them off again’ ‘It’s weird that all those parties are there’
Figure 15. Graph showing hits over time per party
If people search on two specific parties, more parties show up in the graph. This means that people have to deselect all parties expect for the ones they want to see as can be seen in Figure 15.
Secondly, the calendar has 6 complaints of which these are three: ‘I put in 2 of the same dates, but there are no results’
‘Why does the first calendar show the year 1800 by default?’ ‘I really hate this’
36
Figure 16. The calendar on the Political Mashup search engine The text_search has 5 complaints, which were:
‘I cannot type in the number 8’
‘I made a small mistake and now I have to type it all over again’
‘I tried to search for something with parentheses around it, but it doesn’t work’
‘I wonder if the three people that I search for really appear together in a debate. Normally I would do AND, but now I don’t know if that works’
‘How do I change my query? Will it be deleted totally?’
The last complaint has to do with the fact that people do not search in a linear way, but change their query on the way (Bates, 1989).
11.1.3.2.2 Overheid.nl
Feature Positive feedback Negative feedback
Adding subject - 1
Search criteria 1 5
Results page - 4
Calender 2 -
Selection of documents 1 -
Table 17. Participants’ feedback about the Overheid search engine
The search criteria of the Overheid website have most negative feedback. Some quotes are: ‘I need a field to put in a speaker’ and ‘I will just make up something to do this’. The latter was said when a participant had to compare how often CDA and PvdA talked about care over time. There is no tool like a graph to compare this on the website, so he started to search for ‘CDA PvdA care’ during the period of 7-6-2005 and 7-6-2010. After this he searched for the same words, but changed the period to 7-6-2010 to 7-6-2015. For each period he compared the amount of results and this was his answer.
11.2 Expert interview
11.2.1 Sandra Boersma37
Sandra Boersma is part of the team that creates the political block-calendar and works for ProDemos. She often searches for political quotes and facts. Also, she has people who work for her and has to check the facts and figures that they give her. When using the Political Mashup search engine for her own queries, she gave negative and positive feedback about the website.
Positive Negative
1. ‘It’s great that you can search on speaker.
1. ‘I would like to search on a speaker only and not put in a search term’
2. ‘The word that you search on are highlighted in the results’
2. ‘Sometimes the search engine doesn’t recognize the speakers properly’
3. ‘The facets are helpful’
4. ‘Some discussions which I cannot find through Google are easy to find in Political Mashup’
Table 18. Sandra Boersma’s feedback on the Political Mashup search engine
Point 2 of the negative feedback needs some explanation. When Sandra searches on the speaker Wittewall van Stoetwegen and puts in ‘gekkenwerk’ as text search, she was not able to find it. This is due to the fact that the XML code did not recognize Wittewaall van Stoetwegen as a speaker. Figure 17 shows how Nederhorst is recognized as a speaker, while he is not speaking.
Figure 17. Visualization of negative point 2 from table 17. Nederhorst is recognized as a speaker while this is not the case
11.3 Suggestions for improvement
After the first rounds of testing (the user feedback sessions), several suggestions were made to improve the Political Mashup search engine. Table 19 shows the points that were
integrated before the final usability testing took place:
Facets to filter on senate, house of representatives, vernigde vergadering or UCV and OCV were added
38
Text_Search.
Graphs were improved. It was made possible to look at ‘hits over time per party’ and ‘hits over time per speaker’
Default of granularity was set to topic Calender was added
Advanced search was added
Table 19. Suggested improvements that were integrated after the user feedback sessions. After the final usability tests, new points of improvements were discovered. Some of these points were already raised after the user feedback sessions, but the programmer decided not to implement them, so that’s why they are suggested again.
It should be possible to search on speaker without putting in text in the Text_search. Granularity default should be on speech.
It should be possible to adjust the input of text_search. At the moment it is only possible to delete the whole query, but often people want to adjust this.
There should be an explanation about the text_search options (Can a user enter AND for example?).
If a query has no results, it is difficult to understand this, so this should get more clear. The calendar should be replaced by an easier to use one.
The calender should not start in 1800, but in a more recent year.
The calender should give results when a user is searching on two similar dates. It should be possible to turn off advanced search.
When searching on a speaker it should be selected as a facet automatically. The website should be in Dutch - or at least have the option.
Table 20. Suggested improvements based on the final usability tests
After the user feedback sessions, it was suggested to put the granularity on topic as default. This was to make sure that the search engine would search through the whole debate. The problem with topic was that the graph showed things that the participant did not expect to show up. To give an example, when someone wanted to compare how often two parties discussed a certain topic, the person did not see what was expected. He or she wanted to see the two parties in the graph, but all parties showed up in the graph. When the user would have searched on speech, only the two parties would show up.
39
12.
Conclusion & Discussion
12.1 Hypotheses
1. The Political Mashup Search Engine will score well on effectiveness, efficiency and satisfaction.
The first hypothesis has to be rejected.
The hypothesis considers effectiveness, efficiency and satisfaction. First of all, effectiveness will be discussed. 4 out of 6 queries were successfully completed by all participants on the Political Mashup search engine. The queries that were difficult to complete were look-up searches. 40% of the participants was able to complete the following look-up task: ‘Find kamerstuk with dossiernumber 25604, number 8 about zorg (care). This was published on 10-11-1997.’ One of the other look-up tasks was completed by 60% of the participants. Therefore the website is not 100% effective when it comes to look-up queries, but, on the other hand, the exploratory tasks were completed by all participants.
The efficiency is the lowest for 2 and 3 which are both look-up questions. The mean completion time for these two queries was more than 4 minutes. The last question has the highest efficiency of 1:19 minutes which was an exploratory question.
The mean score on satisfaction was 3,23 (on a scale from 1 to 5, 5 being the highest satisfaction). Participants were asked to think aloud during which they complained about specific functionalities. Most complaints were about the graphs that didn’t show exactly what they wanted it to show. Also there were a lot of complaints about the calendar that can be used to find debates from a certain period.
2. The Political Mashup website will have a higher appreciation of effectiveness, efficiency and satisfaction compared to Overheid.nl. This means that it will be most suitable for both look-up search and exploratory search.
The second hypothesis also has to be rejected.
The Political Mashup website has a higher appreciation of effectiveness compared to the Overheid Website, but it scores relatively lower on satisfaction. It is difficult to compare the efficiency of both websites as only query 2 and 3 can be compared.
The participants were able to complete all three exploratory queries on the Political Mashup website. On the Overheid website 1 exploratory query was completed by 40% of the
participants. The other two were completed by none of the participants.
The look-up searches have a mean score of 66,66% success for both the Political Mashup Search Engine and the Overheid.nl.
This means that the Political Mashup website is more suitable for exploratory searches, but for look-up searches both websites are suitable.
40
12.2 Conclusion
Throughout this research paper, the 5 subquestions have been answered. The first three subquestions were:
Q1. How can lookup-searches be supported?
Q2. How can learn-searches be supported when aiming for knowledge acquisition,
comparison and aggregation?
Q3. How can investigate-searches be supported by looking at transformation?
These were answered by looking at tools that support the different kind of searches.
Through use cases and user testing it became clear how each search could be supported in the Political Mashup search engine.
The fourth subquestion was:
Q4. What is the difference in functionality between the Political Mashup Search Engine and the existing websites Staten Generaal Digitaal, Overheid.nl and Google Advanced Search?
These websites were compared. It became clear that all functionalities (except for one) of Google Advanced Search are offered on the Political Mashup Search Engine. It also became clear that the Political Mashup website offers functionalities that the search engine
Overheid.nl and Staten Generaal Digitaal do not offer. To give examples: on both
Overheid.nl and Staten Generaal Digitaal there are no facets to further explore the results. Neither is it possible to view the search results in a graph.
The fifth subquestion was:
Q5. How does the user experience the effectiveness, efficiency and satisfaction of the Political Mashup search engine and how does this differ from Overheid.nl? The effectiveness was measured by looking at whether people could complete their search goals. For the exploratory tasks carried out on the Political Mashup search engine this was 100%, while for the look-up searches this was 66%. The mean efficiency was highest for an exploratory search task (1:19 minutes to complete it), while it was lowest for a look-up search task (4:35 minutes to complete it). The satisfaction had a score of 3,23 on a scale from 1 to 5, 5 being the highest satisfaction.
On the other hand, Overheid.nl scored lower on the exploratory tasks compared to the Political Mashup search engine. 13,33% of the exploratory tasks was completed on the Overheid website. The look-up searches had the same score as the Political Mashup website: 66% was completed by the participants. It is difficult to compare the efficiency, because not enough information searches were completed on the Overheid website, so data is missing. Looking at satisfaction, the Political Mashup website has a mean score of 3,23, while the Overheid website has a score of 3,5.
41
Finally, the subquestions will feed the main research question, which is:
MQ: What is the optimal search interface for exploring political debates? By looking at ways that can help support exploratory search, implementing these in a website and testing it, it became clear that the Political Mashup search engine enables people to explore political debates. When carrying out exploratory search tasks, users are able to find what they need. Although the search engine scores high on effectiveness, the satisfaction is still to be improved. This is because users experience discomfort when using certain features of the website.
Therefore, the final conclusion will be that the Political Mashup Search Engine is the optimal search engine for exploring political debates when considering effectiveness. On the other hand, the website causes discomfort by the users and therefore cannot be called the optimal search interface when looking at satisfaction.
12.3 Discussion
12.3.1 Unfinished website
During the final user tests, the Politcal Mashup search engine was tested. Although it was working properly, it was far from finished. This had a great influence on the satisfaction of the participants. Various participants complained about small malfunctions, like that they did not like the fact that the website was in English or that they could not use the calendar properly. These were complaints that are easy to solve, but have a great impact on the satisfaction of people.
12.3.2 Comparison of the two websites
The Political Mashup search engine was compared to the Overheid search engine. This is not a 100% fair comparison since the Political Mashup engine is not finished yet.
12.3.3 One observer
When the websites were tested, there should’ve been two people present. One should act as host to give instructions and the other should take notes (Hearst, 2009). Due to
constraints, there was one person who carried out both tasks. This might have had influence on the results as the observer had to carry out various tasks at once.
12.3.4 Further research
For further research, it would be interesting to carry out the same usability tests for the Political Mashup search engine after the suggested improvements are implemented. This would be interesting, because the satisfaction is expected to go up with every improvement made.
It would also be interesting to have more experts testing the Political Mashup search engine to find out what they exactly demand from the website.
12.3.5 Self reflection
42
First of all, this was the first time that I had to communicate with a programmer which was interesting. I could’ve spent more time with him, because I only visited him once. After the first meeting we only communicated through email. If I would’ve met him more often in real life, we could’ve had discussions about why he didn’t want to implement a certain aspect that I proposed. Via email, this was more difficult.
Secondly, for me it was difficult to set up the structure of the thesis. It was not a linear process. Sometimes I was finding bugs myself but did not know where to mention this in my thesis. In my thesis it seems like I was setting up use cases and finding mistakes, but before that I also did testing myself. Because of this non-linear process, it took me some time to get the right structure in my thesis.
The user tests were very interesting for me, because it showed the importance of the users. By letting other people carry out queries, I found out about bugs which I could not find without the tests. On the other hand, I could’ve used more experts to test the website. I made up the queries for the user tests myself, while the intended users might demand different things from the website.
![Carter, G.A. 1973. J.L. Hobbs's local history and the library. [Book review]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==)