Development of an interactive experiment on the effect of sequential information on the formation of generic beliefs

(1)

Bachelor Informatica

Development of an interactive

experiment on the effect of

sequential information on the

formation of generic beliefs

Bodi Boel´

e

August 18, 2020

Supervisor(s): dr. Patricia Mirabile

Inf

orma

tica

—

Universiteit

v

an

Ams

terd

am

(2)

(3)

Abstract

Building on earlier (ongoing) investigations on the semantics of generics, this thesis will discuss the online implementation of a dedicated tool to host an interactive experiment and then use this tool to execute a usability survey focused on optimising the efficiency and participant experience of the tool.

The purpose of the experiment is to investigate the claim that generics are formed about a target group through a process of associative learning, which considers both the target of the learning as well as a relevant contrast class.

The experiment described in this thesis is part of an ongoing investigation by an NWO research project on generics at the Institute for Logic, Language and Computation (ILLC) of the University of Amsterdam.

(4)

(5)

4 Results 21 5 Discussion 23 5.1 Gathered results . . . 23 5.1.1 Form results . . . 23 5.2 Process . . . 23 5.2.1 Setting up . . . 23 5.2.2 Python . . . 24 5.2.3 Webtool . . . 24 5.2.4 Late changes . . . 24 5.2.5 Future work . . . 24 5.3 Limitations . . . 24 5.4 Ethical aspects . . . 25 6 Conclusions 27 Bibliography 29 Appendices 31 A ReadMe 33 B Form results 37 B.1 Dutch form response . . . 37

B.2 English form response . . . 37

(6)

(7)

CHAPTER 1

Introduction

1.1 Generic statements

Generic statements are sentences that are used to attribute features, patterns, sentiments and other beliefs to a group. ‘Ducks lay eggs’, ‘tigers are striped’, ‘lions have manes’ and ‘ticks spread Lyme disease’ are examples of generic statements. Generic statements express generalisations about the members of a kind and lack any form of quantification and are therefore ‘bare’ state-ments. Other than bare, there are also other types of statements that describe generalisations, such as quantified statements. Quantifiers that can be found in statements are adverbs such as ‘most’, ‘all’ and ‘many’, examples of the (universally) quantified form of previously mentioned examples are ‘all ducks lay eggs’ and ‘all tigers are striped’. This research will focus on ‘bare generics’ and disregard any other type of statements.

‘Bare’ generic statements express useful generalisations, but an unambiguous conclusion on when people think these statements are true can not be formed. There is no unique critical point of statistical prevalence of the feature in the kind where people in general tend to desig-nate a statement as true. Intuitively a prevalence threshold of above 50% would seem a good assumption of when the predominant judgement will be true. However, when people have to judge the statement ‘lions have manes’, the predominant judgement will be that the statement is true, although less than 50% of lions (only male lions above the age of two years) have manes. When asked about the statement ‘ticks spread Lyme disease’ the predominant conclusion will as well tend to be that the sentence is true, even though the statement is only true for 2.7% (RIVM 2019) of tick bites that actually transfer the disease whereas 20% of ticks carry the dis-ease. These examples show that only assessing prevalence does not seem to be a good way to determine the way a generic will be judged. Leslie, Khemlani, and Glucksberg 2011 investigated how participants interpret a given generic with the way participants interpret the corresponding universally quantified statement in order to test whether generic statements have the same mean-ing as universal statements. They were able to show the existence of a generic overgeneralization effect. This effect occurs for statements describing minority characteristics1as being a universal characteristic when the corresponding generic statement is interpreted as true. In other words, this effect describes that people have the tendency to interpret the statements ‘ducks lay eggs’ and ‘all ducks lay eggs’ as having the same implication about the proportion of ducks that lay eggs. This thesis is about implementing an experimental paradigm, by creating an online tool which can be used in further research to investigate the differences between the theorised formal meaning of generic sentences and the empirical meaning people give to these sentences.

1.2 Formal and Empirical research

There are two main types of research on generics, empirical and formal research. Where empirical research focuses on direct observations and experiences, formal research focuses on modelling

(8)

and formulating formal rules and proving the correctness of a rule regarding the production and interpretation of natural language. The formal approach therefore is more limited in covering all actual cases. The formal approach that posits that generics are interpreted solely on the basis of a fixed prevalence threshold generic is criticised by many because of the proposed interpretation that any formal rule should be based solely on probabilistic considerations e.g. the majority rule. In formal theory rules have been defined that are correct based on the assumption that people’s judgements are based on statistical information. Even though researchers themselves know of the various counter-examples in natural language to these rules. The existence of these mismatches is interpreted as a sign that this formal theory is flawed.

In this project on generics the formal approach and the empirical approach are closely related and are viewed as being complementary with each other. The experimental design is therefore motivated by both approaches. The goal of the design is to enable collection of empirical data, but the design itself and the implementation of that design are determined by the formal theory that will be investigated. The goal of the experiment is to test the rules proposed by a new formal theory, using data gathered through empirical research, this theory takes contrasts in prevalence and the role of learning into consideration.

1.3 Goal

The goal of this research project is to design and implement an interactive experiment able to collect empirical evidence to test the claims made by the proposed formal theory of generics. The experiment focuses on the importance of the learning process in the formation of beliefs regarding generic statements. This project seeks to implement an online tool and the usability of this tool will be tested through a usability survey on a small sample of participants. The usability of the tool will then be optimised using the data gathered in the usability survey to make sure the tool can be used in an ongoing research project by Kochari, Van Rooij, and Schulz 2020 on ‘Generics and Alternatives‘.

In their research, Kochari, Van Rooij, and Schulz 2020 use a formal approach that takes into account the contrasts in prevalence and the learning process on the meaning of generic sentences. Their paper discusses three studies that tested their proposal that generics are evaluated in light of differences in prevalence between the described group and a relevant contrast group. They presented participants visual stimuli with different levels of prevalence in either a no contrast or high contrast condition and asked them to judge the assertability of generic sentences describing these stimuli. They found that the higher the level of prevalence of the non-contrastive case, the less the level of prevalence in the contrastive case would impact the assertion. In their con-clusion they mention a shortcoming of their research that should be the focus of future work, and the implementation developed in this thesis tries to find a way to satisfy that shortcom-ing. The shortcoming mentioned is that the experiment did not take into account the learning dimension that is described in the theory from Kochari, Van Rooij, and Schulz 2020, because the visual stimuli in those experiments were static. According to their theory, the way people take the given contrast into account is the result of a learning process, that can be modelled by the Rescorla–Wagner learning model, which conceptualises learning in terms of an association between conditioned and unconditioned stimuli (Wilson 2012). To be able to better test the theory, the interactive design developed in this thesis will let participants sequentially receive data (one by one), which is the learning process assumed by associative learning. The experiment will also try to improve the way information is absorbed, by forcing the participant to actively participate in the experiment, instead of passively looking at the experiment.

Ivan Pavlov started research on, what is nowadays called, Classical Conditioning in the 1890s. The model he created showed the correlation between trigger (A) and result (B), and how the influence of A ⇒ B was related to A ⇒ ¬B. For example, Pavlov used dogs in his investigation and he found that dogs start drooling when they are offered food. Giving a signal (A) before giving food(B) to the dog would let the dog associate that signal with getting food. This association would get stronger every time signal and food happened together and weakened if they do not happen together. This would eventually result in the dog starting drooling every time the signal(A) was given, even though no food (¬B) was offered (Gormezano and Moore

(9)

1966). The Rescorla–Wagner model, which further develops this paradigm, would for example suggest that pairing the Pavlov signal with a very tasty piece of food (e.g. a juicy steak) would have a greater likelihood of causing salivation than pairing the signal with a less tasty piece of food.

The Rescorla-Wagner model can also be applied to this thesis. Because we are also investi-gating how the information a participant has been exposed to influences their belief. According to the theory, people’s beliefs in a generic at a time t are the result of the number of times people have observed the association between group and feature compared to the number of times people have not observed that same association until the given time t. At time t + N , after observing new information, the belief in a generic will adapt again, depending on the content of the information received. For example, ‘Birds lay eggs’ will be interpreted, at a given time t, as true because people have been exposed to more information about cases where something was a bird and it laying eggs compared to cases where something was a bird and it not laying eggs. The strength of this belief will decrease if people observe a new case, at time t + 1, where a bird does not lay eggs. To test this theory on associative learning, the implementation of the tool discussed in this thesis provides a setting where information is received sequentially.

(10)

(11)

CHAPTER 2

Theoretical background

2.0.1 Intuitions observed by empirical research

Experimental psychology has highlighted a number of biases and preferences in the way people interpret generic statements. One of these is the generic overgeneralization effect, the bias for people to accept a generic statement based on minority characteristics1_{. For example people} tend to classify ‘Men attack people’ as false and ‘Sharks attack people’ as true, even though the chance of men attacking people is vastly higher than the chance of sharks attacking people. This ‘negative’ bias towards sharks, and non-human entities in general, is firstly caused by the striking type of the used predicate. Secondly people are more negatively biased towards non-human categories, as has been indicated in experiments conducted by Tasimi et al. 2017. In different experiments, participants were assigned a domain, either people or tools and things. They were then given a statement about one of these domains and had to rate the chosen domain as either helpful or dangerous. This experiment showed a preference where people tend to be more negative towards generics describing non-human entities. The implementation of the tool this thesis is about, shows different types images to the participants. If the experiment would incorporate images showing different domains of entities (e.g. human entities and non-human entities) then such a bias towards certain entities could influence how a statement is rated and therefore it is important to acknowledge the possibility of such biases during the experiment.

A bare generic statement, such as ‘ducks lay eggs’, is interpreted as true, possibly on the con-sideration that many ducks lay eggs. The universally quantified version of this generic statement, ‘All ducks lay eggs’, should on the other hand be rejected because there are counter-examples that falsify the universal statement, namely that it is physically impossible for male ducks to lay eggs. Different experiments have shown that the universal statement ‘all ducks lay eggs’ tends to be interpreted as true, even though it only correctly describes approximately 50% of the cases. The paper by Leslie, Khemlani, and Glucksberg 2011 set out different experiments focusing on this phenomenon. In one of their experiments they found that, despite knowing the relevant prevalence of a feature in a group that lead to the adoption of a generic statement, universally quantified statements would often be inferred to be true for that same group on the basis of the generic statement. For example, after they were given information on the percentage of ducks that lay eggs, participants would still choose to classify ‘all ducks lay eggs’ as true. In another experiment they offered the participant a correct alternative statement to the bare generic state-ment. For example, the participant was offered the choice between: ‘female ducks lay eggs’ and ‘all ducks lay eggs’ and even then the participants would hold on to the overgeneralised state-ment. Their research shows that people tend to falsely accept (over)generalised statements even when offered a more descriptive alternative, as described in the previous section.

(12)

2.0.2 Formal research

The classical approach to the formal analysis of generic sentences is based on the adoption of the majority rule. Assuming a sentence ‘tigers are striped’ (form: ‘Gs are f’) the majority rule for a generic sentence is defined as:

Definition 2.0.1. Majority rule A generic sentence of the form ‘Gs are f’ is true i.f.f. P (f |G) > 1₂ The majority rule has various counter-examples in natural language, such as the ‘ticks’ ex-ample mentioned in the introduction. With 2.7% of ticks carrying Lyme disease this statement is predicted to be false by definition of the majority rule, but this conflicts with people’s in-tuitions. In their paper Kochari, Van Rooij, and Schulz 2020 suggest that the reason for the mismatch between the majority rule’s predictions and people’s intuitions is that the proponents of the majority rule do not take into account various notions of alternatives. They suggest that many shortcomings of the majority rule can be overcome taking into account various notions of alternatives.

Rooij and Schulz 2019 based their analysis of generic statements on the majority rule, which they borrow from the definition in Cohen 1996. In their paper they analysed the intuitions behind the majority rule (Definition 2.0.1) that some authors saw to be a natural approach to formalise generic statements. Just like many, they doubt the correctness of the rule, partly due to the vagueness and context-dependence of generics. For example, they mention that in many cases people tend to accept and interpret generics based on a distorted picture of the generic statement, such as stereotyping. This is caused by the fact that during their lifetime (i.e. learning process) people are exposed to information provided by (e.g. (social) media) instead of the actual frequencies of cues. As previously mentioned, the formal analysis is based on the majority rule, but because the majority rule is unable to describe people’s judgements accurately enough, Rooij and Schulz 2019 develop a more complex rule. The first component of that rule is the consideration of the prevalence in the described group. The second component of the rule is that prevalence is not considered in an absolute fashion, but rather in a relative fashion by comparing the prevalence of a feature to a relevant contrast group chosen on the basis of context and additional background knowledge. The third component they add to the rule is that the prevalence comparison will also be affected by the impact of the feature. The fourth and currently last aspect that the rule takes into account the learning process throughout people’s life which impacts the way people receive information.

2.0.3 Role of generic beliefs

Experimental evidence indicating that the truth of a generic statement does not straightforwardly depend on the underlying statistical prevalence of a feature among the described group had already been reported in 1966 by Gilson and Abelson 1965. Using the ‘ticks spread Lyme disease’ example, people tend to believe that all ticks spread Lyme disease and therefore people tend to be more afraid of ticks than you would expect from a statistical point of view. Another example is the fear of Rottweilers or the assumption that flying as a means of transport is unsafe. The fear of Rottweilers for example is caused by the way information is spread and the severeness of a Rottweiler’s bite, because when another dog would attack a person, it would probably not make any headlines. That flying is unsafe is caused by the same phenomena. Accidents are rare, but when an accident does happen, the effects are huge, also because of the large number of people involved in a single airplane accident. As suggested by Rooij and Schulz 2019, these stereotypical assumptions are partially caused by the influence of the media. Striking news, for example, gets more attention and is therefore becoming the way news media make headlines and draw attention to their media websites. Due to the extra attention and the way social media aid spreading these kinds of ‘trending’ articles, an imbalance can occur in the way people gather and receive information. Along with the way social media use user data to fill their user’s social media feed with items that interest the user, the interest of a user in a particular matter can influence the information the user gets and therefore expose him to a distorted picture about the subject of interest. Because in those cases, the information the user receives is mostly one-sidedly in favour of a given generalisation, this can influence how people interpret generics. People base their judgement on the information they have received instead of the entire body of existing

(13)

data. This shows that the little evidence that is needed for a statement to be seen as true has far-reaching consequences (e.g. people being disproportionately afraid of Rottweilers). This underlines the importance of the overarching research on how people form generic beliefs and what the effect of sequential information is on their judgement.

2.1 Hypothesis

The tool implemented in this thesis is meant to enable the collection of evidence regarding two hypotheses derived from the theoretical proposal by Kochari, Van Rooij, and Schulz 2020. Hypothesis 1. When information is received sequentially the pattern of responses will match the pattern observed in experiments where the information is received simultaneously.

Hypothesis 2. The pattern of responses will be the pattern predicted based on the relative difference of a feature. Assertability is higher when P (f |G) > P (f |Alt(G)) defined in the paper by Kochari, Van Rooij, and Schulz 2020 as

Assertability of G’s are f’ = P (f |G) − P (f | ∪ Alt(G)) .

The implementation can also be used to test the role of impact on generic statements. In this thesis the focus has been on the tool being used to verify the hypotheses mentioned above, but it can be extended to test other scenarios, for example in a scenario where a feature is described as dangerous or not.

(14)

(15)

CHAPTER 3

Methods

3.1 Experimental Setup

This experiment builds on the basis created by Kochari, Van Rooij, and Schulz 2020. In their experiment they show the participant a grid consisting of two differently coloured animals, as shown in Figure 3.1. The participant would be shown three possible pairs of animals, which were randomly presented in a condition with high contrast, no contrast and in a condition used as an attention check. In their experiment, the conditions and positions of the items in the grids could not be randomly generated for each participant, our interactive experiment allows for randomisation of the grids as well as varying the prevalence of the features per question. The information would all be shown at once to the participant, who would then receive the information passively and would then be asked to rate the generic statement using a continuous scale. This was also a point of discussion because the scale might be too precise and the expected meaning of a certain number rating too uncertain. This has been solved by using a discrete scale, as can be seen in Figure 3.4. To be able to investigate the influence of sequential data on the formation of generic beliefs, initially the experiment would be set up similar to the original experiment shown in Figure 3.1. The interactive implementation has a similar appearance, to be able to compare across studies. To be able to test the first hypothesis, ”When information is received sequentially the pattern of responses will match the pattern observed in experiments where the information is received simultaneously”, a requirement of the tool is that the contrast between the grids is variable. The participant must also interact with the interface and sequentially (in random order) turn around all tiles, one by one. This is required to let the participant receive the information sequentially, to test hypothesis two, ”The pattern of responses will be the pattern predicted based on the relative difference of a feature. Assertability is higher when P (f |G) > P (f |Alt(G)) defined in the paper by Kochari, Van Rooij, and Schulz 2020 as

Assertability of G’s are f’ = P (f |G) − P (f | ∪ Alt(G))

”. The participant will switch turns with the computer, turning around one tile at a time on their own side of the board. The experiment has been framed to be played as a game against a virtual opponent, this to keep the participant’s attention to the contrast between the two grids and to make the repetitiveness of the task less boring. After all tiles have been uncovered, the participant is asked to rate a statement describing the animals from the participant’s side of the grid on a 7-point Likert scale. For the usability survey there were 5 different types of animals, such as beetles and touracos. Each of the stimuli is divided into two groups, one version with the salient feature and one without the feature. These groups are shown in a non particular order and then randomly assigned to one of the conditions, to exclude the chance of a bias from participants for a certain feature. In the particular case of the usability survey there were 5 types of animals and four different conditions. For the experiment all users need to get the same information, but the participant may not be biased in any way. To avoid this, randomisation of the data is important, which is done by randomising the order of the feature ratio and the feature itself. An example of the initialised board is given in Figure 3.2.

(16)

Figure 3.1: Sample of experiments done by Kochari, Van Rooij, and Schulz 2020 as seen by the participant

The experiment forces the participant to have a sequential learning process. By only allowing the participant to see the current information, participants follow a certain randomised timeline to complete their game. This is achieved by closing tiles after they have been uncovered. To avoid focusing on the prevalence of a feature instead of the desired focus on sequential learning, the experiment hides tiles a few seconds after they have been uncovered, which makes it hard to keep track of the absolute prevalence of the feature.

3.2 Implementation

The tool is created as a web based tool, which can be accessed from any non-phone device. This implementation choice was made to prevent participants from getting distracted while taking the survey. Initially Python was one of the languages of choice, but because of the focus on interaction with the tool, Python was ruled out after about a week. The tool has been written using the following web-development languages: AJAX, CSS, HTML, JavaScript, jQuery, PHP and SQL.

The tool was developed particularly for this experiment and was tested on participants in a particular use case, but has features to allow modification by a user with administrator rights. The size of the board can be changed or even randomised on a given horizontal and vertical range. The researcher can add as many conditions as they wish. By changing the length of the experiment and the grid size, the researcher has the possibility to influence the time it takes to complete the experiment, (e.g. by increasing the board size and reducing the number of questions, the time the experiment takes can remain approximately the same). Even though the tool was designed to test two hypotheses of a specific theory it is flexible enough to be adapted to test further hypotheses of the theory of generics. By allowing for this flexibility in terms of

(17)

Figure 3.2: Sample of the experiments initial set-up

settings and changeable items, the tool is applicable for experiments in various other research topics.

3.2.1 Participant experience

For the particular use case of the usability survey, after opening the tool in their web browser the participant is prompted to one of two pages. If a participant does not join through a particular link, they are directed to a thank you page, with a small description of the study. If the participant does join through the correct link the participant is referred to the welcome page, showing a welcome note and a button to start the questionnaire introduction. After clicking the button the participant is given information about the terms of the research and is asked for their informed consent and asked to check their prolific ID.1 During these first 3 pages, if enabled by the investigator, Google’s ‘reCAPTCHA’ judges if the participant is not a robot, before allowing the participant to continue. If ‘reCAPTCHA’ allows the user to continue, the introduction to the study itself is shown.

The next page that the participant sees introduces them to the experiment by explaining the setting of the experiment. The next page the user is referred to is an instructions page. On this page, the participant sees what is expected and the participant can become familiar with the game-like setting used in the experiment in a small 2x2 practice setup. After completing these steps, the participant is referred to a last page before they are finally able to start the experiment. This last page asks the participant to enlarge their screen and to take the survey in a calm environment.

3.2.2 Interaction with the tool

The use case experiment presented in the usability survey is set up as two grids of 4 by 4 tiles each. The player (on the left side) plays against an opponent (randomised turns by a computer on the right side). By clicking on a tile, the participant can interact with the tool and discover

1_{Participants are recruited using Prolific.ac, an online platform aimed at connecting researchers and}

partici-pants willing to fill in surveys and questionnaires in exchange for compensation for their time Palan and Schitter 2018, to be able to reward the students we need a way to verify who they are.

(18)

Figure 3.3: Sample of an ongoing experiment. Player has to match the symbol currently visible on the opponent’s side.

the features of the group they are asked to learn about. The opponent turns a tile with the goal of matching the feature of the card that’s already visible. After each counter move, the tiles are visible for two seconds to show whether the features match, before both cards disappear again. Before the cards disappeared the user gets feedback and if the cards matched a point will be awarded, the points are kept in a per game scoreboard. After the cards have disappeared the opponent is again able to select a card after which the player is able to try to match the computer’s card again, as Figure 3.3 shows. This continues until all cards have been discovered. The participant then sees a generic statement which the participant must rate on preset scale. For the experiment the preset scale is a 7-point Likert scale with values ranging from ‘Strongly disagree’ to ‘Strongly agree’, using this scale the board is similar to the board shown in Figure 3.4. To conclude, each user gets a randomised combination of the prevalence of the target and the contrast with the contrast group. The goal of the experimental setup is to promote an active learning process. For this particular use case of the implementation, the hypotheses would be supported if ratings follow the same pattern as the ones reported in Kochari, Van Rooij, and Schulz 2020, which means that higher ratings are expected for higher prevalence and higher contrast compared to lower prevalence and lower contrast.

3.3 Data collection plan

The data to verify the usability of the tool was collected in a usability survey using the interactive web questionnaire created for this purpose, participants for this usability survey were recruited using social (media) connections. To also be able to receive feedback on the questionnaire itself, these participants received a link to a google form either in Dutch or in English. The form shortly introduces the experiment and asks the participant to evaluate the experiment after completing it. If the results to the usability tool satisfy the expectations, the tool is ready to be used in the described experiment on the theory of generics.

(19)

Figure 3.4: Completed game, waiting for the user to rate the statement ‘Hide Beetles from Marchena have dotted wings.’.

(20)

(21)

CHAPTER 4

Results

A usability survey was run on a small sample and allowed us to verify the usability of the design. Even though the goal of the usability survey was to check the usability of the tool and not to draw inferences about the predictions, we checked the data gathered to see if any conclusion could be drawn from the data, but the data had too low statistical power to allow us to draw any inferences regarding the empirical predictions.

Based on the scale, the expectation is that the participant will respond to a true statement with an equally true response on the scale, i.e. if a statement would be true for all occasions, in that case we would expect an equally true answer, which would be a score of 6, on our 7-point Likert scale. The scale used is the 7-point Likert scale, with collected integer values ranging from 0 to 6, with 3 being the neutral answer. When doing a short analysis of the limited sample data, it seems that there is a distinction between two categories of feature combinations, these are the combinations with a neutral assertability and those with a predictability smaller than −0, 70 and larger than 0, 70.

According to the theoretical model, we should observe assertability ratings either neutral or negative when a statement has probability P (f |G) − P (f | ∪ Alt(G)) = 0 is either neutral or negative. In our data, we find that out of 47 results, 57% (27) correspond with this theoretical expectation. On the other hand, when the theoretically predicted assertability of G’s is smaller than −0, 70 and larger than 0, 70, 69% (11 out of 16) of responses correspond to the hypothesis, whereas when the theoretically predicted assertability is between −0, 70 and 0, 70 52% (16 out of 31) of responses correspond to the hypothesis (See Appendix C).

With a sample of 12 participants the experimental set-up has gotten some helpful feedback. During the final phase of development, the tool was distributed to the first participants to be able to find and fix issues. Throughout this phase changes have been made using the feedback given by the participants, improving the reliability and user-friendliness of the experiment. For example, the first three participants all mentioned that the interactive dimension was too slow, especially the time they had to wait for the opponent (computer) to make its turn. These responses helped improve the timing of the interactive dimension and after these changes had been implemented participants did not mention the slowness of the interactive dimension.

(22)

(23)

CHAPTER 5

Discussion

5.1 Gathered results

5.1.1 Form results

When comparing the form and tool results, one might find that the input with timestamp ‘14-6 16:30’ does not have a counterpart in the tool results (Appendix C). The ‘empty’ record was due to a problem during the experiment which was that the participant had an empty screen. However, there was a record of this person in the database. After testing the tool on multiple devices, a lot device dependable causes could be ruled out. The most likely explanation is that the participant had turned off JavaScript for their browser, which results in a blank page. To prevent this from happening in the future, a message will be shown to a participant with JavaScript turned off that JavaScript is needed to participate in the experiment.

The initial responses were really helpful, but unfortunately users that rated the experiment more negatively did not specifically mention any points of improvement. The gathered results helped rule out usability and implementation problems. Excluding the result with timestamp ‘14-6 1‘14-6:30’ the average score to the user experience question is 2,45, which is only slightly towards the ‘user friendly’ side of the scale, even though a mode rating of 4 was observed. At the start of the experiment participants are made aware that the experiment will take 10 to 15 minutes. There is a difference between the subjective rating of length (i.e. average rating 3,25, which is slightly long and the mode value ’exactly right’) and the actual time it took the participant to complete the experiment. The data showed that total length of the visit to the website, where the experiment was executed using the final set-up, took between 8.5 and 12 minutes to complete, which is shorter than the estimated 10 - 15 minutes which had been announced to participants. All in all, the usability survey was very useful for locating and solving possible weaknesses of the tool.

5.2 Process

5.2.1 Setting up

The goal of the tool is to encourage participants to sequentially learn something about the appearance of the items of a group and to apply these findings to a generic statement, which they then should rate. During the implementation and comparison to the previous experiment, some shortcomings were raised. For example the scale of the previous experiment was too precise, which can cause fluctuation in the results. Throughout the process of implementing the tool, the expected functionalities of the tool have changed quite a bit.

(24)

5.2.2 Python

The first week the focus was on implementing the tool through Python. This became rather difficult after going further into detail on what the tool must be capable of doing. Therefore early in the second week the switch was made to using web development languages as mentioned in the implementation section.

5.2.3 Webtool

The initial tool was going to look like a game of memory where the user had the possibility to uncover any tile one-by-one, until being shown the full grid. Mid-April Robert van Rooij, the lead researcher on the project, attended one of the weekly meetings and a number of changes were discussed to the design and implementation of the tool. Among them were the introduction of a computer player, varying the size of the grid and to interrupt the flow of clicking of the user so that the participant does not focus directly on the numerical quantity of items that present the feature of interest. After implementation of those changes, the tool was ready to be optimised using a usability survey midway through May.

5.2.4 Late changes

That same week, mid-May, Katrin Schulz, senior researcher on the project, provided additional feedback during a meeting. This prompted further changes on the design of the experiment and with that on the implementation of the tool. Those were changes such as letting the user experience the tool more like a game of memory, not showing the participant the tiles that had already been uncovered and later on changing the game of memory into a card matching game against a computer opponent and keeping score of the matches that have been made by both players. The ‘behaviour’ of the computer also needed tuning, in order for the game to be more perceived as a back and forth between the player’s turns and the computer’s turns. These late changes to the experimental design made the process of finalising the project less structured and limited the time for the usability survey to be executed. Therefore the number of participants in the usability survey was very limited.

5.2.5 Future work

During future work, similar situations could be prevented by initiating the project with a meet-ing with all actors involved which could help to better outline the project and specifically the objective, which in this case was the way the tool was implemented. Other improvements would be, better defining a timeline, with enough space to accommodate changes.

5.3 Limitations

The participants so far have been more and more positive in their feedback about the tool. The tool has multiple variables that can be changed, this way the tool can be set up for multiple experiments in different ways, while the specific use case reported here maintains the focus on testing the role of sequential information on learning. This means the tool could be adapted by the researcher to focus on the experiment and gather more relevant data, but could also be adapted by other researchers to be used in new experiments.

From the first few results, came a complaint about using the tool as a game. The goal set for the participants is to win as much points as possible and to beat the computer by making more matches than the computer. From the participants’ point of view this seems unfair, since a participant has no way of knowing which symbol is under which tile. Therefore the participant has no influence on their own score, which means winning or losing the game is based on pure luck. This could demotivate participants and make them pay less attention to the information shown to them even though there is no reward structure for winning and/or losing, which means the stakes of winning are low. From the experiment’s point of view this was done on purpose. The game is only used as a cover, because showing all tiles in advance would allow participants

(25)

to learn all the information simultaneously instead of sequentially and in that case the tool would not be valid towards the goal of the experiment. These conflicting needs between fairness and experiment goals could be a limitation of this experimental set-up.

5.4 Ethical aspects

This thesis and the tool that has been developed takes into account various ethical aspects, such as fully informing the participants. This is done by shortly introducing the subject on the first page: ‘In this survey you will be asked to judge whether a sentence can be correctly asserted in a given scenario’. Further, during the survey, people get more information regarding the survey and how it works. Also participants are asked to give (written) permission for data collection by accepting the terms written in the consent form. This was done to follow ethical recommendations issued by ethics boards about conducting experiments with human beings, which requires to make sure that participants are willing to let the researcher store and process the data they generated. As defined by the declaration of Helsinki, the interest of the participant is paramount, therefore the participant also has the option to stop at any time. The confidentiality of this investigation is ensured by the fact that personal data is not stored and therefore cannot be linked to a participant. The last ethical aspect that has been covered by the tool is the debriefing. After submitting their responses, participants have the ability to read about the how and why of the research and if necessary they can even go into further detail using the provided literature references and by contacting the lead investigator. The tool itself meets certain ethical aspects, by taking a lot of care to commenting the source code in a clear way and by making the source code publicly available. This makes the experiment reproducible and at the same time allows other researchers to see how the experiment was set up. Also taking a lot of care to write a clear ReadMe file (Appendix A), will help towards fully informing interested parties on how to use the tool and what it was designed for.

(26)

(27)

CHAPTER 6

Conclusions

The online implementation of a dedicated tool to host an interactive experiment and the use of this tool to execute a usability survey focused on optimising the efficiency and participant experience of the tool and corresponding literature have been discussed in this thesis. The use of a formal approach to the meaning of generic sentences has been discussed. The shortcomings of the experiments by Kochari, Van Rooij, and Schulz 2020 were what guided the design behind this online interactive tool. According to the theory proposed in that paper, people’s generalisations result from a learning process similar to that of the Rescorla-Wagner learning model, which conceptualises learning as resulting from the contrast between conditioned and unconditioned stimuli.

The goal of this thesis was to design a tool that could be used to verify both hypotheses; 1. When information is received sequentially the pattern of responses will match the pattern

observed in experiments where the information is received simultaneously.

2. The pattern of responses will be the pattern predicted based on the relative difference of a feature. Assertability is higher when P (f |G) > P (f |Alt(G)).

The tool was able to capture the data needed for analysis. Unfortunately, collecting the data necessary to the quantitative analysis of the proposed hypotheses falls outside the scope of this thesis. This tool is an alternative to conventional surveying software which did not have the capability of executing the tasks that were needed for this experiment, such as the randomisation of the questions together with the randomised creation of the grid.

The usability survey has been useful in helping improve the tool. Using the feedback given throughout the usability survey, the tool has been adjusted and improved. Therefore the ex-pectation is that the implementation of the tool can be used in the overarching research project without issues, as many issues have already been found and resolved using the usability survey. Future work will focus on collecting enough data using the tool, to be able to verify the hypothesis of this thesis as also defined by Kochari, Van Rooij, and Schulz 2020.

(28)

(29)

Bibliography

Cohen, Ariel Mordechai (1996). Think generic! The meaning and use of generic sentences. Carnegie Mellon University.

Gilson, Charlotte and Robert P Abelson (1965). “The subjective use of inductive evidence.” In: Journal of Personality and Social Psychology 2.3, p. 301.

Gormezano, Isidore and JW Moore (1966). “Classical conditioning”. In: Experimental methods and instrumentation in psychology 1, pp. 385–420.

Kochari, Arnold, Robert Van Rooij, and Katrin Schulz (2020). “Generics and Alternatives”. In: Frontiers in Psychology 11, p. 1274.

Leslie, Sarah-Jane, Sangeet Khemlani, and Sam Glucksberg (2011). “Do all ducks lay eggs? The generic overgeneralization effect”. In: Journal of Memory and Language 65.1, pp. 15–31. doi: 10.1016/j.jml.2010.12.005.

Palan, Stefan and Christian Schitter (2018). “Prolific.ac—A subject pool for online experiments”. In: Journal of Behavioral and Experimental Finance 17, pp. 22–27. issn: 2214-6350. doi: https://doi.org/10.1016/j.jbef.2017.12.004. url: http://www.sciencedirect.com/ science/article/pii/S2214635017300989.

RIVM (Oct. 2019). Ziekte van Lyme. url: https://www.rivm.nl/ziekte-van-lyme.

Rooij, Robert van and Katrin Schulz (2019). “Generics and typicality: a bounded rationality approach”. In: Linguistics and Philosophy 43.1, pp. 83–117.

Tasimi, Arber et al. (2017). “Differences in the evaluation of generic statements about human and non-human categories”. In: Cognitive Science 41.7, pp. 1934–1957. doi: 10.1111/cogs. 12440.

Wilson, W.J. (Mar. 2012). The Rescorla-Wagner Model, Simplified. url: https : / / campus . albion.edu/wjwilson/files/2012/03/RWSimplified.pdf.

(30)

(31)

(32)

(33)

APPENDIX A

ReadMe

Figure A.1: ReadMe file (dated June 14th 2020). The most updated file can be found here: ‘https://github.com/BodiB/UvA Generics/blob/master/README.md’

(34)

contributors 1 forks 0 stars 1 issues 0 open license not specified LinkedIn

A tool to be used in an interactive experiment on the effect of sequential information on the formation of generic beliefs A tool to be used in an interactive experiment on the effect of sequential information on the formation of generic beliefs

The purpose of the experiment is to investigate the claim that generics are formed about a target group through a process of associative learning, which considers both the target of the learning as well as a relevant contrast class. The experiment described in this thesis is part of an ongoing investigation by an NWO research project on generics at the Institute for Logic, Language and Computation (ILLC) of the

University of Amsterdam.

Explore the docs » Explore the docs » View Demo · Request Feature

Table of Contents Table of Contents

About the Project Getting Started Prerequisites Installation Usage Contributing Contact Acknowledgements

About The Project About The Project

Languages used Languages used

AJAX CSS HTML

(35)

Getting Started Getting Started

To get a local copy up and running follow these simple steps.

Prerequisites Prerequisites

A web server and a MySQL database .

Installation Installation

1. Clone the UvA_Generics

git clone https://github.com/BodiB/UvA_Generics.git

2. Copy all files from the "Code/website" directory into your public_html folder.

3. Import the database file stored in "Code/database" directory into the database you just set-up 4. Open the db.php file and fill in your database information.

5. Open init.php and change the variables to your prefered settings.

6. Go to your_url/admin.php and login with the username and password set in step 5. 7. Set up all necessary values in the Settings , Statements and Features menu. *Notes:

1. The Max # questions setting must match the number of statements and features in the other menu's.* 2. The Referal link back to Prolific setting can be left empty if Prolific is not used to recruit participants.

8. If you wish to use reCAPTCHA , make sure that the $_SESSION["recaptcha"] value in your init.php is to 0 . Go to https://developers.google.com/recaptcha and set up your reCAPTCHA . Then move to init.php in your public_html folder and set the necessary values there.

If you do not wish to use reCAPTCHA , in your init.php set the $_SESSION["recaptcha"] value to 1 .

9. The img folder is pre_loaded with images, to use other images, just add/replace these to the folder and they will be available in the admin menu. ** map.png is used in introduction.php .

Usage Usage

Gathering data Gathering data

Spread the experiment using either the link: your_url/new_user.php or when using Prolific, fill in the following link there: your_url/prolific.php , using the PROLIFIC_PID option as: your_url/prolific.php?PROLIFIC_PID=<?>

Evaluating data Evaluating data Feedback Feedback

Participants can give feedback on their experience after their participation. An overview of this feedback is available on the Feedback section of the admin menu .

Results Results

In the Results section of the admin menu , you are able to see the gathered results in a table form. You can also download the data as a CSV-file .

Contributing Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciatedgreatly appreciated. 1. Fork the Project

2. Create your Feature Branch ( git checkout -b feature ) 3. Commit your Changes ( git commit -m 'commit comment' ) 4. Push to the Branch ( git push origin feature ) 5. Open a Pull Request

Contact Contact

Bodi Boele - bodiboele@gmail.com

Project Link: https://github.com/BodiB/UvA_Generics

Acknowledgements Acknowledgements

(36)

(37)

APPENDIX B

Form results

B.1 Dutch form response

The results gathered through the Dutch form can be found on Google Docs Link:

‘https://docs.google.com/spreadsheets/d/1PLSfR wIK-oo5AGtuL2LPdUuQ AfAn5 -anSjShhfN1k/edit?usp=sharing’

B.2 English form response

The results gathered through the English form can be found on Google Docs Link:

‘https://docs.google.com/spreadsheets/d/1GP28ru4fdBwNUb0sQuQf VmA-vZmnl0hx7HvkwK254g/edit?usp=sharing’

B.3 Form responses

*Note: The form has changed during the experiment and besides the dutch form an English form has been added too. To add context to statements made, all answers are shown here. The dutch answers have been translated in Figure B.1

(38)

Timestamp

Did you encounter any problems during the

experiment? If so, which?

How did you experience the user interface?

How do you think the user interface could be improved?

Did you like the experiment?

What would you like to see different? (Also consider timing and fairness of the game)

Was the explanation necessary to understand the experiment?

How clear did you find the explanation?

Do you have any suggestions to improve the explanation?

How did you find the length of the

experiment? (The time it took to complete it?) Taking into account that the the time window is 10-15 minutes.

Do you have feedback or suggestions to improve the experiment (user interface, appearance, explanations)?

27-5-2020 10:53:57 Yes

The computer is too slow and the pop-up messages are

annoying 2 Yes

Well faster timing and better explained the

purpose of the game. Yes 3

A simpler use of language would

help Too long

27-5-2020 17:31:14 Yes

Not sure if it was intended, but the first puzzle only had

blank spaces. 3 No

The opponent could

make the moves faster Yes 4 Too long

27-5-2020 18:25:39 No 3

It was a little

long Yes 4 Too long

28-5-2020 11:27:12 No 4 Yes

I would make the grid a bit smaller. 3x3 or 4x4 would have been enough for me. Now it took a long time for my feeling. The frequency of the pop-ups should have

been slightly lower Yes 5 Too long

Also try to enable it for mobile devices 29-5-2020 13:44:21 No I didn't really understand why it was a game as you have no influence over whether you win or not. Can't a kind of memorial be

made of it? 1 Not! Very special

Being able to affect your scores

I would have liked a

little more depth 3

Describe more depth or really the goal that you should remember how many matches or something there are or what the characteristics of your insect are

especially Exactly right No No

2-6-2020 10:55:01 No 1 Yes Exactly right

2-6-2020 14:14:50 No 4 Yes Too short

10-6-2020 21:03:18 Yes

the length it was taking too long and it did not

make sense 2 No Way too long

14-6-2020 16:30:17 Yes Blank screen 5 Blank screen Way too short Doesn't work

(39)

Figure B.1: Responses to the Dutch and English feedback form combined. (Google Translation) (dated June 15th 2020)

(40)

Tijdstempel

Ben je tijdens het experiment op

problemen gestuit? Zo ja, welke?

Hoe heb je de gebruikersinterface ervaren?

Hoe zou de gebruikersinterface volgens jou verbeterd kunnen worden?

Vond je het experiment leuk om te doen?

Wat zou je graag anders zien? (Denk bijvoorbeeld ook aan timing en eerlijkheid van het spel)

Was de uitleg nodig om het experiment te begrijpen? Hoe duidelijk vond je de uitleg? Heb je tips om de uitleg te verbeteren?

Hoe vond je de lengte van het experiment? (De tijd die nodig was om deze uit te voeren?)

Heeft u feedback of suggesties om het experiment te verbeteren (gebruikersinterface, uiterlijk, uitleg)? Verder nog vragen/opmerkinge n over dit google form?

Timestamp

Did you encounter any problems during the

experiment? If so, which?

How did you experience the user interface?

Did you like the experiment?

How did you find the length of the experiment? (The time it took to complete it?) Taking into account that the the time window is 10-15 minutes.

Do you have feedback or suggestions to improve the experiment (user interface, appearance, explanations)? Do you have feedback or suggestions to improve the experiment (user interface, appearance, explanations)? 27-5-2020 10:53:57 Ja De computer doet te langzaam en die popups

zijn vervelend 2 Ja

Nou snellere timing dus en het doel van het spel

beter uitgelegd. Ja 3

Ik denk meer Jip en Janneke taal. Te lang

27-5-2020 17:31:14 Ja

.Niet zeker of het de bedoeling was, maar de eerste puzzel had alleen

maar blanco velden. 3 Nee

De tegenstander zou de zetten sneller uit kunnen

voeren Ja 4 Te lang

27-5-2020 18:25:39 Nee 3 Het was wel wat lang Ja 4 Te lang

28-5-2020 11:27:12 Nee 4 Ja

Ik zou de grid iets kleiner maken. 3x3 of 4x4 was genoeg geweest voor mij. Nu duurde het voor mn gevoel erg lang. Ook de frequentie van de pop-ups had iets lager

mogen liggen Ja 5 Te lang

Probeer hem ook mogelijk te maken voor mobiele devices

29-5-2020 13:44:21 Nee

ik snapte niet zo goed waarom het een spel was aangezien je geen invloed hebt of je wint of niet. Kan er geen soort memorie van gemaakt

worden? 1 Niet! erg bijzonder

invloed hebben op je scores

Iets meer diepgang had

van mij erbij gemogen 3

Meer diepgang of echt het doel omschrijven dat je daardoor moet onthouden hoeveel matches of iets er zijn of wat de kenmerken van jou insect vooral

zijn Precies goed Nee Nee

2-6-2020 10:55:01 Nee 1 Ja Precies goed

2-6-2020 14:14:50 Nee 4 Ja Te kort

10-6-2020 21:03:18 Ja

de lengte het duurde te

lang en het sloeg op kak 2 Nee Veel te lang

12-6-2020 19:07:14 Nee 4 Ja Precies goed

14-6-2020 16:30:17 Ja Geen beeld 5 Geen beeld Veel te kort Werkt niet

(41)

Figure B.2: Responses to the Dutch and English feedback form combined. (dated June 15th 2020)

(42)

(43)

APPENDIX C

Experiment results

(44)

Tijdstempel user_id

question_numbe

r statement vertical_tiles horizontal_tiles feature_A_left feature_B_left feature_A_right feature_B_right response Expected response using formula from hypthesis 2 Response shifted to scale between -1 to 1 Corresponds to hypothesis 2?

27-5-2020 10:34:38 wCJzyAYlP2voMYQ 3 Chicks from Baltasar are pink. 5 5 0 25 16 9 1 -1 -0,6666666667 TRUE

27-5-2020 17:13:20 eYkluhn6Og5p7De 4

Chicks from Marchena are

pink. 5 5 0 25 16 9 1 -1 -0,6666666667 TRUE

27-5-2020 22:01:56 hf1ZGPQitt4lqUY 3

pink. 5 5 0 25 16 9 0 -1 -1 TRUE

29-5-2020 13:17:18 IxdNaHH9Z14rr9c 4

Hide Beetles from Marchena

have dotted wings. 4 4 0 15 10 5 0 -1 -1 TRUE

12-6-2020 18:59:43 kFahMsWhyeTjj6n 2

Genovesa barfishes have pink

fins 4 4 0 16 9 7 1 -1 -0,6666666667 TRUE

12-6-2020 23:13:09 EcuyBpyDjOhIiW3 4

fins 4 4 0 16 9 7 0 -1 -1 TRUE

15-6-2020 11:10:32 BJoGsZAtg5Y84qz 4

Genovesa beetles have blue

spots. 4 4 0 16 9 7 0 -1 -1 TRUE

15-6-2020 11:10:32 BJoGsZAtg5Y84qz 5

Genovesa ticks have red

backs. 4 4 0 16 0 16 0 -1 -1 TRUE

27-5-2020 18:04:22 neGLIrCnkPDl0AS 3

have dotted wings. 5 5 0 25 16 9 3 -1 0 FALSE

2-6-2020 14:00:12 LzXt1YBkc2vpMGz 3

spots. 4 4 0 16 9 6 5 -1 0,6666666667 FALSE

10-6-2020 20:53:03 APWONvOqBmXtRoQ 4

backs. 4 4 0 16 9 7 4 -1 0,3333333333 FALSE

15-6-2020 8:33:25 FKTJKxD4hanFvVv 4

backs. 4 4 0 16 9 7 3 -1 0 FALSE

27-5-2020 22:01:56 hf1ZGPQitt4lqUY 4

Genovesa chameleons are

green. 5 5 5 20 10 15 2 -0,6 -0,3333333333 TRUE

27-5-2020 17:13:20 eYkluhn6Og5p7De 2

Hedgehogs from Marchena

have yellow hogs. 5 5 5 20 10 15 5 -0,6 0,6666666667 FALSE

27-5-2020 18:04:22 neGLIrCnkPDl0AS 2

have dotted wings. 5 5 5 20 10 15 3 -0,6 0 FALSE

29-5-2020 13:17:18 IxdNaHH9Z14rr9c 3

Marchena flies have red

wings. 4 4 3 12 6 9 5 -0,6 0,6666666667 FALSE

27-5-2020 10:34:38 wCJzyAYlP2voMYQ 4 Brasilian flies have red wings. 5 5 5 19 10 15 4 -0,5833333333 0,3333333333 FALSE

29-5-2020 13:17:18 IxdNaHH9Z14rr9c 1

Marchena flies have red

wings. 4 4 7 8 8 5 4 -0,06666666667 0,3333333333 FALSE

27-5-2020 22:01:56 hf1ZGPQitt4lqUY 1

pink. 5 5 12 13 12 8 1 -0,04 -0,6666666667 TRUE

27-5-2020 17:13:20 eYkluhn6Og5p7De 3

green. 5 5 12 13 12 8 5 -0,04 0,6666666667 FALSE

27-5-2020 18:04:22 neGLIrCnkPDl0AS 1

Hedgehogs from Marchena

have yellow hogs. 5 5 12 13 12 8 5 -0,04 0,6666666667 FALSE

27-5-2020 10:34:38 wCJzyAYlP2voMYQ 2 French chameleons are green. 5 5 0 0 0 0 5 0 0,6666666667 FALSE

27-5-2020 10:34:38 wCJzyAYlP2voMYQ 1

Hedgehogs from Amsterdam

have yellow hogs. 5 5 10 10 12 12 1 0 -0,6666666667 TRUE

27-5-2020 17:13:20 eYkluhn6Og5p7De 1

pink. 5 5 0 0 0 0 3 0 0 TRUE

27-5-2020 18:04:22 neGLIrCnkPDl0AS 4

green. 5 5 10 10 12 12 1 0 -0,6666666667 TRUE

(45)

Tijdstempel user_id

question_numbe

r statement vertical_tiles horizontal_tiles feature_A_left feature_B_left feature_A_right feature_B_right response Expected response using formula from hypthesis 2 Response shifted to scale between -1 to 1 Corresponds to hypothesis 2? 12-6-2020 18:59:43 kFahMsWhyeTjj6n 4

Genovesa mice have brown

stripes on their backs. 4 4 8 8 8 8 1 0 -0,6666666667 TRUE

12-6-2020 23:13:09 EcuyBpyDjOhIiW3 1

backs. 4 4 8 8 0 16 3 0 0 TRUE

12-6-2020 23:13:09 EcuyBpyDjOhIiW3 3

Genovesa touracos have

green crests. 4 4 8 8 8 8 2 0 -0,3333333333 TRUE

fins 4 4 8 8 8 8 3 0 0 TRUE

spots. 4 4 8 8 0 16 3 0 0 TRUE

15-6-2020 11:10:32 BJoGsZAtg5Y84qz 1

fins 4 4 8 8 8 8 0 0 -1 TRUE

29-5-2020 13:17:18 IxdNaHH9Z14rr9c 2

pink. 4 4 6 6 8 8 4 0 0,3333333333 FALSE

spots. 4 4 8 8 0 16 4 0 0,3333333333 FALSE

fins 4 4 8 8 8 8 4 0 0,3333333333 FALSE

2-6-2020 14:00:12 LzXt1YBkc2vpMGz 1

backs. 4 4 8 7 0 16 4 0,06666666667 0,3333333333 TRUE

2-6-2020 14:00:12 LzXt1YBkc2vpMGz 2

green crests. 4 4 8 7 8 7 1 0,06666666667 -0,6666666667 FALSE

12-6-2020 18:59:43 kFahMsWhyeTjj6n 3

green crests. 4 4 13 3 13 3 5 0,625 0,6666666667 TRUE

12-6-2020 23:13:09 EcuyBpyDjOhIiW3 2

stripes on their backs. 4 4 13 3 13 3 4 0,625 0,3333333333 TRUE

stripes on their backs. 4 4 13 3 13 3 3 0,625 0 FALSE

15-6-2020 11:10:32 BJoGsZAtg5Y84qz 2

green crests. 4 4 13 3 0 16 0 0,625 -1 FALSE

15-6-2020 11:10:32 BJoGsZAtg5Y84qz 3

stripes on their backs. 4 4 13 3 13 3 0 0,625 -1 FALSE

Mosquitoes have orange

heads. 4 4 14 2 14 2 4 0,75 0,3333333333 TRUE

green crests. 4 4 14 2 0 16 3 0,75 0 FALSE

2-6-2020 14:00:12 LzXt1YBkc2vpMGz 4

stripes on their backs. 4 4 14 1 14 1 6 0,8666666667 1 TRUE

2-6-2020 14:00:12 LzXt1YBkc2vpMGz 5