Gender bias in children's books: using generics to derive bias from a corpus

(1)

Universiteit van Amsterdam Amsterdam, June 2019

Gender bias in children’s books

Using generics to derive bias from a corpus

Anna Stalknecht

Supervisors: Robert van Rooij and Martha Steward

Bachelor Thesis Kunstmatige Intelligentie

Universiteit van Amsterdam

As it is used by children to obtain general knowledge bias in children’s books is very problematic. This research will compare a probabilistic method and a syntactic method to extract features that are part of the group "male" or "female" from a children’s book corpus. Both ways successfully derived features from the corpus but also a lot of noise was found. The syntactic method appeared to be best at extracting bias but that could be because it was the approach with the least derived noise.

(2)

1 Introduction

As it is written by humans, human text contains biases. In machine learning the concept of bias is used to explain an under-fitted model which will generalize too much on the given data. In human language bias can be seen in the same way. The lack of knowledge (or the lack of willingness to know) causes a person to make assumptions about certain concepts in the world. Lack of information or the use of already biased data leads to the creation of more biased data. This becomes particularly problematic when this information is derived from aspects of human culture and are known to lead to harmful behavior. When this bias appears in text that is used to educate humans, or to teach intelligent algorithms, existing biases will continue to exist in the knowledge of the learner. Therefore understanding and detecting bias is crucial for the minimization of future bias.

Research by the Observer newspaper found gender biases in children’s books. It found that overall males outnumbered females and most main characters were played by males. The males also played tough and big roles where female roles were kept smaller (Ferguson (2018)). Research by Peterson and Lach (1990) has shown that gender stereotypes in children’s books can be harmful for the cognitive development of children because books play a big role in teaching children to understand the world. This causes that bias is transferred to children through learning. Because bias is especially problematic in places where children obtain their general knowledge this research will use a children’s book corpus obtained from the Gutenberg project.

The linguistic field that studies the meaning in sentences and words is called semantics. In earlier research it was discovered that human biases can be found by using machine learning to derive semantics from a corpus (Aylin Caliskan (2017)). In that research a common semantic extraction method,

(4)

2

Word Embedding, was used. In this research it will be demonstrated that there are multiple ways to extract semantics from a corpus.

In this research a different approach to deriving semantics will be used. To find bias two groups will be made: Male and female. Then the features that occur with that group, given the corpus, will be derived with two different methods. First a theory about generics, using probabilities and then a syntactic method called Hearst patterns.

Generics are statements like: "Ducks lay eggs" and "Tigers are striped". These statements would be considered true by humans even though they are not necessarily true (Andrei Cimpian and Gelman (2010)). Where probably most tigers are indeed striped, less than half of the ducks (adult females) lay eggs. Also the sentence "Mosquito’s carry malaria" would be considered true even though only a small percentage of mosquito’s does carry malaria. These generic sentences play a big role in learning, representing and reasoning (Leslie et al. (2011)) and it is therefore important to study generics, especially in places where humans will obtain their general knowledge. Interesting for this research is that generic sentences contain a group, for example "ducks" and a feature about that group; "lay eggs". To find out whether a feature is accepted in relation with the group Robert van Rooij (pear) states that the probability that that feature appears in relation with the group minus the probability that that feature appears in relation with an alternative group needs to be considered. With this analysis of generics Van Rooij is able to explain why not only majority generics like "Tigers are striped" but also generics like "mosquito’s carry malaria" are accepted as true by humans. Because even though the probability that mosquito’s carry malaria is low, the probability that something else carry’s malaria is even lower.

Hearst patterns are another way to extract semantics, because they extract common sense from a corpus. The patterns like "a cat is a mammal" or "NP_x is a NP_y" give us general information about cats and mammals.

(5)

3

Consider the following sentence: "Anna is a brave young woman". There is a chance that this sentences is biased, as it is written by me and it makes a claim about me, but these bias will become bigger when generalizing this sentence more. For example, if Anna is part of the group female than and one would state that all females are brave young girls. This is the way we will use Hearst patterns to extract bias from the corpus.

In summary, this research will answer the following research question:

What features can be extracted about the male and female groups, and how do the two extraction methods compare to each other

First we hypothesize that in the children’s book corpus there will be a difference in the features that occur with males and the features that occur in females. We also hypothesize that the probabilistic method by van Rooij would be better in deriving these biased features because it can explain "weak" features of a group. Secondly we hypothesize that there will be an overlap in obtained features by the two extraction methods but that Hearst patterns will also obtain many non biased features where the probabilistic method will be getting more noise.

(6)

4

2 Theoretical Background

In order to successfully perform this research every aspect requires assumptions made based on the specifics of this research and on the corresponding theoretical background. This chapter will look further into the steps and aspects of this research and the assumptions that must be made.

2.1 Generics

As this research uses generics it is necessary to make a number of assumtions in order to be able to extract features from the groups. A commonly used semantic extraction method is Word Embedding which states that a word is characterized by the company it keeps (Schakel and Wilson (2015)). So we can assume that the feature of the group will be occurring close to the occurrence of the group. Because children’s book have many characters and to narrow the chance of using features of the wrong group in this research the assumption will be made that when a group occurs in a sentence, features of that group will be occurring in the same sentence. As a result of this all words in that sentence will be considered a feature of that group.

2.2 The Probabilistic Approach

For the probabilistic approach we will simply count all occurrences of features with their groups. To minimize the noise with stop words the words are not counted separately but as noun phrases, small parts of the sentence where at least a noun is present. This is also because we assume that the feature of a group in a generic sentence is not a single word but a phrase, like "lay eggs" for the group birds. So we assume that the group is represented as a single word, where the features are noun phrases that occur surrounding that word.

(7)

2.3 Hearst Patterns 5

2.3 Hearst Patterns

Hearst Patterns were initially designed to derive hyponymy from a text. Hyponymy is a relationship between words. Words that are in hyponymy relation have a more generalized word, "the hypernym", and a more specific word, "the hyponym". For example, the word "mammal" is the hypernym of "cat" but the word cat can also be the hypernym of another word, for example "kitten". The main idea behind extracting hyponymy from text was to find a way to feed more information to WordNet, a lexical database where at that time every input was done by hand. Marti Hearst discovered a way to do it computational with the use of Hearst Patterns. In this case it is also necessary to determine what the groups and what the features are. Because the hypernym is a more generalized way of saying something about the hyponym the hypernym is considered the feature where the hyponym is considered the group. The pattern "NP_x is a NP_y" the NP_x is the hyponym and the NP_y is the hypernym. And when taking the sentence "Anna is a brave young woman" as an example one can indeed see that "Anna" belongs to the group female where "a brave young woman" is a feature of that group. In appendix A1 there is list of all patterns used, there you can see the hyponym als NP and the hypernym as NP_H to make it clear.

2.4 Generics versus hyponymy

Generics and hyponymy derived from Hearst patterns can not be considered the same, but some hyponymy extracted by Hearst patterns can appear to look like a generic, for example: "A mosquito is a malaria carrying insect". Another way Hearst patterns can act like generics is when they are generalized to a group. The sentence "Anna is a brave human being" isn’t a generic, but generalized to the group female it becomes a generic: "A girl is a brave human

(8)

6 2.5 Nouns and adjectives

being". That brings us to the assumption that we make that the groups and the features extracted by the both methods can be compared to each other.

2.5 Nouns and adjectives

In order to visualize the results the features are split up in nouns and adjectives. In a generic sentence and also in a Hearst pattern the words that are best in describing a group are the nouns and the adjectives, because they say something about the object (Miller and Fellbaum (1991)). Therefore the assumption is made that the difference in nouns and adjectives per group will tell us the most about the bias.

(9)

7

3 Methodology

3.1 The corpus

In previous research by Facebook (Jason Weston (2015)) a children’s book corpus was used. After the research Facebook made this corpus available for public use1. The corpus consist of 1.6GB of children’s books drawn from the

Gutenberg Project2_{. The Gutenberg Project is a volunteered project which}

makes books available for public use. Facebook scraped the books from the Gutenberg Project in text format. The corpus consist of 108 children’s books from 12 different authors. Four of these authors were female. A list of the book titles and the authors can be found in appendix A. Because the corpus was already used in a research the corpus was cleaned well. The book titles are all uppercase and were deleted for this research. More cleaning was not necessary.

3.2 Determining the groups

To derive bias from a corpus using generics it was necessary to form two different groups consisting of the two genders; "male" and "female". The python nltk corpus package 3 _{offers a list of names that are classified as male or female.}

With these names and some other gender specific pronouns and words as “he” and “she” and “mother” and “father” the male and female groups are specified. The groups were made into lists to make it manageable to compare and to loop over.

1_{https://research.fb.com/downloads/babi/} 2_{https://www.gutenberg.org/}

(10)

8 3.3 Deriving the features

3.3 Deriving the features

The features were derived with the two different approaches. First with the probabilistic method and secondly with Hearst patterns. In both ways the extracted features are independently analyzed and saved as numpy4 arrays.

3.3.1 Deriving features using probability

In this section we use the probability p(f|G) - p(f|¬G). Where G stands for the group, either male or female, and ¬G is the opposite group. The corpus was loaded as a raw text file and with the ntlk tokenize package the corpus was made into a list of sentences. While looping over the sentences, if a sentence contained a gender specific word that sentence was saved in a list for female or male sentences. The sentences that had no gender specific words in it were not used. After obtaining the sentences each sentences was split into noun phrase chunks. A noun phrase chunk is a part of a sentence that has at least a noun in it. Noun phrase chunks can be found when parsing a sentence.

3.3.1.1 Nounpharse chunks

To make the chunks each sentence had to be parsed. For this particular case it was most convenient to use a RegexParser with the following grammar:

"NP: (<V+>|<NN?>)+.*<NN?>"

This grammar is obtained from code online5_{. The grammar excepts all noun}

phrases that starts with at least one and max 2 verbs or nouns and ends with a noun. The noun phrase chunks are saved in a dictionary with two keys: male and female. The chunks from the female sentences were saved with the female

4_{https://www.numpy.org/}

5

(11)

3.3 Deriving the features 9

key and the male with the male key. After that the probability of a feature was calculated using p(f|G) - p(f|¬G), by simply counting the features of each group. This resulted in a lot of noise, for example: when a book was only about boys, a lot of specifics of that book were counted as a feature of that boy. This caused a lot of story specific words to get high probabilities. Therefore we decided not to look at the probability of a feature, but to accept all features with a probability larger than 0. In total 3321 male features and 2781 female were extracted.

3.3.2 Deriving features using Hearst patterns

For the Hearst patterns the starting point was not the groups but the patterns. First a list of patterns was derived from both the Hearst pattern paper by Marti Hearst herself, which contained 6 patterns, and the Hearst revisited research (Roller et al. (2018)) providing 16 more patterns. In total 22 patterns were used, a list of the patterns can be found at appendix A2. As with the probability the corpus was made into a list of sentences and each sentence was searched for a pattern. Every sentence that contained a pattern was saved in a dictionary with the pattern as a key. This resulted more than 5000 sentences together containing 18 of the 22 patterns. At this point the sentences were not yet filtered for gender groups. As expected the pattern with the most result was the pattern "is a". That pattern was also the one that gave the most noise. It gave a lot of sentences like "there is a bridge". Which doesn’t contain a hyponymy expression. Therefore when splitting the sentences into hyponyms and hypernyms the sentences that contained the following words as a hyponym were not saved:

’it’, ’that’, ’here’, ’there’, ’this’

While splitting the hyponyms and hypernyms both were scanned for the gender specific words and were only saved when they contained one of them. The

(12)

10 3.3 Deriving the features

hypernyms were saved as a feature of the gender. In total there were around 700 male features and 500 female features found.

(13)

11

4 Analysis

The analysis of the results is done independently so the difference between the two methods can be compared. To analyze the results a number of steps were made. After extracting the features for each group the features were labeled as nouns or as adjectives.

4.1 Using GenSim to generate wordvectors

Every noun and adjective that was extracted is translated into a wordvector. This research used GenSim to generate these wordvectors6. The GenSim

word2vec algorithm was trained on the whole corpus. The lists of nous and adjectives were translated to 25 dimensional vectors.

4.2 Plotting the wordvectors

To get a better understanding of the differences and similarities of the words the words were plotted in a graph. In order to do that the scikit learn PCA function7

was used to transform the 25 dimension wordvectors into a 2 dimensional object that can be plotted in a graph. These are the results:

It becomes clear that it is difficult to draw conclusions about the results when the words are plotted this way. The nouns and adjectives have a certain overlap and a number of differences but there are no obvious clusters or big differences between the male and female words.

6_{https://pypi.org/project/gensim/}

(14)

12 4.3 Agglomerative clustering of the word vectors

Figure 4.1: Male and female adjectives and nouns extracted with Hearst patterns

Figure 4.2: Male and female adjectives and nouns extracted with P(f|G)-P(f|¬G)

4.3 Agglomerative

clustering

of

the

word

vectors

To get a better understanding of the features that were extracted and how they compare to each other a agglomerative clustering method was used. Therefore the python Scipy package with agglomerative clustering was used8 _{with the}

ward linkage and after trying different amount of clusters the decision was made to use 5 clusters.

(15)

4.3 Agglomerative clustering of the word vectors 13

4.3.1 Hearst Patterns

(a) Female noun clusters

(b) Male noun clusters

Figure 4.3: Extracted nouns with Hearst patterns

In these plots there is quite a difference between the plots of the female and male noun clusters, but it is still difficult to determine significant differences between the clusters.

As visible in the plots there are big differences in adjectives that are used for males and females. There are some overlapping features but males and females have totally different clusters. In the cluster section these clusters will be looked more into.

(16)

14 4.3 Agglomerative clustering of the word vectors

(a) Female adjective clusters

(b) Male adjective clusters

Figure 4.4: Extracted adjectives with Hearst patterns

4.3.2 Probability

Derived from probabilities the noun features from males and females don’t have any big differences, the biggest difference is that the ammount of noun features for the male group is bigger.

There is a big difference in adjectives derived from probabilities though. Because of this graphs we will look into the clusters of the adjectives closer in the cluster section.

(17)

4.4 The clusters 15

(a) Female noun clusters

(b) Male noun clusters

Figure 4.5: Noun clusters with nouns extracted with probability

4.4 The clusters

To get a better understanding of the plots of the adjectives the lists of adjectives are described below. The colors of the clusters are stated with the lists but the individual clusters can not be compared to the cluster with the same colour from the different gender group, as these are not necessarily the same kind of cluster.

(18)

16 4.4 The clusters

(a) Female adjective clusters

(b) Male adjective clusters

Figure 4.6: Adjective clusters extracted with probability

4.4.1 Hearst

(19)

4.4 The clusters 17

4.4.1.1 female

Blue: Words like strong, bright and delicate that describes the group in general Orange: Words that mostly describe a mental state like ’proud’, ’jealous’ and ’grateful’

Green: Mostly words that describe the personality, like ’ladylike’, ’professional’, ’housewifely’ and ’good-for-naught’ Red: Also words describing

the group in general like ’respectful’, ’dearest’, ’christian’

Purple: Words that describe the appearance of the group like: ’handsome’ and ’young’, but also words to describe the group in general like: ’famous’, ’strange’, ’different’ and ’timid’.

4.4.1.2 male

Blue: Mostly words to in general describe the group like: ’big’, ’serious’ and ’favourite’ a real coherence can not be found. Orange: Words to describe the personality like: ’good’, ’honest’ and ’clever’. Green: also words to describe the personality: ’half-crazy’, ’cold-blooded’, ’self-satisfied’. Red and purple: Not really coherent words generally describing the group: ’monstrous’, ’German’, ’fearsome’.

In general the male features obtained by Hearst patterns are way more generalized, they can tell many different things about males. Where the female features are a lot more specific, they tell something about the appearance of the female, or about the personality. Where the personality is strong or weak, but nothing in between.

(20)

18 4.5 Comparing the methods

4.4.2 Van Rooij

A list of all adjective clusters obtained with the van Rooij method can be found in appendix A4

4.4.2.1 female

Blue: incoherent words with a some noise like: ’Toady’, ’married’, ’dreadful’ and ’Quiquern’. Orange: Mostly verbs that were tagged as adjectives: ’lose’, ’understand’, ’give’ Green: Words that describe an apperance: ’square’, ’dark’, ’pale’ Red and purple: Again words that don’t really express something about

the groups like ’natural’, ’mexican’, ’daily’ and ’broken’

4.4.2.2 male

Blue: Wrongly tagged verbs: ’hear’, ’obey’, ’live’ Orange: general describing words like: ’good’, ’desperate’, ’Greek’. Green: Not usefull. Red: Words like ’worthy’, ’proud’ and ’afraid’ that give a description about the group

As seen in the clusters above the method by van Rooij isn’t giving us clear answers about the features of the groups. That is probably because the method by van Rooij didn’t contain a lot of adjectives.

4.5 Comparing the methods

To compare the two methods table 4.1. It becomes clear that the method by van Rooij extracts significantly more features than the Hearst method. But as found in the results the method also extracted a lot of noise. Another big difference between Hearst patterns and probability is that with the probability little adjectives were extracted. That could be expected as generics don’t necessarily contain a lot of adjectives.

(21)

4.5 Comparing the methods 19

Hearst van Rooij overlapping Total male features 700 3321 not measured Total female features 500 2781 not measured

Male adjectives 132 106 12

Female adjectives 123 76 5

Male nouns 366 2345 138

female nouns 294 2018 124

(22)

20

5 Conclusion and discussion

As expected, some features derived from the corpus about the male and female groups are biased but the approach used contained a lot of noise which made it hard to see the amount of bias. Especially with the use of probabilities many features were derived where a lot of features were not use full. A way to clean these features is necessary to make a better comparison between the groups. Also both extraction methods extracted significantly more male features. This for itself can already be seen as bias but it can cause problems in analyzing the results because the corpus just generally tells us significantly more about male characters than about female characters. A problem we found when extracting the features was that the probability of a feature was to a large extent determined by the context of the book. When a large book was all male, a lot of story specific words were counted as a feature of male, and not counted as a feature of females. This resulted in a lot of noise because words that were actually male features were less common and thus had a smaller probability. For example in the male group the features with the highest probabilities are words like "Farmer Brown" and "Old man Coyote". This noise would probably be a lot smaller with a bigger corpus and even then it could be necessary to filter out these types of words to get better results. There is a bigger children’s book corpus available by the university of Oxford but it can only be used for specific research. Unfortunately this research did not meet the requirements for Oxford University’s approval. It would be interesting to use that corpus for future research. A larger corpus may also enable future researchers to detect differences in gender features between male and female writers.

Aside from the noise, the approaches had some overlap in the results. Especially in derived nouns almost half of the results found with Hearst patterns was also found with probabilities.

(23)

References 21

References

Andrei Cimpian, A. C. B. and Gelman, S. A. (2010). Generic statements require little evidence for acceptance but have powerful implications. Aylin Caliskan, Joanna J Bryson, A. N. (2017). Semantics derived automatically

from language corpora contain human-like biases. 356(4).

Ferguson, D. (2018). Must monsters always be male? huge gender bias revealed in children’s books. The Guardian.

Jason Weston, Antoine Bordes, S. C. e. a. (2015). Towards ai complete question answering: A set of prerequisite toy tasks.

Leslie, S.-J., Khemlani, S., and Glucksberg, S. (2011). Do all ducks lay eggs? the generic overgeneralization effect. Journal of Memory and Language, 65(1):15 – 31.

Miller, G. A. and Fellbaum, C. (1991). Semantic networks of english. Cognition, 41(1):197 – 229.

Peterson, S. B. and Lach, M. A. (1990). Gender stereotypes in children’s books: their prevalence and influence on cognitive and affective development. Gender and Education, 2(2):185–197.

Robert van Rooij, K. S. (To appear). Generics and typicality: A bounded rationality approach. Linguistics and Philosophy.

Roller, S., Kiela, D., and Nickel, M. (2018). Hearst patterns revisited: Automatic hypernym detection from large text corpora. CoRR, abs/1806.03191.

Schakel, A. M. J. and Wilson, B. J. (2015). Measuring word significance using distributed representations of words. CoRR, abs/1508.02297.

(24)

22

Appendix

A1

Hearst patterns used

NP which is an example of NP_H NP which is a class of NP_H NP which is a kind of NP_H NP and any other N_H NP or any other NP_H NP and some NP_H NP or some NP_H NP which is called NP_H NP is a special case of NP_H NP is a NP_H that NP is an NP_H that unlike most NP_H, NP unlike all NP_H, NP unlike any NP_H, NP unlike other NP_H, NP like most NP_H NP like all NP_H NP like any NP_H NP like other NP_H NP NP_H including NP* NP_H such as NP* NP_especially NP*

(25)

A2 List of books and authors 23

(26)

24 A3 Hearst adjective clusters

A3

Hearst adjective clusters

female Blue: [’great’, ’strong’, ’little’, ’bright’, ’delicate’, ’fine’, ’sweet’, ’grand’, ’beautiful’, ’small’, ’rough’, ’black’, ’large’, ’few’, ’new’, ’yellow’, ’red’, ’friendly’] Orange: [’better’, ’smart’, ’dear’, ’innocent’, ’wicked’, ’stupid’, ’much’, ’proud’, ’worth’, ’good’, ’true’, ’nice’, ’dreadful’, ’grateful’, ’jealous’, ’ignorant’, ’real’, ’sensible’, ’generous’, ’fortunate’, ’bad’, ’foolish’, ’lucky’, ’clever’, ’lazy’, ’important’, ’hard’, ’sure’, ’tired’] Green: [’good-for-naught’, ’easy-going’, ’fastidious’, ’flighty’, ’ladylike’, ’professional’, ’nicelooking’, ’estimable’, ’stingy’, ’touchy’, ’disgruntled’, ’braw’, ’housewifely’, ’stuck-up’, ’imaginative’, ’honorable’, ’unencumbered’, ’Episcopalian’, ’carnal’] Red: [’kind-hearted’, ’industrious’, ’daily’, ’respectful’, ’former’, ’well-bred’, ’hospitable’, ’alarming’, ’truest’, ’dearest’, ’GREAT’, ’obedient’, ’nae’, ’Christian’, ’instinct’, ’distracted’, ’shorter’, ’coral’, ’loose’] Purple: [’remarkable’, ’respectable’, ’sacred’, ’regular’, ’old’, ’safe’, ’perfect’, ’pleasant’, ’quiet’, ’particular’, ’other’, ’terrible’, ’young’, ’famous’, ’stern’, ’strange’, ’most’, ’present’, ’splendid’, ’rich’, ’handsome’, ’holy’, ’different’, ’capable’, ’curious’, ’fearful’, ’wonderful’, ’funny’, ’sorrowful’, ’poor’, ’ten’, ’many’, ’best’, ’own’, ’excellent’, ’odd’, ’awful’, ’timid’] male Blue: [’long’, ’thin’, ’great’, ’Little’, ’big’, ’cold’, ’brown’, ’daily’, ’different’, ’regular’, ’strange’, ’excellent’, ’young’, ’old’, ’richer’, ’sweet’, ’remarkable’, ’harsh’, ’nineteen’, ’prejudiced’, ’jealous’, ’determined’, ’only’, ’disagreeable’, ’little’, ’ignorant’, ’serious’, ’favorite’, ’poor’, ’particular’, ’terrible’, ’earnest’, ’free’, ’least’, ’sight’, ’desperate’, ’dead’, ’fat’, ’right’, ’holy’, ’fearful’, ’nest’, ’famous’, ’new’, ’dearly’, ’powerful’, ’latter’, ’such’, ’else’, ’extra’, ’attractive’, ’ten’, ’other’, ’yellow’, ’horrible’, ’responsible’, ’unpleasant’, ’smaller’, ’despondent’, ’deep’, ’more’, ’bigger’, ’Green’, ’helpless’, ’two-legged’, ’odd’, ’independent’, ’awful’, ’French’] Orange: [’fine’, ’smart’, ’handsome’, ’bad’, ’good’, ’honest’, ’beautiful’, ’sure’, ’real’, ’hard’, ’stupid’, ’true’, ’nice’, ’sensible’, ’perfect’, ’ridiculous’, ’dear’, ’splendid’, ’rich’, ’born’, ’clever’, ’dreadful’, ’proud’,

(27)

A4 van Rooij adjective clusters 25

’worse’, ’lazy’, ’late’, ’important’, ’bold’, ’lucky’, ’best’, ’glad’, ’afraid’, ’easy’, ’ill’] Green: [’half-crazy’, ’indefensible’, ’but-parast’, ’cold-blooded’, ’clumsy-looking’, ’grass-tangled’, ’readier’, ’engine-driver’, ’octogenarian’, ’self-satisfied’, ’carnal’] Red [’monstrous’, ’German’, ’hard-working’, ’skilful’, ’stingy’, ’unreasonable’, ’polished’, ’stunned’, ’intelligent’, ’minor’, ’similar’, ’friendless’, ’Irish’, ’experienced’, ’expensive’, ’obstinate’ Purple [’fearsome’, ’unheard’]

A4

van Rooij adjective clusters

female Blue [’dear’, ’future’, ’right’, ’*’, ’Big’, ’Little’, ’favourite’, ’English’, ’answered’, ’mad’, ’Queen’, ’miserable’, ’Irish’, ’nice’, ’lean’, ’Toady’, ’up-stairs’, ’turned’, ’rid’, ’married’, ’Lady’, ’Green’, ’Methodist’, ’grown’, ’plenty’, ’Spencervale’, ’North’, ’enough’, ’wrinkled’, ’verandah’, ’cavern’, ’Quiquern’, ’dreadful’, ’Bumble’] Orange [’lose’, ’understand’, ’draw’, ’give’, ’send’, ’wish’, ’imagine’, ’suppose’, ’want’, ’enjoy’, ’hear’, ’bear’]

Green [’square’, ’dark’, ’blue’, ’gray’, ’crimson’, ’pale’, ’marble’, ’brown’, ’grey’]

Red [’twentieth’, ’Natural’, ’Daily’, ’Improvement’, ’Sunday-school’, ’Mexican’, ’storm-spirit’]

Purple [’sat’, ’poured’, ’fine’, ’broken’, ’willow’, ’lay’, ’dusky’, ’front’, ’nest’, ’shower’, ’light’, ’gable’, ’porch’, ’sweet’] male Blue [’give’, ’understand’, ’hear’, ’obey’, ’touch’, ’leave’, ’live’, ’attend’, ’write’, ’accomplish’, ’Latin’, ’knowest’, ’lead’, ’want’] Orange [’Little’, ’Good’, ’plenty’, ’Queen’, ’British’, ’Tower’, ’dear’, ’Lady’, ’Greek’, ’fairy’, ’Yellow’, ’younger’, ’forest’, ’Big’, ’stood’, ’blue’, ’North’, ’bright’, ’present’, ’loved’, ’stole’, ’Lock-out’, ’sat’, ’English’, ’attract’, ’taught’, ’desperate’, ’saw’, ’Charlottetown’, ’wiped’, ’gathered’, ’tall’, ’enough’, ’Middle’, ’replied’, ’Fifteen’, ’Best’, ’High’, ’Stranger-man’, ’Wonderful’, ’Great’, ’Fear’, ’Green’, ’hollow’, ’straight’, ’urged’, ’continued’,

(28)

26 A4 van Rooij adjective clusters

’Chief’, ’goose’, ’Wild’, ’Spotty’, ’darkest’, ’aspen’, ’tree’, ’Whitefoot’]

Green [’Hereditary’, ’Moorish’, ’slouch’, ’Bitter’, ’Arctic’, ’Scottish’, ’Marblehead’, ’inordinate’, ’Gratian’, ’Hessian’, ’Norval’, ’Grampian’]

Red [’ready’, ’dead’, ’sure’, ’stiff’, ’worthy’, ’proud’, ’fine’, ’sorry’, ’glad’, ’wasted’, ’hungry’, ’afraid’, ’right’, ’pleasant’, ’weary’, ’answered’, ’wicked’, ’careless’, ’dreadful’]

Purple [’Natural’, ’Seven-league’, ’Terrible’, ’Southern’, ’latitude’, ’Neolithic’]

Gender bias in children's books: using generics to derive bias from a corpus