WHAT ENCOURAGES MEN AND WOMEN TO LIKE A JOB ADVERTISEMENT?
SUBMITTED IN PARTIAL FULFILLMENT FOR THE
DEGREE OF MASTER OF SCIENCE
MARIT BEEREPOOT
10983430
MASTER INFORMATION STUDIES:
DATA SCIENCE
FACULTY OF SCIENCE
UNIVERSITY OF AMSTERDAM
2019-06-26
First Examiner Second Examiner
Title, Name Affiliation Email
Prof. Dr. Evangelos Kanoulas UvA, ILPS
e.kanoulas@uva.nl
Dr. Yuval Engel
UvA, Amsterdam Business School y.engel@uva.nl
What encourages men and women to like a job advertisement?
Marit Beerepoot
University of Amsterdam Amsterdam, The Netherlandsmarit16-1@live.nl
ABSTRACT
Gender bias is still an ongoing problem in the job market, especially in areas associated with brilliance such as science, technology, engineering and mathematics fields. The main objective of this paper is to detect whether it is possible to distinguish differences and similarities in important features between men and women, when analyzing a model that predicts whether a person will like a job advertisement. The data that is used for this research contained information about companies, their job advertisements, job seekers and whether a job seeker liked or disliked the advertisements. The model is created using a random forest classifier, which combines several variables about the job seekers, the companies and the jobs. Several differences and similarities between men and women were found, such as that words related to brilliance and technology are more important for men, while words related to communal traits are more important for women. Full lists of features important for both genders and an analysis of the similarities and differences between those lists are presented.
KEYWORDS
Gender bias, Gender discrimination, Job advertisements, Text analysis, Textual representations, Prediction model
1 INTRODUCTION
The last few years the European Commission has been busy with trying to bridge the gender gap in employment rates. Only 66.6% of the women worked in 2017, while 78.1% of men did. Women are especially underrepresented in sectors such as the IT sector and the engineering sector (European Commission, 2018).
In the past, a lot of interesting research has been done in to why there is such a difference in employment rates between men and women. The lower employment rates of women can partly be explained by gender bias in job recruitment (Wille & Derous, 2018). Murciano-Goroff (2017) for example found that job recruiters are 12.37% less likely to contact women with comparable qualifications as men. Besides that, Walker et al. (2009) found that organizations can also (unintentionally) influence a job seeker’s perceptions of the company by their web design, which could also lead to a (unintentional) gender bias.
Gender bias is embedded into human brains early on. Bain et al. (2017) found that a common stereotype associated with men are high-level intellectual abilities. Girls at age 6 already showed less interest into games associated with high-level intellectual ability, than boys did. The stereotypical thinking continues over the years and only gets stronger. Bain et al. (2018) confirms that
this brilliance stereotype is present by job seekers and states that women seem to be less interested in jobs that require brilliance. People do not seem to be aware of gender biases. Gupta et al. (2008) found that business students implicitly confirmed the gender bias that males are better suitable for entrepreneurship, but would not confirm this when it was explicitly asked.
Previous research has shown that job advertisement can be gender biased in different ways, the bias can for example exists in the wording (by using certain grammatical elements) or by the type of personality traits that are described in the advertisements (Wille & Derous, 2017; Bem & Bem, 1973). Interestingly enough, most of the research into gender bias in job advertisements, and gender bias in general, has been done based on quantitative and qualitative research while using for example interviews, experiments and questionnaires. There is not a lot of research done based on a data science approach, where textual data is analyzed using machine learning. This paper will therefore discuss whether it is possible to predict if a man or woman will like a job advertisements based on mostly textual data. This article thereby tries to find differences in what encourages men and women to like a job advertisement. The dataset that is used for this research contains information about startup companies that offer the job, information about the job itself, information about job seekers and information about whether these job seekers like or dislike a job advertisement. The research question that is asked is:
What are differences or similarities in important features between men and women, when analyzing a model which predicts whether
a person will like a startup job advertisement?
To answer this question, four sub questions have to be answered:
1. Which classifier performs best when predicting whether a person will like or dislike a startup job advertisement? 2. Which (textual) features are most important in this
prediction model?
3. What are the differences or similarities between men and women?
4. How do these differences and similarities relate to previous research?
The paper is structured as follows. First section two discusses some related work about what gender bias is, what causes it and how this influences the job market. After that, a method is introduced to filter the data, to extract textual features from the data and to create a prediction model. In section 4, the created model is discussed and the sub questions are answered. The article ends with a conclusion and future work section.
2
2 RELATED WORK
Before differences and similarities between men and women can be identified in a prediction model, it is important to look at why there could be differences. Therefore, the next two subsections will explain what gender bias is, how gender bias influences the job market and how gender bias can be present in job advertisements.
2.1 What is gender bias?
Gender bias means that men and women with the same ability levels are treated differently. This is sometimes also called gender discrimination (Childs, 1990). In general, women are perceived as more communal, caring and interpersonally oriented, while men are perceived with words associated with leadership and agency (Gaucher et al., 2011). These stereotypes influence the way people are treated and how people define themselves, which also affects how people think and remember information about themselves and others. An example of this is that girls often underestimate their grades, even though they scored well in the past (Ellemers, 2018).
These gender stereotypes also have influence on how people think men and women should behave. Women in male dominated jobs or women that are ambitious, competent and competitive (agentic traits) are for example seen as less feminine, while women who behave as the stereotype are evaluated more positively. Men are seen as weak when they act more communal or violate other masculine stereotype traits (Ellemers, 2018).
Gender bias thinking is created at a young age. As discussed earlier girls at age 6 already show less interest in games associated with high-intellectual abilities, since this trait is seen as a masculine trait (Bain et al., 2017). This is caused by that our society contains a lot of gender bias in the day-to-day life, in for example advertisements. Gender bias even appears in books for children. Women in these books are often portrayed as caretakers, princesses or nurses, while men are portrayed as adventurers, knights and heroes (Temple, 1993).
People are not always aware of gender biases. As already mentioned in the introduction, Gupta et al. (2008) found that students implicitly confirmed that men are more suitable for entrepreneurship, while they denied it when it was explicitly asked. In other cases, people seem to acknowledge gender bias and gender discrimination. A study of Cundiff & Vescio (2016) found that when women were asked about why they were underrepresented in leadership positions, the women often used gender discrimination as an explanation. The women thus identified and acknowledged the gender bias.
Since it is unfair that men and women with the same abilities are treated differently and people are not always aware of gender biases, this could lead to problems in the job market.
2.2 Gender bias in the job market
Women form a minority in the job market, since the percentage of women that work is lower than the percentage of men (European Commission, 2018). Research in the past has
shown that job seekers from an ethic minority hold meta stereotypes, which means that they think that people outside the minority group, think about people from inside the ethnic minority group in a certain (negative) way. This can lead to that certain people do not apply to a job, because they think the organization holds negative meta stereotypes (Wille & Derous, 2018). However, organizations actually want to hire these minorities, since these minorities can increase the diversity of ideas in the organization (Gee. 2018). Since there is still a negative stereotype around women, which form a minority in the job market, women might think organizations hold negative stereotypes against them, which can prevent women from applying for that job, even though organizations do not really hold these negative stereotypes (Cundiff & Vescio, 2016).
One of the negative stereotypes hold against women is that they are less intellectual. Women are therefore underrepresented in jobs that require brilliance. They are especially underrepresented in the science, technology, engineering and mathematics fields (Meyer et al., 2015; Storage et al., 2016). As discussed earlier, this stereotype was already present by six-year-old girls, and thus does not seem to disappear as people get six-year-older. This emphasizes the importance of being aware of gender bias by individuals and organizations, since it can easily go unnoticed (Gupta et al., 2008).
The attractiveness of a job is related to the signals and cues sent by an organization through the job advertisement or recruitment tools. It is also dependent on the inferences a person makes based on these signals or cues and the social identity concerns of the person related to these inferences. Ethnic minorities therefore prefer organizations that seem open to diversity, because this could take away their minority concerns (Highhouse, 2007). This openness can be communicated in different ways, for example by explicitly stating it, or by implicitly mentioning it through wording.
Wording could be a reason to not apply for a job. Wille & Derous (2017) found that women were less likely to apply for a job when the advertisement was worded using adjectives, or trait like wording (‘expect hard workers’), when masculine traits were described, but when behavioral wording (verbs, ‘work hard’) was used, this difference disappeared. The type of wording can thus influence the way a person perceives a job advertisement. It can make people feel threatened when they do not feel like they behave like the stereotype or when the wording raises minority concerns. The linguistic category model, described by Semin (2008) confirms that wording influences how people perceive the text, since the model states that the wording of a profile affects how its evaluated and perceived.
Next to wording, the type of personality traits or job requirements could also have influence on whether a person will apply for a job or not. Bem & Bem (1973) for example found that explicitly referencing to men in the job advertisements, or naming a job like a traditionally male dominated job, decreased the amount of women that would apply for the job. Another study showed that that women are often unintentionally described as more communal, and men as more agentic, which means that
3 written text can (unintentionally) contain a gender bias (Gaucher
et al., 2011).
There are thus three main categories of reasons that can lead to gender bias during the recruitment process. First, it could be that women or men think that the organization will hold negative meta stereotypes against them, which can lead to minority concerns. Secondly, it could be that the wording (unintentionally) attracts or scares off men or women. Finally, it could be that the type of personality traits, described in the advertisements, (unintentionally) attracts or scares off men or women. Evidence for the second and the third reason can be found in the text of a job advertisement, for example in the job requirements. Since this research focuses on textual data, it will focus on the second and the third reason, by trying to identify words and type of words that encourage men and women to like or dislike a job advertisement.
3 METHODOLOGY
The Methodology is divided into 4 sections. First, the data is described, followed up by the preprocessing steps. After that, it is described how a model can be evaluated and finally it is discussed how the model is created.
3.1 The data
The data that is used for this research is collected by an app. In the app companies could fill out their information, such as the name, the age, the average employment age, the gender distribution in the company and a short tag line. Next to that, the companies could create job advertisements by providing information about the job title, the required skills, the job tasks, the job requirements and the location. Job seekers can create a profile by providing information about themselves, such as their graduation year, their gender, their education, their location, their work experience, their skills and their languages. Based on the information about the company, job advertisements and job seekers, the app selects job advertisements that it thinks the person is interested in. The job seeker can choose to like or dislike the advertisement. The company can choose to get in contact with the people that liked their job advertisement.
The data was provided in a .xlsl document. The dataset contains information about 383 unique companies that together created 2588 unique job advertisements. A total of 8920 job seekers signed up for the app, who liked or disliked an advertisement 675787 times in total. The data was obtained in the years 2016 and 2017.
3.2 Preprocessing
Python was used to preprocess the data. The preprocessing consisted of three parts: first, a selection of data was made to train the model on. After that, textual features were created. Finally, an oversampling method was used to deal with the imbalance of the dataset. Each step is described more elaborate below.
3.2.1 Selecting the data
The dataset was loaded into Python by using the Pandas library1.The dataset contained information about a lot of different kind of companies, job seekers and job advertisements. To create a more uniform looking dataset, some data points were removed from the dataset, using Pandas.
First of all, the dataset contained job advertisements in different languages. This research limits itself to English textual features, and therefore all the non-English advertisements were removed from the dataset. The dataset also contained job advertisements for part-time jobs, fulltime jobs and internships. Since internships can be seen as a completely different job category and application process, the advertisements for internships were also removed. Next to that, the dataset contained some job advertisements from non-startup companies. A startup company in this research is seen as a company that exists less than 10 years and has no more than 100 employees. Since the majority of the companies is a startup company, the bigger/non-startup companies were also removed from the dataset. Besides that, all likes and dislikes by users with more than 5 years of experience were removed. This was done since there are a lot of different users, with different ages and a different amount of experience. Since the majority of people in the dataset had less than or equal to 5 years of experience, it was decided to focus on these users. This resulted in that the maximum age of the users was 30. Finally, some job advertisements were not shown to a lot of people, and therefore not a lot of rating data is gathered about these job advertisements. Therefore, all advertisements that were rated less than 85 times were removed from the dataset, since this removed only 5% of the data.
The dataset that was left after removing some of the datapoints thus only contains English part-time or fulltime job advertisements from startup companies, which were rated by job seekers younger than 30 years old, which have less than or equal to 5 years of work experience. This dataset consisted of 226 unique companies, 927 unique job advertisements, 2491 users (users that only liked job advertisements from companies that were not in the dataset anymore were also removed) and 133622 ratings. 1314 of the 2491 users were female, 1171 male. 67370 of the ratings came from women and 65896 from men. 28392 ratings were likes and 105444 ratings were dislikes.
The dataset contains the following interesting textual data: The job requirements: This is the actual job
advertisement text provided by the company. This text is usually longer than a paragraph.
The company tagline: This is a tagline for the company. The tagline mostly consists of one sentence.
The job title: The title of the job mostly consists of one to seven words describing the job.
4
3.2.2 Creating new features
Textual data has to be converted into numerical data before it can be used to train a model. Before textual representations can be created, the textual data has to be cleaned up. First, the HTML was removed from the text using BeautifulSoup2. After that, the punctuation and numbers were removed using regular expressions. The words were also lowercased and stop words were removed using NLTK3. Finally, the words were brought back to their dictionary form using lemmatization and added to a list. Afterwards the following text representations were created:
Bag-of-words (BOW): A bag of words representation is a binary representation that indicates whether a job advertisement contains a certain word. This representation was created using the CountVectorizer from scikit-learn 4 . In this representation every advertisement is represented by a vector, where every column is a word with a value, either 0 or 1 (0 means that the word is not present, 1 means present).
tf-idf: The tf-idf score is a different way to represent textual features in a vector and is calculated using the tf and the idf. The tf stands for the term frequency, which can be seen as the raw count of a term in a job advertisement. The idf stands for the Inverse Document Frequency and contains information about how common or rare a word is when looking at all the job advertisements. The tf-idf can be seen as a weight that is given to a word. This representation was created using the scikit-learn TfidfVectorizer.
Part-of-speech (POS) tagging: POS tagging will be used to differentiate between the different part of speech (for example, noun, verb, adjective etc.). The POS tags were created using the NLTK POS tags on the uncleaned text (before applying the numbers removal, punctuation removal and lemmatization). A numerical vector representation of these POS tags was created using scikit-learn CountVectorizer.
Length: The length of the text was also used as a textual representation.
Since it could also be interesting to look at contiguous sets of words/proceeding words, bigrams and trigrams were also included in the bag-of-words representations and tf-idf scores. All the representations were placed in a Pandas Dataframe to make further processing of the data easier. All these textual representations were created for the job requirements and the company tagline. The job title is only represented by a BOW model and the length.
Next to these textual representations, some variables about the user, company and job (USJ variables) were selected. The following features were used:
The age of the user The gender of the user
2 https://www.crummy.com/software/BeautifulSoup/bs4/doc/ 3 https://www.nltk.org/
4 https://scikit-learn.org/stable/
The years of experience of the user
The education field of the user (IT or not IT) The company age
The average employee age
The percent of females working in the company The size of the company
The company title (can be seen as a textual feature, but is here processed using one hot encoding)
The job function code (this could be one of the following 6 occupational classifications: management; business and financial operations, computer occupations; arts, design and media; sales or office and administrative support positions)
The education field, size of the company and the function code were transformed into numerical features using one-hot-encoding. The variables were scaled using a min-max scaler.
3.2.3 Imbalanced dataset
Class imbalance occurs when there are many more instances of a certain class in the dataset than there are of any other classes. A classification algorithm can thereby have difficulty in correctly classifying the classes that are in the minority, since these classes are underrepresented in the training data. This problem can be fixed using two methods: over-sampling the class that is in the minority (e.g. creating more data) or down-sampling the class that is in the majority (e.g. leaving data out of the model creation). Both methods are effective (Muller & Guido, 2017; Japkowicz, 2000).
Synthetic Minority Over-sampling Technique (SMOTE) is a popular oversampling method. SMOTE generates synthetic examples of the minority class, and thereby creates a more balanced dataset. It thus provides extra data to train a model on, that is similar to the data in the minority class (Chawla et al., 2002).
The dataset used here is imbalanced, since there are way more instances of the negative class (160557) than there are of the positive class (40475). SMOTE was therefore used to create a more balanced dataset.
3.3 Evaluation measures
The most common ways to measure the effectiveness of a model are by using the precision, the recall, the accuracy and the F1-score (Manning et al., 2010; Muller & Guido, 2017). Since the dataset is imbalanced, the accuracy of the model will be high (Muller & Guido, 2017). By using a naïve classifier, an accuracy of 80% can be achieved, when predicting that the user will always dislike the job advertisement (shown in table 1). Since most datapoints are actually a dislike, the amount of True Negatives is high and thereby the accuracy is high. The f1-score can be a better evaluation measure while working with imbalanced datasets, since the f1-score takes the precision and the recall into account and is therefore less sensitive to a high amount of True Negatives.
5
Table 1: Baseline scores
Accuracy Precision Recall F1-score
Random 0.499 0.169 0.500 0.252
Predict 0 0.800 0 0 0
Predict 1 0.169 0.169 1 0.289
3.4 Creating the model
The first step to create a good model is to find which classifier to use. After that, the right combination of features has to be found. Finally, that a model can be explained and analyzed.
3.3.1 Which classifier performs best for predicting whether a person will like or dislike a startup job advertisement?
The aim of the model is to predict whether a person will like or dislike a job advertisement, which makes this a binary classification problem. It is a supervised learning problem since ground-truth labels are available in the dataset. The following methods can be used when creating a binary supervised model (Muller & Guido, 2017):
Logistic Regression (LR): This algorithm is a linear algorithm that uses a sigmoid function to distinguish classes.
Support Vector Machine (SVM): This algorithm tries to find a hyperplane with the biggest margin, where the hyperlane can be seen as the boundary between classes. Gaussian Naïve Bayes (GNB): This algorithm learns
parameters by collecting class specific statistics while looking at each class individually.
K-nearest neighbors (KNN): This algorithm classifies a datapoint into the same class as its neighbors.
Random Forest Classifier (RFM): This algorithm consists of a number of decision trees, where each decision tree can be seen as a hierarchy of if/else questions that lead to a decision.
Gradient Boosting Classifier (GBC): This algorithm uses a similar approach as the Random Forest Classifier, but creates the decision trees in a serial manner, where each new tree tries to fix the mistakes of the last tree. All classifiers were trained and evaluated using a 10 fold cross validation. The performance of the different algorithms was measured and compared by calculating and evaluating the accuracy, precision, recall and f1-score (as discussed in section 3.3). Since determining which classifier to use and which features to use are intertwined (different classifiers could perform better while using different features), the classifiers were trained on a three different kinds variables: the BOW representation, the POS tags representation and based on the UCJ variables. The best performing classifier on these three representations was then used to find the optimal feature combination (method of finding the optimal feature combination is described in the next section).
Three baselines (shown in table 1) are created, to evaluate whether a created model performs better than a naive model. First a baseline was created for a model that randomly predicts a like or dislike. After that, a baseline was created for a model that always predicts a dislike (0). Finally, a baseline for a model that always
predicts a like (1) was created. As predicted, it was found that a high accuracy could be found due to the imbalanced dataset. It was therefore decided that the performance goal of the model was to get a higher f1-score than 0.289.
3.3.2 Which textual features are most important in the prediction model?
Before important features can be extracted from a model, it needs to be decided which features are included in the model. Several different features were created, as discussed in section 3.2.2. By testing different feature combinations, it is tried to find how much the different features contribute to the performance of the model. Features that do not increase the performance, are kept out of the final model to keep the model as compact as possible. The following feature configurations were tested:
1. Only the UCJ variables 2. Only the BOW representation 3. Only the POS tags
4. Only the BOW representation of the title 5. Only the Tf-idf representation
6. UCJ varibles + BOW + POS tags
7. 6 but Tf-idf word representation instead of BOW 8. 6 + company names
9. 6 + BOW representation company tagline
10. 6 + company names + BOW representation company tagline
11. 6 + use of second degree polynomials
These experiments were again evaluated using 10 fold cross validation. The performance of the models were measured and evaluated using the measures described in section 3.3.
After the optimal classifier and optimal feature combination were found, suitable methods to explain the model and extract the important features from the model were selected. These will be discussed in section 4.1.
3.3.3 What are the differences or similarities between men and women?
To identify the differences between men and women, the model is divided into two models. One model is made using the data of the men, and one using the data of the women. For both models suitable methods to explain the models will be used to extract important features, which can then be compared.
4 RESULTS
4.1 Which classifier performs best for predicting
whether a person will like or dislike a startup
job advertisement?
The performance of the different classifiers is displayed in table 3. The results from training the model based on a Bag-of-Words model the POS tags and the UCJ variables all indicated that a Random Forest Classifier performs the bes. It was therefore decided to use the Random Forest Classifier to find the best feature combination.
A random forest can be seen as a black box model, which means that it is opaque what the underlying process exactly does (Sametinger, 1997). Scikit-learn makes it possible to extract the
6 most important features from the model. These features are the
most distinctive in the decision trees. The feature importance does however only show which feature has an influence on the outcome, it does not show how these features influence the outcome. To show how the features influence the outcome the partial dependence can be calculated for each feature.
The partial dependence measures how a target feature is influencing the prediction of a model, and thereby shows the effect an input value has on the model. To calculate the partial dependence first n number of points are selected from all possible values of a target variable. These n points are selected so that the points represent the distribution of the dataset. In an ideal situation, where much computational power is present, n = the total number of different values for the target variable. For every selected datapoint, scikit-learn calculates the outcome of the model, trying each value for each feature/variable and all their combinations, while the target variable remains a constant number, equal to the selected datapoint. By doing this, the partial dependence does not contain the effects of the other variables on the predicted outcome, but only of the target variable. After all outcomes are calculated for all possible values for each feature combination, the average of all these outcomes is calculated. The resulting number is the marginal average of all possible outcomes, and is called the partial dependence. This process is than repeated for all other selected datapoints (Ehrlinker, 2015; Friedman 2001; Hastie et al., 2009).
In classification problems where two classes need to be distinguished from each other, the partial dependence can be seen as the probability of a value resulting in the positive class. This means that if there is a binary variable (like the gender), where 1 has a partial dependence of 0.510, and 0 has a partial dependence of 0.490, the variable contributes more to a positive class than it does to a negative class. By non-binary variables, the partial dependence can be plotted. The value on the Y-axis can represents the partial dependence, which can be seen as the probability that the corresponding x-value will result in a prediction of positive class, based on only the plotted variable (Hastie et al., 2009; C.Molnar, 2019).
Table 3: Performance of the classifiers while using 10 fold cross validation
Model BOW POS tags UCJ variables LR P: 0.555 R: 0.558 F1: 0.557 P: 0.460 R: 0.533 F1: 0.494 P: 0.589 R: 0.553 F1: 0.570 SVM P: 0.556 R: 0.557 F1: 0.556 P: 0.450 R: 0.543 F1: 0.501 P: 0.573 R: 0.529 F1: 0.551 GNB P: 0.486 R: 0.545 F1: 0.514 P: 0.450 R: 0.526 F1: 0.488 P: 0.630 R: 0.550 F1: 0.589 KNN P: 0.552 R: 0.558 F1: 0.555 P: 0.550 R: 0.556 F1: 0.551 P: 0.645 R: 0.658 F1: 0.652 RFC P: 0.612 R: 0.601 F1: 0.606 P: 0.606 R: 0.600 F1: 0.607 P: 0.651 R: 0.706 F1: 0.677 GBC P: 0.555 R: 0.575 F1: 0.565 P: 0.525 R: 0.581 F1: 0.552 P: 0.649 R: 0.700 F1: 0.674
A is accuracy, P is precision score, R is recall score, F1 is the f1 score.
The range of the Y-axis, or the difference between the partial dependence of the binary values, can differ quite a lot. Features with a higher importance have a higher chance in having a large range of values or large difference, since they are more important and distinctive in the random forest. Features with a lower importance often have a smaller range of values on the Y-axis or a smaller difference, since they are less distinctive (Hastie et al., 2009; Friedman 2001).
4.2 Which textual features are most important in
the prediction model?
Table 4: Feature combinations and their performance
Feature combination Precision Recall F1 score 1. Only the UCJ variables 0.651 0.706 0.677 2. Only the BOW representation 0.612 0.601 0.606 3. Only the POS tags 0.606 0.600 0.607 4. Only the BOW representation of the title 0.483 0.591 0.532 5. Only the Tf-idf representation 0.607 0.597 0.603
6. UCJ variables + BOW + POS tags 0.768 0.901 0.839
7. 6 but Tf-idf word representation instead of BOW
0.732 0.881 0.805
8. 6 + company names 0.767 0.901 0.840 9. 6 + BOW representation company tagline 0.768 0.900 0.839 10. 6 + company names + BOW
representation company tagline
0.767 0.901 0.840
11. 6 + use of second degree polynomials 0.766 0.903 0.840
4.2.1 What is the optimal feature configuration?
A summary of the experiments executed to find the best feature combination can be found in table 4. It was found that the UCJ variables, the BOW representation, the POS tags and the BOW representation of the title did not perform really well on their own, but when combined (experiment 6) they did perform decent. Swapping the BOW representation for the tf-idf representation caused a decrease in precision, recall and in the f1-score. Adding the company tagline did not seem to make a difference and adding the name of the company only made a really small difference. To keep the model more compact, it was decided to not add the company name or tagline to the model, since there was only such a small increasement in performance. Adding polynomials did also not increase the performance enough to keep them in the model. Experiment 6, combining the UCJ variables, the BOW representation, the POS tags and the BOW representation of the title, did therefore perform best. The model performs a lot better than the f1-score baseline (0.839 > 0.289) and will be further analyzed.
4.2.2 What are the most important features?
In table 5 the feature importance of the 30 most important features of the model are shown. What is visible here, is that the UCJ variables seem to be the most important for the model, especially the user years of experience. The length of the text and the length of the title are also important. Next to that, the use of personal pronouns, verbs in base form, adverbs, verbs in the past participle, superlative adjectives and predeterminers seem to be important. Finally the absence or presence of the words to,
experience, Dutch, knowledge, content, English, skills, work, team , media, years, strong and creative have an important role in the
7 are important. To find what the effect of these important features
is on the outcome, and thereby how they influence the outcome, the partial dependencies of all these features have to be explored.
4.2.3 What are the partial dependencies between the input variables and the outcome?
The partial dependence plots of the non-binary values are shown in figure 1, and of the binary values in table 6. The years of experience plot shows that people with more than half a year of experience are likely to like a job advertisement, since the probability that the outcome is of the positive class is around 0.8. The model also predicts that when a person has more than 2 years of experience this positive effect becomes weaker since the probability that a person will like the advertisement decreases to 0.5 and later to 0.4. The plot of the age shows a different trend: people between 21 and 22.5 have the most positive relation with the chance that someone likes the advertisement. The older people get, the less likely they seem to like the advertisement. The company age plot shows that companies with an age higher than eight seem to have a higher chance in receiving a like from a person. In the average employee age plot it is visible that when the average employee age is between 20 and 25 or 34 and 40 a company also has a higher chance in receiving a like, than when the average age is between 25 and 34. Furthermore, when the percentage of females is above 0.55, it seems to have a positive effect on whether a person will like the advertisement, as shown in the percentage females plot.
Advertisement lengths follow a different trend. Short advertisements (50 words or less) seem to have a positive effect on whether a person will like the advertisement. When advertisements become longer, this positive effect becomes weaker. When advertisements are longer than 380 words, this effect becomes a lot stronger again. It thus seems like people prefer short advertisement or extensive ones. When a job title is short, people seem to be more likely to like a job advertisement than when a title is longer. This effect could however be explained by the fact that only 3069 of the 133622 ratings, which is less than 1%, were done on a job advertisement which title had the length of one word. It is thus not likely that short job titles really results in a higher like chance, it is probably caused by a lack of data of job advertisements with a title consisting of one word.
The partial dependencies of the binary features are shown in table 6. The values in the 0-column show the effect on the positive class in a situation where the feature has value 0. This thus means that when gender is 0 (male), the effect is 0.506, and when the gender is 1 (female), the effect is 0.469. This effect is equal to the probability that someone will like the advertisement, based on only that feature. Here a male thus seems more likely to like the advertisement, same as people educated in IT (0 = no, 1 = yes).
Besides that, when personal pronouns and verbs in past participle are absent or when verbs (base form), adverbs, or superlative adjectives are present in the text, it has a more positive effect on whether a person will like the advertisement, then the other way around. When the words to, knowledge, content,
English, skills, work, team, media, strong or creative are used in
the job advertisement or when the word designer is used in the job
advertisement title, the model also predicts that this will have a positive influence. Besides that, a job is not in Business and Financial operations (0=no, 1=yes) it seems to have a more positive effect on whether a person will like an advertisement. Finally, the absence of words years, Dutch and experience also seem to have a positive effect on whether a person will like the advertisement.
Table 5: The 30 most important features
Feature Importance Feature type
1. User years of Experience 0.489418 UCJ
2. User age 0.179613 UCJ
3. User gender 0.021870 UCJ
4. Company age 0.018643 UCJ
5. Length of the advertisement 0.016138 Length 6. Job in Business and Financial operations 0.015999 UCJ 7. Is the user educated in IT? 0.014282 UCJ 8. Average age employees company 0.009847 UCJ 9. Average percentage female in company 0.007615 UCJ 10. Prp (Personal Pronoun) 0.006924 POS 11. Vb (Verb in base form) 0.006322 POS 12. Rb (Adverb) 0.005922 POS
13. To 0.005823 BOW
14. Experience 0.005818 BOW
15. Vbn (Verb in past participle) 0.005308 POS
16. Dutch 0.005083 BOW
17. Knowledge 0.004729 BOW
18. ‘Designer’ in Title 0.004031 BOW title
19. Content 0.003976 BOW
20. Jjs (adjective, superlative) 0.003595 POS 21. Length of the title 0.003064 Length
22. English 0.003049 BOW 23. Skills 0.002987 BOW 24. Work 0.002960 BOW 25. Pdt (Predeterminer) 0.002883 POS 26. Team 0.002727 BOW 27. Media 0.002443 BOW 28. Years 0.002353 BOW 29. Strong 0.002241 BOW 30. Creative 0.002130 BOW
8
Table 6: The partial dependencies of the binary features
Feature 0 1 Feature type
Gender 0.506 0.469 UCJ
Job in Business and Financial operations
0.510 0.391 UCJ
Is the user educated in IT? 0.484 0.497 UCJ Prp (Personal Pronoun) 0.498 0.501 POS Vb (Verb in base form) 0.500 0.501 POS Rb (Adverb) 0.499 0.501 POS
To 0.499 0.501 BOW
Experience 0.504 0.499 BOW Vbn (Verb in past participle) 0.502 0.498 POS
Dutch 0.502 0.492 BOW
Knowledge 0.500 0.501 BOW ‘Designer’ in Title 0.495 0.517 BOW title
Content 0.497 0.516 BOW
Jjs (adjective, superlative) 0.499 0.502 POS
English 0.497 0.503 BOW Skills 0.500 0.501 BOW Work 0.498 0.503 BOW Pdt (Predeterminer) 0.497 0.518 POS Team 0.499 0.501 BOW Media 0.499 0.502 BOW Years 0.501 0.497 BOW Strong 0.498 0.503 BOW Creative 0.497 0.501 BOW
4.3 What are the differences and similarities in
important features when comparing men and
women?
4.3.1 Differences and similarities in important features
The 30 most important features of the model created based on the data of the men and the women as displayed in table 7 and 8 respectively. The first 9 features by both men and women are the UCJ variables, but the order of them seems to differ a bit. When comparing the order and the coefficients of the features it can be found that the age of the user seems to be more important for women than for men, while the years of experience are more important in the model for men. Based on the coefficients, the UCJ variables seem to be more important in the model of the women. This automatically means that the words representations and the POS tags are more important for the decisions in the model of the men.
The use of personal pronouns, verb in base form & adverbs and the use of the words to, experience, Dutch, skills, work &
English seems to be important for both men and women, just as
the length of the title. For women the words developer, designer,
junior, data and engineer in the title, and the use of the words creative, look, social, team and love in the requirements seem to
be important, while for men the words degree, sales, design,
program, team, strong, knowledge, JavaScript, ability and product
are important. Finally, superlative adjectives seem to influence the model for women, but do not seem to be important in the model for men.
4.3.2 Differences in the partial dependencies of the important features
There are several differences in the partial dependencies of the features that are most important for men and women as visible in table 9 and 10. When men are educated in IT, the model predicts that this will have a positive effect on whether a male will like the advertisement, while when women are not educated in IT this
seems to have a positive effect on whether a female will like the advertisement. Additionally, a job categorized in business and financial operations has a positive influence on a like for women, but the absence of this variable seems to influence the like chance for men. The presence of the words Dutch, skills and team positively contribute to a like for men and absence of these variables contribute to a like for women.
Table 7: The 30 most important features for men
Feature Importance Feature type
1. User years of Experience 0.546575 UCJ
2. User age 0.128378 UCJ
3. Length of the advertisement 0.019349 Length
4. Company age 0.017347 UCJ
5. Is the user educated in IT? 0.016664 UCJ 6. Job in Business and Financial operations 0.011337 UCJ 7. Average age employees company 0.010980 UCJ 8. Average percentage female in company 0.008533 UCJ 9. Prp (Personal Pronoun) 0.008302 POS 10. Vb (Verb in base form) 0.007707 POS
11. Rb (Adverb) 0.07702 POS
12. To 0.006987 BOW
13. Experience 0.006507 BOW
14. Vbn (Verb in past participle) 0.006170 POS
15. English 0.005766 BOW
16. Product 0.004599 BOW
17. Dutch 0.003706 BOW
18. Length of the title 0.003313 BOW title
19. Skills 0.003200 BOW 20. Work 0.003117 BOW 21. Ability 0.002707 BOW 22. Javascript 0.002624 BOW 23. Strong 0.002605 BOW 24. Knowledge 0.002481 BOW 25. Team 0.002420 POS 26. Business 0.002342 BOW 27. Program 0.002217 BOW 28. Design 0.00211 BOW 29. Sales 0.002195 BOW 30. Degree 0.002174 BOW
Table 8: The 30 most important features for women
Feature Importance Feature type
1. User age 0.411771 UCJ
2. User years of Experience 0.185838 UCJ 3. Job in Business and Financial operations 0.030936 UCJ 4. Is the user educated in IT? 0.028442 UCJ
5. Company age 0.025529 UCJ
6. Length of the advertisement 0.017535 Length 7. Average age employees company 0.013017 UCJ
8. Content 0.009672 BOW
9. Average percentage female in company 0.013017 UCJ 10. Prp (Personal Pronoun) 0.007244 POS 11. Vb (Verb in base form) 0.006700 POS 12. Jjs (adjective, superlative) 0.006642 POS
13. To 0.005992 BOW
14. ‘developer’ in Title 0.005971 BOW title 15. Rb (Adverb) 0.005749 POS 16. ‘designer’ in Title 0.005393 BOW title
17. Experience 0.005365 BOW
18. ‘junior’ in Title 0.005271 BOW title 19. Length of the title 0.005064 Length
20. Dutch 0.004278 POS
21. ‘data’ in Title 0.003969 BOW title
22. Skills 0.003910 BOW
23. Creative 0.003888 BOW
24. Work 0.003843 BOW
25. English 0.003722 POS
26. Look 0.003685 BOW
27. ‘Engineer’ in Title 0.003575 BOW title
28. Social 0.003357 BOW
29. Team 0.003290 BOW
9
Table 9: The partial dependencies of the binary features of men
Feature 0 1 Feature type
Is the user educated in IT? 0.431 0.507 UCJ Job in Business and Financial
operations
0.472 0.504 UCJ
Prp (Personal Pronoun) 0.497 0.500 POS Vb (Verb in base form) 0.498 0.499 POS Rb (Adverb) 0.498 0.499 POS
To 0.497 0.499 BOW
Experience 0.503 0.497 BOW Vbn (Verb in past participle) 0.501 0.498 POS English 0.496 0.500 BOW Product 0.497 0.508 BOW Dutch 0.500 0.494 BOW Skills 0.499 0.498 BOW Work 0.497 0.502 BOW Ability 0.498 0.501 BOW Javascript 0.500 0.485 BOW Strong 0.497 0.502 BOW Knowledge 0.499 0.494 BOW Team 0.498 0.500 BOW Business 0.497 0.506 BOW Program 0.498 0.504 BOW Design 0.497 0.505 BOW Sales 0.498 0.499 BOW Degree 0.497 0.509 BOW
Table 10: The partial dependencies of the binary features of women
Feature 0 1 Feature type
Job in Business and Financial operations
0.506 0.336 UCJ
Is the user educated in IT? 0.498 0.479 UCJ Content 0.488 0.514 BOW Prp (Personal Pronoun) 0.490 0.493 POS Vb (Verb in base form) 0.491 0.494 POS Jjs (adjective, superlative) 0.490 0.494 POS
To 0.491 0.493 BOW
‘developer’ in Title 0.494 0.503 BOW title Rb (Adverb) 0.491 0.493 POS ‘designer’ in Title 0.485 0.510 BOW title Experience 0.497 0.490 BOW ‘junior’ in Title 0.493 0.487 BOW title
Dutch 0.495 0.497 BOW
‘data’ in Title 0.501 0.498 BOW title Skills 0.491 0.493 BOW Creative 0.489 0.503 BOW
Work 0.491 0.495 BOW
English 0.487 0.495 BOW
Look 0.501 0.498 BOW
‘Engineer’ in Title 0.487 0.514 BOW title Social 0.491 0.529 BOW
Team 0.492 0.491 BOW
Love 0.491 0.503 BOW
On the other hand, some similarities can be found between the dependencies. The model predicts that the use of personal pronouns and adverbs have a positive effect on a like for men and women, as well as the use of the words work, to and English. The absence word experience has a positive influence on a like of an advertisement for both men and women.
The dependency plots of the non-binary features (length of the job title, the average employee age, the company age, the length of the job advertisement, the percentage of females and the years of experience) only differ in really small ways and are therefore included in appendix A & B. One small difference worth mentioning is that when looking at the advertisement length in words, women seem to have a more positive effect to a longer job advertisement than men do. The plots of the other non-binary variables follow the same trends as discussed in section 4.2.3. There are no differences between men and women in these other
variables. It for example seems like both men and women prefer to work in a company where at least 55% of the employees are female.
Next to the before-mentioned words, the words content,
creative, social and love seem to have a positive effect on whether
a female will like the add, while the word look seems to have a positive effect on whether a female will dislike the advertisement. Besides that, the model predicts that the absence of the words
junior and data in the job title, and the presence of developer, designer or engineer seem to have a positive effect on whether a
women will like the advertisement.
Furthermore, the words business, program, design, sales,
degree, strong, JavaScript, ability & product, and the use of
adverbs, have a positive influence on whether a male will like the job advertisement, while the words, knowledge & JavaScript, and the use of verbs in past participle, have a positive effect on a dislike.
4.4 How do these differences and similarities
relate to previous research?
There are some differences and similarities between men and women in features that could (partly) be explained by the literature. It has to be noted that these are all speculations. Further research has to be done to confirm or reject these statements.
First of all, the educated in IT feature: IT falls under the category technology, which is associated with brilliance, as described in Storage et al. (2016). It could therefore be the case that men educated in IT are more likely to like an advertisement than women that are educated in IT, since they do not associate themselves with the trait brilliance. This could also be a reason that a job categorized in business and financial operations has a positive influence on whether a woman will like the advertisement, since this category is not directly linked areas associated with brilliance such as science, technology, engineering and mathematics fields. Next to that, this theory could explain why the absence of the word data in the job title has a positive effect on a like for a woman. This theory does however not explain why the words designer and engineer in the title have a positive effect on a like, since these are also technical terms. This theory could also explain why more technical terms, such as
program, design and JavaScript turn up in the important features
for men. Finally, it could explain that the word degree shows up in the model of the men, since a degree can also be related to brilliance.
Next to that, women are perceived as more communal, caring and interpersonally oriented (Gaucher et al., 2011). It is therefore interesting that the words love and social are in the top 30 of important features for women. The word strong, which has a positive effect on a like by man, is a masculine word, and could be linked to the more masculine stereotype of men. Some of the words could thus be linked back to the stereotypes of men and women.
There was also some evidence was found that contradicts with the theory from Wille & Derous (2017) about trait wording and behavioral. Adjectives, that belong to more masculine wording,
10 actually seem to have a positive effect on whether a women will
like the advertisement. The theory stated that women dislike this masculine wording.
The words Dutch, skills, look, content, sales, ability, product,
knowledge and team cannot be linked back to any literature. It
could be that men prefer to work in Dutch environments, and prefer to work in teams. Next to that, it could be that the word skills or the skill requirements in the text scare off women. These are however all speculations, more research has to be done to confirm or reject these statements.
Gee (2018) found that women like to have more information when searching for a job and that providing this additional information can lead to a higher likelihood of applying for the job. This could be the reason that women have a more positive effect to longer job advertisement as shown in the partial dependence plots: they might enjoy having more information in the long job advertisement.
The partial dependence plots of men and women showed similarities in which ages and how much years of experience have the most positive effect on whether a person will like the advertisement. It was found that people with less than 2 years of work experience seemed more likely to like an advertisement. This could be the case because these people are actively searching for a job to gain experience, and therefore they might like more advertisements. As soon as they have more experience, they can get more selective. Besides that, people with an age of 21 to 22.5 probably just finished their degree and are actively searching for a job. As they get older, they get more selective. More research has to be done to confirm of reject these statements.
The partial dependence plots also showed that companies with an age higher than 8 years have a higher chance of receiving a like. These companies could have already made a name and have success, which could lead to more likes. Again, more research has to be done to confirm of reject this statement.
5 CONCLUSION
The main objective of this paper was to find what the differences or similarities are in important features between men and women, when analyzing a model that predicts whether a person will like a startup job advertisement. First, a classifier had to be found, that performed best when predicting whether a person would like or dislike the job advertisement. It was found that a random forest classifier performed best.
After that, the most important features of the model, created with the random forest classifier, were identified. It was found that the variables about the user (age, gender, whether person had education in IT and years of experience), the company (age, average employee age and percentage of female employees) and the job (whether the job is in business and financial operations) are the most important for the model. Next to that the use of personal pronouns, verbs in base form, adverbs, verbs in past participle and adjectives are important. Finally it was found that the words to, experience, Dutch, knowledge, content, English,
skills, work, team, media, years, strong and creative are also
important for the model.
To find the differences or similarities between men and women the model was divided into a model based on the data of the men and a model based on the data of the women. It was found that there were quite some differences and similarities. First of all, the variables about the user, company and job were the most important for men and women, but the order was different. The age seemed to be more important in the model of the women than in the model of the men. Besides that, the years of experience were more important for men. Overall, it was found that the user, company and job variables in general were more important for the women. It was also found that men and women showed similarities in which ages and how much years of experience have the most influence on whether someone likes a job advertisement, and also in what percentage of women they prefer a company to have an how old a company should be.
When analyzing the differences and similarities in textual representations between men and women, it was found that by the men some more technical terms showed up by the most important features, like JavaScript, program and design. By women more communal words showed up like love and social. For women it also seemed more important that certain words were used in the title, while none of 30 most important features for men showed words from the title. In addition to the title being important, women also seem to like longer advertisements than men do. Next to these differences in words and lengths, there were also some differences found in which grammatical elements were important. For women superlative adjectives seemed to be important in the model, but in the 30 most important features for men these did not show up.
Finally, it was tried to relate the differences and similarities to the literature. Some of the aforementioned differences and similarities could be related to the literature, while others could not.
6 FUTURE WORK
This research was done using data from startup companies and users with 5 or less years of experience. It could be interesting to repeat this research using a different dataset, which contains data about longer existing companies and users with more experience to check whether there are differences in findings.
Since this research was mostly focusing on using representations of the textual data, not all variables from the user, company or job were used. It could also be interesting to further analyze the variables that were not used and create a model based that combines the variables that were not already used with the variables used here.
Finally, some statements that could not be confirmed or denied were made about differences and similarities between men and women in section 4.4. For example the statement: “It was found that people with less than 2 years of work experience seemed more likely to like an advertisement. This could be because these people are actively searching for a job to gain experience, and therefore they might like more advertisements.” It could be interesting to investigate if these statements made in section 4.4 actually hold.
11
REFERENCES
Bem, S. L., & Bem, D. J. (1973). Does Sex‐biased Job Advertising “Aid and Abet” Sex Discrimination? 1. Journal of Applied Social Psychology, 3(1), 6-18. Bian, L., Leslie, S. J., & Cimpian, A. (2017). Gender stereotypes about intellectual
ability emerge early and influence children’s interests. Science, 355(6323), 389-391.
Bian, L., Leslie, S. J., Murphy, M. C., & Cimpian, A. (2018). Messages about brilliance undermine women's interest in educational and professional opportunities. Journal of Experimental Social Psychology, 76, 404-420. Childs, R. A. (1990). Gender bias and fairness. Practical Assessment, Research &
Evaluation, 2(3), 3.
Cundiff, J. L., & Vescio, T. K. (2016). Gender stereotypes influence how people explain gender disparities in the workplace. Sex Roles, 75(3-4), 126-138. Ehrlinger, J. (2015). Ggrandomforests: visually exploring a random forest for
regression. arXiv preprint arXiv:1501.07196.
Ellemers, N. (2018). Gender stereotypes. Annual review of psychology, 69, 275-298. European Commission. (2018). 2018 Report on equality between women and men in the EU. Retrieved from https://publications.europa.eu/en/publication-detail/-/publication/950dce57-6222-11e8-ab9c-01aa75ed71a1
Friedman, J. H. (2001). Greedy function approximation: a gradient boosting machine. Annals of statistics, 1189-1232.
Gaucher, D., Friesen, J., & Kay, A. C. (2011). Evidence that gendered wording in job advertisements exists and sustains gender inequality. Journal of personality and social psychology, 101(1), 109.
Gee, L. K. (2018). The more you know: information effects on job application rates in a large field experiment. Management Science.
Gupta, V. K., Turban, D. B., & Bhawe, N. M. (2008). The effect of gender stereotype activation on entrepreneurial intentions. Journal of applied psychology, 93(5), 1053.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). Boosting and additive trees. In The elements of statistical learning (pp. 337-387). Springer, New York, NY. Highhouse, S., Thornbury, E. E., & Little, I. S. (2007). Social-identity functions of
attraction to organizations. Organizational Behavior and Human Decision Processes, 103(1), 134-146.
Japkowicz, N. (2000). Learning from imbalanced data sets: a comparison of various strategies. In AAAI workshop on learning from imbalanced data sets (Vol. 68, pp. 10-15).
Manning, C., Raghavan, P., & Schütze, H. (2010). Introduction to information retrieval. Natural Language Engineering, 16(1), 100-103.
Meyer, M., Cimpian, A., & Leslie, S. J. (2015). Women are underrepresented in fields where success is believed to require brilliance. Frontiers in Psychology, 6, 235.
Molnar, C. (2019) Interpretable Machine Learning, Section 5.1.
Muller, A. C., & Guido, S. (2017). Introduction to machine learning with Python: a guide for data scientists. O'Reilly Media.
Murciano-Goroff, R. (2017). Missing Women in Tech: The Labor Market for Highly Skilled Software Engineers. Wall Street Journal.
Sametinger, J. (1997). Software engineering with reusable components. Springer Science & Business Media.
Semin, G. R. (2008). Language puzzles: A prospective retrospective on the linguistic category model. Journal of Language and Social Psychology, 27(2), 197-209. Storage, D., Horne, Z., Cimpian, A., & Leslie, S. J. (2016). The frequency of
“brilliant” and “genius” in teaching evaluations predicts the representation of women and African Americans across fields. PloS one, 11(3), e0150194. Temple, C. (1993). " What if Beauty Had Been Ugly?" Reading against the Grain of
Gender Bias in Children's Books. Language Arts, 70(2), 89-93.
Walker, H. J., Feild, H. S., Giles, W. F., Bernerth, J. B., & Short, J. C. (2011). So what do you think of the organization? A contextual priming explanation for recruitment web site characteristics as antecedents of job seekers’ organizational image perceptions. Organizational Behavior and Human Decision Processes, 114(2), 165-178.
Wille, L., & Derous, E. (2017). Getting the words right: when wording of job ads affects ethnic minorities’ application decisions. Management Communication Quarterly, 31(4), 533-558.
Wille, L., & Derous, E. (2018). When Job Ads Turn You Down: How Requirements in Job Ads May Stop Instead of Attract Highly Qualified Women. Sex Roles, 79(7-8), 464-475.
12
Appendix A: Partial dependence plots of
13
