Hangovers into Formulas
Frank Woltinge 10175458
Bachelors thesis Psychological Methodology July 2017
For my sister Hannah, who always believed in me.
For my high school math teacher, who never believed in me.
And most of all: for everybody with a fear of math, a negative feeling towards statistics, a sense of boredom towards methodology, or all of the above.
3 Table of Contents:
How to Use This Story 4
Part I: A Story of Statistics 5
Part II: Why Learning About Music is Like Learning about Social Science 21
Part III: Rubber Suit Science 31
Discussion 40
Sources for Discussion 44
Additional Notes Part I 41 Part II 43 Part III 44 Inspirations 48 Acknowledgments 49 Sources 50
How to Use this Story:
This is the paper version of Hangovers into Formulas.
Although originally created as a story with an accompanying app, the whole story is now embedded in the app. In this paper version, screenshots are embedded at places where an interactive component has been added to the app.
The app can be accessed in either a read by day version [click] or a read by night version [click]. Although written with scientific sources as a background or influence, this story has been written to be read by people with just a basic knowledge of methodology, statistics and philosophy of science. For this reason, the whole story has been written in a way that makes it interesting to continue reading. It is also for this reason that there are references to public pages like Wikipedia in the additional notes. Wikipedia used to be quite controversial as a source, however, the platform is growing and with its active forum and community members it is actually a great example of improvement by open source - much like science, only not as closed. The use of Wikipedia for sources also seemed fitting because it can be accessed by anyone (unlike scientific research done with public money) and because the notes that reference Wikipedia either don’t really seem to be about controversial topics or just refer to an introduction to a subject, for which Wikipedia is fine. Where possible, notes that reference papers include an open source link. Other papers or books that have been used as a source in the creation of this story have been referenced in the list of sources.
Additional notes are indicated with superscript, with each part starting at number 1.
5 Part I:
A Story of Statistics
Or what Giants, and a Day at a Fun Fair have to do with Social Science
Your friend from Italy is on exchange in Denmark and she sends you a picture in which she is standing amidst a group of Danish guys. ‘Look at how tall they are!’ she writes in a comment on the picture. You reply with ‘that’s nothing, Dutch guys are at least as tall, probably taller!’1. She asks if you want to bet on it, the loser has to pay the other person’s train ticket to visit. You agree. There is a catch, she is majoring in methodology in social sciences, so she wants to make a proper study out of it. Luckily you have Anna as a friend! She is also familiar with the strange language of statistics and methodology, and you ask her for some assistance.
On a nice spring day you send out a group message to a group of your tallest male friends. Fifteen decide to show up. You have taken Stats101, but it’s been a few years. You remember though, that you have to measure all of them in centimetres and add it all up, then divide it by the number of people in the group to get a mean height in cm. In this way, explains Anna, you equally divide a total amount over a group of people, each getting an equal share. The tallest of your friends is 197cm and the shortest one is 167cm. The total height in centimetres of all your friends if you put them all on top of each other is 2821cm. This gives a mean height of 188.1cm. There is no way those Danish guys are this tall, and you look forward to the nice train journey all the way to your friend. Anna says the next step is to calculate the standard deviation. Just like you calculated the mean and divided total length by the number of people in the group, you will now calculate the total deviation and divide it by the number of people in the group. If the mean height is 188.1 cm and the smallest of your friends is 167cm, this means he deviates from your calculated mean by -21.1 cm. The tallest of your friends is 196cm so he deviates by +7.9 cm. After you have calculated all the deviations, you add them all up… And get zero. Anna looks at you smiling: ‘It makes sense if you think about it, right? If you equally divide a total by a certain number of people, which we did with the mean, then if you take all the deviations from this mean and add them all up, it should be zero. Otherwise, if you were to calculate 15 times your mean (188.1cm) you wouldn’t get the same total of 2821cm again! Thus what we actually do is square the deviations before adding them all up. Then we sum all these differences and equally divide the outcome by the number of people we have in the group again. We’ll take the root of this number to get it back to the same scale as before we squared it.2 Now we have a standard deviation!’
Happily, you send your friend a message that you have a mean and standard deviation, you also send with it a nice picture of your friends standing in line while Anna holds a ruler next to them. Your Italian friend sends back ‘That is not fair, you took a group of tall friends. You should have taken a random sample, this is a convenience sample and a biased one’. You show the message to Anna, who nods a bit. She meant to tell you, but you had already started measuring.
‘And we had a fun day right?’
‘But what is this about a random sample? How do I get this random sample?’ You ask her. She will explain it to you, later, first it’s time for a refreshing drink!
Measuring Friends by their Length
Introducing the Basics
When you and Anna are sitting in a nearby bar later that day, you ask her again. How do you get a random sample and what does she mean about bias?
‘Let us assume you are acting in a film.’ She tells you. ‘And let us assume the scene is about you trying your luck digging in a barrel full of presents, you have to grab one. The present itself won’t be in the shot though. So, you stick your hand in the barrel, grab a present and it turns out to be a teddy bear! The director yells cut! She wants another take at the scene. You put the teddy bear back in the barrel, give it a proper shuffle, put your hand back in there and you grab a hairbrush! The director yells cut again, retakes the scene, you give the barrel a proper shuffle, stick your hand in it and grab a plastic flower this time! Now in this scenario you did the same thing three times. You put your hand in a barrel and grabbed a present. But it was never the same present. And this is important. We will call the barrel of presents the population, and the present you grabbed a sample from this population. This means that if you want to say
something about the entire population - everything in the barrel - you would have to account for all the samples you didn’t grab this time, but that you could have grabbed. Just like a retake of a scene. It turned out differently three times in the same circumstances, which is because of the shuffle you gave the barrel. Now bias comes in if for some reason you had grabbed the plastic flower three times, because it had not mixed well in the barrel.’
7 You look at her in confusion.
Here, try it on my laptop. With one button you draw a random sample, and the other one draws a biased sample. Both barrels contain the same population, meaning that on average, you should draw the same number of plastic flowers. Click the checkbox if you found the plastic flower.
‘Plastic flowers Anna?’
‘Okay, different example. Let’s say you want to establish something about the average height of male students in the city of Amsterdam. Let’s also say that we can take all the male students in Amsterdam and put them in a barrel. Now we bring in a giant and let it grab a handful of them, about fifty students. But we want to say something about all the students in the barrel. So we need to make sure of two things: first that we are grabbing students from the same barrel, in this case, male students in Amsterdam, and second that we don’t have a bigger chance of grabbing the ones on top for instance. Because, if we don’t make sure of those things, we will not be able to use our sample of fifty male students from Amsterdam to establish something about the population of all male students in Amsterdam. If we just grab the ones with big feet, we have a bias in our sample. People with big feet are probably taller. But not everyone in the barrel has big feet. Thus if we just use people with big feet to say something about all the people in our population, we will be making a mistake.
Here, let me show you on my laptop. Look, this is what we did earlier today. Took a few tall guys and measured them, calculated a mean and a standard deviation. And you sent it to your friend. But she called you out for being biased. Now let’s also take a random sample. Do you see the difference in mean height?
Now fortunately, length is easy. A real problem comes in if you try to say something that’s not about something easy to measure like height, but about something hard to measure like “how susceptible you are to positive motivation” for instance.’
‘Why is that harder then?’ you ask while you take a sip of your drink.
‘Well, because if we take something like height I can measure it directly, assuming I have a ruler. But to say something about susceptibility to positive motivation, what would I measure? It’s not as if your hair has 46 different colours that all correspond to a certain quality, so I can just measure the strain of hair with the “motivational colour” and say something like “Your
motivation is about 5cm in dark blue”. I could compare it with somebody else whose motivation is 6cm in dark blue, and conclude that her motivation is 1cm longer than yours. We would look at a hypothetical table that tells us that 1cm corresponds to being prepared to work 2 hours overtime, and thus conclude that the woman with 6cm would be more suited for the job. It would be a nice, direct and concrete measure of your motivation; would also make the social sciences a lot easier… Maybe even a bit boring.’
‘You laugh a bit, at Anna and her weird examples. ‘So how do we measure motivation?’ ‘Indirectly. Wait, I’ll go to the loo, give me a sec to think about it.’
9 When she comes back she has a full pint and takes a big gulp from it. ‘Ah, nothing like beer and statistics. Did you know “Turning Hangovers into Formula’s” is my study motto?’
‘Erhm, right Anna… Interesting… So how do we measure motivation?’
‘Try to measure motivation. Since you are doing something indirect, some people would like to argue that you are estimating3. As close as you can get, but still estimating, not measuring… First off, we want to make sure we’re not just grabbing the people with big feet, right, because of the bias?’
‘Right! But how do we get this random sample that we need?’
‘Give the barrel a stir! Give the giant a huge wooden spoon and let it stir that barrel of male students around before grabbing a sample out of it. You see, what we are trying to accomplish here is just like when we divide a total measurement of length equally by a certain number of people. We want to divide a certain amount of bias in all different kinds of directions equally over a group of people. Because, in most studies we won’t know what kind of quality will bring advantage in what situation so we want to equally divide all the kinds of bias that could exist. And the best way we have found to do this so far, is to take a random sample; meaning that everybody needs to have the same basic chance of being selected for your sample.’
You take a sip of your drink and think about this for a bit. It makes sense; given that you want to say something about a whole group, you need something that equally divides all the little advantages or disadvantages that a sample of people has compared to the population.
‘Okay! Now we can get on with the fun stuff. Do you know those fortune tellers that sometimes sit in the park here? They always tell you they see in the cards that your day will be great and everything will work out. They cost about 2 euros right? Now, what if you wanted to know if you could advise your male friends whether they – in general – should just invest the 2 euro,
because you will indeed have a great day if somebody tells you that. Since your friends are part of the population of Amsterdam, we could do a study where we take the male student-population of Amsterdam as the student-population. Back to our giant! He takes a huge wooden spoon, stirs the students around, and grabs about a hundred students from barrel. Now we have a hundred random people. A nice start, but we will need a way to randomly divide them between conditions. One condition will go to the fortune teller, one will not.’
‘Couldn’t we just divide them in two and put half of them in one, and half in the other?’
‘Nope, we can’t. The reason for this is clear if we go back to the barrel with presents. You need to account for what could have been the situation given the same circumstances. Less abstractly phrased: we need to divide the bias equally. If by coincidence you put a lot of the same bias in one of the two conditions, you have a random sample but will still get a possibly biased conclusion. Even if in this situation it will work out, you have to account for every sample that could have been drawn given the same circumstances. And there is no way you will be able to
divide the bias equally by hand, because you don’t know what the bias is and how much everybody has of it, so you want to divide them between conditions randomly. So you ask a friend who studies maths to create a table with randomly generated numbers for you, or you go on the internet and get one and divide your sample accordingly. Now, we have a random sample of people who are randomly divided between conditions! Okay, let’s say our fortune teller has a tent at a fun fair. At this fun fair there is also a shooting gallery where you can shoot ducks. We ask the woman in the gallery to tally the number of ducks that people who come to the gallery shoot in two minutes. We will also give our participants a participation number that she has to write down so we will know what score of the tally goes with what condition. We will use the tally of ducks shot per minute as a measurement of success, now I think of it we will exclude people from our sample who have fired a gun before, because they will be likely to bring bias. The tricky part is now that we send one condition straight to the fortune teller, they give him two euros, he tells them everything is going to be great because he saw it in the cards, and after this they go to the shooting gallery. But, we need a control condition. Otherwise we don’t know how many ducks people shoot who don’t hear everything is going to be great. Also, we need to make sure that they have the same general experience of going to a fortune-teller in this control condition, although he won’t tell them everything is going to be great. Instead he will tell them nothing because he isn’t there! There will be a note saying “Out for lunch”.’
‘Out for lunch?’
‘Yeah, did you think those guys live off air or moon rays? They need to eat.’ ‘…’
‘Alright, now we have two conditions, one with a manipulation and a measurement of “number of ducks in two minutes” as a score for “how susceptible you are to positive motivation”. ‘But how does the number of ducks shot tell us something about how susceptible you are to positive motivation?’
‘That’s a great question, I will get back to that4. For now the thing that is missing is: what if we compare our two groups and they turn out about the same?’
‘Well, then it would not be a good choice to invest two euros in a fortune-teller?’
‘But how do we know if they were motivated by the fortune-teller in the first place? We need to find that out first. We need a manipulation check! Okay so we have to create two questionnaires both consisting of one question. The first one asks: How motivated do you feel on a scale of 1 – 100 with 1 being “I’ll stay in bed today binge-watching Netflix thank you” and 100 being “I could become president of the world!”? And the second one asks: How positive are you feeling today on a scale of -25 to 25 with -25 being “Give me a building. I’ll jump off it”, zero being “quite alright thank you” and 25 being “today is the best day of my life”? To make it fun we take two different scales and they both try to estimate not exactly but almost the same thing, and…’
11 ‘Anna. Slow down. You lost me. I get that we need these questions, but when will we give them these? Before or after?’
‘Both’
‘Right, and won’t they know what we are trying to figure out? I thought you should try to keep your participants in the study unaware of what you want to measure. It’s called blinding them, right?’5
‘You remembered that part did you? Yeah, that is true. For confusion’s sake we will also give them two questions about something irrelevant but plausible, for instance whether they like fun fairs on a five-point-Likert scale and if they prefer shooting galleries, merry-go-rounds, fortune tellers or the house of mirrors… You know we could simulate this study? We could cook up a nice hypothesis, decide how we are going to analyse this hypothesis, simulate some data and see how it turns up!’
‘Alright. But thirst things first, fancy another pint Anna?’ ‘Count me in.’
Measuring Moustaches and Drawing Squares
Anna’s Intermezzo
As Anna starts typing away you picture the scenario. So we gathered a hundred random men out of the student population of Amsterdam, divided them randomly over two different groups named conditions. With one condition getting the positive advice from the fortune teller and one getting there and seeing he’s out for lunch. We also gave them four questions before and after they went to the fortune teller. Just to see if the fortune teller actually has some influence. We are now going to estimate this influence based on the number of ducks they are each able to shoot. We are also assuming that everybody is about as good at shooting ducks because we excluded people who have shot before.
‘Ok, let’s add a group. What fun is two groups when you can have three right? Three’s a crowd!’ ‘Sure Anna, but what about this third group?’
‘Well, maybe, it’s not so much what a fortune teller tells you, but the whole majesty around him that makes the experience so positive. So we will bring in another group that goes to the fortune
teller, and he just sits there and stares at them, to bring about some confusion. And after standing there in some confusion, most likely they will leave and visit the shooting gallery…’ ‘You like to make your subjects uncomfortable, don’t you?’
‘… Ok so what do we think is going to happen?’
‘Well, let’s say visiting the fortune teller makes a difference. But how are we going to find out if it makes a difference?’
‘Ah the beauty of statistics! So, just now we played around with statistics based on length. Length is great because it has a defined unit, centimetres in this case, and this defined unit has a defined correspondence with the world we live in. In this way we can compare it and define differences.’
‘Right... But if motivation doesn’t have a defined correspondence, how do we try to measure it?’ ‘Get a defined unit! Without a defined unit, we can’t do any calculations. Just like with length, we have means and individual variation again. Something new is the grand mean, but this is just the mean, of the group means.’
‘So, we have a mean per group, and a mean of these means?’
‘Yep. Now, key to understanding statistics is to realize that we do the exact same thing the whole time. We take a point of reference and a unit in which to measure the distance between this point of reference and some other point. We use these within a certain context, which gives us the right to say ‘given this context and this point of reference, this thing we measured is x amounts of our defined unit.’ However, it is very important to keep in mind that our distance as well as our unit only has meaning within its context. Our context contains two parts, there is the shape and its representation. This is easy to distinguish. If I draw the outlines of a paper coffee cup seen from above, it looks like a circle, but it represents a coffee cup. ‘
‘Like the art movement?’6
‘ Yep, or like we draw maps and use a square to represent a church.’
‘But aren’t things always bell-curve shaped in statistics? At least, in stats 101, we always used this bell-shaped curve?’
‘I’ll explain that in a second, for now we need to figure out what we want to find out by measuring all these simulated people, or simulated responses more properly.’
‘We wanted to find out if visiting a fortune teller had any positive influence on your day or something right?’
13 ‘Yes, so in a sense, we are trying to predict the future right? We want to say if it is true that for most people, visiting a fortune teller, or more generally, being told they will have a great day, will have a positive influence on their day.
The way we do this is by taking a sample, and testing this supposition on them. How we actually test this is by discovering if our different groups have the same means before they visited the fortune teller and if they have different means after they visited the fortune teller.
Here comes the importance of our random sample of people that we randomly divided. If we didn’t do this, we couldn’t assume that whatever bias they have, would be equally divided. We couldn’t be sure anymore if we were comparing the same groups with each other. Our context might have been changed in its representation, although we wouldn’t be able to see it, because the shape might still be the same.‘
‘Is this what you meant with: we need to account for everything that could happen, but didn’t happen?’
‘Yes, that is going to get really important now, ok, fortunately we did randomly sample them and randomly divide them, thus we can assume that whatever bias they have is randomly divided. So let’s look at our hypotheses. We assume that the mean of each group, is about the same in the before data. If this is the case, our grand mean and our group means are about the same. We also assume that our experimental condition has a higher mean in the after data, compared to the two other group means. And thus, that our grand mean is quite like our two other means, but not at all like our experimental group mean.’
‘So our point of reference will be the two means?’
‘One part of it will be, yes. The other part will be between the individual variation and the grand mean, and individual variation and the group mean. Because if the grand mean and the group means are about the same, then this distance will also be about the same. In this way we can compare...’
‘Anna, you lost me a bit with all this means and variation talk.’
'No worries, here you can view the data. This is the data that we simulated for motivation before visiting the fortune teller for the three groups. As you can see there is individual variation, a grand mean, group means, and there are the distances to each other. Now, when we are using the grand mean to compare it to our group means, the grand mean becomes our model. We would like to know, if our grand mean can be used to predict scores from these different groups.’
'Ah, predicting the future! So, we are comparing our model, which is the grand mean, with... With what... Our group means?’
‘Yes! And we compare our individual variation with both as well! If these distances in grey are about the same as the distance in blue, this would lead to the supposition that our groups don't differ in some respect. Meaning that in the data before visiting the fortune teller, our groups appear to be no different on motivation which is what we set out to find out about!’
'So how do we actually calculate this?
‘I’ll get to that, for now we will take the ratio between these colours as our point of reference to continue.’
‘And the bell curve? You were going to tell me about that.’
‘Yes, if we go back to our giant and the barrel, let us say we gave the barrel a proper stir every time, but we accidentally grabbed a plastic flower two times. It is an unlikely, but possible event. Now, we want to know how unlikely this event would be. In the same way, our means are quite the same, we want to find out how unlikely it is that we get this kind of difference between means by accident. But just like with the barrel, we thus need to account for every situation that could have happened but did - not necessarily – end up happening. Since we are making a prediction about a population, it is not so much about how unlikely this event is now, as it is about how unlikely this event is ever. What we need is knowledge about positions that our value
15 could take when we would do this again and again and again. We need a distribution. Because this distribution will look slightly different depending on the number of people in the group, which makes sense if we go back to the barrel - if you have only 3 items in the barrel, the
possibility that you grab the same item in two consecutive grabs is bigger than when you have a 100 items in the barrel - and our distribution represents all the different situations that could have happened given these circumstances. Thus to make sure our distribution has the right representation and the right shape, we will consider the number of groups and the number of people.’
‘But we don’t know the shape right?’
‘In this case, we know the shape because Fisher calculated it. In a lot of other cases, in comes the bell curve! No matter what a distribution looks like, if you take a sample of it, take the mean of that sample and you keep doing that, your distribution will look like a bell curve or ‘normal’ or ‘Gaussian.’
‘No matter what it looks like, Anna?’
‘Here, give it a try, I created a weird distribution, you can decide how big a sample you want to take out of this distribution and how often you want to do this, see for yourself.
Remember I said we need a point of reference, a unit and a context consisting of shape and representation? This amazing quality - that in the long run, the means of samples give a normal distribution - gives us the advantage of having one part of our context in the pocket. We know
what it looks like. If we know what it looks like, we can calculate area in it. Now to calculate this we need a point of reference and a unit. For ours, as I said, Fisher did it. For a lot of others if I transform a mean to zero, and consequently transform a unit or standard deviation to being ‘one’ we have 3 out of the 4 things we need to do statistics. If you transform something, the value changes, but the relation stays the same. So we change the value our point of reference takes, but we don’t change the relation it has with its unit. You could say that it takes you five minutes to get somewhere, when you start counting at moment zero in time till five minutes. You could also say it takes you 300 seconds from 12:00. That would be an easy transformation. You could also say that you measure everything by the time it costs you to grow a moustache. In that case getting to school would cost you 1/144th of 1/4th of a moustache. A year would be twelve moustaches, or five beards. That would be a somewhat harder transformation. The point is as long as the relation between the point of reference and the unit stay the same given the context, you can change it to whatever you see fit. In our case, we have the distribution, a point of reference, and a unit. Now we can go out and say something about how unlikely the sameness of our means would be.’
‘How would we do that?’
‘Drawing squares. Because we know that the length times the height of a square will give its area, we just draw squares and add them together. You could measure the room we sit in by centimetres or metres but we could also take boxes with some defined size and say the amount of space in this room compares to adding up 23 boxes. We would calculate the amount of space in a box, multiply it with 23, and we would have the space in this room.’7
‘What about empty space, too small for a box?’
‘Great question! This is exactly how it works with drawing squares. We draw squares that have the same unit of measurement, but hypothetically we can always create a square that is half the distance of our current square. For this reason we don’t actually draw the squares and use the smallest distance of a square you could create in the limit, which is infinitesimal. You can try the concept for yourself with this simulation I made. You see how 5 squares still have a lot of area uncovered, but how 200 squares basically cover all of it.
17 Ok. Now we can calculate the area that has our point of reference, or values more extreme because we use the limit which is...’
‘No way, the p-value?’ ‘Yep, the p value.’
‘Seriously Anna, this wall of text only explains how to get to a p-value?’ ‘Well it mostly explains what a p-value represents.’
’... So what does this tell us about fortune tellers?’
‘Go ahead to the next tab and look at the rest of our simulated fun fair data and its statistics and you tell me.’
Measuring Motivation by Shooting Ducks
Interpreting the Fun Fair data
Here you can look at the different sets of data, and see how it was calculated in three ‘different’ ways. In the stats by hand argument the results were calculated manually. Remember that the action of squaring and taking the root is just done to get around summing up to zero. Compare these manually calculated results with the output of the ANOVA and regression functions and see if you can find the similarities. Keep in mind that when doing statistics it is all about points of
reference and units of distance. If you want to see the formulas used for these calculations, check out the Hall of Formula’s tab.
19 ‘So, according to the statistics the before groups are the same and the after groups are not the same?'
‘Well, we can’t say anything about them being the same. We set out to discover how unlikely it was that they would represent this sameness. In science, we try to refute statements, instead of trying to confirm them. If you state, all dogs are blue, and I found a blue dog, that wouldn’t change much. But if I bring you a red one that would ruin your statement. So we can say that it is unlikely that our groups were different before visiting the fortune teller and that is likely that our groups are different after visiting the fortune teller.’
‘So it’s the same as with the dog ? You bring a blue dog, I can’t say ‘all the dogs are blue’, but I could say, it’s unlikely that these dogs have a different colour.’
‘Yes! And bringing you a red dog, it is quite likely they have a different colour. You have to remember you are doing statements about all kinds of different possibilities that you don’t know. If you could collect all the dogs on the planet here, you could say something about all the dogs on this planet. But since we don’t know all the different possibilities there are or what kind of influence one thing has on another, which is the reason we test it in the first place, we refute the sameness, we don’t confirm it. ‘8
‘Okay, so according to the statistics it is unlikely these groups were different before, and quite likely they were different after visiting a fortune teller. Which means, visiting a fortune teller… makes sense?’
‘Well not entirely, we forgot to do a power analysis and a correction for multiple tests.
‘Anna, seriously… Statistics seems like a never ending story, there is always a plot twist! What does that mean?’
‘That there is a lot more I can tell you about, but it will have to wait until tomorrow at least, for now I’m tired.’
Although statistics may seem like a never ending story, this one has an end and you are now at one third, congratulations! This is the end of the basics. If you got this far, you should be able to understand at least conceptually how Null Hypothesis testing works, which pits two points of reference against each other and tries to state something about how unlikely the finding of that distance in units is.
If you find it confusing how it is about not being different and being different, don’t worry, it is confusing. We will pick up this topic and the conceptual explanation of how to summarize data and statistics with the technique of Meta-Analysis in part II. You have also learned what you actually do when you perform an Analysis of Variance (ANOVA), and maybe spotted some similarities between ANOVA and Regression. The key is in the point of reference. Check out the formula tab to see all the formulas you have learned to understand.
21 Part II
Why Learning about Music is Like Learning
about Social Sciences
Or what Beyoncé stuck in a Giant’s Giant Jukebox has to do with the Art of Meta-Analysis
It was long past midnight when you and Anna finally decided to call it a night. The next morning you get up with a bit of a dry throat. ‘Hangovers in to Formulas’ you think, finally understanding Anna’s study motto while you brush your teeth and look at your still half-sleeping face in the mirror. As you Netflix your way through the first part of the day there is something gnawing on the inside of your brain. You decide to send Anna a message about it. Before you know it, you have agreed to meet this afternoon in a record shop in the city centre, where you can also get coffee. When you arrive Anna is already sitting there with this enormous soy-latte, in which she is drawing sine waves using sugar.
‘Now, in our simulation we found out that, given certain assumptions, we can discover what the chances are of something happening in the long run and thus how unlikely some events are. Which is already mighty cool.’
She takes a sip of her coffee and is left with a soy-milk moustache.
‘But understandably, you felt that I left you hanging with our data results, because I said, we didn’t do a power analysis.’
‘Right, why would you want to do a power analysis? I mean, we found a significant effect, right? Why would it matter?’
‘It has to do with a few things. But these all come together in the amount of uncertainty about an estimated distribution. Now, distributions are a somewhat confusing concept, a great way to think about a distribution is just as a collection; a collection of all the possible chances for a certain event given certain circumstances.’
‘Yeah…Right, just a collection…’ You nod politely to Anna while you take a sip of your tea. She notices your confusion. ‘To make this more comprehensible, let us give our giant from yesterday a friend. This new giant could be a collector of music. But whereas we, the people, collect music on records, she collects music as bands. Instead of a jukebox with records, she has a jukebox filled to the brim with bands. This is the reason why such a lot of musicians are confused or using drugs, they are trying to suppress the dream-like memories of being trapped inside a huge box, in which they have to play songs on command.’
‘…’
‘No really! My friend who plays in a band told me this… Anyway, if our giant could take all music, she would have a complete collection of music. But this complete collection of music would also contain a few sub-collections consisting of only rock bands or just of all-male bands, or only of all-female bands. In this collection of all-female bands, you have Destiny’s Child. Beyoncé’s music would then be a sub-collection of Destiny’s Child, but at the same time it would be a part of the collection of solo-female singers. You get that?’
‘Sure. Sports is a category, but you can divide it into certain sports, and you can divide those sports into teams and the teams into players.’
‘Exactly. Statistics works very similarly. If we were to call these different collection levels – in a non-judgmental hierarchy –different levels to use for analysis, you can have higher level and lower level analysis. Lower level analysis being narrower: so it would be about Beyoncé’s music for instance, not about all female solo singers’ music. Or it would be about Bruce Springsteen, not about all male solo singers. An analysis about all male solo singers would be a higher level. You get that?’
‘Yeah, that is quite clear. They are part of a collection, which is again part of a collection. But what has this got to do with social sciences?’
‘Now if we go back to the mistakes we made in our study of susceptibility for positive motivation, the problem that we didn’t address was the power of our study. And you asked, understandably: but if we found an effect, how is this relevant?’
‘Yeah. I mean, I get that a significant finding does not necessarily imply that it is true, but it is an effect, right? If you find a statistically significant result, clearly you had enough power to do so. I mean, I learned in stats101 that power is the complement of a type II error, which is the
possibility of not finding a result that is there. If we found a result, clearly we didn’t make a type II error, thus we had enough power right? So how can it be a problem?’
‘Well we could have made a type I error, but the real question is if Beyoncé really exists! And if she exists, how big is her part in the total collection of female solo singers. And how well are we able to listen to her voice in a sea of noise.’
‘… What?’
‘In statistical terms: is there actually an effect of susceptibility to positive motivation. Does this exist? And if it exists, how big is it, what is the size of this effect, and how able are we to measure this?’
‘The size of it?’
‘Yeah, let me explain. Let us say you think you are listening to Beyoncé. Only problem is, you don’t know if it is Beyoncé. You just assume it is Beyoncé. Our giant had this problem one time,
23 when she heard an amazing song coming out of her jukebox but was not able to find out who was singing it. It could have been Beyoncé, but because the jukebox was full of all kinds of artists and the volume was quite low, it wasn’t really clear who was singing it. Beyoncé is a really small part of the whole collection of music. Now, our giant had a few options. She could turn the volume on the jukebox up, to amplify the sound in this way. This would basically be the equivalent of taking 300 Beyoncé’s all singing at the same time1. The giant could also move closer to the jukebox and put its ear on the amplifier, in this way making the noise of the surroundings less noticeable, or, find a way to get this same song again and listen.’ ‘You lost me Anna. What has statistics got to do with Beyoncé?’
‘Everything! To go back to our fun fair example: we were talking about susceptibility to positive motivation, right? But this is a sub-category of susceptibility in general, as well as a sub-category of motivation in general. Like Beyoncé who is both part of the collection of female-solo-singers and of the collection of all-female bands. Learning something about susceptibility to positive motivation, could be quite hard. Like Beyoncé, it is probably a small part of its collection, if it exists. But it would probably be easier to learn something about motivation or susceptibility in general. So just like when we want to learn something about female-solo singers, we have the option of not just listening to Beyoncé: we could also listen to Alanis Morissette, Lorde, Edith Piaf, Big Mama Thornton, M.I.A., and Aretha Franklin.’
‘That makes sense. It is probably easier to learn the rules of some sport in general instead of about a certain team. And it is probably easier again to learn about a team than about every player. Or, if it’s not a team sport, a certain scene that people belong to. In general, it is probably easier to classify people into certain groups and know something about those groups because they have certain things in common; while knowing all the individuals is a lot more complicated. So to generalize statements about groups makes sense! I mean…’
‘Unfortunately we are not going to solve the problem of overgeneralization here, but it’s good you grasp the concept. Now, to continue the story of our Giant: something happened. Our giant, who was still looking for Beyoncé, scared her into hiding inside the jukebox. Somewhere inside the whole of the giant’s music collection, Beyoncé is hiding, unable to sing because the giant would hear her. And thus she is not able to keep fulfilling her role as a figure of positive empowerment for women all over the world2… Six generations pass and most people have forgotten about Beyoncé. She has become a myth. Until one night, a scientist who has been reading a lot about the rise of female empowerment wakes up with the idea that there had to have been something like Beyoncé. There should be a logical reason for why, in such a relatively short period of time, women all over the world started to demand equal rights3. From the time when women being expected to take care of the household and children was the norm to the moment that the first woman almost became president of the United States was only sixty or
seventy years! This transition didn’t make logical sense to our scientist without something like a Beyoncé figure. If there had been something like a Beyoncé, surely from wherever she was hiding she would do whatever she could to positively influence the world. Maybe, just maybe, she was singing really softly, just a whisper in the wind. But maybe the sound waves of her whisper would still spread in the world. And maybe, just maybe, the song she was singing was the mythical song that contains a part of Chimamanda Ngozi Adichi’s famous “We Should all be Feminists” speech4. Now if this were all the case – the scientist thought – where would these soundwaves most likely turn up? Obviously at female empowerment conferences! And so it was that he started recording sounds he heard at female empowerment conferences, and sounds at general conferences as a control, and compared these recordings to a spectrogram of
Chimamanda Ngozi Adichi’s famous speech. Because myth told us that this had once also been part of a Beyoncé song. If he could find the soundwaves of this famous speech in recordings he made at female empowerment conferences, but not at general conferences, this would be proof for the existence of Beyoncé! Still trapped inside a giant’s giant jukebox!’
‘… Seriously Anna, what have you been using? Microdosing LSD or something?’ ‘No, this is just me. Welcome to my brain. Do you get the story?’
‘I think I sort of do… Let me get it clear. You explained earlier about the social sciences that they try to measure things indirectly. Like you said, it is not possible to use a ruler and measure certain colour hair strings for motivation, right? Thus we used the questions and number of ducks shot as a measure for susceptibility to positive motivation. In the same way in your story it is not possible to measure these assumed sound waves of Beyoncé directly, by for instance recording her singing. Also because you made a myth out of the existence of Beyoncé, it is not even sure these sound waves or Beyoncé exist.’
‘Right. So if we go back to what we know of our scientist so far, he has some loosely specified idea, or what he calls a theory, about why something like Beyoncé should exist. He also has an idea of how he should measure this assumed existence of Beyoncé. Now there is an enormous gap in this form of reasoning.’
‘Why is that? Isn’t this what the scientific method is all about?’
‘Hold that thought, keep it in the back of your mind. Now, if these sound waves of Beyoncé are here they are quite hard to notice. Also, there is a lot of sound in the world, so the collection of sound waves of Beyoncé is a really, really small, and really specific, sub-collection of the huge collection of all soundwaves in the world. And of this already really specific sub-collection, if it exists, the only part that we can use to find these Beyoncé sound waves is the famous speech of Chimamanda Ngozi Adichi because we have recordings of this. So somewhere inside a lot of things we don’t want to know about, other sounds, noise, somewhere, maybe, there are the sound waves of Beyoncé, and inside these soundwaves is the speech that we can measure.
25 Remember what we could do about noise?’
‘… Amplify, get closer, record it a lot of times or all of the above?’
‘Exactly! Now, our scientist can’t amplify. Because Beyoncé, the source of the sound waves which is still only present according to myth and his thoughts, is stuck in the giants jukebox and, is nowhere to be found. Getting closer is hard for the same reason; if we don’t know the source, we can’t get closer to it. But he can record it a lot!’
‘Wait, is this like grabbing multiple presents out of a barrel again?’
‘Yes! And like our other giant, our scientist needs to make sure he is always using the same barrel of presents to get presents from. Thus he has to make sure he samples the same population of sounds at conferences. Using the same recording device and the same way to graph the sound waves that he recorded every time. Basically, he could try to control for as many of the differences that could occur because of different methods as possible. This way the amount that sounds at normal conferences and the speech of Adichi compare would be the control condition, the amount that the sounds at female empowerment conferences and the speech of Adichi compare would be the experimental effect. Then the difference between his control and experimental condition would be the effect-size! Now we get back to power
analysis; because to understand power, you need to understand the size of an effect. The bigger the effect, the easier you can find it in the noise.’
‘Aha! So if Beyoncé, like our giant, was also a giant, it would be really easy to find her in the huge collection of bands stuck in the jukebox?’
‘Exactly! She would literally stand out like a giant amongst all the other noise. The effect-size would be big, getting enough power to find the effect wouldn’t be hard. But Beyoncé isn’t a giant, so the effect-size our scientist could get at most would also be small. Now in social sciences, most effects are small. A power analysis is a tool that gives the possibility to make a calculation that, based on how small your effect is, how many participants you have, and how important the result of your study is, tells you how likely it is that if this effect is there, you will be able to find it in the long run. Even cooler, if you have any three of these four, it will tell you the fourth one since they are all dependent on each other.’
Here you can see how they interact with each other. Watch how each of them has an impact on the three other ones. You can’t directly manipulate the size of your sample, however it is represented in the standard deviation, which as you might remember is a direct result of the number of people in your sample.
Where is Beyoncé hiding?
Can we replicate our way to a higher power?
‘Anna, this kind of represents our fun fair study, right? This is what we did: do a study, get some result. But you were explaining to me why a power analysis was important. While in both cases, the scientist with Beyoncé and our susceptibility to positive motivation study, we found a
27 significant result. We wouldn’t have found that if our effect-size couldn’t be detected right? Clearly we had enough power to detect the effect then?’
Well, yes and no. Look at the relations in the graph between power, type I, and type II error. Type I error is defined by alpha. Which is conventionally put at .05 but it just has to do with what you would like to call a ‘significant’ result. If the decision you are going to make based on the results of the study has severe consequences, an alpha of .05 would not be a very good convention to have.
‘Yeah Anna, but other people do like conventions.’
‘They sure do… The problem is, because we didn’t do a power analysis, we don’t know what our power was in the first place for finding any effect. But because we also didn’t correct for
multiple tests we might have an enlarged alpha: not .05 but larger. ‘So, what you are saying is, we made a type I error?’
‘We might have done.’
‘Couldn’t we just do a replication of the study then, but this time do a power analysis beforehand?’
‘The problem is in the accumulation of knowledge that science is built upon. For a power analysis, we would need the size of the effect. The most common place to look for the size of an effect, is in other papers studying the same thing. But if we made a type I error, we would have an overestimation of this effect5. If somebody saw our study results and wanted to replicate it sufficiently powered, the study would be powered to our stated effect-size. But the size of our effect was an overestimation to begin with. This makes the new study underpowered again, but without knowing it. We use our alpha to make a decision about how unlikely a result has to be before we refute the null hypothesis, we use it to control our studies. But in this scenario, we lost control without knowing. ‘
‘But you explained to me that we can perform the study a lot of times, right? Record it multiple times in the case of our scientist or grab several presents out of the same barrel. There must be a way to add these small studies together?’
‘Yes, you could do a direct-effect Meta-Analysis. However, better practice would be to just do a large study to begin with. Because in a sense, we are assuming to do the same thing. If I do one study with 300 people, or 10 studies with 30 people and add those results together, I get the same result in this type of Meta-analysis. Remember when I told you about the hierarchy in
collection levels? In fixed-effect we assume that there isn’t a hierarchy. Every draw is from the same collection. The ten different studies all draw from the same barrel. This means all our samples have only one source of noise; you can see what I mean here.
‘Then why would 10 studies of 30 people be the same as one study with three hundred people?’ ‘Because of the way the noise is calculated in a direct-effect Meta-Analysis. We calculate the noise by multiplying the number of studies with the number of participants, and then dividing the noise by the outcome. This means you either do 10*30 or 1*300, which is both 300. Mathematically, we are doing the same thing, and because we are assuming that we are
grabbing from the same barrel, taking samples of effect-sizes from the distribution of the same true effect-size, we can add the noise together.
Now our scientist couldn’t get funding for doing his experiment ten times, so he did it one time and found a significant result which he thought was promising. Now, the next thing he does is get in touch with colleagues from around the world. In a discussion panel of scientists that study the border of the unknown he presents his idea and his results. Our scientist still thinks the missing link between the number of women’s rights in the 1950s and 2016 is Beyoncé.6 But another scientist at the panel totally disagrees with him, it is clearly because of Alanis Morisette. A third scientist says it should be measured not at conferences but at music festivals, to measure it at conferences is ridiculous. Yet another scientist agrees with the missing link part of the story, but thinks it has to do with the pitch of a voice. He suggests it should be measured
29 comparing the pitch of the voices of the women that go to these different conferences. He
assumes that women who feel empowered have a deeper voice, so they more resemble men, which is what empowers them… Being like men.’
‘Isn’t that sexist?’
‘Unfortunately being a scientist is no barrier for being a sexist. The point is that these other scientists believe in somewhat the same loosely defined theory, but they think it should either be measured with regard to a different origin, or in a different setting, or with a different method. Now it would be nonsensical to assume that Beyoncé is still the only thing being measured. But we could assume that all of these studies make an assumption about the rise of female empowerment. Because what our scientists agree on is that all of these elements, pitch of voice, Alanis Morissette, and Beyoncé, have something to do with the rise of female
empowerment. So now, we do assume a hierarchy. We assume that ‘the rise of female
empowerment’ is a collection that is higher and all the studies are lower. If they now summarize all their studies, they have to keep in mind that the measurements of sound waves of Beyoncé are not the exact sound waves of Beyoncé, they contain some noise since she isn’t singing only the song with the speech by Adichi the whole time. In the same way, the exact sound waves of Beyoncé with the speech are not exactly the same thing as the rise of female empowerment, but it is assumed to be part of its collection! This means we have two parts of noise, two levels of uncertainty that we need to account for and that we need to find something about.
‘You mean, if we weren’t able to measure the length of male students in Amsterdam but instead could measure the length of males between the ages of 18 and 25 who work in bars around the
university and use that? The mean length of the males in bars we measured, wouldn’t be precisely the mean length of all males that work in bars. And that mean length wouldn’t be precisely the mean length of male students in Amsterdam.’
‘Exactly! Only once again length has an advantage… Now let’s assume they took into account the two levels of noise, summed up all their found effects and got a result of p = 0.05532.’
‘So what does that mean?’
‘That is a great question… what does that mean?’
As Anna says that, she puts on her coat and grins, wiping her soymilk moustache away with the back of her hand. Leaving you, sitting there with a mug of now ice-cold tea and a lot of
confusion.
Congratulations! Two-thirds-mark of the story! You should now have a feeling for what a
distribution is, also you have by now got a conceptual grasp of what power is, why it is important, and what you conceptually do when performing a meta-analysis. You should also be able to see the clear difference between a fixed-effect analysis and a random-effects one, the key is already in that one is singular and the other is plural.
The story will continue and end in part III, in which Anna will hopefully stop being mysterious because it is getting quite annoying. However, it has some meaning.
31 Part III
Rubber suit science
Or why social science should look at the ceiling in the hall of mirrors that it’s lost in
After spending the afternoon in the coffee place talking about Collections, Giants, Beyoncé, Power and finally Meta-Analysis Anna left you in a lot of confusion when you asked her what the results meant. The next day at dawn she sends you a message to ask if you’re ready for the last part of the journey through what she refers as her story of statistics. She invites you to this abandoned fun fair site at the edge of town around sundown. When you arrive she is wearing a suit.
‘Anna, what is this? Wearing a suit, an abandoned funfair site, are we in a film-noir? Should I be wearing a long trench coat?’
‘No, but you should wear a suit. I brought one for you too. We are going to visit this place because it is a representation of the state of social science. The only problem being, we forgot that we got lost in this place. Thousands upon thousands of researchers are stuck in this place and I think I understand why. I can’t get them out, they have to do that themselves. But I can get you in –first of all – and then get you out. Which at least shows a path out of this classic movie location. So, here is the suit, get dressed.’
The suit fits well enough for you to be comfortable, and you join Anna. As she opens the gate and walks in, it starts raining a bit. You look at her with some amusement and confusion.
‘That is the right attitude you will need for this place. Because it is amusing, you can have great fun in here and discover wonderful things that surprise you and make you kind to the world we live in. The confusion will come from the realisation of all the worldviews you have and what influence they have. Just one last thing, like Dolores1, I won’t be the damsel in distress in this trope story.’
As she finishes her sentence, lightning flashes above the sea in the distance, followed by a huge rumble. Anna flicks on a flashlight and puts up an umbrella, she hands you one as well. You follow her footsteps until you arrive at a large building, Anna wants to step in, but you grab her by the shoulder.
‘Anna, seriously. What is this all about? Why do we have all these movie tropes here? And don’t turn around now in a Walter Neff kind of fashion. You were explaining science to me, right? I could still put up with the Beyoncé and giants thing, but at least it had some subtlety to it, there
was a clear cut between the story you were telling me, and being part of some story! But this is just going overboard! Just explain to me what the meaning of all this is!’
She turns around in a Walter Neff kind of fashion ‘Precisely!’ ‘What do you mean?’
‘What do you mean?’
‘Not this again. Don’t play games Anna. What do all these tropes mean? What does it all have to do with meta-analysis or social sciences in general? And don’t say everything!’
‘I can’t explain it, but I can show it to you, we need this overboard kind of storytelling to get to the bottom of the story of statistics, the bottom of understanding social science. So let’s get us out of this rain and into this hall of mirrors.’
Once inside, Anna walks through a door and the lights flick on. Inside it is indeed dry and somewhat warm even. The lights give a dim glow and the sound system starts playing some soothing soul music.
‘There, the surrounding needs to be relaxing, like a day at the beach or a picnic at the park. Otherwise it wouldn’t make any sense that all these researchers don’t see a reason for getting out of here. Outside of this funfair it is a confusing place.’ Satisfied she looks around. ‘Do you remember what we ended with in the coffee place?’
‘Yeah, you simulated some graphs that represented different studies for the rise of female empowerment.’
‘Right, and how did we come up with female empowerment?’
‘Well, there was this scientist that figured there had to be a reason for the rise of female empowerment, and so he figured it was Beyoncé hiding in a giants giant jukebox.’
‘Right. And then he went to his panel and they did all kind of studies, and they did a meta-analysis. And what did we find?’
‘We found some result, but you walked out on me when I asked you what it meant.’ ‘What do you think it means?’
‘That it is not very likely that Beyoncé is hiding in a giants giant jukebox? ‘ ‘Did you find that very likely to begin with?’
‘Well. No. For starters, there are no giants, Beyoncé is not missing and you made the whole story up.’
‘How about if I told you that it wasn’t Beyoncé singing while hiding in a jukebox, but Beyoncé singing on the radio or a radio channel?’
‘Well, that is obviously way more likely.’ ‘And what do the results mean in that case?’
33 empowerment?’
Do you now see what the problem was when you were asking me what the results meant?’ ‘Yeah, I guess that depends on the theory doesn’t it?’
‘Exactly. I told you before about how in statistics we use a point of reference and a unit to get a certain distance within a context that has a shape and a representation. The only thing I just changed was the representation, and it changed your whole explanation of our calculated results.’
‘Does that matter?’
‘If you want to know what your results actually mean, it does, but not if you just want them to mean what you want them to mean.’
‘You are speaking in puzzles again Anna.’
‘I’ll show you. Close your eyes and spin in circles till you hear my voice again.’
You do as she says. Just as you start to get a little dizzy, you hear Anna’s voice come from a microphone in the ceiling.
‘Alright. Come and find me… Also, last movie homage, promised. But you need to see what I mean to get it. At this moment, I am the point of reference. The good thing about me is, I am really here and there is actually a way to find me. You have to figure out what the distance is while you take steps one unit at the time. But, although in shape quite common, this place’s representation is designed to work against you when you try to follow your basic senses and rules of thumb. We should never forget how Feynman2 said that it should be first principle that you must not fool yourself and you’re the easiest person to fool.’
‘Sure Anna’. You mumble, getting where she is going. ‘You shouldn’t just trust a person in a lab coat because he wears a lab coat. Neither trust a dentist when he has some advice about how to repair your car, in the same way you shouldn’t trust your car repair woman about your teeth. They might be right, but it’s not their area of expertise so it’s not really likely.’
‘…However’ she continues ‘knowingly or unknowingly, social scientists like to wander around and be fooled; with all the consequences.’
You hear Anna’s voice now, not through the microphone in the ceiling, but somewhere quite near you. Suddenly you see her a few meters in front of you ‘Found ya!’ You yell as you approach her with swift steps… and hit your head against a mirror.
She comes towards you from another corner ‘You okay?’ ‘Yeah, I got fooled by the mirror.’
‘I know, I was waiting for that. You see, you just literally got hit in the face by reality. You really believed that you could walk through this mirror, but you found out, you couldn’t.’
‘Gee, thanks for pointing that out Anna.’
‘The thing is, you were wrong, while you believed you were right. And this small fact, is something that you found out by testing your assumption – walking into the mirror – and getting the results that you were wrong. But it was plausible you were right. After all, you probably heard my voice starting to be closer, you saw with your eyes that you could walk in that direction, you know you can walk, this all created a certainty that made you walk in that direction… still you were dead wrong. You were fooled.’
‘Yes Anna, I was fooled, stop gloating.’
‘Sorry. It is just, how different would it have been if you were able to believe that I was actually at that place you thought I was, and that would make me be there.’
‘To be honest Anna, I’m really lost in what you are saying or what you are trying to illustrate. I hurt my head, I’m quite confused, I don’t understand any of it anymore.’
‘We have to go back to a few days ago, how did we end up together in this spiral of stories?’ ‘Because my friend in Denmark said Dutch guys were shorter than Danes?’
‘Right! And when you collected a group of tall friends, why did you get called out for that?’ ‘Because it was biased. You can’t use it to say something about Dutch guys in general.’
‘That’s it! With science and in the practice of scientific research we try to tell something about the world surrounding us. It is about truth, not about being right.’
‘What do you mean by that?’
‘That to convince you of being right just took me some smoke and mirrors. You thought I was standing somewhere, but actually I wasn’t. But, to actually be right, your assumption would have had to fit in with all other relevant parts of this world as we know it. Not just with your belief that you could walk in that direction, but with how I can’t at the same time be where I was actually standing and where you thought I was standing. Thus to be right your assumption had to fit in with all basic scientific assumptions, theories and facts we have gathered so far, and even the ones we don’t have yet. So, how do we know that this is the case?’
‘We test it?’
‘Ok, then what do the results in our Beyoncé study mean?’
‘Well obviously the fact that there are no giants as far as we know and that Beyoncé isn’t missing, but that radios do exist and that Beyoncé can be heard on the radio, makes that the second one fits better within the basic rules of physics.’
‘Alright, then what about our study on susceptibility to positive motivation? Let’s say we couldn’t get any funding to do our study again. But another scientist could get some funding but didn’t want to do the same study, because that wouldn’t be new enough. But he did want to study ‘susceptibility to positive motivation’. He decided he would measure the size of the tip people gave in restaurants when they were either surrounded by posters with motivational
35 quotes or when they weren’t. Now, is that also about susceptibility to positive motivation?’ ‘Well, it might be.’
‘How would we know?’
‘Isn’t this what statistics does? I mean, we could measure the distance between them.’ ‘So what do we need to do so?’
‘Come on Anna, don’t be annoying. We need a point of reference, a unit and a context, you’ve repeated it to death.’
‘So what is the context?’
‘The shape and the representation.’ ‘And what is the representation?’
‘The susceptibility to positive motivation?’
‘So how would we know if both studies, both represent susceptibility to positive motivation?’ ‘…’
´Let us start walking out of here.´
It’s not raining outside anymore and the abandoned funfair is just silent. There are no squeaking sounds or wooden panels or frogs outside. You just see some abandoned buildings on an
abandoned site. Night has fallen but the sky is quite bright. ‘Last place to visit: the fun house.’
As you enter, Anna flicks a switch that turns on a light. In the corner you see a rodent running towards the wall; other than hearing its soft paws on the wooden board, the place is silent. Although the lights are on, the different parts of the fun house haven’t been turned on. Not much fun going on.
‘Okay Anna, then what does actually define our representation?’
‘Our representation is defined by a story. A story of beautifully constructed rhetoric, designed to be convincing of what it states, but unable to discover truth.’
‘Wait, why would it be unable to discover truth?’
‘Go stand in front of one of those mirrors against the back wall. What do you see?’ ‘I look skinny.’
‘Does your suit still fit?’ ‘Well obviously.’
‘Go stand in front of the other one. How about now?’ ‘Now I look big.’
‘Anna, of course it does right? I still have the same body, the mirror doesn’t change that.’ ‘You see, the interesting thing is, you are absolutely right. That suit won’t stretch like rubber when looking in to a mirror. What changes is what you are seeing, you are being fooled. The problem is, in social science we do create our theories from something like rubber: words, sentences. Words are symbols, but they are not very narrowly defined. When you look for me in a hall of mirrors, there is a place I’m actually standing, while where you think I’m standing can be true or false. And you can find out by testing it. But words like susceptibility or positive or motivation, what do they mean? What you mean with ‘susceptibility’ or ‘positive’ or
‘motivation’, might not be what I mean with it. What I meant with susceptibility to positive motivation earlier, might not be what I mean with it now. The problem is, our results have no meaning without their representation, and they are represented by a rubber suit. And if they are represented by a rubber suit, when are they true or false?’
‘Didn’t you tell me about the art of meta-analysis? Accumulating results and then you can see whether units in distance are about the same?’
‘I did, but…’
‘But, that doesn’t tell us anything about whether they actually belong to the same context? For all we know, they could have nothing to do with each other? But it’s easy then right Anna? If it is not true, science will tell. There will be no results in the long term for susceptibility to positive motivation, given that we have enough power, control for type I error and inflation of our alpha, randomize our conditions and take a random sample!’
‘Look in that mirror again, what do you see?’
‘Anna, what do you mean? I still look big, but I am not, nothing has changed.’
‘You see, the harsh thing about social reality is, you might not have changed physically, but if I would give you a mirror like this all of the time, your perception of yourself would change… Or maybe even worse is the other way around. You see, the convincing part is still easy, that is just constructing rhetoric in a way that it sounds or appears to be plausible. It is key to making something a social truth. The best way to look at how to create social truths is to look at history. Not so very long ago it was a social fact that women were worse at maths than men3. And this made ‘sense’, right? I mean, women were supposed to be softer and they reasoned with emotion whereas maths is abstract and there is no emotion involved4. And because you knew women were worse at maths, why - as a male professor or teacher - should you give equal attention to men and women when they had trouble to understand maths? There was room for
improvement for one of them, and it wasn’t the one that was going to raise the kids. Why would you waste the energy? Why would we try to teach somebody something that they would
obviously never get? That is the problem of social sciences. Although the claim you make is dead wrong, as long as enough people believe it, you make it true in the social reality because there is
