Visualizations of the Bayes factor

Visualizations of the Bayes Factor

June 24th _{, 2016}

Bachelorproject Psychology – Psychological Methodology Student: Bram Timmers (10174230)

Supervisor: Mr. R. (Ravi) Selker PhD Second Supervisor:

Academic Year: 2015-2016 Semester 2, Block 3

Abstract

In this study, we tested whether three visualization techniques (e.g. pie charts, stacked bar charts and dot charts) would help people in establishing the posterior probability given a certain Bayes factor opposed to numerical or textual displays. We tested this by means of an online experiment, where participants were asked to provide a posterior probability given a Bayes factor in a hypothetical situation. In total 354 participants were analyzed by calculating a Brier score and comparing the four conditions using Bayesian t-tests. The results of this study suggests that the interpretation of the visual representations influences the estimated posterior probability in a positive way. Concluding, we argue that visualization techniques can help people interpreting the Bayes factor in a more correct and consistent manner.

1.0. Introduction

In psychological research experimental psychologists often use statistical methods and procedures for making inferences from experimental data (Dienes, 2011). They devise hypotheses and deduce their implications from testing data with empirical material (e.g., in the form of a survey, experiment or observation). To evaluate the data, statistical interference is often derived by accepting or rejecting opposing hypotheses by means of Null Hypothesis

Testing - calculating p-values and deriving their meaning from a demarcation criteria (i.e., α =

0.05) -.

NHTs has been widely used in psychological research mostly due to its easiness to use and widely available software to conduct NHTs (Dienes et al. , 2011). Recently, the method however has come under pressure by academic researchers (Johnson, 1991; Dienes, 2011; OSF, 2015; Wetzels et al. 2011), as it holds some serious drawbacks in interpreting its results. An alternative method often proposed to defend against some of these drawbacks is Bayesian analysis.

Bayesian analysis and statistics was introduced by Jeffreys (1961) as an initial

approach for getting scientific interference. In contrast to NHTs, where we can only reject or accept theories or hypotheses, Bayesian analysis is used to determine the posterior probability of a theory being correct or true given the data. First, we allocate a prior probability to our two hypotheses of them being true, which we can denote as P(H1) and P(H2). Then, given the

collected data, we can transform our prior probabilities to a posterior probability of the hypothesis being correct. This is done through multiplying our prior probabilities with the likelihood of the data occurring under both hypotheses, also denoted as the Bayes Factor (Kass & Raftery 1995).

In a general form the Bayes formula looks as follows: 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 𝑝𝑝𝑜𝑜𝑜𝑜𝑝𝑝 𝑝𝑝𝑟𝑟𝑝𝑝𝑝𝑝𝑝𝑝 = 𝑏𝑏𝑟𝑟𝑏𝑏𝑝𝑝𝑝𝑝 𝑓𝑓𝑟𝑟𝑓𝑓𝑝𝑝𝑝𝑝𝑝𝑝 ∗ 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 𝑝𝑝𝑜𝑜𝑜𝑜𝑝𝑝 𝑝𝑝𝑟𝑟𝑝𝑝𝑝𝑝𝑝𝑝 =𝑝𝑝 ( 𝐻𝐻1|𝐷𝐷) 𝑝𝑝 ( 𝐻𝐻2|𝐷𝐷)= 𝑝𝑝(D|H1) 𝑝𝑝(D|𝐻𝐻2)× 𝑝𝑝(H1) 𝑝𝑝(𝐻𝐻2) (1)

The benefits of doing Bayesian analysis instead of NHTs is that it counters some of its shortcomings. First of all, there is no stopping rule needed to do Bayesian analysis. In contrast to NHTs the researcher does not have to specify a condition under which one would stop collecting data (Dienes et al., 2011). The researcher can check each posterior probability after collecting a datum and does not have to determine what more extreme data are for the p-value as required by NHTs (Moreijn, 2014). The reason for this is that when the null hypothesis is true and the number of events or data increases, the Bayes factor is just driven to zero (Dienes, 2011). Bayesian analysis thus also does not require large sizes to interpret an effect (Berger, 2006). Furthermore, you can add old findings into your formula as the prior odds for future analysis. Another argument for Bayesian analysis is that it does not put the same constraints to the null hypothesis as with NHTs. It is theoretically possible to find evidence for the null hypothesis in Bayesian analysis (e.g. with an Bayes factor of less than 1).

Bayesian analysis, therefore, does not violate the likelihood principle as NHTs does (Wetzels et al., 2011).

An argument for the use of Bayesian analysis in psychology is thus easily made. Although Bayesian Analysis can sometimes get mathematically complex and one can argue about the objectivity of establishing prior odds, it functions as an alternative for NHTs. Primarily research of comparing p-values with Bayes factor done by Wetzels et al. (2011) showed that although the results mostly agree with each other, p-values between 0.01 and 0.05 only showed Bayes factors of below three. One can argue whether data three times as likely is enough evidence for the alternative hypothesis.

Nevertheless, a problem with Bayesian analysis is how to interpret the Bayes factor. In short, Bayes factors are just numbers that quantify the relative evidence for two competing hypotheses. Where a Bayes factor of one implies ambiguity of the truth of both hypotheses and a Bayes factor of two implies twice the likelihood of the alternative hypothesis, it does not give a clear answer to the truthfulness of both hypotheses (Dienes, 2011). Interpretation of the meaning of a Bayes factor can therefore be arbitrary and mostly depends on what the researchers thinks is a reasonable likelihood. Unless the researcher had specified a specific prior odd for both hypotheses beforehand, his degree of belief given a specific Bayes factor is quite similar to an odds ratio for a betting in a casino (Romeijn, 2014). The degree of belief of a hypothesis would in this way be the price of a betting that he is willing to accept for the occurrence of the expected event. Unfortunately this makes interpretation of the Bayes factor thus highly subjective, as this often depends on the researchers or betters own preliminary expectations about both hypothesis.

One of the shortcomings in making a more objective interpretation given a odds ratio is that people do not to quantify the odds ratio into a prevalence of the outcome of interest (Davis et al., 2008). Odds ratios simply tell us the number of events with the given outcome. It however does not tell us the total cases of the event occurring in the whole sample

(Holcomb, Tinnakorn, Douglas, & Burgdorf, 2001). Although one instinctively would think about an odds ratio as a risk ratio, it is not. When odds ratios are thus seen as a relative risk (e.g. as the chance of the occurrence of the event divided by the chance of the occurrence of no event), one often overstates its effect size. This overstating seems to even divert more when the odds are very large or small. A same effect could exist with interpreting Bayes factors as relative to the truthfulness of one of the hypotheses. In interpreting Bayes factor lower than 1 the researchers would overstate the probability of H1 given the data and would

Another problem with interpreting Bayes factors is that people are often conservatism in interpreting additional data in Bayesian analysis (Ducharme & Peterson, 1968). After adding more data, one tend to adjust their posterior in the right direction, but when the Bayes factor seems to be extreme, people overstate or understate the optimal posterior odds in the Bayesian formula. The reason for this understating of the posterior odds is probably due to people misperceiving the added value that the extreme Bayes factor adds to their prior beliefs. People anchor themselves to their initial beliefs about the truthfulness of both hypothesis and are unwilling to change them fully given a certain odds ratio (Corner, Harris & Hahn, 2010). Helping researchers with interpreting the Bayes factor has never truly been researched. However, research on helping people interpreting odds ratios in casino’s or insurance

companies have stressed the importance of visual representations (Lipkus, 1998). Visual displays of the odds ratios, such as graphs, have shown that it helps people in comprehending the associate risk ratios in a consistent manner in contrast to numerical or textual displays. Furthermore graphical visualizations showed that it can ease the communicating of

information between individuals and help them understand the information quickly (Gresh et al., 2012). For example, Hollands and Spence (1998) found that visualization representations of part-to-whole relationships, such as pie and bar charts, helped people in correctly

interpreting an odds ratio relative to its proportion to the whole sample. They suggested that because these visual charts possessed a physical object representing the whole, people could easier make a proportional estimate. Moreover, they found that pie charts were superior to bar charts in interpreting the proportion, mainly because subjects compared the selected

component to the remainder instead of the whole with the bar chart. In accordance with these findings, a study done by Ancker, Senathirajah, Kukafka and Starren (2006) showed that charts displaying part-to-whole relationships helped people with interpreting odds ratios as a risk ratio.

Bayes factors are fairly similar to odds ratios. In this paper we, therefore, want to stress the importance of visualization representations in interpreting the Bayes factor. In a similar way to interpreting odds ratios, graphical displays could help people and researchers in interpreting the Bayes factor in a more consistent manner.

In this study, We therefore presented three different visualization techniques that can help people with correctly interpreting Bayes factors (i.e. pie charts, stacked bar charts and a dot chart) and overcome some of its problems of interpretation. Participants were recruited through the Amazon Mturk crowdsource platform and randomly assigned to a numerical condition or one of the three visualization conditions. Participants were then presented with different odd ratios and requested to provide the posterior probability of one hypothesis relative to the other. We hypothesize that the visualization techniques could help people in correctly estimating the posterior probability of the hypotheses. In addition, we expected the pie chart to be superior to the other charts, because people correctly interpret “the pie” as a whole sample space opposed to the stacked bar and dot chart.

Finally, as a relative risk ratio would not tell us anything about the belief that the participant holds about the truthfulness of one of the hypothesis, we also asked the

participants to rate their strength of evidence on an ordinal scale to see whether there were any individual differences between the representations in the interpreting of the Bayes factor.

2.0 Methods

2.1 Design

The purpose of the experiment in this study was to test whether three visualization techniques help people in interpreting the Bayes factor correctly compared to numerical representations. Most of our visualizations were based on the academic literature about the visualizing of risks and odds ratios (Lipkus et al., 1998; Gresh et al., 2012). Before conducting our experiment a pilot survey was run among acquaintances to see which visualizations techniques we wanted to test. In this pilot survey four visualization techniques were considered as they mostly are

used to depict proportional probabilities: the pie chart, bar chart, stacked bar chart and a dot chart. After careful consideration we chose to use pie charts, stacked bar charts and dot charts. Both the pilot and final survey were made using the online questionnaire tool Qualtrics. Qualtrics is an internet-mediated platform that offers an easy way of making surveys and providing them through the internet.

Participants were randomly assigned to one of the four conditions. Each condition consisted of 35 trails, consisting of odd ratios ranging from 1-50 for both hypotheses. In our research design, the selected visualizations and Bayes factor were primarily the independent variables. Our key dependent variable was the posterior probability that the participants provided given the specific odds ratio.

2.2 Participants

Participants were recruited through Mturk, an online platform of Amazon where people can complete small online jobs for a small remuneration. In total, 453 participants from the United States were recruited and filled in the online survey. Participants were payed a small amount upon completion of the survey.

2.3 Procedure

After some demographic questions and asking whether any of the participants suffered from color vision deficiency, participants were randomly assigned to one of the four conditions. Before starting with the online survey, people where shown an hypothetical situation, where on an alien planet researchers were testing some medicines and found an certain likelihood of one being better than the other. The alien researchers had no information beforehand, which illustrated the uniformity of the prior odds. Participants were asked whether they understood the hypothetical situation. After being assigned to one of the four conditions, people were

shown an example item of the visualization were both hypothesis were equally as likely to cure the disease. The hypothetical data were a representation of the support in favor of one of the medicines. Participants were asked to provide the posterior probability of the first drug curing the disease relative to the other. After providing the answer, the correct answer of 50 was shown. This was done to see whether they understood the basic interpretation of the Bayes factor. Prior to the experiment, a set of 35 odds ratios was selected, ranging from 1-50 for both hypotheses. Similar to the example item, participants were asked to provide a posterior probability of the first medicine being better than the other given the odds ratio.

After completion of the survey, participants were also asked whether they used any mathematical formula to derive to their answer. This was done to see whether they based their answer only on the visualization or numerical representation shown.

3.0 Results

3.1 The Brier Score

For each participant we used the Brier score to measure their performance on the items. The Brier score is a score function that measures the accuracy of probabilistic predictions (Brier, 1950). It quantifies the difference between what the participant estimated as a relative risk and the true relative risk ratios associated with the given Bayes factors. It does so by taking the mean squared difference between all their 35 estimates. The formula for the brier score is:

(2) 𝐵𝐵𝐵𝐵 = 1

𝑁𝑁 ∑𝑁𝑁𝑡𝑡=1 (𝑓𝑓𝑡𝑡− 𝑝𝑝𝑡𝑡)2

Where N is the number of items, f t the estimated relative risk and ot the true relative risk ratio.

In this way, a Brier score of 0 would imply perfect fit between the actual estimate and the true estimate.

Eventually, 453 participants completed the questionnaire. Twelve participants were excluded from further analysis, because they showed duplicated IP-addresses. Furthermore, 24

participants were excluded because they indicated that they did not originate from the United States or had English as their first language.

We also excluded participants if they gave an answer not between 25% and 75% to the item that represented a Bayes factor of 1. The reason for this is that the correct answer (i.e. 50%) for this item was also given in the example item, so an answer far off 50% would suggest that the participant did not pay any attention to the experiment when filling in the questionnaire. This further resulted in 29 participants getting excluded.

In accordance with this, participants were also excluded when there median

completion time was 2.5 standard deviations off compared to the mean condition completion time. This was done as we expected that participants taking much shorter than the mean completion time, would have not filled in the questionnaire seriously. Furthermore, people taking much longer than the mean completion time in the group, could have overthought their answer. As we were interested in the initial added value that visualizations would hold, we did not want them to think too hard about their estimates and base them only on the visualization or numerical representation shown. This resulted in 20 participants getting excluded in the text condition, two participants in the pie chart condition, three participants in the stacked bar chart condition and four participants in the dot chart condition.

Finally, we also excluded participants who had an Brier score of more than 2.5

standard deviations from the mean Brier score in that condition. Participants with such a high difference between their estimate and true relative risk ratio would not have understand the general context of odds ratios or did not fill in the questionnaire truthfully. This resulted in one participant getting excluded in the text condition, three participants in the pie chart

condition, four in the stacked bar chart condition and six participants in the dot chart condition.

In the end, the overall sample consisted of 354 participants with 75 participants in the text condition, 101 participants in the pie chart condition, 89 participants in the stacked bar chart condition and 99 participants in the dot chart condition.

3.2 Main Analysis – Comparing the Conditions

To compare the four conditions, we performed a series of Bayesian t-test using a default Cauchy prior of 0.707. To conduct these Bayesian t-tests, we used the software package JASP. This package is developed by the psychological methods department of the University of Amsterdam to provide Bayesian statistical methods to the broader psychological and scientific community (https://jasp-stats.org). In total six Bayesian t-test were run: (1) comparing the text condition versus the pie chart condition, (2) comparing the pie chart condition versus the stacked bar chart condition, (3) comparing the text condition versus the stacked bar chart condition, (4) comparing the text condition versus the dot chart condition, (5) Comparing the pie chart condition versus the dot chart condition and (6) comparing the stacked bar chart condition versus the dot chart condition. We used an uniform prior and had no prior expectations about the direction of an effect. The number of cases, mean Brier scores and standard deviations for all four conditions are listed in table 1.

table 1. Number of Cases, Mean Brier scores and standard deviations for all four conditions

N Mean Brier Score Standard Deviation

Numerical Condition 75 0.111 0.064

Pie Chart Condition 101 0.012 0.018

Bar Chart Condition 89 0.029

Comparing the text condition versus the pie chart condition resulted in a BF10 of 9.947e+28,

suggesting that the data almost fully supported the alternative hypothesis of 𝐻𝐻₁: 𝜇𝜇_{𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡} ≠ 𝜇𝜇_{𝑝𝑝𝑝𝑝𝑡𝑡}. Looking at the mean Brier scores and standard deviations this effect seems to be contributed to the fact that pie chart largely outperformed the text condition to a considerable amount. Looking at our second Bayesian t-test, comparing the pie chart condition versus the stacked bar condition, resulted in a BF10 of 264.3. Given the means and standard deviations of

both conditions, the pie chart condition seemed to outperform the stacked bar chart condition and it was 264.3 as likely to occur under the hypothesis that 𝐻𝐻₁: 𝜇𝜇_{𝑝𝑝𝑝𝑝𝑡𝑡} ≠ 𝜇𝜇_{𝑠𝑠𝑡𝑡𝑠𝑠𝑠𝑠𝑠𝑠𝑡𝑡𝑠𝑠 𝑏𝑏𝑠𝑠𝑎𝑎} than 𝐻𝐻0: 𝜇𝜇𝑝𝑝𝑝𝑝𝑡𝑡= 𝜇𝜇𝑠𝑠𝑡𝑡𝑠𝑠𝑠𝑠𝑠𝑠𝑡𝑡𝑠𝑠 𝑏𝑏𝑠𝑠𝑎𝑎 supporting the evidence found in the study of Hollands and Spence

(1998).

Our third and fourth Bayesian t-test led respectively to BF10 of 5.414e+18 and 1.680 e+18.

Both illustrating the difference between the text condition and the other two visualizations. The fifth Bayesian t-test resulted in a BF10 of 719.6, illustrating the difference between the

mean brier score of the pie and dot charts. The final Bayesian t-test however resulted in a BF10 of 0.184, supporting the hypothesis that the Brier scores are relatively equal to each other

in the stacked bar chart condition and dot chart condition.

In summary, all visual representations outperformed the numerical representation in line with the expectations. Furthermore, the pie chart condition seemed to also outperform both other visual representations.

3.3 Exploratory Analysis - Conservatism

In order to see whether participants over- or underestimated the relative risk ratio given that the Bayes factor deviated largely form 1 and effects of conservatism occurred, we plotted the mean estimated relative risk ratios of each item to the true relative risk ratios in both the visualization representation conditions and numeric condition similar to the method used by

Holcomb et al. (2001). Graph 1 displays the mean estimated risk ratios of each item compared to the true relative risk ratios of the items in the text condition. As can be seen on the graph, there was a large discrepancy when the true relative risk ratio was below 50%, where people overestimate the true relative risk ratio by a significant amount. Although we think that this is partly a flaw in the design of the study, where participants could not clearly distinguish between what Bayes factor belonged to which medicine, it clearly indicates the difficulties that participants had with interpreting the numerical values. Moreover, we also see an consistent underestimation of the true relative risk ratio when above 50%. This could be a sign of conservatism, where people anchor themselves to the prior odds of ½ and do not account for all the information that the new Bayes factor holds.

In contrast, when we graph the mean estimated risk ratios compared to the true relative risk ratios in the visual representations, we see that people comprehend the Bayes factor considerably better. There seems to be almost no deviation from the optimal true estimation and no effects of conservatism seems to occur, illustrating once again that the visual

Graph 1. Estimated mean risk ratios for all items compared to the true relative risk ratios in

the numerical condition .

Graph 2. Estimated mean risk ratios for all items compared to the true relative risk ratios in

4.0 Discussion

In this study, we aimed to contribute to the problems with interpreting the Bayes factor through using visualization techniques to comprehend the associated relative ratios of the truthfulness of both hypotheses. In our experiment we tested three visualization techniques (e.g., pie charts, stacked bar charts and dot charts) in comparison to only a numerical representation of the Bayes factor. We investigated whether these visualization techniques would help people with comprehending the Bayes factor in a more consistent manner and understand the relative risk ratios associated with them. The result of our experiment suggests that visualization techniques might help people in interpreting the Bayes Factor. Comparing the three visualization techniques with the numerical representation using Bayesian t-tests indicated that all visualization techniques helped participants in correctly interpreting the associated posterior probabilities. Furthermore, we also found evidence in line with the study of Hollands and Spence (1998) that pie chart worked best to represent the proportional probability of the Bayes factors compared to the other two visual representations. The reason for this could be that participants rightly comprehend “the pie” as representing the sample space and the pie as the proportional likelihood of the hypothesis.

In addition to this, we wanted to see whether there would be any effects of under- and overestimating the posterior probability by large deviates from a Bayes factor of 1. We found evidence that such effects do occur when the Bayes factor is displayed in a numerical or textual way. However, when visualizing the Bayes factor it seems that participants

comprehend the posterior probability correctly. This illustrates the fact that participants find it hard to estimate a proportional likelihood, when odds ratios are only represented in a

numerical way. Finally, it also seems that participants underestimate the posterior probability when odds ratios are given numerically. This can be seen as a clear distinction of

conservatism and can stem from the anchoring of the prior odds given that people are not represented with a well-defined example of the sample in the numerical condition.

However, the great discrepancy in the estimated posterior probabilities in the textual condition can be partly explained by the misinterpreting of the Bayes factors as the likelihood of the wrong medicine due to how it was displayed. This could have resulted in the high discrepancies with Bayes factors favoring the alternative medicine. When we plotted the estimated risk ratios compared to the true relative risk ratios in the textual condition, it is clear that people not understood that the medicines were turned around in the item. Although this clearly demonstrates the importance of visualization techniques for communicating odds ratios as it gives clarification, it also shows that when the textual representation clearly indicates which Bayes factor favors which hypotheses the effect of visual techniques might be more limited than found in this study.

A more clear example of this problem, was found in the added question for

exploratory research about the strength of evidence, where a lot of participants only indicated their belief about the strength of evidence for one of the hypothesis, when they were asked to provide their beliefs about the other hypothesis. This problem not only occurred with the text condition, but also in the visualization conditions.

Another general limitation of this study is that although the visualization techniques helped participants in calculating the posterior probability as a relative risk ratio, it did not investigate whether how to interpret these posterior probabilities. The belief of the strength of evidence for both hypothesis is still subjective to the researchers own expectations (Dienes, 2011). Furthermore, it also does not give a clear answer to the researcher with how to interpret the posterior probability relative to the truthfulness of both hypotheses. Further research is needed whether how visualizations techniques influences the decisions that researchers make regarding the meaning of Bayes factors and posterior probabilities.

Overall, This study however served as an first step in helping people with interpreting the Bayes factor in a more correct and consistent manner by using visualizations techniques. Although interpreting the Bayes factors will almost always be bound to subjective reasoning, Bayesian analysis serves as a refreshing alternative to classical NHTs and can overcome some of NHTs main drawbacks.

References/Citations

Ancker, J. S., Senathirajah, Y., Kukafka, R., & Starren, J. B. (2006). Design features of graphs in health risk communication. A systematic review. Journal of the American

Medical Informatics Association, 13 (6), 608-618.

Berger, J.O., & Sellke, T. (1987). Testing a point null hypothesis: the irreconcilability of P values and evidence. Journal of the American Statistical Association, 82 (397), 112-122.

Berger, J. (2006). The case for objective Bayesian analysis. Bayesian analysis, 1 (3), 385-402. Box, G. E., & Tiao, G.C. (2011). Bayesian inference in statistical analysis (Vol. 40). John

Wiley & Sons.

Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly

weather review, 78 (1), 1-3.

Corner, A., Harris, A. J., & Hahn, U. (2010). Conservatism in belief revision and participant skepticism. In Proceedings of the 32nd annual conference of the Cognitive Science Society (Vol. 1625, p. 1630). Austin, TX: Cognitive Science Society.

Davies, H.T.O., Crombie, I. K., & Tavakoli, M. (1998). When can odds ratios mislead?. Bmj, 316(7136), 989-991.

Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives

on Psychological Science, 6 (3), 274-290.

DuCharme, W. M., & Peterson, C. R. (1968). Intuitive inference about normally distributed populations. Journal of Experimental Psychology, 78 (2p1), 269.

Gresh, D., Deleris, L.A., Gasparini, L., & Evans, D. (2011). Visualizing risk. In Proceedings of IEEE Information Visualization Conference.

Johnson, D. H. (1999). The insignificance of statistical significance testing. The journal of

wildlife management, 763-772.

Holcomb, W. L., Chaiworapongsa, T., Luke, D.A., & Burgdorf, K.D. (2001). An odd measure of risk: use and misuse of the odds ratio. Obstetrics & Gynecology, 98 (4), 685-688. Hollands, J. G., & Spence, I. (1998). Judging proportion with graphs: The summation model.

Applied Cognitive Psychologyy, 12 (2), 173-190.

Lavine, M., & Schervish, M. J. (1999). Bayes factors: what they are and what they are not. The American Statistician, 53 (2), 119-122.

Lipkus, I. M., & Hollands, J.G. (1998). The visual communication of risk. Journal of the National Cancer Instititute. Monographs, (25), 149-163.

Open Science Collaboration (2015). Estimating the reproducibility of psychological science.

Science, 349 (6251), aac4716.

Romeijn, J.W. (2014). Philosophy of statistics. The Stanford Encyclopedia of Philosophy (Autumn 2014 Edition).

Weinstein, N.D., & Sandman, P.M. (1993). Some criteria for evaluating risk messages. Risk Analysis, 13 (1), 103-114.

Wetzels, R., Matzke, D., Lee, M. D. , Rouder, J. N., Iverson, G.J., & Wagemakers, E.J. (2011). Statistical evidence in experimental psychology an emperical comparison using 855 t tests. Perspectives on Psychological Science, 6 (3), 291-298.