The binomial test is arguably the conceptually simplest of all statistical tests: It has only one parameter and an easy to understand distribution for the data. When introducing null hypothesis significance testing it is puzzling that the binomial test is not the first example of a test but sometimes is introduced long after the ttest and the ANOVA (as here) and sometimes is not introduced at all (as here and here). When introducing a new concept, why not start with simplest example? It is not like there is a problem with students understanding the concept of null hypothesis significance testing too well. I’m not doing the same misstake so here follows the Bayesian First Aid alternative to the binomial test!
Bayesian First Aid is an attempt at implementing reasonable Bayesian alternatives to the classical hypothesis tests in R. For the rationale behind Bayesian First Aid see the original announcement. The development of Bayesian First Aid can be followed on GitHub. Bayesian First Aid is a work in progress and I’m grateful for any suggestion on how to improve it!
The Model
Coming up with a Bayesian alternative for the binomial test is almost a nobrainer. The data is a count of the successes and failures in some task (where what is labeled success is an arbitrary choice most of the time). Given this impoverished data (no predictor variables and no dependency between trials) the distribution for the data has to be a binomial distribution with the number of Bernoulli trials ($n$) fixed and the relative frequency of success ($\theta$) as the only free parameter. Notice that $\theta$ is often called the probability of success but from a Bayesian perspective I believe this is almost a misnomer and, at the very least, a very confusing name. The result of a Bayesian analysis is a probability distribution over the parameters and calling $\theta$ a probability can result in hardtodecipher statements like: “The probability that the probability is larger than 50% is more than 50%”. Calling $\theta$ the relative frequency of success also puts emphasis on that $\theta$ is a property of a process and reserves “probability” for statements about knowledge (keeping probabilities nicely “inside the agent”).
The only part of the model that requires some thought is the prior distribution for $\theta$. There seems to mainly be three priors that are proposed as noninformative distributions for $\theta$: The flat prior, Jeffrey’s’ prior and Haldanes prior. These priors are beautifully described by Zhu and Lu (2004) and the flat and Jeffrey’s are pictured below.
The Haldane prior is trickier to visualize as it puts an infinite amount of mass at $\theta=0$ and $\theta=1$.
As explained by Zhu and Lu, the Haldane prior could be considered the least informative prior, so isn’t that what we want? Nope, I’ll go with the flat distribution which can also be considered as “lest informative” when we know that both successes and failures are possible (Lee, 2004). I believe that in most cases when binomial data is analyzed it is known (or at least suspected) a priori that both successes and failures are possible thus it makes sense to use the flat prior as the default prior.
The flat prior is conveniently constructed as a $\mathrm{Beta}(1,1)$ distribution. A $\mathrm{Uniform}(0,1)$ could equally well be used but the $\mathrm{Beta}$ has the advantage that it is easy to make it more or less informative by changing the two parameters. For example, if you want to use Jeffrey’s prior instead of the flat you would use a $\mathrm{Beta}(0.5,0.5)$ and a Haldane prior can be approximated by $\mathrm{Beta}(\epsilon,\epsilon)$ where $\epsilon$ is close to zero, say 0.0001.
The final Bayesian First Aid alternative to the binomial test is then:
The diagram to the left is a Kruschke style diagram. For an easy way of making such a diagram check out this post.
The bayes.binom.test
function
Having a model is fine but Bayesian First Aid is also about being able to run the model in a nice way and getting useful output. First let’s find some binomial data and use it to run a standard binomial test in R. The highly cited Nature paper Poleward shifts in geographical ranges of butterfly species associated with regional warming describes how the geographical areas of a sample of butterfly species have moved northwards, possibly as an effect of the rising global temperature. In that paper we find the following binomial data:
Here is the corresponding binomial test in R with “retracting northwards” as success and “extending southwards” as failure:
1


1 2 3 4 5 6 7 8 9 10 11 

Not a significant result at the 5% $\alpha$ level. C’est la vie.
Fun fact: The resulting pvalue is not the same as that reported in the Nature paper. By closer inspection it seems like the reported pvalue is correct as Table 2 shows that there actually were ten species retracting northwards and not nine, “9 retracting north” was a typo and should have read “10 retracting north”. I leave it as an excecise for the interested reader to find the other wrongly reported binomial test pvalue in the paper. Despite being a typo, here I’ll continue with nine successes and two failures.
Now to run the Bayesian First Aid alternative you simply prepend bayes.
to the function call.
1 2 

1 2 3 4 5 6 7 8 9 10 11 

I’ve chosen to make the output similar to binom.test
. You get to know what data you put in, you get a point estimate (here the mean of the posterior of $\theta$) and you get a credible interval. You don’t get a pvalue, you do get the probability that $\theta$ is lower or higher that a comparison value which defaults to 0.5. Looking at these probabilities and the credible interval it seems like there is pretty good evidence that butterflies are generally moving northwards.
A goal with Bayesian First Aid is complete compliance with the calling structure of the original test functions. You should be able to use the same arguments in the same way as you are used to with binom.test
. For example, the call
1


is handled by bayes.binom.test
by using an 80% credible interval and using 0.75 as the comparison value. I haven’t figured out anything useful to do with alternative="greater"
so that simply issues a warning:
1


1


1 2 3 4 5 6 7 8 9 10 11 

If you run the same call to bayes.binom.test
repeatedly you’ll notice that you get slightly different values every time. This is because Markov chain Monte Carlo (MCMC) approximation is used to approximate the posterior distribution of $\theta$. The accuracy of the MCMC approximation is governed by the n.iter
parameter which defaults to 15000. This should be enough for most purposes but if you feel it is not enough, just max it up!
1


Generic Functions
Every Bayesian First Aid test have corresponding plot
, summary
, diagnostics
and model.code
functions.
plot
plots the parameters of interest and, if appropriate, a posterior predictive distribution. In the case of bayes.binom.test
there is just one parameter $\theta$. The plot of the posterior is a design by John D. Kruschke copied from his Bayesian estimation supersedes the t test program. To get the plot, first save the output of bayes.binom.test
to a variable and then plot
it!
1 2 

summary
prints out a comprehensive summary of all the parameters in the model:
1


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 

diagnostics
prints and plots MCMC diagnostics calculated using the coda package. In the case of bayes.binom.test
MCMC diagnostics is overkill, still it is nice to get that warm fuzzy feeling you get when looking at converging MCMC chains:
1


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 

Last but not least model.code
prints out R and JAGS code that runs the model and that you can copynpaste directly into an R script:
1


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 

For example, in order to change the flat prior on $\theta$ into Jeffrey’s prior we simply change dbeta(1, 1)
into dbeta(0.5, 0.5)
and rerun the code:
1 2 3 4 5 6 7 8 9 10 11 

1 2 3 4 5 6 

1 2 3 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 
