AY2021-2022, Semester 1, Week 14 Monday

A biased coin?

2.1 Investigate whether the coin is biased

Flipping 10 times a coin that was thought to be fair results in 7 heads and 3 tails.

  1. Show that this event is not statistically significant.
  2. Find out how far you have to reduce the total number of tosses for 7 heads to be statistically significant.

Do you have any priors about the potential bias? (more likely to be heads or tails?)

2.1 Investigate whether the coin is biased

1. Show that this event is not statistically significant.

trials <- 10000
responses <- numeric(trials)
response.true <- 7
tosses <- 10

for(i in 1:trials){
  results <- sample(c("H","T"), size = tosses, replace = TRUE)
  responses[i] <- sum(results=="H")
}
hist(responses, freq=FALSE, breaks=c(seq(-0.5,tosses+0.5,1)))
abline(v=quantile(responses,prob=c(0.025,0.975)), col="orange", lwd=2)
abline(v=response.true, col="blue", lwd=2)

2.1 Investigate whether the coins is biased

1. Show that this event is not statistically significant.

Our observed result is within the 95% confidence interval. The value is not as extreme as what is needed to rule out the possibility that the result is just a random outcome, with a certainty of 95%.

2.1 Investigate whether the coins is biased

2. Find out how far …

trials <- 10000
responses <- numeric(trials)
response.true <- 7

par(mfrow=c(2,2))
for(tosses in seq(10,7,-1)) {
  for(i in 1:trials){
    results <- sample(c("H","T"), size = tosses, replace = TRUE)
    responses[i] <- sum(results=="H")
  }
  hist(responses, freq=FALSE, breaks=c(seq(-0.5,tosses+0.5,1)),
       main=paste("Total of",tosses,"tosses"))
  abline(v=quantile(responses,prob=c(0.025,0.975)), col="orange", lwd=2)
  abline(v=response.true, col="blue", lwd=2)
}
par(mfrow=c(1,1))

2.1 Investigate whether the coin is biased

2. Find out how far …

We had to reduce all the way to 7 tosses for 7 heads to be statistically significant.

2.2 Investigate whether the coins are biased

We have two coins that are both believed to be fair. Flipping the two coins simultaneously 10 times results in them both landing heads once; the other 9 times they were both tails, or one of each.

  1. Show that this event is not statistically significant.
  2. How far do you need to increase the total number of coin flips for only one double head to be statistically significant?

2.2 Investigate whether the coins are biased

  1. Show that this event is not statistically significant.
trials <- 10000
responses <- numeric(trials)
response.true <- 1
tosses <- 10

for(i in 1:trials){
  results <- sample(c("HH","HT","TH","TT"), size = tosses, replace = TRUE)
  responses[i] <- sum(results=="HH")
}
hist(responses, freq=FALSE, breaks=c(seq(-0.5,tosses+0.5,1)))
abline(v=quantile(responses,prob=c(0.025,0.975)), col="orange", lwd=2)
abline(v=response.true, col="blue", lwd=2)

2.2 Investigate whether the coins are biased

  1. Show that this event is not statistically significant.

Our observed result is within the 95% confidence interval. The value is not as extreme as what is needed to rule out the possibility that the result is just a random outcome, with a certainty of 95%.

2.2 Investigate whether the coins are biased

After a few experiments, we discover that we need more than 19 tosses for significance. Using this code, we try to solve the problem.

trials <- 10000
responses <- numeric(trials)
response.true <- 1

par(mfrow=c(2,1))
for(tosses in 19:20) {
  for(i in 1:trials){
    results <- sample(c("HH","HT","TH","TT"), size = tosses, replace = TRUE)
    responses[i] <- sum(results=="HH")
  }
  hist(responses, freq=FALSE, breaks=c(seq(-0.5,tosses+0.5,1)),
       main=paste("Total of",tosses,"tosses"))
  abline(v=quantile(responses,prob=c(0.025,0.975)), col="orange", lwd=2)
  abline(v=response.true, col="blue", lwd=2)
}
par(mfrow=c(1,1))

2.2 Investigate whether the coins are biased

On two consecutive runs of the same code, we get different answers!

We had to increase all the way to 20 or 21 for statistical significance. To decide whether it should be 20 or 21 we may need more trials. Think about how probability is defined; 10000 is a large number but probability is defined in terms of that number getting larger and larger.

What we have learned

  1. Using permutation tests to examine the statistical significance of different events
  2. Checking statistical significance with different benchmarks
  3. To have greater confidence in the conclusion of our permutation test, we may need more trials