Before Monday’s class, you were all asked to complete a short survey.

For today’s activity, we will use the cleaned and anonymized survey data to find out more about the distributions of some of the quantitative indicators that we collected through the survey and that describe you (QR students).

Consistent with this week’s focus on the normal distribution, the overarching question in today’s activity is: How “Normal” are QR students?

The goals of this activity are as follows:

• Develop intuition about which data we expect to be normally distributed
• Develop intuition about which data we expect to be non-normally distributed
• Learn how to assess whether the data are normally distributed (using R)

In doing so, we will use some of the R commands and techniques featured in the YouTube R-tutorials to visually assess whether or not a distribution is “normal” (e.g. hist, dnorm, curve, qqnorm, and qqline)

Let us start by importing survey.csv as a data frame survey into R. Let us then examine the data:

survey <- read.csv("survey.csv")
head(survey)

##   gender     nationality height phone facebook  youtube shoe postcode boxoffice
## 1   Male Non-Singaporean    190    43        5      267 29.4        6        NA
## 2 Female     Singaporean    168    27        0   386200 24.7        7 311605581
## 3 Female     Singaporean    154    40       NA     8000 23.9        5        NA
## 4   Male     Singaporean    180    48       NA  1200000 40.0        9  64337744
## 5 Female Non-Singaporean    163    35      493 58658325 25.4        6 258814783
## 6 Female     Singaporean    167    92        0        0 23.0        9     10000

To recall, you were asked to report the following information:

• Categorical Indicators

  • gender: Your gender identity (Female / Male / Neither, both, it depends, something else, etc.)
  • nationality: Your nationality (Singaporean / Non-Singaporean / Other or prefer not to say)

• Quantitative Indicators

  • height: Your height (measured in centimetres)
  • phone: The last two digits of your Singapore hand phone number
  • youtube: The number of views on the video on YouTube that you most recently watched

• Are the heights of QR students normally distributed?

      All Students: Not exactly normally distributed, but...
      Female: Closer to normally distributed with mean 162.8 and sd 6.5
      Male: Closer to normally distributed with mean 175.4 and sd 7.2

• Are the last two digit of the Singapore hand phone numbers of QR students normally distributed?

      Not normally distributed at all, closer to a uniform distribution

• Are the number of views on the videos that QR students most recently watched on YouTube normally distributed?

      Not normally distributed, skewed to the right tail.
      But when we transform...still not normally distributed!

The distribution of non-transformed YouTube views is clearly right skewed and not normal. But even after transforming the number of YouTube views, the data are not normally distributed and the distribution instead appears multi-modal.

• What might have contributed to the odd shape of the distribution?

• Are YNC students different from other YouTube users in terms of their viewing habits in some way that might help to explain?

  • Hint: Consider what you were doing before completing the survey

Yes, YNC students are different: Unlike most YouTube users, YNC students must take QR in Sem 1 of their first year.

• As part of QR, students complete preparatory work before coming to class.
• This usually involves viewing one or more of the excellent series of R Tutorial videos created by Prof. Michael Gastner.
• We know that the last video assigned as part of their preparatory materials before the survey was R Tutorial 25: Z-scores with R.

Let us check if this can help us to explain the odd shape.

How many views are there on QR Tutorial 25?

QR Tutorial Video 25: Z-scores with R

Let’s calculate the log transformed number of views.

tutorial <- log10(3622 + 1) # transform the number of views on QR Tutorial Video 25
tutorial                    # print to console

## [1] 3.559068

Now let’s check if the resulting value is similar to any of the modes of the distribution by adding abline() to the histogram and QQ plots.

The log transformed number of views appears to line up with one of the modes.

We might reasonably conclude the shape is at least partly due to how QR students differ from other users: Many probably watched the same video.

The purpose of today’s lesson was to become more familiar with the normal model. Through the activities, we have worked together to do the following:

• Examined real demographic data and whether it follows a normal model
  • How to diagnose, think about, and deal with outliers in real data
• Develop intuition about which data we might expect to follow a normal model as well as which data might not follow a normal model
• Learn how to assess whether or not data are normally distributed (using R)

When you completed the survey and first examined the dataset, you may have noticed that it contained some additional information:

• facebook: The number of reactions to the Facebook post to which you most recently reacted
• shoe: Your shoe size in centimetres
• postcode: The last digit of the postal code of your permanent residence
• boxoffice: The total worldwide gross of your favorite film

hist(survey$facebook, main="Facebook Reacts", xlab = "Reacts",
     las=1, col="grey", freq = FALSE)

survey$log_facebook<-log10(survey$facebook+1) #Many 0 in reacts. Add one to avoid missing.
mean(survey$log_facebook, na.rm=TRUE)

## [1] 1.225786

sd(survey$log_facebook, na.rm=TRUE)

## [1] 1.222481

max(survey$log_facebook, na.rm=TRUE)

## [1] 4.81292

min(survey$log_facebook, na.rm=TRUE)

## [1] 0

hist(survey$log_facebook, main="Facebook Reacts", xlab = "Reacts (log10)",
     las=1, col="grey", breaks=c(seq(0,6,1)), freq = FALSE)

qqnorm(survey$log_facebook)
qqline(survey$log_facebook, col="red")

hist(survey$shoe, main="Shoe Size", xlab="Size",
     xlim=c(20,40), las=1, col="grey", breaks=c(seq(20,40,1)), freq = FALSE)

mean(survey$shoe, na.rm=TRUE)

## [1] 27.74524

sd(survey$shoe, na.rm=TRUE)

## [1] 4.368099

hist(survey$shoe, main="Shoe Size", xlab="Size",
     xlim=c(20,40), las=1, col="grey", breaks=c(seq(20,40,1)))
curve(dnorm(x,
            mean = mean(survey$shoe, na.rm=TRUE),
            sd = sd(survey$shoe, na.rm=TRUE)),
      from = 20, to = 40, add = TRUE, col = "red")

qqnorm(survey$shoe)
qqline(survey$shoe, col="red")

mean(survey$shoe[survey$gender=="Female"], na.rm=TRUE)

## [1] 26.82661

sd(survey$shoe[survey$gender=="Female"], na.rm=TRUE)

## [1] 4.723942

mean(survey$shoe[survey$gender=="Male"], na.rm=TRUE)

## [1] 29.07531

sd(survey$shoe[survey$gender=="Male"], na.rm=TRUE)

## [1] 3.510396

hist(survey$postcode, main="Last digit of Postcode", xlab="Last digit",
     las=1, breaks=c(seq(-0.5, 9.5, 1)), freq = FALSE)

qqnorm(survey$postcode)
qqline(survey$postcode, col="red")

hist(survey$boxoffice, main="Boxoffice Gross", xlab="Gross",
     las=1, freq = FALSE)

survey$log_boxoffice<-log10(survey$boxoffice+1) #Add one to avoid any missing.
mean(survey$log_boxoffice, na.rm=TRUE)

## [1] 8.089664

sd(survey$log_boxoffice, na.rm=TRUE)

## [1] 0.8525104

max(survey$log_boxoffice, na.rm=TRUE)

## [1] 9.454425

min(survey$log_boxoffice, na.rm=TRUE)

## [1] 4.000043

hist(survey$log_boxoffice, main="Boxoffice Gross", xlab="Gross (log10)",
     las=1, col="grey", breaks=c(seq(0,10,1)), freq = FALSE)
curve(dnorm(x,
            mean = mean(survey$log_boxoffice, na.rm=TRUE),
            sd = sd(survey$log_boxoffice, na.rm=TRUE)),
      from = 0, to = 10, add = TRUE,col = "red")

qqnorm(survey$log_boxoffice)
qqline(survey$log_boxoffice, col="red")