Now we have introduced the concept of random variables, let’s look at some common random variables, starting with the easier cases of discrete random variables. They are common because we can easily relate them to our real-life events. Once we get familiar with them, we can use them to model and solve problems.
Bernoulli Random Variables
Like I said before, when we first come across a new thing, always ALWAYS start with simple examples to get our intuition right. Then, what is the simplest discrete random variable? Well … probably a random variable that can take only two numbers. A random variables that takes two values, usually 1 (head, success, on, etc.) or 0 (tail, failure, off, etc.), is a Bernoulli random variable.
Whenever we talked about a discrete random variable, it is important for us to know its PMF, expectation and variance. The PMF of a Bernoulli random variable is:
$$p_X(k)=\begin{cases} p & \textmd{, if } k=1\\[10pt] 1-p & \textmd{, if } k=0 \end{cases}$$
In the lecture we see that it is relatively easy to compute the expectation and variance of a Bernoulli random variable by the definitions:
$$\mathbb{E}[X] = \sum_{x}xp_X(x) = 1 \times p + 0 \times (1-p) = p$$
and
$$\begin{aligned} \mathbb{V}\textmd{ar}(X) &= \mathbb{E}\left[ (X-\mathbb{E}[X])^2 \right] = \sum_x(X-p)^2p_X(x)\\[12.5pt] &= (1-p)^2 \times p + (0-p)^2(1-p)\\ &= p(1-p) \end{aligned}$$
Binomial Random Variables
The sum of $n$ independent Bernoulli random variables is a binomial random variable. A basic example would be keep tracking the number of heads (or tails) after $n$ independent flips of a coin. We could also think of it as flipping $n$ coins independently at the same time. For each outcome with $k$ heads, the probability is the same: $p^k(1-p)^k$. Then the probability of the random variable taking the value of $k$ is essentially the number of possible outcomes with $k$ heads multiplied by $p^k(1-p)^k$. How many possible outcomes are there with $k$ heads after $n$ flips? We can see it is $\binom{n}{k}$. Therefore, the PMF of a binomial random variable is:
$$ p_X(x) = \begin{cases} \binom{n}{k} p^k (1-p)^{n-k} & \textmd{, when } k=0,1,2,3,\cdots,n\\[10pt] 0 & \textmd{, otherwise} \end{cases} $$
The PMF makes sense to us, but we need to do a sanity check to see if it is a valid probabilistic model. That is:
$$\textmd{whether this is true: } \sum_{k=0}^{n}\binom{n}{k}p^k(1-p)^{n-k} = 1 \textmd{ ?}$$
Recall the binomial theorem from the math class:
$$(a+b)^n = \binom{n}{0}a^0b^n + \binom{n}{1}a^1b^{n-1} + \binom{n}{2}a^2b^{n-2} + \cdots + \binom{n}{n}a^nb^0$$
Note that the right-hand side is $\sum_{k=0}^n\binom{n}{k}a^kb^{n-k}$. If we let $a=p$ and $b=1-p$, then we have:
$$\sum_{k=0}^{n}\binom{n}{k}p^k(1-p)^{n-k} = [p + (1-p)]^n = 1$$
We can also calculate that $\mathbb{E}[X]=np$ and $\mathbb{V}\textmd{ar}(X)=np(1-p)$. The proof can be found in the Extra Reading Material from this lecture.
Poisson Random Variables
Poisson in French means fish. A Poisson random variable is used to model the number of occurrence of a rare event in a time or space interval. We have introduced a few examples during the lecture. The Poisson PMF can be derived by treating it as a special case of the binomial random variable.
Let $\lambda = np$, then $p=\cfrac{\lambda}{n}$. The binomial PMF becomes: $\binom{n}{k}\left( \frac{\lambda}{n} \right)^k \left(1-\frac{\lambda}{n} \right)^{n-k}$. Now if we let $n \to \infty$, it becomes the Poisson PMF.
Before we begin the derivation, we need to recall from the math class: what is $\lim_{n \to \infty} \left(1+\frac{a}{n}\right)^n$, where $a$ is a constant. If we let $n=ax$, then we have:
$$\lim_{n \to \infty} \left(1+\cfrac{a}{n}\right)^n = \lim_{x \to \infty} \left(1+\frac{a}{ax}\right)^{ax} = \lim_{n \to \infty} \left(1+\frac{1}{x}\right)^{ax} = \lim_{n \to \infty} \left[ \left(1+\frac{1}{x}\right)^x \right]^a$$
Since $a$ is a constant, we could take it out of the limit:
$$\lim_{n \to \infty} \left(1+\cfrac{a}{n}\right)^n = \left[\lim_{n \to \infty} \left(1+\frac{1}{x}\right)^x \right]^a$$
Note that $\lim_{n \to \infty} \left(1+\frac{1}{x}\right)^x$ is one of the way to get $e$. Therefore, we have:
$$\begin{equation} \lim_{n \to \infty} \left(1+\cfrac{a}{n}\right)^n = e^a \end{equation} $$
Now we are ready to derive the Poisson PMF:
$$ \begin{aligned} & \lim_{n \to \infty} \binom{n}{k}\left( \cfrac{\lambda}{n} \right)^k \left(1-\cfrac{\lambda}{n} \right)^{n-k} \\[12.5pt] &= \lim_{n \to \infty} \cfrac{n(n-1)\cdots(n-k+1)}{k!} \cdot \cfrac{\lambda ^k}{n^k} \cdot \left(1-\cfrac{\lambda}{n} \right)^n \cdot \left(1-\cfrac{\lambda}{n} \right)^{-\lambda} \\[12.5pt] &= \cfrac{\lambda^k}{k!} \lim_{n \to \infty} \cfrac{n(n-1)\cdots(n-k+1)}{n^k} \cdot \left(1-\cfrac{\lambda}{n} \right)^n \cdot \left(1-\cfrac{\lambda}{n} \right)^{-\lambda} \end{aligned} $$
Now we have reached stage where we need to calculate the limit of a product. The product consists of three terms. If the limit of each term exists, the limit of the product is essentially the product of the limit of each term. Now let’s look at the limit of each of them.
The first term is $\frac{n(n-1)\cdots(n-k+1)}{n^k}$. Note the numerator has $k$ factors, so we have:
$$\cfrac{n(n-1)\cdots(n-k+1)}{n^k} = \cfrac{n^k + \beta_1 \cdot n^{k-1} + \beta_2 \cdot n^{k-2} + \cdots + \beta_{k-1} \cdot n + C}{n^k}$$
When $n \to \infty$, we do not really care about the values of $\beta_i$ and the constant term $C$. $n^k$ grows the fastest, so the limit is 1:
$$\lim_{n \to \infty}\cfrac{n(n-1)\cdots(n-k+1)}{n^k} = 1$$
The second term is $\left(1-\frac{\lambda}{n} \right)^n$. Let $a = -\lambda$ in equation $(1)$, we have:
$$\lim_{n \to \infty} \left(1-\cfrac{\lambda}{n} \right)^n = e^{-\lambda}$$
The third term is $\left(1-\frac{\lambda}{n} \right)^{-\lambda}$. The limit is relatively easy to get:
$$\lim_{n \to \infty} \left(1-\cfrac{\lambda}{n} \right)^{-\lambda} = \left(1-0\right)^{-\lambda} = 1$$
The limits of those three terms do exist, so we have:
$$\lim_{n \to \infty} \binom{n}{k}\left( \cfrac{\lambda}{n} \right)^k \left(1-\cfrac{\lambda}{n} \right)^{n-k} = \cfrac{\lambda^k}{k!} \cdot e^{-\lambda}$$
Therefore, the PMF of a Poisson random variable is:
$$ p_X(x)=\begin{cases} \cfrac{\lambda^k}{k!} \cdot e^{-\lambda} & \textmd{, when } k=0,1,2,3,4,\cdots\\[10pt] 0 & \textmd{, otherwise} \end{cases} $$
Again, let’s first make sure it is a valid probabilistic model:
$$ \begin{aligned} \sum_{k=0}^{\infty}\cfrac{\lambda^k}{k!}\cdot e^{-\lambda} &= \cfrac{\lambda^0}{0!} \cdot e^{-\lambda} + \cfrac{\lambda^1}{1!} \cdot e^{-\lambda} + \cfrac{\lambda^2}{2!} \cdot e^{-\lambda} + \cdots \\[10pt] &= \left(\cfrac{\lambda^0}{0!} + \cfrac{\lambda^1}{1!} + \cfrac{\lambda^2}{2!} + \cdots \right) \cdot e^{-\lambda} \end{aligned} $$
Remember the Taylor series of $e^x$ is $e^x=\frac{x^0}{0!} + \frac{x^1}{1!} + \frac{x^2}{2!} + \cdots$, so we have:
$$\sum_{k=0}^{\infty}\cfrac{\lambda^k}{k!}\cdot e^{-\lambda} = e^{\lambda} \cdot e^{-\lambda} = 1$$
We could also calculated that $\mathbb{E}[X]=\lambda$ and $\mathbb{V}\textmd{ar}(X)=\lambda$. The proof can be found in the Extra Reading Material from this lecture.