Probability Theory Essentials Episode 1 | Moment Generating Function | Covariance | Student Theorem

preview_player
Показать описание
### Unqiueness of Moment Generating Function

Let the CDF of $\boldsymbol{X}, \boldsymbol{Y}$ are $F_{\boldsymbol{X}}(x)$ and $F_{\boldsymbol{Y}}(y)$, MGF are $M_{\boldsymbol{X}}(t)=E[e^{t{\boldsymbol{X}}}]$ and $M_{\boldsymbol{Y}}(t)=E[e^{t{\boldsymbol{Y}}}]$. Then $F_{\boldsymbol{X}}(z)=F_{\boldsymbol{Y}(z)}$ for all $z\in R$ iff $M_{\boldsymbol{X}}(t)=M_{\boldsymbol{Y}}(t)$ for all $t\in (-h,h)$ for some $h \gt 0$.

### Independence By MGF

$X = [X_1, X_2]'$ are independent iff $M(\boldsymbol{t}) = E[e^{\boldsymbol{t}'\boldsymbol{X}}] = M_{X_1}(t_1)M_{X_2}(t_2)$

Proof:

$X = [X_1, X_2]'$ independent, then $M(\boldsymbol{t}) \implies E[e^{\boldsymbol{t}'\boldsymbol{X}}] = M_{X_1}(t_1)M_{X_2}(t_2)$.

$M(\boldsymbol{t}) \impliedby E[e^{\boldsymbol{t}'\boldsymbol{X}}] = M_{X_1}(t_1)M_{X_2}(t_2)$, by uniqueness of MGF, $f(x,y)$ of $M(\boldsymbol{t})$ equals $f_xf_y$ of $M_{X_1}(t_1)M_{X_2}(t_2)$

### Pearson correlation coefficient between -1 and 1

$\rho = \frac{Cov(X,Y)}{\sigma_{\boldsymbol{X}}\sigma_{\boldsymbol{Y}}} \in [-1, 1]$

Proof:

By Holder's inequality (Generalization of Cauch Schwarz inequality) in $L^2$ space: $|Cov(X,Y)| \le \sigma_{\boldsymbol{X}}\sigma_{\boldsymbol{Y}}$

### Commonly used Properties and Formulas

1. $Cov(cX, Y) = cCov(X,Y)$
2. $Cov(X+Y, Z) = Cov(X, Z) + Cov(Y, Z) = Cov(X, Y+Z)$
3. $\displaystyle Cov(\sum_{i=1}^n{X_i}, \sum_{j=1}^m{Y_j}) = \sum_{i=1}^n\sum_{j=1}^mCov(X_i, Y_j)$
4. $\displaystyle Var(\sum_{i=1}^nX_i) = Cov(\sum_{i=1}^nX_i, \sum_{j=1}^nX_j) = \sum_{i=1}^{n}Var(X_i) + 2\sum_{i=1}^n\sum_{j\gt{i}} Cov(X_i, Y_j)$
If $X_i$ are has common $\mu, \sigma^2$, then $\displaystyle Var(\sum_{i=1}^nX_i) = n\sigma^2 + n(n-1)\rho \sigma^2$
5. If $X_1, X_2, ..., X_n$ are iid, then, $Cov(\bar{X}, X_i-\bar{X}) = 0$.
Proof:
$$
\begin{aligned}
\displaystyle Cov(\bar{X}, X_i-\bar{X}) &= Cov(\bar{X}, X_i) - Cov(\bar{X}, \bar{X}) \\
&= \frac{1}{n}Cov(X_i + \sum_{j\ne{i}}X_j, X_i) - Var(\bar{X})
&= \frac{\sigma^2}{n} - \frac{\sigma^2}{n} = 0
\end{aligned}
$$
6. If $Z ~ N(0,1)$, then $Z^2 \sim \chi^2(1)$. Calculate $P(Z^2 \le z)$. Same, if $Z_1, Z_2,..., Z_n$ are iid standard normal, then $\displaystyle\sum_{i=1}^nZ_i^2 \sim \chi^2(n)$
7. $E_Y[E_X[X|Y]] = \int\int{xf(x|y)} g_Y(y) dxdy = \int\int{x\frac{f(x,y)}{f_Y(y)}} f_Y(y) dxdy = \int\int{xf(x,y)} dxdy = E_X[X]$
8. $\displaystyle Var[X] = E_Y[Var[X|Y]] + Var[E_X[X|Y]]$, $Var[E_X[X|Y]] \le Var[X]$
Proof:
$$
\begin{aligned}
E[Var[X|Y]] &= E[E[X^2|Y]] - E[E[X|Y]^2] = E[X^2] - E[E[X|Y]^2] \\
Var[E[X|Y]] &= E[E[X|Y]^2] - E[X]^2
\end{aligned}
$$

## Student Theorem
------

Let $X_1, X_2, \dots, X_n$ iid $N(\mu, \sigma^2)$. Then:

1. $\bar{X} \sim N(\mu, \frac{\sigma^2}{n})$
2. $\bar{X}$ and $S^2$ are independent
3. $\frac{(n-1)S^2}{\sigma^2} \sim \chi^2(n-1)$
4. $T=\frac{\frac{(\bar{X}-\mu)}{\frac{\sigma}{\sqrt{n}}}}{\sqrt{\frac{(n-1)S^2}{\sigma^2(n-1)}}} =\frac{\sqrt{n}(\bar{X}-\mu)}{S} \sim t(n-1)$

Proof:

1. Prove by MGF, or by the fact that linear combination of iid normals are also normal with $E[\sum{\frac{X_i}{n}}] = n\frac{\mu}{n}$ and $Var(\sum{\frac{X_i}{n}})=n\frac{\sigma^2}{n^2}$
2. Use the fact that $\bar{X}$ and ${X_1-\bar{X}, X_2-\bar{X},...,X_n-\bar{X}}$ are all normal and the fact that the covoriance of $\bar{X}$ and the random vector ${X_i-\bar{X}}$ is zero, they are indenpdent, so is the functino of independent variables.
3. Refactor sample variance using $\sum{(\frac{(X_i-\mu) + (\mu-\bar{X})}{\sigma})^2}$ and use independence of $S^2, \bar{X}$ and mgf to derive that the final mgf is the mgf of $\chi^2(n-1)$
4. Is a result of 1-3 and the definition of student distribution.
Рекомендации по теме