Collection of Foundational Math Problems

Foundational math problems to keep me on track after a long time of not doing math.

Discrete Mathematics

  1. Derive sample mean update formula. Given $\mu_{n-1}=\frac{1}{n}\sum_{i=0}^{n-1}x_i$, derive efficient calculation of $\mu_{n}$.
  2. Derive sample variance update formula. Given $\sigma_{n-1}^2=\frac{1}{n}\sum_{i=0}^{n-1}(x_i-\mu_{n-1})^2$, derive efficient calculation of $\sigma_{n}^2$.

Linear Algebra

  1. Factor for $v-Av$
  2. Proof that $(a-b)^T D (a-b)$ is a form of weighted mean squared error with weight on the diagonal matrix D
  3. Define what frobenius norm is, what is the relation with Trace, and what is the derivative of frobenius norm wrt input matrix.

Vector Calculus and Probabilistic Modeling

Interpret $w_0$ correspond to the solution for maximum likelihood estimation given data $\mathcal D={(x_i, t_i)}_i$

\[t=y(x, w) + \epsilon\]

Find $w$ that maximize maximum likelihood given data $\mathcal D={(x_i, t_i)}_i$

\[\begin{align*} t &= \mathcal N(y, \sigma^2) \\ y &= w_0 + w_1x_0 + w_2x_1 + \dots + w_{n+1}x_n \end{align*}\]

Find the closed form of $w_0$ and interpret the function of it.

\[p(t\mid x, w) = \mathcal N(t\mid 3x+2, 1)\]

What is the assumed form of the data-generating model $t$?

\[p(y=1\mid x, w) = \sigma(w^Tx)=\frac{1}{1+\exp(-w^Tx)}\]

What is the full expression for the likelihood $p(y\mid x, w)$ and naturally lead loss function.

\[t\sim Poisson(\lambda(x))\quad \lambda(x)=\exp(w_0+w_1x)\]

Write the expression for $p(t\mid x,w)$ and best parameter $w$ using maximum likelihood estimation.

\[\frac{1}{2}\sum_i(t_i-w^T\phi(x_i))^2 + \frac{\lambda}{2}w^Tw\]

Solve for $w$ that minimize above expression

\[\begin{align*} p(\vec t\mid x, W, \sigma^2) &= \mathcal N(\vec t\mid W^T\phi(x), \sigma^2 I) \\ &= \frac{1}{\sqrt{(2\pi)^K\vert\sigma^2 I\vert}}\exp\left(-\frac{1}{2}(\vec t - W^T\phi(x))^T(\sigma I)^{-1}(\vec t - W^T\phi(x))\right) \end{align*}\]

Find the likelihood of above gaussian given $\mathcal D={(x_i, t_i)}_i$ then find the best $w$ that maximize. What optimization we could use to solve for batch dataset (e.g. using the whole dataset instead of using one by one).

Given $T=[t_i]_i\in R^{N\times K}, X=[\phi(x_i)]_i\in R^{N\times D}, W=[w^T_i]_i \in R^{D\times K}$, find $w$ that minimize \(\mathcal L(w)= \frac{1}{2}\sum_i^N \sum_j^K (w^T\phi(x_i)_j - t_{ij})^2\)

Express the above into matrix form utilizing forbenius norm and then solve for $w$ that minimize above expression.

Given observation state age $x_0$ (old or young) and covid test result $x_1$. Given prior assumption of covid $p(y)=0.1$ and $p(x_1=1\mid y=1)=0.875$ and $p(x_1=0\mid y=0)=0.975$ (no matter the age). Given $\ell(x, a)$ as below.

State Do nothing Give drug
No COVID-19, young 0 8
COVID-19, young 60 8
No COVID-19, old 0 8
COVID-19, old 10 8

For each state, find the optimal action $a^*$ that minimize the expected loss.

Given $\ell(y, a)$ as below. Assume we have a model $P(y\mid x)$

Predicted label a = 0 a = 1
True label    
y = 0 $\ell_{00}$ $\ell_{01}$
y = 1 $\ell_{10}$ $\ell_{11}$

Find the optimal decision rule $a^*$ that minimize the expected loss using our model $P$.

Given $\ell(y, a)$ below, find the optimal decision rule $a^*$ that minimize the expected loss given model $P(Y\mid X)$. \(\ell l(y, a) = \begin{cases} 0&\text{if }a=\hat y\\ \lambda_r&\text{if }a=\text{reject}\\ \lambda_e&\text{otherwise} \end{cases}\)

Find closed form solution $w^*$ that maximize MLE for model below on dataset $\mathcal D={(x_i, y_i)}_i$.

\[p(y=1\mid x) = \frac{1}{1+\exp(-w^Tx+b)}\]

Find closed form solution $w^*$ that maximize MLE for model below on dataset $\mathcal D={(x_i, y_i)}_i$.

\[p(y=k\mid x) = \mathcal S(XW_n)\]