ML 数学面试主线：从概率到估计量

Given logits $z\in\mathbb R^K$, Softmax returns a categorical distribution:

$$p_i=\frac{e^{z_i}}{\sum_j e^{z_j}},\qquad \sum_i p_i=1.$$

For label distribution $y$, Cross-Entropy returns a scalar loss:

$$\mathcal L_{\mathrm{CE}}=-\sum_i y_i\log p_i.$$

For one-hot class $c$:

$$\mathcal L=-\log p_c=-z_c+\operatorname{LSE}(z),\qquad \operatorname{LSE}(z)=\log\sum_j e^{z_j}.$$

The key derivatives are

$$\frac{\partial p_i}{\partial z_j}=p_i(\mathbf1_{i=j}-p_j),\qquad \nabla_z\operatorname{LSE}(z)=\operatorname{softmax}(z),$$

$$\frac{\partial\mathcal L}{\partial z_i}=p_i-y_i.$$

Exponentials can overflow. With $m=\max_i z_i$, compute

$$\operatorname{LSE}(z)=m+\log\sum_i e^{z_i-m}.$$

Softmax is invariant to a common shift, so subtracting $m$ is exact. In code, use log_softmax or cross_entropy rather than taking the log of an already underflowed probability.

For independent examples, the conditional likelihood is

$$L(\theta)=\prod_{n=1}^N p_\theta(y_n\mid x_n).$$

Hence

$$ \arg\max_\theta\prod_n p_\theta(y_n\mid x_n) \iff \arg\max_\theta\sum_n\log p_\theta(y_n\mid x_n) \iff \arg\min_\theta-\sum_n\log p_\theta(y_n\mid x_n). $$

For one-hot classification, the final expression is empirical Cross-Entropy. More precisely, minimizing one-hot CE performs maximum conditional likelihood for the categorical model. This assumes probabilistic outputs and the usual factorization over examples. Label smoothing changes the target and is no longer the same MLE on the original hard labels.

$$\mathcal L_{L1}=\mathcal L_{\text{data}}+\lambda\|w\|_1,\qquad \mathcal L_{L2}=\mathcal L_{\text{data}}+\frac\lambda2\|w\|_2^2.$$

L2 has gradient $\lambda w$, so its shrinkage fades near zero. Away from zero, the L1 subgradient is $\lambda\operatorname{sign}(w)$; at zero it is the interval $[-\lambda,\lambda]$.

For

$$\min_w\frac12(w-a)^2+\lambda|w|,$$

the optimizer is the soft-thresholding operator

$$w^*=\operatorname{sign}(a)\max(|a|-\lambda,0).$$

When $|a|\le\lambda$, zero satisfies the subgradient condition, mapping a whole interval of inputs to an exact zero. The comparable L2 solution is $a/(1+\lambda)$, which normally shrinks without becoming zero. Geometrically, the L1 constraint has corners on the axes; the L2 boundary is smooth.

If $y=f(x)+\epsilon$, $\mathbb E[\epsilon]=0$, and $\operatorname{Var}(\epsilon)=\sigma^2$, then under squared error

$$ \mathbb E[(y-\hat f(x))^2] =\sigma^2 +\big(\mathbb E[\hat f(x)]-f(x)\big)^2 +\mathbb E\big[(\hat f(x)-\mathbb E[\hat f(x)])^2\big]. $$

These are irreducible noise, squared bias, and variance. Bias measures systematic error averaged over possible training sets; variance measures sensitivity to which training set was sampled. More flexible models tend to reduce bias and increase variance, while regularization often trades some bias for lower variance. The exact additive identity above is a squared-loss result.

Maximum likelihood chooses the parameters that best explain the data:

$$\hat\theta_{\mathrm{MLE}}=\arg\max_\theta p(D\mid\theta).$$

MAP adds a prior:

$$\hat\theta_{\mathrm{MAP}} =\arg\max_\theta\big[\log p(D\mid\theta)+\log p(\theta)\big].$$

A Gaussian prior

$$p(w)\propto e^{-\frac\lambda2\|w\|_2^2}$$

produces an L2 penalty; a Laplace prior

$$p(w)\propto e^{-\lambda\|w\|_1}$$

produces L1. Regularization can therefore be read as a negative log-prior in MAP. Both MLE and MAP depend on the likelihood assumptions, while MAP also depends on the prior. With enough data the prior often has less relative influence, but it does not repair misspecification or non-identifiability automatically.

Here LSE means Least Squares Estimator, not LogSumExp. Under

$$y=X\beta+\epsilon,$$

OLS is

$$\hat\beta=(X^\top X)^{-1}X^\top y.$$

The Gauss–Markov conditions are linearity in parameters, full column rank of $X$, $\mathbb E[\epsilon\mid X]=0$, and $\operatorname{Var}(\epsilon\mid X)=\sigma^2I$: homoscedastic, uncorrelated errors. Normality is not required for BLUE.

Proof sketch: let $A_0=(X^\top X)^{-1}X^\top$. Any other linear unbiased estimator can be written as $(A_0+C)y$, and unbiasedness implies $CX=0$. Cross terms vanish, giving

$$ \operatorname{Var}((A_0+C)y)-\operatorname{Var}(A_0y) =\sigma^2CC^\top\succeq0. $$

Thus OLS has minimum covariance among linear unbiased estimators. With heteroscedastic or correlated errors, OLS may remain unbiased under conditional mean zero but is no longer best; known covariance structure motivates GLS.