Multivariate Normal, Wishart and Quadratic Forms
CSIR NET statistics: multivariate normal properties, marginals and conditionals, quadratic forms and chi-square, Cochran's theorem, Wishart and Hotelling T².
- 17 min
On this page · 12 sections
Prerequisites
- Quadratic Forms: positive definite matrices, spectral decomposition and rank.
- Eigenvalues and the Cayley–Hamilton Theorem: an idempotent matrix has eigenvalues 0 and 1, so its rank equals its trace.
- Gauss–Markov Models, BLUE and Linear Hypotheses: the sums of squares whose distributions this page derives.
The Hook: Two Normals That Are Not Jointly Normal
The Multivariate Normal Distribution
A random vector \( X \in \mathbb{R}^p \) is multivariate normal, \( X \sim N_p(\mu, \Sigma) \), if \( a^TX \) is univariate normal (possibly degenerate) for every \( a \in \mathbb{R}^p \). Then \( E(X) = \mu \), \( \operatorname{Cov}(X) = \Sigma \) (symmetric, positive semidefinite) and the moment generating function is
$$M_X(t) = \exp\left(t^T\mu + \tfrac{1}{2}t^T\Sigma t\right).$$If \( \Sigma \) is positive definite, \( X \) has the density
$$f(x) = \frac{1}{(2\pi)^{p/2}\lvert\Sigma\rvert^{1/2}}\exp\left(-\tfrac{1}{2}(x - \mu)^T\Sigma^{-1}(x - \mu)\right).$$If \( \Sigma \) is singular, there is no density on \( \mathbb{R}^p \): \( X \) lives on the affine subspace \( \mu + \mathcal{C}(\Sigma) \).
The density is constant on the ellipsoids \( (x - \mu)^T\Sigma^{-1}(x - \mu) = c \), with axes along the eigenvectors of \( \Sigma \) and semi-axes proportional to \( \sqrt{\lambda_i} \).
Properties to have ready
| Property | Statement |
|---|---|
| linear maps | \( AX + b \sim N_q(A\mu + b, \; A\Sigma A^T) \) for any \( q \times p \) matrix \( A \) |
| marginals | every sub-vector is normal, with the corresponding sub-vector of \( \mu \) and sub-block of \( \Sigma \) |
| independence | for jointly normal blocks, \( X_1 \perp X_2 \iff \Sigma_{12} = 0 \) |
| standardising | \( \Sigma^{-1/2}(X - \mu) \sim N_p(0, I) \) |
| Mahalanobis | \( (X - \mu)^T\Sigma^{-1}(X - \mu) \sim \chi^2_p \) (\( \Sigma \) positive definite) |
| sums | independent \( N_p(\mu_i, \Sigma_i) \) add to \( N_p(\sum\mu_i, \sum\Sigma_i) \) |
| characterisation | \( \bar X \) and the sample covariance are independent for an iid normal sample |
The converse of “marginals normal” is false (the hook), and “uncorrelated implies independent” needs joint normality.
Conditional distributions
Partition \( X = \begin{pmatrix} X_1 \\ X_2 \end{pmatrix} \), \( \mu = \begin{pmatrix} \mu_1 \\ \mu_2 \end{pmatrix} \), \( \Sigma = \begin{pmatrix} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{pmatrix} \), with \( \Sigma_{22} \) invertible.
Three facts hide in this formula:
- The conditional mean is linear in \( x_2 \): the regression of \( X_1 \) on \( X_2 \) is exactly linear, with coefficient matrix \( \Sigma_{12}\Sigma_{22}^{-1} \).
- The conditional covariance \( \Sigma_{11\cdot2} \) (the Schur complement) does not depend on \( x_2 \): the regression is homoscedastic.
- \( X_1 - \Sigma_{12}\Sigma_{22}^{-1}X_2 \) is independent of \( X_2 \). This is the proof of the theorem in one line.
In two dimensions with correlation \( \rho \):
$$E(X_1 \mid X_2 = x_2) = \mu_1 + \rho\frac{\sigma_1}{\sigma_2}(x_2 - \mu_2), \qquad \operatorname{Var}(X_1 \mid X_2) = \sigma_1^2(1 - \rho^2).$$The precision matrix and conditional independence
Let \( \Sigma^{-1} = (\sigma^{ij}) \) be the precision (concentration) matrix. For a jointly normal vector,
$$X_i \perp X_j \mid \text{(all other coordinates)} \iff \sigma^{ij} = 0.$$Zeros of \( \Sigma \) give marginal independence; zeros of \( \Sigma^{-1} \) give conditional independence. They are different, as the Level 3 practice shows. The partial correlation is \( -\frac{\sigma^{ij}}{\sqrt{\sigma^{ii}\sigma^{jj}}} \) (see Partial and Multiple Correlation).
Quadratic Forms in Normal Variables
Moments (no normality needed)
For any random vector with \( E(X) = \mu \), \( \operatorname{Cov}(X) = \Sigma \), and symmetric \( A \):
$$\boxed{E(X^TAX) = \operatorname{tr}(A\Sigma) + \mu^TA\mu.}$$Proof: \( X^TAX = \operatorname{tr}(AXX^T) \) and \( E(XX^T) = \Sigma + \mu\mu^T \). With normality, \( \operatorname{Var}(X^TAX) = 2\operatorname{tr}(A\Sigma A\Sigma) + 4\mu^TA\Sigma A\mu \).
When is it chi-square?
- If \( X \sim N_n(0, I) \) and \( A \) is symmetric, then \( X^TAX \sim \chi^2_r \) iff \( A \) is idempotent (\( A^2 = A \)), and then \( r = \operatorname{rank}A = \operatorname{tr}A \).
- If \( X \sim N_n(\mu, \Sigma) \) with \( \Sigma \) positive definite, then \( X^TAX \) has a (non-central) chi-square distribution with \( r = \operatorname{rank}A \) degrees of freedom iff \( A\Sigma \) is idempotent (equivalently \( A\Sigma A = A \)). The non-centrality parameter is \( \lambda = \mu^TA\mu \) (some texts use \( \frac{1}{2}\mu^TA\mu \)); it is central iff \( \mu^TA\mu = 0 \).
Why idempotent: a symmetric \( A \) is \( Q\Lambda Q^T \); with \( Y = Q^TX \sim N(0, I) \), \( X^TAX = \sum\lambda_iY_i^2 \). This is chi-square exactly when every \( \lambda_i \) is 0 or 1.
With the convention \( \lambda = \mu^TA\mu \), a non-central \( \chi^2_r(\lambda) \) has mean \( r + \lambda \) and variance \( 2r + 4\lambda \).
Independence
Let \( X \sim N_n(\mu, \Sigma) \), with \( A, B \) symmetric.
- \( X^TAX \) and \( X^TBX \) are independent iff \( A\Sigma B = 0 \).
- A linear form \( LX \) and \( X^TAX \) are independent iff \( L\Sigma A = 0 \).
For \( \Sigma = \sigma^2I \) these read \( AB = 0 \) and \( LA = 0 \).
The classic application. For \( X_1, \dots, X_n \overset{\text{iid}}{\sim} N(\mu, \sigma^2) \), write \( \bar X = \frac{1}{n}\mathbf{1}^TX \) and \( (n - 1)S^2 = X^T\left(I - \frac{1}{n}J\right)X \). Since \( \mathbf{1}^T\left(I - \frac{1}{n}J\right) = 0 \), \( \bar X \) and \( S^2 \) are independent, and \( I - \frac{1}{n}J \) is idempotent of rank \( n - 1 \), so \( \frac{(n - 1)S^2}{\sigma^2} \sim \chi^2_{n - 1} \).
Cochran’s theorem
This is why the ANOVA sums of squares (treatments, error) are independent chi-squares under \( H_0 \): their matrices are projections that add to \( I - \frac{1}{n}J \), and their ranks add to \( n - 1 \).
The Wishart Distribution
If \( Z_1, \dots, Z_n \overset{\text{iid}}{\sim} N_p(0, \Sigma) \), the \( p \times p \) random matrix
$$W = \sum_{i=1}^n Z_iZ_i^T = Z^TZ \qquad (Z \text{ the } n \times p \text{ data matrix})$$has the Wishart distribution \( W_p(n, \Sigma) \) with \( n \) degrees of freedom and scale matrix \( \Sigma \). It is the matrix generalisation of \( \sigma^2\chi^2_n \): for \( p = 1 \), \( W_1(n, \sigma^2) = \sigma^2\chi^2_n \).
| Property | Statement |
|---|---|
| mean | \( E(W) = n\Sigma \) |
| entries | \( \operatorname{Var}(w_{ij}) = n(\sigma_{ij}^2 + \sigma_{ii}\sigma_{jj}) \); in particular \( w_{ii} \sim \sigma_{ii}\chi^2_n \) |
| additivity | independent \( W_p(n_1, \Sigma) + W_p(n_2, \Sigma) = W_p(n_1 + n_2, \Sigma) \) |
| linear transformation | \( AWA^T \sim W_q(n, A\Sigma A^T) \) for \( q \times p \) \( A \) |
| quadratic forms | \( \frac{a^TWa}{a^T\Sigma a} \sim \chi^2_n \) for fixed \( a \ne 0 \) |
| invertibility | \( W \) is non-singular with probability 1 iff \( n \ge p \) (\( \Sigma \) positive definite); it then has a density |
| inverse mean | \( E(W^{-1}) = \frac{\Sigma^{-1}}{n - p - 1} \) for \( n > p + 1 \) |
| generalised variance | \( \frac{\lvert W \rvert}{\lvert\Sigma\rvert} \) is distributed as a product of independent \( \chi^2_n, \chi^2_{n-1}, \dots, \chi^2_{n-p+1} \) |
| sample covariance | for an iid \( N_p(\mu, \Sigma) \) sample of size \( n \): \( (n - 1)S \sim W_p(n - 1, \Sigma) \), independent of \( \bar X \sim N_p(\mu, \frac{1}{n}\Sigma) \) |
Hotelling’s T²
The multivariate analogue of the one-sample t statistic. For an iid \( N_p(\mu, \Sigma) \) sample of size \( n > p \), testing \( H_0: \mu = \mu_0 \):
$$T^2 = n(\bar X - \mu_0)^TS^{-1}(\bar X - \mu_0), \qquad \boxed{\frac{n - p}{p(n - 1)}T^2 \sim F_{p,\; n - p} \text{ under } H_0.}$$For \( p = 1 \), \( T^2 = t^2 \) with \( t \sim t_{n - 1} \), and the scaling reduces to \( F_{1, n - 1} = t^2_{n - 1} \). \( T^2 \) is invariant under non-singular linear transformations of the data. The two-sample version uses the pooled covariance and \( \frac{n_1n_2}{n_1 + n_2} \) in place of \( n \).
graph LR
A["Z_i iid N_p(0, Sigma)"] --> B["a'Z_i: univariate normal"]
A --> C["Sum Z_i Z_i': Wishart W_p(n, Sigma)"]
B --> D["Sum of squares: sigma^2 chi^2_n"]
C --> D
C --> E["(n-1)S ~ W_p(n-1, Sigma)"]
E --> F["Hotelling T^2 = n (xbar - mu)' S^-1 (xbar - mu)
scaled to F_{p, n-p}"]Memory Tricks
- Jointly normal: zero covariance = independence. Zero precision = conditional independence. Read \( \Sigma \) for the first, \( \Sigma^{-1} \) for the second.
- Trace plus mean term. \( E(X^TAX) = \operatorname{tr}(A\Sigma) + \mu^TA\mu \): the variance part and the signal part.
- Idempotent means chi-square, rank means df. And for independence, sandwich \( \Sigma \): \( A\Sigma B = 0 \).
- Wishart is chi-square in matrix clothes. Mean \( n\Sigma \); \( p = 1 \) gives \( \sigma^2\chi^2_n \); lose one df for estimating the mean.
- Hotelling’s constant: (n − p) over p(n − 1). It turns \( T^2 \) into \( F_{p, n - p} \).
Don’t Confuse
| This | Is not this | The difference |
|---|---|---|
| each coordinate normal | jointly normal | the \( Y = SX \) example: normal marginals, \( X + Y \) not normal |
| uncorrelated (jointly normal) | uncorrelated (in general) | only joint normality turns zero covariance into independence |
| \( \Sigma_{ij} = 0 \): \( X_i \perp X_j \) | \( (\Sigma^{-1})_{ij} = 0 \): \( X_i \perp X_j \mid \) rest | marginal versus conditional independence |
| \( A \) idempotent (for \( N(0, I) \)) | \( A\Sigma \) idempotent (for \( N(\mu, \Sigma) \)) | the condition must include \( \Sigma \) |
| independence of forms: \( A\Sigma B = 0 \) | \( AB = 0 \) | the same only when \( \Sigma = \sigma^2I \) |
| \( W_p(n, \Sigma) \) from known mean | \( W_p(n - 1, \Sigma) \) from \( (n - 1)S \) | centring at \( \bar X \) costs one degree of freedom |
| non-centrality \( \mu^TA\mu \) | \( \frac{1}{2}\mu^TA\mu \) | a convention; check which the question uses |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C, one or more options may be correct.
Level 1: Foundation
- If \( X \sim N_p(\mu, \Sigma) \), what is the distribution of \( AX + b \)?
- What is \( E(W) \) for \( W \sim W_p(n, \Sigma) \)?
- If \( X \sim N_n(0, I) \), what is the distribution of \( X^TX \)?
Answers
- \( N_q(A\mu + b, A\Sigma A^T) \).
- \( n\Sigma \).
- \( \chi^2_n \) (here \( A = I \) is idempotent of rank \( n \)).
Level 2: Part B Style
Practice Q1. Let \( (X_1, X_2) \sim N_2\left(\begin{pmatrix} 1 \\ 2 \end{pmatrix}, \begin{pmatrix} 4 & 3 \\ 3 & 9 \end{pmatrix}\right) \). The conditional distribution of \( X_1 \) given \( X_2 = 5 \) is:
(a) \( N(2, 3) \) (b) \( N(2, 4) \) (c) \( N(4, 3) \) (d) \( N(1, 3) \)
Answer: (a)
Mean \( 1 + \frac{3}{9}(5 - 2) = 2 \); variance \( 4 - \frac{9}{9} = 3 \). Option (c) adds the whole deviation \( 5 - 2 = 3 \) to the mean, ignoring the regression coefficient \( \frac{\sigma_{12}}{\sigma_{22}} = \frac{1}{3} \); option (b) forgets to reduce the variance.
Practice Q2. Let \( X \sim N_3(0, \Sigma) \) with \( \Sigma = \begin{pmatrix} 2 & 1 & 0 \\ 1 & 2 & 1 \\ 0 & 1 & 2 \end{pmatrix} \). Then \( \operatorname{Var}(X_1 + X_2 + X_3) \) is:
(a) 6 (b) 8 (c) 10 (d) 12
Answer: (c)
\( \operatorname{Var}(\mathbf{1}^TX) = \mathbf{1}^T\Sigma\mathbf{1} \) = sum of all entries \( = 6 + 2(1 + 0 + 1) = 10 \).
Practice Q3. Let \( X \sim N_2(\mu, I) \) with \( \mu = (1, 1)^T \). Then \( E\left[(X_1 + X_2)^2\right] \) is:
(a) 2 (b) 4 (c) 6 (d) 8
Answer: (c)
\( (X_1 + X_2)^2 = X^TJX \), so \( E = \operatorname{tr}(J) + \mu^TJ\mu = 2 + 4 = 6 \). Directly: \( X_1 + X_2 \sim N(2, 2) \), \( E = 2 + 2^2 = 6 \).
Practice Q4. For a sample of size \( n = 20 \) from \( N_2(\mu, \Sigma) \), Hotelling’s \( T^2 \) for \( H_0: \mu = \mu_0 \) satisfies \( cT^2 \sim F_{2, 18} \) under \( H_0 \), where \( c \) is:
(a) \( \frac{18}{38} \) (b) \( \frac{38}{18} \) (c) \( \frac{1}{2} \) (d) \( \frac{19}{18} \)
Answer: (a)
\( c = \frac{n - p}{p(n - 1)} = \frac{18}{2 \cdot 19} = \frac{18}{38} = \frac{9}{19} \). Equivalently \( T^2 \sim \frac{38}{18}F_{2, 18} \), which is option (b) read the wrong way round.
Level 3: Part C Style
Practice Q5. Let \( X = (X_1, X_2, X_3)^T \sim N_3(0, \Sigma) \) with \( \Sigma = \begin{pmatrix} 2 & 1 & 0 \\ 1 & 2 & 1 \\ 0 & 1 & 2 \end{pmatrix} \). Which statements are true?
(a) \( X_1 \) and \( X_3 \) are independent. (b) \( X_1 \) and \( X_3 \) are conditionally independent given \( X_2 \). (c) \( X_1 + X_3 \) and \( X_1 - X_3 \) are independent. (d) \( X_2 \) and \( X_1 - X_3 \) are independent.
Answer: (a), (c) and (d)
All the pairs below are jointly normal, so zero covariance means independence.
- (a) True. \( \sigma_{13} = 0 \).
- (b) False. \( \operatorname{Cov}(X_1, X_3 \mid X_2) = \sigma_{13} - \frac{\sigma_{12}\sigma_{23}}{\sigma_{22}} = 0 - \frac{1}{2} \ne 0 \). Equivalently, \( \Sigma^{-1} = \frac{1}{4}\begin{pmatrix} 3 & -2 & 1 \\ -2 & 4 & -2 \\ 1 & -2 & 3 \end{pmatrix} \) has a non-zero \( (1, 3) \) entry.
- (c) True. \( \operatorname{Cov}(X_1 + X_3, X_1 - X_3) = \operatorname{Var}X_1 - \operatorname{Var}X_3 = 2 - 2 = 0 \).
- (d) True. \( \operatorname{Cov}(X_2, X_1 - X_3) = \sigma_{21} - \sigma_{23} = 1 - 1 = 0 \).
Practice Q6. Let \( X \sim N_3(0, I) \), \( Q_1 = \frac{(X_1 - X_2)^2}{2} \), \( Q_2 = \frac{(X_1 + X_2 - 2X_3)^2}{6} \). Which statements are true?
(a) \( Q_1 \sim \chi^2_1 \). (b) \( Q_1 \) and \( Q_2 \) are independent. (c) \( X^TX - Q_1 - Q_2 \sim \chi^2_1 \). (d) \( Q_1 + Q_2 \sim \chi^2_1 \).
Answer: (a), (b) and (c)
- (a) True. \( \frac{X_1 - X_2}{\sqrt{2}} \sim N(0, 1) \).
- (b) True. The matrices \( A_1 = \frac{v_1v_1^T}{2} \), \( A_2 = \frac{v_2v_2^T}{6} \) satisfy \( A_1A_2 = 0 \) because \( v_1^Tv_2 = 1 - 1 + 0 = 0 \).
- (c) True. The remainder is \( \frac{(X_1 + X_2 + X_3)^2}{3} \), a rank-one projection; Cochran’s theorem with ranks \( 1 + 1 + 1 = 3 \).
- (d) False. A sum of two independent \( \chi^2_1 \) variables is \( \chi^2_2 \).
Practice Q7. Let \( W \sim W_3(10, \Sigma) \) with \( \Sigma \) positive definite. Which statements are true?
(a) \( E(W) = 10\Sigma \). (b) \( W \) is non-singular with probability 1. (c) \( E(W^{-1}) = \frac{1}{6}\Sigma^{-1} \). (d) For a fixed non-zero \( a \in \mathbb{R}^3 \), \( \frac{a^TWa}{a^T\Sigma a} \sim \chi^2_{10} \).
Answer: all four, (a), (b), (c) and (d)
- (a) True. \( E(W) = n\Sigma \).
- (b) True. \( n = 10 \ge p = 3 \).
- (c) True. \( E(W^{-1}) = \frac{\Sigma^{-1}}{n - p - 1} = \frac{\Sigma^{-1}}{6} \), valid as \( n > p + 1 \).
- (d) True. \( a^TWa = \sum(a^TZ_i)^2 \) with \( a^TZ_i \overset{\text{iid}}{\sim} N(0, a^T\Sigma a) \).
Quick Revision Table
| Idea | Fact |
|---|---|
| Definition | every \( a^TX \) normal; mgf \( e^{t^T\mu + t^T\Sigma t/2} \) |
| Linear map | \( AX + b \sim N(A\mu + b, A\Sigma A^T) \) |
| Independence | jointly normal: \( \Sigma_{12} = 0 \iff \) independent |
| Conditional | mean \( \mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(x_2 - \mu_2) \), covariance \( \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21} \) |
| Conditional independence | \( (\Sigma^{-1})_{ij} = 0 \) |
| Mahalanobis | \( (X - \mu)^T\Sigma^{-1}(X - \mu) \sim \chi^2_p \) |
| \( E(X^TAX) \) | \( \operatorname{tr}(A\Sigma) + \mu^TA\mu \) |
| Chi-square form | \( A\Sigma \) idempotent; df = rank \( A \) |
| Independent forms | \( A\Sigma B = 0 \); linear and quadratic: \( L\Sigma A = 0 \) |
| Cochran | \( \sum A_i = I \), \( \sum r_i = n \) \( \iff \) independent \( \chi^2_{r_i} \) |
| Wishart | \( \sum Z_iZ_i^T \); mean \( n\Sigma \); \( (n - 1)S \sim W_p(n - 1, \Sigma) \) |
| \( E(W^{-1}) \) | \( \frac{\Sigma^{-1}}{n - p - 1} \) |
| Hotelling | \( \frac{n - p}{p(n - 1)}T^2 \sim F_{p, n - p} \) |
Key Takeaways
What’s Next
- Partial and Multiple Correlation: the conditional covariance turned into correlations.
- PCA, Discriminant, Cluster and Canonical Correlation Analysis: eigen-analysis of \( \Sigma \) and of its blocks.
- Quadratic Forms: the linear algebra behind every result on this page.
- Back to the Linear Models and Multivariate Analysis chapter overview.