Skip to content

Multivariate Normal, Wishart and Quadratic Forms

CSIR NET statistics: multivariate normal properties, marginals and conditionals, quadratic forms and chi-square, Cochran's theorem, Wishart and Hotelling T².

  • 17 min read
  • Updated
On this page · 12 sections
  1. Prerequisites
  2. The Hook: Two Normals That Are Not Jointly Normal
  3. The Multivariate Normal Distribution
  4. Quadratic Forms in Normal Variables
  5. The Wishart Distribution
  6. Memory Tricks
  7. Don’t Confuse
  8. Common Mistakes
  9. Practice
  10. Quick Revision Table
  11. Key Takeaways
  12. What’s Next

Prerequisites

The Hook: Two Normals That Are Not Jointly Normal

The Multivariate Normal Distribution

Multivariate normal

A random vector \( X \in \mathbb{R}^p \) is multivariate normal, \( X \sim N_p(\mu, \Sigma) \), if \( a^TX \) is univariate normal (possibly degenerate) for every \( a \in \mathbb{R}^p \). Then \( E(X) = \mu \), \( \operatorname{Cov}(X) = \Sigma \) (symmetric, positive semidefinite) and the moment generating function is

$$M_X(t) = \exp\left(t^T\mu + \tfrac{1}{2}t^T\Sigma t\right).$$

If \( \Sigma \) is positive definite, \( X \) has the density

$$f(x) = \frac{1}{(2\pi)^{p/2}\lvert\Sigma\rvert^{1/2}}\exp\left(-\tfrac{1}{2}(x - \mu)^T\Sigma^{-1}(x - \mu)\right).$$

If \( \Sigma \) is singular, there is no density on \( \mathbb{R}^p \): \( X \) lives on the affine subspace \( \mu + \mathcal{C}(\Sigma) \).

The density is constant on the ellipsoids \( (x - \mu)^T\Sigma^{-1}(x - \mu) = c \), with axes along the eigenvectors of \( \Sigma \) and semi-axes proportional to \( \sqrt{\lambda_i} \).

Properties to have ready

PropertyStatement
linear maps\( AX + b \sim N_q(A\mu + b, \; A\Sigma A^T) \) for any \( q \times p \) matrix \( A \)
marginalsevery sub-vector is normal, with the corresponding sub-vector of \( \mu \) and sub-block of \( \Sigma \)
independencefor jointly normal blocks, \( X_1 \perp X_2 \iff \Sigma_{12} = 0 \)
standardising\( \Sigma^{-1/2}(X - \mu) \sim N_p(0, I) \)
Mahalanobis\( (X - \mu)^T\Sigma^{-1}(X - \mu) \sim \chi^2_p \) (\( \Sigma \) positive definite)
sumsindependent \( N_p(\mu_i, \Sigma_i) \) add to \( N_p(\sum\mu_i, \sum\Sigma_i) \)
characterisation\( \bar X \) and the sample covariance are independent for an iid normal sample

The converse of “marginals normal” is false (the hook), and “uncorrelated implies independent” needs joint normality.

Conditional distributions

Partition \( X = \begin{pmatrix} X_1 \\ X_2 \end{pmatrix} \), \( \mu = \begin{pmatrix} \mu_1 \\ \mu_2 \end{pmatrix} \), \( \Sigma = \begin{pmatrix} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{pmatrix} \), with \( \Sigma_{22} \) invertible.

Conditional distribution
$$X_1 \mid X_2 = x_2 \;\sim\; N\left(\mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(x_2 - \mu_2), \;\; \Sigma_{11\cdot2}\right), \qquad \Sigma_{11\cdot2} = \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}.$$

Three facts hide in this formula:

  1. The conditional mean is linear in \( x_2 \): the regression of \( X_1 \) on \( X_2 \) is exactly linear, with coefficient matrix \( \Sigma_{12}\Sigma_{22}^{-1} \).
  2. The conditional covariance \( \Sigma_{11\cdot2} \) (the Schur complement) does not depend on \( x_2 \): the regression is homoscedastic.
  3. \( X_1 - \Sigma_{12}\Sigma_{22}^{-1}X_2 \) is independent of \( X_2 \). This is the proof of the theorem in one line.

In two dimensions with correlation \( \rho \):

$$E(X_1 \mid X_2 = x_2) = \mu_1 + \rho\frac{\sigma_1}{\sigma_2}(x_2 - \mu_2), \qquad \operatorname{Var}(X_1 \mid X_2) = \sigma_1^2(1 - \rho^2).$$

The precision matrix and conditional independence

Let \( \Sigma^{-1} = (\sigma^{ij}) \) be the precision (concentration) matrix. For a jointly normal vector,

$$X_i \perp X_j \mid \text{(all other coordinates)} \iff \sigma^{ij} = 0.$$

Zeros of \( \Sigma \) give marginal independence; zeros of \( \Sigma^{-1} \) give conditional independence. They are different, as the Level 3 practice shows. The partial correlation is \( -\frac{\sigma^{ij}}{\sqrt{\sigma^{ii}\sigma^{jj}}} \) (see Partial and Multiple Correlation).

Quadratic Forms in Normal Variables

Moments (no normality needed)

For any random vector with \( E(X) = \mu \), \( \operatorname{Cov}(X) = \Sigma \), and symmetric \( A \):

$$\boxed{E(X^TAX) = \operatorname{tr}(A\Sigma) + \mu^TA\mu.}$$

Proof: \( X^TAX = \operatorname{tr}(AXX^T) \) and \( E(XX^T) = \Sigma + \mu\mu^T \). With normality, \( \operatorname{Var}(X^TAX) = 2\operatorname{tr}(A\Sigma A\Sigma) + 4\mu^TA\Sigma A\mu \).

When is it chi-square?

Chi-square quadratic forms
  1. If \( X \sim N_n(0, I) \) and \( A \) is symmetric, then \( X^TAX \sim \chi^2_r \) iff \( A \) is idempotent (\( A^2 = A \)), and then \( r = \operatorname{rank}A = \operatorname{tr}A \).
  2. If \( X \sim N_n(\mu, \Sigma) \) with \( \Sigma \) positive definite, then \( X^TAX \) has a (non-central) chi-square distribution with \( r = \operatorname{rank}A \) degrees of freedom iff \( A\Sigma \) is idempotent (equivalently \( A\Sigma A = A \)). The non-centrality parameter is \( \lambda = \mu^TA\mu \) (some texts use \( \frac{1}{2}\mu^TA\mu \)); it is central iff \( \mu^TA\mu = 0 \).

Why idempotent: a symmetric \( A \) is \( Q\Lambda Q^T \); with \( Y = Q^TX \sim N(0, I) \), \( X^TAX = \sum\lambda_iY_i^2 \). This is chi-square exactly when every \( \lambda_i \) is 0 or 1.

With the convention \( \lambda = \mu^TA\mu \), a non-central \( \chi^2_r(\lambda) \) has mean \( r + \lambda \) and variance \( 2r + 4\lambda \).

Independence

Independence of forms (Craig's theorem and its linear version)

Let \( X \sim N_n(\mu, \Sigma) \), with \( A, B \) symmetric.

  • \( X^TAX \) and \( X^TBX \) are independent iff \( A\Sigma B = 0 \).
  • A linear form \( LX \) and \( X^TAX \) are independent iff \( L\Sigma A = 0 \).

For \( \Sigma = \sigma^2I \) these read \( AB = 0 \) and \( LA = 0 \).

The classic application. For \( X_1, \dots, X_n \overset{\text{iid}}{\sim} N(\mu, \sigma^2) \), write \( \bar X = \frac{1}{n}\mathbf{1}^TX \) and \( (n - 1)S^2 = X^T\left(I - \frac{1}{n}J\right)X \). Since \( \mathbf{1}^T\left(I - \frac{1}{n}J\right) = 0 \), \( \bar X \) and \( S^2 \) are independent, and \( I - \frac{1}{n}J \) is idempotent of rank \( n - 1 \), so \( \frac{(n - 1)S^2}{\sigma^2} \sim \chi^2_{n - 1} \).

Cochran’s theorem

Cochran's theorem
Let \( X \sim N_n(0, I) \) and \( X^TX = \sum_{i=1}^k X^TA_iX \), with symmetric \( A_i \) of rank \( r_i \) and \( \sum A_i = I \). Then the forms \( X^TA_iX \) are independent with \( X^TA_iX \sim \chi^2_{r_i} \) iff \( \sum r_i = n \).

This is why the ANOVA sums of squares (treatments, error) are independent chi-squares under \( H_0 \): their matrices are projections that add to \( I - \frac{1}{n}J \), and their ranks add to \( n - 1 \).

The Wishart Distribution

Wishart distribution

If \( Z_1, \dots, Z_n \overset{\text{iid}}{\sim} N_p(0, \Sigma) \), the \( p \times p \) random matrix

$$W = \sum_{i=1}^n Z_iZ_i^T = Z^TZ \qquad (Z \text{ the } n \times p \text{ data matrix})$$

has the Wishart distribution \( W_p(n, \Sigma) \) with \( n \) degrees of freedom and scale matrix \( \Sigma \). It is the matrix generalisation of \( \sigma^2\chi^2_n \): for \( p = 1 \), \( W_1(n, \sigma^2) = \sigma^2\chi^2_n \).

PropertyStatement
mean\( E(W) = n\Sigma \)
entries\( \operatorname{Var}(w_{ij}) = n(\sigma_{ij}^2 + \sigma_{ii}\sigma_{jj}) \); in particular \( w_{ii} \sim \sigma_{ii}\chi^2_n \)
additivityindependent \( W_p(n_1, \Sigma) + W_p(n_2, \Sigma) = W_p(n_1 + n_2, \Sigma) \)
linear transformation\( AWA^T \sim W_q(n, A\Sigma A^T) \) for \( q \times p \) \( A \)
quadratic forms\( \frac{a^TWa}{a^T\Sigma a} \sim \chi^2_n \) for fixed \( a \ne 0 \)
invertibility\( W \) is non-singular with probability 1 iff \( n \ge p \) (\( \Sigma \) positive definite); it then has a density
inverse mean\( E(W^{-1}) = \frac{\Sigma^{-1}}{n - p - 1} \) for \( n > p + 1 \)
generalised variance\( \frac{\lvert W \rvert}{\lvert\Sigma\rvert} \) is distributed as a product of independent \( \chi^2_n, \chi^2_{n-1}, \dots, \chi^2_{n-p+1} \)
sample covariancefor an iid \( N_p(\mu, \Sigma) \) sample of size \( n \): \( (n - 1)S \sim W_p(n - 1, \Sigma) \), independent of \( \bar X \sim N_p(\mu, \frac{1}{n}\Sigma) \)

Hotelling’s T²

The multivariate analogue of the one-sample t statistic. For an iid \( N_p(\mu, \Sigma) \) sample of size \( n > p \), testing \( H_0: \mu = \mu_0 \):

$$T^2 = n(\bar X - \mu_0)^TS^{-1}(\bar X - \mu_0), \qquad \boxed{\frac{n - p}{p(n - 1)}T^2 \sim F_{p,\; n - p} \text{ under } H_0.}$$

For \( p = 1 \), \( T^2 = t^2 \) with \( t \sim t_{n - 1} \), and the scaling reduces to \( F_{1, n - 1} = t^2_{n - 1} \). \( T^2 \) is invariant under non-singular linear transformations of the data. The two-sample version uses the pooled covariance and \( \frac{n_1n_2}{n_1 + n_2} \) in place of \( n \).

graph LR
    A["Z_i iid N_p(0, Sigma)"] --> B["a'Z_i: univariate normal"]
    A --> C["Sum Z_i Z_i': Wishart W_p(n, Sigma)"]
    B --> D["Sum of squares: sigma^2 chi^2_n"]
    C --> D
    C --> E["(n-1)S ~ W_p(n-1, Sigma)"]
    E --> F["Hotelling T^2 = n (xbar - mu)' S^-1 (xbar - mu)
scaled to F_{p, n-p}"]

Memory Tricks

  • Jointly normal: zero covariance = independence. Zero precision = conditional independence. Read \( \Sigma \) for the first, \( \Sigma^{-1} \) for the second.
  • Trace plus mean term. \( E(X^TAX) = \operatorname{tr}(A\Sigma) + \mu^TA\mu \): the variance part and the signal part.
  • Idempotent means chi-square, rank means df. And for independence, sandwich \( \Sigma \): \( A\Sigma B = 0 \).
  • Wishart is chi-square in matrix clothes. Mean \( n\Sigma \); \( p = 1 \) gives \( \sigma^2\chi^2_n \); lose one df for estimating the mean.
  • Hotelling’s constant: (n − p) over p(n − 1). It turns \( T^2 \) into \( F_{p, n - p} \).

Don’t Confuse

ThisIs not thisThe difference
each coordinate normaljointly normalthe \( Y = SX \) example: normal marginals, \( X + Y \) not normal
uncorrelated (jointly normal)uncorrelated (in general)only joint normality turns zero covariance into independence
\( \Sigma_{ij} = 0 \): \( X_i \perp X_j \)\( (\Sigma^{-1})_{ij} = 0 \): \( X_i \perp X_j \mid \) restmarginal versus conditional independence
\( A \) idempotent (for \( N(0, I) \))\( A\Sigma \) idempotent (for \( N(\mu, \Sigma) \))the condition must include \( \Sigma \)
independence of forms: \( A\Sigma B = 0 \)\( AB = 0 \)the same only when \( \Sigma = \sigma^2I \)
\( W_p(n, \Sigma) \) from known mean\( W_p(n - 1, \Sigma) \) from \( (n - 1)S \)centring at \( \bar X \) costs one degree of freedom
non-centrality \( \mu^TA\mu \)\( \frac{1}{2}\mu^TA\mu \)a convention; check which the question uses

Common Mistakes

Practice

All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C, one or more options may be correct.

Level 1: Foundation

  1. If \( X \sim N_p(\mu, \Sigma) \), what is the distribution of \( AX + b \)?
  2. What is \( E(W) \) for \( W \sim W_p(n, \Sigma) \)?
  3. If \( X \sim N_n(0, I) \), what is the distribution of \( X^TX \)?
Answers
  1. \( N_q(A\mu + b, A\Sigma A^T) \).
  2. \( n\Sigma \).
  3. \( \chi^2_n \) (here \( A = I \) is idempotent of rank \( n \)).

Level 2: Part B Style

Practice Q1. Let \( (X_1, X_2) \sim N_2\left(\begin{pmatrix} 1 \\ 2 \end{pmatrix}, \begin{pmatrix} 4 & 3 \\ 3 & 9 \end{pmatrix}\right) \). The conditional distribution of \( X_1 \) given \( X_2 = 5 \) is:

(a) \( N(2, 3) \) (b) \( N(2, 4) \) (c) \( N(4, 3) \) (d) \( N(1, 3) \)

Answer: (a)

Mean \( 1 + \frac{3}{9}(5 - 2) = 2 \); variance \( 4 - \frac{9}{9} = 3 \). Option (c) adds the whole deviation \( 5 - 2 = 3 \) to the mean, ignoring the regression coefficient \( \frac{\sigma_{12}}{\sigma_{22}} = \frac{1}{3} \); option (b) forgets to reduce the variance.

Practice Q2. Let \( X \sim N_3(0, \Sigma) \) with \( \Sigma = \begin{pmatrix} 2 & 1 & 0 \\ 1 & 2 & 1 \\ 0 & 1 & 2 \end{pmatrix} \). Then \( \operatorname{Var}(X_1 + X_2 + X_3) \) is:

(a) 6 (b) 8 (c) 10 (d) 12

Answer: (c)

\( \operatorname{Var}(\mathbf{1}^TX) = \mathbf{1}^T\Sigma\mathbf{1} \) = sum of all entries \( = 6 + 2(1 + 0 + 1) = 10 \).

Practice Q3. Let \( X \sim N_2(\mu, I) \) with \( \mu = (1, 1)^T \). Then \( E\left[(X_1 + X_2)^2\right] \) is:

(a) 2 (b) 4 (c) 6 (d) 8

Answer: (c)

\( (X_1 + X_2)^2 = X^TJX \), so \( E = \operatorname{tr}(J) + \mu^TJ\mu = 2 + 4 = 6 \). Directly: \( X_1 + X_2 \sim N(2, 2) \), \( E = 2 + 2^2 = 6 \).

Practice Q4. For a sample of size \( n = 20 \) from \( N_2(\mu, \Sigma) \), Hotelling’s \( T^2 \) for \( H_0: \mu = \mu_0 \) satisfies \( cT^2 \sim F_{2, 18} \) under \( H_0 \), where \( c \) is:

(a) \( \frac{18}{38} \) (b) \( \frac{38}{18} \) (c) \( \frac{1}{2} \) (d) \( \frac{19}{18} \)

Answer: (a)

\( c = \frac{n - p}{p(n - 1)} = \frac{18}{2 \cdot 19} = \frac{18}{38} = \frac{9}{19} \). Equivalently \( T^2 \sim \frac{38}{18}F_{2, 18} \), which is option (b) read the wrong way round.

Level 3: Part C Style

Practice Q5. Let \( X = (X_1, X_2, X_3)^T \sim N_3(0, \Sigma) \) with \( \Sigma = \begin{pmatrix} 2 & 1 & 0 \\ 1 & 2 & 1 \\ 0 & 1 & 2 \end{pmatrix} \). Which statements are true?

(a) \( X_1 \) and \( X_3 \) are independent. (b) \( X_1 \) and \( X_3 \) are conditionally independent given \( X_2 \). (c) \( X_1 + X_3 \) and \( X_1 - X_3 \) are independent. (d) \( X_2 \) and \( X_1 - X_3 \) are independent.

Answer: (a), (c) and (d)

All the pairs below are jointly normal, so zero covariance means independence.

  • (a) True. \( \sigma_{13} = 0 \).
  • (b) False. \( \operatorname{Cov}(X_1, X_3 \mid X_2) = \sigma_{13} - \frac{\sigma_{12}\sigma_{23}}{\sigma_{22}} = 0 - \frac{1}{2} \ne 0 \). Equivalently, \( \Sigma^{-1} = \frac{1}{4}\begin{pmatrix} 3 & -2 & 1 \\ -2 & 4 & -2 \\ 1 & -2 & 3 \end{pmatrix} \) has a non-zero \( (1, 3) \) entry.
  • (c) True. \( \operatorname{Cov}(X_1 + X_3, X_1 - X_3) = \operatorname{Var}X_1 - \operatorname{Var}X_3 = 2 - 2 = 0 \).
  • (d) True. \( \operatorname{Cov}(X_2, X_1 - X_3) = \sigma_{21} - \sigma_{23} = 1 - 1 = 0 \).

Practice Q6. Let \( X \sim N_3(0, I) \), \( Q_1 = \frac{(X_1 - X_2)^2}{2} \), \( Q_2 = \frac{(X_1 + X_2 - 2X_3)^2}{6} \). Which statements are true?

(a) \( Q_1 \sim \chi^2_1 \). (b) \( Q_1 \) and \( Q_2 \) are independent. (c) \( X^TX - Q_1 - Q_2 \sim \chi^2_1 \). (d) \( Q_1 + Q_2 \sim \chi^2_1 \).

Answer: (a), (b) and (c)
  • (a) True. \( \frac{X_1 - X_2}{\sqrt{2}} \sim N(0, 1) \).
  • (b) True. The matrices \( A_1 = \frac{v_1v_1^T}{2} \), \( A_2 = \frac{v_2v_2^T}{6} \) satisfy \( A_1A_2 = 0 \) because \( v_1^Tv_2 = 1 - 1 + 0 = 0 \).
  • (c) True. The remainder is \( \frac{(X_1 + X_2 + X_3)^2}{3} \), a rank-one projection; Cochran’s theorem with ranks \( 1 + 1 + 1 = 3 \).
  • (d) False. A sum of two independent \( \chi^2_1 \) variables is \( \chi^2_2 \).

Practice Q7. Let \( W \sim W_3(10, \Sigma) \) with \( \Sigma \) positive definite. Which statements are true?

(a) \( E(W) = 10\Sigma \). (b) \( W \) is non-singular with probability 1. (c) \( E(W^{-1}) = \frac{1}{6}\Sigma^{-1} \). (d) For a fixed non-zero \( a \in \mathbb{R}^3 \), \( \frac{a^TWa}{a^T\Sigma a} \sim \chi^2_{10} \).

Answer: all four, (a), (b), (c) and (d)
  • (a) True. \( E(W) = n\Sigma \).
  • (b) True. \( n = 10 \ge p = 3 \).
  • (c) True. \( E(W^{-1}) = \frac{\Sigma^{-1}}{n - p - 1} = \frac{\Sigma^{-1}}{6} \), valid as \( n > p + 1 \).
  • (d) True. \( a^TWa = \sum(a^TZ_i)^2 \) with \( a^TZ_i \overset{\text{iid}}{\sim} N(0, a^T\Sigma a) \).

Quick Revision Table

IdeaFact
Definitionevery \( a^TX \) normal; mgf \( e^{t^T\mu + t^T\Sigma t/2} \)
Linear map\( AX + b \sim N(A\mu + b, A\Sigma A^T) \)
Independencejointly normal: \( \Sigma_{12} = 0 \iff \) independent
Conditionalmean \( \mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(x_2 - \mu_2) \), covariance \( \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21} \)
Conditional independence\( (\Sigma^{-1})_{ij} = 0 \)
Mahalanobis\( (X - \mu)^T\Sigma^{-1}(X - \mu) \sim \chi^2_p \)
\( E(X^TAX) \)\( \operatorname{tr}(A\Sigma) + \mu^TA\mu \)
Chi-square form\( A\Sigma \) idempotent; df = rank \( A \)
Independent forms\( A\Sigma B = 0 \); linear and quadratic: \( L\Sigma A = 0 \)
Cochran\( \sum A_i = I \), \( \sum r_i = n \) \( \iff \) independent \( \chi^2_{r_i} \)
Wishart\( \sum Z_iZ_i^T \); mean \( n\Sigma \); \( (n - 1)S \sim W_p(n - 1, \Sigma) \)
\( E(W^{-1}) \)\( \frac{\Sigma^{-1}}{n - p - 1} \)
Hotelling\( \frac{n - p}{p(n - 1)}T^2 \sim F_{p, n - p} \)

Key Takeaways

What’s Next