Skip to content

Gauss–Markov Models, BLUE and Linear Hypotheses

CSIR NET statistics: the Gauss–Markov linear model, normal equations, estimability, BLUE, estimating sigma squared and the F test for linear hypotheses.

  • 19 min read
  • Updated
On this page · 14 sections
  1. Prerequisites
  2. The Hook: Weigh Them Together
  3. The Gauss–Markov Model
  4. Estimability
  5. The Gauss–Markov Theorem
  6. Estimating \( \sigma^2 \)
  7. Testing Linear Hypotheses
  8. Memory Tricks
  9. Don’t Confuse
  10. Common Mistakes
  11. Practice
  12. Quick Revision Table
  13. Key Takeaways
  14. What’s Next

Prerequisites

The Hook: Weigh Them Together

The Gauss–Markov Model

Gauss–Markov linear model
$$Y = X\beta + \varepsilon, \qquad E(\varepsilon) = 0, \qquad \operatorname{Var}(\varepsilon) = \sigma^2 I_n,$$

where \( Y \) is an \( n \times 1 \) vector of observations, \( X \) is a known \( n \times p \) design matrix of rank \( r \le p \), \( \beta \) is a \( p \times 1 \) vector of unknown parameters and \( \sigma^2 > 0 \) is unknown. The errors are uncorrelated with equal variance. Normality is not assumed. Adding \( \varepsilon \sim N_n(0, \sigma^2 I) \) gives the normal linear model, needed for exact tests.

The model is full rank when \( r = p \) (regression with independent columns) and less than full rank when \( r < p \) (ANOVA models with an overall mean and effects, such as \( y_{ij} = \mu + \alpha_i + \varepsilon_{ij} \)).

Least squares and the normal equations

Least squares minimises \( S(\beta) = (Y - X\beta)^T(Y - X\beta) \). Setting the gradient to zero gives the normal equations

$$\boxed{X^TX\beta = X^TY}.$$
  • They are always consistent: \( X^TY \) lies in the column space of \( X^T \), which equals the column space of \( X^TX \).
  • Full rank: unique solution \( \hat\beta = (X^TX)^{-1}X^TY \).
  • Less than full rank: infinitely many solutions \( \hat\beta = (X^TX)^-X^TY \), one for each generalised inverse \( (X^TX)^- \). The fitted vector \( X\hat\beta \) is the same for all of them.

The fitted vector is the orthogonal projection of \( Y \) onto the column space \( \mathcal{C}(X) \):

$$\hat Y = X\hat\beta = PY, \qquad P = X(X^TX)^-X^T.$$
Property of \( P \) (the hat or projection matrix)Why it matters
\( P^T = P \), \( P^2 = P \)orthogonal projection onto \( \mathcal{C}(X) \)
\( PX = X \)fitted values of an exact fit are exact
\( P \) does not depend on the choice of g-inverse\( \hat Y \) and residuals are unique
\( \operatorname{rank} P = \operatorname{tr} P = r \)model degrees of freedom
\( I - P \) is also a symmetric idempotent, rank \( n - r \)residual degrees of freedom
residuals \( e = (I - P)Y \) satisfy \( X^Te = 0 \)residuals are orthogonal to every column of \( X \)

Estimability

Estimable function
A linear parametric function \( \lambda^T\beta \) is estimable if it has a linear unbiased estimator: some \( a \in \mathbb{R}^n \) with \( E(a^TY) = \lambda^T\beta \) for every \( \beta \).

Since \( E(a^TY) = a^TX\beta \), the condition is \( a^TX = \lambda^T \). So

$$\boxed{\lambda^T\beta \text{ estimable} \iff \lambda^T \in \text{row space of } X \iff \lambda^T(X^TX)^-X^TX = \lambda^T.}$$

Useful consequences:

  • Every \( E(y_i) = x_i^T\beta \) (each row of \( X\beta \)) is estimable, and so is every linear combination of them.
  • There are exactly \( r = \operatorname{rank} X \) linearly independent estimable functions.
  • In a full-rank model every \( \lambda^T\beta \) is estimable.
  • If \( r < p \), the individual parameters are typically not estimable.

The Gauss–Markov Theorem

Gauss–Markov theorem

In the model \( E(Y) = X\beta \), \( \operatorname{Var}(Y) = \sigma^2I \), let \( \lambda^T\beta \) be estimable and \( \hat\beta \) any solution of the normal equations. Then \( \lambda^T\hat\beta \) is the same for every such solution, is unbiased for \( \lambda^T\beta \), and has the smallest variance among all linear unbiased estimators. It is the unique BLUE (best linear unbiased estimator), with

$$\operatorname{Var}(\lambda^T\hat\beta) = \sigma^2\lambda^T(X^TX)^-\lambda.$$

In the full-rank case, \( \operatorname{Var}(\hat\beta) = \sigma^2(X^TX)^{-1} \).

Read the fine print, because Part C does:

  • Linear and unbiased are both restrictions. A biased estimator (ridge regression) or a non-linear one can have smaller mean squared error.
  • No normality is needed. Only the first two moments enter.
  • Under normality, \( \lambda^T\hat\beta \) is also the MLE and the UMVUE among all unbiased estimators, not only linear ones.
  • The equal-variance, uncorrelated assumption matters. If \( \operatorname{Var}(\varepsilon) = \sigma^2V \) with known positive definite \( V \ne I \), ordinary least squares is still unbiased but no longer best. The BLUE is the generalised (Aitken) least squares estimator $$\hat\beta_{\text{GLS}} = (X^TV^{-1}X)^{-1}X^TV^{-1}Y, \qquad \operatorname{Var} = \sigma^2(X^TV^{-1}X)^{-1}.$$ With \( V = \operatorname{diag}(v_i) \) this is weighted least squares, with weights \( \frac{1}{v_i} \).

Estimating \( \sigma^2 \)

The residual (error) sum of squares is a quadratic form:

$$\text{SSE} = e^Te = Y^T(I - P)Y = Y^TY - \hat\beta^TX^TY.$$

Using \( E(Y^TAY) = \operatorname{tr}(A\Sigma) + \mu^TA\mu \) with \( A = I - P \), \( \Sigma = \sigma^2I \), \( \mu = X\beta \) and \( (I - P)X = 0 \):

$$E(\text{SSE}) = \sigma^2\operatorname{tr}(I - P) = (n - r)\sigma^2, \qquad \boxed{s^2 = \hat\sigma^2 = \frac{\text{SSE}}{n - r}} \text{ is unbiased.}$$

The divisor is \( n - r \) (rank), not \( n - p \) (number of parameters), when the model is not of full rank. The MLE under normality divides by \( n \) and is biased.

Under normality (\( \varepsilon \sim N_n(0, \sigma^2I) \)):

  • \( \lambda^T\hat\beta \sim N\left(\lambda^T\beta, \; \sigma^2\lambda^T(X^TX)^-\lambda\right) \) for estimable \( \lambda^T\beta \);
  • \( \dfrac{\text{SSE}}{\sigma^2} \sim \chi^2_{n - r} \);
  • \( \hat Y = PY \) and \( \text{SSE} \) are independent, because \( P(I - P) = 0 \);
  • hence \( t = \dfrac{\lambda^T\hat\beta - \lambda^T\beta}{s\sqrt{\lambda^T(X^TX)^-\lambda}} \sim t_{n - r} \), the basis of confidence intervals.

In the weighing example, the residuals are \( (-\frac{1}{3}, -\frac{1}{3}, \frac{1}{3}) \), \( \text{SSE} = \frac{1}{3} \), \( n - r = 1 \), so \( s^2 = \frac{1}{3} \).

Testing Linear Hypotheses

A linear hypothesis is \( H_0: L\beta = c \), with \( L \) a \( q \times p \) matrix of rank \( q \). It is testable when every row of \( L\beta \) is estimable.

The general linear hypothesis F test

In the normal linear model with \( \operatorname{rank} X = r \), for a testable \( H_0: L\beta = c \) of rank \( q \):

$$F = \frac{(L\hat\beta - c)^T\left[L(X^TX)^-L^T\right]^{-1}(L\hat\beta - c)/q}{\text{SSE}/(n - r)} \sim F_{q,\; n - r} \quad \text{under } H_0.$$

Equivalently, if \( \text{SSE}_H \) is the residual sum of squares when the model is fitted subject to \( H_0 \),

$$F = \frac{(\text{SSE}_H - \text{SSE})/q}{\text{SSE}/(n - r)}.$$

Reject \( H_0 \) for large \( F \). Under the alternative, \( F \) has a non-central F distribution.

The second form is the one to remember. It says: impose the hypothesis, refit, and see how much the residual sum of squares grows per constraint, measured against the error variance. Every ANOVA F test, the overall regression F test and the partial F test for dropping variables are special cases.

graph TD
    A["Model Y = X beta + e
rank X = r"] --> B["Solve X'X b = X'Y
(any g-inverse)"] B --> C{"Is lambda' beta in
row space of X?"} C -->|no| D["Not estimable:
no unbiased linear estimator"] C -->|yes| E["BLUE = lambda' b
Var = sigma^2 lambda'(X'X)^- lambda"] B --> F["SSE = Y'(I-P)Y
s^2 = SSE/(n-r)"] E --> G["Test L beta = c:
F = ((SSE_H - SSE)/q) / (SSE/(n-r))"] F --> G

Restricted least squares

Minimising \( S(\beta) \) subject to \( L\beta = c \) (full rank \( X \)) gives

$$\hat\beta_H = \hat\beta - (X^TX)^{-1}L^T\left[L(X^TX)^{-1}L^T\right]^{-1}(L\hat\beta - c),$$

and \( \text{SSE}_H - \text{SSE} = (L\hat\beta - c)^T[L(X^TX)^{-1}L^T]^{-1}(L\hat\beta - c) \ge 0 \). That identity is why the two forms of the F statistic coincide.

Memory Tricks

  • “Estimable = row of X, or built from rows.” If you cannot write \( \lambda^T \) as a combination of the rows of \( X \), stop: no unbiased linear estimator exists.
  • Contrasts survive. In any effects model with an overall mean, \( \sum c_i\alpha_i \) is estimable exactly when \( \sum c_i = 0 \).
  • Degrees of freedom are ranks are traces. \( \operatorname{tr} P = r \), \( \operatorname{tr}(I - P) = n - r \). Divide SSE by the trace.
  • BLUE needs two moments; F needs normality. Gauss–Markov is a second-moment theorem. Exact t and F distributions need the normal model.
  • F = extra SSE per constraint over error mean square. Fit with and without \( H_0 \), subtract, divide by \( q \), divide by \( s^2 \).

Don’t Confuse

ThisIs not thisThe difference
\( n - r \) (rank)\( n - p \) (parameters)they differ when \( X \) is not of full rank
BLUEminimum variance unbiased (UMVUE)BLUE competes only with linear estimators; under normality it is also UMVUE
\( \hat\beta \) unique\( X\hat\beta \) uniquewith \( r < p \), \( \hat\beta \) depends on the g-inverse, \( X\hat\beta \) and estimable \( \lambda^T\hat\beta \) do not
estimableidentifiable from infinite data onlyestimability is exact: \( \lambda^T \) in the row space of \( X \)
OLS under \( \operatorname{Var} = \sigma^2V \)BLUE under \( \operatorname{Var} = \sigma^2V \)OLS stays unbiased, but GLS \( (X^TV^{-1}X)^{-1}X^TV^{-1}Y \) is best
\( \hat\sigma^2_{\text{MLE}} = \frac{\text{SSE}}{n} \)\( s^2 = \frac{\text{SSE}}{n - r} \)only \( s^2 \) is unbiased
testable hypothesisany linear restrictiontestable needs every row of \( L\beta \) estimable

Common Mistakes

Practice

All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C, one or more options may be correct.

Level 1: Foundation

  1. State the condition on \( \lambda \) for \( \lambda^T\beta \) to be estimable in \( E(Y) = X\beta \).
  2. In a Gauss–Markov model with \( n = 20 \) and \( \operatorname{rank} X = 4 \), what is \( E(\text{SSE}) \)?
  3. Which assumptions does the Gauss–Markov theorem use: normality, uncorrelated errors, equal variances, zero mean?
Answers
  1. \( \lambda^T \) lies in the row space of \( X \); equivalently \( \lambda^T(X^TX)^-X^TX = \lambda^T \).
  2. \( (20 - 4)\sigma^2 = 16\sigma^2 \).
  3. Zero mean, uncorrelated errors and equal variances. Not normality.

Level 2: Part B Style

Practice Q1. In the model \( y_{ij} = \mu + \alpha_i + \varepsilon_{ij} \), \( i = 1, 2, 3 \), \( j = 1, \dots, 4 \), which function is estimable?

(a) \( \mu \) (b) \( \alpha_1 \) (c) \( \alpha_1 - 2\alpha_2 + \alpha_3 \) (d) \( \mu + \alpha_1 + \alpha_2 \)

Answer: (c)

\( \lambda = (\lambda_0, \lambda_1, \lambda_2, \lambda_3) \) is in the row space iff \( \lambda_0 = \lambda_1 + \lambda_2 + \lambda_3 \).

  • (a) \( (1, 0, 0, 0) \): \( 1 \ne 0 \). Not estimable.
  • (b) \( (0, 1, 0, 0) \): \( 0 \ne 1 \). Not estimable.
  • (c) \( (0, 1, -2, 1) \): \( 0 = 1 - 2 + 1 \). Estimable; it is a contrast.
  • (d) \( (1, 1, 1, 0) \): \( 1 \ne 2 \). Not estimable.

Practice Q2. A full-rank linear model with \( n = 25 \) observations and \( p = 5 \) parameters has residual sum of squares 40. The unbiased estimate of \( \sigma^2 \) is:

(a) 1.6 (b) 2 (c) 8 (d) 10

Answer: (b)

\( s^2 = \frac{\text{SSE}}{n - p} = \frac{40}{20} = 2 \). Option (a) divides by \( n \) (the biased MLE).

Practice Q3. Let \( y_1 = \theta + \varepsilon_1 \), \( y_2 = 2\theta + \varepsilon_2 \), \( y_3 = -\theta + \varepsilon_3 \) with uncorrelated errors of variance \( \sigma^2 \). The variance of the BLUE of \( \theta \) is:

(a) \( \frac{\sigma^2}{3} \) (b) \( \frac{\sigma^2}{6} \) (c) \( \frac{\sigma^2}{4} \) (d) \( \frac{\sigma^2}{2} \)

Answer: (b)

\( X = (1, 2, -1)^T \), \( X^TX = 1 + 4 + 1 = 6 \). So \( \hat\theta = \frac{y_1 + 2y_2 - y_3}{6} \) and \( \operatorname{Var}(\hat\theta) = \frac{\sigma^2}{X^TX} = \frac{\sigma^2}{6} \).

Practice Q4. In a normal linear model with \( n = 30 \) and \( \operatorname{rank} X = 6 \), a testable hypothesis consists of 3 linearly independent estimable constraints. Under \( H_0 \) the F statistic has distribution:

(a) \( F_{3, 30} \) (b) \( F_{3, 24} \) (c) \( F_{6, 24} \) (d) \( F_{3, 27} \)

Answer: (b)

Numerator degrees of freedom \( q = 3 \) (number of independent constraints); denominator \( n - r = 30 - 6 = 24 \).

Level 3: Part C Style

Practice Q5. Let \( Y = X\beta + \varepsilon \), with \( X \) of order \( n \times p \) and rank \( r < p \), \( E(\varepsilon) = 0 \), \( \operatorname{Var}(\varepsilon) = \sigma^2I \), and \( P = X(X^TX)^-X^T \). Which statements are true?

(a) \( P \) is symmetric and idempotent. (b) \( \operatorname{tr} P = p \). (c) \( X\hat\beta \) is the same for every solution \( \hat\beta \) of the normal equations. (d) Every component \( \beta_j \) of \( \beta \) is estimable.

Answer: (a) and (c)
  • (a) True. \( P \) is the orthogonal projection onto \( \mathcal{C}(X) \), and it does not depend on the g-inverse.
  • (b) False. \( \operatorname{tr} P = \operatorname{rank} P = r < p \).
  • (c) True. \( X\hat\beta = PY \) for every solution.
  • (d) False. Only \( r \) linearly independent functions are estimable. With \( r < p \), at least one unit vector \( e_j^T \) is outside the row space of \( X \) (otherwise all \( p \) unit vectors would be in it and the rank would be \( p \)).

Practice Q6. In the weighing model \( y_1 = \theta_1 + \varepsilon_1 \), \( y_2 = \theta_2 + \varepsilon_2 \), \( y_3 = \theta_1 + \theta_2 + \varepsilon_3 \) (normal errors, variance \( \sigma^2 \)), the data are \( y = (3, 5, 9) \). Which statements are true?

(a) The BLUE of \( \theta_1 \) is \( \frac{10}{3} \). (b) The unbiased estimate of \( \sigma^2 \) is \( \frac{1}{3} \). (c) The F statistic for \( H_0: \theta_1 = \theta_2 \) is 6, on \( (1, 1) \) degrees of freedom. (d) \( \operatorname{Var}(\hat\theta_1 - \hat\theta_2) = \frac{2\sigma^2}{3} \).

Answer: (a), (b) and (c)
  • (a) True. \( \hat\theta_1 = \frac{2(3) - 5 + 9}{3} = \frac{10}{3} \).
  • (b) True. Fitted values \( \frac{10}{3}, \frac{16}{3}, \frac{26}{3} \); residuals \( -\frac{1}{3}, -\frac{1}{3}, \frac{1}{3} \); SSE \( = \frac{1}{3} \) on \( 3 - 2 = 1 \) degree of freedom.
  • (c) True. \( L\hat\theta = -2 \), \( L(X^TX)^{-1}L^T = 2 \), \( F = \frac{4/2}{1/3} = 6 \).
  • (d) False. \( \sigma^2(1, -1)(X^TX)^{-1}(1, -1)^T = \frac{\sigma^2}{3}(2 + 1 + 1 + 2) = 2\sigma^2 \). The covariance \( \operatorname{Cov}(\hat\theta_1, \hat\theta_2) = -\frac{\sigma^2}{3} \) adds to the variance of a difference.

Practice Q7. Consider \( Y = X\beta + \varepsilon \) with \( X \) of full column rank, \( E(\varepsilon) = 0 \) and \( \operatorname{Var}(\varepsilon) = \sigma^2V \), where \( V \) is a known positive definite matrix, \( V \ne I \). Which statements are true?

(a) The ordinary least squares estimator \( (X^TX)^{-1}X^TY \) is unbiased for \( \beta \). (b) The ordinary least squares estimator is always the BLUE of \( \beta \). (c) \( (X^TV^{-1}X)^{-1}X^TV^{-1}Y \) is the BLUE of \( \beta \). (d) The Gauss–Markov theorem in its standard form requires the errors to be normally distributed.

Answer: (a) and (c)
  • (a) True. \( E[(X^TX)^{-1}X^TY] = (X^TX)^{-1}X^TX\beta = \beta \); unbiasedness uses only \( E(\varepsilon) = 0 \).
  • (b) False. Take \( X = (1, 1)^T \), \( V = \operatorname{diag}(1, 4) \). OLS is \( \frac{y_1 + y_2}{2} \) with variance \( \frac{\sigma^2(1 + 4)}{4} = 1.25\sigma^2 \); the weighted mean \( \frac{4y_1 + y_2}{5} \) has variance \( \frac{\sigma^2(16 + 4)}{25} = 0.8\sigma^2 \).
  • (c) True. Transform by \( V^{-1/2} \): the model \( V^{-1/2}Y = V^{-1/2}X\beta + V^{-1/2}\varepsilon \) has error variance \( \sigma^2I \), and its least squares estimator is the GLS formula (Aitken’s theorem).
  • (d) False. Only the first two moments are used.

Quick Revision Table

IdeaFact
Model\( Y = X\beta + \varepsilon \), \( E\varepsilon = 0 \), \( \operatorname{Var}\varepsilon = \sigma^2I \), rank \( X = r \)
Normal equations\( X^TX\beta = X^TY \): always consistent
Hat matrix\( P = X(X^TX)^-X^T \): symmetric, idempotent, trace \( r \), g-inverse free
Estimable\( \lambda^T \in \) row space of \( X \); exactly \( r \) independent ones
Effects model\( \mu + \alpha_i \) and contrasts \( \sum c_i\alpha_i \) (\( \sum c_i = 0 \)) estimable
Gauss–Markov\( \lambda^T\hat\beta \) is BLUE, \( \operatorname{Var} = \sigma^2\lambda^T(X^TX)^-\lambda \); no normality needed
GLS (Aitken)\( (X^TV^{-1}X)^{-1}X^TV^{-1}Y \) when \( \operatorname{Var} = \sigma^2V \)
SSE\( Y^T(I - P)Y \), \( E = (n - r)\sigma^2 \), \( s^2 = \frac{\text{SSE}}{n - r} \)
Normal model\( \text{SSE}/\sigma^2 \sim \chi^2_{n - r} \), independent of \( \hat Y \)
F test\( F = \frac{(\text{SSE}_H - \text{SSE})/q}{\text{SSE}/(n - r)} \sim F_{q, n - r} \)

Key Takeaways

What’s Next