Gauss–Markov Models, BLUE and Linear Hypotheses
CSIR NET statistics: the Gauss–Markov linear model, normal equations, estimability, BLUE, estimating sigma squared and the F test for linear hypotheses.
- 19 min
On this page · 14 sections
Prerequisites
- Inner Product Spaces and Orthonormal Bases: least squares is an orthogonal projection, and the normal equations come from it.
- Rank, Determinants and Linear Equations: consistency of \( Ax = b \) and the rank of \( A^TA \).
- Quadratic Forms: sums of squares are quadratic forms \( Y^TAY \), and their ranks become degrees of freedom.
- Descriptive Statistics and Exploratory Data Analysis: the two regression lines, which this page generalises.
The Hook: Weigh Them Together
The Gauss–Markov Model
where \( Y \) is an \( n \times 1 \) vector of observations, \( X \) is a known \( n \times p \) design matrix of rank \( r \le p \), \( \beta \) is a \( p \times 1 \) vector of unknown parameters and \( \sigma^2 > 0 \) is unknown. The errors are uncorrelated with equal variance. Normality is not assumed. Adding \( \varepsilon \sim N_n(0, \sigma^2 I) \) gives the normal linear model, needed for exact tests.
The model is full rank when \( r = p \) (regression with independent columns) and less than full rank when \( r < p \) (ANOVA models with an overall mean and effects, such as \( y_{ij} = \mu + \alpha_i + \varepsilon_{ij} \)).
Least squares and the normal equations
Least squares minimises \( S(\beta) = (Y - X\beta)^T(Y - X\beta) \). Setting the gradient to zero gives the normal equations
$$\boxed{X^TX\beta = X^TY}.$$- They are always consistent: \( X^TY \) lies in the column space of \( X^T \), which equals the column space of \( X^TX \).
- Full rank: unique solution \( \hat\beta = (X^TX)^{-1}X^TY \).
- Less than full rank: infinitely many solutions \( \hat\beta = (X^TX)^-X^TY \), one for each generalised inverse \( (X^TX)^- \). The fitted vector \( X\hat\beta \) is the same for all of them.
The fitted vector is the orthogonal projection of \( Y \) onto the column space \( \mathcal{C}(X) \):
$$\hat Y = X\hat\beta = PY, \qquad P = X(X^TX)^-X^T.$$| Property of \( P \) (the hat or projection matrix) | Why it matters |
|---|---|
| \( P^T = P \), \( P^2 = P \) | orthogonal projection onto \( \mathcal{C}(X) \) |
| \( PX = X \) | fitted values of an exact fit are exact |
| \( P \) does not depend on the choice of g-inverse | \( \hat Y \) and residuals are unique |
| \( \operatorname{rank} P = \operatorname{tr} P = r \) | model degrees of freedom |
| \( I - P \) is also a symmetric idempotent, rank \( n - r \) | residual degrees of freedom |
| residuals \( e = (I - P)Y \) satisfy \( X^Te = 0 \) | residuals are orthogonal to every column of \( X \) |
Estimability
Since \( E(a^TY) = a^TX\beta \), the condition is \( a^TX = \lambda^T \). So
$$\boxed{\lambda^T\beta \text{ estimable} \iff \lambda^T \in \text{row space of } X \iff \lambda^T(X^TX)^-X^TX = \lambda^T.}$$Useful consequences:
- Every \( E(y_i) = x_i^T\beta \) (each row of \( X\beta \)) is estimable, and so is every linear combination of them.
- There are exactly \( r = \operatorname{rank} X \) linearly independent estimable functions.
- In a full-rank model every \( \lambda^T\beta \) is estimable.
- If \( r < p \), the individual parameters are typically not estimable.
The Gauss–Markov Theorem
In the model \( E(Y) = X\beta \), \( \operatorname{Var}(Y) = \sigma^2I \), let \( \lambda^T\beta \) be estimable and \( \hat\beta \) any solution of the normal equations. Then \( \lambda^T\hat\beta \) is the same for every such solution, is unbiased for \( \lambda^T\beta \), and has the smallest variance among all linear unbiased estimators. It is the unique BLUE (best linear unbiased estimator), with
$$\operatorname{Var}(\lambda^T\hat\beta) = \sigma^2\lambda^T(X^TX)^-\lambda.$$In the full-rank case, \( \operatorname{Var}(\hat\beta) = \sigma^2(X^TX)^{-1} \).
Read the fine print, because Part C does:
- Linear and unbiased are both restrictions. A biased estimator (ridge regression) or a non-linear one can have smaller mean squared error.
- No normality is needed. Only the first two moments enter.
- Under normality, \( \lambda^T\hat\beta \) is also the MLE and the UMVUE among all unbiased estimators, not only linear ones.
- The equal-variance, uncorrelated assumption matters. If \( \operatorname{Var}(\varepsilon) = \sigma^2V \) with known positive definite \( V \ne I \), ordinary least squares is still unbiased but no longer best. The BLUE is the generalised (Aitken) least squares estimator $$\hat\beta_{\text{GLS}} = (X^TV^{-1}X)^{-1}X^TV^{-1}Y, \qquad \operatorname{Var} = \sigma^2(X^TV^{-1}X)^{-1}.$$ With \( V = \operatorname{diag}(v_i) \) this is weighted least squares, with weights \( \frac{1}{v_i} \).
Estimating \( \sigma^2 \)
The residual (error) sum of squares is a quadratic form:
$$\text{SSE} = e^Te = Y^T(I - P)Y = Y^TY - \hat\beta^TX^TY.$$Using \( E(Y^TAY) = \operatorname{tr}(A\Sigma) + \mu^TA\mu \) with \( A = I - P \), \( \Sigma = \sigma^2I \), \( \mu = X\beta \) and \( (I - P)X = 0 \):
$$E(\text{SSE}) = \sigma^2\operatorname{tr}(I - P) = (n - r)\sigma^2, \qquad \boxed{s^2 = \hat\sigma^2 = \frac{\text{SSE}}{n - r}} \text{ is unbiased.}$$The divisor is \( n - r \) (rank), not \( n - p \) (number of parameters), when the model is not of full rank. The MLE under normality divides by \( n \) and is biased.
Under normality (\( \varepsilon \sim N_n(0, \sigma^2I) \)):
- \( \lambda^T\hat\beta \sim N\left(\lambda^T\beta, \; \sigma^2\lambda^T(X^TX)^-\lambda\right) \) for estimable \( \lambda^T\beta \);
- \( \dfrac{\text{SSE}}{\sigma^2} \sim \chi^2_{n - r} \);
- \( \hat Y = PY \) and \( \text{SSE} \) are independent, because \( P(I - P) = 0 \);
- hence \( t = \dfrac{\lambda^T\hat\beta - \lambda^T\beta}{s\sqrt{\lambda^T(X^TX)^-\lambda}} \sim t_{n - r} \), the basis of confidence intervals.
In the weighing example, the residuals are \( (-\frac{1}{3}, -\frac{1}{3}, \frac{1}{3}) \), \( \text{SSE} = \frac{1}{3} \), \( n - r = 1 \), so \( s^2 = \frac{1}{3} \).
Testing Linear Hypotheses
A linear hypothesis is \( H_0: L\beta = c \), with \( L \) a \( q \times p \) matrix of rank \( q \). It is testable when every row of \( L\beta \) is estimable.
In the normal linear model with \( \operatorname{rank} X = r \), for a testable \( H_0: L\beta = c \) of rank \( q \):
$$F = \frac{(L\hat\beta - c)^T\left[L(X^TX)^-L^T\right]^{-1}(L\hat\beta - c)/q}{\text{SSE}/(n - r)} \sim F_{q,\; n - r} \quad \text{under } H_0.$$Equivalently, if \( \text{SSE}_H \) is the residual sum of squares when the model is fitted subject to \( H_0 \),
$$F = \frac{(\text{SSE}_H - \text{SSE})/q}{\text{SSE}/(n - r)}.$$Reject \( H_0 \) for large \( F \). Under the alternative, \( F \) has a non-central F distribution.
The second form is the one to remember. It says: impose the hypothesis, refit, and see how much the residual sum of squares grows per constraint, measured against the error variance. Every ANOVA F test, the overall regression F test and the partial F test for dropping variables are special cases.
graph TD
A["Model Y = X beta + e
rank X = r"] --> B["Solve X'X b = X'Y
(any g-inverse)"]
B --> C{"Is lambda' beta in
row space of X?"}
C -->|no| D["Not estimable:
no unbiased linear estimator"]
C -->|yes| E["BLUE = lambda' b
Var = sigma^2 lambda'(X'X)^- lambda"]
B --> F["SSE = Y'(I-P)Y
s^2 = SSE/(n-r)"]
E --> G["Test L beta = c:
F = ((SSE_H - SSE)/q) / (SSE/(n-r))"]
F --> GRestricted least squares
Minimising \( S(\beta) \) subject to \( L\beta = c \) (full rank \( X \)) gives
$$\hat\beta_H = \hat\beta - (X^TX)^{-1}L^T\left[L(X^TX)^{-1}L^T\right]^{-1}(L\hat\beta - c),$$and \( \text{SSE}_H - \text{SSE} = (L\hat\beta - c)^T[L(X^TX)^{-1}L^T]^{-1}(L\hat\beta - c) \ge 0 \). That identity is why the two forms of the F statistic coincide.
Memory Tricks
- “Estimable = row of X, or built from rows.” If you cannot write \( \lambda^T \) as a combination of the rows of \( X \), stop: no unbiased linear estimator exists.
- Contrasts survive. In any effects model with an overall mean, \( \sum c_i\alpha_i \) is estimable exactly when \( \sum c_i = 0 \).
- Degrees of freedom are ranks are traces. \( \operatorname{tr} P = r \), \( \operatorname{tr}(I - P) = n - r \). Divide SSE by the trace.
- BLUE needs two moments; F needs normality. Gauss–Markov is a second-moment theorem. Exact t and F distributions need the normal model.
- F = extra SSE per constraint over error mean square. Fit with and without \( H_0 \), subtract, divide by \( q \), divide by \( s^2 \).
Don’t Confuse
| This | Is not this | The difference |
|---|---|---|
| \( n - r \) (rank) | \( n - p \) (parameters) | they differ when \( X \) is not of full rank |
| BLUE | minimum variance unbiased (UMVUE) | BLUE competes only with linear estimators; under normality it is also UMVUE |
| \( \hat\beta \) unique | \( X\hat\beta \) unique | with \( r < p \), \( \hat\beta \) depends on the g-inverse, \( X\hat\beta \) and estimable \( \lambda^T\hat\beta \) do not |
| estimable | identifiable from infinite data only | estimability is exact: \( \lambda^T \) in the row space of \( X \) |
| OLS under \( \operatorname{Var} = \sigma^2V \) | BLUE under \( \operatorname{Var} = \sigma^2V \) | OLS stays unbiased, but GLS \( (X^TV^{-1}X)^{-1}X^TV^{-1}Y \) is best |
| \( \hat\sigma^2_{\text{MLE}} = \frac{\text{SSE}}{n} \) | \( s^2 = \frac{\text{SSE}}{n - r} \) | only \( s^2 \) is unbiased |
| testable hypothesis | any linear restriction | testable needs every row of \( L\beta \) estimable |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C, one or more options may be correct.
Level 1: Foundation
- State the condition on \( \lambda \) for \( \lambda^T\beta \) to be estimable in \( E(Y) = X\beta \).
- In a Gauss–Markov model with \( n = 20 \) and \( \operatorname{rank} X = 4 \), what is \( E(\text{SSE}) \)?
- Which assumptions does the Gauss–Markov theorem use: normality, uncorrelated errors, equal variances, zero mean?
Answers
- \( \lambda^T \) lies in the row space of \( X \); equivalently \( \lambda^T(X^TX)^-X^TX = \lambda^T \).
- \( (20 - 4)\sigma^2 = 16\sigma^2 \).
- Zero mean, uncorrelated errors and equal variances. Not normality.
Level 2: Part B Style
Practice Q1. In the model \( y_{ij} = \mu + \alpha_i + \varepsilon_{ij} \), \( i = 1, 2, 3 \), \( j = 1, \dots, 4 \), which function is estimable?
(a) \( \mu \) (b) \( \alpha_1 \) (c) \( \alpha_1 - 2\alpha_2 + \alpha_3 \) (d) \( \mu + \alpha_1 + \alpha_2 \)
Answer: (c)
\( \lambda = (\lambda_0, \lambda_1, \lambda_2, \lambda_3) \) is in the row space iff \( \lambda_0 = \lambda_1 + \lambda_2 + \lambda_3 \).
- (a) \( (1, 0, 0, 0) \): \( 1 \ne 0 \). Not estimable.
- (b) \( (0, 1, 0, 0) \): \( 0 \ne 1 \). Not estimable.
- (c) \( (0, 1, -2, 1) \): \( 0 = 1 - 2 + 1 \). Estimable; it is a contrast.
- (d) \( (1, 1, 1, 0) \): \( 1 \ne 2 \). Not estimable.
Practice Q2. A full-rank linear model with \( n = 25 \) observations and \( p = 5 \) parameters has residual sum of squares 40. The unbiased estimate of \( \sigma^2 \) is:
(a) 1.6 (b) 2 (c) 8 (d) 10
Answer: (b)
\( s^2 = \frac{\text{SSE}}{n - p} = \frac{40}{20} = 2 \). Option (a) divides by \( n \) (the biased MLE).
Practice Q3. Let \( y_1 = \theta + \varepsilon_1 \), \( y_2 = 2\theta + \varepsilon_2 \), \( y_3 = -\theta + \varepsilon_3 \) with uncorrelated errors of variance \( \sigma^2 \). The variance of the BLUE of \( \theta \) is:
(a) \( \frac{\sigma^2}{3} \) (b) \( \frac{\sigma^2}{6} \) (c) \( \frac{\sigma^2}{4} \) (d) \( \frac{\sigma^2}{2} \)
Answer: (b)
\( X = (1, 2, -1)^T \), \( X^TX = 1 + 4 + 1 = 6 \). So \( \hat\theta = \frac{y_1 + 2y_2 - y_3}{6} \) and \( \operatorname{Var}(\hat\theta) = \frac{\sigma^2}{X^TX} = \frac{\sigma^2}{6} \).
Practice Q4. In a normal linear model with \( n = 30 \) and \( \operatorname{rank} X = 6 \), a testable hypothesis consists of 3 linearly independent estimable constraints. Under \( H_0 \) the F statistic has distribution:
(a) \( F_{3, 30} \) (b) \( F_{3, 24} \) (c) \( F_{6, 24} \) (d) \( F_{3, 27} \)
Answer: (b)
Numerator degrees of freedom \( q = 3 \) (number of independent constraints); denominator \( n - r = 30 - 6 = 24 \).
Level 3: Part C Style
Practice Q5. Let \( Y = X\beta + \varepsilon \), with \( X \) of order \( n \times p \) and rank \( r < p \), \( E(\varepsilon) = 0 \), \( \operatorname{Var}(\varepsilon) = \sigma^2I \), and \( P = X(X^TX)^-X^T \). Which statements are true?
(a) \( P \) is symmetric and idempotent. (b) \( \operatorname{tr} P = p \). (c) \( X\hat\beta \) is the same for every solution \( \hat\beta \) of the normal equations. (d) Every component \( \beta_j \) of \( \beta \) is estimable.
Answer: (a) and (c)
- (a) True. \( P \) is the orthogonal projection onto \( \mathcal{C}(X) \), and it does not depend on the g-inverse.
- (b) False. \( \operatorname{tr} P = \operatorname{rank} P = r < p \).
- (c) True. \( X\hat\beta = PY \) for every solution.
- (d) False. Only \( r \) linearly independent functions are estimable. With \( r < p \), at least one unit vector \( e_j^T \) is outside the row space of \( X \) (otherwise all \( p \) unit vectors would be in it and the rank would be \( p \)).
Practice Q6. In the weighing model \( y_1 = \theta_1 + \varepsilon_1 \), \( y_2 = \theta_2 + \varepsilon_2 \), \( y_3 = \theta_1 + \theta_2 + \varepsilon_3 \) (normal errors, variance \( \sigma^2 \)), the data are \( y = (3, 5, 9) \). Which statements are true?
(a) The BLUE of \( \theta_1 \) is \( \frac{10}{3} \). (b) The unbiased estimate of \( \sigma^2 \) is \( \frac{1}{3} \). (c) The F statistic for \( H_0: \theta_1 = \theta_2 \) is 6, on \( (1, 1) \) degrees of freedom. (d) \( \operatorname{Var}(\hat\theta_1 - \hat\theta_2) = \frac{2\sigma^2}{3} \).
Answer: (a), (b) and (c)
- (a) True. \( \hat\theta_1 = \frac{2(3) - 5 + 9}{3} = \frac{10}{3} \).
- (b) True. Fitted values \( \frac{10}{3}, \frac{16}{3}, \frac{26}{3} \); residuals \( -\frac{1}{3}, -\frac{1}{3}, \frac{1}{3} \); SSE \( = \frac{1}{3} \) on \( 3 - 2 = 1 \) degree of freedom.
- (c) True. \( L\hat\theta = -2 \), \( L(X^TX)^{-1}L^T = 2 \), \( F = \frac{4/2}{1/3} = 6 \).
- (d) False. \( \sigma^2(1, -1)(X^TX)^{-1}(1, -1)^T = \frac{\sigma^2}{3}(2 + 1 + 1 + 2) = 2\sigma^2 \). The covariance \( \operatorname{Cov}(\hat\theta_1, \hat\theta_2) = -\frac{\sigma^2}{3} \) adds to the variance of a difference.
Practice Q7. Consider \( Y = X\beta + \varepsilon \) with \( X \) of full column rank, \( E(\varepsilon) = 0 \) and \( \operatorname{Var}(\varepsilon) = \sigma^2V \), where \( V \) is a known positive definite matrix, \( V \ne I \). Which statements are true?
(a) The ordinary least squares estimator \( (X^TX)^{-1}X^TY \) is unbiased for \( \beta \). (b) The ordinary least squares estimator is always the BLUE of \( \beta \). (c) \( (X^TV^{-1}X)^{-1}X^TV^{-1}Y \) is the BLUE of \( \beta \). (d) The Gauss–Markov theorem in its standard form requires the errors to be normally distributed.
Answer: (a) and (c)
- (a) True. \( E[(X^TX)^{-1}X^TY] = (X^TX)^{-1}X^TX\beta = \beta \); unbiasedness uses only \( E(\varepsilon) = 0 \).
- (b) False. Take \( X = (1, 1)^T \), \( V = \operatorname{diag}(1, 4) \). OLS is \( \frac{y_1 + y_2}{2} \) with variance \( \frac{\sigma^2(1 + 4)}{4} = 1.25\sigma^2 \); the weighted mean \( \frac{4y_1 + y_2}{5} \) has variance \( \frac{\sigma^2(16 + 4)}{25} = 0.8\sigma^2 \).
- (c) True. Transform by \( V^{-1/2} \): the model \( V^{-1/2}Y = V^{-1/2}X\beta + V^{-1/2}\varepsilon \) has error variance \( \sigma^2I \), and its least squares estimator is the GLS formula (Aitken’s theorem).
- (d) False. Only the first two moments are used.
Quick Revision Table
| Idea | Fact |
|---|---|
| Model | \( Y = X\beta + \varepsilon \), \( E\varepsilon = 0 \), \( \operatorname{Var}\varepsilon = \sigma^2I \), rank \( X = r \) |
| Normal equations | \( X^TX\beta = X^TY \): always consistent |
| Hat matrix | \( P = X(X^TX)^-X^T \): symmetric, idempotent, trace \( r \), g-inverse free |
| Estimable | \( \lambda^T \in \) row space of \( X \); exactly \( r \) independent ones |
| Effects model | \( \mu + \alpha_i \) and contrasts \( \sum c_i\alpha_i \) (\( \sum c_i = 0 \)) estimable |
| Gauss–Markov | \( \lambda^T\hat\beta \) is BLUE, \( \operatorname{Var} = \sigma^2\lambda^T(X^TX)^-\lambda \); no normality needed |
| GLS (Aitken) | \( (X^TV^{-1}X)^{-1}X^TV^{-1}Y \) when \( \operatorname{Var} = \sigma^2V \) |
| SSE | \( Y^T(I - P)Y \), \( E = (n - r)\sigma^2 \), \( s^2 = \frac{\text{SSE}}{n - r} \) |
| Normal model | \( \text{SSE}/\sigma^2 \sim \chi^2_{n - r} \), independent of \( \hat Y \) |
| F test | \( F = \frac{(\text{SSE}_H - \text{SSE})/q}{\text{SSE}/(n - r)} \sim F_{q, n - r} \) |
Key Takeaways
What’s Next
- ANOVA, ANCOVA and Fixed, Random and Mixed Effects: the F test of this page applied to designed experiments.
- Simple, Multiple and Logistic Regression: the full-rank case, with diagnostics.
- Multivariate Normal, Wishart and Quadratic Forms: why SSE is chi-square and independent of \( \hat Y \).
- Back to the Linear Models and Multivariate Analysis chapter overview.