Nonparametric Tests, Rank Correlation and Bayesian Inference
CSIR NET statistics: sign, Wilcoxon, Mann–Whitney, runs, Kruskal–Wallis and KS tests; Spearman and Kendall correlation; conjugate priors, Bayes estimators.
- 19 min
On this page · 14 sections
Prerequisites
- Hypothesis Testing: MP, UMP and Likelihood Ratio Tests: size, power and p-values.
- Sampling Distributions, Standard Errors and Order Statistics: order statistics and the distribution-free interval for the median.
- Descriptive Statistics and Exploratory Data Analysis: Pearson’s correlation \( r \), which Spearman’s coefficient applies to ranks.
- Combinatorics and Counting: null distributions of rank statistics are counting problems.
The Hook: When You Cannot Trust the Bell Curve
Why Nonparametric Tests
A test is distribution-free if the null distribution of its statistic is the same for every continuous population. Ranks and signs achieve this: under \( H_0 \), every arrangement of ranks or signs is equally likely, so null distributions are pure counting.
| Parametric (\( t \), \( F \)) | Nonparametric (rank, sign) | |
|---|---|---|
| Assumptions | normal population (or large \( n \)) | continuity; sometimes symmetry |
| Data needed | measurements | ranks or signs suffice (ordinal data) |
| Outliers | strongly affected | resistant |
| Efficiency when the population is normal | best | slightly lower (see below) |
| Hypotheses | usually means | medians, locations, “same distribution” |
Efficiency. The Pitman asymptotic relative efficiency (ARE) compares the sample sizes two tests need for the same power. Under normal populations, the Wilcoxon signed-rank and Mann–Whitney tests have ARE \( \frac{3}{\pi} \approx 0.955 \) relative to the \( t \)-test, and the sign test \( \frac{2}{\pi} \approx 0.637 \). For heavy-tailed populations the rank tests can be far more efficient than \( t \).
One-Sample and Paired Tests
Sign test
\( H_0 \): the population median is \( m_0 \) (continuous population). Statistic: \( S = \#\{i : X_i > m_0\} \). Under \( H_0 \), \( S \sim \) Binomial\( (n, \frac{1}{2}) \). Observations equal to \( m_0 \) are dropped and \( n \) reduced. Paired data: apply it to the differences \( D_i = Y_i - X_i \) with \( m_0 = 0 \). Large \( n \): \( Z = \frac{S - n/2}{\sqrt{n}/2} \approx N(0, 1) \).
The sign test assumes only continuity. It ignores the sizes of the differences, which is why it is less powerful.
Wilcoxon signed-rank test
\( H_0 \): the distribution of \( D \) is symmetric about 0. Statistic: rank \( \lvert D_1 \rvert, \dots, \lvert D_n \rvert \) from 1 to \( n \); \( T^+ \) is the sum of ranks belonging to positive \( D_i \), \( T^- \) the sum for negative ones.
$$T^+ + T^- = \frac{n(n+1)}{2}, \qquad E(T^+) = \frac{n(n+1)}{4}, \qquad \operatorname{Var}(T^+) = \frac{n(n+1)(2n+1)}{24}.$$Under \( H_0 \) each rank is positive or negative with probability \( \frac{1}{2} \) independently, so \( T^+ = \sum_{k=1}^n kB_k \) with \( B_k \) i.i.d. Bernoulli\( (\frac{1}{2}) \): that gives the mean and variance at once (\( \operatorname{Var} = \frac{1}{4}\sum k^2 \)). \( P(T^+ = t) = \frac{c(t)}{2^n} \), where \( c(t) \) is the number of subsets of \( \{1, \dots, n\} \) summing to \( t \).
Symmetry is an assumption here: the signed-rank test is a test of the median (or centre) only under symmetry.
Two-Sample and \( k \)-Sample Tests
Mann–Whitney \( U \) / Wilcoxon rank-sum test
\( H_0 \): the two populations \( F \) and \( G \) are identical. Alternative: one is shifted (stochastically larger). Statistics: pool the \( N = n_1 + n_2 \) observations and rank them. \( W \) = sum of the ranks of sample 1. \( U \) = number of pairs \( (X_i, Y_j) \) with \( X_i > Y_j \).
$$W = U + \frac{n_1(n_1 + 1)}{2}, \qquad E(U) = \frac{n_1n_2}{2}, \qquad \operatorname{Var}(U) = \frac{n_1n_2(N + 1)}{12}, \qquad E(W) = \frac{n_1(N + 1)}{2}.$$The two counts \( U_{X>Y} \) and \( U_{Y>X} \) add to \( n_1n_2 \). \( \frac{U}{n_1n_2} \) estimates \( P(X > Y) \). For large samples \( \frac{U - E(U)}{\sqrt{\operatorname{Var}U}} \approx N(0, 1) \).
Other standard tests
| Test | Hypothesis | Statistic and null behaviour |
|---|---|---|
| Wald–Wolfowitz runs test (one sample) | randomness of a sequence of two symbols | \( R \) = number of runs; \( E(R) = 1 + \frac{2n_1n_2}{N} \), \( \operatorname{Var}(R) = \frac{2n_1n_2(2n_1n_2 - N)}{N^2(N - 1)} \); too few runs suggest clustering or trend, too many suggest alternation |
| Median test (two or more samples) | equal medians | classify each observation above/below the pooled median; \( \chi^2 \) test on the resulting table |
| Kruskal–Wallis (\( k \) samples) | \( k \) identical populations | \( H = \frac{12}{N(N+1)}\sum_{i=1}^k \frac{R_i^2}{n_i} - 3(N + 1) \approx \chi^2_{k-1} \), \( R_i \) = rank sum of sample \( i \) |
| Kolmogorov–Smirnov (one sample) | \( F = F_0 \), \( F_0 \) continuous and fully specified | \( D_n = \sup_x \lvert F_n(x) - F_0(x) \rvert \); null law the same for every continuous \( F_0 \) |
| Kolmogorov–Smirnov (two sample) | \( F = G \) | \( \sup_x \lvert F_{n_1}(x) - G_{n_2}(x) \rvert \) |
Kruskal–Wallis is the rank version of one-way ANOVA; with \( k = 2 \) it is equivalent to the two-sided Mann–Whitney test. The KS test compares whole distributions, unlike \( \chi^2 \) goodness of fit it needs no grouping, and it loses its distribution-free property if parameters of \( F_0 \) are estimated from the same data.
Rank Correlation
Given \( n \) pairs, rank each variable separately (\( 1 \) to \( n \)), and let \( d_i \) be the difference between the two ranks of pair \( i \).
It is Pearson’s correlation coefficient computed on the ranks. With ties, assign average (mid-) ranks and compute Pearson’s \( r \) on them (or apply a tie correction to the formula).
A pair of observations \( (x_i, y_i), (x_j, y_j) \) is concordant if \( (x_i - x_j)(y_i - y_j) > 0 \) and discordant if \( < 0 \). With \( C \) concordant and \( D \) discordant pairs (no ties),
$$\tau = \frac{C - D}{\binom{n}{2}} = \frac{C - D}{n(n - 1)/2}.$$Properties shared by both.
- \( -1 \le r_s, \tau \le 1 \), with \( +1 \) iff the ranks agree completely (a strictly increasing relationship) and \( -1 \) iff they are reversed.
- Unchanged by any strictly increasing transformation of either variable (\( \log \), cube, square on positive data), because the ranks do not change. Pearson’s \( r \) is not.
- Under independence (continuous variables): mean 0; \( \operatorname{Var}(r_s) = \frac{1}{n - 1} \), \( \operatorname{Var}(\tau) = \frac{2(2n + 5)}{9n(n - 1)} \); both are asymptotically normal, so \( r_s\sqrt{n - 1} \approx N(0, 1) \).
- \( \sum d_i^2 \) ranges from 0 (identical ranks) to \( \frac{n(n^2 - 1)}{3} \) (reversed ranks).
Elementary Bayesian Inference
Prior to posterior
Treat the parameter as a random variable with a prior density \( \pi(\theta) \). After observing \( x \), Bayes’ theorem gives the posterior
$$\pi(\theta \mid x) = \frac{f(x \mid \theta)\,\pi(\theta)}{\int f(x \mid \theta')\pi(\theta')\,d\theta'} \propto L(\theta; x)\,\pi(\theta).$$Posterior \( \propto \) likelihood \( \times \) prior. The denominator is the marginal (prior predictive) density of \( x \); in practice you write the product, drop constants, and recognise the kernel of a known density.
Conjugate families
A family of priors is conjugate to a model if the posterior is in the same family.
| Model | Prior | Posterior after data \( x_1, \dots, x_n \) |
|---|---|---|
| Binomial: \( x \) successes in \( m \) trials | Beta\( (a, b) \) | Beta\( (a + x,\; b + m - x) \) |
| Bernoulli, \( n \) trials, \( T = \sum x_i \) | Beta\( (a, b) \) | Beta\( (a + T,\; b + n - T) \) |
| Poisson\( (\lambda) \) | Gamma\( (\alpha, \beta) \), rate \( \beta \) | Gamma\( (\alpha + \sum x_i,\; \beta + n) \) |
| Exponential, rate \( \lambda \) | Gamma\( (\alpha, \beta) \) | Gamma\( (\alpha + n,\; \beta + \sum x_i) \) |
| \( N(\theta, \sigma^2) \), \( \sigma^2 \) known | \( N(\mu_0, \tau^2) \) | \( N(\mu_n, \tau_n^2) \), \( \frac{1}{\tau_n^2} = \frac{1}{\tau^2} + \frac{n}{\sigma^2} \), \( \mu_n = \tau_n^2\left(\frac{\mu_0}{\tau^2} + \frac{n\bar{x}}{\sigma^2}\right) \) |
In the normal case, precisions add and the posterior mean is the precision-weighted average of the prior mean and \( \bar{x} \). In the beta–binomial case,
$$E(p \mid x) = \frac{a + x}{a + b + m} = \frac{a + b}{a + b + m}\cdot\frac{a}{a + b} + \frac{m}{a + b + m}\cdot\frac{x}{m},$$a weighted average of the prior mean and the sample proportion, with the prior acting like \( a + b \) extra observations. As the sample grows, the data dominate.
Bayes estimators
A loss function \( L(\theta, d) \) measures the cost of estimating \( \theta \) by \( d \). The Bayes estimator minimises the posterior expected loss:
| Loss | Bayes estimator |
|---|---|
| squared error \( (\theta - d)^2 \) | posterior mean |
| absolute error \( \lvert \theta - d \rvert \) | posterior median |
| 0–1 loss (limit of small windows) | posterior mode (MAP estimate) |
The Bayes risk is the expected loss averaged over both data and prior. Bayes estimators under proper priors are usually biased in the frequentist sense (they shrink towards the prior mean), but they are consistent under mild conditions.
Credible intervals, priors and prediction
- A \( 100(1 - \alpha)\% \) credible interval is any \( C \) with \( P(\theta \in C \mid x) = 1 - \alpha \). Unlike a confidence interval, it is a probability statement about \( \theta \) given the data. The highest posterior density (HPD) interval is the shortest one.
- Non-informative priors. The uniform prior; improper priors (not integrating to 1, such as \( \pi(\mu) \propto 1 \) on \( \mathbb{R} \)), allowed if the posterior is proper; Jeffreys’ prior \( \pi(\theta) \propto \sqrt{I(\theta)} \), invariant under reparametrisation. For the binomial it is Beta\( (\frac{1}{2}, \frac{1}{2}) \); for a Poisson mean, \( \propto \lambda^{-1/2} \); for a normal mean, flat.
- With a flat prior on a normal mean (known \( \sigma \)), the posterior is \( N(\bar{x}, \sigma^2/n) \): the 95% credible interval coincides numerically with the classical \( \bar{x} \pm 1.96\sigma/\sqrt{n} \), though its interpretation differs.
- Posterior predictive. For the beta–binomial model, the probability that the next trial is a success is \( E(p \mid x) = \frac{a + x}{a + b + m} \).
- Sequential updating. Today’s posterior is tomorrow’s prior; updating with all data at once gives the same answer as updating one observation at a time.
graph LR
P["Prior pi(theta)"] --> M["multiply by likelihood L(theta; x)"]
M --> Q["Posterior pi(theta | x)"]
Q --> E1["Mean: squared-error estimate"]
Q --> E2["Median: absolute-error estimate"]
Q --> E3["Mode: MAP estimate"]
Q --> CI["Credible / HPD interval"]
Q -->|"new data"| MMemory Tricks
- “Signs need continuity, signed ranks need symmetry.” The sign test assumes almost nothing; the signed-rank test assumes a symmetric distribution of differences.
- “6, n cubed minus n.” Spearman: \( 1 - \frac{6\sum d^2}{n^3 - n} \).
- “n(n+1)/4 and n1 n2 / 2.” Null means of \( T^+ \) and \( U \): half of the maximum possible value in each case.
- “Beta adds successes and failures; gamma adds counts and sample size.” Beta\( (a + x, b + n - x) \), Gamma\( (\alpha + \sum x, \beta + n) \).
- “Mean, median, mode for square, absolute, zero-one.” The three losses and their Bayes estimators, in order.
Don’t Confuse
| This | Is not this | The difference |
|---|---|---|
| sign test (median, continuity only) | signed-rank test (needs symmetry) | the second uses magnitudes through ranks |
| Mann–Whitney (two independent samples) | Wilcoxon signed-rank (paired data) | independent vs paired |
| Spearman \( r_s \) | Kendall \( \tau \) | different scales; in the example 0.8 vs 0.6 |
| rank correlation \( = 1 \) | Pearson \( r = 1 \) | ranks need only a monotone relation; Pearson needs a line |
| credible interval | confidence interval | probability about \( \theta \) given data vs coverage of a procedure |
| posterior mean | MLE | the posterior mean shrinks the MLE toward the prior mean |
| KS with fully specified \( F_0 \) | KS with estimated parameters | estimation destroys the distribution-free null law |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C, one or more options may be correct.
Level 1: Foundation
- Write Spearman’s rank correlation formula (no ties).
- Which prior family is conjugate for a Poisson mean?
- Under squared-error loss, what is the Bayes estimator?
Answers
- \( r_s = 1 - \frac{6\sum d_i^2}{n(n^2 - 1)} \).
- The gamma family.
- The posterior mean.
Level 2: Part B Style
Practice Q1. For 5 pairs of ranks, \( \sum d_i^2 = 4 \). Spearman’s rank correlation is:
(a) 0.6 (b) 0.7 (c) 0.8 (d) 0.9
Answer: (c)
\( 1 - \frac{6 \times 4}{5 \times 24} = 1 - 0.2 = 0.8 \).
Practice Q2. In 10 paired comparisons (no ties), 9 differences are positive. The one-sided sign-test p-value for \( H_0 \): median difference \( = 0 \) is:
(a) \( \frac{1}{1024} \) (b) \( \frac{10}{1024} \) (c) \( \frac{11}{1024} \) (d) \( \frac{1}{2} \)
Answer: (c)
\( P(S \ge 9) \) for \( S \sim \) Binomial\( (10, \frac{1}{2}) \) is \( \frac{10 + 1}{1024} \approx 0.0107 \). Option (b) is \( P(S = 9) \) alone.
Practice Q3. A success probability has prior Beta\( (2, 2) \). In 10 trials there are 7 successes. The Bayes estimate under squared-error loss is:
(a) 0.6 (b) \( \frac{9}{14} \) (c) \( \frac{2}{3} \) (d) 0.7
Answer: (b)
Posterior Beta\( (9, 5) \), mean \( \frac{9}{14} \approx 0.643 \). Option (c) is the posterior mode; (d) the MLE.
Practice Q4. Samples of sizes \( n_1 = 4 \) and \( n_2 = 5 \) are ranked together. Under \( H_0 \) (identical populations), the expected Mann–Whitney \( U \) and the expected rank sum of the first sample are:
(a) 10 and 20 (b) 10 and 18 (c) 20 and 20 (d) 9 and 20
Answer: (a)
\( E(U) = \frac{4 \times 5}{2} = 10 \); \( E(W) = \frac{n_1(N + 1)}{2} = \frac{4 \times 10}{2} = 20 \). (Check: \( W = U + \frac{4 \times 5}{2} = U + 10 \).)
Level 3: Part C Style
Practice Q5. Counts \( 3, 5, 4 \) are observed from Poisson\( (\lambda) \), and \( \lambda \) has a Gamma prior with shape 2 and rate 1. Which statements are true?
(a) The posterior is Gamma with shape 14 and rate 4. (b) The Bayes estimate under squared-error loss is 3.5, between the prior mean 2 and the MLE 4. (c) The posterior mode is 3.5. (d) The posterior variance is 0.875.
Answer: (a), (b) and (d)
- (a) True. Shape \( 2 + 12 = 14 \), rate \( 1 + 3 = 4 \).
- (b) True. \( \frac{14}{4} = 3.5 \); prior mean \( \frac{2}{1} \), MLE \( \bar{x} = 4 \).
- (c) False. Mode \( \frac{14 - 1}{4} = 3.25 \).
- (d) True. \( \frac{14}{4^2} = 0.875 \).
Practice Q6. Five pairs have \( x \)-ranks 1, 2, 3, 4, 5 and \( y \)-ranks 2, 1, 4, 3, 5. Which statements are true?
(a) Spearman’s \( r_s = 0.8 \). (b) Kendall’s \( \tau = 0.6 \). (c) Replacing every \( y \) value by its cube leaves both coefficients unchanged. (d) Kendall’s \( \tau \) always equals Spearman’s \( r_s \).
Answer: (a), (b) and (c)
- (a) True. \( \sum d^2 = 4 \).
- (b) True. 8 concordant, 2 discordant pairs: \( \frac{6}{10} \).
- (c) True. Cubing is strictly increasing, so the ranks are unchanged.
- (d) False. This very example has \( 0.6 \ne 0.8 \).
Practice Q7. Paired differences: \( 1.2, -0.4, 2.5, 0.8, -1.5, 3.1, 0.6, 2.0 \). Which statements are true?
(a) The Wilcoxon signed-rank statistic \( T^+ = 30 \). (b) Under \( H_0 \), \( E(T^+) = 18 \) and \( \operatorname{Var}(T^+) = 51 \). (c) The one-sided sign-test p-value (about 0.145) is larger than the exact one-sided signed-rank p-value (\( \frac{14}{256} \approx 0.055 \)). (d) The signed-rank test requires the differences to be normally distributed.
Answer: (a), (b) and (c)
- (a) True. Negative differences have ranks 1 and 5; \( T^+ = 36 - 6 \).
- (b) True. \( \frac{8 \cdot 9}{4} = 18 \); \( \frac{8 \cdot 9 \cdot 17}{24} = 51 \).
- (c) True. Sign test \( \frac{37}{256} \); signed-rank \( P(T^- \le 6) = \frac{14}{256} \).
- (d) False. It needs symmetry about the hypothesised centre, not normality.
Practice Q8. \( X_1, \dots, X_n \) are i.i.d. \( N(\theta, 1) \), and \( \theta \) has prior \( N(0, 1) \). Which statements are true?
(a) The posterior mean is \( \frac{n\bar{X}}{n + 1} \). (b) The posterior variance is \( \frac{1}{n + 1} \). (c) The Bayes estimator under squared-error loss is unbiased for \( \theta \). (d) The posterior median equals the posterior mean.
Answer: (a), (b) and (d)
- (a) True and (b) True. Precisions add: \( 1 + n \). Mean \( \frac{0 \cdot 1 + n\bar{X}}{n + 1} \).
- (c) False. \( E\left(\frac{n\bar{X}}{n+1}\right) = \frac{n\theta}{n + 1} \ne \theta \) for \( \theta \ne 0 \): shrinkage toward the prior mean.
- (d) True. The posterior is normal, hence symmetric.
Quick Revision Table
| Item | Fact |
|---|---|
| Sign test | \( S \sim \) Bin\( (n, \frac{1}{2}) \); continuity only |
| Signed-rank | \( E\,T^+ = \frac{n(n+1)}{4} \), \( \operatorname{Var} = \frac{n(n+1)(2n+1)}{24} \); symmetry |
| Mann–Whitney | \( E\,U = \frac{n_1n_2}{2} \), \( \operatorname{Var} = \frac{n_1n_2(N+1)}{12} \); \( W = U + \frac{n_1(n_1+1)}{2} \) |
| Runs | \( E\,R = 1 + \frac{2n_1n_2}{N} \) |
| Kruskal–Wallis | \( \frac{12}{N(N+1)}\sum \frac{R_i^2}{n_i} - 3(N+1) \approx \chi^2_{k-1} \) |
| KS | \( \sup \lvert F_n - F_0 \rvert \); distribution-free for continuous, fully specified \( F_0 \) |
| ARE vs \( t \) (normal) | Wilcoxon \( 3/\pi \approx 0.955 \); sign \( 2/\pi \approx 0.637 \) |
| Spearman | \( 1 - \frac{6\sum d^2}{n(n^2-1)} \); null variance \( \frac{1}{n-1} \) |
| Kendall | \( \frac{C - D}{n(n-1)/2} \) |
| Posterior | \( \propto \) likelihood \( \times \) prior |
| Conjugates | beta–binomial, gamma–Poisson, gamma–exponential, normal–normal |
| Bayes estimators | mean (squared), median (absolute), mode (0–1) |
| Jeffreys | \( \propto \sqrt{I(\theta)} \); binomial: Beta\( (\frac{1}{2}, \frac{1}{2}) \) |
Key Takeaways
What’s Next
- Distributions and Statistical Inference: Quick Revision: the whole chapter on one page.
- Point Estimation and Confidence Intervals: compare Bayes estimators with UMVUEs and MLEs.
- Chi-Square Goodness of Fit and Large-Sample Tests: the median test and contingency-table methods used above.
- Back to the chapter overview or the Mathematical Sciences overview.