Probability Distributions and the Central Limit Theorem
CSIR NET Physical Sciences: random variables, binomial, Poisson and normal distributions, random walks, counting errors and the central limit theorem.
- 16 min
On this page · 15 sections
- Prerequisites
- The Hook: Why a Geiger Counter Never Ticks Evenly
- Random Variables, Mean and Variance
- The Binomial Distribution
- The Poisson Distribution
- The Normal (Gaussian) Distribution
- The Central Limit Theorem
- Propagation of Errors
- Memory Tricks
- Don’t Confuse
- Common Mistakes
- Practice
- Quick Revision Table
- Key Takeaways
- What’s Next
Prerequisites
- Counting with permutations and combinations, from Counting, Probability and Number Properties (Part A).
- Gaussian integrals and the Fourier transform of a Gaussian, from Fourier Series, Fourier and Laplace Transforms.
- Mean, standard deviation and error propagation as used in the lab, from Mean and Standard Deviation (Chemical Sciences).
The Hook: Why a Geiger Counter Never Ticks Evenly
Random Variables, Mean and Variance
A random variable \( X \) takes values \( x_i \) with probabilities \( p_i \) (discrete) or has a probability density \( f(x) \) with \( P(a < X < b) = \int_a^bf\,dx \) (continuous). In both cases the probabilities sum or integrate to 1.
| Quantity | Discrete | Continuous |
|---|---|---|
| Mean \( \mu = \langle X\rangle \) | \( \sum x_ip_i \) | \( \int xf(x)\,dx \) |
| Variance \( \sigma^2 \) | \( \langle X^2\rangle - \langle X\rangle^2 \) | same |
| Standard deviation | \( \sigma = \sqrt{\text{variance}} \) | same |
Rules that do most of the work.
- \( \langle aX + b\rangle = a\langle X\rangle + b \) and \( \operatorname{Var}(aX + b) = a^2\operatorname{Var}X \).
- Means always add: \( \langle X + Y\rangle = \langle X\rangle + \langle Y\rangle \).
- For independent \( X \), \( Y \): variances add, \( \operatorname{Var}(X + Y) = \operatorname{Var}X + \operatorname{Var}Y \), and \( \operatorname{Var}(X - Y) \) is the same sum, not the difference.
- The characteristic function \( \phi_X(k) = \langle e^{ikX}\rangle \) is the Fourier transform of the density; for independent variables \( \phi_{X+Y} = \phi_X\phi_Y \). The density of a sum is the convolution of the densities.
Two continuous distributions you need in passing.
- Uniform on \( (a, b) \): mean \( \frac{a + b}{2} \), variance \( \frac{(b - a)^2}{12} \).
- Exponential with rate \( \lambda \), \( f(t) = \lambda e^{-\lambda t} \) for \( t \ge 0 \): mean \( 1/\lambda \), variance \( 1/\lambda^2 \). It describes the waiting time between events of a Poisson process (the time to the next decay), and it is memoryless: \( P(T > s + t \mid T > s) = P(T > t) \). A nucleus that has not decayed is as good as new.
The Binomial Distribution
\( n \) independent trials, each a success with probability \( p \) (failure \( q = 1 - p \)). The number of successes \( k \) has
$$\boxed{P(k) = \binom{n}{k}p^kq^{n-k}, \qquad \mu = np, \qquad \sigma^2 = npq}$$The most probable value is near \( np \) (precisely \( \lfloor(n + 1)p\rfloor \), with a tie when \( (n + 1)p \) is an integer). The distribution is symmetric only for \( p = \tfrac{1}{2} \).
Examples. Four fair coin tosses: \( P(\text{2 heads}) = 6/16 = 0.375 \). Ten tosses: \( P(5) = 252/1024 \approx 0.246 \), so even the most likely outcome happens less than a quarter of the time.
The one-dimensional random walk
A walker takes \( N \) steps of length \( l \), each to the right with probability \( p \) and to the left with \( q \). If \( n_R \) steps go right, the displacement is \( x = (2n_R - N)l \). Since \( n_R \) is binomial,
$$\langle x\rangle = N(p - q)l, \qquad \sigma_x^2 = 4Npq\,l^2.$$For the unbiased walk (\( p = q = \tfrac{1}{2} \)): \( \langle x\rangle = 0 \) and
$$\boxed{x_{\text{rms}} = l\sqrt N}$$Diffusion is this result in disguise: with \( N \) proportional to time, the mean-square displacement grows linearly in time, \( \langle x^2\rangle = 2Dt \) in one dimension.
The Poisson Distribution
Let \( n \to \infty \) and \( p \to 0 \) with \( \lambda = np \) fixed. The binomial distribution becomes
$$\boxed{P(k) = \frac{\lambda^ke^{-\lambda}}{k!}, \qquad \mu = \sigma^2 = \lambda}$$Properties.
- Mean equals variance. A count \( N \) therefore has standard deviation \( \sqrt N \) and fractional error \( 1/\sqrt N \): 100 counts give 10%, 10 000 counts give 1%.
- \( P(k + 1)/P(k) = \lambda/(k + 1) \): the probabilities rise while \( k + 1 < \lambda \) and fall afterwards, so the mode is \( \lfloor\lambda\rfloor \) (with a tie at \( \lambda - 1 \) and \( \lambda \) for integer \( \lambda \)).
- The sum of independent Poisson variables is Poisson with the summed mean.
- \( P(0) = e^{-\lambda} \), so \( P(\text{at least one}) = 1 - e^{-\lambda} \).
- For large \( \lambda \) the Poisson distribution approaches a normal distribution with \( \mu = \sigma^2 = \lambda \).
The Normal (Gaussian) Distribution
$$\boxed{f(x) = \frac{1}{\sigma\sqrt{2\pi}}\exp\left[-\frac{(x - \mu)^2}{2\sigma^2}\right]}$$Standardise with \( z = (x - \mu)/\sigma \).
| Range | Probability |
|---|---|
| \( \lvert z\rvert < 1 \) | 0.683 |
| \( \lvert z\rvert < 2 \) | 0.954 |
| \( \lvert z\rvert < 3 \) | 0.997 |
| \( z > 1.645 \) | 0.050 |
| \( z > 1.96 \) | 0.025 |
| \( z > 2 \) | 0.023 |
Facts to have ready.
- Full width at half maximum: \( \text{FWHM} = 2\sqrt{2\ln2}\,\sigma \approx 2.355\sigma \).
- Moments about the mean: \( \langle(x - \mu)^2\rangle = \sigma^2 \), \( \langle(x - \mu)^4\rangle = 3\sigma^4 \); all odd central moments vanish.
- A sum of independent normal variables is normal, with means and variances added.
- Its characteristic function is again a Gaussian, \( e^{i\mu k - \sigma^2k^2/2} \), the probabilistic face of “a Gaussian is its own Fourier transform”.
Normal approximation to the binomial (de Moivre–Laplace): for large \( n \) with \( np \) and \( nq \) both large, the binomial is close to a normal with \( \mu = np \), \( \sigma^2 = npq \). Because the binomial is discrete, use a continuity correction: \( P(X \ge 60) \) becomes \( P(X > 59.5) \) in the normal approximation.
The Central Limit Theorem
Let \( X_1, \ldots, X_N \) be independent and identically distributed with finite mean \( \mu \) and finite variance \( \sigma^2 \). Then as \( N \to \infty \) the standardised sum
$$\boxed{Z_N = \frac{\sum_{i=1}^{N}X_i - N\mu}{\sigma\sqrt N}}$$approaches the standard normal distribution, whatever the distribution of the individual \( X_i \). Equivalently, the sample mean \( \bar X \) is approximately normal with mean \( \mu \) and standard deviation \( \sigma/\sqrt N \) (the standard error).
Why it works. The characteristic function of \( Z_N \) is \( [\phi(k/\sigma\sqrt N)]^N \) (after centring). Expanding \( \phi(k) = 1 - \sigma^2k^2/2 + \cdots \) and taking the limit gives \( e^{-k^2/2} \), the transform of a standard Gaussian. The finite variance is exactly what makes the \( k^2 \) term exist.
Consequences.
- Averaging beats noise as \( 1/\sqrt N \). To reduce the standard error tenfold you need a hundred times as many measurements.
- Gaussian errors are the norm because a measurement error is usually the sum of many small independent effects.
- A classic demonstration: the sum of 12 independent uniform \( (0, 1) \) variables has mean 6 and variance \( 12\times\frac{1}{12} = 1 \), so (sum \( - 6 \)) is already a good approximation to a standard normal variable.
When it fails. If the variance is infinite, the theorem does not apply. The Lorentzian (Cauchy) distribution \( f(x) = \frac{1}{\pi}\frac{\gamma}{x^2 + \gamma^2} \), the line shape of a decaying state, has no finite variance (not even a finite mean in the usual sense). The average of \( N \) Cauchy variables has the same Cauchy distribution as one of them: averaging does not help at all. Its characteristic function \( e^{-\gamma\lvert k\rvert} \) has no \( k^2 \) term, which is where the derivation above breaks.
graph LR
A[Binomial n, p] -->|n large, p small, np = lambda fixed| B[Poisson lambda]
A -->|n large, np and nq large| C[Normal mean np, variance npq]
B -->|lambda large| C
D[Any distribution with finite variance] -->|sum of many independent copies| CPropagation of Errors
For \( f(x_1, \ldots, x_n) \) with independent errors \( \sigma_i \) in each \( x_i \), to first order,
$$\boxed{\sigma_f^2 = \sum_i\left(\frac{\partial f}{\partial x_i}\right)^2\sigma_i^2}$$- Sum or difference: absolute errors add in quadrature.
- Product or quotient: relative errors add in quadrature.
- Power \( f = x^n \): \( \sigma_f/f = \lvert n\rvert\sigma_x/x \).
Memory Tricks
Don’t Confuse
| This | vs | That |
|---|---|---|
| Standard deviation of one measurement \( \sigma \) | Standard error of the mean \( \sigma/\sqrt N \) | |
| \( \operatorname{Var}(X - Y) = \operatorname{Var}X + \operatorname{Var}Y \) (independent) | \( \langle X - Y\rangle = \langle X\rangle - \langle Y\rangle \) | |
| Poisson: \( \sigma^2 = \mu \) | Binomial: \( \sigma^2 = \mu q < \mu \) | |
| Random walk: \( x_{\text{rms}} \propto \sqrt N \) | Ballistic motion: distance \( \propto N \) | |
| CLT: needs finite variance | Cauchy/Lorentzian: averages do not converge | |
| Exponential waiting time: memoryless | Normal or uniform lifetimes: not memoryless |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper.
Level 1: Foundation
- Mean and variance of a binomial distribution with \( n = 10 \), \( p = 0.5 \)?
- For a Poisson distribution with \( \lambda = 2 \), what is \( P(0) \)?
- A Geiger counter registers 10 000 counts. What is the fractional statistical error?
Answers
- Mean 5, variance 2.5. \( np = 5 \), \( npq = 2.5 \).
- \( e^{-2} \approx 0.135 \).
- 1%. \( \sqrt{10000}/10000 = 0.01 \).
Level 2: Part B Style
Practice Q1. A counter records on average 4 counts per second from a steady source. The probability of exactly 2 counts in a given one-second interval is closest to:
(a) 0.073 (b) 0.147 (c) 0.195 (d) 0.25
Answer: (b) 0.147
\( P(2) = \frac{4^2e^{-4}}{2!} = 8e^{-4} \approx 8\times0.0183 = 0.147 \). Option (c) is \( P(3) = P(4) \approx 0.195 \), the mode; (a) is \( P(1) \).
Practice Q2. A particle performs an unbiased one-dimensional random walk of 100 steps, each of length 1 µm. Its root-mean-square displacement from the start is:
(a) 1 µm (b) 10 µm (c) 50 µm (d) 100 µm
Answer: (b) 10 µm
\( x_{\text{rms}} = l\sqrt N = 1\times\sqrt{100} = 10 \) µm. Option (d) is the total path length; (c) confuses the rms displacement with \( N/2 \) steps.
Practice Q3. Twelve independent random numbers are drawn uniformly from \( (0, 1) \) and added. The mean and standard deviation of the sum are:
(a) 6 and 1 (b) 6 and \( \sqrt{12} \) (c) 6 and 12 (d) 0.5 and 1
Answer: (a) 6 and 1
Each term has mean \( \tfrac{1}{2} \) and variance \( \tfrac{1}{12} \). Means add to 6; variances add to 1, so \( \sigma = 1 \). By the central limit theorem the sum is already close to a normal distribution.
Practice Q4. For a Gaussian line shape of standard deviation \( \sigma \), the full width at half maximum is approximately:
(a) \( \sigma \) (b) \( 1.177\sigma \) (c) \( 2\sigma \) (d) \( 2.355\sigma \)
Answer: (d) 2.355σ
Half maximum where \( e^{-x^2/2\sigma^2} = \tfrac{1}{2} \), i.e. \( x = \sigma\sqrt{2\ln2} \approx 1.177\sigma \). The full width is twice that. Option (b) is the half width.
Level 3: Part C Style
Practice Q5. A fair coin is tossed 100 times; \( X \) is the number of heads. Which of the following are correct?
- \( \langle X\rangle = 50 \) and \( \sigma_X = 5 \).
- With the continuity correction, the normal approximation gives \( P(X \ge 60) \approx 0.029 \).
- \( P(40 \le X \le 60) \) is approximately 0.96.
- The Poisson distribution with \( \lambda = 50 \) is a good approximation to the distribution of \( X \).
(a) 1 and 2 only (b) 1, 2 and 3 only (c) 1 and 4 only (d) all four
Answer: (b) 1, 2 and 3 only
Statement 1: \( np = 50 \), \( \sqrt{npq} = \sqrt{25} = 5 \). True.
Statement 2: \( z = (59.5 - 50)/5 = 1.9 \), upper tail \( \approx 0.0287 \). The exact binomial value is 0.0284. True.
Statement 3: with correction, \( P(39.5 < X < 60.5) = P(\lvert z\rvert < 2.1) \approx 0.964 \); the exact value is 0.965. True.
Statement 4: Poisson with \( \lambda = 50 \) has variance 50, twice the true variance 25. \( p = \tfrac{1}{2} \) is not small. False.
Practice Q6. In a counting experiment, a detector records 900 counts in 100 s with a source in place and 400 counts in 100 s with the source removed. Assume Poisson statistics. Which of the following are correct?
- The net count rate due to the source is \( 5.0 \) s\( ^{-1} \).
- The standard deviation of the net rate is about \( 0.36 \) s\( ^{-1} \).
- The standard deviation of the net rate is about \( 0.22 \) s\( ^{-1} \).
- Counting both runs for 400 s instead of 100 s would halve the uncertainty in the net rate.
(a) 1 and 2 only (b) 1, 2 and 4 only (c) 1, 3 and 4 only (d) 1 and 3 only
Answer: (b) 1, 2 and 4 only
Statement 1: \( 9.0 - 4.0 = 5.0 \) s\( ^{-1} \). True.
Statements 2 and 3: the rate errors are \( 30/100 = 0.30 \) and \( 20/100 = 0.20 \) s\( ^{-1} \); add in quadrature: \( \sqrt{0.13} \approx 0.36 \). Statement 2 true; statement 3 (\( \sqrt{0.09 - 0.04} \approx 0.22 \)) subtracts variances. False.
Statement 4: counts grow fourfold, so each count’s relative error falls by \( \sqrt4 = 2 \), and the rate errors become 0.15 and 0.10 s\( ^{-1} \), giving 0.18 s\( ^{-1} \). True.
Practice Q7. Which of the following statements are correct?
- The waiting time between successive decays from a long-lived source is exponentially distributed and memoryless.
- The mean of \( N \) independent measurements, each with standard deviation 0.5, has standard error 0.05 when \( N = 100 \).
- The average of \( N \) independent Lorentzian (Cauchy) distributed variables approaches a Gaussian as \( N \to \infty \).
- If \( X \) and \( Y \) are independent Poisson variables with means 3 and 5, \( X + Y \) is Poisson with mean 8.
(a) 1 and 2 only (b) 1, 2 and 4 only (c) 2, 3 and 4 only (d) all four
Answer: (b) 1, 2 and 4 only
Statement 1: decays form a Poisson process; inter-event times are exponential and memoryless. True.
Statement 2: \( 0.5/\sqrt{100} = 0.05 \). True.
Statement 3: the Cauchy distribution has no finite variance, so the central limit theorem does not apply; the average has the same Cauchy distribution as a single variable. False.
Statement 4: sums of independent Poisson variables are Poisson with added means. True.
Quick Revision Table
| Distribution | \( P \) or \( f \) | Mean | Variance |
|---|---|---|---|
| Binomial | \( \binom{n}{k}p^kq^{n-k} \) | \( np \) | \( npq \) |
| Poisson | \( \lambda^ke^{-\lambda}/k! \) | \( \lambda \) | \( \lambda \) |
| Normal | \( \frac{1}{\sigma\sqrt{2\pi}}e^{-(x-\mu)^2/2\sigma^2} \) | \( \mu \) | \( \sigma^2 \) |
| Uniform \( (a, b) \) | \( 1/(b - a) \) | \( (a + b)/2 \) | \( (b - a)^2/12 \) |
| Exponential | \( \lambda e^{-\lambda t} \) | \( 1/\lambda \) | \( 1/\lambda^2 \) |
| Random walk | \( N \) steps of \( l \) | 0 (unbiased) | \( Nl^2 \) |
| Sample mean (CLT) | approx. normal | \( \mu \) | \( \sigma^2/N \) |
Key Takeaways
What’s Next
- Mathematical Methods: Quick Revision: every result from this chapter on one page.
- Complex Analysis and Residues, the previous topic.
- Data Analysis in Chemical Sciences, for regression and error treatment in experiments.
- Identical Particles, Pauli Principle and Spin-Statistics, where counting arguments meet quantum statistics.