Linear and Nonlinear Curve Fitting and the Chi-Square Test
CSIR NET Physical Sciences: weighted least squares, slope and intercept errors, linearisation, nonlinear fits, chi-square goodness of fit, practice.
- 21 min
On this page · 14 sections
Prerequisites
- Data Interpretation, Error Analysis and Least-Squares Fitting: the core-unit treatment of error types, Poisson counting and the basic straight-line fit. This page goes further: weighting, nonlinear models, the chi-square distribution and goodness-of-fit tests.
- Research aptitude: experimental design, sampling and data: accuracy versus precision, random versus systematic error, mean and standard deviation.
- Linear Algebra, Matrices and Eigenvalues: the normal equations of least squares are a small linear system.
- Graphs of functions and scientific plots: reading log and semi-log plots, which is what linearisation produces.
The Hook: Is That Line Any Good?
The Principle of Least Squares
Suppose you measure \( y_i \) at points \( x_i \) (\( i = 1, \ldots, N \)), each \( y_i \) with standard deviation \( \sigma_i \), and the model is \( y = f(x; a_1, \ldots, a_p) \) with \( p \) parameters. If each \( y_i \) is Gaussian about the true \( f(x_i) \), the probability of the observed data is proportional to
$$\prod_i \exp\left[-\frac{(y_i - f(x_i))^2}{2\sigma_i^2}\right] = \exp\left(-\tfrac{1}{2}\chi^2\right), \qquad \chi^2 = \sum_{i=1}^N \frac{\left[y_i - f(x_i; \mathbf a)\right]^2}{\sigma_i^2}.$$Maximising that likelihood is the same as minimising \( \chi^2 \). That is the whole justification of least squares: it is the maximum-likelihood estimate for Gaussian errors. When every \( \sigma_i \) equals the same \( \sigma \), it reduces to minimising the plain sum of squared residuals.
Straight-Line Fit: \( y = mx + c \)
Equal errors
Setting \( \partial\chi^2/\partial m = \partial\chi^2/\partial c = 0 \) gives the normal equations
$$m\sum x_i^2 + c\sum x_i = \sum x_iy_i, \qquad m\sum x_i + cN = \sum y_i.$$With \( \Delta = N\sum x_i^2 - \left(\sum x_i\right)^2 \):
$$\boxed{m = \frac{N\sum x_iy_i - \sum x_i\sum y_i}{\Delta}, \qquad c = \frac{\sum x_i^2\sum y_i - \sum x_i\sum x_iy_i}{\Delta} = \bar y - m\bar x}$$The best line always passes through the centroid \( (\bar x, \bar y) \). The parameter uncertainties follow from propagating \( \sigma \) through these linear formulas:
$$\boxed{\sigma_m^2 = \frac{N\sigma^2}{\Delta}, \qquad \sigma_c^2 = \frac{\sigma^2\sum x_i^2}{\Delta}}$$If \( \sigma \) is not known in advance, estimate it from the scatter itself, using \( N - 2 \) (not \( N \)) because two parameters were fitted:
$$\sigma^2 \approx \frac{1}{N - 2}\sum_i\left(y_i - mx_i - c\right)^2.$$Unequal errors: weighted least squares
Give each point weight \( w_i = 1/\sigma_i^2 \) and replace every sum: \( N \to \sum w_i \), \( \sum x_i \to \sum w_ix_i \), and so on. Then \( \sigma_m^2 = \sum w_i/\Delta_w \) and \( \sigma_c^2 = \sum w_ix_i^2/\Delta_w \) with \( \Delta_w = \sum w_i\sum w_ix_i^2 - (\sum w_ix_i)^2 \). A point with half the error counts four times as much.
The simplest weighted fit is a constant, \( y = \mu \): the weighted mean
$$\boxed{\bar x_w = \frac{\sum x_i/\sigma_i^2}{\sum 1/\sigma_i^2}, \qquad \sigma_{\bar x_w} = \left(\sum\frac{1}{\sigma_i^2}\right)^{-1/2}}$$which is how independent measurements of the same quantity are combined. For \( N \) equal errors it reduces to the familiar \( \sigma/\sqrt N \).
Correlation between slope and intercept
Slope and intercept are generally correlated: \( \text{cov}(m, c) = -\sigma^2\sum x_i/\Delta \). If all \( x_i > 0 \), tilting the line up lowers the intercept, so the covariance is negative. Shifting the origin to \( \bar x \) (fitting \( y = m(x - \bar x) + c' \)) makes the two estimates independent, which is why \( c' = \bar y \) has the simple error \( \sigma/\sqrt N \).
The linear correlation coefficient
$$r = \frac{\sum(x_i - \bar x)(y_i - \bar y)}{\sqrt{\sum(x_i - \bar x)^2\sum(y_i - \bar y)^2}}, \qquad -1 \leq r \leq 1.$$\( r^2 \) is the fraction of the variance in \( y \) explained by the line. A high \( r \) does not prove the model is right: a gentle parabola sampled over a short range gives \( r > 0.99 \). Judge a fit by its residuals and its \( \chi^2 \), not by \( r \).
Nonlinear Models
Linearisation
Many physics laws become straight lines after a transformation:
| Law | Plot | Slope | Intercept |
|---|---|---|---|
| \( y = Ae^{bx} \) (decay, Arrhenius-type, charging) | \( \ln y \) vs \( x \) | \( b \) | \( \ln A \) |
| \( y = Ax^n \) (power law, Kepler, scaling) | \( \ln y \) vs \( \ln x \) | \( n \) | \( \ln A \) |
| \( y = A/(B + x) \) | \( 1/y \) vs \( x \) | \( 1/A \) | \( B/A \) |
| \( R = R_0e^{E_a/k_BT} \) (thermally activated) | \( \ln R \) vs \( 1/T \) | \( E_a/k_B \) | \( \ln R_0 \) |
| \( y = a + bx^2 \) | \( y \) vs \( x^2 \) | \( b \) | \( a \) |
Polynomial and other linear-in-parameter fits
“Linear least squares” means linear in the parameters, not in \( x \). The model \( y = a_0 + a_1x + a_2x^2 \) or \( y = a\sin x + b\cos x \) is still linear: the normal equations are a \( p \times p \) linear system \( \mathbf{(A^TWA)\,a = A^TWy} \), where \( \mathbf A \) is the design matrix \( A_{ij} = \phi_j(x_i) \) and \( \mathbf W \) the diagonal weight matrix. The parameter covariance matrix is \( (\mathbf{A^TWA})^{-1} \). One exact solve, no iteration.
Truly nonlinear fits
A model such as \( y = A\exp[-(x - x_0)^2/2s^2] + B \) is nonlinear in \( x_0 \) and \( s \). Minimising \( \chi^2 \) then needs iteration from a starting guess:
- Gauss–Newton: linearise \( f \) about the current parameters, solve the linear least-squares problem for the step, repeat. Fast near the minimum, can overshoot far from it.
- Gradient (steepest) descent: step down \( -\nabla\chi^2 \). Robust far from the minimum, slow near it.
- Levenberg–Marquardt: interpolates between the two with a damping parameter; the standard choice in fitting software.
Practical consequences that Part C can test: a nonlinear fit can converge to a local minimum, so starting values matter; it needs at least as many points as parameters (in practice many more); and the curvature of \( \chi^2 \) at the minimum gives the parameter errors.
Reading errors from the chi-square surface
Near its minimum, \( \chi^2 \) is a paraboloid in the parameters. For one parameter \( a \),
$$\chi^2(a) \approx \chi^2_{\min} + \frac{(a - \hat a)^2}{\sigma_a^2}, \qquad \sigma_a^{-2} = \frac{1}{2}\frac{\partial^2\chi^2}{\partial a^2}.$$So the \( 1\sigma \) interval is where \( \chi^2 \) rises by 1 above its minimum (with the other parameters re-optimised at each step). This “\( \Delta\chi^2 = 1 \)” rule works for linear and nonlinear fits alike.
Propagation of Errors
For a result \( q = f(x, y, \ldots) \) with independent errors,
$$\sigma_q^2 = \left(\frac{\partial f}{\partial x}\right)^2\sigma_x^2 + \left(\frac{\partial f}{\partial y}\right)^2\sigma_y^2 + \cdots$$| Form | Rule |
|---|---|
| \( q = x \pm y \) | absolute errors add in quadrature |
| \( q = xy \) or \( x/y \) | fractional errors add in quadrature |
| \( q = x^n \) | \( \sigma_q/q = \lvert n\rvert\,\sigma_x/x \) |
| \( q = \ln x \) | \( \sigma_q = \sigma_x/x \) |
| \( q = e^x \) | \( \sigma_q/q = \sigma_x \) |
Correlated variables need the covariance term \( 2\frac{\partial f}{\partial x}\frac{\partial f}{\partial y}\text{cov}(x, y) \); this is where the slope–intercept correlation matters if you compute, say, the \( x \)-intercept \( -c/m \).
The Chi-Square Statistic and Its Distribution
If the model is right and the \( \sigma_i \) are right, each term \( (y_i - f_i)/\sigma_i \) is a standard normal variable. After fitting \( p \) parameters from \( N \) points, \( \chi^2 \) follows the chi-square distribution with
$$\boxed{\nu = N - p \text{ degrees of freedom}, \qquad \langle\chi^2\rangle = \nu, \qquad \text{Var}(\chi^2) = 2\nu}$$so a good fit has \( \chi^2 \approx \nu \pm \sqrt{2\nu} \). The reduced chi-square \( \chi^2_\nu = \chi^2/\nu \) should be close to 1.
| \( \chi^2_\nu \) | Meaning | What to check |
|---|---|---|
| \( \approx 1 \) | Fit consistent with the errors | nothing; quote the parameters |
| \( \gg 1 \) | Scatter larger than the error bars allow | wrong model, underestimated \( \sigma_i \), outliers, unaccounted systematic error |
| \( \ll 1 \) | Scatter smaller than the error bars imply | overestimated \( \sigma_i \), too many parameters (over-fitting), correlated or fabricated data |
The p-value is the probability that a correct model would give a \( \chi^2 \) at least as large as observed. A p-value below 0.05 (or 0.01) is the usual signal to reject the model. Upper critical values at the 5% level, worth remembering for the first few \( \nu \):
| \( \nu \) | 1 | 2 | 3 | 4 | 5 | 6 | 10 |
|---|---|---|---|---|---|---|---|
| \( \chi^2_{0.05} \) | 3.84 | 5.99 | 7.81 | 9.49 | 11.07 | 12.59 | 18.31 |
For large \( \nu \) the distribution is close to Gaussian with mean \( \nu \) and standard deviation \( \sqrt{2\nu} \), so \( \chi^2 > \nu + 2\sqrt{2\nu} \) is suspicious.
Pearson’s chi-square test for counts
For data in \( k \) bins with observed counts \( O_j \) and expected counts \( E_j \) (from a hypothesis such as “the die is fair” or “the counts are Poisson”):
$$\boxed{\chi^2 = \sum_{j=1}^k\frac{(O_j - E_j)^2}{E_j}}$$This is the same statistic with \( \sigma_j^2 = E_j \) (Poisson variance). Degrees of freedom: \( \nu = k - 1 - (\text{number of parameters estimated from the data}) \). The \( -1 \) is for the constraint that the total count is fixed. The test needs expected counts of about 5 or more per bin; merge sparse bins first.
Memory Tricks
Don’t Confuse
| This | vs | That |
|---|---|---|
| Linear least squares: linear in the parameters (\( a + bx + cx^2 \) qualifies) | Nonlinear: parameters inside functions (\( Ae^{-x/\tau} \) in \( \tau \)); needs iteration | |
| \( \nu = N - p \) for a curve fit | \( \nu = k - 1 - p \) for a binned Pearson test | |
| \( \chi^2_\nu \gg 1 \): bad model or errors too small | \( \chi^2_\nu \ll 1 \): errors too large or over-fitting | |
| Correlation coefficient \( r \): strength of linear association | \( \chi^2 \): agreement with the error bars | |
| Standard deviation of data \( \sigma \) | Standard error of the mean \( \sigma/\sqrt N \) | |
| Precision (small random scatter) | Accuracy (small systematic offset); a fit cannot remove a systematic error |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper.
Level 1: Foundation
- A straight line is fitted to 12 data points. How many degrees of freedom does the \( \chi^2 \) have?
- Which plot turns \( y = Ax^n \) into a straight line, and what is its slope?
- Three measurements of the same quantity have equal errors \( \sigma \). What is the error of their mean?
Answers
- 10. \( \nu = N - p = 12 - 2 \).
- \( \ln y \) against \( \ln x \), slope \( n \), intercept \( \ln A \).
- \( \sigma/\sqrt 3 \). The weighted-mean formula with equal weights.
Level 2: Part B Style
Practice Q1. The least-squares straight line through the points \( (0, 1) \), \( (1, 3) \), \( (2, 4) \), \( (3, 6) \) is:
(a) \( y = 1.5x + 1.2 \) (b) \( y = 1.6x + 1.1 \) (c) \( y = 1.7x + 1.0 \) (d) \( y = 1.6x + 1.0 \)
Answer: (b) y = 1.6x + 1.1
\( N = 4 \), \( \sum x = 6 \), \( \sum y = 14 \), \( \sum x^2 = 14 \), \( \sum xy = 0 + 3 + 8 + 18 = 29 \). \( \Delta = 4(14) - 36 = 20 \).
\( m = (4 \times 29 - 6 \times 14)/20 = 32/20 = 1.6 \). \( c = \bar y - m\bar x = 3.5 - 1.6 \times 1.5 = 1.1 \).
Check: the line passes through the centroid \( (1.5, 3.5) \). Option (d) fails that check.
Practice Q2. A straight line fitted to 20 points with quoted errors gives \( \chi^2 = 45 \). The most reasonable conclusion is:
(a) the fit is excellent (b) the errors are overestimated (c) the errors are underestimated or the model is inadequate (d) more parameters are needed to reduce \( \nu \)
Answer: (c)
\( \nu = 18 \), so \( \chi^2_\nu = 2.5 \). For \( \nu = 18 \), \( \chi^2 \) should lie near \( 18 \pm 6 \); 45 is about 4.5 standard deviations high. Either the error bars are too small or a straight line is the wrong model. Option (b) is the reading for \( \chi^2_\nu \ll 1 \).
Practice Q3. A die is rolled 60 times, giving counts 8, 12, 9, 15, 6, 10 for faces 1 to 6. Testing the hypothesis that the die is fair, the chi-square value and the degrees of freedom are:
(a) 5.0, 5 (b) 5.0, 6 (c) 0.5, 5 (d) 50, 5
Answer: (a) 5.0 and 5
Deviations from \( E = 10 \): \( 8 - 10 = -2 \), \( 12 - 10 = 2 \), \( 9 - 10 = -1 \), \( 15 - 10 = 5 \), \( 6 - 10 = -4 \), \( 10 - 10 = 0 \). Squares: 4, 4, 1, 25, 16, 0, total 50. So \( \chi^2 = 50/10 = 5.0 \).
\( \nu = 6 - 1 = 5 \) (one constraint: the total is fixed at 60). Since \( 5.0 < 11.07 \), the die is consistent with fair. Option (d) forgets to divide by \( E \); option (b) forgets the constraint.
Practice Q4. Values \( y_i \) with constant absolute error \( \sigma \) are to be fitted to \( y = Ae^{-x/\tau} \) by plotting \( \ln y \) against \( x \). The appropriate weight for point \( i \) in the straight-line fit is proportional to:
(a) 1 (b) \( y_i \) (c) \( y_i^2 \) (d) \( 1/y_i^2 \)
Answer: (c) y²
\( \sigma_{\ln y} = \sigma/y \), so \( w = 1/\sigma_{\ln y}^2 = y^2/\sigma^2 \propto y^2 \). The large-\( y \) points, which are the most precise on a log scale, dominate. Option (b) is the weight for Poisson counts (\( \sigma_y = \sqrt y \)).
Level 3: Part C Style
Practice Q5. A short-lived source is counted for equal short intervals starting at \( t = 0, 10, 20, 30 \) min, giving (background-corrected) 1000, 610, 370 and 225 counts. A straight-line fit of \( \ln N \) against \( t \) gives a half-life closest to:
(a) 10 min (b) 14 min (c) 20 min (d) 28 min
Answer: (b) 14 min
\( \ln N = 6.908, 6.413, 5.914, 5.416 \). The successive differences are \( -0.494, -0.500, -0.497 \) per 10 min, so the slope is \( -\lambda \approx -0.0497 \) min\( ^{-1} \) (a full least-squares fit gives \( -0.0497 \)).
\( t_{1/2} = \ln 2/\lambda = 0.693/0.0497 \approx 13.9 \) min.
Two checks: the counts fall by roughly 0.61 every 10 min, and \( 0.61^{1.4} \approx 0.5 \), consistent with 14 min. With counts of order 1000, Poisson errors are about 3%, so the near-perfect line is reasonable; a proper fit would weight each point by \( N_i \).
Practice Q6. Three laboratories report measurements of the same quantity: \( 10.2 \pm 0.2 \), \( 9.8 \pm 0.2 \) and \( 10.5 \pm 0.5 \) (arbitrary units). Consider:
- The weighted mean is about 10.04.
- The error of the weighted mean is about 0.14.
- The \( \chi^2 \) of the three values about the weighted mean is about 2.9 with 2 degrees of freedom, so they are mutually consistent at the 5% level.
- The unweighted mean, 10.17, is a better estimate because it treats all laboratories equally.
Which are correct?
(a) 1 and 2 only (b) 1, 2 and 3 only (c) 3 and 4 only (d) 1, 2, 3 and 4
Answer: (b) 1, 2 and 3 only
Weights \( 1/\sigma^2 = 25, 25, 4 \), total 54.
Statement 1: \( \bar x_w = (25 \times 10.2 + 25 \times 9.8 + 4 \times 10.5)/54 = 542/54 = 10.037 \). True.
Statement 2: \( \sigma = 1/\sqrt{54} = 0.136 \). True.
Statement 3: \( \chi^2 = \left(\frac{0.163}{0.2}\right)^2 + \left(\frac{-0.237}{0.2}\right)^2 + \left(\frac{0.463}{0.5}\right)^2 = 0.66 + 1.41 + 0.86 = 2.93 \). One parameter (the mean) was fitted, so \( \nu = 2 \); critical value 5.99. Consistent. True.
Statement 4: the unweighted mean gives the least precise result as much influence as the best ones; it is unbiased but has a larger error. False.
Practice Q7. Five points with \( \sigma = 0.2 \) each are fitted by \( y = mx + c \), giving \( m = 1.97 \pm 0.06 \), \( c = 0.09 \pm 0.21 \), \( \chi^2 = 2.3 \). A student then fits a quartic polynomial (five parameters) to the same five points. Consider:
- The quartic passes through every point, so its \( \chi^2 = 0 \).
- The quartic has zero degrees of freedom, so its \( \chi^2 \) says nothing about the quality of the model.
- The straight-line fit, with \( \chi^2_\nu \approx 0.76 \), is acceptable.
- Because \( c \) is within one standard deviation of zero, the data are consistent with a line through the origin.
Which are correct?
(a) 1 and 3 only (b) 1, 2 and 3 only (c) 1, 2, 3 and 4 (d) 2 and 4 only
Answer: (c) 1, 2, 3 and 4
Statement 1: five parameters through five points with distinct \( x \) interpolate exactly. True.
Statement 2: \( \nu = 5 - 5 = 0 \); a zero \( \chi^2 \) with no degrees of freedom carries no information. True. This is over-fitting in its extreme form.
Statement 3: \( \nu = 3 \), \( \chi^2_\nu = 2.3/3 \approx 0.76 \), well inside the expected range \( 3 \pm 2.4 \). True.
Statement 4: \( 0.09/0.21 \approx 0.4\sigma \). True. Refitting \( y = mx \) would then give a slope with a smaller error, since one fewer parameter is estimated.
Quick Revision Table
| Item | Result |
|---|---|
| Least squares | maximum likelihood for Gaussian errors: minimise \( \chi^2 = \sum(y_i - f_i)^2/\sigma_i^2 \) |
| Line slope | \( m = (N\sum xy - \sum x\sum y)/\Delta \), \( \Delta = N\sum x^2 - (\sum x)^2 \) |
| Line intercept | \( c = \bar y - m\bar x \); line passes through centroid |
| Errors | \( \sigma_m^2 = N\sigma^2/\Delta \), \( \sigma_c^2 = \sigma^2\sum x^2/\Delta \) |
| \( \sigma \) from scatter | \( \sum r_i^2/(N - 2) \) |
| Weighted mean | \( \sum(x_i/\sigma_i^2)/\sum(1/\sigma_i^2) \), error \( (\sum 1/\sigma_i^2)^{-1/2} \) |
| Linearised log fit | weight \( y_i^2/\sigma_{y_i}^2 \) |
| Degrees of freedom | \( N - p \) (fit); \( k - 1 - p \) (binned test) |
| \( \chi^2 \) distribution | mean \( \nu \), variance \( 2\nu \) |
| 5% critical values | 3.84 (1), 5.99 (2), 7.81 (3), 9.49 (4), 11.07 (5) |
| Parameter error from surface | \( \Delta\chi^2 = 1 \) |
| Pearson test | \( \sum(O - E)^2/E \), need \( E \gtrsim 5 \) per bin |
Key Takeaways
What’s Next
- Transducers and Detectors: where the data come from, and why counting detectors give Poisson errors.
- Noise, Filtering, Shielding and Grounding: the physical origin of the \( \sigma_i \), and how averaging \( N \) times reduces them by \( \sqrt N \).
- Electronics and Experimental Methods (Advanced): Quick Revision.
- Back to the Physical Sciences overview.