Descriptive Statistics and Exploratory Data Analysis
CSIR NET statistics: mean, median, quantiles, moments, skewness, kurtosis, correlation, regression lines, box plots and outlier rules, with practice.
- 20 min
On this page · 14 sections
Prerequisites
- Part A reading of graphs from Histograms, Distributions, Scatter and Box Plots. This page assumes you can read a histogram and a box plot and goes straight to the algebra behind them.
- Weighted averages from Percentages, Ratios, Averages and Mixtures, used for combined means and variances.
- The Cauchy–Schwarz inequality from Inner Product Spaces and Orthonormal Bases. It is the whole reason \( \lvert r \rvert \le 1 \).
The Hook: One Number Is Never Enough
Measures of Location
For observations \( x_1, \dots, x_n \):
| Measure | Definition | Minimises | Resistant to outliers? |
|---|---|---|---|
| Arithmetic mean \( \bar{x} \) | \( \frac{1}{n}\sum x_i \) | \( \sum (x_i - a)^2 \) | no |
| Median \( \tilde{x} \) | middle order statistic (average of the two middle ones if \( n \) even) | \( \sum \lvert x_i - a \rvert \) | yes |
| Mode | most frequent value (grouped: peak of the histogram) | number of \( x_i \ne a \) | yes |
| Geometric mean \( G \) | \( \left(\prod x_i\right)^{1/n} \), \( x_i > 0 \) | no | |
| Harmonic mean \( H \) | \( \frac{n}{\sum 1/x_i} \), \( x_i > 0 \) | no | |
| Trimmed mean | mean after dropping a fixed fraction from each end | partly |
The “minimises” column is worth knowing as a theorem. Expanding,
$$\sum_{i=1}^{n} (x_i - a)^2 = \sum_{i=1}^{n} (x_i - \bar{x})^2 + n(\bar{x} - a)^2,$$so the sum of squares is smallest exactly at \( a = \bar{x} \), and \( \sum (x_i - \bar{x}) = 0 \). The function \( a \mapsto \sum \lvert x_i - a \rvert \) is convex and piecewise linear with slope (number of \( x_i < a \)) minus (number of \( x_i > a \)), which changes sign at the median. For even \( n \) every point between the two middle values is a minimiser.
Means inequality. For positive data \( \bar{x} \ge G \ge H \), with equality iff all \( x_i \) are equal. For two numbers, \( G^2 = \bar{x} H \).
Grouped data. With class midpoints \( m_j \), frequencies \( f_j \), \( N = \sum f_j \): \( \bar{x} = \frac{1}{N}\sum f_j m_j \). Median \( = L + \frac{N/2 - C}{f}\, h \), where \( L \) is the lower boundary of the median class, \( C \) the cumulative frequency before it, \( f \) its frequency and \( h \) its width. Mode \( = L + \frac{f_1 - f_0}{2f_1 - f_0 - f_2}\, h \) for the modal class with neighbouring frequencies \( f_0, f_2 \).
Quantiles
The \( p \)-th quantile is a value with at least a fraction \( p \) of the data at or below it and at least \( 1 - p \) at or above it. Quartiles are \( Q_1, Q_2 = \tilde{x}, Q_3 \). For small samples, software and textbooks use several slightly different conventions (interpolation rules) for \( Q_1 \) and \( Q_3 \). A CSIR NET question either states one or gives data where they agree; if it states nothing, use the median of each half.
Measures of Dispersion
| Measure | Formula | Notes |
|---|---|---|
| Range | \( x_{(n)} - x_{(1)} \) | depends only on the extremes |
| Interquartile range | \( Q_3 - Q_1 \) | resistant; quartile deviation is half of it |
| Mean deviation about \( a \) | \( \frac{1}{n}\sum \lvert x_i - a \rvert \) | least when \( a \) is the median |
| Variance | \( s^2 = \frac{1}{n}\sum (x_i - \bar{x})^2 = \overline{x^2} - \bar{x}^2 \) | descriptive convention: divisor \( n \) |
| Sample variance | \( \frac{1}{n-1}\sum (x_i - \bar{x})^2 \) | unbiased for \( \sigma^2 \) (an inference fact) |
| Standard deviation | \( s = \sqrt{s^2} \) | same units as the data |
| Coefficient of variation | \( \text{CV} = \frac{s}{\bar{x}} \times 100\% \) | unit-free; compares consistency |
| Median absolute deviation | median of \( \lvert x_i - \tilde{x} \rvert \) | the resistant analogue of \( s \) |
Useful bounds. \( \text{MD about mean} \le s \) (Jensen or Cauchy–Schwarz). For data in \( [a, b] \), \( s \le \frac{b - a}{2} \), with equality when half the data sit at each end. Also \( s \le \frac{\text{range}}{2} \) and \( s^2 \ge \frac{(\text{range})^2}{2n} \).
Change of origin and scale
If \( u_i = \frac{x_i - a}{h} \), then
$$\bar{x} = a + h\bar{u}, \qquad s_x^2 = h^2 s_u^2, \qquad s_x = \lvert h \rvert s_u.$$Location measures move with both origin and scale; dispersion measures ignore the origin and scale by \( \lvert h \rvert \); variance scales by \( h^2 \).
Pooling two groups
Groups of sizes \( n_1, n_2 \), means \( \bar{x}_1, \bar{x}_2 \), variances \( s_1^2, s_2^2 \) (divisor \( n \)). With \( \bar{x} = \frac{n_1\bar{x}_1 + n_2\bar{x}_2}{n_1 + n_2} \) and \( d_i = \bar{x}_i - \bar{x} \),
$$\boxed{s^2 = \frac{n_1(s_1^2 + d_1^2) + n_2(s_2^2 + d_2^2)}{n_1 + n_2}}$$This is the data form of the law of total variance (within-group plus between-group), which reappears on the joint and conditional distributions page.
Moments, Skewness and Kurtosis
Raw moments about a point \( A \): \( \mu_r' = \frac{1}{n}\sum (x_i - A)^r \). Central moments: \( \mu_r = \frac{1}{n}\sum (x_i - \bar{x})^r \). So \( \mu_0 = 1 \), \( \mu_1 = 0 \), \( \mu_2 = s^2 \). Expanding binomially, with \( \mu_1' = \bar{x} - A \):
$$\mu_2 = \mu_2' - \mu_1'^2, \qquad \mu_3 = \mu_3' - 3\mu_2'\mu_1' + 2\mu_1'^3, \qquad \mu_4 = \mu_4' - 4\mu_3'\mu_1' + 6\mu_2'\mu_1'^2 - 3\mu_1'^4.$$Central moments do not depend on the origin; \( \mu_r \) scales by \( h^r \).
Pearson’s coefficients.
$$\beta_1 = \frac{\mu_3^2}{\mu_2^3}, \quad \gamma_1 = \frac{\mu_3}{\mu_2^{3/2}}, \qquad \beta_2 = \frac{\mu_4}{\mu_2^2}, \quad \gamma_2 = \beta_2 - 3.$$- \( \gamma_1 \) carries the sign of skewness; \( \beta_1 \ge 0 \) does not. Both are unchanged by origin and by positive rescaling; a negative scale factor flips the sign of \( \gamma_1 \).
- \( \beta_2 = 3 \) for the normal distribution (mesokurtic); \( \beta_2 > 3 \) leptokurtic; \( \beta_2 < 3 \) platykurtic.
- Always \( \beta_2 \ge 1 \), and more sharply \( \boxed{\beta_2 \ge \beta_1 + 1} \). Equality holds only for a distribution on two points.
| Distribution | \( \gamma_1 \) | \( \beta_2 \) |
|---|---|---|
| Normal | 0 | 3 |
| Uniform on an interval | 0 | 1.8 |
| Exponential | 2 | 9 |
| Laplace (double exponential) | 0 | 6 |
| Symmetric two-point (\( \pm 1 \)) | 0 | 1 |
Other skewness measures. Pearson’s first coefficient \( \frac{\bar{x} - \text{mode}}{s} \); Bowley’s quartile coefficient \( \frac{Q_3 + Q_1 - 2Q_2}{Q_3 - Q_1} \), which lies in \( [-1, 1] \). For moderately skewed unimodal data the empirical relation \( \text{mean} - \text{mode} \approx 3(\text{mean} - \text{median}) \) holds; it is a rule of thumb, not a theorem.
For a right-skewed unimodal distribution the usual order is mode \( < \) median \( < \) mean. This too is a tendency: there are distributions that break it. Part C may test this exact point.
Sheppard’s correction. For grouped data with equal class width \( h \), a smooth distribution and tails tapering to zero, the variance computed from midpoints overstates the true variance by about \( \frac{h^2}{12} \); the corrected value is \( \mu_2 - \frac{h^2}{12} \).
Bivariate Data: Correlation and Regression
For pairs \( (x_i, y_i) \):
$$\text{cov}(x, y) = \frac{1}{n}\sum (x_i - \bar{x})(y_i - \bar{y}) = \overline{xy} - \bar{x}\bar{y}, \qquad r = \frac{\text{cov}(x, y)}{s_x s_y}.$$Properties of \( r \).
- \( -1 \le r \le 1 \) by Cauchy–Schwarz on the centred vectors. \( \lvert r \rvert = 1 \) iff all points lie on one non-horizontal, non-vertical line.
- If \( u = \frac{x - a}{h} \), \( v = \frac{y - b}{k} \), then \( r_{uv} = \text{sign}(hk)\, r_{xy} \). Correlation is free of origin and scale, apart from sign.
- \( r \) measures linear association only. \( y = x^2 \) on data symmetric about 0 gives \( r = 0 \) with perfect dependence.
- \( r \) is not resistant: one far point can create or destroy a large correlation.
The two regression lines
Least squares of \( y \) on \( x \) minimises vertical squared distances; of \( x \) on \( y \), horizontal ones. Both pass through \( (\bar{x}, \bar{y}) \).
$$y - \bar{y} = b_{yx}(x - \bar{x}), \quad b_{yx} = r\frac{s_y}{s_x}; \qquad x - \bar{x} = b_{xy}(y - \bar{y}), \quad b_{xy} = r\frac{s_x}{s_y}.$$| Fact | Consequence |
|---|---|
| \( b_{yx} b_{xy} = r^2 \le 1 \) | test for which line is which |
| same sign as \( r \) | both positive, both negative, or both zero |
| \( r = \text{sign} \cdot \sqrt{b_{yx} b_{xy}} \) | \( r \) is the geometric mean of the slopes |
| \( \frac{b_{yx} + b_{xy}}{2} \ge \lvert r \rvert \) when \( r \ne 0 \) | AM–GM |
| \( b_{vu} = \frac{h}{k} b_{yx} \) | slopes ignore origin, not scale |
| \( r = \pm 1 \) | the two lines coincide |
| \( r = 0 \) | lines \( y = \bar{y} \) and \( x = \bar{x} \): perpendicular |
The acute angle \( \theta \) between the lines satisfies
$$\tan\theta = \frac{1 - r^2}{\lvert r \rvert} \cdot \frac{s_x s_y}{s_x^2 + s_y^2}.$$Coefficient of determination. With \( \hat{y}_i \) the fitted values, \( \frac{1}{n}\sum (y_i - \hat{y}_i)^2 = s_y^2(1 - r^2) \). So \( r^2 \) is the fraction of the variance of \( y \) “explained” by the linear fit.
Exploratory Data Analysis
EDA, as developed by John Tukey, means looking at the data with resistant, graphical summaries before fitting models.
Five-number summary: \( x_{(1)}, Q_1, \tilde{x}, Q_3, x_{(n)} \). The box plot draws it: a box from \( Q_1 \) to \( Q_3 \), a line at the median, whiskers to the most extreme points inside the fences.
Tukey’s fences.
$$\text{inner: } Q_1 - 1.5\,\text{IQR}, \ Q_3 + 1.5\,\text{IQR}; \qquad \text{outer: } Q_1 - 3\,\text{IQR}, \ Q_3 + 3\,\text{IQR}.$$Points beyond the inner fences are flagged as possible outliers, beyond the outer fences as extreme. A flag is a reason to look, not to delete.
Other EDA tools.
| Tool | What it shows |
|---|---|
| Stem-and-leaf display | the shape of a histogram while keeping every value |
| Histogram | shape, modality, gaps; area is proportional to frequency |
| Box plot | centre, spread, skew (median off-centre in the box, unequal whiskers), outliers; side-by-side comparison of groups |
| Normal Q–Q plot | sample quantiles against normal quantiles; a straight line means approximately normal, an S-shape means tails heavier or lighter than normal |
| Scatter plot | form, direction and strength of association; outliers and clusters that \( r \) hides |
| Residual plot | pattern in residuals means the linear model is missing something |
| Transformations (log, square root) | reduce right skew; turn multiplicative effects into additive ones |
Resistance. A statistic is resistant if changing a small part of the data, however wildly, changes it only a little. The median, the IQR, the MAD and trimmed means are resistant; the mean, standard deviation, range, \( r \) and least-squares slopes are not.
graph TD
A[Raw data] --> B[Plot first: stem-and-leaf, histogram, box plot, scatter]
B --> C{Outliers or strong skew?}
C -->|Yes| D[Resistant summaries: median, IQR, MAD, consider a transformation]
C -->|No| E[Mean, s, moments, r, least-squares line]
D --> F[Check residuals and Q-Q plot]
E --> FMemory Tricks
Don’t Confuse
| This | vs | That |
|---|---|---|
| \( b_{yx} = r s_y / s_x \) | \( b_{xy} = r s_x / s_y \) | |
| \( r \) free of origin and scale | regression slopes free of origin only | |
| \( \beta_1 = \mu_3^2/\mu_2^3 \) (no sign) | \( \gamma_1 = \mu_3/\mu_2^{3/2} \) (signed) | |
| \( \beta_2 \) (normal = 3) | excess kurtosis \( \gamma_2 = \beta_2 - 3 \) (normal = 0) | |
| variance with divisor \( n \) (description) | divisor \( n - 1 \) (unbiased estimation) | |
| \( r = 0 \): no linear relation | independence | |
| outlier flagged by a fence | an observation that must be removed |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C of Mathematical Sciences, one or more options may be correct.
Level 1: Foundation
- Find the mean and the variance (divisor \( n \)) of 2, 4, 4, 4, 5, 5, 7, 9.
- Which value of \( a \) minimises \( \sum \lvert x_i - a \rvert \)?
- If \( b_{yx} = 0.8 \) and \( b_{xy} = 0.45 \), find \( r \).
Answers
- Sum \( = 40 \), so \( \bar{x} = 5 \). Squared deviations: 9, 1, 1, 1, 0, 0, 4, 16, total 32, so \( s^2 = 4 \) and \( s = 2 \).
- Any median of the data (unique when \( n \) is odd).
- \( r^2 = 0.8 \times 0.45 = 0.36 \), both slopes positive, so \( r = 0.6 \).
Level 2: Part B Style
Practice Q1. Group I has 10 observations with mean 20 and variance 4; group II has 15 observations with mean 25 and variance 9 (variances with divisor \( n \)). The variance of the combined 25 observations is:
(a) 7 (b) 13 (c) 6 (d) 6.5
Answer: (b)
Combined mean \( = \frac{200 + 375}{25} = 23 \), so \( d_1 = -3 \), \( d_2 = 2 \).
$$s^2 = \frac{10(4 + 9) + 15(9 + 4)}{25} = \frac{130 + 195}{25} = 13.$$Option (a) is the within-group part alone, \( \frac{10 \cdot 4 + 15 \cdot 9}{25} = 7 \); option (c) is the between-group part alone, \( \frac{10 \cdot 9 + 15 \cdot 4}{25} = 6 \). The pooled variance is their sum.
Practice Q2. For a bivariate data set the regression lines are \( 3x + 2y = 26 \) and \( 6x + y = 31 \). The correlation coefficient is:
(a) \( -\frac{1}{2} \) (b) \( \frac{1}{2} \) (c) \( -2 \) (d) \( -\frac{1}{4} \)
Answer: (a)
See the worked example above: the only consistent assignment gives \( b_{yx} = -\frac{3}{2} \), \( b_{xy} = -\frac{1}{6} \), so \( r^2 = \frac{1}{4} \) and \( r \) is negative. (d) is \( r^2 \) with a sign attached; (c) is impossible since \( \lvert r \rvert \le 1 \).
Practice Q3. The correlation between \( x \) and \( y \) is 0.4. If \( u = 3 - 2x \) and \( v = 5y + 1 \), the correlation between \( u \) and \( v \) is:
(a) 0.4 (b) \( -0.4 \) (c) \( -4 \) (d) \( -0.16 \)
Answer: (b)
Origins do not matter; the scale factors are \( -2 \) and \( 5 \), whose product is negative, so \( r_{uv} = -r_{xy} = -0.4 \).
Practice Q4. The first four moments of a distribution about the value 4 are 1, 4, 10 and 45. The distribution is:
(a) symmetric and leptokurtic (b) positively skewed and platykurtic (c) symmetric and platykurtic (d) negatively skewed and mesokurtic
Answer: (c)
\( \mu_1' = 1 \), so the mean is 5. Then \( \mu_2 = 4 - 1 = 3 \), \( \mu_3 = 10 - 3(4)(1) + 2(1) = 0 \), \( \mu_4 = 45 - 4(10)(1) + 6(4)(1) - 3(1) = 26 \). So \( \beta_1 = 0 \) (symmetric) and \( \beta_2 = \frac{26}{9} \approx 2.89 < 3 \) (platykurtic).
Level 3: Part C Style
Practice Q5. Consider the data 2, 3, 3, 5, 7, 8, 9, 40. Take \( Q_1 \) and \( Q_3 \) to be the medians of the lower four and upper four values. Which statements are true?
(a) The mean exceeds the median. (b) The interquartile range is 5.5. (c) The value 40 lies beyond the upper inner fence \( Q_3 + 1.5\,\text{IQR} \). (d) Deleting the value 40 changes the median by more than it changes the mean.
Answer: (a), (b) and (c)
- (a) True. Mean \( = \frac{77}{8} = 9.625 \); median \( = \frac{5 + 7}{2} = 6 \).
- (b) True. \( Q_1 = \frac{3 + 3}{2} = 3 \), \( Q_3 = \frac{8 + 9}{2} = 8.5 \), IQR \( = 5.5 \).
- (c) True. Upper inner fence \( = 8.5 + 8.25 = 16.75 < 40 \).
- (d) False. Without 40 the median is 5 (a change of 1) and the mean is \( \frac{37}{7} \approx 5.29 \) (a change of about 4.34). This is resistance in one line: the median barely moves, the mean collapses.
Practice Q6. For bivariate data with non-zero variances, let \( b_{yx} \) and \( b_{xy} \) be the two regression coefficients and \( r \) the correlation. Which statements are always true?
(a) \( b_{yx} \) and \( b_{xy} \) have the same sign. (b) If \( b_{yx} = 1.6 \), then \( b_{xy} \le 0.625 \). (c) \( r \) equals the arithmetic mean of \( b_{yx} \) and \( b_{xy} \). (d) If \( r = 0 \), the two regression lines are perpendicular.
Answer: (a), (b) and (d)
- (a) True. Both equal \( \text{cov}(x, y) \) divided by a positive variance.
- (b) True. \( b_{yx} b_{xy} = r^2 \le 1 \) and \( b_{xy} > 0 \), so \( b_{xy} \le \frac{1}{1.6} = 0.625 \).
- (c) False. \( r \) is (up to sign) the geometric mean; the arithmetic mean is at least \( \lvert r \rvert \) and usually larger. Example: \( b_{yx} = 1.6 \), \( b_{xy} = 0.4 \) gives \( r = 0.8 \) but the average is 1.
- (d) True. With \( r = 0 \) the lines are \( y = \bar{y} \) and \( x = \bar{x} \).
Practice Q7. For 20 observations, \( \sum (x_i - 10) = 40 \) and \( \sum (x_i - 10)^2 = 480 \). Which statements are true (variance with divisor \( n \))?
(a) The mean is 12. (b) The variance is 20. (c) \( \sum (x_i - 12)^2 = 400 \). (d) The coefficient of variation exceeds 40%.
Answer: (a), (b) and (c)
- (a) True. \( \bar{x} = 10 + \frac{40}{20} = 12 \).
- (b) True. \( s^2 = \frac{480}{20} - \left(\frac{40}{20}\right)^2 = 24 - 4 = 20 \).
- (c) True. \( \sum (x_i - \bar{x})^2 = n s^2 = 400 \). Equivalently, \( 480 = 400 + 20(12 - 10)^2 \), the minimising identity.
- (d) False. \( s = \sqrt{20} \approx 4.47 \), CV \( \approx 37.3\% \).
Quick Revision Table
| Idea | Fact |
|---|---|
| Mean, median as minimisers | \( \sum (x_i - a)^2 \) at \( \bar{x} \); \( \sum \lvert x_i - a \rvert \) at the median |
| Means inequality | \( \bar{x} \ge G \ge H \) for positive data |
| Coding \( u = (x - a)/h \) | \( \bar{x} = a + h\bar{u} \), \( s_x^2 = h^2 s_u^2 \), \( \mu_r(x) = h^r \mu_r(u) \) |
| Pooled variance | \( \frac{\sum n_i(s_i^2 + d_i^2)}{\sum n_i} \) |
| Central from raw | \( \mu_2 = \mu_2' - \mu_1'^2 \); \( \mu_3 = \mu_3' - 3\mu_2'\mu_1' + 2\mu_1'^3 \) |
| Shape | \( \beta_1 = \mu_3^2/\mu_2^3 \), \( \beta_2 = \mu_4/\mu_2^2 \), \( \beta_2 \ge \beta_1 + 1 \) |
| Correlation | \( \lvert r \rvert \le 1 \); invariant under origin and scale up to sign |
| Regression | \( b_{yx} = r s_y/s_x \), \( b_{xy} = r s_x/s_y \), product \( r^2 \), both through \( (\bar{x}, \bar{y}) \) |
| Residual variance | \( s_y^2(1 - r^2) \) |
| Tukey fences | \( Q_1 - 1.5\,\text{IQR} \), \( Q_3 + 1.5\,\text{IQR} \) (outer: 3 IQR) |
| Resistant | median, IQR, MAD, trimmed mean |
Key Takeaways
What’s Next
- Sample Space, Independence and Bayes’ Theorem: from frequencies in a data set to probabilities in a model.
- Random Variables, Distribution Functions and Moments: the population versions of the mean, variance, moments, \( \beta_1 \) and \( \beta_2 \) on this page.
- Joint, Marginal and Conditional Distributions; Characteristic Functions: covariance, correlation and regression for random vectors, and \( E(Y \mid X) \) as the best predictor.
- Probability Theory: Quick Revision.