Joint, Marginal and Conditional Distributions; Characteristic Functions
CSIR NET probability: joint, marginal and conditional laws, independence, conditional expectation, bivariate normal, Jacobians, characteristic functions.
- 21 min
On this page · 15 sections
- Prerequisites
- The Hook: Uncorrelated but Completely Dependent
- Joint Distributions
- Covariance and Correlation
- Conditional Distributions and Conditional Expectation
- The Bivariate Normal
- Transformations of Random Vectors
- Characteristic Functions
- Memory Tricks
- Don’t Confuse
- Common Mistakes
- Practice
- Quick Revision Table
- Key Takeaways
- What’s Next
Prerequisites
- Random Variables, Distribution Functions and Moments: CDFs, densities, expectation and MGFs for one variable.
- Sample Space, Independence and Bayes’ Theorem: conditional probability and independence of events, which this page lifts to random variables.
- Metric Spaces and Several Variables: Jacobians and change of variables in double integrals.
- Quadratic Forms: covariance matrices are positive semi-definite, and the bivariate normal density is the exponential of a quadratic form.
The Hook: Uncorrelated but Completely Dependent
Joint Distributions
For a random vector \( (X, Y) \) the joint CDF is \( F(x, y) = P(X \le x, Y \le y) \). It is non-decreasing and right-continuous in each variable, tends to 0 if either argument \( \to -\infty \) and to 1 if both \( \to +\infty \), and satisfies the rectangle inequality: for \( a_1 < b_1 \), \( a_2 < b_2 \),
$$F(b_1, b_2) - F(a_1, b_2) - F(b_1, a_2) + F(a_1, a_2) = P(a_1 < X \le b_1,\ a_2 < Y \le b_2) \ge 0.$$Monotonicity in each variable does not imply the rectangle inequality, so a candidate \( F \) must be checked against it.
Discrete: joint pmf \( p(x, y) \). Continuous: joint density \( f(x, y) \ge 0 \) with \( F(x, y) = \int_{-\infty}^{x}\int_{-\infty}^{y} f \), so \( f = \frac{\partial^2 F}{\partial x \partial y} \) where it is continuous.
Marginals. \( f_X(x) = \int f(x, y)\,dy \), \( p_X(x) = \sum_y p(x, y) \), \( F_X(x) = \lim_{y \to \infty} F(x, y) \). Marginals never determine the joint law (except under independence).
Independence of random variables
\( X \) and \( Y \) are independent iff \( F(x, y) = F_X(x)F_Y(y) \) for all \( x, y \); equivalently \( f(x, y) = f_X(x)f_Y(y) \) (densities), or \( p(x, y) = p_X(x)p_Y(y) \) (pmfs).
If \( X, Y \) are independent, so are \( g(X), h(Y) \) for any (Borel) functions \( g, h \), and \( E[g(X)h(Y)] = E g(X)\,E h(Y) \) whenever these are finite.
Covariance and Correlation
$$\text{Cov}(X, Y) = E[(X - \mu_X)(Y - \mu_Y)] = E(XY) - E(X)E(Y), \qquad \rho = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}.$$- \( \text{Var}(aX + bY) = a^2\sigma_X^2 + b^2\sigma_Y^2 + 2ab\,\text{Cov}(X, Y) \); for a vector, \( \text{Var}(a^T X) = a^T \Sigma a \ge 0 \), so the covariance matrix \( \Sigma \) is positive semi-definite.
- \( \lvert \rho \rvert \le 1 \), with equality iff \( Y = a + bX \) almost surely.
- Independent \( \Rightarrow \) uncorrelated (given finite variances). Not conversely: \( (X, X^2) \) with \( X \) symmetric; or \( (\cos\Theta, \sin\Theta) \) with \( \Theta \sim U(0, 2\pi) \).
- For jointly normal variables, uncorrelated \( \iff \) independent. Two variables that are each normal but not jointly normal can be uncorrelated and dependent: \( X \sim N(0, 1) \), \( Y = SX \) with \( S = \pm 1 \) a fair sign independent of \( X \). Then \( Y \sim N(0, 1) \), \( \text{Cov}(X, Y) = 0 \), but \( \lvert Y \rvert = \lvert X \rvert \), and \( X + Y \) has an atom at 0, so \( (X, Y) \) is not bivariate normal.
Conditional Distributions and Conditional Expectation
Discrete: \( p_{Y \mid X}(y \mid x) = \frac{p(x, y)}{p_X(x)} \) for \( p_X(x) > 0 \). Continuous: \( f_{Y \mid X}(y \mid x) = \frac{f(x, y)}{f_X(x)} \) for \( f_X(x) > 0 \).
For fixed \( x \), this is a pmf or density in \( y \). Then
$$E(Y \mid X = x) = \int y\,f_{Y \mid X}(y \mid x)\,dy,$$and \( E(Y \mid X) \) is the random variable \( m(X) \) where \( m(x) = E(Y \mid X = x) \).
The law of total variance is the population version of the pooled-variance formula on the descriptive statistics page. It also gives \( \text{Var}(E(Y \mid X)) \le \text{Var}(Y) \).
Random sums. If \( N \) is a non-negative integer random variable independent of i.i.d. \( X_1, X_2, \dots \), and \( S = X_1 + \dots + X_N \):
$$E S = E N \cdot E X, \qquad \text{Var}(S) = E N \cdot \text{Var}(X) + \text{Var}(N)(E X)^2.$$Poisson splitting and the binomial conditional
If \( X \sim \text{Poisson}(\lambda) \) and \( Y \sim \text{Poisson}(\mu) \) are independent, then \( X + Y \sim \text{Poisson}(\lambda + \mu) \) and
$$X \mid (X + Y = n) \sim \text{Binomial}\left(n, \frac{\lambda}{\lambda + \mu}\right).$$Conversely, if \( N \sim \text{Poisson}(\lambda) \) and each of the \( N \) items is independently of type 1 with probability \( p \), the type counts are independent \( \text{Poisson}(\lambda p) \) and \( \text{Poisson}(\lambda(1 - p)) \).
The Bivariate Normal
\( (X, Y) \) is bivariate normal with means \( \mu_1, \mu_2 \), variances \( \sigma_1^2, \sigma_2^2 \) and correlation \( \rho \in (-1, 1) \) if its density is
$$f(x, y) = \frac{1}{2\pi\sigma_1\sigma_2\sqrt{1 - \rho^2}}\exp\left\{-\frac{1}{2(1 - \rho^2)}\left[z_1^2 - 2\rho z_1 z_2 + z_2^2\right]\right\}, \quad z_i = \frac{x_i - \mu_i}{\sigma_i}.$$| Fact | Statement |
|---|---|
| Marginals | \( X \sim N(\mu_1, \sigma_1^2) \), \( Y \sim N(\mu_2, \sigma_2^2) \) |
| Conditional | \( Y \mid X = x \sim N\left(\mu_2 + \rho\frac{\sigma_2}{\sigma_1}(x - \mu_1),\ \sigma_2^2(1 - \rho^2)\right) \) |
| Regression | \( E(Y \mid X) \) is linear; conditional variance does not depend on \( x \) |
| Linear combinations | \( aX + bY \sim N(a\mu_1 + b\mu_2,\ a^2\sigma_1^2 + b^2\sigma_2^2 + 2ab\rho\sigma_1\sigma_2) \) |
| Independence | \( \rho = 0 \iff X, Y \) independent |
| Characterisation | \( (X, Y) \) is bivariate normal iff every \( aX + bY \) is normal |
A useful trick: \( X \) and \( Y - cX \) are jointly normal and independent when \( c = \frac{\text{Cov}(X, Y)}{\text{Var}(X)} \). That one line proves the conditional formula.
Transformations of Random Vectors
If \( (U, V) = T(X, Y) \) with \( T \) one-to-one and differentiable on the support, with inverse \( (x, y) = (x(u, v), y(u, v)) \),
$$\boxed{f_{U,V}(u, v) = f_{X,Y}(x(u, v), y(u, v))\,\left\lvert \frac{\partial(x, y)}{\partial(u, v)} \right\rvert}$$To find the law of one function \( U = g(X, Y) \), add a convenient \( V \) (often \( V = X \)), transform, and integrate \( V \) out.
Convolution. For independent \( X, Y \) with densities, \( f_{X+Y}(z) = \int f_X(x) f_Y(z - x)\,dx \).
| Independent inputs | Result |
|---|---|
| \( X, Y \sim U(0, 1) \) | \( X + Y \): triangular density on \( (0, 2) \), peak 1 at \( z = 1 \) |
| \( X \sim \text{Gamma}(\alpha, \lambda) \), \( Y \sim \text{Gamma}(\beta, \lambda) \) | \( X + Y \sim \text{Gamma}(\alpha + \beta, \lambda) \); \( \frac{X}{X + Y} \sim \text{Beta}(\alpha, \beta) \), independent of \( X + Y \) |
| \( X, Y \) i.i.d. exponential, rate \( \lambda \) | \( \min(X, Y) \sim \) exponential, rate \( 2\lambda \); \( X - Y \) Laplace |
| \( X, Y \) i.i.d. \( N(0, 1) \) | \( \frac{X}{Y} \) standard Cauchy; \( X^2 + Y^2 \sim \) exponential with mean 2; \( X + Y \), \( X - Y \) independent |
| \( X_i \) independent, CDFs \( F_i \) | \( \max \): \( \prod F_i(x) \); \( \min \): \( 1 - \prod (1 - F_i(x)) \) |
Characteristic Functions
The characteristic function (CF) of \( X \) is
$$\varphi_X(t) = E(e^{itX}) = E\cos tX + i\,E\sin tX, \qquad t \in \mathbb{R}.$$Because \( \lvert e^{itX} \rvert = 1 \), it always exists.
Inversion. If \( a < b \) are continuity points of \( F \),
$$F(b) - F(a) = \lim_{T \to \infty}\frac{1}{2\pi}\int_{-T}^{T}\frac{e^{-ita} - e^{-itb}}{it}\varphi(t)\,dt.$$If \( \int \lvert \varphi \rvert < \infty \), then \( X \) has a bounded continuous density
$$f(x) = \frac{1}{2\pi}\int_{-\infty}^{\infty} e^{-itx}\varphi(t)\,dt.$$(Integrals of this kind are often evaluated by residues; see Taylor and Laurent Series, Singularities and Residues.)
Atoms. \( P(X = a) = \lim_{T \to \infty}\frac{1}{2T}\int_{-T}^{T} e^{-ita}\varphi(t)\,dt \). If \( \lvert \varphi(t_0) \rvert = 1 \) for some \( t_0 \ne 0 \), \( X \) is a lattice variable: concentrated on \( \{b + \frac{2\pi k}{t_0} : k \in \mathbb{Z}\} \) for some \( b \). By the Riemann–Lebesgue lemma, if \( X \) has a density then \( \varphi(t) \to 0 \) as \( \lvert t \rvert \to \infty \). So a CF that does not tend to 0 (for instance the periodic CF of a lattice variable) belongs to a law with no density.
| Distribution | \( \varphi(t) \) |
|---|---|
| Degenerate at \( c \) | \( e^{ict} \) |
| \( \pm 1 \) with probability \( \frac{1}{2} \) each | \( \cos t \) |
| Bernoulli(\( p \)) / Binomial(\( n, p \)) | \( q + pe^{it} \) / \( (q + pe^{it})^n \) |
| Poisson(\( \lambda \)) | \( \exp\{\lambda(e^{it} - 1)\} \) |
| \( U(-a, a) \) | \( \frac{\sin at}{at} \) |
| Exponential, rate \( \lambda \) | \( \frac{\lambda}{\lambda - it} \) |
| Gamma(\( \alpha \), rate \( \lambda \)) | \( \left(1 - \frac{it}{\lambda}\right)^{-\alpha} \) |
| Laplace, density \( \frac{1}{2}e^{-\lvert x \rvert} \) | \( \frac{1}{1 + t^2} \) |
| \( N(\mu, \sigma^2) \) | \( \exp\{i\mu t - \frac{1}{2}\sigma^2 t^2\} \) |
| Standard Cauchy | \( e^{-\lvert t \rvert} \) |
Note the duality: Laplace has CF \( \frac{1}{1 + t^2} \) (the Cauchy shape), and Cauchy has CF \( e^{-\lvert t \rvert} \) (the Laplace shape).
Is this a characteristic function?
| Candidate | Verdict | Reason |
|---|---|---|
| \( \cos t \), \( \cos^2 t \) | yes | \( \cos t \): mass \( \frac{1}{2} \) at \( \pm 1 \); \( \cos^2 t = \frac{1 + \cos 2t}{2} \): mass \( \frac{1}{2} \) at 0 and \( \frac{1}{4} \) at \( \pm 2 \) (the sum of two independent \( \pm 1 \) variables) |
| \( e^{-\lvert t \rvert^\alpha} \), \( 0 < \alpha \le 2 \) | yes | symmetric stable laws |
| \( e^{-t^4} \) | no | \( \varphi''(0) = 0 \) would force \( E X^2 = 0 \), so \( X = 0 \) and \( \varphi \equiv 1 \) |
| \( \frac{1}{1 + t^4} \) | no | same argument: \( \varphi''(0) = 0 \) |
| \( \sin t \) | no | \( \varphi(0) = 0 \) |
| \( 1 + t^2 \), \( e^{t^2} \) | no | exceeds 1 |
| \( \max(1 - \lvert t \rvert, 0) \) | yes | Pólya’s criterion (real, even, convex on \( (0, \infty) \), decreasing to 0) |
| \( \lvert \varphi \rvert^2 \), \( \text{Re}\,\varphi \), \( \varphi^n \) for a CF \( \varphi \) | yes | \( X - X' \); mixture of \( X \) and \( -X \); sum of \( n \) i.i.d. copies |
| a function equal to 1 on \( [-1, 1] \) but not identically 1 | no | \( \varphi''(0) = 0 \) forces \( E X^2 = 0 \), so \( X = 0 \) and \( \varphi \equiv 1 \) |
Lévy’s continuity theorem. \( X_n \to X \) in distribution iff \( \varphi_{X_n}(t) \to \varphi_X(t) \) for every \( t \). More strongly, if \( \varphi_{X_n} \to g \) pointwise and \( g \) is continuous at 0, then \( g \) is a CF and \( X_n \) converges in distribution to it. This is the engine of the central limit theorem.
Memory Tricks
Don’t Confuse
| This | vs | That |
|---|---|---|
| uncorrelated | independent (equal only for jointly normal) | |
| each marginal normal | jointly (bivariate) normal | |
| \( E(Y \mid X = x) \), a number | \( E(Y \mid X) \), a random variable | |
| best predictor \( E(Y \mid X) \) | best linear predictor (regression line) | |
| \( \varphi^2 \): CF of \( X_1 + X_2 \) (i.i.d.) | \( \varphi(2t) \): CF of \( 2X \) | |
| MGF may not exist | CF always exists |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C of Mathematical Sciences, one or more options may be correct.
Level 1: Foundation
- Write the characteristic function of \( N(0, 1) \).
- Does \( \text{Cov}(X, Y) = 0 \) imply independence?
- \( f(x, y) = 2 \) on \( 0 < x < y < 1 \). Are \( X \) and \( Y \) independent?
Answers
- \( e^{-t^2/2} \).
- No in general (\( X \) and \( X^2 \) for symmetric \( X \)); yes for jointly normal variables.
- No. The support is a triangle, not a product set; for instance \( P(X > \frac{1}{2}, Y < \frac{1}{2}) = 0 \) while both marginal probabilities are positive.
Level 2: Part B Style
Practice Q1. \( (X, Y) \) has density \( f(x, y) = x + y \) on \( [0, 1]^2 \). Then \( \text{Cov}(X, Y) \) is:
(a) 0 (b) \( -\frac{1}{144} \) (c) \( \frac{1}{144} \) (d) \( -\frac{1}{12} \)
Answer: (b)
\( f_X(x) = x + \frac{1}{2} \), so \( E X = \frac{1}{3} + \frac{1}{4} = \frac{7}{12} \), and by symmetry \( E Y = \frac{7}{12} \). \( E(XY) = \int_0^1\int_0^1 xy(x + y)\,dx\,dy = \frac{1}{3}\cdot\frac{1}{2} + \frac{1}{2}\cdot\frac{1}{3} = \frac{1}{3} \). \( \text{Cov} = \frac{1}{3} - \frac{49}{144} = -\frac{1}{144} \).
Practice Q2. \( (X, Y) \) has density 2 on \( 0 < x < y < 1 \). Then \( E(Y \mid X = x) \) for \( 0 < x < 1 \) is:
(a) \( \frac{x}{2} \) (b) \( \frac{1 + x}{2} \) (c) \( \frac{2}{3} \) (d) \( 1 - x \)
Answer: (b)
Given \( X = x \), \( Y \) is uniform on \( (x, 1) \). (c) is the unconditional mean \( E Y \).
Practice Q3. Which of the following is a characteristic function?
(a) \( e^{-t^4} \) (b) \( \frac{1}{1 + t^2} \) (c) \( \sin t \) (d) \( 1 + t^2 \)
Answer: (b)
(b) is the CF of the Laplace density \( \frac{1}{2}e^{-\lvert x \rvert} \). (a) has \( \varphi''(0) = 0 \), forcing a degenerate law at 0 whose CF is 1, a contradiction. (c) has \( \varphi(0) = 0 \). (d) exceeds 1.
Practice Q4. \( X \sim \text{Poisson}(2) \) and \( Y \sim \text{Poisson}(3) \) are independent. Then \( P(X = 1 \mid X + Y = 4) \) is:
(a) \( \frac{216}{625} \) (b) \( \frac{2}{5} \) (c) \( \frac{96}{625} \) (d) \( 2e^{-2} \)
Answer: (a)
\( X \mid (X + Y = 4) \sim \text{Binomial}(4, \frac{2}{5}) \), so \( P = 4 \cdot \frac{2}{5}\cdot\left(\frac{3}{5}\right)^3 = \frac{216}{625} \approx 0.346 \). (c) is \( P(X = 3 \mid X + Y = 4) = 4(0.4)^3(0.6) = \frac{96}{625} \). (By coincidence \( P(X = 2 \mid X + Y = 4) = 6(0.4)^2(0.6)^2 \) also equals \( \frac{216}{625} \).) (d) is the unconditional \( P(X = 1) \).
Level 3: Part C Style
Practice Q5. \( (X, Y) \) is bivariate normal with \( E X = E Y = 0 \), \( \text{Var}\,X = 1 \), \( \text{Var}\,Y = 4 \) and correlation \( \rho = \frac{1}{2} \). Which statements are true?
(a) \( E(Y \mid X = 2) = 2 \). (b) \( \text{Var}(Y \mid X = x) = 3 \) for every \( x \). (c) \( X \) and \( Y - X \) are independent. (d) \( X + Y \sim N(0, 5) \).
Answer: (a), (b) and (c)
\( \text{Cov}(X, Y) = \rho\sigma_X\sigma_Y = \frac{1}{2}\cdot 1 \cdot 2 = 1 \).
- (a) True. \( \rho\frac{\sigma_Y}{\sigma_X}x = \frac{1}{2}\cdot 2 \cdot 2 = 2 \).
- (b) True. \( \sigma_Y^2(1 - \rho^2) = 4 \cdot \frac{3}{4} = 3 \).
- (c) True. \( \text{Cov}(X, Y - X) = 1 - 1 = 0 \), and \( (X, Y - X) \) is jointly normal, so uncorrelated means independent.
- (d) False. \( \text{Var}(X + Y) = 1 + 4 + 2 \cdot 1 = 7 \).
Practice Q6. Let \( \varphi \) be the characteristic function of a random variable \( X \). Which of the following are always true?
(a) \( \lvert \varphi(t) \rvert^2 \) is a characteristic function. (b) \( \text{Re}\,\varphi(t) \) is a characteristic function. (c) \( \varphi(t)^2 \) is the characteristic function of \( 2X \). (d) If \( \varphi \) is real-valued, then \( X \) and \( -X \) have the same distribution.
Answer: (a), (b) and (d)
- (a) True. It is the CF of \( X - X' \) with \( X' \) an independent copy: \( \varphi(t)\overline{\varphi(t)} \).
- (b) True. \( \text{Re}\,\varphi = \frac{\varphi(t) + \varphi(-t)}{2} \), the CF of the mixture that takes \( X \) or \( -X \) with probability \( \frac{1}{2} \) each.
- (c) False. The CF of \( 2X \) is \( \varphi(2t) \). For \( X \sim N(0, 1) \): \( \varphi(t)^2 = e^{-t^2} \) but \( \varphi(2t) = e^{-2t^2} \).
- (d) True. \( \varphi_{-X}(t) = \varphi(-t) = \overline{\varphi(t)} = \varphi(t) \); uniqueness gives equal laws.
Practice Q7. \( X \) and \( Y \) are independent exponential random variables with mean 1. Let \( U = X + Y \) and \( V = \frac{X}{X + Y} \). Which statements are true?
(a) \( U \) has density \( u e^{-u} \) on \( (0, \infty) \). (b) \( V \) is uniform on \( (0, 1) \). (c) \( U \) and \( V \) are independent. (d) \( \min(X, Y) \) has mean 1.
Answer: (a), (b) and (c)
Inverse map \( x = uv \), \( y = u(1 - v) \), Jacobian \( \left\lvert \frac{\partial(x, y)}{\partial(u, v)} \right\rvert = u \). So \( f_{U,V}(u, v) = e^{-u}\cdot u \) on \( (0, \infty) \times (0, 1) \): it factors on a product set.
- (a) True. \( U \sim \text{Gamma}(2, 1) \).
- (b) True. The \( v \)-factor is constant on \( (0, 1) \).
- (c) True. Factorisation on a rectangle.
- (d) False. \( P(\min > t) = e^{-2t} \): exponential with rate 2, mean \( \frac{1}{2} \).
Quick Revision Table
| Idea | Fact |
|---|---|
| Joint CDF | rectangle inequality is the extra condition |
| Independence | \( f = f_X f_Y \); factor on a product support |
| Covariance | \( E XY - E X\,E Y \); \( \Sigma \) positive semi-definite |
| Uncorrelated vs independent | equal only for jointly normal |
| Conditional density | \( f(x, y)/f_X(x) \) |
| Tower, EVE | \( E\,E(Y \mid X) = E Y \); \( \text{Var}\,Y = E\,\text{Var}(Y \mid X) + \text{Var}\,E(Y \mid X) \) |
| Best predictor | \( E(Y \mid X) \) in mean square |
| Bivariate normal | \( Y \mid X = x \sim N(\mu_2 + \rho\frac{\sigma_2}{\sigma_1}(x - \mu_1), \sigma_2^2(1 - \rho^2)) \) |
| Poisson conditional | \( X \mid X + Y = n \sim \text{Bin}(n, \frac{\lambda}{\lambda + \mu}) \) |
| Jacobian | \( f_{U,V} = f_{X,Y}\lvert \partial(x, y)/\partial(u, v) \rvert \) |
| CF basics | \( \varphi(0) = 1 \), \( \lvert \varphi \rvert \le 1 \), uniformly continuous, real iff symmetric |
| CF moments | \( \varphi^{(k)}(0) = i^k E X^k \) |
| Not a CF | \( e^{-t^4} \), \( \sin t \), anything \( > 1 \) in modulus |
| Lévy continuity | convergence of CFs \( \iff \) convergence in distribution |
Key Takeaways
What’s Next
- Inequalities, Convergence and Limit Theorems: Chebyshev, Markov, Jensen, the four modes of convergence, the laws of large numbers and the CLT, proved with characteristic functions.
- Random Variables, Distribution Functions and Moments for the one-variable MGF table this page extends.
- Quadratic Forms for definiteness of covariance matrices.
- Probability Theory: Quick Revision.