Skip to content

Joint, Marginal and Conditional Distributions; Characteristic Functions

CSIR NET probability: joint, marginal and conditional laws, independence, conditional expectation, bivariate normal, Jacobians, characteristic functions.

  • 21 min read
  • Updated
On this page · 15 sections
  1. Prerequisites
  2. The Hook: Uncorrelated but Completely Dependent
  3. Joint Distributions
  4. Covariance and Correlation
  5. Conditional Distributions and Conditional Expectation
  6. The Bivariate Normal
  7. Transformations of Random Vectors
  8. Characteristic Functions
  9. Memory Tricks
  10. Don’t Confuse
  11. Common Mistakes
  12. Practice
  13. Quick Revision Table
  14. Key Takeaways
  15. What’s Next

Prerequisites

The Hook: Uncorrelated but Completely Dependent

Joint Distributions

For a random vector \( (X, Y) \) the joint CDF is \( F(x, y) = P(X \le x, Y \le y) \). It is non-decreasing and right-continuous in each variable, tends to 0 if either argument \( \to -\infty \) and to 1 if both \( \to +\infty \), and satisfies the rectangle inequality: for \( a_1 < b_1 \), \( a_2 < b_2 \),

$$F(b_1, b_2) - F(a_1, b_2) - F(b_1, a_2) + F(a_1, a_2) = P(a_1 < X \le b_1,\ a_2 < Y \le b_2) \ge 0.$$

Monotonicity in each variable does not imply the rectangle inequality, so a candidate \( F \) must be checked against it.

Discrete: joint pmf \( p(x, y) \). Continuous: joint density \( f(x, y) \ge 0 \) with \( F(x, y) = \int_{-\infty}^{x}\int_{-\infty}^{y} f \), so \( f = \frac{\partial^2 F}{\partial x \partial y} \) where it is continuous.

Marginals. \( f_X(x) = \int f(x, y)\,dy \), \( p_X(x) = \sum_y p(x, y) \), \( F_X(x) = \lim_{y \to \infty} F(x, y) \). Marginals never determine the joint law (except under independence).

Independence of random variables

\( X \) and \( Y \) are independent iff \( F(x, y) = F_X(x)F_Y(y) \) for all \( x, y \); equivalently \( f(x, y) = f_X(x)f_Y(y) \) (densities), or \( p(x, y) = p_X(x)p_Y(y) \) (pmfs).

If \( X, Y \) are independent, so are \( g(X), h(Y) \) for any (Borel) functions \( g, h \), and \( E[g(X)h(Y)] = E g(X)\,E h(Y) \) whenever these are finite.

Covariance and Correlation

$$\text{Cov}(X, Y) = E[(X - \mu_X)(Y - \mu_Y)] = E(XY) - E(X)E(Y), \qquad \rho = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}.$$
  • \( \text{Var}(aX + bY) = a^2\sigma_X^2 + b^2\sigma_Y^2 + 2ab\,\text{Cov}(X, Y) \); for a vector, \( \text{Var}(a^T X) = a^T \Sigma a \ge 0 \), so the covariance matrix \( \Sigma \) is positive semi-definite.
  • \( \lvert \rho \rvert \le 1 \), with equality iff \( Y = a + bX \) almost surely.
  • Independent \( \Rightarrow \) uncorrelated (given finite variances). Not conversely: \( (X, X^2) \) with \( X \) symmetric; or \( (\cos\Theta, \sin\Theta) \) with \( \Theta \sim U(0, 2\pi) \).
  • For jointly normal variables, uncorrelated \( \iff \) independent. Two variables that are each normal but not jointly normal can be uncorrelated and dependent: \( X \sim N(0, 1) \), \( Y = SX \) with \( S = \pm 1 \) a fair sign independent of \( X \). Then \( Y \sim N(0, 1) \), \( \text{Cov}(X, Y) = 0 \), but \( \lvert Y \rvert = \lvert X \rvert \), and \( X + Y \) has an atom at 0, so \( (X, Y) \) is not bivariate normal.

Conditional Distributions and Conditional Expectation

Discrete: \( p_{Y \mid X}(y \mid x) = \frac{p(x, y)}{p_X(x)} \) for \( p_X(x) > 0 \). Continuous: \( f_{Y \mid X}(y \mid x) = \frac{f(x, y)}{f_X(x)} \) for \( f_X(x) > 0 \).

For fixed \( x \), this is a pmf or density in \( y \). Then

$$E(Y \mid X = x) = \int y\,f_{Y \mid X}(y \mid x)\,dy,$$

and \( E(Y \mid X) \) is the random variable \( m(X) \) where \( m(x) = E(Y \mid X = x) \).

The law of total variance is the population version of the pooled-variance formula on the descriptive statistics page. It also gives \( \text{Var}(E(Y \mid X)) \le \text{Var}(Y) \).

Random sums. If \( N \) is a non-negative integer random variable independent of i.i.d. \( X_1, X_2, \dots \), and \( S = X_1 + \dots + X_N \):

$$E S = E N \cdot E X, \qquad \text{Var}(S) = E N \cdot \text{Var}(X) + \text{Var}(N)(E X)^2.$$

Poisson splitting and the binomial conditional

If \( X \sim \text{Poisson}(\lambda) \) and \( Y \sim \text{Poisson}(\mu) \) are independent, then \( X + Y \sim \text{Poisson}(\lambda + \mu) \) and

$$X \mid (X + Y = n) \sim \text{Binomial}\left(n, \frac{\lambda}{\lambda + \mu}\right).$$

Conversely, if \( N \sim \text{Poisson}(\lambda) \) and each of the \( N \) items is independently of type 1 with probability \( p \), the type counts are independent \( \text{Poisson}(\lambda p) \) and \( \text{Poisson}(\lambda(1 - p)) \).

The Bivariate Normal

\( (X, Y) \) is bivariate normal with means \( \mu_1, \mu_2 \), variances \( \sigma_1^2, \sigma_2^2 \) and correlation \( \rho \in (-1, 1) \) if its density is

$$f(x, y) = \frac{1}{2\pi\sigma_1\sigma_2\sqrt{1 - \rho^2}}\exp\left\{-\frac{1}{2(1 - \rho^2)}\left[z_1^2 - 2\rho z_1 z_2 + z_2^2\right]\right\}, \quad z_i = \frac{x_i - \mu_i}{\sigma_i}.$$
FactStatement
Marginals\( X \sim N(\mu_1, \sigma_1^2) \), \( Y \sim N(\mu_2, \sigma_2^2) \)
Conditional\( Y \mid X = x \sim N\left(\mu_2 + \rho\frac{\sigma_2}{\sigma_1}(x - \mu_1),\ \sigma_2^2(1 - \rho^2)\right) \)
Regression\( E(Y \mid X) \) is linear; conditional variance does not depend on \( x \)
Linear combinations\( aX + bY \sim N(a\mu_1 + b\mu_2,\ a^2\sigma_1^2 + b^2\sigma_2^2 + 2ab\rho\sigma_1\sigma_2) \)
Independence\( \rho = 0 \iff X, Y \) independent
Characterisation\( (X, Y) \) is bivariate normal iff every \( aX + bY \) is normal

A useful trick: \( X \) and \( Y - cX \) are jointly normal and independent when \( c = \frac{\text{Cov}(X, Y)}{\text{Var}(X)} \). That one line proves the conditional formula.

Transformations of Random Vectors

If \( (U, V) = T(X, Y) \) with \( T \) one-to-one and differentiable on the support, with inverse \( (x, y) = (x(u, v), y(u, v)) \),

$$\boxed{f_{U,V}(u, v) = f_{X,Y}(x(u, v), y(u, v))\,\left\lvert \frac{\partial(x, y)}{\partial(u, v)} \right\rvert}$$

To find the law of one function \( U = g(X, Y) \), add a convenient \( V \) (often \( V = X \)), transform, and integrate \( V \) out.

Convolution. For independent \( X, Y \) with densities, \( f_{X+Y}(z) = \int f_X(x) f_Y(z - x)\,dx \).

Independent inputsResult
\( X, Y \sim U(0, 1) \)\( X + Y \): triangular density on \( (0, 2) \), peak 1 at \( z = 1 \)
\( X \sim \text{Gamma}(\alpha, \lambda) \), \( Y \sim \text{Gamma}(\beta, \lambda) \)\( X + Y \sim \text{Gamma}(\alpha + \beta, \lambda) \); \( \frac{X}{X + Y} \sim \text{Beta}(\alpha, \beta) \), independent of \( X + Y \)
\( X, Y \) i.i.d. exponential, rate \( \lambda \)\( \min(X, Y) \sim \) exponential, rate \( 2\lambda \); \( X - Y \) Laplace
\( X, Y \) i.i.d. \( N(0, 1) \)\( \frac{X}{Y} \) standard Cauchy; \( X^2 + Y^2 \sim \) exponential with mean 2; \( X + Y \), \( X - Y \) independent
\( X_i \) independent, CDFs \( F_i \)\( \max \): \( \prod F_i(x) \); \( \min \): \( 1 - \prod (1 - F_i(x)) \)

Characteristic Functions

The characteristic function (CF) of \( X \) is

$$\varphi_X(t) = E(e^{itX}) = E\cos tX + i\,E\sin tX, \qquad t \in \mathbb{R}.$$

Because \( \lvert e^{itX} \rvert = 1 \), it always exists.

Inversion. If \( a < b \) are continuity points of \( F \),

$$F(b) - F(a) = \lim_{T \to \infty}\frac{1}{2\pi}\int_{-T}^{T}\frac{e^{-ita} - e^{-itb}}{it}\varphi(t)\,dt.$$

If \( \int \lvert \varphi \rvert < \infty \), then \( X \) has a bounded continuous density

$$f(x) = \frac{1}{2\pi}\int_{-\infty}^{\infty} e^{-itx}\varphi(t)\,dt.$$

(Integrals of this kind are often evaluated by residues; see Taylor and Laurent Series, Singularities and Residues.)

Atoms. \( P(X = a) = \lim_{T \to \infty}\frac{1}{2T}\int_{-T}^{T} e^{-ita}\varphi(t)\,dt \). If \( \lvert \varphi(t_0) \rvert = 1 \) for some \( t_0 \ne 0 \), \( X \) is a lattice variable: concentrated on \( \{b + \frac{2\pi k}{t_0} : k \in \mathbb{Z}\} \) for some \( b \). By the Riemann–Lebesgue lemma, if \( X \) has a density then \( \varphi(t) \to 0 \) as \( \lvert t \rvert \to \infty \). So a CF that does not tend to 0 (for instance the periodic CF of a lattice variable) belongs to a law with no density.

Distribution\( \varphi(t) \)
Degenerate at \( c \)\( e^{ict} \)
\( \pm 1 \) with probability \( \frac{1}{2} \) each\( \cos t \)
Bernoulli(\( p \)) / Binomial(\( n, p \))\( q + pe^{it} \) / \( (q + pe^{it})^n \)
Poisson(\( \lambda \))\( \exp\{\lambda(e^{it} - 1)\} \)
\( U(-a, a) \)\( \frac{\sin at}{at} \)
Exponential, rate \( \lambda \)\( \frac{\lambda}{\lambda - it} \)
Gamma(\( \alpha \), rate \( \lambda \))\( \left(1 - \frac{it}{\lambda}\right)^{-\alpha} \)
Laplace, density \( \frac{1}{2}e^{-\lvert x \rvert} \)\( \frac{1}{1 + t^2} \)
\( N(\mu, \sigma^2) \)\( \exp\{i\mu t - \frac{1}{2}\sigma^2 t^2\} \)
Standard Cauchy\( e^{-\lvert t \rvert} \)

Note the duality: Laplace has CF \( \frac{1}{1 + t^2} \) (the Cauchy shape), and Cauchy has CF \( e^{-\lvert t \rvert} \) (the Laplace shape).

Is this a characteristic function?

CandidateVerdictReason
\( \cos t \), \( \cos^2 t \)yes\( \cos t \): mass \( \frac{1}{2} \) at \( \pm 1 \); \( \cos^2 t = \frac{1 + \cos 2t}{2} \): mass \( \frac{1}{2} \) at 0 and \( \frac{1}{4} \) at \( \pm 2 \) (the sum of two independent \( \pm 1 \) variables)
\( e^{-\lvert t \rvert^\alpha} \), \( 0 < \alpha \le 2 \)yessymmetric stable laws
\( e^{-t^4} \)no\( \varphi''(0) = 0 \) would force \( E X^2 = 0 \), so \( X = 0 \) and \( \varphi \equiv 1 \)
\( \frac{1}{1 + t^4} \)nosame argument: \( \varphi''(0) = 0 \)
\( \sin t \)no\( \varphi(0) = 0 \)
\( 1 + t^2 \), \( e^{t^2} \)noexceeds 1
\( \max(1 - \lvert t \rvert, 0) \)yesPólya’s criterion (real, even, convex on \( (0, \infty) \), decreasing to 0)
\( \lvert \varphi \rvert^2 \), \( \text{Re}\,\varphi \), \( \varphi^n \) for a CF \( \varphi \)yes\( X - X' \); mixture of \( X \) and \( -X \); sum of \( n \) i.i.d. copies
a function equal to 1 on \( [-1, 1] \) but not identically 1no\( \varphi''(0) = 0 \) forces \( E X^2 = 0 \), so \( X = 0 \) and \( \varphi \equiv 1 \)

Lévy’s continuity theorem. \( X_n \to X \) in distribution iff \( \varphi_{X_n}(t) \to \varphi_X(t) \) for every \( t \). More strongly, if \( \varphi_{X_n} \to g \) pointwise and \( g \) is continuous at 0, then \( g \) is a CF and \( X_n \) converges in distribution to it. This is the engine of the central limit theorem.

Memory Tricks

Don’t Confuse

ThisvsThat
uncorrelatedindependent (equal only for jointly normal)
each marginal normaljointly (bivariate) normal
\( E(Y \mid X = x) \), a number\( E(Y \mid X) \), a random variable
best predictor \( E(Y \mid X) \)best linear predictor (regression line)
\( \varphi^2 \): CF of \( X_1 + X_2 \) (i.i.d.)\( \varphi(2t) \): CF of \( 2X \)
MGF may not existCF always exists

Common Mistakes

Practice

All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C of Mathematical Sciences, one or more options may be correct.

Level 1: Foundation

  1. Write the characteristic function of \( N(0, 1) \).
  2. Does \( \text{Cov}(X, Y) = 0 \) imply independence?
  3. \( f(x, y) = 2 \) on \( 0 < x < y < 1 \). Are \( X \) and \( Y \) independent?
Answers
  1. \( e^{-t^2/2} \).
  2. No in general (\( X \) and \( X^2 \) for symmetric \( X \)); yes for jointly normal variables.
  3. No. The support is a triangle, not a product set; for instance \( P(X > \frac{1}{2}, Y < \frac{1}{2}) = 0 \) while both marginal probabilities are positive.

Level 2: Part B Style

Practice Q1. \( (X, Y) \) has density \( f(x, y) = x + y \) on \( [0, 1]^2 \). Then \( \text{Cov}(X, Y) \) is:

(a) 0 (b) \( -\frac{1}{144} \) (c) \( \frac{1}{144} \) (d) \( -\frac{1}{12} \)

Answer: (b)

\( f_X(x) = x + \frac{1}{2} \), so \( E X = \frac{1}{3} + \frac{1}{4} = \frac{7}{12} \), and by symmetry \( E Y = \frac{7}{12} \). \( E(XY) = \int_0^1\int_0^1 xy(x + y)\,dx\,dy = \frac{1}{3}\cdot\frac{1}{2} + \frac{1}{2}\cdot\frac{1}{3} = \frac{1}{3} \). \( \text{Cov} = \frac{1}{3} - \frac{49}{144} = -\frac{1}{144} \).

Practice Q2. \( (X, Y) \) has density 2 on \( 0 < x < y < 1 \). Then \( E(Y \mid X = x) \) for \( 0 < x < 1 \) is:

(a) \( \frac{x}{2} \) (b) \( \frac{1 + x}{2} \) (c) \( \frac{2}{3} \) (d) \( 1 - x \)

Answer: (b)

Given \( X = x \), \( Y \) is uniform on \( (x, 1) \). (c) is the unconditional mean \( E Y \).

Practice Q3. Which of the following is a characteristic function?

(a) \( e^{-t^4} \) (b) \( \frac{1}{1 + t^2} \) (c) \( \sin t \) (d) \( 1 + t^2 \)

Answer: (b)

(b) is the CF of the Laplace density \( \frac{1}{2}e^{-\lvert x \rvert} \). (a) has \( \varphi''(0) = 0 \), forcing a degenerate law at 0 whose CF is 1, a contradiction. (c) has \( \varphi(0) = 0 \). (d) exceeds 1.

Practice Q4. \( X \sim \text{Poisson}(2) \) and \( Y \sim \text{Poisson}(3) \) are independent. Then \( P(X = 1 \mid X + Y = 4) \) is:

(a) \( \frac{216}{625} \) (b) \( \frac{2}{5} \) (c) \( \frac{96}{625} \) (d) \( 2e^{-2} \)

Answer: (a)

\( X \mid (X + Y = 4) \sim \text{Binomial}(4, \frac{2}{5}) \), so \( P = 4 \cdot \frac{2}{5}\cdot\left(\frac{3}{5}\right)^3 = \frac{216}{625} \approx 0.346 \). (c) is \( P(X = 3 \mid X + Y = 4) = 4(0.4)^3(0.6) = \frac{96}{625} \). (By coincidence \( P(X = 2 \mid X + Y = 4) = 6(0.4)^2(0.6)^2 \) also equals \( \frac{216}{625} \).) (d) is the unconditional \( P(X = 1) \).

Level 3: Part C Style

Practice Q5. \( (X, Y) \) is bivariate normal with \( E X = E Y = 0 \), \( \text{Var}\,X = 1 \), \( \text{Var}\,Y = 4 \) and correlation \( \rho = \frac{1}{2} \). Which statements are true?

(a) \( E(Y \mid X = 2) = 2 \). (b) \( \text{Var}(Y \mid X = x) = 3 \) for every \( x \). (c) \( X \) and \( Y - X \) are independent. (d) \( X + Y \sim N(0, 5) \).

Answer: (a), (b) and (c)

\( \text{Cov}(X, Y) = \rho\sigma_X\sigma_Y = \frac{1}{2}\cdot 1 \cdot 2 = 1 \).

  • (a) True. \( \rho\frac{\sigma_Y}{\sigma_X}x = \frac{1}{2}\cdot 2 \cdot 2 = 2 \).
  • (b) True. \( \sigma_Y^2(1 - \rho^2) = 4 \cdot \frac{3}{4} = 3 \).
  • (c) True. \( \text{Cov}(X, Y - X) = 1 - 1 = 0 \), and \( (X, Y - X) \) is jointly normal, so uncorrelated means independent.
  • (d) False. \( \text{Var}(X + Y) = 1 + 4 + 2 \cdot 1 = 7 \).

Practice Q6. Let \( \varphi \) be the characteristic function of a random variable \( X \). Which of the following are always true?

(a) \( \lvert \varphi(t) \rvert^2 \) is a characteristic function. (b) \( \text{Re}\,\varphi(t) \) is a characteristic function. (c) \( \varphi(t)^2 \) is the characteristic function of \( 2X \). (d) If \( \varphi \) is real-valued, then \( X \) and \( -X \) have the same distribution.

Answer: (a), (b) and (d)
  • (a) True. It is the CF of \( X - X' \) with \( X' \) an independent copy: \( \varphi(t)\overline{\varphi(t)} \).
  • (b) True. \( \text{Re}\,\varphi = \frac{\varphi(t) + \varphi(-t)}{2} \), the CF of the mixture that takes \( X \) or \( -X \) with probability \( \frac{1}{2} \) each.
  • (c) False. The CF of \( 2X \) is \( \varphi(2t) \). For \( X \sim N(0, 1) \): \( \varphi(t)^2 = e^{-t^2} \) but \( \varphi(2t) = e^{-2t^2} \).
  • (d) True. \( \varphi_{-X}(t) = \varphi(-t) = \overline{\varphi(t)} = \varphi(t) \); uniqueness gives equal laws.

Practice Q7. \( X \) and \( Y \) are independent exponential random variables with mean 1. Let \( U = X + Y \) and \( V = \frac{X}{X + Y} \). Which statements are true?

(a) \( U \) has density \( u e^{-u} \) on \( (0, \infty) \). (b) \( V \) is uniform on \( (0, 1) \). (c) \( U \) and \( V \) are independent. (d) \( \min(X, Y) \) has mean 1.

Answer: (a), (b) and (c)

Inverse map \( x = uv \), \( y = u(1 - v) \), Jacobian \( \left\lvert \frac{\partial(x, y)}{\partial(u, v)} \right\rvert = u \). So \( f_{U,V}(u, v) = e^{-u}\cdot u \) on \( (0, \infty) \times (0, 1) \): it factors on a product set.

  • (a) True. \( U \sim \text{Gamma}(2, 1) \).
  • (b) True. The \( v \)-factor is constant on \( (0, 1) \).
  • (c) True. Factorisation on a rectangle.
  • (d) False. \( P(\min > t) = e^{-2t} \): exponential with rate 2, mean \( \frac{1}{2} \).

Quick Revision Table

IdeaFact
Joint CDFrectangle inequality is the extra condition
Independence\( f = f_X f_Y \); factor on a product support
Covariance\( E XY - E X\,E Y \); \( \Sigma \) positive semi-definite
Uncorrelated vs independentequal only for jointly normal
Conditional density\( f(x, y)/f_X(x) \)
Tower, EVE\( E\,E(Y \mid X) = E Y \); \( \text{Var}\,Y = E\,\text{Var}(Y \mid X) + \text{Var}\,E(Y \mid X) \)
Best predictor\( E(Y \mid X) \) in mean square
Bivariate normal\( Y \mid X = x \sim N(\mu_2 + \rho\frac{\sigma_2}{\sigma_1}(x - \mu_1), \sigma_2^2(1 - \rho^2)) \)
Poisson conditional\( X \mid X + Y = n \sim \text{Bin}(n, \frac{\lambda}{\lambda + \mu}) \)
Jacobian\( f_{U,V} = f_{X,Y}\lvert \partial(x, y)/\partial(u, v) \rvert \)
CF basics\( \varphi(0) = 1 \), \( \lvert \varphi \rvert \le 1 \), uniformly continuous, real iff symmetric
CF moments\( \varphi^{(k)}(0) = i^k E X^k \)
Not a CF\( e^{-t^4} \), \( \sin t \), anything \( > 1 \) in modulus
Lévy continuityconvergence of CFs \( \iff \) convergence in distribution

Key Takeaways

What’s Next

Advertisement
Advertisement