Skip to content

Markov Chains and Stochastic Processes

CSIR NET Mathematical Sciences Unit 4: Markov chains, classification of states, stationary distributions, limits of P^n, Poisson and birth-death processes.

3
Topics
61
Min read

Chapter Overview

Markov chains are the part of Unit 4 (statistics) of the CSIR NET Mathematical Sciences syllabus where probability meets linear algebra. A chain is a stochastic matrix; its \( n \)-step behaviour is the matrix power \( P^n \); its long-run behaviour is a left eigenvector for the eigenvalue 1. The continuous-time half of the chapter, the Poisson process and birth-and-death processes, replaces the matrix by a generator and the powers by an exponential, and gives the standard models of queues and populations.

The notes assume an MSc first course in probability and a first look at stochastic processes (texts such as Ross’s Introduction to Probability Models, Karlin and Taylor’s A First Course in Stochastic Processes, Norris’s Markov Chains or Feller’s An Introduction to Probability Theory and Its Applications, Volume 1). They do not re-prove the limit theorems. They collect the exact statements, the shortcuts that turn a \( 3 \times 3 \) system into two lines, and the counter-examples (periodic, reducible and countable chains) that decide multiple-correct Part C questions.

What You’ll Learn

By the end of this chapter you will be able to:

  • Write the transition matrix of a verbal model and compute \( n \)-step probabilities with Chapman–Kolmogorov
  • Split any finite chain into communicating classes and decide, in seconds, which are closed, recurrent or transient, and their periods
  • Decide recurrence of countable chains (random walks on \( \mathbb{Z} \) and \( \mathbb{Z}^d \), reflecting walks) from \( \sum_n p_{ii}^{(n)} \)
  • Compute absorption probabilities and expected absorption times by first-step analysis or the fundamental matrix
  • Find stationary distributions with the two-state formula, double stochasticity, detailed balance or the degree formula, and convert them to mean recurrence times
  • State exactly when \( p_{ij}^{(n)} \) converges, to what, and what happens for periodic and reducible chains
  • Use the Poisson process (exponential gaps, conditioning on \( N(t) \), superposition, thinning) and solve birth-and-death models: Yule, linear birth–death, M/M/1, M/M/\( \infty \)

Prerequisites


Learning Path

graph TD
    A["1. Markov Chains and Classification of States"] --> B["2. n-Step Transition Limits and Stationary Distributions"]
    A --> C["3. Poisson and Birth-and-Death Processes"]
    B --> C
    B --> D["Quick Revision"]
    C --> D

Topics in This Chapter

#TopicWhat it coversTime
1Markov Chains and Classification of StatesMarkov property, transition matrices, Chapman–Kolmogorov, communicating and closed classes, recurrence and transience, random walks on \( \mathbb{Z}^d \), period, absorption, gambler’s ruin40 min
2n-Step Transition Limits and Stationary Distributionsstationary distributions and mean recurrence times, detailed balance, random walks on graphs, birth–death chains, the convergence theorem, periodic and reducible limits, \( P^n \) in closed form40 min
3Poisson and Birth-and-Death ProcessesPoisson process and its properties, continuous-time chains and Kolmogorov equations, birth-and-death processes, Yule, linear birth–death, M/M/1 and other queues40 min
Quick Revisionevery formula and counter-example on one page10 min

Official syllabus lines and where they are covered

Official topic lineCovered in
Markov chains (finite/countable), classification of statesMarkov Chains and Classification of States (finite chains, random walks and countable chains); reflecting walks and birth–death chains on the non-negative integers in n-Step Transition Limits and Stationary Distributions
n-step transition limits, stationary distributionn-Step Transition Limits and Stationary Distributions; Chapman–Kolmogorov and \( P^n \) are introduced in Markov Chains and Classification of States
Poisson and birth-and-death processesPoisson and Birth-and-Death Processes

How CSIR NET Asks It

Per the NTA information bulletin for December 2026, the Mathematical Sciences paper (code 704) is marked as follows:

Part BPart C
Questions given4060
Questions to attempt2520
Marks per correct answer34.75
Marks per wrong answer−0.750
Maximum7595

NTA publishes no topic-wise weightage, so we make no claim about how many questions come from this chapter. What the format tells you is how to prepare:

  • Part B (single correct) rewards fast computation: the number of classes of a given matrix, a period, an absorption probability, a stationary distribution, a mean recurrence time, \( \lim p_{ij}^{(n)} \), a Poisson or conditional binomial probability, an M/M/1 measure.
  • Part C (one or more correct) rewards structure plus counter-examples. A typical stem fixes a chain (a random walk, a reducible matrix, a birth–death model) and lists four statements about recurrence, periodicity, uniqueness of \( \pi \), convergence of \( P^n \) or extinction. Each is either forced by a theorem or broken by a small chain. Every topic page ends with Level 3 practice in exactly this format, with every option justified.

For the marking scheme and the other units, see the Mathematical Sciences overview. The matrix side of this chapter (eigenvalues, diagonalisation) is in Linear Algebra; the ODE side of Kolmogorov’s equations is in Linear ODEs and Variation of Parameters. Part A (General Aptitude) is common to all subjects. Found an error? See About for how to report it.


Start Learning

Begin with the first topic:

→ Markov Chains and Classification of States

Advertisement
Advertisement