Transition Matrices

A discrete-time Markov chain has the memoryless property: P(X_{n+1}=j | X_n=i, X_{n-1},...) = P(X_{n+1}=j | X_n=i) = p_{ij}. This property means that the future evolution of the chain depends only on the current state, not on how the chain arrived there. The transition matrix P has entries p_{ij} representing the probability of moving from state i to state j in one step. Each row sums to 1 because the chain must go somewhere.

The n-step transition probability p_{ij}^{(n)} gives the probability of moving from state i to state j in exactly n steps. These are computed by matrix multiplication: the n-step transition matrix is P^n. For small state spaces, you can compute P^2, P^3, and so on directly. For larger state spaces or theoretical analysis, eigenvalue decomposition of P simplifies the computation of P^n significantly.

State Classification

States in a Markov chain fall into two fundamental categories. A state i is recurrent if the chain is certain to return to i eventually: P(return to i | start at i) = 1. A state i is transient if there is a positive probability of never returning: P(return to i | start at i) < 1. Recurrent states are further classified as positive recurrent (finite expected return time) or null recurrent (infinite expected return time). In finite-state chains, all recurrent states are positive recurrent.

A state is absorbing if p_{ii} = 1, meaning the chain stays there forever once entered. The period of a state i is the greatest common divisor of all n such that p_{ii}^{(n)} > 0. A state with period 1 is called aperiodic. A chain is irreducible if every state can be reached from every other state. These classifications determine the long-run behavior of the chain.

Computing Transition Probabilities

Consider a three-state Markov chain with transition matrix P = [[0.7, 0.2, 0.1], [0.3, 0.5, 0.2], [0.1, 0.3, 0.6]]. To find the probability of being in state 2 after two steps starting from state 1, compute the (1,2) entry of P^2. The calculation is: p_{12}^{(2)} = p_{11}*p_{12} + p_{12}*p_{22} + p_{13}*p_{32} = 0.7*0.2 + 0.2*0.5 + 0.1*0.3 = 0.14 + 0.10 + 0.03 = 0.27.

For higher powers, matrix multiplication becomes tedious by hand but follows the same pattern. On Exam MAS-I, you will typically deal with two or three state chains where P^2 or P^3 can be computed manually. For larger problems, the eigenvalue decomposition P = Q * D * Q^{-1} gives P^n = Q * D^n * Q^{-1}, where D is a diagonal matrix of eigenvalues.

Stationary Distributions

A stationary distribution pi satisfies pi * P = pi and the sum of all pi_i = 1. Intuitively, if the chain starts in the stationary distribution, it remains in the stationary distribution at every future time step. For an irreducible, positive recurrent chain, a unique stationary distribution exists, and pi_j = 1/E[T_j] where T_j is the expected return time to state j. States visited more frequently have larger stationary probabilities.

To compute the stationary distribution, solve the system of equations pi * P = pi along with the normalization constraint sum(pi_i) = 1. For the three-state example above, the equations are: pi_1 = 0.7*pi_1 + 0.3*pi_2 + 0.1*pi_3, pi_2 = 0.2*pi_1 + 0.5*pi_2 + 0.3*pi_3, pi_3 = 0.1*pi_1 + 0.2*pi_2 + 0.6*pi_3, and pi_1 + pi_2 + pi_3 = 1. Rearranging the first equation gives 0.3*pi_1 = 0.3*pi_2 + 0.1*pi_3, and solving the full system yields the stationary distribution.

Convergence and Limiting Behavior

If a Markov chain is irreducible and aperiodic (called ergodic), then regardless of the starting state, the distribution of X_n converges to the unique stationary distribution as n approaches infinity. This means lim_{n to infinity} p_{ij}^{(n)} = pi_j for all states i. In practical terms, the rows of P^n all converge to the same row vector pi as n grows large.

Periodicity prevents convergence of individual transition probabilities. A chain with period d has p_{ii}^{(n)} = 0 unless n is a multiple of d. However, the Cesaro average (1/n) * sum_{k=1}^{n} p_{ij}^{(k)} still converges to pi_j even for periodic chains. On Exam MAS-I, most problems involve aperiodic chains, so direct convergence applies.

Actuarial Applications

Markov chains are widely used in insurance modeling. Bonus-malus systems (no-claims discount schemes in auto insurance) are modeled as Markov chains where the state represents the discount level. A policyholder moves to a better discount level after a claim-free year and to a worse level after filing a claim. The stationary distribution gives the long-run proportion of policyholders at each discount level, which is essential for pricing the system.

Multi-state models for health and disability insurance use Markov chains with states like "healthy," "disabled," and "dead." Transition probabilities represent the rates of becoming disabled, recovering, or dying. The probability of being in any state after n years, starting from any initial state, is computed using P^n. These models underpin the pricing and reserving for disability income, long-term care, and critical illness insurance products.

Common Pitfalls

A frequent mistake is confusing the row and column conventions for transition matrices. In the convention used on Exam MAS-I, rows represent the current state and columns represent the next state, so row sums equal 1. The stationary distribution equation is pi * P = pi (row vector times matrix). If you accidentally transpose the convention, your answers will be wrong.

Another common error is computing stationary distributions for reducible chains. If the chain has multiple communicating classes, the stationary distribution is not unique. You must identify the recurrent classes and find stationary distributions within each class separately.

Study Tips

Practice computing P^2 quickly for small transition matrices. Memorize the setup for finding stationary distributions: write out pi * P = pi, rearrange into a homogeneous system, drop one redundant equation, and replace it with the normalization constraint. Work through bonus-malus problems that ask for the long-run average premium, which requires the stationary distribution. Be comfortable classifying states as transient or recurrent and determining whether a chain is irreducible and aperiodic. These fundamental skills appear repeatedly on Exam MAS-I.

Markov Chains: Transition Matrices and Stationary Distributions