Time Series Analysis: AR, MA, and ARMA Models for Exam SRM

Stationarity and White Noise

Before studying time series models, you must understand stationarity. A weakly stationary process has three properties: (1) the mean E[Y_t] is constant over time, (2) the variance Var(Y_t) is constant over time, and (3) the autocovariance Cov(Y_t, Y_{t+k}) depends only on the lag k, not on the time t. Stationarity is required for AR and ARMA models to have stable, well-defined properties.

White noise is the simplest stationary process: epsilon_t ~ WN(0, sigma^2) has zero mean, constant variance sigma^2, and zero autocovariance at all nonzero lags. White noise serves as the building block for all time series models. It represents the unpredictable, random component of the series.

The autocorrelation function (ACF) measures the correlation between Y_t and Y_{t+k}: rho(k) = Cov(Y_t, Y_{t+k}) / Var(Y_t). For white noise, rho(0) = 1 and rho(k) = 0 for all k != 0. The ACF and partial autocorrelation function (PACF) are the primary tools for identifying the type and order of a time series model.

Autoregressive (AR) Models

An AR(p) model expresses the current value as a linear combination of the p most recent past values plus white noise: Y_t = c + phi_1*Y_{t-1} + phi_2*Y_{t-2} + ... + phi_p*Y_{t-p} + epsilon_t, where epsilon_t ~ WN(0, sigma^2). The constant c is often set to zero (mean-centered series).

For the AR(1) model Y_t = phi*Y_{t-1} + epsilon_t: the process is stationary when |phi| < 1. When stationary, the mean is 0, the variance is sigma^2 / (1 - phi^2), and the ACF is rho(k) = phi^k. If phi > 0, the ACF decays exponentially toward zero. If phi < 0, the ACF alternates in sign while decaying in magnitude.

The AR(2) model Y_t = phi_1*Y_{t-1} + phi_2*Y_{t-2} + epsilon_t is stationary when the roots of the characteristic polynomial 1 - phi_1*z - phi_2*z^2 = 0 lie outside the unit circle. In terms of the coefficients, the stationarity conditions are: phi_1 + phi_2 < 1, phi_2 - phi_1 < 1, and |phi_2| < 1.

The critical identification signature of AR(p) is: the PACF cuts off after lag p (equals zero for k > p), while the ACF tails off (decays gradually). This pattern is the reverse of MA models and is the primary way to distinguish between model types from sample data.

Moving Average (MA) Models

An MA(q) model expresses the current value as a linear combination of the current error and the q most recent past errors: Y_t = mu + epsilon_t + theta_1*epsilon_{t-1} + ... + theta_q*epsilon_{t-q}. Unlike AR models, MA models are always stationary regardless of the parameter values, because they are finite linear combinations of white noise terms.

For the MA(1) model Y_t = epsilon_t + theta*epsilon_{t-1}: the ACF is rho(1) = theta / (1 + theta^2) and rho(k) = 0 for k >= 2. The ACF cuts off after lag 1. The variance is (1 + theta^2) * sigma^2.

More generally, for MA(q), the ACF cuts off after lag q (rho(k) = 0 for k > q), while the PACF tails off. This is exactly the opposite of the AR pattern. To ensure uniqueness of the MA representation, we require the invertibility condition: the roots of 1 + theta_1*z + ... + theta_q*z^q = 0 must lie outside the unit circle. For MA(1), this means |theta| < 1.

A worked example: consider an MA(2) process Y_t = epsilon_t + 0.6*epsilon_{t-1} - 0.3*epsilon_{t-2} with sigma^2 = 4. The variance is (1 + 0.36 + 0.09) * 4 = 5.80. The ACF at lag 1 is rho(1) = (0.6 + 0.6*(-0.3)) / (1 + 0.36 + 0.09) = (0.6 - 0.18) / 1.45 = 0.290. The ACF at lag 2 is rho(2) = (-0.3) / 1.45 = -0.207. For k >= 3, rho(k) = 0.

ARMA Models

An ARMA(p, q) model combines both AR and MA components: Y_t = phi_1*Y_{t-1} + ... + phi_p*Y_{t-p} + epsilon_t + theta_1*epsilon_{t-1} + ... + theta_q*epsilon_{t-q}. The stationarity conditions depend only on the AR part, and the invertibility conditions depend only on the MA part.

For ARMA models, both the ACF and PACF tail off (neither cuts off cleanly). This makes order identification from ACF/PACF plots more difficult than for pure AR or MA models. In practice, you fit several candidate ARMA(p, q) models with small values of p and q (typically 0, 1, or 2) and compare them using AIC or BIC.

The ARMA(1, 1) model Y_t = phi*Y_{t-1} + epsilon_t + theta*epsilon_{t-1} is the simplest mixed model. Its ACF starts at rho(1) = (phi + theta)(1 + phi*theta) / (1 + 2*phi*theta + theta^2) and then decays as rho(k) = phi * rho(k-1) for k >= 2. The initial value differs from the pure AR pattern, but the subsequent exponential decay resembles AR(1).

ACF and PACF Identification Summary

The following patterns are essential for Exam SRM. For AR(p): ACF tails off (exponential or sinusoidal decay), PACF cuts off after lag p. For MA(q): ACF cuts off after lag q, PACF tails off. For ARMA(p, q): both ACF and PACF tail off. For white noise: both ACF and PACF are zero at all nonzero lags (all values fall within the confidence bands +/- 1.96/sqrt(n)).

When examining sample ACF and PACF plots, individual values outside the confidence bands at isolated high lags may be spurious (roughly 5% will exceed the bands by chance). Look for systematic patterns at low lags rather than isolated spikes at high lags.

Forecasting with AR and MA Models

For an AR(1) model, the h-step ahead forecast from time T is Y_hat_{T+h} = phi^h * Y_T (assuming mean zero). This forecast converges to zero (the mean) as h increases, with the rate of convergence depending on |phi|. The forecast variance increases with h, approaching the unconditional variance sigma^2 / (1 - phi^2) for large h.

For an MA(1) model, the one-step ahead forecast is Y_hat_{T+1} = theta * epsilon_T, but for h >= 2, the forecast is zero (the mean). MA models have a limited forecast horizon because the effect of past shocks dies out after q steps.

Forecast intervals are constructed as Y_hat_{T+h} +/- z_{alpha/2} * sqrt(Var(e_{T+h})), where the forecast error variance depends on the model and the horizon h. These intervals widen with the forecast horizon, reflecting increasing uncertainty about the future.

Study Tips for Exam SRM

For Exam SRM, the most important skills are: (1) identifying model type (AR, MA, ARMA) from ACF and PACF patterns, (2) checking stationarity conditions for AR models, (3) computing the ACF for simple models, and (4) calculating one-step and multi-step forecasts. Practice drawing ACF and PACF patterns for AR(1), AR(2), MA(1), and MA(2) models with various parameter values. Be comfortable with the duality between AR and MA: AR models have ACF that tails off and PACF that cuts off, while MA models show the reverse. This pattern recognition is tested repeatedly on the exam.