Neural Network Basics for Actuarial Applications
Understand neural network fundamentals and their actuarial applications for Exam SRM.
Neural Network Architecture
A feedforward neural network consists of layers of interconnected nodes (neurons). The input layer has one node for each predictor variable. One or more hidden layers perform the computation. The output layer produces the final prediction. Information flows forward from input through hidden layers to output, with no cycles (hence "feedforward").
Each node in a hidden layer computes two things: first, a weighted sum of its inputs plus a bias term (the linear combination z = w_1*x_1 + w_2*x_2 + ... + w_p*x_p + b), and second, a nonlinear activation function applied to that sum: h = g(z). The weights w and biases b are the parameters that the network learns from data.
The nonlinearity is essential. Without activation functions (or with only linear activations), any number of layers would collapse into a single linear transformation, and the network could only model linear relationships. The activation function gives the network the ability to approximate any continuous function (the universal approximation theorem), provided the hidden layer is wide enough.
Activation Functions
The choice of activation function affects both the network's capacity and the ease of training. The most common activation functions for hidden layers are:
ReLU (Rectified Linear Unit): g(z) = max(0, z). This is the default choice in modern networks. It is computationally efficient, avoids the vanishing gradient problem for positive values, and has been shown to work well in practice. Its main drawback is that neurons with negative inputs have zero gradient ("dying ReLU problem"), which can slow training.
Sigmoid: g(z) = 1 / (1 + exp(-z)), which maps inputs to the range (0, 1). It was historically popular but is now less common in hidden layers because it suffers from vanishing gradients (the gradient approaches zero for very large or very small inputs, making learning very slow).
Tanh: g(z) = (exp(z) - exp(-z)) / (exp(z) + exp(-z)), which maps inputs to (-1, 1). It is centered at zero (unlike sigmoid), which can help learning, but still suffers from vanishing gradients at extreme values.
The output layer activation depends on the task. For regression, use a linear (identity) activation so the output can take any value. For binary classification, use sigmoid to produce a probability between 0 and 1. For multi-class classification with K classes, use softmax, which produces K probabilities that sum to 1.
Network Depth and Width
The number of hidden layers (depth) and the number of nodes per layer (width) determine the network's complexity. A single hidden layer with enough nodes can approximate any continuous function, but deeper networks (multiple hidden layers) can often learn the same function with far fewer total parameters by building up increasingly abstract representations.
For actuarial applications on Exam SRM, networks are typically small: one or two hidden layers with 5 to 50 nodes each. The total number of parameters grows quickly with network size. A network with p inputs, one hidden layer of h nodes, and one output has h*(p+1) + (h+1) = h*p + 2h + 1 parameters. With p = 10 and h = 20, that is 231 parameters, which is modest. But a two-hidden-layer network with layers of 100 nodes each could have over 11,000 parameters, requiring much more data to train reliably.
Training with Backpropagation
Training a neural network means finding the weights and biases that minimize a loss function. For regression, the loss is typically mean squared error: L = (1/n) * sum(y_i - f(x_i))^2. For binary classification, the loss is cross-entropy: L = -(1/n) * sum[y_i * ln(p_i) + (1-y_i) * ln(1-p_i)].
The backpropagation algorithm computes the gradient of the loss with respect to every weight in the network using the chain rule. Starting from the output layer and working backward (hence "back" propagation), the gradient at each layer depends on the gradient from the layer above and the local derivatives of the activation functions and weighted sums.
Once the gradients are computed, stochastic gradient descent (SGD) or its variants update the weights. Instead of computing the gradient using all n observations (which is slow), mini-batch SGD uses a random subset (batch) of observations for each update. One pass through the entire dataset is called an epoch. Common optimizers include Adam (which adapts the learning rate for each parameter) and RMSprop, both of which generally converge faster than plain SGD.
Regularization and Overfitting
Neural networks with many parameters are prone to overfitting, especially with limited training data. Several regularization techniques help control this.
Weight decay (L2 regularization) adds a penalty lambda * sum(w^2) to the loss function, analogous to ridge regression. This discourages large weights and produces smoother predictions.
Dropout randomly sets a fraction of hidden nodes to zero during each training step (typically 20% to 50%). This forces the network to learn redundant representations and prevents co-adaptation of neurons. At prediction time, all nodes are used but their outputs are scaled by the dropout rate.
Early stopping monitors the validation error during training and stops when it begins to increase, preventing the network from memorizing the training data. This is one of the simplest and most effective regularization techniques.
Actuarial Applications
Neural networks are increasingly used in actuarial science, though their adoption is more cautious than in other fields due to regulatory requirements for model interpretability. Current applications include telematics data analysis (processing driving behavior data for auto insurance pricing), image-based claims processing (using computer vision to assess damage from photographs), and as components of ensemble models (combining neural network predictions with GLMs).
A practical compromise in actuarial work is the "combined actuarial neural network" (CANN) approach, which uses a GLM as the base model and a neural network to capture residual patterns that the GLM misses. This retains the GLM's interpretability for regulatory purposes while gaining predictive accuracy.
Study Tips for Exam SRM
For Exam SRM, the focus is on conceptual understanding rather than implementation. Know the basic architecture (input, hidden, output layers), the role of activation functions (they introduce nonlinearity), how backpropagation computes gradients (chain rule applied layer by layer), and the key hyperparameters (number and size of hidden layers, learning rate, epochs, regularization). Understand the tradeoff between complexity and interpretability: neural networks can model complex patterns but are "black boxes" compared to GLMs and decision trees. Be able to compare neural networks to other methods and explain when they are or are not appropriate for a given actuarial problem.