Categorical Variable Encoding for Predictive Models

One-Hot Encoding (Dummy Variables)

One-hot encoding creates k-1 binary indicator variables for a categorical variable with k levels (the omitted level is the reference category). For example, a "Region" variable with levels North, South, East, West becomes three dummy variables. In the regression, each coefficient measures the difference between that level and the reference level.

One-hot encoding is the standard approach for linear regression, logistic regression, and GLMs. However, for variables with many levels (e.g., ZIP code), it creates many columns, increasing dimensionality and potentially causing overfitting.

Ordinal and Target Encoding

Ordinal encoding assigns integers (1, 2, 3, ...) to levels. This is appropriate only when the levels have a natural order (e.g., education level, credit rating). Using ordinal encoding for nominal variables (no natural order) imposes a false ranking. Target encoding (mean encoding) replaces each category level with the mean of the target variable for that level. This reduces dimensionality to a single column. However, it can cause data leakage if not done carefully (use cross-validated target encoding or add regularization).

Handling High-Cardinality Variables

Variables with many categories (e.g., vehicle model, occupation code) pose challenges. Approaches include: grouping rare categories into an "Other" level, target encoding with smoothing (blend the category mean with the overall mean, weighted by the number of observations), and embedding layers in neural networks. For tree-based models, many implementations handle categorical variables natively by considering all possible binary splits of the categories. For Exam SRM, know when each encoding method is appropriate and the risks of each approach, especially data leakage with target encoding.