Families of ML Algorithms

Day 2 | WISE Workshop — Addis Ababa 2026

Sean Sylvia, Ph.D.

2026-02-03

Session 2: Algorithm Families

Overview

Linear Methods

  • Linear Regression
  • Ridge (L2)
  • LASSO (L1)
  • Elastic Net

Tree-Based Methods

  • Decision Trees
  • Random Forests
  • Gradient Boosting
  • XGBoost/LightGBM

For supply chain forecasting: We’ll focus on LASSO, Random Forests, and Gradient Boosting

The Regularization Spectrum

Every model class has its own form of regularization:

Function Class Regularization Parameters
Linear LASSO (L1), Ridge (L2), Elastic Net
Decision Trees Depth, leaves, leaf size, info gain
Random Forest # trees, features per split, sample sizes
Gradient Boosting # trees, learning rate, depth, regularization

Cross-cutting insight: Regularization always restricts model capacity to prevent memorizing training data and encourage generalization.

Linear Models with Regularization

Linear Regression: The Foundation

\[\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_p x_p\]

Problem with OLS in high dimensions:

  • Many features (p large) relative to observations (n)
  • Overfitting: model fits noise, not signal
  • Coefficients become unstable

Solution: Add a penalty to constrain coefficients

Ridge vs LASSO: The Key Difference

Ridge Regression (L2)

\[\min_\beta \sum_{i=1}^n (y_i - X_i\beta)^2 + \lambda \sum_{j=1}^p \beta_j^2\]

  • Penalty: Sum of squared coefficients
  • Effect: Shrinks all coefficients toward zero
  • Never sets coefficients exactly to zero

LASSO (L1)

\[\min_\beta \sum_{i=1}^n (y_i - X_i\beta)^2 + \lambda \sum_{j=1}^p |\beta_j|\]

  • Penalty: Sum of absolute coefficients
  • Effect: Shrinks AND selects features
  • Can set coefficients exactly to zero

LASSO: Automatic Variable Selection

λ increasing →

Feature 1:  ═══════════════════════════■──────────▶ 0
Feature 2:  ═════════════════■──────────────────────▶ 0
Feature 3:  ═══════■────────────────────────────────▶ 0
Feature 4:  ═══════════════════════════════════■────▶ 0

■ = coefficient becomes exactly zero
  • As \(\lambda\) increases, more coefficients shrink to zero
  • Automatic feature selection!
  • Useful when you have many potential predictors

When to Use Linear Methods

Advantages:

  • Interpretable coefficients
  • Fast training and prediction
  • Works well with many features
  • Good baseline to compare against
  • Handles high-dimensional data (with regularization)

Limitations:

  • Assumes linear relationships
  • Sensitive to outliers
  • May underfit complex patterns (non-linearities, interactions)

Tree-Based Methods

Decision Trees: The Intuition

How a tree makes predictions:

                    [Month Demand < 500?]
                          /         \
                        Yes          No
                         |            |
              [Rainy Season?]    [Facility Size?]
                 /      \           /      \
               Yes      No       Small    Large
                |        |         |        |
            Predict:  Predict:  Predict:  Predict:
              650       420       550       720
  • Split data recursively based on feature thresholds
  • Each leaf contains a prediction
  • Easy to interpret and visualize!

How Trees Split: Information Gain

At each node, find the split that best separates the data:

For regression (predicting demand): - Minimize within-node variance (MSE)

For classification (predicting stockout: yes/no): - Maximize information gain (reduce entropy) - Or: minimize Gini impurity

Trees automatically capture non-linearities and interactions!

Tree Regularization: Controlling Complexity

Parameter What it controls
Max depth How deep the tree can grow
Min samples per leaf Minimum observations in each leaf
Min samples to split Minimum observations to create a split
Max features How many features to consider at each split

Shallow trees: High bias, low variance (underfit)

Deep trees: Low bias, high variance (overfit)

Decision Trees: Pros and Cons

Advantages ✅

  • Easy to interpret and explain
  • Handles non-linear relationships
  • Captures interactions automatically
  • No feature scaling needed
  • Handles missing values

Limitations ❌

  • Prone to overfitting
  • High variance (unstable)
  • Small data changes → different tree
  • Often outperformed by ensembles

Solution: Combine many trees! (Ensembles)

Ensemble Methods

Random Forest: Wisdom of the Crowd

Key idea: Combine many diverse trees to reduce variance

Algorithm:

  1. Create B bootstrap samples from training data
  2. Train a tree on each sample
  3. At each split, consider only m random features (not all p)
  4. Average predictions (regression) or vote (classification)
Tree 1: 650    Tree 2: 620    Tree 3: 680    ...    Tree B: 640
              ↘        ↓        ↙
              Final Prediction: 648

Why Random Forest Works

Two sources of randomness:

  1. Bootstrap sampling: Each tree sees slightly different data
  2. Feature subsampling: Each split considers different features

Result:

  • Trees are decorrelated (different errors)
  • Averaging reduces variance without increasing bias
  • Much more stable than a single tree

Gradient Boosting: Learning from Mistakes

Key idea: Build trees sequentially, each correcting previous errors

Algorithm:

  1. Train initial model (e.g., predict mean)
  2. Compute residuals (errors) from current model
  3. Train new tree to predict the residuals
  4. Add new tree to ensemble (scaled by learning rate)
  5. Repeat steps 2-4
Iteration 1: Predict mean                    Error: Large
Iteration 2: Predict mean + tree₁            Error: Medium
Iteration 3: Predict mean + tree₁ + tree₂    Error: Smaller
...
Iteration B: Σ trees                         Error: Small

Gradient Boosting: Key Parameters

Parameter What it controls Typical values
n_estimators Number of trees 100-1000
learning_rate Step size for updates 0.01-0.3
max_depth Depth of each tree 3-10
min_samples_leaf Min observations per leaf 1-20
subsample Fraction of data per tree 0.5-1.0

Trade-off: More trees + lower learning rate = better but slower

Random Forest vs Gradient Boosting

Aspect Random Forest Gradient Boosting
Training Parallel (fast) Sequential (slower)
Trees Independent Build on each other
Overfitting risk Lower Higher (needs tuning)
Out-of-the-box Often good Needs more tuning
State-of-art Good Often best

Popular implementations:

  • Gradient Boosting: XGBoost, LightGBM, CatBoost
  • Random Forest: scikit-learn, ranger

Algorithm Selection Guide

                    ┌─────────────────────────────┐
                    │  What's your data like?     │
                    └─────────────────────────────┘
                              │
           ┌──────────────────┼──────────────────┐
           │                  │                  │
       Linear?            Non-linear?       High-dimensional?
           │                  │                  │
           ▼                  ▼                  ▼
      Linear/LASSO      Tree methods         LASSO/Elastic Net
                              │
                    ┌─────────┴─────────┐
                    │                   │
              Need speed?         Need accuracy?
                    │                   │
                    ▼                   ▼
             Random Forest      Gradient Boosting

Our Approach Today

  1. Baseline: Linear regression or LASSO
  2. Random Forest: Good out-of-box performance
  3. Gradient Boosting: Often best, more tuning
  4. Compare: Using cross-validation metrics
  5. Tune: Hyperparameters of best model

Start simple, add complexity only if needed!

Key Takeaways: Session 2

  1. LASSO combines prediction with automatic feature selection
  2. Decision trees capture non-linearities and interactions
  3. Ensembles (Random Forest, Gradient Boosting) beat single models
  4. Random Forest: Parallel trees, good out-of-box
  5. Gradient Boosting: Sequential, often best but needs tuning
  6. Always compare multiple approaches with cross-validation

Let’s Practice!

Time for hands-on: Forecasting demand with these algorithms.

Open Demand Forecasting Notebook