Trees & Ensembles

Day 2 Afternoon | WISE Workshop — Addis Ababa 2026

Sean Sylvia, Ph.D.

2026-02-03

Table of contents

• From Linear to Nonlinear
  • Morning Recap
  • The Limitation of Linear Models
• Decision Trees
  • The Intuition
  • How Trees Split: The Algorithm
  • Example: Titanic Survival
  • Trees: Pros and Cons
  • Tree Regularization
  • The Instability Problem
• Ensemble Methods
  • The Key Insight
• Random Forest
  • Algorithm Overview
  • Why Random Forest Works
  • Random Forest: Key Parameters
  • Out-of-Bag (OOB) Error
• Gradient Boosting
  • The Key Idea
  • Gradient Boosting Algorithm
  • Gradient Boosting: Key Parameters
  • Popular Implementations
• Random Forest vs Gradient Boosting
  • Head-to-Head Comparison
  • When to Use Which
• Feature Importance
  • Understanding What Matters
  • Example: Supply Chain Features
• Algorithm Selection Guide
  • Decision Framework
  • Our Approach This Afternoon
• Connection to Supply Chain
  • From Predictions to Simulation
  • Real-World Example: Drug Shortage Prediction
• Key Takeaways
  • Summary: Trees & Ensembles
  • Discussion: Before Hands-on
  • Next: Full ML Pipeline

From Linear to Nonlinear

Morning Recap

Regularized linear models (LASSO, Ridge):

• Add penalties to constrain coefficients
• LASSO performs feature selection
• Assume linear relationships in features

This afternoon:

• What if the relationship is nonlinear?
• What if there are interactions we can’t specify in advance?

The Limitation of Linear Models

Consider predicting demand:

\[\text{Demand} = \beta_0 + \beta_1 \cdot \text{Season} + \beta_2 \cdot \text{Facility\_Size} + ...\]

Problems:

• What if effect of season differs by facility size?
• What if relationship is threshold-based?
• Manually specifying interactions is tedious and incomplete

Trees discover nonlinearities and interactions automatically

Decision Trees

The Intuition

How a tree makes predictions:

                    [Month Demand < 500?]
                          /         \
                        Yes          No
                         |            |
              [Rainy Season?]    [Facility Size?]
                 /      \           /      \
               Yes      No       Small    Large
                |        |         |        |
            Predict:  Predict:  Predict:  Predict:
              650       420       550       720

• Split data based on feature thresholds
• Each leaf contains a prediction
• Path from root to leaf = decision rules

How Trees Split: The Algorithm

At each node:

1. Consider all features and all possible thresholds
2. Choose the split that best separates outcomes
3. Repeat recursively until stopping criterion

For regression (predicting demand):

• Minimize within-node variance (MSE)
• Each split reduces prediction error

For classification (stockout: yes/no):

• Minimize impurity (Gini index or entropy)
• Each split creates purer groups

Example: Titanic Survival

Classic tree example:

Trees capture interactions naturally: being female AND in first class → high survival

Trees: Pros and Cons

Advantages ✅

• Easy to interpret and explain
• Handles nonlinear relationships
• Captures interactions automatically
• No feature scaling needed
• Handles missing values
• Works for regression and classification

Limitations ❌

• Prone to overfitting
• High variance (unstable)
• Small data changes → different tree
• Greedy algorithm (locally optimal)
• Often outperformed by ensembles

Tree Regularization

Control complexity with hyperparameters:

Parameter	What it controls	Trade-off
max_depth	How deep tree can grow	Deeper = more complex
min_samples_leaf	Min observations per leaf	Higher = simpler
min_samples_split	Min observations to split	Higher = simpler
max_features	Features considered per split	Fewer = more randomness

Shallow trees: High bias, low variance (underfit)

Deep trees: Low bias, high variance (overfit)

The Instability Problem

Run on different random subsets → very different trees

Subset 1:                    Subset 2:
[Season < Summer?]          [Facility Size > Large?]
    /       \                    /          \
  [...]    [...]              [...]        [...]

Solution: Combine many trees into an ensemble

Ensemble Methods

The Key Insight

“The wisdom of crowds”

Single Tree

• High variance
• Sensitive to data
• One perspective

Many Trees (Ensemble)

• Lower variance
• More stable
• Multiple perspectives averaged

Two main approaches:

1. Random Forest: Independent trees, averaged
2. Gradient Boosting: Sequential trees, each corrects previous

Random Forest

Algorithm Overview

1. Create B bootstrap samples from training data
2. Train a tree on each sample
3. At each split, consider only m random features
4. Average predictions (regression) or vote (classification)

Tree 1: 650    Tree 2: 620    Tree 3: 680    ...    Tree B: 640
              ↘        ↓        ↙
              Final Prediction: 648

Why Random Forest Works

Two sources of randomness:

Randomness	How	Effect
Bootstrap sampling	Each tree sees ~63% of data	Different data perspectives
Feature subsampling	Random m features per split	Decorrelated trees

Result:

• Individual trees make different errors
• Errors cancel out when averaged
• Much more stable than single tree

Random Forest: Key Parameters

Parameter	What it controls	Typical values
n_estimators	Number of trees	100-500
max_depth	Depth of each tree	None (full) or limited
max_features	Features per split	sqrt(p) for classification, p/3 for regression
min_samples_leaf	Min samples in leaf	1-5

Good news: Random Forests work well out-of-the-box with defaults!

Out-of-Bag (OOB) Error

Built-in validation:

• Each tree trained on ~63% of data (bootstrap)
• Remaining ~37% is “out-of-bag” for that tree
• Predict OOB samples, average across trees
• Free cross-validation estimate

rf = RandomForestRegressor(n_estimators=100, oob_score=True)
rf.fit(X_train, y_train)
print(f"OOB Score: {rf.oob_score_}")

Gradient Boosting

The Key Idea

Learn sequentially, each tree corrects previous errors

Iteration 1: Predict mean                    Error: Large
Iteration 2: Predict mean + tree₁            Error: Medium
Iteration 3: Predict mean + tree₁ + tree₂    Error: Smaller
...
Iteration B: Σ trees                         Error: Small

Each new tree focuses on the residuals (mistakes) of the current model

Gradient Boosting Algorithm

1. Initialize with simple prediction (e.g., mean)
2. Compute residuals: \(r_i = y_i - \hat{y}_i\)
3. Fit a tree to predict the residuals
4. Update predictions: \(\hat{y} \leftarrow \hat{y} + \eta \cdot \text{tree}(X)\)
5. Repeat steps 2-4

Learning rate (η): How much each tree contributes

• Small η + many trees = better but slower
• Large η + few trees = faster but less accurate

Gradient Boosting: Key Parameters

Parameter	What it controls	Typical values
n_estimators	Number of boosting rounds	100-1000
learning_rate	Step size (η)	0.01-0.3
max_depth	Depth of each tree	3-10 (shallow)
min_samples_leaf	Min samples in leaf	1-20
subsample	Fraction of data per tree	0.5-1.0

Trade-off: More trees + lower learning rate = better but slower

Popular Implementations

Library	Strengths	When to use
XGBoost	Fast, regularization built-in	Production, competitions
LightGBM	Very fast, handles large data	Big datasets
CatBoost	Handles categorical features	Mixed data types
sklearn GradientBoosting	Simple, educational	Learning, small data

XGBoost and LightGBM are current state-of-the-art for tabular data

Random Forest vs Gradient Boosting

Head-to-Head Comparison

Aspect	Random Forest	Gradient Boosting
Training	Parallel (fast)	Sequential (slower)
Trees	Independent	Build on each other
Overfitting risk	Lower	Higher (needs tuning)
Out-of-the-box	Often good	Needs more tuning
State-of-art	Good	Often best
Interpretability	Feature importance	Feature importance

When to Use Which

Random Forest

• Quick baseline
• Less tuning time available
• Want robustness
• Parallel training needed
• Good enough is sufficient

Gradient Boosting

• Maximum accuracy needed
• Time for careful tuning
• Kaggle/competitions
• Production with proper validation
• State-of-the-art required

Practical advice: Start with Random Forest, try Gradient Boosting if you need better

Feature Importance

Understanding What Matters

Both ensembles provide feature importance scores:

Methods:

Method	How it works
Mean Decrease in Impurity	Average reduction in node impurity across trees
Permutation Importance	Drop in accuracy when feature is shuffled

Example: Supply Chain Features

import pandas as pd
importances = pd.DataFrame({
    'feature': feature_names,
    'importance': rf.feature_importances_
}).sort_values('importance', ascending=False)

print(importances.head(10))

Feature	Importance
Lag-1 Demand	0.35
Facility Size	0.18
Season	0.12
Distance to Warehouse	0.08
…	…

Algorithm Selection Guide

Decision Framework

                    ┌─────────────────────────────┐
                    │  What's your data like?     │
                    └─────────────────────────────┘
                              │
           ┌──────────────────┼──────────────────┐
           │                  │                  │
       Linear?            Non-linear?       High-dimensional?
           │                  │                  │
           ▼                  ▼                  ▼
      LASSO/Ridge       Tree methods         LASSO/Elastic Net
                              │
                    ┌─────────┴─────────┐
                    │                   │
              Need speed?         Need accuracy?
                    │                   │
                    ▼                   ▼
             Random Forest      Gradient Boosting

Our Approach This Afternoon

1. Baseline: LASSO (from this morning)
2. Random Forest: Good out-of-box performance
3. Gradient Boosting: Often best, more tuning
4. Compare: Using cross-validation metrics
5. Feature Importance: What drives predictions?

Start simple, add complexity only if needed

Connection to Supply Chain

From Predictions to Simulation

Your ML models will:

1. Predict demand at each facility
2. Identify which features matter most
3. Generate forecasts for multiple scenarios

Days 3-4 ABM will use these as inputs:

ML Predictions → ABM Simulation
   ↓                 ↓
Demand forecasts → Inventory decisions
Risk scores → Prioritization
Feature importance → What-if scenarios

Real-World Example: Drug Shortage Prediction

The Study

Roe et al. (2025) applied ML to predict drug shortages in South Korea:

• Random Forest with Bayesian optimization
• 1,054 shortage cases
• Two prediction tasks

Results

Task	Performance
Duration prediction	62% accuracy
Cause classification	>70% F1

Top features: Shortage frequency, import status, alternatives

Similar approach to our supply chain forecasting

Key Takeaways

Summary: Trees & Ensembles

1. Decision trees capture nonlinearities and interactions automatically

2. Trees alone are unstable and prone to overfitting

3. Random Forest: Average many independent trees → stable predictions

4. Gradient Boosting: Sequential trees that learn from mistakes → often best accuracy

5. Feature importance helps understand what drives predictions

6. Start with RF, try Gradient Boosting if you need more

Discussion: Before Hands-on

1. What nonlinear patterns might exist in medication demand?

2. Which features might interact in ways linear models miss?

3. How could feature importance inform supply chain decisions?

Next: Full ML Pipeline

Apply everything to demand forecasting:

• Compare LASSO, Random Forest, Gradient Boosting
• Use cross-validation for honest comparison
• Analyze feature importance
• Generate predictions for simulation

Open Demand Forecasting Notebook