← Back to portfolio Churn Prediction — Leakage-Free Retention Model preview
Stack
PythonLightGBMscikit-learnpandas / NumPymatplotlibpytestuv
Impact
  • Chronological snapshot split (train/val/test) — no future activity leaks into training
  • Churn label = a future 30-day inactivity window for recently active users only
  • Decision metric recall@top-10% = 0.53, lift@top-10% = 3.07x
  • LightGBM Brier 0.068 vs 0.099 for the balanced logistic baseline
Source View on GitHub → Updated: Sep 18, 2026

Churn Prediction — Leakage-Free Retention Model

Context

A churn model for a subscription product. The value of the project is not the algorithm but the discipline: features are computed “as-of” a snapshot date, the target is a future inactivity window, and the split is chronological, so the model is evaluated the way it would be used in production. It is the same principle as A/B analysis: no leakage from the future into the moment being modelled.

Data & Method

The data is synthetic and deterministic (seed = 42): 12,000 users sign up between Jun 2023 and Jan 2024, a channel-driven tenure (exponential lifetime) and a plan-driven daily activity probability; activity decays slightly approaching churn, giving a churn rate of ~16% per snapshot.

Features and label

  • As-of features — recency, activity over prior 7/14/30 days, tenure, recent trend, average sessions, plus categorical channel / device / country / plan.
  • Leakage-free label — churn = no activity in [snapshot, snapshot+30d] for a user who was active in the prior 30 days. Long-dead users are excluded rather than labelled: predicting on ghosts is not a real task.

Split and metric

  • Chronological split — train (2024-01-15) -> val (2024-02-15) -> test (2024-03-15). A random split would put a user’s future activity into training and their past into test — silent leakage, and eliminating it is the whole reason the project exists.
  • Business metric first — ROC-AUC and PR-AUC are reported, but the decision metric is recall@top-decile and lift@top-decile: if retention acts on the top 10% riskiest users, how many actual churners do we catch.

Model and tests

  • SHAP — LightGBM’s native predict_proba(pred_contrib=True) (TreeSHAP without the shap package). recency_days dominates.
  • Baseline — a balanced logistic regression.
  • Tests — 5 pytest tests: balance, leakage, recency correctness, model beats baseline.
Metric (test snapshot) LightGBM LogReg (balanced)
ROC-AUC 0.904 0.917
PR-AUC 0.809 0.825
Recall@top-10% 0.53 0.54
Lift@top-10% 3.07x 3.08x
Brier (calibration) 0.068 0.099

Run

# Python >=3.10. Deps: pandas, numpy, scikit-learn, lightgbm, matplotlib.
uv run --with pandas --with numpy python data/generate_data.py
uv run --with pandas --with numpy --with scikit-learn --with lightgbm --with matplotlib python run.py
uv run --with pandas --with numpy --with scikit-learn --with lightgbm --with matplotlib --with pytest pytest -q

Outputs land in reports/: metrics.json + evaluation.png (ROC, PR, SHAP bar).

Findings

AUC is a near-tie (the synthetic features are nearly linear, so logistic regression is competitive), but LightGBM is meaningfully better calibrated (Brier 0.068 vs 0.099) — which matters when scores drive retention spend. Honest finding: a strong recency feature makes the lift modest; the value of the project is the leakage-free setup and the business metric, not a GBM trophy.

Impact

  • Chronological snapshot split — no future activity leaks into training.
  • Future-window label — we do not predict on long-departed users.
  • Business metric first — recall@top-10% 0.53, lift@top-10% 3.07x.
  • Calibration — Brier 0.068 (LightGBM) vs 0.099 (LogReg): the score is usable for retention-spend decisions.

Documentation

Case study

Problem

Churn models are often evaluated in a way that does not match how they are used: a random train/test split puts a user's future activity into training, and AUC does not answer the question the model exists for — whom in the top risk bucket should we act on. The project needs a model whose evaluation setup matches production.

Approach

Features are computed as-of a snapshot date: recency, activity in 7/14/30-day windows, tenure, recent trend, average sessions, plus categorical channel/device/country/plan. The leakage-free label defines churn as no activity in [snapshot, snapshot+30d] for a user who was active in the prior 30 days. The split is chronological (train 2024-01-15 -> val 2024-02-15 -> test 2024-03-15); a random split would be silent leakage. The decision metric is recall@top-decile and lift@top-decile, not AUC; SHAP comes from LightGBM's native TreeSHAP, with no shap package.

Result

On the test snapshot LightGBM and the logistic baseline are nearly tied on AUC (0.904 vs 0.917), but LightGBM is meaningfully better calibrated (Brier 0.068 vs 0.099). Within the top 10% riskiest users the model catches 53% of real churners — a 3.07x lift over random. Honest finding: a strong recency feature keeps the lift modest; the value of the project is the leakage-free setup and the business metric.

0.53 Recall@top-10%
3.07x Lift@top-10%
0.904 ROC-AUC
0.068 Brier

Charts

Source: github.com/NikitaBoyarkin/churn-prediction: metrics from run.py (reports/metrics.json) on the seeded synthetic dataset (12,000 users, seed=42); regenerate with `uv run --with pandas --with numpy --with scikit-learn --with lightgbm --with matplotlib python run.py`

LightGBM vs the logistic baseline

ROC-AUC, PR-AUC, recall@top-10% and Brier on the test snapshot (2024-03-15). ROC-AUC and PR-AUC are close; the difference shows in calibration — the metric that matters when the score drives retention spend.

0 0.5 1 ROC-AUC — LightGBM: 0.9 0.9 PR-AUC — LightGBM: 0.81 0.81 Recall@top-10% — LightGBM: 0.53 0.53 Brier — LightGBM: 0.07 0.07 ROC-AUC — LogReg (balanced): 0.92 0.92 PR-AUC — LogReg (balanced): 0.83 0.83 Recall@top-10% — LogReg (balanced): 0.54 0.54 Brier — LogReg (balanced): 0.1 0.1 ROC-AUC PR-AUC Recall@top-10% Brier Metric Value LightGBM LogReg (balanced)
Key takeaways
  • ROC-AUC is a near-tie (0.904 vs 0.917), but LightGBM is much better calibrated: Brier 0.068 vs 0.099.
  • Within the top 10% riskiest users the model catches 53% of real churners — a 3.07x lift over random (the baseline gets 3.08x with worse calibration).

See also

Connection map

Projects, posts and topics connected to this one. Hover a node to see its name; click to open.