Causal / Uplift — CUPED and Individual Treatment Effects
▶ Data-flow diagram — self-contained HTML, opens in a browser — no install, no build
Situation
A standard two-sample t-test answers only “does the treatment work on average” and ignores the pre-period. Experiments therefore ask for more traffic than they need, and retention offers go to everyone while only part of the audience responds.
The project closes both gaps: CUPED reduces variance using a pre-period covariate, and uplift modeling estimates the individual treatment effect (ITE). Both run on a synthetic randomized experiment with a known heterogeneous effect, so the estimates can be checked against ground truth.
Task
I needed the experiment to read both the average effect and who actually responds, so I owned the variance reduction and the uplift estimation, and checked both against a synthetic run with known ground truth.
Actions
Synthetic, deterministic data (seed = 42), 20,000 users in a randomized experiment:
segment—new(30%) /returning(70%) → heterogeneous treatment effect;x_pre— a pre-period covariate correlated with the outcome (ρ ≈ 0.67);treatment— 50/50, independent of everything (clean randomization);tau_true— the ground-truth individual treatment effect (eval only);y_cont— continuous outcome;y_bin— binary conversion.
CUPED — same ATE estimate, less variance:
| Method | ATE | SE | 95% CI |
|---|---|---|---|
| Naive | 0.270 | 0.019 | [0.232, 0.308] |
| CUPED | 0.276 | 0.014 | [0.247, 0.304] |
Uplift — individual effects, T- and S-learners:
| Model | AUUC | QINI | uplift@20% | corr(τ) |
|---|---|---|---|---|
| T-learner | 0.0043 | 0.0014 | 0.072 | 0.50 |
| S-learner | 0.0057 | 0.0029 | 0.041 | 0.66 |
| Random | 0.0037 | 0.0008 | 0.022 | −0.007 |
Run
uv run --with pandas --with numpy python data/generate_data.py
uv run --with pandas --with numpy --with scikit-learn --with lightgbm --with matplotlib python run.py
uv run --with pandas --with numpy --with scikit-learn --with lightgbm --with matplotlib --with pytest pytest -qOutputs: reports/metrics.json + reports/uplift.png (QINI curves + segment uplift vs ground truth).
Result
CUPED buys power for free: no new experiment design is needed, only a better estimator on data you already collected — provided a pre-period covariate exists. The point estimate is unchanged (0.270 → 0.276, within noise), the standard error shrinks by ~26%, and the CI narrows 1.35x — an experiment that needed 10k users per arm now needs ~5.5k.
Uplift answers a different question than A/B testing: not “does the treatment work on average” but “who does it work on”. At n = 6000 the curve metrics are noise-dominated: the T-learner sits 0.81σ from the null baseline (AUUC +0.00293 ± 0.00164), the S-learner 1.72σ, and even the oracle ranking by the known true τ reaches only 2.25σ, while the random scorer sits at 0.46σ — where an independent baseline belongs, which leaves the oracle as the ceiling. What is clean is not AUUC or QINI but the segment recovery and corr(τ) = 0.50–0.66 against −0.01 for randomness. The models recovered the shape of the effect: “new” users respond 6.3–6.8x more than “returning” users (the reference gap is 8.4x) — the targeting signal a discount campaign would act on.
- Ship rule (pre-registered): a break-even threshold of 0.05 — the campaign ships to
newusers only and is not shipped toreturning.
| Segment | Truth (binary uplift) | T-learner | S-learner |
|---|---|---|---|
| new | 0.110 | 0.126 | 0.120 |
| returning | 0.013 | 0.018 | 0.019 |
Limitations
The latent ground-truth τ is 0.60 (new) / 0.08 (returning), but the conversion uplift is ~0.11 / 0.01 because the sigmoid at a high baseline conversion (~71%) damps large latent effects. Comparing predicted binary uplift to latent τ would be a scale mismatch, so the report uses rank correlation (scale-free) and per-segment empirical recovery.
On real data the ITE is never observed — synthetic data with a true heterogeneous effect is the only reason recovery can be checked at all; that is the fundamental problem of causal inference, not a flaw of the method.