A/B Testing Methodology Toolkit

Q: How is a method's correctness verified?

By simulation calibration: an A/A run holds Type I error at α and bootstrap CI coverage sits near 95% — the property is backed by a run, not by a citation.

Q: How much variance does CUPED remove?

ρ² of it, which is the standard error by roughly a factor of ρ.

Q: Why is naive peeking dangerous?

It inflates Type I error; Pocock/OBF and mSPRT keep it under control.

Q: How should ratio metrics like CTR be tested?

The delta method gives the correct SE for CTR and RPC — the naive per-unit t-test is biased on a ratio metric.

Q: What does the end-to-end pipeline contain?

SRM → CUPED → delta-method CTR → per-segment ATE with a BH correction → novelty check.

▶ Architecture diagram — self-contained HTML, opens in a browser — no install, no build

Situation

An A/B testing method is only as good as its Type I error under the null and its power under a real effect. In practice those promises are rarely checked: naive peeking, a wrong SE for ratio metrics, and multiple testing silently break decisions. The toolkit implements 18 methods from the primary literature and calibrates each one by simulation instead of trusting asymptotics.

Task

I needed a testing toolkit whose methods hold up under the null and under a real effect, so I owned the calibration layer and checked every method by simulation instead of trusting asymptotics.

Actions

Modules

Module	Method	What the demo shows
`srm_test.py`	Sample Ratio Mismatch (χ²)	catches bucketing/traffic bugs before any downstream test
`sample_size.py`	Fixed-horizon sizing	n/arm for proportions and means
`delta_method_ratio.py`	Ratio metrics (CTR, RPC)	correct SE for ΣY/ΣX; the naive per-unit t-test is biased
`ratio_cuped.py`	CUPED on a ratio metric	linearise Z = Y − R·X: pre-period data cuts SE without new sample
`cuped.py`	Variance reduction	variance ↓ by ρ², SE by ~ρ, using pre-period data
`cupac.py`	CUPAC	OLS over every pre-period column removes more variance than one covariate
`post_stratification.py`	Post-stratification	repairs a drifted split using pre-registered strata
`group_sequential.py`	Alpha-spending boundaries	Pocock/OBF control Type I while naive peeking inflates it
`msprt_always_valid.py`	Always-valid p-values	mSPRT lets you peek and stop any time, validly
`sequential_ratio.py`	Sequential ratio metrics	delta-method + mSPRT for CTR under continuous monitoring
`sequential_ab_testing.py`	Evan Miller’s sequential rule	reproduces the size table, validates Type I/power, sample savings
`bayesian_ab_test.py`	Analytic Bayesian A/B	Beta-Binomial / Normal-Normal, P(B>A), expected loss, ROPE
`bootstrap_ci.py`	Bootstrap CIs	percentile & BCa for skewed metrics
`heterogeneous_treatment_effects.py`	HTE by segment	interaction model reveals Simpson-paradox-like cancellation
`multiple_comparisons.py`	Multiple-testing correction	Bonferroni (FWER) vs Benjamini-Hochberg (FDR)
`novelty_primacy.py`	Time-varying effects	treat×day interaction detects novelty decay / primacy growth
`switchback.py`	Cluster & switchback designs	cluster-robust SE; naive over-/under-rejects; carryover bias
`test_simulator.py`	Generic test calibration	plug any DGP + test → empirical Type I and power curve

End-to-end pipeline

scripts/run_full_pipeline.py ties the modules into one realistic flow on synthetic data: SRM check → CUPED → delta-method CTR test → per-segment ATE with BH correction → novelty check → a markdown report in outputs/report.md.

Run

uv sync --all-groups
uv run pytest                 # calibration test suite
uv run python scripts/run_full_pipeline.py   # end-to-end demo → outputs/report.md
uv run python scripts/cli.py --help          # run any method from the terminal

Result

The tests/ suite re-runs every calibration with assertions:

Type I error ≈ α (± tolerance) for each method under its null;
CI coverage ≈ 95% for the bootstrap;
naive peeking inflates Type I, always-valid / alpha-spending control it;
the naive per-unit ratio SE is inaccurate, the delta-method SE is accurate;
correctness on known-answer fixtures (SRM splits, segment uplifts, etc.);
CUPED on the pipeline run: variance −17.7%, SE 0.00360 → 0.00327;
delta-method CTR: +0.51 pp lift at p < 0.0001, SRM p = 0.8376.

Limitations

The numbers come from simulations with a known DGP, not production traffic: calibration confirms the implementation is correct, but it does not guarantee that real data satisfies the assumptions. This is a set of tools, not a substitute for a sound experiment design.

Documentation

GitHub → ab_test

Frequently asked

How is a method's correctness verified?

By simulation calibration: an A/A run holds Type I error at α and bootstrap CI coverage sits near 95% — the property is backed by a run, not by a citation.

How much variance does CUPED remove?

ρ² of it, which is the standard error by roughly a factor of ρ.

Why is naive peeking dangerous?

It inflates Type I error; Pocock/OBF and mSPRT keep it under control.

How should ratio metrics like CTR be tested?

The delta method gives the correct SE for CTR and RPC — the naive per-unit t-test is biased on a ratio metric.

What does the end-to-end pipeline contain?

SRM → CUPED → delta-method CTR → per-segment ATE with a BH correction → novelty check.

A/B Testing Methodology Toolkit

Situation

Task

Actions

Modules

End-to-end pipeline

Run

Result

Limitations

Documentation

Frequently asked

Charts

Type I error when peeking at interim looks

False positives under continuous peeking

Sequential test's average sample size

Sample savings by minimum detectable effect

Need this reasoning applied to your data?

A/B Testing Methodology Toolkit

Situation

Task

Actions

Modules

End-to-end pipeline

Run

Result

Limitations

Documentation

Frequently asked

Charts

Type I error when peeking at interim looks

False positives under continuous peeking

Sequential test's average sample size

Sample savings by minimum detectable effect

See also

Linked from

Connection map

Need this reasoning applied to your data?