A/B Testing Methodology Toolkit

Business Context

An A/B testing method is only as good as its Type I error under the null and its power under a real effect. Rather than trusting asymptotic promises, each module simulates the pipeline end-to-end and reports the empirical rates.

Modules

Module	Method	What the demo shows
`srm_test.py`	Sample Ratio Mismatch (χ²)	catches bucketing/traffic bugs before any downstream test
`sample_size.py`	Fixed-horizon sizing	n/arm for proportions and means
`delta_method_ratio.py`	Ratio metrics (CTR, RPC)	correct SE for ΣY/ΣX; the naive per-unit t-test is biased
`cuped.py`	Variance reduction	SE shrinks by ~corr(X,Y)² using pre-period data
`group_sequential.py`	Alpha-spending boundaries	Pocock/OBF control Type I while naive peeking inflates it
`msprt_always_valid.py`	Always-valid p-values	mSPRT lets you peek and stop any time, validly
`sequential_ratio.py`	Sequential ratio metrics	delta-method + mSPRT for CTR under continuous monitoring
`sequential_ab_testing.py`	Evan Miller’s sequential rule	reproduces the size table, validates Type I/power, sample savings
`bayesian_ab_test.py`	Analytic Bayesian A/B	Beta-Binomial / Normal-Normal, P(B>A), expected loss, ROPE
`bootstrap_ci.py`	Bootstrap CIs	percentile & BCa for skewed metrics
`heterogeneous_treatment_effects.py`	HTE by segment	interaction model reveals Simpson-paradox-like cancellation
`multiple_comparisons.py`	Multiple-testing correction	Bonferroni (FWER) vs Benjamini-Hochberg (FDR)
`novelty_primacy.py`	Time-varying effects	treat×day interaction detects novelty decay / primacy growth
`switchback.py`	Cluster & switchback designs	cluster-robust SE; naive over-/under-rejects; carryover bias
`test_simulator.py`	Generic test calibration	plug any DGP + test → empirical Type I and power curve

End-to-end pipeline

scripts/run_full_pipeline.py ties the modules into one realistic flow on synthetic data: SRM check → CUPED → delta-method CTR test → per-segment ATE with BH correction → novelty check → a markdown report in outputs/report.md.

Testing Philosophy

The tests/ suite re-runs every calibration with assertions:

Type I error ≈ α (± tolerance) for each method under its null;
CI coverage ≈ 95% for the bootstrap;
naive peeking inflates Type I, always-valid / alpha-spending control it;
the naive per-unit ratio SE is inaccurate, the delta-method SE is accurate;
correctness on known-answer fixtures (SRM splits, segment uplifts, etc.).

Run

uv sync --all-groups
uv run pytest                 # calibration test suite
uv run python scripts/run_full_pipeline.py   # end-to-end demo → outputs/report.md

Impact

15 calibrated modules — from SRM to switchback, each with an A/A null check.
Checked numbers — Type I ≈ α, coverage ≈ 95%, peeking controlled.
Unbiased ratio metrics — delta method instead of the naive per-unit t-test.
Cohesive pipeline — from SRM check to per-segment ATEs and a novelty check.

Documentation

GitHub → ab_test