Architecture · Diagram Design
A/B testing methodology toolkit
A/B testing methodology toolkit
Architecture of a calibration-driven A/B testing library: a CLI and end-to-end pipeline call four method families — guardrails and sizing, ratio and variance estimators, sequential monitoring, and inference and design — each calibration of which is re-run as an assertion by a simulator-backed harness that emits the pipeline report, the figure gallery, and per-module scenario verdicts.
METHOD MODULES
CALIBRATION & EVIDENCE
SRM CHECK
ESTIMATE
MONITOR
INTERVALS
TYPE I
POWER
COVERAGE
FWER/FDR
REPORT
FIGURES
VERDICTS
ENTRY
Entry points — CLI and full pipeline
scripts/cli.py · run_full_pipeline.py
SRM
Guardrails & sizing
srm_test · sample_size
FOCAL
Ratio & variance
cuped · ratio_cuped · delta_method
SEQ
Sequential tests
group_sequential · msprt
INF
Inference & design
bayesian_ab_test · bootstrap_ci
CAL
Calibration harness
test_simulator.py · tests/ — every calibration re-run as an assertion
OUT
Pipeline report
outputs/report.md · pipeline_*.png
FIG
Figure gallery
plots/ · make_figures.py
EX
Scenario demos
examples/ · one VERDICT per module
LEGEND
Module family
Focal module
Layer (entry · calibration)
Emitted artefact
Call / assert flow