CUPED

On this page
  1. Definition
  2. How to compute
  3. Pitfalls

Definition

CUPED (Controlled-experiment Using Pre-Experiment Data) reduces the variance of a metric using a covariate measured before the experiment. Part of a metric’s spread is explained by a user’s pre-experiment behaviour; subtracting the component predicted from it yields the same unbiased effect estimate with a smaller standard error.

How to compute

The adjusted metric is Y~=Y−θ(X−Xˉ)\tilde{Y} = Y - \theta (X - \bar X), where XX is the same metric or a proxy measured before launch and θ=Cov⁡(Y,X)/Var⁡(X)\theta = \operatorname{Cov}(Y, X)/\operatorname{Var}(X). The optimal θ\theta removes ρ2\rho^2 of the variance, so the standard error shrinks by roughly 1−ρ2\sqrt{1-\rho^2}. At ρ=0.7\rho = 0.7 that is about a 30% saving in sample size.

Pitfalls

XX must be pre-experiment and independent of the treatment, or the estimate is biased. The correlation has to be stable across arms; if the pre-period does not resemble the test window, θ\theta fits noise. For ratio metrics the covariance must be computed through the delta method, not directly.

Related terms

Mentioned in

Connections map

Interactive subgraph: 10 nodes and 28 links around this page. Each node is keyboard-focusable; press Enter or Space to open its page. A full list of nodes and their links follows the graphic.

  • Causal / Uplift — CUPED and Individual Treatment Effects : Correlation and causation (Explicit link), CUPED (Explicit link), Uplift modelling (Explicit link)
  • A/A test : Sample Ratio Mismatch (SRM) (Explicit link), p-value (Explicit link), CUPED (Explicit link), Statistical power (Explicit link), ab-testing (Thematic link)
  • Correlation and causation : Causal / Uplift — CUPED and Individual Treatment Effects (Explicit link), Uplift modelling (Explicit link), CUPED (Explicit link), ab-testing (Thematic link)
  • CUPED : A/A test (Explicit link), Correlation and causation (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), p-value (Explicit link), Statistical power (Explicit link), Minimum detectable effect (MDE) (Explicit link), Causal / Uplift — CUPED and Individual Treatment Effects (Explicit link), Uplift modelling (Explicit link), ab-testing (Thematic link)
  • Minimum detectable effect (MDE) : CUPED (Explicit link), Statistical power (Explicit link), p-value (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), Uplift modelling (Explicit link), ab-testing (Thematic link)
  • p-value : A/A test (Explicit link), CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), Statistical power (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), ab-testing (Thematic link)
  • Sample Ratio Mismatch (SRM) : A/A test (Explicit link), CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), p-value (Explicit link), ab-testing (Thematic link)
  • Statistical power : CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), p-value (Explicit link), A/A test (Explicit link), ab-testing (Thematic link)
  • Uplift modelling : Correlation and causation (Explicit link), CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), Causal / Uplift — CUPED and Individual Treatment Effects (Explicit link), ab-testing (Thematic link)
  • ab-testing : A/A test (Thematic link), Correlation and causation (Thematic link), CUPED (Thematic link), Minimum detectable effect (MDE) (Thematic link), p-value (Thematic link), Sample Ratio Mismatch (SRM) (Thematic link), Statistical power (Thematic link), Uplift modelling (Thematic link)

Terms, articles and projects connected to this concept. Tap or click opens a node.

← All terms