Minimum detectable effect (MDE)

On this page
  1. Definition
  2. How to compute
  3. Pitfalls

Definition

The MDE is the smallest effect size a test will detect with a given power, at significance level α\alpha and the available sample. It is a property of the design’s sensitivity, not a forecast: the MDE states which effects the test can distinguish from noise at all.

How to compute

Solving the power formula for the effect in a two-arm test: Δ=(z1−α/2+z1−β)2σ2/n\Delta = (z_{1-\alpha/2} + z_{1-\beta}) \sqrt{2\sigma^2 / n}. The value is inversely proportional to the square root of the sample: to catch half the effect you need four times the observations. Estimate σ2\sigma^2 from historical data on the same metric.

Pitfalls

The MDE depends on variance, so CUPED lowers it directly. Never confuse the MDE with the expected effect: a small MDE means high sensitivity, not that an effect exists. Computing the MDE on a peeked sample is invalid. Ratio metrics need a correct standard error, or the MDE comes out too optimistic.

Related terms

Mentioned in

Connections map

Interactive subgraph: 11 nodes and 36 links around this page. Each node is keyboard-focusable; press Enter or Space to open its page. A full list of nodes and their links follows the graphic.

  • A/B Testing Methodology Toolkit : Confidence interval (Explicit link), Sample size (Explicit link), Minimum detectable effect (MDE) (Explicit link), p-value (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), Statistical power (Explicit link)
  • A/B test : p-value (Explicit link), Statistical power (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), Minimum detectable effect (MDE) (Explicit link), Confidence interval (Explicit link), Sample size (Explicit link), ab-testing (Thematic link)
  • Confidence interval : A/B Testing Methodology Toolkit (Explicit link), p-value (Explicit link), Minimum detectable effect (MDE) (Explicit link), A/B test (Explicit link), ab-testing (Thematic link)
  • CUPED : Sample Ratio Mismatch (SRM) (Explicit link), p-value (Explicit link), Statistical power (Explicit link), Minimum detectable effect (MDE) (Explicit link), Uplift modelling (Explicit link), ab-testing (Thematic link)
  • Minimum detectable effect (MDE) : A/B test (Explicit link), Confidence interval (Explicit link), CUPED (Explicit link), Statistical power (Explicit link), p-value (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), A/B Testing Methodology Toolkit (Explicit link), Sample size (Explicit link), Uplift modelling (Explicit link), ab-testing (Thematic link)
  • p-value : A/B test (Explicit link), Confidence interval (Explicit link), CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), Statistical power (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • Sample size : A/B Testing Methodology Toolkit (Explicit link), Statistical power (Explicit link), Minimum detectable effect (MDE) (Explicit link), A/B test (Explicit link), ab-testing (Thematic link)
  • Sample Ratio Mismatch (SRM) : A/B test (Explicit link), CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), p-value (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • Statistical power : A/B test (Explicit link), CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), p-value (Explicit link), Sample size (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • Uplift modelling : CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), ab-testing (Thematic link)
  • ab-testing : A/B test (Thematic link), Confidence interval (Thematic link), CUPED (Thematic link), Minimum detectable effect (MDE) (Thematic link), p-value (Thematic link), Sample size (Thematic link), Sample Ratio Mismatch (SRM) (Thematic link), Statistical power (Thematic link), Uplift modelling (Thematic link)

Terms, articles and projects connected to this concept. Tap or click opens a node.

← All terms