Statistical power

On this page
  1. Definition
  2. How to compute
  3. Pitfalls

Definition

Power is the probability of rejecting the null hypothesis when the true effect equals a specified size. It is written 1−β1-\beta, where β\beta is the Type II error rate. Power of 0.8 means an effect of the required size will be found in 80% of repetitions; in the rest the test misses it and returns a non-significant result.

How to compute

Power rises with sample size, effect size, and significance level, and falls with variance. For a two-sample mean, approximately: n≈2(z1−α/2+z1−β)2σ2/Δ2n \approx 2 (z_{1-\alpha/2} + z_{1-\beta})^2 \sigma^2 / \Delta^2 per arm. Take the variance from historical data and the effect from the MDE the product deems meaningful.

Pitfalls

Compute power before the experiment: post-hoc power on an observed effect is meaningless and almost always low. Power does not describe the effect you found; it describes the sensitivity of the design. Several metrics in one test lower the power of each, and CUPED restores it by shrinking σ2\sigma^2.

Related terms

Mentioned in

Connections map

Interactive subgraph: 9 nodes and 28 links around this page. Each node is keyboard-focusable; press Enter or Space to open its page. A full list of nodes and their links follows the graphic.

  • A/B Testing Methodology Toolkit : Sample size (Explicit link), A/A test (Explicit link), Minimum detectable effect (MDE) (Explicit link), p-value (Explicit link), Statistical power (Explicit link)
  • A/A test : p-value (Explicit link), CUPED (Explicit link), A/B Testing Methodology Toolkit (Explicit link), A/B test (Explicit link), Statistical power (Explicit link), ab-testing (Thematic link)
  • A/B test : p-value (Explicit link), Statistical power (Explicit link), A/A test (Explicit link), Minimum detectable effect (MDE) (Explicit link), Sample size (Explicit link), ab-testing (Thematic link)
  • CUPED : A/A test (Explicit link), p-value (Explicit link), Statistical power (Explicit link), Minimum detectable effect (MDE) (Explicit link), ab-testing (Thematic link)
  • Minimum detectable effect (MDE) : A/B test (Explicit link), CUPED (Explicit link), Statistical power (Explicit link), p-value (Explicit link), A/B Testing Methodology Toolkit (Explicit link), Sample size (Explicit link), ab-testing (Thematic link)
  • p-value : A/A test (Explicit link), A/B test (Explicit link), CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), Statistical power (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • Sample size : A/B Testing Methodology Toolkit (Explicit link), Statistical power (Explicit link), Minimum detectable effect (MDE) (Explicit link), A/B test (Explicit link), ab-testing (Thematic link)
  • Statistical power : A/B test (Explicit link), CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), p-value (Explicit link), Sample size (Explicit link), A/A test (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • ab-testing : A/A test (Thematic link), A/B test (Thematic link), CUPED (Thematic link), Minimum detectable effect (MDE) (Thematic link), p-value (Thematic link), Sample size (Thematic link), Statistical power (Thematic link)

Terms, articles and projects connected to this concept. Tap or click opens a node.

← All terms