p-value

On this page
  1. Definition
  2. How to compute
  3. Pitfalls

Definition

A p-value is the probability of observing a test statistic at least as extreme as the one obtained, assuming the null hypothesis is true. It is not the probability that the effect is random, nor P(H0∣data)P(H_0 \mid \text{data}). A small p only shows the data are hard to reconcile with the null; it says nothing about the size of the effect.

How to compute

Compute it from the test statistic and its null distribution: p=P(T≥tobs∣H0)p = P(T \ge t_{\text{obs}} \mid H_0). Analytically for t- and z-tests, or via a permutation test and the bootstrap when the distribution is unknown. The two-sided version doubles the tail.

Pitfalls

A p-value does not replace a confidence interval or an effect size: a significant effect can be trivial. Multiple testing inflates the false-discovery share, so apply a correction (BH, Holm). Peeking and stopping on significance inflate the Type I error. A p-value does not prove a hypothesis, it merely fails to reject it.

Related terms

Mentioned in

Connections map

Interactive subgraph: 11 nodes and 38 links around this page. Each node is keyboard-focusable; press Enter or Space to open its page. A full list of nodes and their links follows the graphic.

  • A/B Testing Methodology Toolkit : Confidence interval (Explicit link), A/A test (Explicit link), Minimum detectable effect (MDE) (Explicit link), p-value (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), Statistical power (Explicit link)
  • A/A test : Sample Ratio Mismatch (SRM) (Explicit link), p-value (Explicit link), CUPED (Explicit link), A/B Testing Methodology Toolkit (Explicit link), A/B test (Explicit link), Statistical power (Explicit link), ab-testing (Thematic link)
  • A/B test : p-value (Explicit link), Statistical power (Explicit link), A/A test (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), Minimum detectable effect (MDE) (Explicit link), Confidence interval (Explicit link), ab-testing (Thematic link)
  • Bootstrap : p-value (Explicit link), Confidence interval (Explicit link), ab-testing (Thematic link)
  • Confidence interval : A/B Testing Methodology Toolkit (Explicit link), p-value (Explicit link), Bootstrap (Explicit link), Minimum detectable effect (MDE) (Explicit link), A/B test (Explicit link), ab-testing (Thematic link)
  • CUPED : A/A test (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), p-value (Explicit link), Statistical power (Explicit link), Minimum detectable effect (MDE) (Explicit link), ab-testing (Thematic link)
  • Minimum detectable effect (MDE) : A/B test (Explicit link), Confidence interval (Explicit link), CUPED (Explicit link), Statistical power (Explicit link), p-value (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • p-value : A/A test (Explicit link), A/B test (Explicit link), Bootstrap (Explicit link), Confidence interval (Explicit link), CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), Statistical power (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • Sample Ratio Mismatch (SRM) : A/A test (Explicit link), A/B test (Explicit link), CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), p-value (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • Statistical power : A/B test (Explicit link), CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), p-value (Explicit link), A/A test (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • ab-testing : A/A test (Thematic link), A/B test (Thematic link), Bootstrap (Thematic link), Confidence interval (Thematic link), CUPED (Thematic link), Minimum detectable effect (MDE) (Thematic link), p-value (Thematic link), Sample Ratio Mismatch (SRM) (Thematic link), Statistical power (Thematic link)

Terms, articles and projects connected to this concept. Tap or click opens a node.

← All terms