Sample Ratio Mismatch (SRM)

On this page
  1. Definition
  2. How to compute
  3. Pitfalls

Definition

Sample Ratio Mismatch is a gap between observed group shares and the intended split. With a planned 50/50, actual 49.7/50.3 on a large sample is not noise but a sign that randomisation, logging, or filtering is broken. SRM poisons the whole test: it distorts both the Type I error and the effect estimate.

How to compute

Use a chi-square goodness-of-fit test: χ2=∑i(Oi−Ei)2/Ei\chi^2 = \sum_i (O_i - E_i)^2 / E_i with k−1k-1 degrees of freedom, where OiO_i are observed and EiE_i expected group sizes. Set a strict threshold, p<0.001p < 0.001: with many daily checks, controlling false alarms matters more than sensitivity. Check the split at the funnel entry, not only on collected metrics.

Pitfalls

SRM is invisible on small samples, so test on full traffic. Bot filtering and survivorship can create or hide a skew. You cannot “fix” SRM by reweighting after the fact: find the cause first, then decide whether the test is usable.

Related terms

Mentioned in

Connections map

Interactive subgraph: 9 nodes and 26 links around this page. Each node is keyboard-focusable; press Enter or Space to open its page. A full list of nodes and their links follows the graphic.

  • A/B Testing Methodology Toolkit : A/A test (Explicit link), Minimum detectable effect (MDE) (Explicit link), p-value (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link)
  • A/A test : Sample Ratio Mismatch (SRM) (Explicit link), p-value (Explicit link), CUPED (Explicit link), A/B Testing Methodology Toolkit (Explicit link), A/B test (Explicit link), ab-testing (Thematic link)
  • A/B test : p-value (Explicit link), A/A test (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), Minimum detectable effect (MDE) (Explicit link), Guardrail metric (Explicit link), ab-testing (Thematic link)
  • CUPED : A/A test (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), p-value (Explicit link), Minimum detectable effect (MDE) (Explicit link), ab-testing (Thematic link)
  • Guardrail metric : A/B test (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), ab-testing (Thematic link)
  • Minimum detectable effect (MDE) : A/B test (Explicit link), CUPED (Explicit link), p-value (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • p-value : A/A test (Explicit link), A/B test (Explicit link), CUPED (Explicit link), Minimum detectable effect (MDE) (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • Sample Ratio Mismatch (SRM) : A/A test (Explicit link), A/B test (Explicit link), CUPED (Explicit link), Guardrail metric (Explicit link), Minimum detectable effect (MDE) (Explicit link), p-value (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • ab-testing : A/A test (Thematic link), A/B test (Thematic link), CUPED (Thematic link), Guardrail metric (Thematic link), Minimum detectable effect (MDE) (Thematic link), p-value (Thematic link), Sample Ratio Mismatch (SRM) (Thematic link)

Terms, articles and projects connected to this concept. Tap or click opens a node.

← All terms