A/A test

On this page
  1. Definition
  2. How to compute
  3. Pitfalls

Definition

An A/A test is an experiment where both groups receive the same experience, so the true effect is zero by construction. It does not test the product but the tooling: the correctness of the split, metric computation, standard error estimation, and the pipeline itself.

How to compute

Run it on real traffic before or alongside the main test. The share of “significant” differences at α=0.05\alpha = 0.05 should be around 5%, and p-values should be uniformly distributed. Also inspect SRM and metric stability day by day. For every new metric type an A/A run is worthwhile, at least on historical data.

Pitfalls

An A/A test cannot catch an error specific to a particular variant; it validates only the shared infrastructure. A passed A/A does not guarantee correctness under a real effect. Spending all traffic on it steals power from useful tests. A single A/A run without a SRM check proves little.

Related terms

Mentioned in

Connections map

Interactive subgraph: 8 nodes and 23 links around this page. Each node is keyboard-focusable; press Enter or Space to open its page. A full list of nodes and their links follows the graphic.

  • A/B Testing Methodology Toolkit : A/A test (Explicit link), p-value (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), Statistical power (Explicit link)
  • A/A test : Sample Ratio Mismatch (SRM) (Explicit link), p-value (Explicit link), CUPED (Explicit link), A/B Testing Methodology Toolkit (Explicit link), A/B test (Explicit link), Statistical power (Explicit link), ab-testing (Thematic link)
  • A/B test : p-value (Explicit link), Statistical power (Explicit link), A/A test (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), ab-testing (Thematic link)
  • CUPED : A/A test (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), p-value (Explicit link), Statistical power (Explicit link), ab-testing (Thematic link)
  • p-value : A/A test (Explicit link), A/B test (Explicit link), CUPED (Explicit link), Statistical power (Explicit link), Sample Ratio Mismatch (SRM) (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • Sample Ratio Mismatch (SRM) : A/A test (Explicit link), A/B test (Explicit link), CUPED (Explicit link), p-value (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • Statistical power : A/B test (Explicit link), CUPED (Explicit link), p-value (Explicit link), A/A test (Explicit link), A/B Testing Methodology Toolkit (Explicit link), ab-testing (Thematic link)
  • ab-testing : A/A test (Thematic link), A/B test (Thematic link), CUPED (Thematic link), p-value (Thematic link), Sample Ratio Mismatch (SRM) (Thematic link), Statistical power (Thematic link)

Terms, articles and projects connected to this concept. Tap or click opens a node.

← All terms