Nikita Boyarkin
← Volta Neobank

Causal · project 23 of 23

A DiD test shows the KYC fix causally lifted M3 retention by +9.09pp (95% CI [+6.21, +11.96]) — flat pre-trends, placebo ≈ 0, max |SMD| 0.157 < 0.2.

Situation
The KYC fix coincided with better retention, but a pre/post t-test is correlation, not causation: it ignores the shared trend
Task
Own the move from correlation to a causal estimate: a difference-in-differences design against a flow the KYC fix did not touch
Action
Treated — in-app KYC, comparison — partner KYC; naive pre/post, DiD, covariate-adjusted DiD; parallel-trends, placebo, SMD gate
Result
M3 retention: DiD ATT +9.09pp (95% CI [+6.21, +11.96]); activation naive +6.29pp vs DiD +4.92pp
Stack
Pythonpandas / NumPySciPy / Statsmodelsscikit-learnMatplotlib / Seabornuv + ruff
Source View on GitHub →
On this page
  1. Situation
  2. Task
  3. Actions
  4. Result
  5. Recommendations
  6. Documentation

Volta — Causal Validation of KYC (DiD)

Situation

Project 3 showed the KYC fix coincided with better retention, but a pre/post Welch t-test is a correlation, not a causal estimate: it ignores the shared monthly trend. This project tests the same claim with a difference-in-differences design against a flow the fix did not touch.

Task

I owned the move from correlation to a causal estimate for the KYC fix: I needed a difference-in-differences design against a flow the fix did not touch, not the earlier before-and-after test.

Actions

  • Design: treated — in-app KYC (subject to the progress bar), comparison — partner KYC (agent-assisted; the fix does not apply), cutoff 2024-09.
  • N = 84,000 (50,400 treated / 33,600 comparison), 24 registration cohorts (2023-01 … 2024-12).
  • Estimates: naive pre/post, 2×2 DiD, and a covariate-adjusted DiD (age, device, pre-activity) with cohort-clustered SE.
  • Diagnostics: parallel trends (pre-period gap slope), a placebo at a fake 2024-01 cutoff, covariate balance (SMD), propensity overlap.
  • The data is synthetic: the generator injects a known ATT (+5.7 / +8.5 / +9.0 pp), so this is a methods demonstration — the estimator must recover the effect.

Result

  • M3 retention: DiD ATT +9.09pp (95% CI [+6.21, +11.96], p<0.001) — the causal estimate matches Project 3 (+9.2pp).
  • Activation: naive +6.29pp → DiD +4.92pp (95% CI [+4.07, +5.77]) — pre/post overstates the effect because it does not net out the shared trend.
  • M1 retention: DiD +7.49pp (95% CI [+5.92, +9.05]).
  • Diagnostics pass: pre-trends flat (p = 0.70 / 0.36 / 0.29), placebo ≈ 0 (every CI covers 0), max |SMD| = 0.157 < 0.2.
  • Recovery: 3/3 95% CIs cover the injected effect — the estimator recovers the truth.

Recommendations

  • Separate the evidence: the randomized A/B (+5.72pp) drives the ship decision; DiD validates observational claims (retention) causally.
  • Label pre/post numbers as correlation: naive activation +6.29pp vs DiD +4.92pp — pre/post overstates.
  • Keep the diagnostic gate (parallel trends + placebo + SMD < 0.2) before publishing any causal claim.

Documentation

Charts

Source: github.com/NikitaBoyarkin/volta-banking — 22 projects; figures recomputed from the repo's own datasets (data/*.csv) via its analysis scripts. Funnel counts from data/volta_funnel_data.csv (10,000 users); A/B, retention, segmentation, churn, RFM, CLV, attribution, anomalies, spend, support, NPS, JTBD, unit economics, premium, KYC deep-dive, referral, assisted CAC, FX sourcing, premium offers, anchor CAC and dormant win-back follow the published project narrative (README + part pages).

Naive pre/post vs DiD by outcome

Three estimates of the KYC-fix effect: naive pre/post, 2×2 DiD, and covariate-adjusted DiD. Pre/post does not net out the shared monthly trend.

0 5 10 Активация — Naive pre/post: 6.29 6.29 M1 retention — Naive pre/post: 7.59 7.59 M3 retention — Naive pre/post: 7.6 7.6 Активация — 2×2 DiD: 4.99 4.99 M1 retention — 2×2 DiD: 7.57 7.57 M3 retention — 2×2 DiD: 9.16 9.16 Активация — Covariate-adjusted DiD: 4.92 4.92 M1 retention — Covariate-adjusted DiD: 7.49 7.49 M3 retention — Covariate-adjusted DiD: 9.09 9.09 Активация M1 retention M3 retention Outcome Effect, pp Naive pre/post 2×2 DiD Covariate-adjusted DiD
Key takeaways
  • M3 retention: DiD ATT +9.09pp vs naive +7.60pp — the causal estimate is higher and confirms Project 3.
  • Activation: naive +6.29pp vs DiD +4.92pp — pre/post overstates the effect by ~1.4pp.

Recovery: DiD recovers the injected effect

Injected ATT vs the DiD estimate: 3/3 95% CIs cover the truth — the estimator is correctly specified.

0 5 10 Активация — Injected (truth): 5.7 5.7 M1 retention — Injected (truth): 8.5 8.5 M3 retention — Injected (truth): 9 9 Активация — Recovered (DiD): 4.92 4.92 M1 retention — Recovered (DiD): 7.49 7.49 M3 retention — Recovered (DiD): 9.09 9.09 Активация M1 retention M3 retention Outcome ATT, pp Injected (truth) Recovered (DiD)
Key takeaways
  • DiD estimates are close to the truth: gaps of 0.78 / 1.01 / 0.09 pp, every CI covers the injected effect.
  • The method is valid: 3/3 CIs cover the truth — the design is fit for real data.

Volta project map

Volta overview →

Core

Extended

Market & Jobs

RAT v2

Causal

  • Volta — Causal Validation of KYC (DiD)