Experiment Lab
Decide with evidence, not attractive creatives
Observed improvement, statistical evidence, causal inference and business impact are four different questions.
Experiment setup
Eney · simulated outcomes you can edit
| Variant | Impressions | Clicks | Trial signups | Paid | Spend $ | Revenue $ | CTR | CVR | CAC | ROAS |
|---|---|---|---|---|---|---|---|---|---|---|
| ControlLess Switching, More Doing | 1.20% | 5.44% | $76 | 0.55× | ||||||
| T1Powerful Assistance. You're in Control. | 1.10% | 7.04% | $56 | 0.75× |
Trial CVR with 95% Wilson intervals
Relative lift
+29.5%
Absolute difference
+1.60 pp
95% CI -0.74 to 3.95 pp
p-value (two-proportion z)
0.178
Required n / arm
13K
Not yet reached
Four different questions
- Observed improvement
Did the treatment's rate come out higher? Yes, in this sample.
- Statistical evidence
Is the gap unlikely under no true difference? Not at the 0.05 level.
- Causal inference
Only valid with randomised assignment and no peeking/leakage. Platform "A/B" splits may not be fully random.
- Business impact
Does it lower CAC or lift LTV? CTR wins can still attract weaker customers.
Decision panel
Transparent rule: data quality → sample size → significance → CAC vs. target
Each arm needs ≈12,981 clicks to detect a 15% relative change. Current evidence is inconclusive (p = 0.178).
Human approval required
No budget changes are executed automatically. This records a reviewed decision only.
Attribution vs. incrementality
Attribution: who gets credit?
Same 830 synthetic conversions, five rules
Observational: attribution redistributes credit; it can't tell you which conversions would have happened anyway.
Incrementality: randomised holdout
Exposed vs. randomly withheld audience
Exposed CVR
1.40%
Holdout CVR
1.18%
Incremental conv.
198
16% of exposed conv. · p=0.073
Assumes random assignment, no contamination between groups, identical measurement, and a fixed analysis window. Incremental = (exposed CVR − holdout CVR) × exposed users.