Experiments, A/B Tests, and Real Learning

Experiments turn arguments into evidence.

When to run an A/B test

• You have enough traffic to reach significance
• The change is reversible
• You can clearly define a winning metric

When NOT to A/B test

• Tiny user base , qualitative interviews beat noisy stats
• Strategic, hard-to-reverse decisions
• When the right answer is already obvious

Speed of learning is the only real moat for an early stage startup.