Bayesian Product Discovery

Vanity metrics feel like control. They don’t make decisions easier.

This tool is for disciplined mind-changing: start with an outside-view belief, then update it when a real test comes back.

Each row below is one test. Say what happened, then answer two questions about that signal: how often you would see it if the bet were working, and how often you would see it anyway if the bet were failing. The gap between those two numbers is all the evidence you actually have.

The bet In plain English, what are you claiming will happen?

Step 1 — Starting belief 0.2 Before any evidence: out of comparable projects you have seen, what share actually worked? Use that base rate, not your enthusiasm for this one. Step 2 — Decision threshold 0.5 We commit if the belief ends up above this. Set it now, before you look at any evidence — deciding the bar afterwards is how a model becomes a rationalisation.

Step 3 — Signals (tests) New signals start at 0.5 / 0.5 — worth nothing until you argue otherwise

Starting belief

0.20

Belief after evidence

—

Decision threshold

0.50

Verdict

—

Step 4 — How the belief moved

Step	What happened	If it works, we’d see it	If it fails, we’d see it anyway	Weight of evidence	Belief after

Weight of evidence is the ratio of those two columns. Above 1 the belief rises; below 1 it falls; at roughly 1 the signal told you nothing, however good it felt. That is the working definition of a vanity metric.

How the update works

Each test takes the belief you walked in with and re-weighs it by how much better the signal predicts success than failure. The result becomes the starting belief for the next test.

belief after = (belief × chance if it works) ÷ (belief × chance if it works + (1 − belief) × chance if it fails)