Notes on buildingXXIII
Small, honest experiments
An experiment without a stated hypothesis isn’t an experiment. It’s a change, followed by a search for a number that improved.
The discipline is entirely in the order. Hypothesis, then metric, then the smallest test that could move it — written down before the work, where they can embarrass you afterwards.
Write the losing condition
The clause people skip is the one that matters: what result would make you revert.
Without it, every outcome gets absorbed. Up means it worked. Flat means the test was underpowered. Down means something else interfered. All three are available after the fact, and all three feel reasonable, which is why they get used.
Deciding in advance costs nothing and is the only thing standing between an experiment and a rationalisation.
Smallest viable test
Most questions don’t need an A/B test. They need a fake door, five user calls, a manually operated version for a week, or a change shipped to one account with the outcome watched by hand.
A/B tests are expensive in traffic and slow at low volume — below a certain scale they cannot resolve anything you’d act on, and running them anyway produces confident conclusions from noise. Small products should be running qualitative tests almost exclusively, and often aren’t, because a significance number looks more serious in a deck.
The cheapest useful test is usually doing the job by hand. Before building routing logic, route it manually for a week and find out whether the outcome anyone promised actually shows up. Before building a dashboard, mail the numbers round and see if a single person reads them. A fake door — a page for a plan that doesn’t exist yet — answers a demand question in a day. None of these produce a p-value, and all of them beat a test you haven’t the traffic to run.
Where it breaks
Experimentation optimises what it can measure over a horizon it can observe, which makes it structurally conservative.
It reliably finds local improvements — the better button, the shorter form — and reliably rejects anything whose value appears after the measurement window. Nothing that takes three months to matter survives a two-week test. Teams that run entirely on experiments drift toward a product that is well-optimised and directionless, and each individual decision along the way was evidence-based.
There’s a quieter version of the problem. Run enough tests and some win by chance; ship those, and you accumulate a product built partly on noise, with no mechanism that ever removes them. Re-testing old wins is unglamorous and almost never done.
So experiments answer “is this version better than that one”, which is a narrow question. They do not answer what to build, which direction to go, or what the product is for. Those come from judgement, and dressing judgement in experimental language doesn’t improve it — it just makes it harder to argue with.
Takeaway: test assumptions before scaling.