Systems are not their parts
You can improve every stage of a pipeline and end up with a slower pipeline. It happens often enough that it should be the default expectation rather than a surprise.
A voice agent is the clean example. Faster transcription that emits partial results more eagerly makes the language model start on incomplete input, which makes it revise, which makes the speech synthesis restart. Three components got better. The conversation got worse. Nothing in any single component’s metrics shows this.
Interactions are where the behaviour lives
The properties people actually care about — feels fast, feels reliable, feels trustworthy — are not owned by any component. They emerge from timing between components, from what happens under load, from what each part does when its neighbour misbehaves.
Which is why component-level dashboards can all be green during an outage everyone can feel. Each part is meeting its contract. The contract was the wrong thing to specify.
The useful move is to instrument the path, not the pieces. One trace that follows a single request end to end tells you more than nine healthy services.
Leverage points
Not all interventions are equal. Changing a parameter is weak. Changing a feedback loop is strong. Changing the goal the system optimises for is stronger still, and usually the one nobody has authority to touch.
The corollary is that most of the effort in a system goes into the weakest tier, because that’s the tier with tickets.
Retry storms have the same shape without the timing subtlety. A service slows; every client retries three times on timeout; the service now takes triple the load precisely when it has least capacity, and a slowdown becomes an outage. Every client is behaving exactly as specified, and the retry policy was added as a reliability feature. What failed is a property of the whole — no component owns it, and no component’s dashboard shows it.
Where it breaks
Systems thinking has a failure mode that is arguably worse than naive component-optimising: it produces people who can explain everything and fix nothing.
Once you’re fluent in feedback loops and emergence, every problem can be framed as systemic, which means no specific change is ever obviously correct — there might be a second-order effect. This is genuinely true and completely paralysing. Meanwhile the person who just made the slow query fast has shipped.
There’s also a diagnostic version of the same trap. “It’s an emergent property of the system” is unfalsifiable and blame-free, which makes it the most comfortable possible conclusion for a post-mortem. Sometimes the system is fine and a specific person merged a specific bad change.
The split I’d draw: use systems thinking to choose where to intervene, then be completely reductionist about the intervention itself. Map the interactions, find the loop, then go fix one thing and measure. The map is not the work.
Takeaway: map interactions before optimising parts.