Profile first, fix second

The system was a real-time ad insertion platform. At any given moment it was processing thousands of jobs per ad break inside a strict latency window, no room to miss, and it needed around 122 worker servers just to keep up at normal load. More for the big events.

That’s a lot of boxes. And we knew the throughput wasn’t where it should be. What we didn’t know, exactly, was why.

So we profiled it.

What profiling actually looks like

The temptation, when a system is slower than it should be, is to start theorizing. You stare at the code long enough and you start seeing suspects: that loop looks wasteful, that library probably has overhead, maybe if we swap this out for something faster. Sometimes you’re right. Often you’re not, and a month later you’ve made a dozen changes, the system is four percent faster, and you have no idea which change did it or why.

The alternative is boring and methodical. Pick one thing. Measure it. Look at what the measurement actually shows, not what you expected it to show. Fix the thing it points at. Measure again. Write down the gain. Move to the next thing.

That’s it. That’s the whole discipline.

Eight months, one at a time

We ran this for about eight months, from early 2019 through the end of the year. Every improvement measured, dated, and documented.

The first one was connection overhead. The system was opening a fresh TCP connection to an internal microservice for every single job, and at the volume we were running, the churn was real. Switching to persistent connections bought 15% throughput without touching a line of business logic. Small change, clean measurement, move on.

The task serializer was next. Profiling the layer between the queue and the workers showed the serializer itself was the bottleneck, not the work it was serializing. Optimizing it gave 130%. That one surprised us. It looked like infrastructure glue, the kind of code nobody suspects; it turned out to be where most of the time was going.

The threading model was costing us in a quiet, invisible way. When a job hit its timeout, the thread didn’t actually stop. It kept running, burning CPU, contending with everything else. Switching to AsyncIO, which has a clean mechanism for cancelling concurrent work once it times out, gave 127%. Not because async is magically faster, but because in this case the threading model was structurally mismatched with how the system needed to handle timeouts.

Fallback processing was too eager, doing work that got thrown away in certain scenarios. Tuning it to run on actual conditions rather than always-on gave 52%.

Later in the year, a debug serialization step that fed an analytics pipeline turned out to lean on a library with real overhead. Pre-serializing the data before handing it off gave 118%.

Each of these was its own analysis cycle. Profile, identify, fix exactly that one thing, measure.

The lesson isn’t the numbers

The point of listing the percentages isn’t to brag about one brilliant optimization. It’s that none of these wins were guessable. You could not look at that codebase and predict the serializer was a 130% opportunity, or that the debug pipeline had 118% hiding in it. Profiling told us. Guessing would have found something eventually, maybe…but not this.

The other thing worth saying: every change was surgical. We didn’t refactor the system, we didn’t redesign the architecture, we didn’t swap the language out from under it. We fixed exactly what the profile showed and nothing else. That kept the gains attributable (we knew where each one came from) and the risk on any single change bounded.

Compound enough targeted improvements and you need materially fewer boxes to carry the same load. That’s the cost win, and it is a real one. But the method doesn’t care what you’re optimizing for. Whether you’re chasing an infrastructure bill, hitting a latency target, or just trying to get a system to stop being the thing that catches fire every week, it’s the same loop.

Profile first. Fix what it shows. Measure after. Move to the next one.