Standing up Vault is the easy part

Standing up HashiCorp Vault takes an afternoon. There’s a demo, a token comes back, everyone nods, and you go home feeling like you solved secrets management. Then you put it in the path of every deploy and every running service, and that afternoon turns into the next several years. That second part is the actual job, and nobody puts it in the screenshot.

My team owned and operated self-hosted Vault on Kubernetes for a globally distributed platform of thousands of servers…mostly cattle, but everyone has their favorite pet. The ask was easy to say and hard to do: machine identity without static secrets, and one trust model that held across CI/CD and every workload. Here’s what that taught me that the quickstart left out.

Static secrets don’t get decided, they accumulate

Nobody sits down and chooses to scatter long-lived credentials across a platform. It happens one reasonable exception at a time. A pipeline needs to reach a registry, so somebody drops in a token. A service needs a database, so somebody adds an environment variable. Each one is defensible on its own. Put together, they’re a map of every place an attacker would love to be.

You don’t fix that with a nicer vault for the same habit. You fix it by removing the reason the habit exists. We moved pipelines and workloads onto Kubernetes auth, so identity came from what the workload actually was, not from a string it carried around and hoped nobody copied. When the credential is short-lived and tied to identity, there’s nothing durable to leak, and the exceptions stop piling up because there’s nothing left to except.

RBAC is a people problem wearing a config file

Least-privilege reads like a configuration task. In practice it’s a series of slightly awkward conversations. Every team is certain its access is the one holding production together, and most of the time the honest answer is that nobody has looked at it in years. Writing the policy is the small part. Agreeing on who should be able to do what, and then holding that line while teams reorganize and services move, is the work.

We treated least-privilege as the default instead of a someday cleanup project, which mostly meant being the team that asked “wait, why does this need that?” early and often. Not the most popular question when we’re all encouraged to move fast and break things. Some things, like prod, can’t be broken, when TSAs enter the equation. Least-privilege vs “someday” is also the difference between an access model and a pile of grants nobody can explain.

The part nobody warns you about: you’re running a service now

The trap with a secrets platform is treating it like an integration you finish. It’s closer to a service you operate, with all the unglamorous parts that implies. Upgrades that can’t take down the deploy path. Failure modes you’ve rehearsed on purpose instead of discovering at 2am. An audit trail that holds up when somebody actually asks who could do what, and when. We did that boring work, and the patterns we built outlived their first use, spreading across developer tooling and platform access control because they were worth reaching for.

The payoff is boring, and that’s the whole point

There’s no dramatic before-and-after for this one. The win was a single, auditable identity layer for pipelines and workloads, replacing credentials that used to live in a dozen different places, carried across years of platform evolution without becoming the thing everyone quietly routed around. Good identity infrastructure is invisible when it’s working. You measure it by how rarely anyone has to think about it, and by how fast you can answer the who-could-do-what question without a meeting. A SOC 2 and ISO 27001 dual audit landed not long after, and both passed. The most exciting thing about that audit was how boring it was.

That’s the part worth getting right. Everything before it really is just the afternoon.