Rotate Everything in Production (Part 2): Admin mTLS + Vault Server TLS (and Why These Two Come First)
This part covers the first two rotations from the run:
- Admin client cert (mTLS to Vault)
- Vault server TLS cert (listener cert)
These are the control-plane certs. If they fail, everything downstream is harder to recover.
TL;DR
- Admin client cert first, so you always have a working identity.
- Server TLS second, because every client depends on it.
- Validate with
vault statusand an SNI handshake before moving on.
1) Admin client cert rotation (mTLS)
What it is
The client cert for privileged CLI access. In this run:
- environment PKI mount
- admin client role
- environment-specific admin client CN
- a bounded TTL suitable for operational use
Files written
- admin private key under a root-owned admin TLS directory
- admin client cert in the same directory
- issuing CA / trust material needed by the CLI
Why it must be first
If server TLS changes and something about trust/SNI/mTLS is wrong, you want a fresh, known-good client identity ready before the listener flips.
Environment exports (pattern)
export VAULT_CLIENT_CERT='/root/<admin-tls-dir>/admin.crt'
export VAULT_CLIENT_KEY='/root/<admin-tls-dir>/admin.key'
# VAULT_CACERT / VAULT_TLS_SERVER_NAME come from your env-specific setup
vault status
Operational note: keep the key 0600, and treat this directory as sensitive. If it leaks, it is not “just a cert”.
2) Vault server TLS rotation
What it is
The cert presented by the Vault listener.
- the local Vault HTTPS listener
- an environment-specific TLS server name
- a dedicated service owner
- an environment PKI mount
- a dedicated TLS directory for the Vault service
Files written
server.key(0600)server.crt(0644)fullchain.crt(server + intermediate)ca_chain.pem(intermediate + root)
That last file matters: it contains the trust chain clients need to validate.
Why the chain layout matters
You can have a perfect leaf cert and still break clients if:
- the server presents an incomplete chain
- the client trusts the wrong CA bundle
- SNI does not match what clients validate
Your clients should set the TLS server name explicitly if they do not connect via the same hostname that appears in the certificate.
Two checks right after server TLS changes
Check A: Vault auth path works
Using the admin exports above:
vault status
If this works, you validated: client auth + server TLS + CA trust + routing.
Check B: TLS handshake works (SNI explicitly)
Use an OpenSSL client check against the local Vault listener with the expected server name.
If this fails with resets or handshake errors, the problem is below Vault auth (listener, certs, chain, process not ready).
Common gotchas in this setup
- File ownership and permissions: use the dedicated Vault service user and
0600for keys. - SNI mismatch: tests without
-servernamecan mislead you. - Restart timing: rotating files does nothing until the process reloads (Part 4).
Next up
Part 3 goes downstream:
- agent mTLS issuance + cert-auth mappings
- app leaf issuance (nginx roles, domain constraints)
- vault-agent restarts (and the version mismatch message)