This part covers the first two rotations from the run:

  1. Admin client cert (mTLS to Vault)
  2. Vault server TLS cert (listener cert)

These are the control-plane certs. If they fail, everything downstream is harder to recover.

TL;DR

  • Admin client cert first, so you always have a working identity.
  • Server TLS second, because every client depends on it.
  • Validate with vault status and an SNI handshake before moving on.

1) Admin client cert rotation (mTLS)

What it is

The client cert for privileged CLI access. In this run:

  • mount: pki-prod
  • role: admin-client
  • CN: vault-admin-prod
  • TTL: 720h

Files written

  • /root/vault/tls-admin/prod/admin.key (0600)
  • /root/vault/tls-admin/prod/admin.crt (0644)
  • /root/vault/tls-admin/prod/issuing_ca.pem

Why it must be first

If server TLS changes and something about trust/SNI/mTLS is wrong, you want a fresh, known-good client identity ready before the listener flips.

Environment exports (from the run)

export VAULT_CLIENT_CERT='/root/vault/tls-admin/prod/admin.crt'
export VAULT_CLIENT_KEY='/root/vault/tls-admin/prod/admin.key'
# VAULT_CACERT / VAULT_TLS_SERVER_NAME come from /root/vault/env/prod.env
vault status

Operational note: keep the key 0600, and treat this directory as sensitive. If it leaks, it is not “just a cert”.

2) Vault server TLS rotation

What it is

The cert presented by the Vault listener.

  • VAULT_ADDR=https://127.0.0.1:22400
  • SNI / TLS server name: vault.prod.privsec.ch
  • owner: vaultprod:vaultprod
  • intermediate mount: pki-prod
  • TLS directory: /home/vaultprod/tls-prod

Files written

  • /home/vaultprod/tls-prod/server.key (0600)
  • /home/vaultprod/tls-prod/server.crt (0644)
  • /home/vaultprod/tls-prod/fullchain.crt (server + intermediate)
  • /home/vaultprod/tls-prod/ca_chain.pem (intermediate + root)

That last file matters: it contains the trust chain clients need to validate.

Why the chain layout matters

You can have a perfect leaf cert and still break clients if:

  • the server presents an incomplete chain
  • the client trusts the wrong CA bundle
  • SNI does not match what clients validate (vault.prod.privsec.ch)

Your run set SNI explicitly with VAULT_TLS_SERVER_NAME=vault.prod.privsec.ch.

Two checks right after server TLS changes

Check A: Vault auth path works

Using the admin exports above:

vault status

If this works, you validated: client auth + server TLS + CA trust + routing.

Check B: TLS handshake works (SNI explicitly)

openssl s_client -connect 127.0.0.1:22400 -servername vault.prod.privsec.ch -brief </dev/null

If this fails with resets or handshake errors, the problem is below Vault auth (listener, certs, chain, process not ready).

Common gotchas in this setup

  • File ownership and permissions: use vaultprod:vaultprod and 0600 for keys.
  • SNI mismatch: tests without -servername can mislead you.
  • Restart timing: rotating files does nothing until the process reloads (Part 4).

Next up

Part 3 goes downstream:

  • agent mTLS issuance + cert-auth mappings
  • app leaf issuance (nginx roles, domain constraints)
  • vault-agent restarts (and the version mismatch message)