Why I Wanted My Own PKI

Most tutorials use either:

  • completely ad‑hoc, self‑signed certificates created with random OpenSSL commands, or
  • public CAs like Let’s Encrypt, focused on browser TLS.

That’s fine for a public website, but it doesn’t match what I want for my VPS:

  • internal mTLS between services (Vault, proxies, apps),
  • short‑lived client certificates for admins and agents,
  • clear separation between test and prod, and
  • a trust model I control from top to bottom.

So I decided to build my own small PKI:

  • with an offline root CA,
  • one intermediate CA per environment (test, prod),
  • and Vault acting as the online CA that actually issues the daily certificates.

This post is the high‑level overview of that PKI design.

PKI Layers in My Setup

The core idea is a three‑level hierarchy:

[ OFFLINE ROOT CA ]
    │
    ├── signs → [ INTERMEDIATE CA (test) ]  → used by Vault pki-test
    │
    └── signs → [ INTERMEDIATE CA (prod) ]  → used by Vault pki-prod

From each intermediate:
 ├─ Vault server certificates
 ├─ reverse proxy certificates
 ├─ admin client certificates (mTLS)
 └─ agent / app certificates

1. Offline Root CA

Role:

  • The ultimate trust anchor.
  • Used only to sign intermediate CA certificates.
  • Never used to sign „normal“ server or client certificates.

Where it lives:

  • On the VPS under something like:
/root/vault/offline-root/test/root-ca.pem
/root/vault/offline-root/prod/root-ca.pem

(plus the private keys, which are the really sensitive part).

How it is used:

  • Only during rare maintenance windows:
  • when I need to create a new intermediate CA,
  • or rotate an existing intermediate CA.
  • In normal operation, the root key is not used and should ideally be offline (encrypted backup, separate machine, etc.).

2. Intermediate CAs (test & prod)

For each environment, I have an intermediate CA:

  • PrivSec Intermediate CA (test)
  • PrivSec Intermediate CA (prod)

The idea:

  • Vault generates a CSR for the intermediate CA,
  • the offline root CA signs that CSR,
  • then I import the signed intermediate back into Vault.

Conceptually:

Vault (pki-test)     Offline Root
------------------   ---------------------------
generate CSR   ───►  sign CSR → test-ca.pem
import signed  ◄───  hand back cert

Once imported, Vault’s pki-test mount is the intermediate CA:

  • it knows the intermediate cert,
  • it has the corresponding private key,
  • it can issue leaf certificates,
  • it can publish a CRL,
  • and it can be configured with TTL limits and roles.

Same for pki-prod in the prod environment.

3. Vault as the Online CA

With the intermediates in place, Vault becomes the online CA:

  • pki-test issues all test certificates,
  • pki-prod issues all prod certificates.

Examples of what comes out of these mounts:

  • Vault server certs (for vault.test.privsec.ch, vault.prod.privsec.ch)
  • reverse proxy certs (for public endpoints and internal revproxies)
  • admin client certs (like vault-admin-setup)
  • agent/app certs for sidecars and services

Instead of manually running openssl req -new -x509 ... every time, I can:

  • define roles in Vault (vault-server, admin-client, proxy-server, …),
  • call pki-*/issue/<role> to get consistent certs,
  • control TTL, allowed SANs, key usages and more via Vault policy.

Where the PKI Files Live on the VPS

I also wanted the filesystem layout to reflect the PKI layers.

Offline and CA layer (root‑only)

/root/vault/offline-root/test/root-ca.pem      # offline root CA (test)
/root/vault/offline-root/prod/root-ca.pem      # offline root CA (prod)

/root/vault/ca/test-ca.pem                     # signed intermediate CA (test)
/root/vault/ca/prod-ca.pem                     # signed intermediate CA (prod)

Only root should touch these, especially the private keys.
They are not used in day‑to‑day application traffic.

Vault admin TLS

Admin mTLS certs live in a separate place:

/root/vault/tls-admin/test/admin.crt
/root/vault/tls-admin/test/admin.key
/root/vault/tls-admin/test/issuing_ca.pem

/root/vault/tls-admin/prod/admin.crt
/root/vault/tls-admin/prod/admin.key
/root/vault/tls-admin/prod/issuing_ca.pem

These are leaf client certificates issued by pki-test / pki-prod,
not by the offline root.

They’re used by:

  • CLI access to Vault (VAULT_CLIENT_CERT, VAULT_CLIENT_KEY),
  • admin scripts that issue server/agent certificates via Vault.

They have short TTLs and are rotated regularly (at least that’s the plan).

Service TLS per Unix user

The actual runtime services each have their own TLS directories under their home:

/home/vaulttest/tls-test/       # Vault test server cert + chain
/home/vaultprod/tls-prod/       # Vault prod server cert + chain

/home/proxytest/tls/            # client/server certs for test proxy
/home/proxyprod/tls/            # client/server certs for prod proxy

/home/appuser/tls-*/...         # optional: certs for apps / agents

This matches the Unix user layout I described in the previous post:

  • vaulttest, vaultprod → own Vault instances
  • proxytest, proxyprod → revproxies
  • appuser → application containers

Each of them only sees the certs it actually needs.

How I Create the PKI (Conceptual Flow)

I won’t dump every single command here, but the flow is:

1. Create offline root CA (once per environment family)

On a secure machine or on the VPS in a very controlled window:

  • Generate a long‑lived root key + cert:
PrivSec OFFLINE Root CA
  • Store the key encrypted and backed up.
  • Copy only the public root CA (root-ca.pem) where needed.

2. Setup Vault PKI mounts

In each Vault environment (test, prod):

  • Enable PKI:
vault secrets enable -path=pki-test pki
vault secrets tune -max-lease-ttl=8760h pki-test
  • Generate an intermediate CSR:
vault write pki-test/intermediate/generate/internal \
  common_name="PrivSec Intermediate CA (test)" \
  key_type="rsa" key_bits=4096 \
  exportable=true \
  | tee /root/vault/ca/test-intermediate.csr
  • Sign that CSR with the offline root CA:
# on the offline/secure side with openssl
openssl x509 -req \
  -in test-intermediate.csr \
  -CA root-ca.pem -CAkey root-ca.key -CAcreateserial \
  -out test-ca.pem -days 3650 -sha256 -extfile <(…)
  • Import the signed intermediate into Vault:
vault write pki-test/intermediate/set-signed \
  certificate=@/root/vault/ca/test-ca.pem

Do the same for pki-prod.

Now:

  • pki-test is effectively “PrivSec Intermediate CA (test)”,
  • pki-prod is effectively “PrivSec Intermediate CA (prod)”.

3. Define roles for different certificate types

Examples:

  • vault-server role:

  • allowed domains: vault.test.privsec.ch, vault.prod.privsec.ch, localhost

  • allows IP SANs for 127.0.0.1, etc.

  • medium TTL (months).

  • admin-client role:

  • client auth usage only.

  • short TTL (e.g. 720h).

  • used to create vault-admin-setup certs.

  • proxy-server / proxy-client roles:

  • separate roles for public endpoints and mTLS to Vault.

So instead of hand‑crafting certs, I can:

vault write pki-test/issue/vault-server \
 common_name="vault.test.privsec.ch" \
 ip_sans="127.0.0.1,::1" \
 alt_names="vault.test.privsec.ch,localhost" \
 ttl="2160h" \
 > /home/vaulttest/tls-test/vault-test.json

Parse the JSON, write the cert/key/chain into files, reload Vault, done.

First Mistakes and Lessons Learned

Of course, I managed to break things.

1. Admin cert TTL too short

I picked a reasonably short TTL for the admin client certs, something like:

720h  (≈ 30 days)

That’s good for security…
until you forget to renew them and have mTLS required on the Vault listener:

  • TLS handshake fails with tls: expired certificate.
  • Vault never even gets to see your token (root or not).
  • Suddenly you’re locked out of your own cluster, with a valid root token but no valid client cert.

Lesson:

  • Short TTLs are great,
  • but you must have:
  • an explicit rotation plan,
  • monitoring or at least a script that tells you „admin.crt expires in X days“,
  • a maintenance path (e.g. secondary listener) to recover without nuking everything.

2. Mixing PKI layers in my head

At the beginning, I constantly confused:

  • offline root CA vs intermediate CAs vs admin client certs,
  • “why can’t my root token fix TLS?” (because TLS fails before tokens matter),
  • what exactly is stored where.

The filesystem structure helps a lot:

  • /root/vault/offline-root → root CAs
  • /root/vault/ca → intermediate CA certs
  • /root/vault/tls-admin → admin mTLS certs
  • /home/*/tls-* → service certs

Once I stuck to that separation, things got clearer.

Where This Fits in the Bigger Picture

This PKI design is the cryptographic foundation for everything else I want to do on this VPS:

  • Vault test/prod instances with real HTTPS and mTLS.
  • Reverse proxies that authenticate to Vault via client certs.
  • Apps and agents that get short‑lived certificates from Vault.
  • Eventually, more advanced things like mutual TLS between services or mapping this pattern into Kubernetes.

In the next posts, I’ll zoom in on:

  • how I actually issue and rotate the admin client certs,
  • how I monitor certificate expiry,
  • and how the PKI ties into the rootless Podman + multi‑user layout from the previous entry.