Quick Commands

# edit + deploy
git status
git add -A
git commit -m "docs: update"
git push

# rebuild static blog output (local)
cd site
npm ci --no-audit --no-fund
npm run build

# VPS: pull only
# (on server)
git pull --ff-only

Runbook – Vault mTLS & proxy setup (as of 2025-09-29)

This document summarizes the exact steps and examples we executed, plus short explanations of how everything fits together and which script to run for which use case.

1) Architecture & terms (short)

• Vault (server): runs with PKI mount pki-test and auth methods like auth/cert (mTLS) and approle.
• Intermediate CA vs. chain:
  • Intermediate = from Vault pki-test/cert/ca.
  • Chain = intermediate + root (e.g. for NGINX as current-ca-chain.pem).
  • For Vault TLS verification, it worked best to use the full chain (otherwise you saw “x509: unknown authority”).
• Agents:
  • App agent (e.g. user nctest) fetches leaf certificates (role nginx-nctest) via mTLS.
  • Proxy CA agent (user proxytest) updates only the CA chain for NGINX (no leaf).
• Two agent units on proxytest:
  • vault-agent-proxytest-ca.service (new, CA chain only).
  • vault-agent-proxytest.service (old, had TLS verify errors) → disable the old one (see troubleshooting).

Standard paths per user:

$HOME/vault/mtls/agent.{crt,key}     # mTLS client cert/key (for auth/cert)
$HOME/vault/ca/ca.pem                # CA file (yours initially was just “Intermediate”)
$HOME/nginx/ca/current-ca-chain.pem  # NGINX-Chain (Intermediate + Root)

2) Scripts and what they do

1. setup-vault-agent-mtls-client-config-v4.5.sh
   Provisions an mTLS client certificate for a Linux user (e.g. nctest, proxytest) including auth/cert mapping + policy.
   Result: $HOME/vault/mtls/agent.crt|key, $HOME/vault/ca/ca.pem, mapping under auth/cert/certs/<mapping>.

2. setup-vault-agent-app-config-v4.3.sh
   Builds the app agent config (leaf issuance, templating, post-hook) using cert auth (mTLS).

3. setup-vault-agent-proxy-config-v4.6.sh
   Builds the proxy CA agent config (reads intermediate from Vault, writes a chain for NGINX, starts a user unit).
   Result: ~/.vault-agent-<app>-ca/... and vault-agent-<app>-ca.service running.

4. distribute_ca_to_agents_v3.sh
   Copies root / intermediate / chain to one or more users (from Vault or locally) to standardize trust material.

5. vault-tls-check.sh
   One-shot TLS/mTLS verification against Vault (or any TLS server). Great to validate CA/mTLS files without root.

   Note: fix a small typo in the script (end → fi).

3) Playbooks (exactly how we did it)

A) App nctest – mTLS + app agent

1. Create the mTLS client cert for nctest
   (Policy/Mapping: pki-issue-nctest, Mapping-Name agent-nctest)

   sudo -E ./setup-vault-agent-mtls-client-config-v4.5.sh \
     --env test --config ./config/apps.yaml --app nctest \
     --cn "agent-nctest.test.privsec.ch" \
     --mapping-name "agent-nctest" \
     --mapping-policy "pki-issue-nctest"
   
   

   Result includes: /home/nctest/vault/mtls/agent.crt|key, /home/nctest/vault/ca/ca.pem

2. Configure the app agent (leaf issuer, auth=cert)

   sudo -E ./setup-vault-agent-app-config-v4.3.sh \
     --env test --config ./config/apps.yaml --app nctest --auth cert
   
   

3. Check (as nctest, without root)

   • Live TLS:

     ./vault-tls-check.sh --addr 127.0.0.1:22300 --sni vault.test.privsec.ch \
       --cafile "$HOME/vault/ca/ca.pem"
     
     

   • If the server requires mTLS:

     ./vault-tls-check.sh --addr 127.0.0.1:22300 --sni vault.test.privsec.ch \
       --cafile "$HOME/vault/ca/ca.pem" \
       --mtls-cert "$HOME/vault/mtls/agent.crt" \
       --mtls-key  "$HOME/vault/mtls/agent.key"
     
     

B) Proxy proxytest – mTLS client + CA-chain agent

1. Create the mTLS client cert for proxytest
   (Policy/Mapping: pki-ca-read-proxytest, Mapping-Name proxytest-test-vault)

   sudo -E ./setup-vault-agent-mtls-client-config-v4.5.sh \
     --env test --app proxytest \
     --mapping-policy "pki-ca-read-proxytest"
   
   

2. Set up the proxy CA agent (writes the chain for NGINX)

   sudo -E ./setup-vault-agent-proxy-config-v4.6.sh \
     --env test --app proxytest
   
   

   Ausgabe (bei dir):
   Root CA gespiegelt → /home/proxytest/vault/ca/ca.pem
CA chain present: /home/proxytest/nginx/ca/current-ca-chain.pem

3. TLS check (as proxytest, without root)

   • With chain (recommended):

     ./vault-tls-check.sh --addr 127.0.0.1:22300 --sni vault.test.privsec.ch \
       --cafile "$HOME/nginx/ca/current-ca-chain.pem"
     
     

   • Public IP:

     ./vault-tls-check.sh --addr <public-ip>:22300 --sni vault.test.privsec.ch \
       --cafile "$HOME/nginx/ca/current-ca-chain.pem"
     
     

   • (Optional) With mTLS:

     ./vault-tls-check.sh --addr 127.0.0.1:22300 --sni vault.test.privsec.ch \
       --cafile "$HOME/nginx/ca/current-ca-chain.pem" \
       --mtls-cert "$HOME/vault/mtls/agent.crt" \
       --mtls-key  "$HOME/vault/mtls/agent.key"
     
     

C) CA an mehrere User verteilen (optional)

• Intermediate aus Vault an App-User pushen:

  sudo -E ./distribute_ca_to_agents_v3.sh \
    --env test --config ./config/apps.yaml \
    --which intermediate --users "nctest apptest"
  
  

• Chain for **proxytest** (NGINX):

  sudo -E ./distribute_ca_to_agents_v3.sh \
    --env test --config ./config/apps.yaml \
    --which chain --users proxytest \
    --dest "/home/proxytest/nginx/ca/current-ca-chain.pem"
  
  

4) Day-2 operations

• Check user services (as the respective user):

  systemctl --user --state=running
  systemctl --user status vault-agent-<app>-ca.service
  journalctl --user -u vault-agent-<app>-ca.service -e -n 60
  
  

• (Proxy only) Apply updated chain: the post-hook triggers an NGINX reload (containers with label tls=true).

5) Troubleshooting (quick reference)

6) Recommended settings (stability)

• Agent HCL ca_cert: for the proxy agent, prefer the chain (same file that makes your OpenSSL checks succeed). Example:

  vault {
    address = "https://127.0.0.1:22300"
    ca_cert = "/home/proxytest/nginx/ca/current-ca-chain.pem"
    # optional: client_cert / client_key if mTLS to Vault is enforced
  }
  
  

• File permissions:
  • agent.key 0600, agent.crt/ca.pem 0644 (that’s what the setup scripts write).

• Always set SNI correctly (checks, NGINX upstreams, etc.).

7) Exact commands from the session (copy/paste)

nctest (App, mTLS + App-Agent):

sudo -E ./setup-vault-agent-mtls-client-config-v4.5.sh \
  --env test --config ./config/apps.yaml --app nctest \
  --cn "agent-nctest.test.privsec.ch" \
  --mapping-name "agent-nctest" \
  --mapping-policy "pki-issue-nctest"

sudo -E ./setup-vault-agent-app-config-v4.3.sh \
  --env test --config ./config/apps.yaml --app nctest --auth cert

proxytest (Proxy-CA-Agent):

sudo -E ./setup-vault-agent-mtls-client-config-v4.5.sh \
  --env test --app proxytest \
  --mapping-policy "pki-ca-read-proxytest"

sudo -E ./setup-vault-agent-proxy-config-v4.6.sh \
  --env test --app proxytest

TLS checks (without root):

# proxy user with chain (local)
./vault-tls-check.sh --addr 127.0.0.1:22300 --sni vault.test.privsec.ch \
  --cafile "$HOME/nginx/ca/current-ca-chain.pem"

# public IP
./vault-tls-check.sh --addr <public-ip>:22300 --sni vault.test.privsec.ch \
  --cafile "$HOME/nginx/ca/current-ca-chain.pem"

# app user with standard CA & (optional) mTLS
./vault-tls-check.sh --addr 127.0.0.1:22300 --sni vault.test.privsec.ch
./vault-tls-check.sh --addr 127.0.0.1:22300 --sni vault.test.privsec.ch \
  --cafile "$HOME/vault/ca/ca.pem" \
  --mtls-cert "$HOME/vault/mtls/agent.crt" \
  --mtls-key  "$HOME/vault/mtls/agent.key"

Note (check script): In vault-tls-check.sh at the very bottom, replace the accidental end with **fi** (otherwise Bash errors).

If you want, I can turn this into a one-page mini checklist for rolling out a new app/proxy based on these playbooks.