Quick Commands

# edit + deploy
git status
git add -A
git commit -m "docs: update"
git push

# rebuild static blog output (local)
cd site
npm ci --no-audit --no-fund
npm run build

# VPS: pull only
# (on server)
git pull --ff-only

NGINX Reverse Proxy & Sidecar — Final State (derived strictly from your files)

1) High-level overview

Traffic path (today’s working chain):

[Internet]
   └── HTTPS :443 (gateway on host)
        └── proxy_pass → http://127.0.0.1:7777
             (host loopback → published port on test-proxy container)
             └── [Test-Proxy container]
                  listen 7777
                  proxy_pass → http://10.0.2.2:2001        # host loopback via slirp gateway
                       └── [Sidecar (shares netns with app)]
                            listen 0.0.0.0:2001
                            proxy_pass → http://127.0.0.1:4001
                                 └── [App]
                                      listen 127.0.0.1:4001

Health endpoints present:

  • Test-proxy: GET /__nginx_ok (port 7777)
  • Sidecar: GET /__gw_ok (port 2001)
  • App: GET /__app_ok (port 4001)

Why this works in your rootless, per-user, isolated pods setup:

  • The app stays bound to container loopback (127.0.0.1:4001).
  • The sidecar shares the same network namespace (network_mode: "service:app88"), so it can reach 127.0.0.1:4001 and expose only :2001 outward.
  • The test-proxy runs in slirp4netns with allow_host_loopback=true, so it can call the host loopback via **10.0.2.2**; you point proxy_pass to http://10.0.2.2:2001.
  • The host gateway forwards HTTPS → 127.0.0.1:7777 (test-proxy publish).

No shared docker/podman network between pods, no service-name coupling.

2) Component inventory (only what’s in your files)

A) Main gateway (host NGINX)

  • Files (from proxyuser dump):
    • nginx/nginx.conf (includes conf.d/*.conf and sites-enabled/*.conf)
    • nginx/sites-enabled/00-blackhole.conf
      • Drop HTTP 80 default (444), reject unknown HTTPS default (ssl_reject_handshake).
    • nginx/sites-enabled/test-proxy-gateway.conf
      • server :80 for *.test.privsec.ch → ACME + redirect to HTTPS.
      • server :443 with LE certs at /etc/letsencrypt/live/test.privsec.ch/....
      • proxy_pass → **http://127.0.0.1:7777** (preserves Host, XFwd headers; upgrade ready).
    • nginx/sites-enabled/n1c1t3st.conf (separate host; also proxies to 127.0.0.1:7777, contains optional auth_request & rate limits).
    • nginx/conf.d/*.conf (proxy defaults, logging options, gates map, limits, an example upstream).
    • Certbot:
      • createCertWild.yml (manual DNS challenge for *.test.privsec.ch, test.privsec.ch)
      • letsencrypt/renewal/*.conf (LE renewal configs)

B) Test-proxy (container)

  • Compose (docker-compose.yml in proxytest):
    • image: nginx:stable
    • **network_mode: "slirp4netns:allow_host_loopback=true"**
    • ports: "127.0.0.1:7777:7777"
    • Mounts: nginx/nginx.conf, nginx/sites-enabled, nginx/logs
  • NGINX global (nginx/nginx.conf):
    • single log_format main, error_log ... info
    • proxy header defaults, limit_conn_zone, limit_req_zone
    • include /etc/nginx/sites-enabled/*.conf
  • vHost (nginx/sites-enabled/test-wildcard.conf):
    • listen 7777;
    • server_name ~^nctest[^.]*\.test\.privsec\.ch$ test.privsec.ch;
    • /__nginx_ok returns 200
    • **proxy_pass http://10.0.2.2:2001;** (explicit slirp host-loopback IP)
    • Per-IP connection & rate limiting (limit_conn perip 50; limit_req zone=reqs ...)

C) App + sidecar pod

  • Compose (docker-compose.yml in test):
    • app88:
      • image: nginx:stable-alpine
      • ports: "127.0.0.1:2001:2001" (publish sidecar’s port to host loopback)
      • mounts: nginx.conf (app), html, logs
      • healthcheck: nginx -t
    • app88-gw (sidecar):
      • image: nginx:stable-alpine
      • **network_mode: "service:app88"** (shares netns; sees app’s 127.0.0.1)
      • mounts: gw.conf, logs
  • Sidecar NGINX (gw.conf):
    • listen 0.0.0.0:2001;
    • /__gw_ok returns 200
    • proxy_pass http://127.0.0.1:4001;
  • App NGINX (nginx.conf in test):
    • listen 127.0.0.1:4001;
    • serves /usr/share/nginx/html (+ index)
    • /__app_ok returns 200
    • split access logs + warn/notice error logs

3) Effective directory trees (from your dumps)

# testproxy (containerized test proxy)
.
├─ docker-compose.yml
└─ nginx/
   ├─ nginx.conf
   └─ sites-enabled/
      └─ test-wildcard.conf

# app pod (app + sidecar share netns)
.
├─ docker-compose.yml
├─ nginx.conf           # app (127.0.0.1:4001)
├─ gw.conf              # sidecar (0.0.0.0:2001 → 127.0.0.1:4001)
├─ html/                # static content (mounted)
└─ logs/                # access/error logs (mounted)

# gateway on host
.
├─ nginx/
│  ├─ nginx.conf
│  ├─ conf.d/
│  │  ├─ gates.conf
│  │  ├─ limits.conf
│  │  ├─ logging.conf
│  │  ├─ proxy.conf
│  │  └─ upstreams.conf
│  └─ sites-enabled/
│     ├─ 00-blackhole.conf
│     ├─ n1c1t3st.conf
│     └─ test-proxy-gateway.conf
├─ letsencrypt/
│  └─ renewal/
│     ├─ test.privsec.ch.conf
│     └─ n1c1t3st.privsec.ch.conf
└─ createCertWild.yml

4) Comms diagram (ports & IPs exactly as configured)

Client (any)
  │  HTTPS :443  SNI: nctest.test.privsec.ch
  ▼
[Host: Main NGINX gateway]
  - 80: ACME + redirect
  - 443: TLS (LE cert *.test.privsec.ch)
  - proxy_pass http://127.0.0.1:7777
  │
  ▼
Host loopback 127.0.0.1:7777  (published from testproxy)
  │
  ▼
[Test-Proxy container (slirp4netns)]
  - listen 7777
  - server_name ~^nctest... test.privsec.ch
  - proxy_pass http://10.0.2.2:2001   ← slirp host-loopback
  │
  ▼
Host (as seen from container): 10.0.2.2:2001
  │
  ▼
[Sidecar (shares netns with app)]
  - listen 0.0.0.0:2001
  - proxy_pass http://127.0.0.1:4001
  │
  ▼
[App]
  - listen 127.0.0.1:4001

Health path examples:

  • GET https://nctest.test.privsec.ch/__app_ok → 443→7777→10.0.2.2:2001→127.0.0.1:4001
  • curl -H 'Host: nctest.test.privsec.ch' http://127.0.0.1:7777/__nginx_ok (test-proxy)
  • curl http://127.0.0.1:2001/__gw_ok (sidecar)
  • curl http://127.0.0.1:4001/__app_ok (app, from inside pod)

5) Logging, limits, and headers (as-is)

  • Logging
    • Test-proxy: access.log with main; error_log ... info (will show upstream errors).
    • App pod: split access logs (access.2xx.log, access.4xx.log, access.5xx.log, access.log) and warn/notice error logs.
    • Gateway: multiple formats in conf.d/logging.conf + per-site behaviors.
  • Limits
    • Test-proxy limit_conn perip 50; limit_req zone=reqs burst=50 nodelay;
    • Global zones defined in test-proxy’s nginx/nginx.conf.
    • Gateway has additional limit zones (limits.conf) and optional auth gate (gates.conf) used by n1c1t3st.conf.
  • Proxy headers
    • All three layers set: Host, X-Real-IP, X-Forwarded-For, X-Forwarded-Proto.
    • Upgrade headers enabled globally where applicable.

6) Operational runbooks (from your current state)

  • Restart test-proxy (reload config)
    podman exec -it testproxy nginx -t && podman exec -it testproxy nginx -s reload

  • Recreate test-proxy from compose
    podman-compose down && podman-compose up -d

  • End-to-end quick tests

    curl -sSI http://127.0.0.1:2001/__gw_ok
podman exec -it testproxy sh -lc 'curl -sSI http://10.0.2.2:2001/__app_ok'
curl -i -H 'Host: nctest.test.privsec.ch' http://127.0.0.1:7777/__app_ok
curl -sv --http1.1 https://nctest.test.privsec.ch/__app_ok

7) Decision log (why each choice)

  • Separate pods / users; no shared networks → reduce cross-tenant blast radius; avoid container-to-container DNS coupling.
  • Sidecar pattern (0.0.0.0:2001 → 127.0.0.1:4001) → app remains local to its namespace; only the sidecar is addressable.
  • Publish only host loopback (127.0.0.1:2001, 127.0.0.1:7777) → nothing exposed publicly.
  • Test-proxy on slirp with **allow_host_loopback=true** → in rootless mode, lets the proxy reach host loopback via 10.0.2.2, keeping pods otherwise isolated.
  • Explicit **10.0.2.2** in **proxy_pass** → avoids host.containers.internal resolution ambiguity; matches your working tests.
  • Gateway blackholes for unknown names on :80/:443 → quiets noise and tightens surface.

8) Known constraints & don’ts (from what broke earlier)

  • Do not point test-proxy upstream to 127.0.0.1:2001 (that’s the test-proxy’s own loopback).
  • Do not point upstream to your WAN IP (109.*) — sidecar is never bound there.
  • Rootless + netavark: you cannot publish directly to 10.0.3.1 with -p 10.0.3.1:... (rootlessport: cannot assign requested address).
  • If you switch back to a hostname upstream, add extra_hosts: ["host.containers.internal:10.0.2.2"] on the test-proxy service.

If you want this as a printable Markdown doc, say the word and I’ll format it 1:1 into a clean “Runbook + Architecture” file using exactly the content above.

Post-fix plan (so it stays stable)

Everything is green now. Here’s a short “post-fix” plan so you can add new apps quickly without falling into the 502 trap again.

Recap (so nobody forgets)

  • Root cause: ports: was on the sidecar, but the NetNS belongs to the app service (network_mode: service:APP).
  • Fix: move ports: to the app service (TEST → 127.0.0.1:22288:2001, PROD → 127.0.0.1:23288:2001).
  • Resulting flow: proxymain → proxy{test|prod} → 127.0.0.1:22x88 → sidecar:2001 → app:4001 ✅

Lock it in

  • Commit + tag: commit the working Compose + NGINX files and tag it (e.g. env-host-loopback-v1).
  • Maintain PORTS.md: keep the reserved ranges (TEST 22000–22999, PROD 23000–23999) and document each app.
  • Keep health checks:
    • proxymain/proxy{test|prod}: location = /__nginx_ok { return 200; }
    • sidecar: __gw_ok → app: __app_ok
  • Don’t break ACME: on the :80 server, serve /.well-known/acme-challenge/ first, then redirect to HTTPS.

Guardrails / security

  • Host-only: keep all env proxies on listen 127.0.0.1:77xx; (don’t expose them publicly).
  • Limit loopback ports: optionally firewall-guard it so only lo can access them; if someone binds 0.0.0.0:22x88 by mistake, it still won’t be reachable externally.
  • No-priv-esc / read-only: keep your proxymain hardening (no-new-privileges, read_only, tmpfs).
  • Separate logs: /home/{proxymain,proxytest,proxyprod}/logs/ stays cleanly separated.

Monitoring & quick diagnosis

  • Tail everything (root): sudo bash -lc 'shopt -s nullglob; tail -F -n0 -v /home/{proxymain,proxytest,proxyprod}/logs/*.log'

  • Smoke test (host):

    # TEST
curl -sSI http://127.0.0.1:22288/__gw_ok
curl -sSI http://127.0.0.1:22288/__app_ok
curl -sSI -H 'Host: app88.test.privsec.ch' http://127.0.0.1:7701/__nginx_ok
curl -sSI -H 'Host: app88.test.privsec.ch' http://127.0.0.1:7701/__app_ok

# PROD
curl -sSI http://127.0.0.1:23288/__gw_ok
curl -sSI http://127.0.0.1:23288/__app_ok
curl -sSI -H 'Host: app88.prod.privsec.ch' http://127.0.0.1:7700/__nginx_ok
curl -sSI -H 'Host: app88.prod.privsec.ch' http://127.0.0.1:7700/__app_ok

  • End-to-end (TLS):

    curl -sSI https://app88.test.privsec.ch/__app_ok
curl -sSI https://app88.prod.privsec.ch/__app_ok

Runbook: add a new app in 6 steps

  1. Pick a port (TEST → 22xxx / PROD → 23xxx) and register it in PORTS.md.
  2. Copy the app folder (apps/{test|prod}/appNN) and add your HTML/runtime.
  3. Adjust Compose:
    • App service: ports: ["127.0.0.1:<PORT>:2001"]
    • Sidecar: network_mode: "service:<app-service>", no ports:
  4. proxy{test|prod} vhost: server_name + proxy_pass http://127.0.0.1:<PORT>;
  5. Reload env proxies: podman exec -it proxytest nginx -t && nginx -s reload (same for prod).
  6. Run curls/health checks: host first, then env proxy, then TLS.

Why it works (rule of thumb)

Whoever owns the NetNS publishes the ports.
With network_mode: "service:APP", the NetNS belongs to the APP service → so ports: must live there.