📚 BusinessExplain 🏠 Home 📊 Analytics
← Back to Skills

Vps Provision

⚙️ DevOps & Infrastructure 23 min read v1.1.0
📋 Contents

VPS Provisioning & Sync

Spin up new Hermes instances on VPSes for parallel work delegation.

Current Fleet

| Alias | IP | Geo | Camofox Port | SSH Key | Role |
|-------|-----|-----|-------------|---------|------|
| hkus2 | 82.21.72.61 | US (NY) | 9377 (local) | default | Primary |
| hkice1 | 82.22.53.61 | US (HK) | 9377 (local) | default | OpenClaw + Hermes |
| hkuk2 | 152.114.193.86 | US (UK?) | 9377 (local) | default | Full Hermes Clone |
| hktr1 / child-vps | 82.26.94.226 | TR (Istanbul) | 9378 (tunnel) | ~/.ssh/id_ed25519 | Secondary |
| pkoc2 / pk-vps | 103.171.122.216 | PK (Islamabad) | 9379 (tunnel) | ~/.ssh/id_pk | PK Browser #1 |
| pkoc3 / pk-vps2 | 103.171.122.217 | PK (Islamabad) | 9380 (tunnel) | ~/.ssh/id_pk3 | PK Browser #2 (default) |

SSH aliases: child-vps/hktr1 → root@82.26.94.226 (id_ed25519), pkoc2/pk-vps → root@103.171.122.216 (id_pk), pkoc3/pk-vps2 → root@103.171.122.217 (id_pk3), hkice1 → root@82.22.53.61 (default key).

HK ICE-1 hosts: OpenClaw (Docker, port 18789, oc.hkice1.s4s.host), Hermes WebUI (port 8788), CamoFox (port 9377). Nginx routes to OpenClaw by default.

Quick Sync to New VPS

hermes-sync  [user]

This script (/usr/local/bin/hermes-sync) does:

  • Installs Hermes if not present

  • Copies .env, config.yaml, auth.json

  • Rsyncs all 107 skills

  • Copies AGENTS.md/CLAUDE.md context

  • Installs Camoufox browser + Xvfb + systemd services

  • Resets CAMOFOX_URL to local (not tunnel)

    • Example: hermes-sync child-vps or hermes-sync 1.2.3.4 root

    Manual Steps (if script fails)

    # 1. SSH key setup
ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519
ssh-copy-id -o StrictHostKeyChecking=no root@

    2. Install Hermes
    
ssh root@ 'curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash'

    3. Sync everything
    
rsync -az --del ~/.hermes/skills/ root@:.hermes/skills/
scp ~/.hermes/.env ~/.hermes/config.yaml root@:.hermes/
scp ~/.hermes/auth.json root@:.hermes/

    4. Camoufox (browser automation)
    
ssh root@ 'cd /usr/local/lib/hermes-agent && source venv/bin/activate && pip install camoufox && python -m camoufox.fetch'

    5. Systemd services for Xvfb + Camofox (see script for service files)

    Using Child VPS

    One-shot LLM queries (preferred over hermes chat)

    llm-ask  "question"
llm-ask groq "What is 7*8?"          # Free, 212 tok/s
llm-ask opencode-go "Explain X"     # Default provider
llm-ask openrouter "Research Y"     # 33+ free models
llm-ask chutes "Analyze Z"          # TEE privacy
llm-ask byteplus-coding "code task" # Coding specialist
    Works on any VPS with /usr/local/bin/llm-ask installed. Auto-loads .env keys. 14 providers available.

    On child VPS: ssh child-vps 'llm-ask groq "question"'

    ⚠️ Do NOT use hermes chat -q for scripted queries — the TUI renderer produces garbled output. Use llm-ask instead.

    Background tasks (SSH + notify_on_complete)

    # Dispatch task to child VPS, get notified when done
terminal(command="ssh child-vps 'llm-ask groq \"question\"'", background=True, notify_on_complete=True)

    Parallel tasks on child VPS

    # Run multiple providers simultaneously on child
terminal(command="ssh child-vps 'llm-ask groq \"task 1\"' & ssh child-vps 'llm-ask opencode-go \"task 2\"' & wait")

    Interactive Hermes session (tmux)

    ssh child-vps 'tmux new-session -d -s worker -x 120 -y 40 "hermes"'
ssh child-vps 'tmux send-keys -t worker "your prompt" Enter'
ssh child-vps 'tmux capture-pane -t worker -p | tail -30'

    Browser via child VPS (different geo/IP)

    ssh child-vps 'child-browse "https://example.com"'
    The child-browse script wraps Camoufox with DISPLAY=:99. Returns page text. For JSON APIs: ssh child-vps 'curl -s https://ipinfo.io/json'

    delegate_task from this Hermes

    Use terminal(command="ssh child-vps 'llm-ask ...'") with background=True for async work. For complex multi-turn tasks, start a tmux session on the child.

    Pitfalls

  • hermes chat -q produces garbled TUI output — always use llm-ask for one-shot scripted queries
  • SSH tunnel port conflicts — stale SSH processes from previous tunnel instances can hold the port, causing "Address already in use" errors that make the tunnel service loop in auto-restart. Fix: ss -tlnp | grep to find the stale PID, kill -9 , then systemctl restart ssh-tunnel-. The journalctl telltale is bind [127.0.0.1]:: Address already in use.
  • Camoufox service: Type=oneshot, RemainAfterExit=yes — the Python launcher forks a Node process then exits. systemctl status shows "active (exited)" which is correct; the browser keeps running.
  • Xvfb must be running before Camofox starts. The systemd dependency After=xvfb.service handles ordering.
  • SSH tunnel persists only while SSH process lives — ssh -f -N -L 9378:localhost:9377 creates a background tunnel, but it dies if the SSH connection drops. Don't rely on tunnels for production use; use direct commands via ssh child-vps instead.
  • Skills sync is destructive mirror — rsync --del means local deletions propagate to child. Intentional to keep fleets identical.
  • CAMOFOX_URL auto-detects locally — don't set it in child's .env unless routing through a tunnel (which you shouldn't for normal use).
  • Nginx auth double-login — if a web app has built-in auth (like Hermes WebUI), do NOT add nginx auth_basic on top. It creates a double-login. Use only the app's built-in auth or only nginx auth, never both.
  • Password preference — user uses "munaf" as the default password for all site auth (basic auth htpasswd, WebUI HERMES_WEBUI_PASSWORD, dashboard access). Applies to all VPSes in the fleet.
  • SSH key for PK VPS — ~/.ssh/id_pk (separate from the ed25519 key used for child-vps).
  • hermes-sync on PK overwrites Node-based Camofox — the sync script tries to install python -m camoufox.server as a systemd service. If the VPS already has the Node-based Camofox from /opt/camofox-browser, the Python service will conflict. After running hermes-sync on a VPS with existing Node Camofox, verify with curl localhost:9377/health that the correct instance is running.
  • Hermes WebUI — deployed at https://webui.hkus2.s4s.host/ (password: munaf), systemd service hermes-webui, internal port 8788.
  • Hermes Dashboard — deployed at https://dash.hkus2.s4s.host/ (basic auth munaf:munaf), internal port 9119, firewalled to localhost only.
  • All sites password-protected — API Hub, LLM Council, Dashboard, WebUI all require auth (munaf:munaf for basic auth, or single password "munaf" for WebUI built-in auth).

    • Multi-Geo Browser Routing

    Persistent SSH tunnels route remote Camofox instances to localhost ports.

    | Port | VPS | Geo | Systemd Service |
    |------|-----|-----|-----------------|
    | 9377 | localhost (US) | New York, US | camofox-server (local) |
    | 9377 | hkice1 | US (HK) | camofox (local to hkice1) |
    | 9378 | child-vps (TR) | Istanbul, Turkey | ssh-tunnel-tr |
    | 9379 | pkoc2 (PK) | Islamabad, Pakistan | ssh-tunnel-pk |
    | 9380 | pkoc3 (PK) | Islamabad, Pakistan | ssh-tunnel-pk2 |

    Switch browser geo

    Set CAMOFOX_URL in ~/.hermes/.env, then /reset:

    # US (auto-detected — no need to set)

    CAMOFOX_URL=http://localhost:9377
    

    Turkey
    
ssh -f -N -L 9378:localhost:9377 -o ServerAliveInterval=60 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes child-vps

    Pakistan (pkoc2)
    

    CAMOFOX_URL=http://localhost:9379
    

    Pakistan (pkoc3) — DEFAULT
    
CAMOFOX_URL=http://localhost:9380

    Create persistent SSH tunnel (systemd)

    For each remote VPS with Camofox:

    # /etc/systemd/system/ssh-tunnel-tr.service
[Unit]
Description=SSH Tunnel to Turkey VPS Camofox (port 9378)
After=network-online.target

    [Service]
    ExecStart=/usr/bin/ssh -N -L 9378:localhost:9377 -o ServerAliveInterval=60 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes child-vps
    Restart=always
    RestartSec=5
    

    [Install]
    WantedBy=multi-user.target

    Then: systemctl daemon-reload && systemctl enable --now ssh-tunnel-tr

    Verify: curl -s http://localhost:9378/health

    Direct API usage (bypass Hermes browser tools)

    When you need to browse from a specific geo without changing CAMOFOX_URL:

    import requests, json, time

    base = "http://localhost:9379"  # PK Camofox via tunnel
    user_id, session_key = "hermes-pk", "pk-session"
    

    Create tab
    
r = requests.post(f"{base}/tabs", json={"userId": user_id, "sessionKey": session_key})
tab_id = r.json()["tabId"]

    Navigate (userId required in navigate/snapshot calls too)
    
requests.post(f"{base}/tabs/{tab_id}/navigate", json={"url": "https://ipinfo.io/json", "userId": user_id})
time.sleep(3)
r = requests.get(f"{base}/tabs/{tab_id}/snapshot", params={"userId": user_id})
print(r.json().get("snapshot", ""))

    Cleanup
    
requests.delete(f"{base}/tabs/{tab_id}", params={"userId": user_id})

    Quick geo-IP verification

    # Via the Camofox API
ssh pkoc2 'curl -s https://ipinfo.io/json'        # PK IP
ssh child-vps 'curl -s https://ipinfo.io/json'     # TR IP

    Via Python API (see above) — returns IP of the browser's VPS

    Hermes Approvals — Auto-Approve

    User preference: all approvals on auto, no confirmation prompts.

    # In ~/.hermes/config.yaml:
approvals:
  mode: auto          # was: manual
  timeout: 60
  cron_mode: auto     # was: deny
  mcp_reload_confirm: false
  destructive_slash_confirm: false

    delegation:
      subagent_auto_approve: true   # was: false

    Full Hermes Clone Deployment (Nginx + SSL + Docker)

    When deploying a complete replica of the gateway (not just a child agent), you need Nginx, SSL, Hindsight (Docker), WA Chat Viewer, MariaDB, and all web content.

    Step-by-step Clone Checklist

  • SSH key setup — ssh-copy-id or manual cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys
  • Install Hermes — Ubuntu 24.04 doesn't have pip3 by default. Use venv:
    •    python3 -m venv /usr/local/lib/hermes-agent/venv
   /usr/local/lib/hermes-agent/venv/bin/pip install hermes-agent
   ln -sf /usr/local/lib/hermes-agent/venv/bin/hermes /usr/local/bin/hermes
  • Copy config — Don't rsync/tar the entire .hermes/ (1.5G+ in plugins alone). Copy only essentials:
    •    scp config.yaml AGENT_PERSONA.md root@NEW:.hermes/
   cd .hermes && tar cf - skills/ | ssh root@NEW "cd .hermes && tar xf -"
    Then sed -i 's/old\.domain/new.domain/g' config.yaml for domain renames.
  • Nginx + SSL — Install nginx + certbot. The critical sequence:
    •    apt-get install -y nginx certbot python3-certbot-nginx
   
   # Generate temp self-signed certs AND dhparam BEFORE starting nginx
   # certbot --nginx fails if nginx can't start (no SSL certs)
   for domain in sub1.new.domain sub2.new.domain ...; do
     mkdir -p /etc/letsencrypt/live/$domain
     openssl req -x509 -nodes -days 1 -newkey rsa:2048 \
       -keyout /etc/letsencrypt/live/$domain/privkey.pem \
       -out /etc/letsencrypt/live/$domain/fullchain.pem \
       -subj '/CN=temp'
   done
   openssl dhparam -out /etc/letsencrypt/ssl-dhparams.pem 2048
   
   # Copy nginx site configs from source, sed domain names
   # Start nginx, THEN get real certs
   nginx -t && systemctl start nginx
   
   # Get ONE SAN cert for ALL subdomains (not per-domain)
   certbot certonly --standalone --non-interactive --agree-tos \
     --email your@email.com \
     -d sub1.new.domain -d sub2.new.domain -d sub3.new.domain ...
   
   # Stop nginx temporarily for standalone mode, then update all
   # site configs to point to the single cert path, then restart
    ⚠️ Pitfall: certbot --nginx fails if /etc/letsencrypt/live// already exists from temp certs. Either rm -rf /etc/letsencrypt/live archive renewal accounts before running, or use --standalone mode (stop nginx first). ⚠️ Pitfall: openssl dhparam can take 30+ seconds. Don't background it — wait for it to finish before starting nginx.
  • Hindsight (Docker) — Needs LLM provider env vars or it crashes:
    • docker run -d --name hindsight --restart unless-stopped \
  -p 9999:9999 -p 9077:8888 \
  -v /root/.hindsight-data:/home/hindsight/.pg0 \
  -e HINDSIGHT_API_KEY=top-secret-key \
  -e HINDSIGHT_API_LLM_PROVIDER=openai \
  -e HINDSIGHT_API_LLM_API_KEY= \
  -e HINDSIGHT_API_LLM_MODEL=kimi-k2.5 \
  -e HINDSIGHT_API_LLM_BASE_URL=https://ollama.com/v1 \
  -e HINDSIGHT_API_HOST=0.0.0.0 \
  -e HINDSIGHT_API_PORT=8888 \
  -e HINDSIGHT_ENABLE_API=true \
  -e HINDSIGHT_ENABLE_CP=true \
  -e HINDSIGHT_CP_ACCESS_KEY=munaf \
  -e HINDSIGHT_CP_DATAPLANE_API_URL=http://localhost:8888 \
  -e HINDSIGHT_API_LOG_LEVEL=info \
  ghcr.io/vectorize-io/hindsight:latest
    ⚠️ CRITICAL: Volume mount is -v /root/.hindsight-data:/home/hindsight/.pg0 — NOT /root/.hermes/hindsight:/app/data. The gateway stores Hindsight data at /root/.hindsight-data/ and the container reads from /home/hindsight/.pg0. Getting this wrong means the clone starts with zero memory.

    ⚠️ Pitfall: Hindsight takes 15-30 seconds to initialize (downloads embedding/reranker models on first run). Check health with curl http://localhost:9077/health — don't assume it's broken just because port isn't responding immediately.

  • Hermes WebUI — Install as systemd service. Needs HERMES_WEBUI_PASSWORD env var. Copy /opt/hermes-webui/ from source (exclude node_modules/.git).
  • WhatsApp Chat Viewer — Python Flask app at /var/www/whatsapp-chat/. Needs flask and flask-cors in the Hermes venv:
    •    /usr/local/lib/hermes-agent/venv/bin/pip install flask flask-cors
  • MariaDB — apt-get install -y mariadb-server for session storage.
  • Web content — Static sites (API Hub, LLM Council, Server4Sale, etc.) just need tar copy to /var/www// and nginx config.
  • Verify all domains — Loop through subdomains checking HTTP status codes:
    •     for domain in apinew dash webui memory llm new seo; do
      curl -sk -o /dev/null -w "%{http_code}" "https://$domain.new.domain/" && echo " $domain"
    done

    SAN Cert vs Per-Domain Certs

    For multi-domain setups, get one SAN cert covering all subdomains rather than separate certs per domain:

  • All nginx server_name blocks reference the same ssl_certificate path

  • Easier to manage, single renewal

  • Use certbot certonly --standalone -d a.domain -d b.domain -d c.domain

  • After getting the real cert, update all nginx configs to point to it, then nginx -t && systemctl restart nginx

    • Full DR Clone — Data Transfer Checklist

    When doing a complete DR clone (not just a child agent sync), you need to transfer ALL data directories. Don't assume .hermes/ is enough.

    What to transfer (in order of size)

    | Source Path | Size (typical) | Purpose |
    |-------------|----------------|---------|
    | ~/.hermes/plugins/ | ~1.5 GB | Installed plugin repos |
    | ~/.hindsight-data/ | ~480 MB | Hindsight vector DB (Docker volume source) |
    | /opt/hermes-webui/ | ~240 MB | WebUI application (exclude node_modules) |
    | ~/.hermes/sessions/ | ~140 MB | Session transcripts |
    | ~/.hermes/skills/ | ~55 MB | Agent skills |
    | /var/www/ | varies | API Hub, LLM Council, Server4Sale, etc. |
    | ~/.hermes/whatsapp/ | ~17 MB | WhatsApp bridge data |
    | ~/.hermes/cron/ | ~1 KB | Cron jobs |
    | ~/.hermes/bin/ | varies | Custom scripts (llm-ask, hermes-sync) |

    Transfer method for large dirs

    # Stop Hindsight on SOURCE for clean data copy
ssh root@SOURCE 'docker stop hindsight'

    Create tar of everything missing from standard sync
    
ssh root@SOURCE 'cd /root && tar cf /tmp/hermes-dump.tar \
  .hindsight-data/ \
  .hermes/sessions/ \
  .hermes/webui/ \
  .hermes/plugins/ \
  .hermes/cron/ \
  .hermes/whatsapp/ \
  .hermes/bin/ \
  .hermes/logs/ 2>/dev/null; \
  tar rf /tmp/hermes-dump.tar /opt/hermes-webui/ 2>/dev/null; \
  tar rf /tmp/hermes-dump.tar /var/www/ 2>/dev/null'

    Compress and transfer
    
ssh root@SOURCE 'gzip -f /tmp/hermes-dump.tar'
scp root@SOURCE:/tmp/hermes-dump.tar.gz /tmp/
scp /tmp/hermes-dump.tar.gz root@DEST:/tmp/
ssh root@DEST 'cd / && tar xzf /tmp/hermes-dump.tar.gz'

    Restart Hindsight on SOURCE
    
ssh root@SOURCE 'docker start hindsight'

    Post-clone verification

    # Check Hindsight data integrity — node/doc/link counts must match SOURCE
ssh root@DEST 'curl -s http://localhost:9077/health | jq .'

    Verify all subdomains respond
    
for domain in apinew dash webui memory api-memory llm new seo techmart digitech wa hindsight; do
  code=$(curl -sk -o /dev/null -w "%{http_code}" "https://$domain.DEST_DOMAIN/")
  echo "$domain: $code"
done

    Check .hermes size matches
    
ssh root@SOURCE 'du -sh ~/.hermes/ ~/.hindsight-data/'
ssh root@DEST   'du -sh ~/.hermes/ ~/.hindsight-data/'

    Pitfalls specific to full DR clones

  • hermes-gateway.service: Enable it (systemctl enable hermes-gateway) but do NOT start it until WhatsApp is paired on the new server. Starting without a paired session causes the bridge to fail silently.
  • Hindsight must be stopped during data copy — copying its data while running produces a corrupt or inconsistent snapshot. Stop the container, copy, then restart.
  • WebUI node_modules: When copying /opt/hermes-webui/, exclude node_modules/ and .git/ to save 200+ MB. Run cd /opt/hermes-webui && npm install on the destination instead.
  • Flask for WA Chat Viewer: The WA Chat Viewer needs flask and flask-cors in the Hermes venv: /usr/local/lib/hermes-agent/venv/bin/pip install flask flask-cors
  • Domain rename: After copying config.yaml, run sed -i 's/old\.domain/new.domain/g' to update all domain references.
  • Memory is almost full: After a full clone, memory may be at 80%+ from the process. Don't try to store the entire transfer log — only store durable facts.
  • Hindsight CP_ACCESS_KEY: Always munaf across all fleet nodes.
  • WebUI passwords: Primary gateway uses munaf. Clone nodes use admin (e.g., hkuk2admin for hkuk2).
  • Hindsight env vars: The clone's Hindsight container needs the same LLM env vars as the gateway. Check the gateway's running container with docker inspect hindsight | jq '.[0].Config.Env' to get the exact values.

    • Notes

  • Child VPS config matches parent (same model, same API keys, same 14 providers)
  • Both VPSes have /usr/local/bin/llm-ask and /usr/local/bin/hermes-sync
  • Both VPSes have /usr/local/bin/child-browse for Camoufox quick-browse

    • Advanced Techniques (2025-2026)

    SSH multiplexing (10x speedup)

    Enable SSH multiplexing to reuse a single TCP connection for all SSH/SCP/rsync operations to the same host. This eliminates the TCP handshake and key exchange overhead for every command:

    # Add to ~/.ssh/config
Host child-vps pkoc2 pkoc3 hkice1
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h-%p
  ControlPersist 600

    mkdir -p ~/.ssh/sockets

    With this, the first SSH connection establishes the master socket; subsequent ssh, scp, and rsync commands to the same host reuse it instantly — often 10× faster for short commands like ssh child-vps 'hostname'.

    rsync --checksum for idempotent sync

    The default rsync -az --del uses modification time + size to decide what to transfer. If timestamps drift across VPSes (NTP skew, filesystem differences), rsync may re-copy unchanged files. Use --checksum for truly idempotent syncs:

    rsync -azc --del ~/.hermes/skills/ root@child-vps:.hermes/skills/

    --checksum computes a file-content hash instead of relying on mtime. Slower on large directories but guarantees no unnecessary transfers.

    SSH config wildcards

    Group common settings for the entire fleet with wildcard patterns in ~/.ssh/config:

    Host .s4s.host 10.
  User root
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  ServerAliveInterval 60
  ServerAliveCountMax 3

    Host child-vps
      HostName 82.26.94.226
      IdentityFile ~/.ssh/id_ed25519
    

    Host pkoc2 pk-vps
      HostName 103.171.122.216
      IdentityFile ~/.ssh/id_pk

    This avoids repeating User, StrictHostKeyChecking, and keepalive settings for every host block.

    pdsh/parallel for mass fleet commands

    When you need to run the same command across all VPSes simultaneously:

    # Install pdsh
apt-get install -y pdsh

    Run across the fleet
    
pdsh -w child-vps,pkoc2,pkoc3 'apt-get update && apt-get upgrade -y'

    Or with GNU parallel for more control:

    # Check disk space across fleet
parallel --nonall -j0 'ssh {} df -h /' ::: child-vps pkoc2 pkoc3

    systemd dependency ordering

    When services depend on each other (e.g., Xvfb must be running before Camofox, SSH tunnels before apps that use them), use explicit ordering:

    [Unit]
After=xvfb.service
Wants=xvfb.service
BindsTo=xvfb.service
  • After= — start this service after the listed one
  • Wants= — start the listed unit if it's not already running (soft dependency)
  • BindsTo= — stop this service if the listed unit stops (hard dependency)
  • Before= — declare this service must start before another

    • For SSH tunnel services that might have port conflicts on restart, add a Conflicts= stanza and use ExecStartPre to kill stale processes:

    [Service]
ExecStartPre=/usr/bin/bash -c 'ss -tlnp | grep :9378 && exit 1 || exit 0'
ExecStart=/usr/bin/ssh -N -L 9378:localhost:9377 ...
RestartSec=3

    Edge Cases / Pitfalls

  • SSH tunnel ghost with RestartSec race — If an SSH tunnel service crashes and RestartSec=5 fires, but the old SSH process hasn't fully released the port yet, the new process gets bind: Address already in use. The service enters a crash loop. Mitigation: use ExecStartPre to check port availability, set RestartSec=10 to give more decay time, or use ExitOnForwardFailure=yes (already in the fleet config) so SSH exits immediately instead of hanging with a dead tunnel.
  • rsync --del following symlinks — rsync --del removes destination files that don't exist at the source. If your source directory contains symlinks pointing outside the transfer tree, rsync by default copies the symlink target content (not the symlink itself). Use -l (copy symlinks as symlinks) or --copy-links (follow and copy content) intentionally — mixing them up can cause --del to remove files on the destination that exist under different symlink paths on the source.

    • OpenTofu as Terraform Alternative

    OpenTofu is a CNCF-incubating project and drop-in replacement for Terraform 1.5.x–1.6.x. After HashiCorp's BSL license change, the community forked Terraform into OpenTofu, which is now governed by the Linux Foundation.

    Key advantages over Terraform:

  • State encryption — native encryption of state files at rest (AES-256-GCM), a feature Terraform lacks without paid Terraform Cloud

  • Drop-in compatibility — same HCL syntax, same CLI commands (tofu init/plan/apply), interoperable state files with Terraform 1.5–1.6

  • Enterprise adoption — Fidelity, large cloud providers, and platform teams have migrated production workloads

  • Community-driven — open governance under CNCF, no vendor lock-in

    • Limitations vs. Terraform:

  • No SAML/SSO (Terraform Cloud/Enterprise-only feature)

  • No Terraform Cloud remote operations (state backend works fine with S3/GCS/Azure Blob)

  • Provider/plugin ecosystem identical (shares the same registry for now; diverging over time)

    • Usage on VPS fleet:

    # Install OpenTofu
    curl -fsSL https://get.opentofu.org/install.sh | bash
    

    Drop-in replacement — just alias
    
alias tf=tofu

    Encrypt state at rest
    
tofu init -backend-config="encrypt=true"

    For VPS provisioning, OpenTofu can manage Hetzner/DO/Vultr instances as easily as Terraform but with state encryption built in. Prefer tofu over terraform for all new infrastructure code.

    Caddy as Nginx Alternative

    Caddy is an excellent reverse proxy for simple VPS setups — especially when you need automatic HTTPS with zero configuration.

    Key advantages over Nginx:

  • Automatic HTTPS — provisions and renews Let's Encrypt certificates automatically, zero config needed. No certbot cron jobs, no manual certificate management

  • Single binary — one static binary, no dependencies, apt install caddy or download directly

  • Simpler config — Caddyfile is dramatically simpler than nginx.conf for common patterns:

    •   webui.example.com {
          reverse_proxy localhost:8788
      }

    vs. Nginx's ~40 lines for the same (server block, SSL config, proxy headers, cert paths, HSTS)
  • Performance — benchmarks at 142K req/s (≈22% faster than Nginx for typical reverse proxy workloads)

  • HTTP/3 built-in — QUIC support out of the box, no extra compilation flags

  • JSON config API — runtime config changes via REST API without restarts

    • When to use Caddy vs. Nginx:

  • ✅ Caddy — simple reverse proxies, automatic HTTPS, few upstreams, single-server setups, quick prototyping

  • ✅ Nginx — complex load balancing, rate limiting, custom auth modules, large-scale production (existing expertise), Lua/OpenResty needs

    • Caddy on VPS:

    apt install -y caddy
    

    Minimal Caddyfile for Hermes WebUI
    
cat > /etc/caddy/Caddyfile << 'EOF'
webui.hkus2.s4s.host {
    reverse_proxy localhost:8788
}
dash.hkus2.s4s.host {
    basicauth * {
        munaf $2a$14$HASH_HERE
    }
    reverse_proxy localhost:9119
}
EOF

    systemctl restart caddy

    ⚠️ Caddy auto-HTTPS requires ports 80+443 open and DNS pointing to the server. For internal-only services, use http:// prefix to disable auto-HTTPS: http://internal.local { ... }.

    Systemd Security Hardening Templates

    Apply systemd sandboxing directives to all Hermes-related services (Camofox, Xvfb, SSH tunnels, etc.) to limit blast radius if any service is compromised.

    Recommended hardening directives (add to [Service] section):

    [Service]

    Filesystem isolation
    
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/log/hermes /run/hermes
ProtectHome=read-only

    Capability restrictions
    
NoNewPrivileges=true
PrivateDevices=true
ProtectClock=true
ProtectHostname=true

    Memory / execution restrictions
    
MemoryDenyWriteExecute=true
LockPersonality=true
RestrictRealtime=true
RestrictSUIDSGID=true

    Namespace restrictions
    
RestrictNamespaces=true

    Or granular: RestrictNamespaces=net ipc
    

    System call filtering
    
SystemCallFilter=@system-service
SystemCallFilter=~@mount @keyring @privileged

    Network restrictions (if service doesn't need network)
    

    IPAccounting=yes
    

    IPAddressDeny=any
    

    IPAddressAllow=localhost
    

    Ambient capabilities
    
AmbientCapabilities=
CapabilityBoundingSet=

    Applying to existing fleet services:

    # Check current security score (lower = more hardened)
systemd-analyze security camofox-server.service
systemd-analyze security xvfb.service
systemd-analyze security hermes-webui.service

    Create override for any service (avoids editing packaged unit files)
    
systemctl edit camofox-server.service

    Paste the hardening directives above into the override file
    

    Reload and restart
    
systemctl daemon-reload
systemctl restart camofox-server.service

    Per-service tuning:

  • Camofox/Xvfb — needs X11 socket, shared memory, and network access; use ProtectSystem=full (not strict), PrivateDevices=false (GPU access), exclude @mount from syscall filter

  • SSH tunnels — only need network access; can use maximum hardening including IPAddressDeny=any + IPAddressAllow=localhost for SOCKS-only

  • Hermes WebUI — needs network access and write paths for logs; ProtectSystem=strict with explicit ReadWritePaths=

    • Pitfalls:

  • ProtectSystem=strict blocks writes to /usr, /etc, /boot. Add ReadWritePaths= for any dirs the service needs

  • MemoryDenyWriteExecute=true breaks JIT-compiled runtimes (Node.js, Python's PyGTK). Test services after enabling

  • PrivateDevices=true prevents access to /dev/ devices. Camofox needs /dev/dri/* for GPU — use PrivateDevices=false there

  • Always run systemd-analyze security after changes — aim for score ≤ 5.0

    • Let's Encrypt Certificate Changes

    Let's Encrypt is significantly changing certificate lifetimes and renewal practices. These changes affect all VPS fleet services using auto-HTTPS (Caddy) or certbot.

    Certificate Lifetime Timeline

    | Date | Change | |------|--------| | May 2026 | 45-day certificates available (opt-in) | | June 2025 | Expiration notification emails discontinued — no more renewal reminders | | Feb 2027 | 64-day certificates available | | Feb 2028 | 45-day certificates become default (90-day certs no longer issued) |

    Critical Impacts

  • Expiration emails ended June 2025 — no more email warnings. You must implement your own monitoring (certwatch scripts, Prometheus alerts, or health checks)
  • More frequent renewals — 45-day certs mean renewing every ~30 days instead of every 60. Automate everything
  • ARI (ACME Renewal Information) — new extension that tells your ACME client the optimal window to renew. Prevents thundering-herd renewals and reduces failure risk:
    • # Certbot with ARI support (certbot >= 2.8)
certbot renew --deploy-hook "systemctl reload nginx"

    Caddy uses ARI natively — no action needed
  • DNS-PERSIST-01 challenge (coming 2026) — new challenge type that allows persistent DNS validation without re-creating TXT records every renewal. Reduces DNS API calls and supports smoother automation.

    • Action Items for VPS Fleet

    # 1. Update Certbot (for Nginx-based hosts)
apt update && apt install -y certbot python3-certbot-nginx
certbot --version  # Ensure >= 2.8 for ARI support

    2. Monitor certificate expiry without email
    

    Add to cron on all VPSes:
    
cat > /etc/cron.daily/cert-check << 'SCRIPT'
#!/bin/bash

    Alert if any cert expires in < 14 days
    
for cert in /etc/letsencrypt/live/*/cert.pem; do
    days=$(( ( $(date -d "$(openssl x509 -enddate -noout -in "$cert" | cut -d= -f2)" +%s) - $(date +%s) ) / 86400 ))
    if [ "$days" -lt 14 ]; then
        echo "WARNING: $cert expires in $days days" >&2
    fi
done
SCRIPT
chmod +x /etc/cron.daily/cert-check

    3. For Caddy — already handles ARI and auto-renewal natively
    

    No changes needed; Caddy will automatically use shorter lifetimes when available

    ⚠️ Caddy users — Caddy handles all of this automatically (ARI, renewal, shorter lifetimes). No action needed unless you want custom monitoring. Certbot/Nginx users must update certbot and add expiry monitoring.

    Cloud-init Templates

    Cloud-init is the industry standard for first-boot VM initialization across all major cloud providers (AWS, Azure, GCP, DigitalOcean, Hetzner, Vultr, Linode). It runs on first boot to configure users, packages, files, and commands from a single YAML file.

    Basic VPS Setup Template

    #cloud-config

    Save as: cloud-init.yaml
    

    Usage: Pass as user-data when creating VPS, or:
    

      cloud-init -c cloud-init.yaml (on existing host)
    

    Timezone
    
timezone: UTC

    Locale
    
locale: en_US.UTF-8

    Hostname
    
hostname: hermes-vps
manage_etc_hosts: true

    Users
    
users:
  - name: hermes
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAA... hermes@fleet
    lock_passwd: false
    passwd: "$6$rounds=4096$salt$hash..."  # mkpasswd -m sha512crypt

    - name: root
        ssh_authorized_keys:
          - ssh-ed25519 AAAA... root@fleet
    

    SSH hardening
    
ssh_deletekeys: true
ssh:
  emit_dsa: false
  emit_ecdsa: true
  emit_ed25519: true
  emit_rsa: true

    Packages
    
package_update: true
package_upgrade: true
packages:
  - curl
  - git
  - rsync
  - tmux
  - htop
  - jq
  - python3
  - python3-pip
  - python3-venv
  - caddy           # or nginx
  - podman          # container runtime (Docker alternative)
  - pdsh            # parallel SSH

    Write files
    
write_files:
  - path: /etc/ssh/sshd_config.d/hardened.conf
    content: |
      PermitRootLogin prohibit-password
      PasswordAuthentication no
      X11Forwarding no
      MaxAuthTries 3
      ClientAliveInterval 300
      ClientAliveCountMax 2
    owner: root:root
    permissions: '0644'

    - path: /etc/systemd/system/hermes-webui.service
        content: |
          [Unit]
          Description=Hermes Agent WebUI
          After=network-online.target
          Wants=network-online.target
    

    [Service]
          Type=simple
          ExecStart=/usr/local/bin/hermes webui
          Environment=HERMES_WEBUI_PASSWORD=munaf
          PrivateTmp=true
          ProtectSystem=strict
          ReadWritePaths=/var/log/hermes
          NoNewPrivileges=true
          Restart=always
          RestartSec=5
    

    [Install]
          WantedBy=multi-user.target
        owner: root:root
        permissions: '0644'
    

    - path: /etc/caddy/Caddyfile
        content: |
          hermes.example.com {
              reverse_proxy localhost:8788
          }
        owner: root:root
        permissions: '0644'
    

    Run commands
    
runcmd:
  # Enable services
  - systemctl daemon-reload
  - systemctl enable --now caddy
  - systemctl enable --now hermes-webui

    # Install Hermes Agent
      - curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
    

    # Firewall
      - ufw default deny incoming
      - ufw allow ssh
      - ufw allow http
      - ufw allow https
      - ufw --force enable
    

    # Create log directory
      - mkdir -p /var/log/hermes
      - chown hermes:hermes /var/log/hermes

    Quick Provision with Cloud-init

    # Hetzner
hcloud server create --name hermes-vps --type cx22 --image ubuntu-24.04 --user-data-from-file cloud-init.yaml

    DigitalOcean
    
doctl compute droplet create hermes-vps --image ubuntu-24-04-x64 --size s-2vcpu-4gb --user-data-file cloud-init.yaml

    Vultr
    
vultr instance create --label hermes-vps --plan vc2-2c-4gb --os 1863 --userdata "$(cat cloud-init.yaml)"

    Cloud-init Debugging

    # Check cloud-init status
cloud-init status

    Read cloud-init logs
    
cat /var/log/cloud-init.log
cat /var/log/cloud-init-output.log

    Re-run cloud-init (for testing)
    
cloud-init clean
cloud-init init
cloud-init modules -m config
cloud-init modules -m final

    Pitfalls:

  • First line must be #cloud-config — without this header, cloud-init won't parse the file

  • YAML must be valid — use cloud-init -c cloud-init.yaml to validate before deploying

  • No interactive commands — runcmd runs non-interactively; don't use commands that expect user input

  • Ordering — write_files runs before runcmd, so config files are in place when commands execute

  • Idempotency — package_upgrade: true runs on every boot if not disabled. Set package_upgrade: false after first boot, or use cloud-init clean carefully

    • 💬 Ask about Vps Provision