Vps Provision

Category: DevOps & Infrastructure | Read: 23 min | v1.1.0

VPS Provisioning & Sync

Spin up new Hermes instances on VPSes for parallel work delegation.

Current Fleet

| Alias | IP | Geo | Camofox Port | SSH Key | Role |
|-------|-----|-----|-------------|---------|------|
| hkus2 | 82.21.72.61 | US (NY) | 9377 (local) | default | Primary |
| hkice1 | 82.22.53.61 | US (HK) | 9377 (local) | default | OpenClaw + Hermes |
| hkuk2 | 152.114.193.86 | US (UK?) | 9377 (local) | default | Full Hermes Clone |
| hktr1 / child-vps | 82.26.94.226 | TR (Istanbul) | 9378 (tunnel) | ~/.ssh/id_ed25519 | Secondary |
| pkoc2 / pk-vps | 103.171.122.216 | PK (Islamabad) | 9379 (tunnel) | ~/.ssh/id_pk | PK Browser #1 |
| pkoc3 / pk-vps2 | 103.171.122.217 | PK (Islamabad) | 9380 (tunnel) | ~/.ssh/id_pk3 | PK Browser #2 (default) |

SSH aliases: child-vps/hktr1 → root@82.26.94.226 (id_ed25519), pkoc2/pk-vps → root@103.171.122.216 (id_pk), pkoc3/pk-vps2 → root@103.171.122.217 (id_pk3), hkice1 → root@82.22.53.61 (default key).

HK ICE-1 hosts: OpenClaw (Docker, port 18789, oc.hkice1.s4s.host), Hermes WebUI (port 8788), CamoFox (port 9377). Nginx routes to OpenClaw by default.

Quick Sync to New VPS

hermes-sync  [user]

This script (/usr/local/bin/hermes-sync) does:

  • Installs Hermes if not present

  • Copies .env, config.yaml, auth.json

  • Rsyncs all 107 skills

  • Copies AGENTS.md/CLAUDE.md context

  • Installs Camoufox browser + Xvfb + systemd services

  • Resets CAMOFOX_URL to local (not tunnel)
  • Example: hermes-sync child-vps or hermes-sync 1.2.3.4 root

    Manual Steps (if script fails)

    # 1. SSH key setup
    ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519
    ssh-copy-id -o StrictHostKeyChecking=no root@
    

    2. Install Hermes

    ssh root@ 'curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash'

    3. Sync everything

    rsync -az --del ~/.hermes/skills/ root@:.hermes/skills/ scp ~/.hermes/.env ~/.hermes/config.yaml root@:.hermes/ scp ~/.hermes/auth.json root@:.hermes/

    4. Camoufox (browser automation)

    ssh root@ 'cd /usr/local/lib/hermes-agent && source venv/bin/activate && pip install camoufox && python -m camoufox.fetch'

    5. Systemd services for Xvfb + Camofox (see script for service files)

    Using Child VPS

    One-shot LLM queries (preferred over hermes chat)

    llm-ask  "question"
    llm-ask groq "What is 7*8?"          # Free, 212 tok/s
    llm-ask opencode-go "Explain X"     # Default provider
    llm-ask openrouter "Research Y"     # 33+ free models
    llm-ask chutes "Analyze Z"          # TEE privacy
    llm-ask byteplus-coding "code task" # Coding specialist
    
    Works on any VPS with /usr/local/bin/llm-ask installed. Auto-loads .env keys. 14 providers available.

    On child VPS: ssh child-vps 'llm-ask groq "question"'

    ⚠️ Do NOT use hermes chat -q for scripted queries — the TUI renderer produces garbled output. Use llm-ask instead.

    Background tasks (SSH + notify_on_complete)

    # Dispatch task to child VPS, get notified when done
    terminal(command="ssh child-vps 'llm-ask groq \"question\"'", background=True, notify_on_complete=True)
    

    Parallel tasks on child VPS

    # Run multiple providers simultaneously on child
    terminal(command="ssh child-vps 'llm-ask groq \"task 1\"' & ssh child-vps 'llm-ask opencode-go \"task 2\"' & wait")
    

    Interactive Hermes session (tmux)

    ssh child-vps 'tmux new-session -d -s worker -x 120 -y 40 "hermes"'
    ssh child-vps 'tmux send-keys -t worker "your prompt" Enter'
    ssh child-vps 'tmux capture-pane -t worker -p | tail -30'
    

    Browser via child VPS (different geo/IP)

    ssh child-vps 'child-browse "https://example.com"'
    
    The child-browse script wraps Camoufox with DISPLAY=:99. Returns page text. For JSON APIs: ssh child-vps 'curl -s https://ipinfo.io/json'

    delegate_task from this Hermes

    Use terminal(command="ssh child-vps 'llm-ask ...'") with background=True for async work. For complex multi-turn tasks, start a tmux session on the child.

    Pitfalls

  • hermes chat -q produces garbled TUI output — always use llm-ask for one-shot scripted queries
  • SSH tunnel port conflicts — stale SSH processes from previous tunnel instances can hold the port, causing "Address already in use" errors that make the tunnel service loop in auto-restart. Fix: ss -tlnp | grep to find the stale PID, kill -9 , then systemctl restart ssh-tunnel-. The journalctl telltale is bind [127.0.0.1]:: Address already in use.
  • Camoufox service: Type=oneshot, RemainAfterExit=yes — the Python launcher forks a Node process then exits. systemctl status shows "active (exited)" which is correct; the browser keeps running.
  • Xvfb must be running before Camofox starts. The systemd dependency After=xvfb.service handles ordering.
  • SSH tunnel persists only while SSH process livesssh -f -N -L 9378:localhost:9377 creates a background tunnel, but it dies if the SSH connection drops. Don't rely on tunnels for production use; use direct commands via ssh child-vps instead.
  • Skills sync is destructive mirrorrsync --del means local deletions propagate to child. Intentional to keep fleets identical.
  • CAMOFOX_URL auto-detects locally — don't set it in child's .env unless routing through a tunnel (which you shouldn't for normal use).
  • Nginx auth double-login — if a web app has built-in auth (like Hermes WebUI), do NOT add nginx auth_basic on top. It creates a double-login. Use only the app's built-in auth or only nginx auth, never both.
  • Password preference — user uses "munaf" as the default password for all site auth (basic auth htpasswd, WebUI HERMES_WEBUI_PASSWORD, dashboard access). Applies to all VPSes in the fleet.
  • SSH key for PK VPS~/.ssh/id_pk (separate from the ed25519 key used for child-vps).
  • hermes-sync on PK overwrites Node-based Camofox — the sync script tries to install python -m camoufox.server as a systemd service. If the VPS already has the Node-based Camofox from /opt/camofox-browser, the Python service will conflict. After running hermes-sync on a VPS with existing Node Camofox, verify with curl localhost:9377/health that the correct instance is running.
  • Hermes WebUI — deployed at https://webui.hkus2.s4s.host/ (password: munaf), systemd service hermes-webui, internal port 8788.
  • Hermes Dashboard — deployed at https://dash.hkus2.s4s.host/ (basic auth munaf:munaf), internal port 9119, firewalled to localhost only.
  • All sites password-protected — API Hub, LLM Council, Dashboard, WebUI all require auth (munaf:munaf for basic auth, or single password "munaf" for WebUI built-in auth).
  • Multi-Geo Browser Routing

    Persistent SSH tunnels route remote Camofox instances to localhost ports.

    | Port | VPS | Geo | Systemd Service |
    |------|-----|-----|-----------------|
    | 9377 | localhost (US) | New York, US | camofox-server (local) |
    | 9377 | hkice1 | US (HK) | camofox (local to hkice1) |
    | 9378 | child-vps (TR) | Istanbul, Turkey | ssh-tunnel-tr |
    | 9379 | pkoc2 (PK) | Islamabad, Pakistan | ssh-tunnel-pk |
    | 9380 | pkoc3 (PK) | Islamabad, Pakistan | ssh-tunnel-pk2 |

    Switch browser geo

    Set CAMOFOX_URL in ~/.hermes/.env, then /reset:

    # US (auto-detected — no need to set)
    

    CAMOFOX_URL=http://localhost:9377

    Turkey

    ssh -f -N -L 9378:localhost:9377 -o ServerAliveInterval=60 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes child-vps

    Pakistan (pkoc2)

    CAMOFOX_URL=http://localhost:9379

    Pakistan (pkoc3) — DEFAULT

    CAMOFOX_URL=http://localhost:9380

    Create persistent SSH tunnel (systemd)

    For each remote VPS with Camofox:

    # /etc/systemd/system/ssh-tunnel-tr.service
    [Unit]
    Description=SSH Tunnel to Turkey VPS Camofox (port 9378)
    After=network-online.target
    

    [Service]
    ExecStart=/usr/bin/ssh -N -L 9378:localhost:9377 -o ServerAliveInterval=60 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes child-vps
    Restart=always
    RestartSec=5

    [Install]
    WantedBy=multi-user.target

    Then: systemctl daemon-reload && systemctl enable --now ssh-tunnel-tr

    Verify: curl -s http://localhost:9378/health

    Direct API usage (bypass Hermes browser tools)

    When you need to browse from a specific geo without changing CAMOFOX_URL:

    import requests, json, time
    

    base = "http://localhost:9379" # PK Camofox via tunnel
    user_id, session_key = "hermes-pk", "pk-session"

    Create tab

    r = requests.post(f"{base}/tabs", json={"userId": user_id, "sessionKey": session_key}) tab_id = r.json()["tabId"]

    Navigate (userId required in navigate/snapshot calls too)

    requests.post(f"{base}/tabs/{tab_id}/navigate", json={"url": "https://ipinfo.io/json", "userId": user_id}) time.sleep(3) r = requests.get(f"{base}/tabs/{tab_id}/snapshot", params={"userId": user_id}) print(r.json().get("snapshot", ""))

    Cleanup

    requests.delete(f"{base}/tabs/{tab_id}", params={"userId": user_id})

    Quick geo-IP verification

    # Via the Camofox API
    ssh pkoc2 'curl -s https://ipinfo.io/json'        # PK IP
    ssh child-vps 'curl -s https://ipinfo.io/json'     # TR IP
    

    Via Python API (see above) — returns IP of the browser's VPS

    Hermes Approvals — Auto-Approve

    User preference: all approvals on auto, no confirmation prompts.

    # In ~/.hermes/config.yaml:
    approvals:
      mode: auto          # was: manual
      timeout: 60
      cron_mode: auto     # was: deny
      mcp_reload_confirm: false
      destructive_slash_confirm: false
    

    delegation:
    subagent_auto_approve: true # was: false

    Full Hermes Clone Deployment (Nginx + SSL + Docker)

    When deploying a complete replica of the gateway (not just a child agent), you need Nginx, SSL, Hindsight (Docker), WA Chat Viewer, MariaDB, and all web content.

    Step-by-step Clone Checklist

  • SSH key setupssh-copy-id or manual cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys
  • Install Hermes — Ubuntu 24.04 doesn't have pip3 by default. Use venv:
  •    python3 -m venv /usr/local/lib/hermes-agent/venv
       /usr/local/lib/hermes-agent/venv/bin/pip install hermes-agent
       ln -sf /usr/local/lib/hermes-agent/venv/bin/hermes /usr/local/bin/hermes
       
  • Copy config — Don't rsync/tar the entire .hermes/ (1.5G+ in plugins alone). Copy only essentials:
  •    scp config.yaml AGENT_PERSONA.md root@NEW:.hermes/
       cd .hermes && tar cf - skills/ | ssh root@NEW "cd .hermes && tar xf -"
       
    Then sed -i 's/old\.domain/new.domain/g' config.yaml for domain renames.
  • Nginx + SSL — Install nginx + certbot. The critical sequence:
  •    apt-get install -y nginx certbot python3-certbot-nginx
       
       # Generate temp self-signed certs AND dhparam BEFORE starting nginx
       # certbot --nginx fails if nginx can't start (no SSL certs)
       for domain in sub1.new.domain sub2.new.domain ...; do
         mkdir -p /etc/letsencrypt/live/$domain
         openssl req -x509 -nodes -days 1 -newkey rsa:2048 \
           -keyout /etc/letsencrypt/live/$domain/privkey.pem \
           -out /etc/letsencrypt/live/$domain/fullchain.pem \
           -subj '/CN=temp'
       done
       openssl dhparam -out /etc/letsencrypt/ssl-dhparams.pem 2048
       
       # Copy nginx site configs from source, sed domain names
       # Start nginx, THEN get real certs
       nginx -t && systemctl start nginx
       
       # Get ONE SAN cert for ALL subdomains (not per-domain)
       certbot certonly --standalone --non-interactive --agree-tos \
         --email your@email.com \
         -d sub1.new.domain -d sub2.new.domain -d sub3.new.domain ...
       
       # Stop nginx temporarily for standalone mode, then update all
       # site configs to point to the single cert path, then restart
       
    ⚠️ Pitfall: certbot --nginx fails if /etc/letsencrypt/live// already exists from temp certs. Either rm -rf /etc/letsencrypt/live archive renewal accounts before running, or use --standalone mode (stop nginx first). ⚠️ Pitfall: openssl dhparam can take 30+ seconds. Don't background it — wait for it to finish before starting nginx.
  • Hindsight (Docker) — Needs LLM provider env vars or it crashes:
  • docker run -d --name hindsight --restart unless-stopped \
      -p 9999:9999 -p 9077:8888 \
      -v /root/.hindsight-data:/home/hindsight/.pg0 \
      -e HINDSIGHT_API_KEY=top-secret-key \
      -e HINDSIGHT_API_LLM_PROVIDER=openai \
      -e HINDSIGHT_API_LLM_API_KEY= \
      -e HINDSIGHT_API_LLM_MODEL=kimi-k2.5 \
      -e HINDSIGHT_API_LLM_BASE_URL=https://ollama.com/v1 \
      -e HINDSIGHT_API_HOST=0.0.0.0 \
      -e HINDSIGHT_API_PORT=8888 \
      -e HINDSIGHT_ENABLE_API=true \
      -e HINDSIGHT_ENABLE_CP=true \
      -e HINDSIGHT_CP_ACCESS_KEY=munaf \
      -e HINDSIGHT_CP_DATAPLANE_API_URL=http://localhost:8888 \
      -e HINDSIGHT_API_LOG_LEVEL=info \
      ghcr.io/vectorize-io/hindsight:latest
    
    ⚠️ CRITICAL: Volume mount is -v /root/.hindsight-data:/home/hindsight/.pg0 — NOT /root/.hermes/hindsight:/app/data. The gateway stores Hindsight data at /root/.hindsight-data/ and the container reads from /home/hindsight/.pg0. Getting this wrong means the clone starts with zero memory.

    ⚠️ Pitfall: Hindsight takes 15-30 seconds to initialize (downloads embedding/reranker models on first run). Check health with curl http://localhost:9077/health — don't assume it's broken just because port isn't responding immediately.

  • Hermes WebUI — Install as systemd service. Needs HERMES_WEBUI_PASSWORD env var. Copy /opt/hermes-webui/ from source (exclude node_modules/.git).
  • WhatsApp Chat Viewer — Python Flask app at /var/www/whatsapp-chat/. Needs flask and flask-cors in the Hermes venv:
  •    /usr/local/lib/hermes-agent/venv/bin/pip install flask flask-cors
       
  • MariaDBapt-get install -y mariadb-server for session storage.
  • Web content — Static sites (API Hub, LLM Council, Server4Sale, etc.) just need tar copy to /var/www// and nginx config.
  • Verify all domains — Loop through subdomains checking HTTP status codes:
  •     for domain in apinew dash webui memory llm new seo; do
          curl -sk -o /dev/null -w "%{http_code}" "https://$domain.new.domain/" && echo " $domain"
        done
        

    SAN Cert vs Per-Domain Certs

    For multi-domain setups, get one SAN cert covering all subdomains rather than separate certs per domain:

  • All nginx server_name blocks reference the same ssl_certificate path

  • Easier to manage, single renewal

  • Use certbot certonly --standalone -d a.domain -d b.domain -d c.domain

  • After getting the real cert, update all nginx configs to point to it, then nginx -t && systemctl restart nginx
  • Full DR Clone — Data Transfer Checklist

    When doing a complete DR clone (not just a child agent sync), you need to transfer ALL data directories. Don't assume .hermes/ is enough.

    What to transfer (in order of size)

    | Source Path | Size (typical) | Purpose |
    |-------------|----------------|---------|
    | ~/.hermes/plugins/ | ~1.5 GB | Installed plugin repos |
    | ~/.hindsight-data/ | ~480 MB | Hindsight vector DB (Docker volume source) |
    | /opt/hermes-webui/ | ~240 MB | WebUI application (exclude node_modules) |
    | ~/.hermes/sessions/ | ~140 MB | Session transcripts |
    | ~/.hermes/skills/ | ~55 MB | Agent skills |
    | /var/www/ | varies | API Hub, LLM Council, Server4Sale, etc. |
    | ~/.hermes/whatsapp/ | ~17 MB | WhatsApp bridge data |
    | ~/.hermes/cron/ | ~1 KB | Cron jobs |
    | ~/.hermes/bin/ | varies | Custom scripts (llm-ask, hermes-sync) |

    Transfer method for large dirs

    # Stop Hindsight on SOURCE for clean data copy
    ssh root@SOURCE 'docker stop hindsight'
    

    Create tar of everything missing from standard sync

    ssh root@SOURCE 'cd /root && tar cf /tmp/hermes-dump.tar \ .hindsight-data/ \ .hermes/sessions/ \ .hermes/webui/ \ .hermes/plugins/ \ .hermes/cron/ \ .hermes/whatsapp/ \ .hermes/bin/ \ .hermes/logs/ 2>/dev/null; \ tar rf /tmp/hermes-dump.tar /opt/hermes-webui/ 2>/dev/null; \ tar rf /tmp/hermes-dump.tar /var/www/ 2>/dev/null'

    Compress and transfer

    ssh root@SOURCE 'gzip -f /tmp/hermes-dump.tar' scp root@SOURCE:/tmp/hermes-dump.tar.gz /tmp/ scp /tmp/hermes-dump.tar.gz root@DEST:/tmp/ ssh root@DEST 'cd / && tar xzf /tmp/hermes-dump.tar.gz'

    Restart Hindsight on SOURCE

    ssh root@SOURCE 'docker start hindsight'

    Post-clone verification

    # Check Hindsight data integrity — node/doc/link counts must match SOURCE
    ssh root@DEST 'curl -s http://localhost:9077/health | jq .'
    

    Verify all subdomains respond

    for domain in apinew dash webui memory api-memory llm new seo techmart digitech wa hindsight; do code=$(curl -sk -o /dev/null -w "%{http_code}" "https://$domain.DEST_DOMAIN/") echo "$domain: $code" done

    Check .hermes size matches

    ssh root@SOURCE 'du -sh ~/.hermes/ ~/.hindsight-data/' ssh root@DEST 'du -sh ~/.hermes/ ~/.hindsight-data/'

    Pitfalls specific to full DR clones

  • hermes-gateway.service: Enable it (systemctl enable hermes-gateway) but do NOT start it until WhatsApp is paired on the new server. Starting without a paired session causes the bridge to fail silently.
  • Hindsight must be stopped during data copy — copying its data while running produces a corrupt or inconsistent snapshot. Stop the container, copy, then restart.
  • WebUI node_modules: When copying /opt/hermes-webui/, exclude node_modules/ and .git/ to save 200+ MB. Run cd /opt/hermes-webui && npm install on the destination instead.
  • Flask for WA Chat Viewer: The WA Chat Viewer needs flask and flask-cors in the Hermes venv: /usr/local/lib/hermes-agent/venv/bin/pip install flask flask-cors
  • Domain rename: After copying config.yaml, run sed -i 's/old\.domain/new.domain/g' to update all domain references.
  • Memory is almost full: After a full clone, memory may be at 80%+ from the process. Don't try to store the entire transfer log — only store durable facts.
  • Hindsight CP_ACCESS_KEY: Always munaf across all fleet nodes.
  • WebUI passwords: Primary gateway uses munaf. Clone nodes use admin (e.g., hkuk2admin for hkuk2).
  • Hindsight env vars: The clone's Hindsight container needs the same LLM env vars as the gateway. Check the gateway's running container with docker inspect hindsight | jq '.[0].Config.Env' to get the exact values.
  • Notes

  • Child VPS config matches parent (same model, same API keys, same 14 providers)
  • Both VPSes have /usr/local/bin/llm-ask and /usr/local/bin/hermes-sync
  • Both VPSes have /usr/local/bin/child-browse for Camoufox quick-browse
  • Advanced Techniques (2025-2026)

    SSH multiplexing (10x speedup)

    Enable SSH multiplexing to reuse a single TCP connection for all SSH/SCP/rsync operations to the same host. This eliminates the TCP handshake and key exchange overhead for every command:

    # Add to ~/.ssh/config
    Host child-vps pkoc2 pkoc3 hkice1
      ControlMaster auto
      ControlPath ~/.ssh/sockets/%r@%h-%p
      ControlPersist 600
    
    mkdir -p ~/.ssh/sockets
    

    With this, the first SSH connection establishes the master socket; subsequent ssh, scp, and rsync commands to the same host reuse it instantly — often 10× faster for short commands like ssh child-vps 'hostname'.

    rsync --checksum for idempotent sync

    The default rsync -az --del uses modification time + size to decide what to transfer. If timestamps drift across VPSes (NTP skew, filesystem differences), rsync may re-copy unchanged files. Use --checksum for truly idempotent syncs:

    rsync -azc --del ~/.hermes/skills/ root@child-vps:.hermes/skills/
    

    --checksum computes a file-content hash instead of relying on mtime. Slower on large directories but guarantees no unnecessary transfers.

    SSH config wildcards

    Group common settings for the entire fleet with wildcard patterns in ~/.ssh/config:

    Host .s4s.host 10.
      User root
      StrictHostKeyChecking no
      UserKnownHostsFile /dev/null
      ServerAliveInterval 60
      ServerAliveCountMax 3
    

    Host child-vps
    HostName 82.26.94.226
    IdentityFile ~/.ssh/id_ed25519

    Host pkoc2 pk-vps
    HostName 103.171.122.216
    IdentityFile ~/.ssh/id_pk

    This avoids repeating User, StrictHostKeyChecking, and keepalive settings for every host block.

    pdsh/parallel for mass fleet commands

    When you need to run the same command across all VPSes simultaneously:

    # Install pdsh
    apt-get install -y pdsh
    

    Run across the fleet

    pdsh -w child-vps,pkoc2,pkoc3 'apt-get update && apt-get upgrade -y'

    Or with GNU parallel for more control:

    # Check disk space across fleet
    parallel --nonall -j0 'ssh {} df -h /' ::: child-vps pkoc2 pkoc3
    

    systemd dependency ordering

    When services depend on each other (e.g., Xvfb must be running before Camofox, SSH tunnels before apps that use them), use explicit ordering:

    [Unit]
    After=xvfb.service
    Wants=xvfb.service
    BindsTo=xvfb.service
    
  • After= — start this service after the listed one
  • Wants= — start the listed unit if it's not already running (soft dependency)
  • BindsTo= — stop this service if the listed unit stops (hard dependency)
  • Before= — declare this service must start before another
  • For SSH tunnel services that might have port conflicts on restart, add a Conflicts= stanza and use ExecStartPre to kill stale processes:

    [Service]
    ExecStartPre=/usr/bin/bash -c 'ss -tlnp | grep :9378 && exit 1 || exit 0'
    ExecStart=/usr/bin/ssh -N -L 9378:localhost:9377 ...
    RestartSec=3
    

    Edge Cases / Pitfalls

  • SSH tunnel ghost with RestartSec race — If an SSH tunnel service crashes and RestartSec=5 fires, but the old SSH process hasn't fully released the port yet, the new process gets bind: Address already in use. The service enters a crash loop. Mitigation: use ExecStartPre to check port availability, set RestartSec=10 to give more decay time, or use ExitOnForwardFailure=yes (already in the fleet config) so SSH exits immediately instead of hanging with a dead tunnel.
  • rsync --del following symlinksrsync --del removes destination files that don't exist at the source. If your source directory contains symlinks pointing outside the transfer tree, rsync by default copies the symlink target content (not the symlink itself). Use -l (copy symlinks as symlinks) or --copy-links (follow and copy content) intentionally — mixing them up can cause --del to remove files on the destination that exist under different symlink paths on the source.
  • OpenTofu as Terraform Alternative

    OpenTofu is a CNCF-incubating project and drop-in replacement for Terraform 1.5.x–1.6.x. After HashiCorp's BSL license change, the community forked Terraform into OpenTofu, which is now governed by the Linux Foundation.

    Key advantages over Terraform:

  • State encryption — native encryption of state files at rest (AES-256-GCM), a feature Terraform lacks without paid Terraform Cloud

  • Drop-in compatibility — same HCL syntax, same CLI commands (tofu init/plan/apply), interoperable state files with Terraform 1.5–1.6

  • Enterprise adoption — Fidelity, large cloud providers, and platform teams have migrated production workloads

  • Community-driven — open governance under CNCF, no vendor lock-in
  • Limitations vs. Terraform:

  • No SAML/SSO (Terraform Cloud/Enterprise-only feature)

  • No Terraform Cloud remote operations (state backend works fine with S3/GCS/Azure Blob)

  • Provider/plugin ecosystem identical (shares the same registry for now; diverging over time)
  • Usage on VPS fleet:

    # Install OpenTofu
    curl -fsSL https://get.opentofu.org/install.sh | bash

    Drop-in replacement — just alias

    alias tf=tofu

    Encrypt state at rest

    tofu init -backend-config="encrypt=true"

    For VPS provisioning, OpenTofu can manage Hetzner/DO/Vultr instances as easily as Terraform but with state encryption built in. Prefer tofu over terraform for all new infrastructure code.

    Caddy as Nginx Alternative

    Caddy is an excellent reverse proxy for simple VPS setups — especially when you need automatic HTTPS with zero configuration.

    Key advantages over Nginx:

  • Automatic HTTPS — provisions and renews Let's Encrypt certificates automatically, zero config needed. No certbot cron jobs, no manual certificate management

  • Single binary — one static binary, no dependencies, apt install caddy or download directly

  • Simpler config — Caddyfile is dramatically simpler than nginx.conf for common patterns:

  •   webui.example.com {
    reverse_proxy localhost:8788
    }

    vs. Nginx's ~40 lines for the same (server block, SSL config, proxy headers, cert paths, HSTS)
  • Performance — benchmarks at 142K req/s (≈22% faster than Nginx for typical reverse proxy workloads)

  • HTTP/3 built-in — QUIC support out of the box, no extra compilation flags

  • JSON config API — runtime config changes via REST API without restarts
  • When to use Caddy vs. Nginx:

  • Caddy — simple reverse proxies, automatic HTTPS, few upstreams, single-server setups, quick prototyping

  • Nginx — complex load balancing, rate limiting, custom auth modules, large-scale production (existing expertise), Lua/OpenResty needs
  • Caddy on VPS:

    apt install -y caddy

    Minimal Caddyfile for Hermes WebUI

    cat > /etc/caddy/Caddyfile << 'EOF' webui.hkus2.s4s.host { reverse_proxy localhost:8788 } dash.hkus2.s4s.host { basicauth * { munaf $2a$14$HASH_HERE } reverse_proxy localhost:9119 } EOF

    systemctl restart caddy

    ⚠️ Caddy auto-HTTPS requires ports 80+443 open and DNS pointing to the server. For internal-only services, use http:// prefix to disable auto-HTTPS: http://internal.local { ... }.

    Systemd Security Hardening Templates

    Apply systemd sandboxing directives to all Hermes-related services (Camofox, Xvfb, SSH tunnels, etc.) to limit blast radius if any service is compromised.

    Recommended hardening directives (add to [Service] section):

    [Service]
    

    Filesystem isolation

    PrivateTmp=true ProtectSystem=strict ReadWritePaths=/var/log/hermes /run/hermes ProtectHome=read-only

    Capability restrictions

    NoNewPrivileges=true PrivateDevices=true ProtectClock=true ProtectHostname=true

    Memory / execution restrictions

    MemoryDenyWriteExecute=true LockPersonality=true RestrictRealtime=true RestrictSUIDSGID=true

    Namespace restrictions

    RestrictNamespaces=true

    Or granular: RestrictNamespaces=net ipc

    System call filtering

    SystemCallFilter=@system-service SystemCallFilter=~@mount @keyring @privileged

    Network restrictions (if service doesn't need network)

    IPAccounting=yes

    IPAddressDeny=any

    IPAddressAllow=localhost

    Ambient capabilities

    AmbientCapabilities= CapabilityBoundingSet=

    Applying to existing fleet services:

    # Check current security score (lower = more hardened)
    systemd-analyze security camofox-server.service
    systemd-analyze security xvfb.service
    systemd-analyze security hermes-webui.service
    

    Create override for any service (avoids editing packaged unit files)

    systemctl edit camofox-server.service

    Paste the hardening directives above into the override file

    Reload and restart

    systemctl daemon-reload systemctl restart camofox-server.service

    Per-service tuning:

  • Camofox/Xvfb — needs X11 socket, shared memory, and network access; use ProtectSystem=full (not strict), PrivateDevices=false (GPU access), exclude @mount from syscall filter

  • SSH tunnels — only need network access; can use maximum hardening including IPAddressDeny=any + IPAddressAllow=localhost for SOCKS-only

  • Hermes WebUI — needs network access and write paths for logs; ProtectSystem=strict with explicit ReadWritePaths=
  • Pitfalls:

  • ProtectSystem=strict blocks writes to /usr, /etc, /boot. Add ReadWritePaths= for any dirs the service needs

  • MemoryDenyWriteExecute=true breaks JIT-compiled runtimes (Node.js, Python's PyGTK). Test services after enabling

  • PrivateDevices=true prevents access to /dev/ devices. Camofox needs /dev/dri/* for GPU — use PrivateDevices=false there

  • Always run systemd-analyze security after changes — aim for score ≤ 5.0
  • Let's Encrypt Certificate Changes

    Let's Encrypt is significantly changing certificate lifetimes and renewal practices. These changes affect all VPS fleet services using auto-HTTPS (Caddy) or certbot.

    Certificate Lifetime Timeline

    | Date | Change | |------|--------| | May 2026 | 45-day certificates available (opt-in) | | June 2025 | Expiration notification emails discontinued — no more renewal reminders | | Feb 2027 | 64-day certificates available | | Feb 2028 | 45-day certificates become default (90-day certs no longer issued) |

    Critical Impacts

  • Expiration emails ended June 2025 — no more email warnings. You must implement your own monitoring (certwatch scripts, Prometheus alerts, or health checks)
  • More frequent renewals — 45-day certs mean renewing every ~30 days instead of every 60. Automate everything
  • ARI (ACME Renewal Information) — new extension that tells your ACME client the optimal window to renew. Prevents thundering-herd renewals and reduces failure risk:
  • # Certbot with ARI support (certbot >= 2.8)
    certbot renew --deploy-hook "systemctl reload nginx"
    

    Caddy uses ARI natively — no action needed

  • DNS-PERSIST-01 challenge (coming 2026) — new challenge type that allows persistent DNS validation without re-creating TXT records every renewal. Reduces DNS API calls and supports smoother automation.
  • Action Items for VPS Fleet

    # 1. Update Certbot (for Nginx-based hosts)
    apt update && apt install -y certbot python3-certbot-nginx
    certbot --version  # Ensure >= 2.8 for ARI support
    

    2. Monitor certificate expiry without email

    Add to cron on all VPSes:

    cat > /etc/cron.daily/cert-check << 'SCRIPT' #!/bin/bash

    Alert if any cert expires in < 14 days

    for cert in /etc/letsencrypt/live/*/cert.pem; do days=$(( ( $(date -d "$(openssl x509 -enddate -noout -in "$cert" | cut -d= -f2)" +%s) - $(date +%s) ) / 86400 )) if [ "$days" -lt 14 ]; then echo "WARNING: $cert expires in $days days" >&2 fi done SCRIPT chmod +x /etc/cron.daily/cert-check

    3. For Caddy — already handles ARI and auto-renewal natively

    No changes needed; Caddy will automatically use shorter lifetimes when available

    ⚠️ Caddy users — Caddy handles all of this automatically (ARI, renewal, shorter lifetimes). No action needed unless you want custom monitoring. Certbot/Nginx users must update certbot and add expiry monitoring.

    Cloud-init Templates

    Cloud-init is the industry standard for first-boot VM initialization across all major cloud providers (AWS, Azure, GCP, DigitalOcean, Hetzner, Vultr, Linode). It runs on first boot to configure users, packages, files, and commands from a single YAML file.

    Basic VPS Setup Template

    #cloud-config
    

    Save as: cloud-init.yaml

    Usage: Pass as user-data when creating VPS, or:

    cloud-init -c cloud-init.yaml (on existing host)

    Timezone

    timezone: UTC

    Locale

    locale: en_US.UTF-8

    Hostname

    hostname: hermes-vps manage_etc_hosts: true

    Users

    users: - name: hermes sudo: ALL=(ALL) NOPASSWD:ALL shell: /bin/bash ssh_authorized_keys: - ssh-ed25519 AAAA... hermes@fleet lock_passwd: false passwd: "$6$rounds=4096$salt$hash..." # mkpasswd -m sha512crypt

    - name: root
    ssh_authorized_keys:
    - ssh-ed25519 AAAA... root@fleet

    SSH hardening

    ssh_deletekeys: true ssh: emit_dsa: false emit_ecdsa: true emit_ed25519: true emit_rsa: true

    Packages

    package_update: true package_upgrade: true packages: - curl - git - rsync - tmux - htop - jq - python3 - python3-pip - python3-venv - caddy # or nginx - podman # container runtime (Docker alternative) - pdsh # parallel SSH

    Write files

    write_files: - path: /etc/ssh/sshd_config.d/hardened.conf content: | PermitRootLogin prohibit-password PasswordAuthentication no X11Forwarding no MaxAuthTries 3 ClientAliveInterval 300 ClientAliveCountMax 2 owner: root:root permissions: '0644'

    - path: /etc/systemd/system/hermes-webui.service
    content: |
    [Unit]
    Description=Hermes Agent WebUI
    After=network-online.target
    Wants=network-online.target

    [Service]
    Type=simple
    ExecStart=/usr/local/bin/hermes webui
    Environment=HERMES_WEBUI_PASSWORD=munaf
    PrivateTmp=true
    ProtectSystem=strict
    ReadWritePaths=/var/log/hermes
    NoNewPrivileges=true
    Restart=always
    RestartSec=5

    [Install]
    WantedBy=multi-user.target
    owner: root:root
    permissions: '0644'

    - path: /etc/caddy/Caddyfile
    content: |
    hermes.example.com {
    reverse_proxy localhost:8788
    }
    owner: root:root
    permissions: '0644'

    Run commands

    runcmd: # Enable services - systemctl daemon-reload - systemctl enable --now caddy - systemctl enable --now hermes-webui

    # Install Hermes Agent
    - curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash

    # Firewall
    - ufw default deny incoming
    - ufw allow ssh
    - ufw allow http
    - ufw allow https
    - ufw --force enable

    # Create log directory
    - mkdir -p /var/log/hermes
    - chown hermes:hermes /var/log/hermes

    Quick Provision with Cloud-init

    # Hetzner
    hcloud server create --name hermes-vps --type cx22 --image ubuntu-24.04 --user-data-from-file cloud-init.yaml
    

    DigitalOcean

    doctl compute droplet create hermes-vps --image ubuntu-24-04-x64 --size s-2vcpu-4gb --user-data-file cloud-init.yaml

    Vultr

    vultr instance create --label hermes-vps --plan vc2-2c-4gb --os 1863 --userdata "$(cat cloud-init.yaml)"

    Cloud-init Debugging

    # Check cloud-init status
    cloud-init status
    

    Read cloud-init logs

    cat /var/log/cloud-init.log cat /var/log/cloud-init-output.log

    Re-run cloud-init (for testing)

    cloud-init clean cloud-init init cloud-init modules -m config cloud-init modules -m final

    Pitfalls:

  • First line must be #cloud-config — without this header, cloud-init won't parse the file

  • YAML must be valid — use cloud-init -c cloud-init.yaml to validate before deploying

  • No interactive commandsruncmd runs non-interactively; don't use commands that expect user input

  • Orderingwrite_files runs before runcmd, so config files are in place when commands execute

  • Idempotencypackage_upgrade: true runs on every boot if not disabled. Set package_upgrade: false after first boot, or use cloud-init clean carefully