INFRA — deployment reality on agnes and the dev box

Scope: what is actually running on the two hosts, verified by read-only SSH on 2026-11-09. Every claim below was produced by a command run against the live host in that session; anything I could not verify is marked [UNVERIFIED]. Nothing on either host was changed.

	prod agnes.pukogames.com	dev 192.168.1.250
Public IP	195.154.152.46	none (LAN only)
OS	Debian 13 (trixie), kernel 6.12.74+deb13+1-amd64	Arch Linux
Uptime / load	197 d, load 1.32 / 1.34 / 1.10	14 d, load 0.08 / 0.20 / 0.17
CPU / RAM	12 cores / 31 GiB	12 cores / 31 GiB
Root disk	/dev/md1 40 G, 14 G free (63 % used)	/dev/sdb2 436 G, 65 G free (85 % used)
Big disk	/dev/md2 1.8 T → /data, 692 G free (61 % used)	/dev/sda2 3.7 T → /mnt/hdd, 609 G free (slow HDD)
GPU	none	RTX 4060 8 GB (driver 615.71.09, CUDA 13.4)
Docker	29.9.3, storage driver overlayfs, data-root /data/docker	29.8.0, data-root /var/lib/docker (on /)
Compose	v5.7.0	v5.5.1
Containers running	64 (67 total incl. stopped)	11

---

1. Prod — what terminates 80/443

Host nginx 1.26.3 (Debian package, no Docker), systemctl is-active nginx → active.

ss -lntp | grep ':443 '
LISTEN 0 511 195.154.152.46:443 ... users:(("nginx",pid=2187625,fd=25),(...workers...))

The master is /usr/sbin/nginx -g daemon on; master_process on; (pid 2187625). A second nginx master with -g daemon off (pid 3985810) binds 212.83.136.221:80/443 and 212.83.131.57:80/443 — that is the adbtd-mta-edge mail container, unrelated to web traffic. No Caddy, no Traefik on prod (which caddy → not found).

Main config /etc/nginx/nginx.conf is the Debian default: tls 1.2/1.3, server_tokens off, gzip on, and it includes /etc/nginx/conf.d/*.conf then /etc/nginx/sites-enabled/*. There is no default_server on 443 (see grep -rn default_server /etc/nginx/sites-available/ — only :80 and commented lines).

Layout

Path	What lives there
/etc/nginx/sites-available/<name>.conf	real vhost files, one per app
/etc/nginx/sites-enabled/<name>	symlink → ../sites-available/<name>.conf
/etc/nginx/snippets/	probe-drop.conf, quest-security-headers.conf, muse-security-headers.conf, fastcgi-php.conf
/etc/nginx/conf.d/	stub_status.conf (Prometheus), elec-limits.conf, cadastre-*.conf, adbtd-log-format.conf
/etc/letsencrypt/live/<fqdn>/	certs; /etc/letsencrypt/renewal/<fqdn>.conf holds the renewal params

Live vhosts (15, i.e. all of sites-enabled): adbtd-admin, adbtd-check, adbtd-mcp.conf, adbtd-web, adbtd-webmail, eau.pukogames.com, elec.pukogames.com, gamesrv.pukogames.com, grafana.pukogames.com, k0.pukogames.com, lagrandecarte.fr, muse.pukogames.com, pukogames-landing (apex), quest.pukogames.com, wochew.pukogames.com.

Every vhost follows the same shape: server_name <fqdn>; + listen 195.154.152.46:443 ssl; + certbot-managed TLS lines + proxy_pass http://127.0.0.1:<port>, and a second server { listen 195.154.152.46:80; return 301 https://...; } block, also certbot-managed. sites-available is littered with *.pre-iter4-le.bak, *.bak.YYYYMMDD-HHMMSS copies — that backup-before-edit convention is the house style.

Declaring a vhost for a new app

1. Write /etc/nginx/sites-available/<fqdn> (copy quest.pukogames.com and change server_name, the upstream ports, the snippet includes).
2. ln -s /etc/nginx/sites-available/<fqdn> /etc/nginx/sites-enabled/<fqdn>
3. certbot --nginx -d <fqdn> — obtains the cert and injects the four # managed by Certbot TLS lines + the HTTP→301 block into the file.
4. nginx -t && systemctl reload nginx (reload, never restart: the box serves 15 vhosts).

TLS

• Let's Encrypt via certbot --nginx (authenticator nginx, installer nginx, key_type = ecdsa), verified in /etc/letsencrypt/renewal/quest.pukogames.com.conf. 18 certs under /etc/letsencrypt/live/.
• Renewal: certbot.timer (/usr/lib/systemd/system/certbot.timer, enabled): OnCalendar=*-*-* 00,12:00:00, RandomizedDelaySec=43200, Persistent=true → runs daily in practice. Ends in certbot.service, which reloads nginx through the nginx installer.
• Only one deploy hook exists (adbtd-mx-edge-reload.sh, for the mail edge); nothing app-specific needs a hook — certs are re-read by the nginx reload.
• quest.pukogames.com cert: CN=quest.pukogames.com, valid 2026-10-20 → 2027-01-18.
• No CAA record on pukogames.com (dig +short pukogames.com CAA → empty), so any ACME CA is allowed.

*.pukogames.com resolves (wildcard, section 5). With no default_server on 443, an unknown hostname is served by the first loaded 443 server block, which is sites-enabled/adbtd-admin (alphabetically first):

dig +short galaxy.pukogames.com            → agnes.pukogames.com. 195.154.152.46
curl -H 'Host: galaxy.pukogames.com' https://195.154.152.46/  → 307, location: /login  (adbtd-admin)
openssl s_client -servername galaxy.pukogames.com …          → subject=CN=admin.adbtd.com

So galaxy.pukogames.com already answers today — with the wrong cert and someone else's app. A new app needs its vhost + cert before anyone can use the name; there is no harmless "not configured" state.

Firewall (affects game traffic)

iptables/netfilter-persistent, persisted in /etc/iptables/rules.v4, INPUT policy DROP. Open: 22, 80, 443, 4242 (muse ssh), mail 25/465/587/143/993, udp 51820 (WireGuard wg-quest), plus -s 172.16.0.0/12 ACCEPT (Docker bridges) and one manual drop of 185.203.118.171.

No rule mentions the game ports 30000-30021, yet gamesrv-* containers are reachable because Docker publishes go through the DOCKER/PREROUTING DNAT path, not INPUT — ss -lnu shows 30006, 30007, 30013, 30015, 30021 bound and docker ps shows 0.0.0.0:30000-30021->…. [INFERENCE] on the mechanism, direct evidence on the published/bound ports. Consequence: a UDP game server published by compose needs no firewall edit; a host process bound directly (like cadastre on 0.0.0.0:3100) would be blocked from the internet.

---

2. Quest (quest.pukogames.com) — the reference deployment, end to end

Codename quest = the PuKoren/learn-chinese private repo (evidence: gh search code "deploy-remote.sh" --owner PuKoren returns PuKoren/learn-chinese:infra/agnes/deploy-remote.sh and docs/04-infra.md; the CI runner is registered to https://github.com/PuKoren/learn-chinese). Its own docs/04-infra.md is the authoritative narrative; this section is what the box actually shows.

Files on disk

Path	Owner:mode	Contents
/home/koren/quest/docker-compose.yml	koren 644	the whole stack, copied by every deploy from infra/agnes/docker-compose.yml
/home/koren/quest/deploy-remote.sh	koren 755	runs on agnes, copied by every deploy
/home/koren/quest/.env	koren 600	all secrets + runtime config, 5162 B, never in git; 11 rotated .env.bak.* copies next to it
/home/koren/quest/deploy.log	koren 644	<ISO8601> <sha7> per successful deploy (121 entries, last 2026-11-07T16:16:55Z 7912410)
/home/koren/quest/backup.sh	koren 755	nightly pg_dump (section 3)
/srv/quest/media	symlink → /data/quest/media	static media; nginx www-data cannot traverse /home/koren (700)

Compose stack

docker inspect confirms com.docker.compose.project.config_files=/home/koren/quest/docker-compose.yml, workdir /home/koren/quest, restart=unless-stopped on everything but quest-migrate.

Service	Image	Host binding	Notes
quest-postgres	postgres:17-alpine (17.11)	none	volume quest-pgdata, also on grafana_monitoring as quest-db
quest-migrate	quest-api:<sha>	none	one-shot node dist/migrate.mjs, restart: "no"
quest-api	quest-api:<sha>	127.0.0.1:3051	env_file: .env, 512 MB, bind /srv/quest/media/audio/tts-cache
quest-web	quest-web:<sha>	127.0.0.1:3050	nginx:80 in the container, 64 MB, wget healthcheck
quest-speech	quest-speech:<tag>	none	TTS/ASR on quest-net only, 3 GB / 4 CPUs at half weight, built on agnes (infra/agnes/speech-build.sh), pull_policy: never

Networks: quest-net (project network) and external grafana_monitoring. All logs json-file, 20 MB × 5. No systemd unit — restart policy + Docker is the supervisor.

Build, tag, ship

Images are never pushed to a registry. They are built on agnes' own docker daemon by a self-hosted GitHub Actions runner and tagged with the 7-char git sha:

• .github/workflows/deploy.yml (runs-on: [self-hosted, quest], concurrency group deploy-production): tag=$(git rev-parse --short=7 HEAD) → docker build -f apps/api/Dockerfile -t quest-api:$TAG . and -f apps/app/Dockerfile -t quest-web:$TAG . → installs a restrict-ed deploy key from secrets AGNES_SSH_KEY / AGNES_KNOWN_HOSTS → scp infra/agnes/{docker-compose.yml,deploy-remote.sh} koren@agnes:/home/koren/quest/ → ssh … bash /home/koren/quest/deploy-remote.sh $TAG → smoke test https://quest.pukogames.com/api/v1/health == {"status":"ok","database":"up"} and / contains <div id="root"> → deletes the key.
• Trigger: workflow_run of CI on main, or gh workflow run deploy.yml --ref main. Only main may dispatch.
• Runner: container quest-runner (quest-runner:local, 2.338.2, restart: unless-stopped, cpus: 4, cpu_shares: 512, mem_limit: 6g, /var/run/docker.sock mounted, home = bind /data/quest/runner/home), listening with label quest; plus a throwaway quest-ci-postgres on quest-ci-net. Registration is per repo (RUNNER_URL=https://github.com/PuKoren/learn-chinese).
• deploy-remote.sh <sha7>: requires quest-api:$TAG/quest-web:$TAG already loaded, docker compose up -d --remove-orphans with QUEST_TAG=$tag (migrate must exit 0), waits up to 120 s on quest-api+quest-web health, curls both on loopback, appends the tag to deploy.log, retags latest only after health passes, then prunes every other quest-api/quest-web tag except current, latest and the previous one (rollback slot).
• Rollback: ssh koren@agnes.pukogames.com '~/quest/deploy-remote.sh <previous sha7>', or cd ~/quest && QUEST_TAG=<sha7> docker compose up -d without retagging. Migrations are forward-only.
• Fallback without Actions: scripts/deploy.sh [--checks] [--speech] [<ref>] from a clone (git archive → /home/koren/quest/src, build on agnes, same script).

Image state on the box right now: quest-api:7912410 / quest-api:latest / quest-api:6175cf7 (265 MB) and quest-web:7912410 / :latest / :6175cf7 (108 MB), plus quest-speech:latest (2.43 GB).

Secrets

/home/koren/quest/.env (0600) is injected with env_file: into quest-api; compose interpolates POSTGRES_PASSWORD/AUTH_SECRET into DATABASE_URL too, and declares AUTH_SECRET/POSTGRES_PASSWORD with :? so a missing value fails the deploy loudly instead of crash-looping. Key names currently present (values not read): POSTGRES_PASSWORD, AUTH_SECRET, AUTH_BASE_URL, AUTH_TRUSTED_ORIGINS, APP_URL, CORS_ORIGINS, LOG_LEVEL, GOOGLE_CLIENT_ID/SECRET, APPLE_* (sign-in + IAP: APPLE_CLIENT_ID, APPLE_SIGNIN_*, APPLE_IAP_ENABLED, APPLE_KEY_ID, APPLE_ISSUER_ID, APPLE_PRIVATE_KEY, bundle ids…), GOOGLE_PLAY_*, STRIPE_SECRET_KEY, STRIPE_PUBLISHABLE_KEY, STRIPE_WEBHOOK_SECRET, PAYWALL_ENABLED, SMTP_HOST/PORT/USER/PASS/FROM.

Postgres data

Named volume quest-pgdata → /data/docker/volumes/quest-pgdata/_data (Docker data-root is /data/docker, so named volumes land on the big disk for free). Never bind-mounted from /home.

---

3. Postgres on prod

One instance per app, in Docker. No shared server. Eight Postgres containers, each with its own volume, none reachable from the internet:

Container	Image	Version	Host port
quest-postgres	postgres:17-alpine	17.11	none
quest-ci-postgres	postgres:17-alpine	17.11	none (CI only)
muse-postgres	postgres:17-alpine	17.11	none
cadastre-postgres	postgis/postgis:17-3.5	17.5	127.0.0.1:54320
wochew-db-1	pgvector/pgvector:pg16	16.15	127.0.0.1:5432
adbtd-postgres	postgres:16-alpine	16.13	none
gamesrv-postgres	postgres:16-alpine	16.13	none
elec-postgres	postgres:18-alpine	18.6	none

No host-installed Postgres (ps aux | grep postgres shows only container PIDs). /data/postgres (11 GB, uid 70) is an orphan left over from an older bind-mounted instance — no running container mounts it (docker inspect mount scan → empty).

Backups — what exists, exactly

Job	Mechanism	Schedule	Target	Retention
quest DB	koren's crontab (not /etc/cron.d) → /home/koren/quest/backup.sh	45 3 * * *	/home/koren/quest-backups/quest-YYYY-MM-DD.dump (pg_dump --format=custom inside the container, verified with pg_restore --list)	14 days (-mtime +14 -delete); 23 MB/dump, last 6 present
elec DB	koren crontab /home/koren/elec/infra/backup.sh	30 3 * * *	/home/koren/elec-backups/	script-owned
cadastre	systemd cadastre-db-backup.timer → scripts/backup-db.sh	daily ~04:19	/data/cadastre-backups (238 GB), TimeoutStartSec=6h	script-owned
adbtd (app + DB + maildirs)	root /etc/cron.d/adbtd-backup → /usr/local/bin/adbtd-backup.sh	0 3 * * *	Scaleway Object Storage s3://adbtd-backups/<date>/ via MinIO client mc, ~75 MiB/run	bucket-side 7 d, plus a Grafana system_health row per run
host config	root /etc/cron.d/agnes-host-backup → /usr/local/sbin/agnes-host-backup.sh	30 3 * * *	s3://<bucket>/host/<date>/: /etc/nginx, /etc/letsencrypt, fail2ban, iptables, custom systemd units, cron, ssh host keys, grafana-data, prometheus-data, manifest with sha256	bucket-side 7 d; last run 2026-11-09 02:31, 2367 MiB, 14 artefacts

No restic, no borg, no rsync-to-remote job (which restic borg → nothing; /usr/bin/rsync present but no job uses it for off-box copies). mc is the only S3 client.

Not backed up: quest media volume (/data/quest/media — regenerable), gameservers data (101 GB), muse, wochew, k0, elec's volumes beyond its own script, and every Postgres volume except quest's/elec's dump and adbtd's. The host-backup script states this explicitly in its header.

---

4. Disk layout

docker info | grep -i root   →  Docker Root Dir: /data/docker   (Storage Driver: overlayfs)
df -h  →  /dev/md1 40G 24G used 14G avail 63% /      /dev/md2 1.8T 1.1T used 692G avail 61% /data

du -sx -m per top-level entry of /data (2026-11-09):

Dir	Size	What
/data/docker	395 GB	Docker data-root: images, volumes, build cache
/data/cadastre-backups	238 GB	cadastre dumps (why /data is 61 % full)
/data/gameservers	101 GB	save/world data of the game-server containers
/data/cache	95 GB	misc cache
/data/postgres	11 GB	orphan, unused
/data/quest	4.4 GB	media + runner home
/data/loki	887 MB	Loki chunks
/data/wochew / /data/adbtd / /data/inkbs / /data/muse / /data/alpaca	2.2 G / 303 M / 298 M / 143 M / 65 M	app data

docker system df: images 74.45 GB (30.51 GB reclaimable), build cache 147.2 GB (133.6 GB reclaimable), local volumes 394.8 GB (763 MB reclaimable). docker builder prune alone would free ~130 GB — that is the cheapest headroom on the box.

/ holds only the OS, /home/koren/* project dirs and /var/log — 14 GB free, and the quest doc explicitly warns to be frugal there. Anything large must go to /data (bind mount or named volume; named volumes land on /data automatically).

---

5. DNS

Registrar/DNS provider: online.net = Scaleway (dig +short pukogames.com NS → ns0.online.net., ns1.online.net.; SOA ns0.online.net. hostmaster.pukogames.com. 2026110501). Mail: Google Workspace (MX 1 aspmx.l.google.com).

dig +short pukogames.com            → 195.154.152.46
dig +short agnes.pukogames.com      → 195.154.152.46
dig +short quest.pukogames.com      → 195.154.152.46
dig +short grafana.pukogames.com    → agnes.pukogames.com. 195.154.152.46
dig +short muse | gamesrv | wochew | k0 | cadastre | elec | eau  (same CNAME → agnes)
dig +short galaxy.pukogames.com           → agnes.pukogames.com. 195.154.152.46   ← wildcard
dig +short random-test-xyz.pukogames.com  → agnes.pukogames.com. 195.154.152.46   ← wildcard proof

• A wildcard *.pukogames.com exists (any label resolves to agnes; MyDNS-style CNAME to agnes.pukogames.com). A new subdomain therefore needs no DNS change at all — only an nginx vhost + cert.
• No AAAA on quest.pukogames.com (or the wildcard) — IPv4 only.
• No CAA record.
• Names that resolve to agnes (dig +short <name> A): pukogames.com (apex landing), agnes, quest, muse, gamesrv, wochew, k0, cadastre, elec, eau, grafana, alpaca, binance, inbks, promoter, lagrandecarte.fr, plus the adbtd.com family. Because the wildcard answers for every label, dig cannot tell an explicit record from wildcard fallback — the authoritative list of live names is the nginx vhost set (15) plus the 18 LE certs.
• No separate staging zone: every *.pukogames.com name points at production agnes. There is nowhere to put a staging host except a subdomain on the same box.

---

6. Resource headroom on prod

uptime      → load average: 1.32, 1.34, 1.10     (12 cores)
free -h     → total 31Gi, used 12Gi, free 497Mi, buff/cache 19Gi, available 18Gi
              Swap: 1.0Gi total, 1.0Gi used, 96Ki free     ← swap is exhausted
vmstat 1 3  → us 4-5  sy 1-7  id 84-93  wa 0-5
docker ps -q | wc -l → 64

CPU: ~90 % idle, 12 cores ⇒ roughly 11 cores of headroom in the current steady state. Memory: 18 GiB available, but swap is 100 % consumed, so real memory pressure exists already (the kernel has paged everything it can). Disk: 692 GB on /data, 14 GB on /.

Top consumers by CPU at sample time: TShock.Server (terraria, ~12 %), cadvisor (~11 %), valheim_server.x86_64 (~9 %), one wochew Postgres backend (~8 %).

Ports currently taken on prod (TCP): 22 25 80 110 143 443 465 587 993 995 3000 3001 3002 3010 3011 3020 3025 3030 3040 3050 3051 3060 3061 3091 3100 4000 4190 4242 5432 8080 8081 8891 8892 8893 9999 10001 10002 11332 11334 11335 18091 30000 30012 30015 30021 54320 56380 63790; UDP: 68 546 30006 30007 30013 30015 30021 51820. Free and safe for a new loopback-bound app: 127.0.0.1:3052-3090 (minus 3060/3061), 3092-3099, or 3200-3999; the 30000-30999 band belongs to gamesrv (GAMESERVER_PORT_MIN=30000, GAMESERVER_PORT_MAX=30999, PORT_STRIDE=3, allocated atomically in its Postgres). Note 3100 is cadastre (host node build/index.js on 0.0.0.0:3100, blocked externally by INPUT DROP).

---

7. Monitoring on prod

Stack lives in /home/koren/grafana (compose project grafana), network grafana_monitoring — which is the integration point: any app container joined to it is scrapable and its Postgres can be read by Grafana.

Container	Image	Exposure
grafana	grafana/grafana-oss:latest	127.0.0.1:3000 → https://grafana.pukogames.com
prometheus	prom/prometheus:latest	no host port; volume grafana_prometheus-data; --storage.tsdb.retention.time=30d --web.enable-admin-api
blackbox	prom/blackbox-exporter:latest	internal
node-exporter	prom/node-exporter:latest	internal, pid: host, /proc /sys / read-only, --collector.textfile.directory=/textfile (/var/lib/node_exporter/textfile)
cadvisor	gcr.io/cadvisor/cadvisor:latest	internal, privileged, /dev/kmsg, docker root read-only
nginx-exporter	nginx/nginx-prometheus-exporter:latest	scrapes http://172.22.0.1:8081/stub_status
loki	grafana/loki:latest	internal, user: "0", data /data/loki
promtail	grafana/promtail:latest	ships /data/docker/containers/* + /var/log to Loki

Prometheus scrape jobs (/home/koren/grafana/prometheus/prometheus.yml): prometheus, node-exporter:9100, cadvisor:8080, nginx:9113, and blackbox-http (30 s) probing the app health endpoints — including https://quest.pukogames.com/api/v1/health (service: quest-api) and /api/v1/health/speech.

Containers on grafana_monitoring right now: grafana prometheus blackbox loki promtail node-exporter cadvisor nginx-exporter quest-api quest-postgres adbtd-admin gamesrv-api gamesrv-orchestrator muse-api wochew-db-1.

Grafana provisioning is file-based for dashboards (/home/koren/grafana/grafana/dashboards/*.json, 15 of them: quest, quest-riders, server, docker, nginx, security, muse, muse-adoption, elec, wochew, cadastre, adbtd, adbtd-check, adbtd-campaigns, adbtd-traffic). The datasource list could not be read: curl -u admin:<compose password> /api/datasources → 401 password-auth.failed, so the password in docker-compose.yml/GF_SECURITY_ADMIN_PASSWORD is stale relative to the Grafana DB. [UNVERIFIED] on the exact datasource roster; from the compose comments and network aliases the intended ones are Prometheus, Loki, and Postgres datasources quest-db, cadastre-pg, adbtd-pg (each app's grafana_ro / read-only role).

The node_exporter textfile directory /var/lib/node_exporter/textfile holds agnes_host_backup.prom (written by the host-backup script at 03:31), i.e. the backup's own health metric. There is no alerting channel beyond GF_UNIFIED_ALERTING_ENABLED=true and SMTP mail.adbtd.com:587; the adbtd backup writes a system_health row so a missing backup fires an alert.

---

8. Dev box — koren@192.168.1.250

Arch Linux, 12 cores, 31 GiB, RTX 4060 8 GB, 11 containers, load 0.08/0.20/0.17. free -h: 11 Gi used, 19 Gi available; swap 4.0 Gi total / 4.0 Gi used / 5.9 Mi free (also exhausted).

Ports

Listening TCP: 22 53 631 1883 3000 3001 5000 5355 6379 8086 8096 8123 8189 8190 8554 8555 8920 9090 9191 11434 11435 18554 18555 27017 40835. Nothing is published through a proxy — every service binds 0.0.0.0 (or loopback) directly.

Free and safe for a dev stack: 3200-4999, 8200-8499, 9000-11433, 11500-18553, 18556-27016. Avoid 3000-3001 (sensors app + Grafana), 5000 (frigate), 8554-8555 (frigate RTSP), 8920 (emby), 9090/9191 (s3mock), 11434/11435 (ollama), 27017 (mongo).

Reverse proxy

None active. nginx is installed but /etc/nginx/ contains only mime.types and systemctl is-active nginx → inactive. caddy is installed (/usr/bin/caddy), disabled, with the stock welcome Caddyfile (serves /usr/share/caddy on http://) and an empty /etc/caddy/conf.d/. traefik: not installed. Dev services are reached by http://192.168.1.250:<port>.

Datastores

• Postgres: absent. pacman -Q postgresql → not found, systemctl is-enabled postgresql → not-found, no postgres container (docker ps -a | grep -i postgres → empty). Adding one means docker run postgres:17-alpine (or a compose service) — pattern exists on prod, nothing to reuse on dev.
• Present: mongo 4.4 :27017, redis 7 127.0.0.1:6379, mosquitto :1883, influxdb 2.7 :8086, ollama :11434 (+ muse-ollama 127.0.0.1:11435), frigate :5000, emby :8096/:8920, homeassistant, esp32-home-sensors :3000, sensors-grafana :3001, docker-s3mock :9090/:9191.

Toolchain

Tool	Version
node	v26.8.2
npm	12.0.2
bun	not installed
go	go1.27.1-X:nodwarf5 linux/amd64
rustc / cargo	1.98.1 (2026-09-01)
python3	3.14.7
docker / compose	29.8.0 / v5.5.1

GPU and 3D tooling

lspci | grep -i vga → 09:00.0 VGA compatible controller: NVIDIA Corporation AD107 [GeForce RTX 4060] (rev a1)
nvidia-smi → 615.71.09, CUDA 13.4, 196MiB / 8188MiB used, 0% util, 37 °C

Blender is not installed: which blender → nothing, ls /usr/bin/blender /usr/local/bin/blender → no such file, pacman -Q blender → error: package 'blender' was not found. No glTF/GLB pipeline tool is on the box either (comfyui exists for image generation only).

Docker root is /var/lib/docker on / (65 GB free) — the fast SSD, consistent with the owner's rule to keep hot files off /mnt/hdd (3.7 T HDD, very slow, used for media).

---

9. What shipping a new app on agnes actually costs

A "galaxy" deployment follows exactly the quest path — nothing new has to be invented, but these are the decisions already made for you:

1. Code lives at /home/koren/galaxy on agnes (compose + deploy-remote.sh + .env 0600 + deploy.log); source keeps infra/agnes/ copies.
2. App containers bind 127.0.0.1:<port> from the free band; only nginx sees the internet.
3. Static/large data → /data/galaxy (symlink /srv/galaxy if nginx must serve it, because /home/koren is 0700).
4. Postgres = its own postgres:17-alpine container + named volume (lands on /data/docker/volumes).
5. Vhost copied from quest.pukogames.com + certbot --nginx -d galaxy.pukogames.com + nginx -t + systemctl reload nginx. DNS needs nothing (wildcard).
6. Images built on agnes and tagged <sha7>; deploy = scp compose+script then deploy-remote.sh <sha7>; keep latest + previous tag, prune the rest.
7. Monitoring = join grafana_monitoring, expose /metrics, add a blackbox probe; logs = json-file 20 MB×5, promtail picks up galaxy-* automatically.
8. Backups = own cron + pg_dump --format=custom with pg_restore --list verification, 14-day retention, mc to Scaleway S3 if it must leave the box.

---

Gaps and risks

• galaxy.pukogames.com already serves adbtd-admin (307 → /login, cert CN=admin.adbtd.com) because 443 has no default_server; the new vhost + cert must land before the name is usable, and any other typo'd subdomain also hits that app.
• No staging environment: every *.pukogames.com resolves to production agnes, so there is no host or DNS name to test a release on before it is public.
• / has 14 GB free and Docker's data-root is on /data, so anything installed outside /data (node_modules caches, build outputs, /home/koren/<app>/src) eats the last of the root disk.
• Swap is fully consumed on prod (1.0 Gi/1.0 Gi, 96 KiB free) and on dev (4.0 Gi/4.0 Gi, 5.9 Mi free): memory headroom is available, not free RAM — a memory-hungry game process will page from the start.
• /data is 61 % full and 133 GB of it is stale Docker build cache (147 GB total build cache); nothing prunes it automatically, and cadastre-backups alone is 238 GB.
• Backups are per-app and manual to adopt: no shared backup service, no restic/borg, off-box copies only for adbtd and host config (7-day bucket retention); a new app silently has none until someone writes a cron.
• No registry: images only exist on agnes' daemon; losing /data/docker means rebuilding, and a rollback only works for the one previous tag kept per repo.
• The self-hosted CI runner is per repo (RUNNER_URL=https://github.com/PuKoren/learn-chinese, label quest, one job at a time, no fork PRs, host docker socket = root on agnes); a Galaxy repo needs its own runner registration, and a shared runner would serialize deploys with quest.
• TLS automation is certbot-with-nginx only: certbot --nginx rewrites the vhost file in place, so a hand-edited vhost must be backed up before the first issuance, and there is no CAA record to restrict CAs (fine for LE, worth knowing).
• Firewall has no allowance for host-bound game traffic: only 22/80/443/4242/mail/51820 are ACCEPTed; anything bound by a host process (not Docker) is unreachable until rules.v4 is edited by hand (never netfilter-persistent save).
• UDP game ports collide with gamesrv: the 30000-30999 band is owned by the existing game-server orchestrator (PORT_STRIDE=3, atomic allocation in its Postgres); Galaxy's dedicated-server ports must be allocated elsewhere or registered with it.
• Blender is missing on dev, and so is any glTF pipeline: the low-poly ship/station asset pipeline has no tooling on either box today.
• Dev box has no Postgres and no reverse proxy, and its Docker root is on the 65 GB root SSD — a local stack with a database will need both added, with data pointed at /mnt/hdd or a new volume.
• Grafana's admin password in the compose file no longer matches the DB (401 password-auth.failed), so dashboards cannot be managed through the API from the box until someone resets it in the UI.
• No monitoring of the box's own headroom: nothing alerts on /data filling, swap exhaustion or a missing app backup except the adbtd system_health row.