Skip to content

ADR 011: Every job on the worker VM, by priority

  • Status: Accepted
  • Date: 2026-10-23
  • Deciders: Owner
  • Technical Context: apps/server (Solid Queue, config/deploy.yml), infra/worker, infra/basemap, infra/depth, infra/oci; extends ADR 008, moves the build of ADR 004 out of GitHub Actions

Context

The worker VM (ADR 008, live 2026-10-22) ran only the depth build. Every Rails job (Solid Queue) still ran inside Puma on the small app VM (1 OCPU / 6 GB), on one default queue: an OpenStreetMap refresh (an hour of downloads and osmium) or a backup competed with watch uploads and with the web requests themselves. The monthly basemap extract ran in GitHub Actions. The owner wants the worker VM to run all jobs, with the tile builds at the lowest priority and upload processing at the highest.

Decision

  • Rails jobs on the worker VM: a Kamal role jobs of the server app (bin/jobs) on ssh-worker.sailever.com, same image and secrets as web. The worker VM reaches Postgres: NSG db from NSG worker, the db VM's iptables and pg_hba.conf from the private subnet, TLS as for the app.
  • Queues by priority: realtime (TrackSegmentIngest, TrackRecompute, FitBatchImport, AccountErasure) in its own process; default (wind, tides, segment sweep) before background (OSM places, backups, cleanups) in a second one. The recurring schedule runs only on the worker VM.
  • Fallback on the app VM: one Puma thread works the realtime queue only, so uploads are processed while the worker VM is down or rebooting (owner's choice, 2026-10-23).
  • Tile builds last: the worker's own service (depth layer, and now the basemap) gets 128 CPU shares against the Rails jobs' 1024, a memory cap of 8 of 12 GB, nice 19 and idle I/O. The two builds share one lock and never run at the same time (disk).
  • Basemap on the worker VM: infra/basemap replaces the build and cleanup jobs of tiles.yml, with five latitude bands of at most about 10 GB instead of three (the worker has less free disk than a runner), smoke test before the switch, and a daily cleanup two days after a switch.

Consequences

  • The app VM only answers requests (plus the one fallback thread); a long job no longer slows the website.
  • A worker VM outage delays everything but realtime: wind, tides, backups and cleanups catch up when it is back; a missed 03:00 backup runs at the next one.
  • Deploys of the server also deploy the jobs role to the worker VM; data tasks (deploy:after, OSM modes) run with app exec --primary, on the app VM only.
  • No GitHub Actions minutes or secrets for the tiles any more; the R2 token is used only by the worker.
  • More basemap band files (5) per version; the tiles Worker reads their headers for tiles.json once an hour.
  • The Always Free A1 capacity stays fully used (app 1 OCPU, db 1, worker 2); more job load means a bigger worker or paid capacity.