Fleet baseline runbook

The common build for all four OCI hosts: swap, SSH hardening, UFW, Fail2Ban, NGINX with a /health endpoint, Certbot TLS, the Projects Monitor collector, unattended upgrades and the weekly maintenance timer. Anything vinhedo1-specific (PiVPN / WireGuard, Grimoire API) is in the vinhedo1 build runbook.

Where a phase is already scripted in this repo, run the script — don’t retype it. The scripts are idempotent and are the source of truth; this page only says when to run them.

Host matrix

Host Public IPv4 FQDN Workload Extras
vinhedo1 144.22.174.243 vinhedo1.apibr.com Fresh rebuild (Aug 2026) PiVPN + WireGuard (51820/udp); Grimoire API later
vinhedo2 147.15.44.128 vinhedo2.apibr.com NGINX → webhooks.straccini.com, Docker UFW + Docker caveat, see Phase 3
vinhedo3 157.151.24.216 vinhedo3.apibr.com Idle — nothing listening but SSH Still on stock OCI iptables, no UFW
vinhedo4 157.151.18.221 vinhedo4.apibr.com NGINX → api.handler.straccini.com, PHP-FPM 8.4, webhooks-* units Leave until last when rebooting

All four: Ubuntu 24.04 LTS, VM.Standard.E2.1.Micro (1 GB RAM, 2 vCPU as seen by nproc), 45 GB ext4 boot volume, timezone Etc/UTC, login as ubuntu with passwordless sudo.

Where each host stands (surveyed 2026-09-12)

Phase vinhedo1 vinhedo2 vinhedo3 vinhedo4
1.1 Swap ✓ ✓ ✓ ✓
1.2 Journal cap ✓ — — —
1.3 needrestart ✓ — — —
2 SSH hardening ✓ — — ✓
3 UFW (OCI rules purged) ✓ ✓ — ✓
4 Fail2Ban ✓ (+recidive) ✓ — ✓
5 NGINX /health ✓ (text) site for another domain no nginx site for another domain
6 Certbot ✓ ✓ (other domain) — ✓ (other domain)
7 Projects Monitor collector ✓ ✓ ✓ ✓
8.1 Unattended auto-reboot ✓ 04:30 — — —
8.2 Weekly maintenance timer ✓ ✓ ✓ ✓

vinhedo1 was a clean build; vinhedo2 and vinhedo4 are not. Phases 3 and 5 behave differently on a host that already has services listening — the pre-flight audits there are not optional. Enabling UFW blind on vinhedo4 cuts off the webhooks ingress; flushing iptables on vinhedo2 without restarting Docker breaks container networking.

Run everything as ubuntu with sudo. Keep a second SSH session open throughout Phase 3 on every host.

Phase 0 — Prerequisites (per host, before touching anything)

0.1 DNS

Each vinhedoN.apibr.com needs an A record pointing at that host’s current IP. vinhedo1’s changed with the rebuild. All four currently match. From the laptop (PowerShell):

foreach ($n in 1..4) {
  $h = "vinhedo$n.apibr.com"
  "{0,-22} dns={1,-16} actual={2}" -f $h,
    ((Resolve-DnsName $h -Type A).IPAddress -join ","),
    (ssh -o ConnectTimeout=5 "vinhedo$n" 'curl -s -m 5 ifconfig.me')
}

Mismatches must be fixed before Phase 6 — Certbot’s HTTP-01 challenge resolves the name from the public internet.

0.2 OCI Security Lists / NSGs

UFW is irrelevant if OCI drops the packet upstream. Stateful ingress from 0.0.0.0/0 on every host:

Protocol Port Hosts
TCP 22 all
TCP 80 all
TCP 443 all
UDP 51820 vinhedo1 only

Plus whatever application ports each host already exposes — check the Phase 3.1 audit before assuming.

Phase 1 — Base system (all hosts) 

sudo apt update && sudo apt full-upgrade -y

sudo apt install -y \
  curl wget ca-certificates gnupg git jq \
  vim htop iputils-ping dnsutils \
  unattended-upgrades apt-listchanges needrestart \
  python3-systemd

Timezone. All four hosts are on Etc/UTC today. Timer times in this runbook (OnCalendar, Automatic-Reboot-Time) are interpreted in the host’s local zone, so whatever you pick, keep the fleet consistent. If you want local time:

sudo timedatectl set-timezone Europe/Dublin

Hostname. Set it explicitly to the alias (OCI usually already does this from the instance name) and make it resolvable so sudo doesn’t stall:

H=vinhedo1   # <-- set per host
sudo hostnamectl set-hostname "$H"
grep -q "127.0.1.1 $H" /etc/hosts || echo "127.0.1.1 $H $H.apibr.com" | sudo tee -a /etc/hosts

1.1 Swap — scripted

The E2.1.Micro shape ships with no swap. Run from the laptop:

cd baseline
./deploy-swap.sh                 # all four; idempotent, skips hosts with active swap
VERIFY_ONLY=1 ./deploy-swap.sh   # just report

That gives every host a 2 GB /swapfile in /etc/fstab plus vm.swappiness=10 and vm.vfs_cache_pressure=50 in /etc/sysctl.d/99-swap.conf. See baseline/setup-swap.sh for the per-host logic. Done on all four as of 2026-09-12.

1.2 Cap the journal 

sudo mkdir -p /etc/systemd/journald.conf.d
printf '[Journal]\nSystemMaxUse=200M\nSystemMaxFileSize=50M\n' \
  | sudo tee /etc/systemd/journald.conf.d/size.conf
sudo systemctl restart systemd-journald

Worth doing on vinhedo4 first — a crash-looping worker can fill a small boot volume fast. Only vinhedo1 has this today.

1.3 Stop needrestart prompting during unattended upgrades 

sudo sed -i "s/^#\?\$nrconf{restart}.*/\$nrconf{restart} = 'a';/" /etc/needrestart/needrestart.conf

Phase 2 — SSH hardening (all hosts)

Ubuntu 24.04 uses socket activation for SSH, and OCI images already drop 60-cloudimg-settings.conf under /etc/ssh/sshd_config.d/. Add a higher-numbered drop-in so it wins:

sudo tee /etc/ssh/sshd_config.d/99-hardening.conf >/dev/null <<'EOF'
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
X11Forwarding no
MaxAuthTries 3
ClientAliveInterval 300
ClientAliveCountMax 2
EOF

sudo sshd -t && sudo systemctl restart ssh

AllowUsers is deliberately omitted here. If you add it, list every account that needs SSH (ubuntu, projects_monitor, and on vinhedo4 also webhooks) — the collector’s setup script appends projects_monitor to an existing AllowUsers line, but it can’t know about your other service accounts. Add it after Phase 7.

Before applying PermitRootLogin no anywhere, confirm you log in as a non-root user with a key and can sudo. All four currently log in as ubuntu.

If you ever want SSH on a non-standard port, on 24.04 you must edit the socket, not just sshd_config: sudo systemctl edit ssh.socket → [Socket] / ListenStream= (empty, to clear) / ListenStream=2222.

Phase 3 — UFW (all hosts)

3.1 Audit first — mandatory on vinhedo2 and vinhedo4 

sudo ss -tulpn | grep LISTEN

Write down every port bound to 0.0.0.0 or *. Anything listening only on 127.0.0.1 or 10.x doesn’t need a UFW rule. What to expect:

  • All hosts: rpcbind on 111/tcp+udp, pulled in by the OCI image. Nothing here uses NFS, so the default deny is enough — or remove it: sudo systemctl disable --now rpcbind.socket rpcbind.
  • vinhedo2: NGINX on 80/443 fronting webhooks.straccini.com; Docker. Docker publishes container ports straight into iptables, bypassing UFW, so a -p 8080:80 container is public whether UFW allows it or not. Prefer binding published ports to 127.0.0.1: and proxying via NGINX.
  • vinhedo4: NGINX on 80/443 fronting api.handler.straccini.com; PHP-FPM 8.4 on a unix socket; webhooks-* units.

3.2 Clear the OCI iptables rules

Canonical’s OCI images ship iptables-persistent rules with a REJECT at the end of INPUT, sitting below UFW’s chain and silently dropping traffic UFW believes it allowed. vinhedo3 still has these; the other three are clean.

sudo iptables -L INPUT -n --line-numbers        # look before you flush
sudo iptables -P INPUT ACCEPT                   # policies first, then flush —
sudo iptables -P FORWARD ACCEPT                 # never leaves you behind a DROP
sudo iptables -P OUTPUT ACCEPT
sudo iptables -F && sudo iptables -X
sudo apt purge -y netfilter-persistent iptables-persistent

If Docker is installed (vinhedo2): the flush also removes Docker’s DOCKER* chains and the FORWARD rules containers rely on. Run sudo systemctl restart docker afterwards so it recreates them.

3.3 Configure and enable 

sudo apt install -y ufw

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp  comment 'HTTP'
sudo ufw allow 443/tcp comment 'HTTPS'

# vinhedo1 only
sudo ufw allow 51820/udp comment 'WireGuard'

# Add one line per additional public port found in 3.1, e.g.:
# sudo ufw allow 8080/tcp comment 'webhooks ingress'

sudo ufw logging low
sudo ufw enable
sudo ufw status verbose

From your second session, confirm SSH still works before moving on. Then confirm the host’s own services still respond — on vinhedo4, journalctl -u webhooks-service -n 20 and a test webhook; on vinhedo2, docker ps and a request through NGINX.

Phase 4 — Fail2Ban (all hosts)

Two Ubuntu 24.04 gotchas: there is no /var/log/auth.log (journald only), and the package doesn’t pull in python3-systemd, so the sshd jail silently fails to read the journal without it. Phase 1 installed it.

sudo apt install -y fail2ban

sudo tee /etc/fail2ban/jail.local >/dev/null <<'EOF'
[DEFAULT]
backend   = systemd
banaction = ufw
bantime   = 1h
findtime  = 10m
maxretry  = 5
# loopback, the WireGuard subnet, the fleet itself, and the Projects Monitor host (osasco)
ignoreip  = 127.0.0.1/8 ::1 10.141.230.0/24 144.22.174.243 147.15.44.128 157.151.24.216 157.151.18.221 177.73.237.236

[sshd]
enabled  = true
mode     = aggressive
maxretry = 3
bantime  = 24h

[recidive]
enabled  = true
bantime  = 1w
findtime = 1d
maxretry = 5
EOF

sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd

Whitelisting the fleet means a fumbled inter-host script can’t ban a machine you rely on. Whitelisting osasco (177.73.237.236) means a broken collector key on the Projects Monitor side degrades to “no data” rather than a 24-hour ban that also blinds the dashboard. vinhedo1’s current ignoreip has loopback and 10.6.0.0/24 — but PiVPN actually chose 10.141.230.0/24, so the VPN is not whitelisted at all today. Bring it in line.

After Phase 5, append the NGINX jails:

sudo tee -a /etc/fail2ban/jail.local >/dev/null <<'EOF'

[nginx-http-auth]
enabled = true

[nginx-botsearch]
enabled = true
EOF
sudo systemctl restart fail2ban
sudo fail2ban-client status

vinhedo2 and vinhedo4 have the sshd and NGINX jails but not recidive; add it there.

Phase 5 — NGINX and the /health endpoint (all hosts)

5.1 Install (skip if already present) 

command -v nginx || sudo apt install -y nginx
sudo systemctl enable --now nginx

On a host that already serves traffic (vinhedo2, vinhedo4), do not remove sites-enabled/default or existing site files. Add the vinhedoN.apibr.com server block alongside them; NGINX picks the block by server_name.

5.2 Hardening drop-in 

sudo tee /etc/nginx/conf.d/00-hardening.conf >/dev/null <<'EOF'
server_tokens off;
client_max_body_size 20m;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
EOF

5.3 Site with /health

The $host in the health payload makes the response self-identifying, so a monitor hitting all four gets distinguishable output. location = /health is an exact match so /healthz or /health/foo fall through to the app.

H=$(hostname -s)

sudo tee /etc/nginx/sites-available/$H.apibr.com >/dev/null <<EOF
server {
    listen 80;
    listen [::]:80;
    server_name $H.apibr.com;
    root /var/www/$H;
    index index.html;

    location = /health {
        default_type application/json;
        access_log off;
        return 200 '{"status":"ok","host":"$H"}\n';
    }

    location / { try_files \$uri \$uri/ =404; }
}
EOF

sudo mkdir -p /var/www/$H
echo "<h1>$H</h1>" | sudo tee /var/www/$H/index.html
sudo ln -sf /etc/nginx/sites-available/$H.apibr.com /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

curl -s -H "Host: $H.apibr.com" http://localhost/health

The -H Host: matters on vinhedo2/4, where localhost would otherwise hit the other site. It stops working once Certbot adds the HTTP→HTTPS redirect in Phase 6; after that use the public URL.

vinhedo1 currently serves a plain-text ok from a prefix location /health — switch it to the JSON block above so all four look alike (UptimeRobot’s keyword check for ok still matches).

On vinhedo1 and vinhedo3 only (nothing else served) you can also drop the default site:

sudo rm -f /etc/nginx/sites-enabled/default && sudo nginx -t && sudo systemctl reload nginx

Phase 6 — TLS with Certbot (all hosts)

The apt package is simpler than snap on a Minimal image and its renewal timer is enabled automatically.

sudo apt install -y certbot python3-certbot-nginx
H=$(hostname -s)
sudo certbot --nginx -d $H.apibr.com --agree-tos -m <your-email> --redirect

sudo certbot renew --dry-run
systemctl list-timers certbot.timer
curl -sI https://$H.apibr.com/health | head -1

On vinhedo2 and vinhedo4 Certbot already manages a certificate for the app domain; this adds a second certificate and a second renewal, nothing else changes. --redirect only touches the vinhedoN.apibr.com server block.

If issuance fails, the cause is almost always Phase 0: DNS pointing at the wrong IP, or port 80 closed in the OCI Security List.

Phase 7 — Projects Monitor collector — scripted (all hosts)

From the laptop, in projects-monitor-access-to-OCI/:

./deploy-projects-monitor.sh               # provision / update all four
VERIFY_ONLY=1 ./deploy-projects-monitor.sh # collect a snapshot from each

Same account, same key, same monitor-report on all four; WireGuard is detected at runtime, so vinhedo2/3/4 report "wireguard": {"available": false} and there is no host-specific variant to maintain. What the scripts do is documented in Projects Monitor Access.

Per-host variation lives in /etc/projects-monitor/services.conf, auto-detected on first run and never overwritten afterwards. After adding NGINX or Fail2Ban to a host that was provisioned before they existed (vinhedo3), add them by hand:

ssh vinhedo3 'sudo sed -i "s/^MONITOR_SERVICES=\"/&nginx fail2ban /" /etc/projects-monitor/services.conf && cat /etc/projects-monitor/services.conf'

Confirm the script is identical everywhere:

for h in vinhedo1 vinhedo2 vinhedo3 vinhedo4; do
  printf '%-10s %s\n' "$h" "$(ssh $h sha256sum /usr/local/bin/monitor-report | cut -c1-16)"
done

Phase 8 — Staying updated (all hosts)

8.1 Unattended upgrades with a staggered reboot

Only vinhedo1 has this today; the others run the stock config with Automatic-Reboot off, so a kernel CVE sits unpatched until someone reboots by hand.

REBOOT_AT=04:00   # <-- stagger per host: 04:00 vinhedo1, 04:15 vinhedo2, 04:30 vinhedo3, 04:45 vinhedo4

sudo tee /etc/apt/apt.conf.d/51-my-unattended-upgrades >/dev/null <<EOF
Unattended-Upgrade::Allowed-Origins {
    "\${distro_id}:\${distro_codename}";
    "\${distro_id}:\${distro_codename}-security";
    "\${distro_id}:\${distro_codename}-updates";
};
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-WithUsers "false";
Unattended-Upgrade::Automatic-Reboot-Time "$REBOOT_AT";
EOF

sudo tee /etc/apt/apt.conf.d/20auto-upgrades >/dev/null <<'EOF'
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::AutocleanInterval "7";
EOF

sudo unattended-upgrade --dry-run --debug | tail -20

The reboot slots sit after the weekly maintenance window (Sun 03:00–04:15 across the fleet, see 8.2) and before apt-daily-upgrade (~06:00–07:00), so no two things fight over the box. vinhedo1 is currently set to 04:30 — move it to 04:00 when you touch the others.

No Ubuntu Pro means no Livepatch, so the automatic reboot is the only path by which kernel CVEs actually take effect. On vinhedo4, confirm the webhooks-* units all have Restart=always and come back cleanly after a reboot — otherwise set Automatic-Reboot "false" there and reboot it by hand when the weekly maintenance report says REBOOT REQUIRED.

Without Pro you get noble-updates and noble-security only. That covers main; universe packages (Fail2Ban among them) are community-maintained for security fixes.

8.2 Weekly maintenance + healthchecks.io — scripted

From the laptop, in weekly-maintenance/:

./deploy-weekly-maintenance.sh              # all four
TEST_RUN=1 ./deploy-weekly-maintenance.sh   # deploy, then fire once to check healthchecks.io

This installs the maintenance script, the per-host healthchecks.io URL, a timer staggered 20 minutes apart per host, and the boot/shutdown ping. Details in Weekly Maintenance. Already deployed on all four.

8.3 UptimeRobot

Point one monitor per host at https://vinhedoN.apibr.com/health, expecting keyword ok. That covers NGINX, TLS expiry and basic reachability in one check.

8.4 What still needs you

Unattended upgrades don’t cross release boundaries. 24.04 LTS is supported to mid-2029; wait for 26.04.1 and run do-release-upgrade interactively, one host at a time.

Phase 9 — Verification

Per host:

sudo ufw status verbose
sudo fail2ban-client status
sudo systemctl is-enabled nginx fail2ban unattended-upgrades certbot.timer weekly-maintenance.timer
curl -s https://$(hostname -s).apibr.com/health
free -h
systemctl list-units --state=failed --no-legend

From the laptop:

for h in vinhedo1 vinhedo2 vinhedo3 vinhedo4; do
  printf '%-10s %s\n' "$h" "$(ssh $h curl -s -m 5 https://$h.apibr.com/health)"
done

VERIFY_ONLY=1 baseline/deploy-swap.sh
VERIFY_ONLY=1 projects-monitor-access-to-OCI/deploy-projects-monitor.sh

(curl.exe from a Windows shell may fail TLS handshakes to these hosts; running the check through SSH sidesteps that.)

Then reboot each host one at a time, re-running the checks after each, to prove everything returns unattended:

ssh vinhedo3 'sudo reboot'

Leave vinhedo4 until last, and watch the webhooks workers come back:

ssh vinhedo4 'systemctl list-units "webhooks-*" --no-pager'

vinhedo1 has extra checks — see the vinhedo1 build runbook.

This site uses Just the Docs, a documentation theme for Jekyll.