Webhooks — Production VM Setup, From Scratch

Target: example.com · Ubuntu 24.04 LTS · Nginx + PHP 8.4-FPM (no Apache/.htaccess anywhere in the stack) · systemd daemons · GitHub Actions over SSH, deploying via git pull + composer install directly on the box.

This reflects the current state of this repo — scripts/setup-vps.sh and .github/workflows/deploy.yml do almost everything below automatically. It assumes zero users on the VM, so this is the exact order of operations for a fresh box.

GitHub Actions secrets checklist

Secret Purpose
SSH_HOST VM’s public IP or hostname
SSH_PORT Usually 22, or a custom port if you hardened SSH
SSH_USER Set to webhooks — GitHub Actions SSHes in directly as the app-owning user, no separate deploy account
SSH_PRIVATE_KEY Private half of the CI → VM keypair (Part E)
JOB_RUNNER_TOKEN Consumed by Src/lib/runner.php for HTTP-triggered worker runs — generate with openssl rand -hex 32
HEALTH_CHECK_URL Base URL used by the post-deploy Swagger check (e.g. https://example.com) — the workflow fails at that step if this is unset

FTP_SERVER, FTP_USERNAME, FTP_PASSWORD, and INSTALLATION_ENDPOINT are leftovers from the old FTP-based deploy and referenced nowhere in the current workflow — delete them once you’re confident in the cutover.

Part A — Oracle Cloud: instance & network

  1. Instance: Compute → Instances. Reuse an existing Always-Free instance or create fresh: shape VM.Standard.E2.1.Micro (Always Free x86) or a paid E4/E5 flex shape, image Canonical Ubuntu 24.04, attach your SSH public key at creation.
  2. Reserve the public IP: Networking → IP Management → reserve as a Reserved Public IP so it survives stop/start and so the IP you whitelist on the DB host (Part C) doesn’t change later. Point your domain’s DNS A record at it now (low TTL, e.g. 300s).
  3. Open the cloud-level firewall (Security List / NSG — separate from anything configured inside the VM):
    • Add stateful ingress: TCP/22 (ideally restricted to your own IP CIDR), TCP/80, TCP/443 from 0.0.0.0/0.
    • Leave egress open — outbound AMQPS (5671) to CloudAMQP, HTTPS to GitHub/OneSignal/HealthChecks.io, and the MariaDB port to your DB host.
  4. Oracle’s default-iptables gotcha: the stock Ubuntu image ships iptables rules (via netfilter-persistent) that allow only SSH + ICMP inbound, independently of the Security List and of UFW. Handle this in Part B before installing Nginx, or port 80/443 will be silently dropped even after ufw allow.

Part B — First login & OS hardening 

ssh -i ~/.ssh/your_key ubuntu@<VM_PUBLIC_IP>
sudo apt-get update && sudo apt-get -y upgrade

The default ubuntu user is already sudo-enabled and key-only — use it directly for provisioning. No separate deploy user is needed: GitHub Actions connects as webhooks itself, set up in Part E.

SSH hardening

/etc/ssh/sshd_config.d/99-hardening.conf:

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
X11Forwarding no
MaxAuthTries 3
AllowUsers ubuntu webhooks

sudo systemctl restart ssh

Test the new connection in a second terminal before closing your current session.

Gotcha: AllowUsers rejects a user before sshd even looks at authorized_keys — so if this line ever lists a renamed or removed account, GitHub Actions gets Permission denied (publickey) even with a perfectly correct key installed, and it looks exactly like a bad-key problem. Confirm with sudo journalctl -u ssh -n 30 — the real error reads User webhooks ... not allowed because not listed in AllowUsers, not a pubkey rejection. Fix: sudo sed -i 's/AllowUsers .*/AllowUsers ubuntu webhooks/' /etc/ssh/sshd_config.d/99-hardening.conf && sudo sshd -t && sudo systemctl reload ssh.

UFW — resolve Oracle’s pre-installed iptables rules first 

sudo apt-get install -y ufw
sudo systemctl disable --now netfilter-persistent 2>/dev/null || true
sudo iptables -F && sudo iptables -X
sudo ip6tables -F && sudo ip6tables -X
sudo iptables -P INPUT ACCEPT && sudo iptables -P FORWARD ACCEPT && sudo iptables -P OUTPUT ACCEPT

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw logging on
sudo ufw enable

Fail2ban 

sudo apt-get install -y fail2ban

/etc/fail2ban/jail.local:

[DEFAULT]
bantime  = 1h
findtime = 10m
maxretry = 5
backend  = systemd

[sshd]
enabled = true

[nginx-http-auth]
enabled = true

[nginx-botsearch]
enabled = true

sudo systemctl enable --now fail2ban

Housekeeping 

sudo apt-get install -y unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades
sudo timedatectl set-timezone Etc/UTC

Part C — Reaching the MariaDB host by IP whitelist (no VPN)

  1. Firewall on the DB host: allow inbound TCP/3306 from this VM’s Reserved Public IP specifically.
  2. MySQL/MariaDB grants: 
    CREATE USER IF NOT EXISTS 'webhooks'@'<VM_PUBLIC_IP>' IDENTIFIED BY '<password>';
GRANT ALL PRIVILEGES ON <database>.* TO 'webhooks'@'<VM_PUBLIC_IP>';
FLUSH PRIVILEGES;
  3. Confirm from the VM: mysql -h <db-host> -P 3306 -u webhooks -p -e "SELECT 1"
  4. Enforce TLS in transit (no VPN means plaintext otherwise): ALTER USER 'webhooks'@'<VM_PUBLIC_IP>' REQUIRE SSL; on the DB host.

Src/secrets/mySql.secrets.php on this VM (written by CI, Part E) points $mySqlHost at the DB host’s real address, not 127.0.0.1.

Part D — PHP 8.4 

sudo apt-get install -y software-properties-common
sudo add-apt-repository -y ppa:ondrej/php
sudo apt-get update
sudo apt-get install -y \
  php8.4-cli php8.4-fpm php8.4-common \
  php8.4-mbstring php8.4-curl php8.4-mysql php8.4-bcmath \
  php8.4-sockets php8.4-xml php8.4-zip php8.4-opcache

php -v
php -m | grep -E 'pcntl|sockets'   # confirm both are loaded

Both git and Composer are needed on this VM, not just in CI — setup-vps.sh installs them for you in the next step. deploy.yml runs git pull + composer install directly over SSH on every push; nothing is built in CI and shipped over.

Part E — Run setup-vps.sh (does almost everything)

This one script creates the webhooks user, both SSH keypairs’ plumbing, clones the repo, installs Composer dependencies, writes and enables the full Nginx site (HTTP-only for now), installs and enables all four systemd units, and installs a scoped sudoers policy — all idempotently. Because of the chicken-and-egg key registration below, you’ll run it twice.

scp scripts/setup-vps.sh ubuntu@<VM_PUBLIC_IP>:/tmp/setup-vps.sh
ssh ubuntu@<VM_PUBLIC_IP> 'sudo bash /tmp/setup-vps.sh'

First run creates the webhooks user and prints a new public key — this is the VM’s own outbound key so it can git pull the private repo. Add it read-only at your repo’s Settings → Deploy keys page.

It also creates an empty authorized_keys for the opposite direction (GitHub Actions connecting into this VM) and asks you to populate it with the public half of a separate keypair you generate yourself:

ssh-keygen -t ed25519 -C "github-actions-deploy" -f deploy_key -N ""
ssh ubuntu@<VM_PUBLIC_IP> "sudo tee -a /opt/webhooks/.ssh/authorized_keys" < deploy_key.pub

The matching private half (deploy_key) goes into the SSH_PRIVATE_KEY GitHub secret.

The script then aborts at the git fetch step, by design, until the first key is registered.

Second run (after registering the deploy key): completes the clone, Composer install, Nginx site (enabled and reloaded — plain HTTP), systemd units (installed + enabled, will crash-loop harmlessly every 10s until secrets exist), and the sudoers policy at /etc/sudoers.d/webhooks-deploy:

webhooks ALL=(webhooks) NOPASSWD: ALL
webhooks ALL=(root) NOPASSWD: /usr/bin/chown -R webhooks:webhooks /opt/webhooks, /usr/bin/systemctl restart webhooks-service, /usr/bin/systemctl restart webhooks-cleanup, /usr/bin/systemctl restart webhooks-database-service, /usr/bin/systemctl restart webhooks-maintenance, /usr/bin/systemctl reload php8.4-fpm

That’s the entire privilege footprint — nothing broader. Verify any time with ssh webhooks@<VM_PUBLIC_IP> sudo -l.

If you ever re-copy and re-run setup-vps.sh after pulling a newer version of the script, it’s safe — every step is idempotent, and re-running is exactly how you pick up fixes (e.g. a missing sudoers policy) without hand-editing the VM.

Part F — TLS

Nginx is already serving your domain over plain HTTP after Part E. Issue the cert and let certbot upgrade that same server block in place (adds the 443 listener + redirect, preserves every rewrite/deny rule already there — nothing to hand-edit):

sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d example.com --redirect -m you@example.com --agree-tos

Part G — PHP-FPM pool (still manual)

Edit /etc/php/8.4/fpm/pool.d/www.conf:

user = webhooks
group = webhooks
listen = /run/php/php8.4-fpm.sock
listen.owner = www-data
listen.group = www-data

sudo systemctl restart php8.4-fpm

Cutover checklist

  1. Push to main (or gh workflow run deploy.yml). Watch it SSH in as webhooks, git pull, composer install, write secrets, restart the four daemons, reload PHP-FPM.
  2. sudo systemctl status webhooks-service webhooks-cleanup webhooks-database-service webhooks-maintenance — all active (running).
  3. curl https://example.com/api/health → {"status":"Healthy", ...}.
  4. Smoke-test /api/v1/swagger and /api/v1/broker (with your Authorization header).
  5. Re-point external webhook sources (GitHub App URL, HealthChecks.io, etc.) to your domain.
  6. journalctl -u webhooks-service -f and tail -f /var/log/nginx/webhooks.error.log while watching first real traffic.
  7. Once stable for a few days: delete the old FTP_*/INSTALLATION_ENDPOINT secrets and decommission any previous host.

Appendix — quick reference 

# Service status / logs
sudo systemctl status webhooks-service
sudo journalctl -u webhooks-cleanup -f

# Sudoers sanity check
ssh webhooks@<VM_PUBLIC_IP> sudo -l

# Firewall
sudo ufw status verbose
sudo fail2ban-client status sshd

# TLS renewal (automatic via systemd timer, manual test)
sudo certbot renew --dry-run

# DB reachability
mysql -h <db-host> -P 3306 -u webhooks -p -e "SELECT 1"

# Nginx
sudo nginx -t && sudo systemctl reload nginx

Known follow-ups, not covered here:

  • Broker failover regression in Src/lib/queue.php — a single dead CloudAMQP broker still stalls all three queues.
  • Local Docker dev (docker-compose.yml) has no Src/secrets/*.php provisioned, so /api/health reports every dependency unhealthy when run locally — cosmetic for local dev, not a production concern.
This site uses Just the Docs, a documentation theme for Jekyll.