Broker Queues

The RabbitMQ/LavinMQ broker (see broker failover, Src/lib/queue.php) holds 12 queues — 4 logical destinations, each provisioned as a {name} / {name}-retry / {name}-error triple. All 12 are pre-provisioned on the production broker; declareQueueAndDLX() only declares/binds them when running against the test/CI broker.

Broker Queue Map — a legend showing the shared retry cycle (main queue nacks to retry, retry's 10-minute TTL dead-letters back to main, up to 10 times, then the 11th failure moves the message permanently to error), followed by four lanes: github, github-database, github-hooks, and github-forbidden, each with its producer, its three queues, and its consuming worker

The retry/error pattern (shared by all 4)

Every logical queue follows the same dead-letter cycle, built once in declareQueueAndDLX() and driven by nackOrDeadLetter() (both Src/lib/queue.php):

  1. A consumer nack()s a message it failed to process for a transient reason (e.g. a DB connection blip). RabbitMQ dead-letters it to {name}-retry.
  2. {name}-retry holds the message for a 10-minute TTL (x-message-ttl), then dead-letters it back to {name} for another attempt. Nothing ever consumes -retry queues directly — the TTL is the only thing that moves messages out of them.
  3. RabbitMQ’s own x-death header tracks how many times a given message has cycled through {name}. getDeathCount() reads it; once it reaches 10, nackOrDeadLetter() calls sendToErrorQueue() instead of nacking again, publishing the message to {name}-error and acking it off the retry cycle for good.
  4. {name}-error is terminal — nothing consumes it automatically. It exists for manual inspection and (if needed) manual replay.

A permanent failure (a duplicate delivery, a payload with no body) skips this cycle entirely — the consumer just ack()s and drops the message, since retrying it would never produce a different outcome.

The 4 logical queues

github / github-retry / github-error

  • Producer: Src/handlers/github.php, when the request’s source IP is inside the Src/meta.json CIDR allow list. Publishes to the github-exchange fanout exchange, which is bound to both github and github-database — one publish reaches both queues. The hooks worker also republishes here (see github-hooks below) when replaying a previously-undelivered hook.
  • Consumer: the service daemon worker (runServiceJob() → receiveQueue(..., "github", "processMessage", "github-exchange"), Src/services/consumer.github.php). For every message: saveGithubToDatabase() writes the raw event to the github log table and upserts its github_statistics_6h bucket (see ADR-0004); then handleGitHub() (Src/lib/github.handler.php) composes and sends the outbound email/push Alert — it does not touch any entity table, despite the name.
  • This queue is output-facing: persist the raw log, notify a human. The entity-table dispatch lives on github-database below, not here.

github-database / github-database-retry / github-database-error

  • Producer: the same github-exchange fanout publish as github above — every valid GitHub delivery lands in both queues from a single publish, not two separate sendQueue() calls.
  • Consumer: the database-service daemon worker (runDatabaseServiceJob() → receiveQueue(..., "github-database", "processDatabaseMessage", "github-exchange"), Src/services/consumer.github.database.php). Calls saveGithubValidatedToDatabase(), which:
    • unconditionally runs ReleaseTrackingHandler first, watching release/workflow_run events across every repository this service sees (not just GStraccini-bot’s) into github_releases/github_release_workflow_runs (see ADR-0006);
    • then, scoped to the GStraccini-bot GitHub App only (isGitHubXToBot()/ validateEventAndAction()), dispatches to the OOP pipeline (GitHubEventHandler → *Repository classes, for create/delete, discussion, issues, pull_request) or the older procedural save*GitHubToDatabase() calls (check_run, comments, installation, etc.), writing into whichever typed github_* table the event maps to.
  • Kept as a separate queue/worker from github so a slow or failing entity-table write here can’t stall or drop the raw log + Alert on github, and vice versa.

github-hooks / github-hooks-retry / github-hooks-error

  • Producer: the hooks worker itself. Before it drains this queue each cycle, checkHooks() (Src/services/check.hooks.github.php) polls the GStraccini-bot GitHub App’s /app/hook/deliveries API for deliveries that didn’t get a 202 the first time, and publishes one message per undelivered hook onto github-hooks.
  • Consumer: the hooks daemon worker (runHooksJob() → receiveQueue(..., "github-hooks", "processMessageHook"), Src/services/consumer.github.php). retryGitHubHook() re-fetches the original request (headers + payload) for that hook delivery from GitHub’s API and republishes it onto the github queue — i.e. this queue’s job is to feed github, not to process events itself.
  • This is the self-healing loop for deliveries that arrived while this app (or the broker) was down or erroring: GitHub already recorded them as failed deliveries, and this queue is what re-injects them.

github-forbidden / github-forbidden-retry / github-forbidden-error

  • Producer: Src/handlers/github.php, when the request’s source IP is not in the Src/meta.json CIDR allow list (getQueueName() returns github-forbidden instead of github). Published to the default exchange directly — not through github-exchange, since this path is meant for one queue only.
  • Consumer: the cleanup worker, once per run (runCleanupJob() → receiveQueue(60 * 1, "github-forbidden", "processMessageForbidden"), Src/services/consumer.github.php). Runs the same handleGitHub() pipeline as the github queue, just tagged "Forbidden" — the event is still processed, this queue only exists to keep IP-disallowed traffic out of the primary github queue’s throughput/latency budget.
This site uses Just the Docs, a documentation theme for Jekyll.