Follow WebhooksHandler to another Broker only once the current one is drained

WebhooksHandler publishes GitHub Events through one of several free-tier CloudAMQP Brokers and rotates to the next one on its own when the current one reaches its monthly message limit (its ADR-0003). This app configures the same Brokers ($rabbitMqConnectionStrings) but reads exactly one, chosen by the broker.active_index Setting, and until now only the Operator could change it (POST /api/v1/broker/switch/{index}). After an automatic rotation, new Events piled up on the new Broker while every Worker kept waiting on the old, silent one — for as long as it took someone to notice.

The producer now tells this app where it went (its ADR-0005: POST /api/v1/broker/pending-switch, the target named by host and virtual host), and this app moves on its own. Two things shaped how:

• The producer says where, this app decides when. Switching the moment the notification arrives would read the new Broker while Events are still waiting on the old one, and the typed github_* tables care about order: a pull request’s synchronize after its closed means a different Processing State than the other way round. So the notification only records a Pending Switch (the broker.pending_switch Setting, as JSON), and the flip happens in the Workers: at the start of every consumer window, receiveQueue() runs completePendingBrokerSwitchIfDrained(), which passively declares the four logical Queues and their -retry siblings on the active Broker and, when all eight report zero ready messages, writes the target into broker.active_index and clears the Pending Switch. Each Worker re-reads the active Broker per window, so all of them follow within one cycle (five minutes for service and database-service). Several Workers may check at once; they all compute the same answer and the writes are idempotent.
• Matched by endpoint, not by index. The producer reads its connections from .env, this app from its secrets, and nothing keeps the two lists in the same order. findBrokerIndexByEndpoint() resolves the host/vhost pair the producer sends against getBrokers(); an unknown pair is a 404, not a guess.

Two corners of the drain are worth spelling out:

• -retry counts, -error does not. A message nacked for a transient failure sits in {queue}-retry for ten minutes and then comes back to {queue} (ADR-0009). A Broker whose main Queues are empty but whose -retry Queues are not is therefore not drained — flipping then would strand those messages on a Broker nobody reads. -error is terminal on every Broker and is inspected by hand, so it is left out.
• A Leftover Drain backs the flip. The passive declare counts ready messages, not the one a Worker holds unacked (prefetch 1), so a nack can land on the old Broker’s -retry just as another Worker’s check passed — and the producer may rotate twice before the first Pending Switch completes, leaving messages on a Broker that was never active here. So before its window on the active Broker, every Worker looks at each other configured Broker (except a Pending Switch’s target, which the producer is filling and which must not be read early) for messages on its own Queue and processes them first with basic_get (drainLeftoverBrokers()), which returns the moment the Queue is empty instead of waiting out a window. Those messages are older than anything on the active Broker, so order is kept. The pass is capped at 500 messages per cycle and a Broker that cannot be reached only logs a warning: a spare that is down must neither fail the run on HealthChecks nor hold the active Broker hostage.

The Operator’s hand still wins: POST /api/v1/broker/switch/{index} moves immediately and drops any Pending Switch, and DELETE /api/v1/broker/pending-switch cancels one without moving. A notification whose switchedAt is older than the Pending Switch already on record is ignored, so late or out-of-order deliveries from the producer’s retry queue cannot move the Broker backwards.

Consequences

• receiveQueue() costs one Settings read per window plus, while a switch is pending, one connection and eight passive declares against the active Broker; and one connection plus one passive declare per non-active Broker per window for the Leftover Drain. Passive declares are free on CloudAMQP’s message count; basic_get deliveries count like any other.
• Between the producer’s rotation and the flip, new Events wait on the target Broker. With the Workers’ five-minute windows the lag is normally one window once the old Broker is empty; a long -retry tail (up to ten minutes per cycle, ten cycles at most) extends it. That lag is the price of ordering and is accepted.
• An unreachable active Broker cannot be verified drained, so the Pending Switch waits (logged as a warning) rather than leaving messages behind on a Broker that will come back. If it will not come back, the Operator switches by hand.
• The settings.value column is NOT NULL, so “no pending switch” is stored as an empty string. A stored target that no longer exists in $rabbitMqConnectionStrings reads as no Pending Switch.
• basic_get messages carry no consumer tag, and the consumer callbacks cancel their consumer when the window is about to end; the Leftover Drain therefore hands them a window that never ends (PHP_INT_MAX). If those callbacks ever grow a second use of the window arguments, revisit this.
• broker.last_switch records the last completed Pending Switch for GET /api/v1/broker; nothing reads it to decide anything.