Worker Schedules

When each background worker actually runs its job, and for how long.

There is no crontab and no systemd timer in production. Every worker is a long-lived systemd unit started with --daemon (scripts/setup-vps.sh), so “schedule” means the daemon loop’s cadence:

loop:  new run id  →  /start ping  →  run one cycle of the job  →  success (or /fail) ping  →  sleep sleepBetweenCycles  →  repeat

The time between two job starts is cycle duration + sleepBetweenCycles, not a fixed clock time. Nothing is aligned to wall-clock minutes. Every push to main restarts all units (deploy.yml), and systemd restarts a crashed unit after 10 s (Restart=always, RestartSec=10); both reset the cadence.

The loop lives in runAsDaemon() (Src/lib/runner.php). Each entry script (Src/<worker>.php) passes its own onceWork (cron/http modes), daemonCycleWork (daemon mode) and sleepBetweenCycles to runJob().

Summary

Worker systemd unit One daemon cycle Sleep after cycle Job starts roughly every
service webhooks-service Consume github for 5 min (10 min during peak hours) 1 s ~5 min (~10 min peak), effectively continuous
database-service webhooks-database-service Consume github-database for 5 min 1 s ~5 min, effectively continuous
hooks webhooks-hooks checkHooks(), then consume github-hooks for 10 min 60 s ~11 min
cleanup webhooks-cleanup Trim github, sync meta, drain github-forbidden for 1 min, delete successful workflows 300 s ~6 min + DB/API time
maintenance webhooks-maintenance Update workflow run conclusions, re-request cancelled workflows 60 s 1 min + however long the GitHub API calls take
idle-pull-requests webhooks-idle-pull-requests One Idle Retry for the first due Idle Pull Request, round-robin across repositories (two quick queries) 3600 s ~1 h, plus once on every unit (re)start
queue-relay webhooks-queue-relay No cycles: a continuous Unix-socket accept loop n/a Always running

How a queue-consuming cycle ends

service, database-service, hooks and the forbidden-queue step of cleanup all go through receiveQueue($timeout, ...) (Src/lib/queue.php). The consume window is time-bounded, not message-bounded:

  • The consume loop runs until $timeout seconds have passed since it started, even if the queue is empty for the whole window. An idle cycle still lasts the full window.
  • Every message callback (processMessage, processDatabaseMessage, processMessageHook, processMessageForbidden) cancels the consumer once fewer than 5 s are left in the window. A message that arrives near the end therefore closes the cycle slightly early.
  • In daemon mode a SIGTERM/SIGINT/SIGHUP breaks the loop the next time wait() returns (isDaemonRunning()), so stopping a unit doesn’t wait for the window to run out.
  • If connecting fails, the cycle retries once after 5 s and then gives up until the next cycle.

Per worker

service (GitHub queue consumer)

  • Daemon: runServiceJob(5), a 5-minute window on the github queue, then 1 s sleep.
  • Peak hours: on Monday and Tuesday, 00:00–04:00 UTC (gmdate(), in Src/runners/service.runner.php) the window is doubled, to 10 minutes in daemon mode. The day and hour are checked when each cycle starts, so a cycle that begins at 03:59 UTC still runs a doubled window.
  • Cron / HTTP: runServiceJob(15), a 15-minute window (30 minutes during peak hours).
  • Run log event: {"event":"queue_consumed","queue":"github", ...} with broker, minutes_per_broker, peak_hours and consumed (messages handed to processMessage() in that window).

database-service (GitHub database queue consumer)

  • Daemon: runDatabaseServiceJob(5), a 5-minute window on github-database, then 1 s sleep. There is no peak-hours doubling.
  • Cron / HTTP: runDatabaseServiceJob(10), a 10-minute window.
  • Run log event: {"event":"queue_consumed","queue":"github-database", ...} with broker, minutes_per_broker and consumed.

hooks (GitHub hook redelivery)

  • Every mode: runHooksJob() calls checkHooks() (catch-up pagination against GitHub’s hook deliveries) and then consumes github-hooks for a fixed 10 minutes.
  • Daemon: 60 s sleep, so a new checkHooks() pass starts about every 11 minutes.
  • It was split out of maintenance so that slow workflow-run loops can’t hold back its cadence. It has its own HealthChecks.io check (HEALTHCHECKSIO_HOOKS).
  • Run log event: {"event":"hooks", ...} with failed_deliveries_queued (deliveries checkHooks() pushed to github-hooks), check_skipped (true when another run held the hooks.txt lock) and consumed for the 10-minute window.

cleanup

  • Every mode: runCleanupJob() runs these steps in order:
    1. cleanupTable("github") removes rows older than the retention window in config/config.json.
    2. downloadMeta() syncs GitHub meta.
    3. The github-forbidden queue is drained for a fixed 1 minute.
    4. deleteSuccessfulWorkflows() removes workflow runs that succeeded.
  • Daemon: 300 s sleep, so a cycle starts about every 6 minutes plus the time steps 1, 2 and 4 take.
  • Run log event: {"event":"cleanup", ...} with github_rows_deleted, meta_hook_ranges, forbidden_consumed and workflow_runs_deleted, one per step above.

maintenance

  • Every mode: runMaintenanceJob() runs WorkflowRunUpdater::updateWorkflowRunConclusions() and then rerequestCancelledWorkflows().
  • Daemon: 60 s sleep. There’s no consume window, so the cycle length depends on how many runs need updating: the work is sequential, unbatched GitHub API calls (sometimes 100+), with no signal check between them. For that reason the unit has TimeoutStopSec=60 instead of 30.
  • Run log event: {"event":"maintenance", ...} with conclusions_checked (outdated runs looked up on GitHub) and rerequested (cancelled runs re-requested).

idle-pull-requests

  • Every mode: runIdlePullRequestsJob() calls retryIdlePullRequest() (Src/lib/bot.database.php). That function gives one Idle Retry (see ADR-0010) to the first DUE row of github_pull_requests_idle_view (Sql/0038, Sql/0039, Sql/0040, Sql/0042). The view lists open pull requests in the PROCESSED state with no update for at least a day, and gives each one an IdleStatus:
    • DUE: idle for at least IdleRetryCount + 1 days, with IdleRetryCount < 10
    • WAITING: still inside its backoff window; WaitingUntil (UpdatedAt + IdleRetryCount + 1 days) is when it turns DUE, and is NULL on the other two
    • GAVE_UP: has reached 10 Idle Retries

    It left joins github_repositories (the most recent row per owner/name) to report RepositoryPrivate, which is NULL for a pull request whose repository has no github_repositories row. The join never decides membership: Sql/0039 made it an inner join and the view shrank from 200+ rows to 35 in production, so Sql/0040 relaxed it.

    Order (Sql/0042): round-robin across repositories. Ordered by UpdatedAt alone, the repository with the oldest pull requests got one Idle Retry after another while every other repository waited. The view now orders by RepositoryIdleRank (1 for the repository’s least retried DUE pull request, oldest among equals, then its other DUE ones by IdleRetryCount and age, then WAITING, then GAVE_UP), then RepositoryLastPullRequestUpdate (the most recent UpdatedAt among all of the repository’s pull requests, open or closed), then UpdatedAt: every repository’s first pull request, least recently active repository first; then every repository’s second, and so on. Once the bot processes an Idle Retry, that repository’s last update jumps to that moment and it goes to the back of the line, so consecutive Idle Retries go to different repositories whenever more than one has a DUE pull request. A repository with daily activity (dependabot, the bot’s own next-pull-request chain) sorts last on purpose: the normal workflow is already moving its pull requests along, and a pull request stuck in the middle of that chain is a manual check.

    Least retried first (Sql/0043). A retried pull request’s backoff made it look older than the never-retried ones next to it (with 2 retries it is DUE again after 3 idle days, with 0 after 1), so it kept rank 1 and got its third Idle Retry while the others waited for their first. IdleStatus (DUE, WAITING, GAVE_UP) and then IdleRetryCount now come before everything else in the order, and IdleRetryCount comes before age inside the repository rank: every DUE pull request with 0 retries is listed before any with 1, and so on, with the round-robin above applied inside each count. No pull request gets its second Idle Retry while another is still waiting for its first, whatever repository either is in.

  • The Idle Retry sets ProcessingState = 'IDLE' and IdleRetryCount = IdleRetryCount + 1, and assigns UpdatedAt to itself. GStraccini-bot then claims the pull request like a RE_REQUESTED one, near the front of its ORDER BY UpdatedAt queue. The UPDATE only applies while the row is still PROCESSED, so an Event that arrives in between wins.
  • Backoff: each Idle Retry waits one day longer than the last: 1, 2, 3 … 10 idle days, about 55 days in total. Every Event-driven write resets IdleRetryCount to 0, and so does the bot’s own re-request when another pull request in the repository closes.
  • Daemon: 3600 s sleep. The job runs immediately whenever the unit starts, so every deploy (which restarts all units) adds one extra run on top of the hourly cadence. The async SIGTERM handler interrupts the sleep(), so stopping the unit doesn’t wait out the hour.
  • GET /api/v1/pull-requests/idle returns the whole view in that order, including WAITING and GAVE_UP rows, each with repositoryPrivate, repositoryLastPullRequestUpdate, repositoryIdleRank and waitingUntil. Its first DUE row is the next one this worker will retry.
  • It has its own HealthChecks.io check (HEALTHCHECKSIO_IDLE_PULL_REQUESTS).
  • Run log event: {"event":"idle_retry", ...} with the repository, pull request number and URL, sequence, updated_at and the new idle_retry_count, or {"event":"nothing_due"} when no pull request was due.

queue-relay

  • It doesn’t use runJob(). Src/queue-relay.php always runs runQueueRelayJob(), which accepts connections on the Unix socket continuously and checks isDaemonRunning() every 2 s.
  • It has no cron or HTTP mode, and POST /api/v1/workers/queue-relay/run is deliberately not offered.

Running a job outside the daemon

  • Cron mode: php Src/<worker>.php with no --daemon runs onceWork once and exits. Nothing in the repo schedules this. If you add your own crontab entry, make the interval longer than the cron-mode window (15 min for service, 30 min if the run can start during peak hours; 10 min for database-service and hooks) so runs don’t overlap. Also keep in mind that the systemd daemon is probably running too.
  • HTTP: POST /api/v1/workers/{name}/run (Src/api/v1/workers.php) returns 202 and then runs the job once inside the PHP-FPM request, with the same window as cron mode. While it runs, it holds a PHP-FPM worker for that whole window.

HealthChecks.io implications

Every run — a cron execution, one daemon cycle, or an HTTP trigger — is one tracked run on the worker’s check (runTracked() in Src/lib/runner.php): a fresh HealthChecks client with a new run id (?rid=<GUID> on every ping), a /start ping before the job, and a success ping after it, or /fail carrying the exception message if the job threw (the exception is then rethrown, so a daemon still exits and is restarted by systemd). HealthChecks.io therefore shows each run as a start → end pair with its own duration, and the global $hc always points at the current run, so $hc->error()/$hc->log() calls inside a job and fatal_handler() land on the same run id. Every job also sends one /log event per run with what it did (the Run log event lines above, always carrying run_id), so the check’s event list reads start → log → end per run, and the same run_id (from currentRunId()) is on every LogStream line of that run, so a run can be followed across HealthChecks.io and LogStream. The step functions report what the events need: receiveQueue() returns the messages consumed, cleanupTable() / deleteSuccessfulWorkflows() the rows deleted, downloadMeta() the hook ranges written, checkHooks() the deliveries queued (or null when it skipped on the lock), and WorkflowRunUpdater’s two methods the runs they touched. Because /start is pinged on every cycle, workers with a short sleepBetweenCycles (service, database-service) ping twice per cycle. When you configure a check’s period and grace, use the “Job starts roughly every” column above, with headroom for peak hours (service) and for slow GitHub API passes (maintenance, cleanup). Every runJob() worker has its own check, so each one’s pings come from a single cadence. queue-relay has no check: it’s a continuous accept loop with no cycles to ping after.

This site uses Just the Docs, a documentation theme for Jekyll.