Skip to content

How a job runs

How runnerd fits togetherrunnerd serve polls GitHub over HTTPS with the gh CLI's token and registers a just-in-time runner for each queued job it starts. Inside the daemon are the engine, the job history in SQLite, the API and the web dashboard. runnerctl and runnerctl tui reach the API over the Unix socket; a browser reaches the dashboard on 127.0.0.1:7477, which reads the same API. The daemon starts containers through the Docker API, in the slice whose MemoryMax is the budget: one per slot, here jvm 1, jvm 2, node 1 and node 2, plus the Docker Hub mirror.GitHubqueued jobs, JIT runners, stepsHTTPS, the gh CLI's token:polls, registers runnersrunnerd serveengine: polls, plans, startsjob history (SQLite)API: JSON and eventsdashboardrunnerctlUnix socketrunnerctl tuiUnix socketbrowser127.0.0.1:7477Docker APIrunnerd.sliceMemoryMax is the budgetjvm 1jvm 2node 1node 2mirrora slot's runner: one job, its own Docker, sysbox
One daemon between GitHub and Docker; every client goes through its API.

One daemon, runnerd serve, owns every slot. Clients never talk to GitHub or Docker: they read the daemon’s status and history and send it commands over its Unix socket ($XDG_RUNTIME_DIR/runnerd.sock). The dashboard runs inside the daemon’s process but uses the same socket.

  1. Queued. The job waits on GitHub, asking for labels that one of the repository’s profiles serves.
  2. Listed. Every poll_interval the engine lists each repository’s queued jobs, and three times as often while the repository is active: a job in the last 10 minutes, or a runner running one. It lists this machine’s runners of a repository first, every third time and after one of them starts or goes, so a job whose runner just took it does not get a second one. A command lists every repository at once.
  3. Admitted. The planner picks a free slot in the repository’s profile whose labels match the job. A job starts only if its runner’s memory fits the budget: its predicted peak (its cap while the job has too little history, or under admission = "caps"), within what the host leaves the slice. A job that does not fit stays queued, and runnerctl status says why.
  4. Runner started. The engine registers an ephemeral runner with the pool’s labels and starts a container for it in the slot, under sysbox, over the profile’s snapshot. Up to four runners start at once, and their registrations go to GitHub one after another. The runner takes one job that matches those labels, not necessarily the one that prompted the start, and exits after it.
  5. Job. While it runs, the engine records the job’s steps, its log and, every 2 seconds, its container’s CPU and memory in the history store. Clients stream this through GET /v1/events.
  6. Recorded. When the runner exits, its credentials are scrubbed and, on a Sysbox profile that opts in, the image probes a sample of the calls Sysbox serves. Docker’s event wakes the engine, which inspects the container, removes it, frees the slot and starts a runner for the next listed job there without waiting for the next poll. GitHub drops the runner’s registration with its job; runnerd deletes one it kept without holding the slot. If Docker’s events are not coming through, the next poll finds the exit. The job’s record is final once GitHub reports it completed. A failed job of a profile with record_failures leaves a record first, which runnerctl shell opens.
A job's lifecycleA job is queued on GitHub with labels a repository's labels match. runnerd takes the lowest free slot of the repository's profile; with no slot or no memory free, the job stays queued on GitHub until the next poll. It registers a just-in-time runner for one job, removed if it gets no job in 5 minutes, and starts a container with the slot's volume over the profile's snapshot. While the job runs, the job history records its steps, log and samples every 2 seconds. When it finishes, the runner exits and the image deletes its secrets. If the job failed and the profile has record_failures, the container is first committed to an image, runnerd-record:job, that runnerctl shell opens. Then the container and the runner's registration are removed and the slot is free.queuedGitHub lists it; its labels matchslotthe profile's lowest free slotJIT runnerregisters for one jobcontainerslot volume over the snapshotrunningsteps, log, samples every 2 sfinishedrunner exits, scrubs its secretsremovedcontainer and runner; slot freeno slot or memoryfree: it stays queueduntil the next pollno job in 5 min: removedpassed, or not recordedfailed, withrecord_failuresrecordan image to opena shell in
A job's way through runnerd. Only a failed job of a profile with record_failures leaves a record.
  • It reads the gh CLI’s token at start, and again when GitHub rejects it. Every poll_interval (15 s by default) it asks GitHub for each repository’s queued jobs, and three times as often, every 5 s by default, for a repository with jobs lately.
  • The pool whose labels include every label in a job’s runs-on serves it: one [[repositories]] entry, or one of several.
  • It registers a just-in-time runner and starts it in a free slot of the pool’s profile if the slice’s MemoryMax has room for the profile’s memory (under measured admission, the job’s predicted peak) and, under CPU admission, the CPU budget has room too.
  • A job that does not fit lets younger ones past, until it has waited 10 minutes for memory or CPU, or 2 if its pool already went first and still has jobs queued. Then it goes first. What it wins is the next runner of its pool: GitHub gives a runner any job its labels match, so another job of that pool may take it.
  • A runner takes one job, then runnerd removes it. A runner with no job after 5 minutes is removed unless it is warm.
  • Each tick removes the idle runners it no longer needs first, all at once, then starts runners for queued jobs. A runner that no longer fits what runs by then waits; the others start, then the tick plans again from the same listings (at most three times) instead of waiting for the next poll.
  • When a runner’s container stops, Docker’s event wakes runnerd between polls: it removes the runner and starts one for the next job the last poll listed in its slot within seconds. A job cancelled or taken elsewhere since that poll can still get a runner, which then waits like any idle one. A job that went first holds these starts back too.
  • A job’s page shows when runnerd first saw it queued (Seen by runnerd; a job a waiting runner took before any listing showed it has none), so its wait splits into the time before runnerd knew of it, the time it took to start a runner, and the runner’s boot.
  • A restarted runnerd adopts running runner containers.

When the hour’s rate limit runs short, runnerd reads running jobs less often first, then lists active repositories at the idle pace, then every repository two or four times less often, one step at a time, and logs each change; no registration or delete waits for this, but what a tick decides, such as stopping an idle runner or topping up a warm one, comes with the slower ticks.

Where each part lives maps these parts to the code.