Skip to content

Terms

One concurrent job of a profile. Each job gets a new container; the slot keeps a volume over the profile’s read-only snapshot, so the next job in it finds the Docker images, ~/.cache and tool cache the last one left. A queued job starts in the free slot that last ran the same job (repository, workflow, job name), else one that ran a job of its repository, else the lowest free slot. Caches says what a slot keeps.

A profile is a runner size and image: cpus, memory, slots and image under [profiles.<name>]. A pool is one [[repositories]] entry: a repository, the profile its jobs run on, and the labels it serves. A repository can have several pools, one per profile (Split light and heavy jobs).

Every runner container runs in one systemd slice, runnerd.slice as runnerd init writes it. Its MemoryMax is the budget: a runner starts only when it fits beside what already runs, and what the host’s memory leaves the slice. A runner reserves its job’s predicted peak (measured admission, the default), or its profile’s memory while the job has too little history and always under admission = "caps". Below, every runner is at its cap:

The memory budgetrunnerd check-config adds every slot's memory cap and the mirror's: two jvm slots of 9G, two node slots of 7G and 512M for the mirror make 32.5G, which fits a MemoryMax of 33G. Each tick, what runs takes its cap: here jvm 1, node 1, a shell in a jvm record at 9G and the mirror use 25.5G, leaving 7.5G. A queued node job needs 7G, which fits, so it starts in node 2. A queued jvm job needs 9G, which does not fit, so it waits in GitHub's queue though jvm 2 is free.runnerd check-configevery slot's cap and the mirror: 32.5G of 33Gjvm 19Gjvm 29Gnode 17Gnode 27Gmirror 512M, 512M freeeach tickwhat runs takes its cap; a job starts if its capfits what is leftjvm 19Gnode 17Gjvm shell9G7.5G free7Ga node job fits: it starts in node 29Ga jvm job waits, though jvm 2 is free
The example config in a slice with MemoryMax=33G, to scale. A shell in a record and a reseed take memory too, so a free slot is not always room.

runnerd check-config [file] reads the slice’s MemoryMax from systemd, or takes --memory-max. Under measured admission it fails when one runner of a profile with the mirror does not fit, and refuses a slice without a MemoryMax. Under caps it fails when slots × memory over all profiles plus the mirror does not fit, and only warns about a slice without one:

$ runnerd check-config runnerd.toml
runnerd.toml: valid under measured admission: the largest runner with the mirror needs 9.5G of the 33G MemoryMax; every slot at its cap at once would take 32.5G
$ runnerd check-config runnerd.toml --memory-max 9G
error: one runner of jvm and the mirror take 9.5G, the slice runnerd.slice's MemoryMax is 9G; lower its `memory` or raise the slice's MemoryMax
$ runnerd check-config caps.toml --memory-max 24G
error: slots × memory over all profiles plus the mirror is 32.5G, the slice runnerd.slice's MemoryMax is 24G; lower `slots` or `memory`, or raise the slice's MemoryMax (`runnerd doctor` prints its unit)

A warm runner is registered before GitHub queues the job and takes it within a second or two, where a job that finds no runner waits for a new one to start (how long).

  • Each pool keeps warm runners (1 by default) while its repository is active: one of its jobs was waiting, taken or finished in the last 10 minutes. Otherwise they go after 5 idle minutes.
  • When a job is seen, the other jobs of its run get runners too: per pool, the median number that workflow’s runs had there over 14 days, a run with none there counting as 0, less those seen. A pool needs 3 runs with jobs there first, and one that most runs skip gets none.
  • Warm runners start last, once every queued job that fits has a runner, and reserve their profile’s memory (and cpus under CPU admission). A queued job of another repository takes their room. After a job in slot 1, the profile measures what slot 1 changed without stopping anything, at most every 10 minutes. Only a reseed due by that measure (slot 1 reached reseed_at) or asked for removes them, once its other runners are idle.

With record_failures = true on a profile, a failed job’s stopped runner is kept as an image, and runnerctl shell <job> opens a shell in it at the failing step. Records of failed jobs says what it keeps and who can read it.

What needs you now, most urgent first, each with what clears it: queued jobs runnerd cannot start with the config it runs with (a paused repository, or labels no pool serves), a job that waited 30 minutes or more for the host’s memory, the doctor’s warnings and failures (or that its checks are not current), warning insights, a repository GitHub will not list, a token GitHub refuses, and a spent rate limit. The dashboard’s overview and the TUI’s show it as one line, a count and the most urgent item, that opens to the list; GET /v1/attention serves it.

runnerd records every job its runners may run in SQLite; [history] sets its path and retention. For each job:

What Kept
its timeline (queued, container started, job started, finished) and steps, as GitHub shows them with the job
the container’s CPU, memory and network every 2 s, its CPU throttling and the bytes the Docker Hub mirror served it samples for history.samples; totals and peaks with the job
its log, secrets still masked, up to history.log_max for history.logs
how long it waited for each cause runnerd’s plans gave, such as the host’s memory or a barrier (why jobs queued) with the job

A job’s wait runs from queued to started; the part that passed while runnerd was not running is kept as unseen and left out of the waits sizing reads.

A job that leaves GitHub’s queue without one of this machine’s runners is recorded as elsewhere (the dashboard’s “Not run here”), as cancelled in queue when its run was cancelled first, or as its conclusion, such as skipped. No wait, median or cost counts it. A finished job goes after history.jobs.

A failed job reads runner fault when Sysbox, not its code, failed it; Why did this job fail? says how runnerd tells, and runnerd can re-run its run.