Admission and sizing

A runner starts only when it fits the slice’s MemoryMax beside what already runs, and what the host’s memory leaves the slice beside its other programs. By default it reserves its profile’s memory, and the budget check holds slots × memory to MemoryMax. runnerd also keeps every job’s sampled peaks, and from them it can:

• advise each profile’s memory and slots (Sizing), and apply the advice itself (Auto sizing);
• reserve each job’s predicted peak instead of its cap (Measured admission);
• hold back jobs the CPUs cannot take (CPU admission).

Advice is always there; the other three are off by default. runnerd simulate replays your own history through measured and CPU admission before you turn either on. The keys and their defaults are in Configuration.

What a job’s memory is

A container is sampled every 2 s, so a shorter spike goes unseen. Its memory is an estimate of what the job holds without page cache, which the kernel mostly reclaims before it kills anything: anonymous memory, tmpfs and kernel memory, read as memory.current less the file page cache lists. That is lower than docker stats, which still counts page cache read twice. A job that only fills its cap with page cache has not met it.

The slice’s memory, which measured admission’s high-water mark reads, is memory.current less inactive page cache, as docker stats counts it, so it errs toward holding starts back. Kills and out-of-memory events come from the cgroup’s memory.events, and so does sock_throttled: how many times the kernel held the container’s downloads to a trickle because its reclaim was going badly. The job page and runnerctl job show it, and the memory pressure insight names a repository whose jobs it slowed.

The host’s memory

The slice’s MemoryMax is fixed, but the rest of the host is not: a desktop, local builds or other services take RAM the budget assumed was free. When the host runs short, the kernel reclaims memory from every cgroup at once, and while that reclaim goes badly it throttles every runner’s sockets, so checkouts and downloads stall for a minute or more. runnerd keeps new runners out of that state on its own, under either policy:

• Every 2 s it reads the host’s MemAvailable and /proc/pressure/memory. The slice may hold what its runners hold now, without page cache or reclaimable kernel caches, plus MemAvailable, less a margin kept free for the host’s page cache: the larger of 2G and a sixteenth of the host’s memory, 4G on 64G. A new runner starts only when what it reserves fits that beside what the running ones reserve, as it must fit MemoryMax.
• Nothing starts while the host has under twice the margin available and every task that could run on it waits for memory 5% of the time or more over the last 10 s (full pressure), until that falls under 1%. With more available, such stalls come from runners reclaiming page cache at their own caps, which a start elsewhere does not make worse.
• Once a job waits for the host while runners run, the margin grows by half until the host has room again, so starts do not resume at the edge and push it straight back. The first runner into an empty slice needs only the margin.
• Running jobs are never stopped for it. A job that waits says so: The host is short of memory: needs 6G; the host has 2G to spare; programs outside CI hold 30G, most java 9.3G and qemu-system-x86 5.8G. It names up to three programs outside the slice that hold 512M or more each, by process name only, read every 30 s. The status line, runnerctl watch, the TUI and the dashboard show the host’s line beside the budget (host: 31G of 64G available; leaves the slice 40G, followed by the same programs when there are any). When the host leaves the slice less room than MemoryMax does, the line says what that leaves free, leaves the slice 30G of its 33G, so 3.75G is free, the budget bar’s free part is what of that is past every slot’s block, and the rest of the room is hatched, held by the host. Without MemoryMax the host alone bounds the slice, and the line says what it leaves free: leaves the slice 40G, so 12G is free. runnerd doctor warns while the host has less to spare than the smallest profile’s memory, or stalls short of memory, and says which programs hold it.
• Each job’s record keeps how long it waited for the host. Sizing and auto sizing take that out of its wait for a slot, so a busy desktop does not make auto sizing add slots.

Without a reading from the last few seconds, only MemoryMax bounds starts.

Sizing

runnerctl sizing, the dashboard’s Sizing page and the TUI turn each job’s recent sampled memory and CPU peaks (what memory counts) into advice for each profile’s memory and slots. A job’s recent runs are its last 60 successful ones in the 30 days (3 × 20, or 3 × min_jobs when that is more), with any that failed among them, so a job that got lighter stops holding its cap up after 60 runs, not a month:

• The need is the largest peak plus a margin, or plus 1G when that is more, rounded up to 256M. The cap kills a job that goes over it, so the need follows the heaviest run, not a percentile: a profile whose builds fit in 2G but whose end-to-end job takes 6G keeps room for the 6G. The percentiles are shown to say how rare that run is. A run at a larger cap than now counts at no more than that cap: a JVM grows its heap toward the room it has, so its peak under a 14G cap says little about what it needs under 8G.
• The 1G is room for page cache. Peaks leave page cache out, and a job that writes files fills its cap with it. With less than about 500M of it left above the job’s own memory, the kernel’s reclaim at the cap goes badly and throttles the job’s downloads; 1G keeps twice that.
• Below min_jobs jobs there is no advice, only the peaks.
• A run met its cap when its container’s own limit ran out of memory (memory.events.local oom): the job needs more than it shows, and counts at least its cap. A peak near the cap with no oom at it is not a need for more. A kill for memory while the container’s own limit held (the slice ran out, or a service container hit its own limit) holds the cap but raises nothing.
• A job with a run that met its cap or was killed for memory is on its cap until 3 of its runs in a row succeed clear of it. While any job is, the profile is not lowered, and the advice line says how many are.
• When the need is a tenth or more under memory, the advice is to lower it. If jobs waited a minute or more for a slot (p95), or the slots were busy half the time, it also adds as many slots as the profile’s memory now holds. A slot counts as busy from when its runner’s job was queued: a warm runner waiting for a job does not.
• A raise reads only runs that met the current memory since the profile has had it, toward the larger of what they need and what the same job needed at larger caps. A job whose runs at a larger cap need more than memory, with none running out at it, keeps it, and the advice line says so. Slots go down only if MemoryMax cannot hold them. When it cannot hold even one, the advice is to grow the slice first, and there is nothing to apply.
• Under admission = "measured", memory is only each container’s limit; runners reserve their predicted peaks, not it. So advice never changes slots there: a raise only has to fit one runner beside the mirror, as the budget check does, and a lower keeps the slots. When jobs waited for a slot or the slots were busy, the advice line says more slots may fit; auto sizing adds them itself, and without it runnerd simulate --slots replays the history with more before you raise slots.

margin and min_jobs under [sizing] set the margin and the fewest jobs.

What sizing cannot see:

• How much a job killed for memory needed. A job that idles at 3G and is killed within one 2 s interval shows a 3G peak; the kill keeps its cap from being lowered, but its need still reads 3G. Failed runs add their peaks but do not push successful ones out of a job’s recent runs. Cancelled jobs are left out, since a cancel stops a job before its peak, unless one met its cap.
• Time the daemon was not running. Jobs queued while it was down show long waits, which can make the advice add slots.

Without auto sizing, advice is only written when you apply it: runnerctl sizing apply <profile>, a in the TUI, or the Sizing page’s button, which opens the change in the config editor. Each goes through the same check as any config, so a change MemoryMax cannot hold is refused. Under caps, lowering never asks for more than the profile’s slots take now, and raising takes only what MemoryMax has spare, so two raises applied one after the other can be refused. runnerd doctor warns when a profile’s jobs need more than its memory. Run without a daemon, it reads the history database read-only. SQLite still leaves the -wal and -shm files a WAL database needs next to it when it may write there, so run it as the user who runs the daemon. Where it may not write, it reads the database file alone, and skips the sizing check while the -wal holds changes the file lacks. Sizing reads the slice’s MemoryMax on each request, so one raised after systemctl daemon-reload counts without restarting runnerd.

Auto sizing

With auto = true runnerd sizes each profile’s memory, slots and cpus itself; runnerd init turns it on. When it starts and every 5 minutes after, it works out each profile’s change and writes it into the config file the way runnerctl sizing apply does: the same budget check, the rest of the file kept as written, comments included, and the daemon runs with it.

• memory follows the advice, which takes min_jobs runs. No lowering while a job is on its cap.
• A lowering that went too far is taken back at the next look, without waiting for raise_interval: when a run meets the lowered cap, or a job that succeeded at the old memory fails 3 times in a row at it (failure or timed_out, one per workflow run, on at least two branches). memory goes back to what the lowering came from and, under caps, slots to what they were; for 30 days no lowering goes under it again. A memory set by hand is not taken back: a run that meets it raises it straight to what the job needs.
• slots: a profile whose jobs waited a minute or more for a slot at p95 (not counting time they waited for the host’s memory), or whose slots were busy half the time, gets one more while every profile’s slots still fit MemoryMax with the mirror: at their caps under caps, at their jobs’ p95 peaks under measured. Under caps a lowered memory also adds the slots the advice gives, and with a CPU budget only the slots whose cpus it holds beside every other profile’s. Slots are never lowered under what runs.
• cpus read each job’s average CPU over its run, never its peaks: a parallel build bursts to whatever cap it gets. Only successful runs with a measured average count, and only those at the profile’s current cpus since its last change and since runnerd started running those cpus, so a hand edit, a reload or an undo starts the evidence over. They go up when one job (repository, workflow and name) has 3 such runs averaging 70% of the cap or more at the median, up to the slice’s CPUs (and under caps, the CPU budget when one is set); a raise those leave no room for is not written, and runnerctl sizing says “No CPU raise” and why, beside any memory or slot change. They go down when min_jobs such runs say so and every job among them has a measured average, toward twice the heaviest job’s p95 average, rounded up to whole CPUs, at least 1. Other programs slowing the runs for a day can bring them down a step, which the next raise takes back.

A change that raises comes at most once every raise_interval (1 hour) and reads only the runs since the profile’s last change, so one slot that fixed a month of waits is not followed by another every hour. A change that lowers memory or cpus waits interval (1 hour): a lowering that goes too far costs one job, as the first run killed at it takes it back. Slot and CPU raises need 3 runs, so a new profile gets them before it has min_jobs of memory history. A lowering of memory, and any change of cpus, moves by at most max_step percent, memory in steps of 256M, so a profile far above its need gets there over a few changes. A raise of memory comes only from a run that met the cap, and goes straight to what the job needs.

Bounds are optional: min_memory and max_memory keep memory between them, and max_slots caps slots, for instance to leave another profile room. Without max_slots and without a MemoryMax nothing bounds slots, so none is added.

[sizing]
auto = true

[profiles.node]
image = "ghcr.io/ayagmar/runnerd-runner:node-0.0.20"
cpus = 8
memory = "7G"
slots = 2
min_memory = "2G"   # optional: never lower than this
max_slots = 6       # optional: never more slots than this

Changes picked in one look each count against the budget the ones before them take, so two profiles never both take the one slot that fits. A change is never written over a file you edited on disk: until runnerd runs with your edit (restart it, or make the edit in the dashboard’s editor instead), auto sizing changes nothing and logs why. An editor opened before an auto change, or before an edit on disk, cannot save over it either: the save is refused, and reloading the page shows the new file. The daemon also refuses a change that would drop a slot a runner is in at the moment it writes.

Each change is recorded in the job history before the file is written, logged, sent as a sizing.changed event once written, and listed by runnerctl sizing, the dashboard’s Sizing page and the TUI with what it replaced. A change the history cannot record is not written; one a crash interrupted is settled from the file at the next start, with auto on or off: done if the profile has its values, else listed as one that may not have been written, still holding the interval. runnerctl sizing undo <profile> writes the old values back, cpus included, as the Sizing page’s link does through the config editor; with auto still on, a raise can come after raise_interval and a lowering after interval, so bound the profile to keep it.

Measured admission

By default a runner starts only when its profile’s memory fits MemoryMax beside the caps of everything running, and the budget check holds slots × memory to MemoryMax. Most jobs peak far under their caps, so that memory sits reserved and unused. admission = "measured" under [host] has each runner reserve its job’s predicted peak instead:

[host]
slice = "runnerd.slice"
isolation = "sysbox"
admission = "measured"

[profiles.node]
image_stage = "node"
cpus = 8
memory = "7G"   # still each container's hard limit
slots = 8       # how many run at once; 8 × 7G may exceed MemoryMax

• A job is predicted from its own last 20 successful runs (repository, workflow and job name), or [sizing] min_jobs when that is more, with any that failed among them: with min_jobs of them, their p95 peak plus [sizing] margin, rounded up to 256M. A run that met its cap counts at least its cap.
• With fewer, the larger of its profile’s p95 and its own largest peak, plus the margin. Once 2 of its runs succeeded, one copy of it at a time reserves its own largest peak plus the margin instead; a copy queued or running beside another, as when a burst follows a cleared cache, takes the larger, as does every job of a repository with a busy runner whose job is not known yet. A job with no runs yet gets its profile’s largest peak plus the margin, since a new workflow may be the heaviest. These need min_jobs runs of the profile; without them, or when every run of the job was cancelled, the cap.
• A run that met its cap or was killed for memory, as sizing tells it, puts its job back on its cap until 3 of its runs in a row succeed clear of it, 7 days at most, cancelled runs included. A run that went past what it reserved, cancelled or not, has its job reserve the larger of its recent runs’ largest peak and that run’s while that run is among its recent runs, 7 days at most.
• A runner may take any of its repository’s queued jobs that its labels match, so it reserves the largest of their predictions (splitting light and heavy jobs keeps a heavy job’s out of a linter’s runner), and more if a heavier job queues while it waits; when that no longer fits, the runner is stopped and the job waits for one that fits. Once its job is known, it reserves that job’s prediction.
• A running runner counts at the larger of what it reserved and what it last used, so a job past its prediction leaves less for the next starts. Without a sample of it or of the slice from the last few seconds, it counts at its cap.
• Nothing starts while the slice uses 90% of MemoryMax or more, counted as docker stats does. Without a sample of the slice from the last few seconds, runners reserve their caps. Switching back to admission = "caps" counts every running runner at its cap.
• The check asks only that one runner of each profile, with the mirror, fits MemoryMax, and that the slice has one.

The caps still stop a container that goes over them. What measured admission trades for the shorter waits is that several jobs going over their predictions at once can fill the slice, which the kernel then answers by reclaiming page cache and killing a process inside it, a runner’s before the mirror’s. Each job’s record says what its runner reserved and why (runnerctl job <id>, the dashboard’s job page). The job page’s charts draw the reservation as a line under the cap, so a container capped at 22 CPUs that reserved 3.6 shows both.

Try it first

runnerd simulate replays the last 30 days through both policies, each job queued when it was and running as long as it ran, and changes nothing:

$ runnerd simulate --memory-max 33G --slots node=8
196 finished jobs replayed against a MemoryMax of 33G, each queued when it was and running as long as it ran.
PEAKS is every running job's peak added up at the busiest moment, with the mirror: more than the slice ever held, but a lower bound on what the daemon could overlap. The replay holds each running job at its peak, where the daemon admits against what jobs use while they warm up, and it leaves out start latency, seeds, shells and cancelled jobs.

POLICY    SLOTS                   WAIT P50  WAIT P95  WAIT MAX      MEAN AT ONCE    PEAKS   UNDER  HELD    CPU CPU HELD
caps      jvm 2, node 6                 0s     4m19s     7m38s       57s       5   18.95G       0     0    6.2        -
caps*     jvm 3, node 8                 0s     4m19s     7m38s       57s       5   18.95G       0     0    6.2        -
measured  jvm 3, node 8                 0s     1m55s     3m20s       18s       8   24.35G       4     0      7        -
cpu ×1    jvm 3, node 8                 0s     1m55s     3m20s       18s       8   24.35G       4     0      7        0

* more slots than the caps fit in MemoryMax, which the config check refuses with admission = "caps".
UNDER: jobs that peaked above what admission reserved for them. HELD: times the high-water mark held back a job the predictions had room for.
cpu ×N: measured admission, and each runner reserves its job's predicted average CPU of the slice's 32 CPUs times N. CPU: the most the running jobs' average CPUs added up to at once. CPU HELD: times the CPU budget held back a job memory and its slots had room for. The replay keeps each job's run time, so it shows what a CPU budget holds back, not how much sharing the CPUs slows the jobs.

Without --memory-max it reads the slice’s from systemd, and without --slots it gives each profile as many slots as MemoryMax holds of it alone, or its configured ones if more; --since 7d (or 12h, 2w) sets the window. A PEAKS over MemoryMax is marked !. A PEAKS well under MemoryMax is what the replay shows, not a promise: leave the daemon room for what it does not model. The cpu ×N rows replay CPU admission.

CPU admission

Memory admission does not see CPU: eight image builds that each fit in memory can still ask for more CPUs than the host has, and every job on it slows down. cpu_overcommit under [host] adds a CPU budget, the slice’s CPUs times the factor, and a runner starts only when what it reserves of that fits too:

[host]
slice = "runnerd.slice"
isolation = "sysbox"
admission = "measured"
cpu_overcommit = 1.0

• The slice’s CPUs are its AllowedCPUs, or the host’s, and fewer when its CPUQuota allows less. runnerd reads them from systemd at start, and again whenever a config is applied. A config that sets cpu_overcommit is refused when they cannot be read; without it they only scale the dashboard’s CPU chart.
• Under admission = "measured" a runner reserves its job’s average CPU, its CPU time over its run time, predicted from the job history as memory is: with [sizing] min_jobs runs of the job, their p95; with fewer, its own largest once 2 succeeded, else the larger of that and its profile’s p95; with none, its profile’s largest. Without min_jobs runs of the profile, its cpus. There is no margin: the factor is the margin.
• Under admission = "caps" every runner reserves its profile’s cpus, and every running one counts at its container’s CPU limit, also one admitted by its prediction before a reload.
• A runner reserves the largest of its pool’s queued jobs until it knows its job, then that job’s; never more than the whole budget, so one runner can always start.
• The mirror, seed containers and shells reserve no CPU.

A job that had a slot and memory but not CPU waits. Once it has waited 2 minutes, no younger job starts until it fits, so a stream of light jobs cannot keep a heavy one waiting forever. The wait counts from when the job was first held for CPU until it starts or leaves the queue, also through ticks that hold it for memory or a slot instead. The status line and the dashboard say so (CPU 30 of 32 CPUs reserved; a job waits for CPU), the daemon logs it, and the job’s record says how long it waited (runnerctl job <id>).

It is off by default, and auto sizing does not set it. Linux shares the slice’s CPUs between containers by weight, and a slice with a low CPUWeight, such as 20, leaves the desktop its share however many jobs run, so without a budget a busy host costs jobs time, not failures. A job’s average is far under its peak: the image builds that peak at 6 CPUs average under one, so reserving peaks would leave most of the host idle, and averages admit nearly everything on a host with many CPUs. On one host’s history at 32 CPUs, the running jobs’ averages never added up past 8 CPUs, even with twice the node slots, so a budget at 1.0 held nothing back. It matters on a small host. The same history replayed against 8 CPUs:

$ runnerd simulate --memory-max 33G --slots node=8 --cpus 8 --cpu-overcommit 1 --cpu-overcommit 1.5
POLICY    SLOTS                   WAIT P50  WAIT P95  WAIT MAX      MEAN AT ONCE    PEAKS   UNDER  HELD    CPU CPU HELD
caps      jvm 2, node 6                 0s     4m19s     7m38s       57s       5   18.95G       0     0    6.2        -
caps*     jvm 3, node 8                 0s     4m19s     7m38s       57s       5   18.95G       0     0    6.2        -
measured  jvm 3, node 8                 0s     1m55s     3m20s       18s       8   24.35G       4     0      7        -
cpu ×1    jvm 3, node 8                 0s     2m30s     4m21s       32s       8   24.16G       4     0      7      281
cpu ×1.5  jvm 3, node 8                 0s     2m08s     4m21s       21s       8   24.35G       4     0      7        8

Start at 1.0: the running jobs’ averages fill the CPUs, and peaks that land together cost a short slowdown, not a kill. Raise it when jobs wait for CPU the host shows idle; lower it to keep CPU free for the host itself. --cpus replays another number of CPUs, and --cpu-overcommit, repeated, compares factors.