Running

One daemon between GitHub and Docker; every client goes through its API.

The service

runnerd init writes a user unit that runs runnerd serve:

[Unit]
Description=runnerd: ephemeral GitHub Actions runners
Documentation=https://ayagmar.github.io/runnerd/

[Service]
Type=exec
ExecStart="/home/dana/.local/bin/runnerd" serve --config "/home/dana/.config/runnerd/runnerd.toml"
Restart=on-failure
RestartSec=10

[Install]
WantedBy=default.target

systemctl --user enable --now runnerd   # start now and at login
loginctl enable-linger "$USER"          # keep it running while you are logged out
journalctl --user -u runnerd -f         # its log
systemctl --user restart runnerd        # after editing runnerd.toml by hand
systemctl --user stop runnerd           # waits for running jobs, up to systemd's stop timeout

A config saved from the dashboard or PUT /v1/config applies without a restart, except [web], history.database and caches.build.directory. runnerd stop waits for running jobs too; runnerd stop --force, or a second signal, kills them.

To back up the config, copy runnerd.toml itself: runnerctl config show shows notification URLs and headers as "<secret>", and runnerd refuses to start with those.

Runner containers live in Docker, not in the unit: a restarted runnerd adopts the ones still running, and keeps the repositories it had paused paused.

How it schedules

• It reads the gh CLI’s token at start, and again when GitHub rejects it.
• Every poll_interval it asks GitHub for each repository’s queued jobs.
• A job is served by the repository’s pool whose labels include every label in its runs-on: one [[repositories]] entry, or one of several.
• For each one it registers a just-in-time runner and starts it in a free slot of that pool’s profile, if the slice’s MemoryMax still has room for the profile’s memory, or with measured admission for the job’s predicted peak, and under CPU admission if the CPU budget has room for it too. Other jobs wait in GitHub’s queue.
• The runner takes one job, its container exits, and runnerd removes it. A runner with no job after 5 minutes is removed, unless it is a warm runner.
• A restarted runnerd adopts the runner containers still running and deletes this host’s runners that have none.
• Before a profile’s first runner it pulls the profile’s image, logging the pull’s progress.

A job's way through runnerd. Only a failed job of a profile with record_failures leaves a record.

Warm runners

A job that finds no runner waits for the next poll, a registration and a container start: 15 to 25 seconds. A warm runner is already registered when GitHub queues the job, and takes it within a second or two.

• Each pool keeps warm runners (1 by default) while its repository is active: a job of it was queued in the last hour, or it is an hour of the day (UTC) its jobs came in on 3 of the last 14 days. Outside those, its warm runners go after 5 minutes idle and none starts again until a job comes.
• When a job of a workflow run is seen, the run’s other jobs get runners too: in each pool, as many as that workflow’s runs usually had there (the median of the last 14 days, from 3 runs), less the ones already seen.
• Warm runners start last: only once every queued job that fits has its runner, and while no job waits for a slot, memory or CPU. Each reserves its profile’s memory (and cpus under CPU admission, at most the CPU budget).
• A queued job of another repository that needs the room takes it: runnerd removes as many warm runners as make the job fit and starts its runner in the same poll.
• A profile due a reseed (slot 1 ran a job) removes its warm runners once its other runners are idle, so the snapshot still gets rebuilt. The shared caches’ check does the same across all profiles, at most once an hour, until it ends.
• A warm runner that ends without a job after waiting, as when Docker restarts, does not hold its repository’s next runners back; a second in a row does.

runnerctl watch, the TUI and the dashboard show them as warm runners waiting for a job; runnerctl status shows them idle. Sizing tells the jobs a warm runner took from the others, with each kind’s median wait.

Warm caches

A slot is one concurrent job of a profile. Each job gets a new container, but the slot keeps a volume over the profile’s read-only snapshot. What a job deletes of the image, or replaces, stays deleted for every slot once it is in the snapshot. The images point every common toolchain’s cache into ~/.cache: rustup and cargo, Go modules and builds, Gradle, Maven, npm, Yarn, Bun, pip, uv, Poetry, pub, Deno and coursier.

Two slots of the jvm profile. A slot's changes stay in its own volume; slot 1's become the snapshot when the profile is idle.

Cache	Kept
the slot’s Docker images, ~/.cache, tool cache, Android SDK	per slot, between jobs, shared by the jobs of every repository and branch on the profile: a job can tag an image, such as postgres:16, that a later job in the slot then runs without pulling
the slot’s Docker build cache, BuildKit’s cache mounts included	per slot, between jobs. After each job the build cache is trimmed, least recently used first, to about half the slot’s share, and a store still over the share is emptied, images too. The slots together get a quarter of the free disk plus what their volumes hold, at least 1 GB and at most 20 GB per slot. A store the slot’s Docker Engine does not start on, as after a crash, is emptied when the next container starts. A slot closed by a lower slots loses its volume at the next check of the caches. The containers, volumes and networks a job starts go after it
cargo’s credentials and config, Gradle’s gradle.properties in ~/.cache	never: no later job or record shell sees them, even from an older snapshot
the profile’s snapshot	replaced by slot 1’s changes once they reach reseed_at and the profile is idle; a snapshot that would hold more than snapshot_max starts over from the image instead, with every slot’s changes; once measured above snapshot_max, it starts over at the next moment the profile is idle, however little slot 1 changed. After a reseed that did not finish, as when the host went down during one, the profile’s jobs run on the image alone until the next reseed starts it over
pnpm store, Maven repository	shared by all slots, emptied above their max_size when no runner runs: warm runners stop for it at most hourly, at the first moment no job runs
Docker Hub mirror	runs while any runner does, warm ones too; stops, and its storage is emptied above mirror.max_size, with the shared caches’ check
build cache	per repository, shared by every slot; written only by the jobs that may: pushes, schedules and manual runs of the default branch or build.branches, and opted-in pull requests; its least recently used entries go above build.max_size with the shared caches’ check

Build cache

A Gradle task built in slot 2 is a cache miss in slot 3: each slot has its own ~/.cache. The build cache is one store every runner reaches, kept by the daemon in ~/.cache/runnerd/build-cache, one per repository. A pull request’s build gets what main built, whichever slot built it.

Tool	What it gets	When
Gradle	its HTTP build cache as the remote cache, through an init script in $GRADLE_USER_HOME/init.d	the build turns the build cache on (org.gradle.caching=true or --build-cache) and configures no remote cache of its own
sccache	its WebDAV backend (SCCACHE_WEBDAV_ENDPOINT)	the workflow runs it (RUSTC_WRAPPER=sccache) with no other backend: S3, GCS, GitHub Actions’ cache and the rest take precedence
Bazel	--remote_cache in /etc/bazel.bazelrc, with --remote_download_outputs=all	the workspace’s .bazelrc sets no --remote_cache of its own

Who writes:

• A job writes only when GitHub says it runs a push, a schedule or a manual run (workflow_dispatch) of the repository’s default branch, as GitHub said it within the last hour, or of one of build.branches, which add to it. runnerd asks GitHub which runner holds the job, what started its run, and whether the branch has the run’s commit, so a tag named main does not pass for the branch; what the job writes in its runner’s log or environment counts for nothing.
• A repository with build_cache_writers = "pull-requests" (Configuration) also lets its own branches’ pull requests write, on the pull_request event only. GitHub must say a person, not a bot, opened the pull request, opened, reopened or pushed to it for this run, and started or re-ran the run, and the branch must still have the run’s commit. Who made the commits is not checked: a person’s pull request from a branch a bot pushed writes. A fork’s pull request, Renovate’s or Dependabot’s, and a run a bot pushed to a person’s pull request read only. Whoever can push a branch to the repository can then change what main builds load, so turn it on only where that is you. With strict up-to-date pull requests it pays: the merge builds the tree the pull request’s last run built, so main finds every task in the cache. It helps only where a job’s build goes through the cache, with Gradle, sccache or Bazel; Maven, Turborepo and Flutter builds gain nothing from it.
• Every other job, pull requests from forks or not, reads. Its uploads are answered as stored and dropped, so Gradle keeps the cache on for the rest of the build.
• What a job uploads before GitHub has answered, usually its first seconds, waits and is kept or dropped once it has. A job that ends first and uploaded something is asked about right after its runner is removed; its uploads are dropped only if GitHub does not answer then.
• A repository reads only its own entries.

sccache uses the shared cache instead of its disk cache in the slot, not as well as: a pull request’s own compilations are not kept, so a second push to it compiles again what the first did. A Rust repository with many pull requests may do better with its own backend, or build_cache = false.

Each runner reaches the cache through a socket of its own, which no other runner sees; the image forwards it to 127.0.0.1:41771. A cache that does not answer, as while runnerd restarts, fails no build: Gradle warns and builds without it. Containers a job starts itself, such as a container: job’s or docker build’s, do not reach it.

The job page and runnerctl job show each job’s hits and what it read and stored; the Caches page and runnerctl caches, the cache’s size and its hit rate over 7 days. The Cold cache insight names a job whose writing runs find almost nothing in it.

To keep a repository out, set build_cache = false under it. A workflow turns off Gradle’s part with RUNNERD_BUILD_CACHE_URL: "" in its env, and any tool’s by configuring a remote cache of its own. runnerctl caches --clean runnerd-build-cache empties it while no runner runs a job.

Records of failed jobs

With record_failures on a profile (Configuration), a failed job’s stopped runner is committed to an image, runnerd-record:<job>, before it is removed. runnerctl shell <job> starts a container from it and opens bash in the job’s workspace, after printing the step that failed and what it ran:

$ runnerctl shell 120055991489
runnerd: the record of job 120055991489 of halyard/ledger, CI / build
It failed at step 5: Unit tests (exit code 1)
Its log shows it ran:
    ./gradlew test --no-daemon
    shell: /usr/bin/bash -e {0}
You are in /home/runner/_work/ledger/ledger, as the job left it. Not kept: the job's Docker containers and images; what it changed in ~/.cache, the tool cache and the Android SDK, which show the profile's snapshot; and what it added or changed among your home's dot-files and in /tmp (logins, and tools in ~/.local or ~/.nvm), except ~/.rustup and ~/.cargo's bin, registry and git. Exit to remove this container.
runner@4be1c0d2a7e9:~/_work/ledger/ledger$

--step 3 shows step 3 instead.

• The terminal goes over the API socket and keeps the size it was opened with.
• The shell’s container counts against the budget with the profile’s memory cap, and is refused when the budget cannot hold it, or under measured admission while the slice is at 90% of MemoryMax.
• It mounts the profile’s snapshot and the shared pnpm and Maven caches read-only. While it is open, the profile does not reseed (runnerctl reseed answers 409) and the shared caches are not trimmed or cleaned. It is refused while the profile reseeds or the shared caches are trimmed.
• It goes when the shell exits, when the client goes away, or after 8 hours.

What	In a record
the workspace, _work/<repository>, with build output and test reports	yes
the runner’s _diag logs, secrets masked as the runner masks them on GitHub	yes
a Rust toolchain in ~/.rustup and ~/.cargo’s bin, registry and git, for a job that sets RUSTUP_HOME and CARGO_HOME there (the images put both in ~/.cache)	yes
anything else the job wrote outside volumes and the places below, such as a tool unpacked in /opt	yes
other dot-files and dot-directories the job added or changed in the runner’s home: logins (.gitconfig, .aws, .gnupg, .docker, ~/.cargo/credentials.toml and config.toml, the runner’s .credentials) and tools installed there (~/.local, ~/.nvm, ~/.dotnet, ~/.bun, ~/.sdkman); what it added or changed in /tmp and /var/tmp	never
the JIT config, _work/_temp (step scripts), the token actions/checkout puts in .git/config	never
the slot’s caches (rustup and cargo included), tool cache, Android SDK, and the job’s Docker images	no; volumes, which a shell sees as the profile’s snapshot

The runner image deletes the “never” rows when the runner exits, and runnerd refuses to keep a container whose changes still hold any of them. A step that saves a secret into the workspace or elsewhere outside those places leaves it in the record: runnerd does not read file contents.

Records are readable by anyone who can use Docker on this host (docker run runnerd-record:<job>), and by the user runnerd runs as, through its socket, which is the only client the socket accepts. Turn record_failures on only for profiles whose jobs you would let those users see.

Records take disk: committing one takes a second or two, and it takes about twice what the job wrote outside volumes, since Docker keeps the layer packed and unpacked. runnerctl records lists them with their limits; runnerctl records rm <job>, --all or --older-than 2d removes them. The job history keeps, after a record is gone, when it was made, its size, and why it went (count, size, age or user).

Job history

runnerd records every job its runners may run in a SQLite database; [history] sets its path and how long each part is kept. For each job:

What	Kept
its timeline: queued, container started, job started, finished, container and runner registration removed	with the job
its steps with their start and end, as GitHub shows them	with the job
CPU, memory and network of its container, sampled every 2 s	samples for history.samples; totals and peaks with the job
the bytes the Docker Hub mirror served it	with the job
its log, as GitHub shows it with masked secrets still masked, up to history.log_max	for history.logs

A job that leaves GitHub’s queue without one of this machine’s runners is recorded as elsewhere (the dashboard’s “Not run here”): cancelled in queue when its run was cancelled first, as a newer push does, or its conclusion otherwise, such as skipped. It has no start, wait or run time, and no median or cost counts it. Only a job another machine’s runner took keeps the word elsewhere.

A finished job’s record goes after history.jobs, with its steps and runner. The database file does not shrink when rows go; SQLite reuses the space. runnerd also keeps the sizes of the cache volumes and the slice’s CPU and memory over time.

Where the data comes from:

• Which job a runner took: a small script runnerd runs in the container (docker exec, as root) reads the job’s ID from the runner’s _diag/Worker_*.log within a second of the job starting. GitHub’s list of queued jobs still shows the job for several seconds after that, and runnerd does not start a second runner for it.
• Steps: GitHub’s jobs API, read every poll while a job runs. The runner’s own logs name steps differently from GitHub (Set up runner for Set up job) and change between runner versions. The API’s times have second precision.
• The log: the runner’s _diag/pages, which the same script follows. The runner writes the log there in 4 KiB blocks, so lines arrive a block at a time, and a quiet step’s lines arrive when it ends.
• CPU and memory: the container’s and the slice’s cgroup files, every 2 s; Admission and sizing says what the memory counts. Network from /proc/<pid>/net/dev.

API

JSON on $XDG_RUNTIME_DIR/runnerd.sock, mode 0600: only the user runnerd runs as may connect. The types are in crates/core/src/api.rs and its modules. Times are seconds since the Unix epoch, sizes bytes, CPU millicores.

Request	Response
GET /v1/status	profiles and their slots, running runners, repositories with their pools and queued jobs, each job’s pool and why it waits, the memory budget with the mirror’s and each seed container’s share, what the host’s memory leaves the slice, the slice’s CPUs, and under cpu_overcommit the CPU budget
POST /v1/repositories/{owner}/{name}/pause?force=	no new runners; 409 naming the running job, unless force=true cancels its run
POST /v1/repositories/{owner}/{name}/resume	204
POST /v1/repositories/pause?force=, POST /v1/repositories/resume	the same for every configured repository; while any runs a job, 409 naming the jobs, and none is paused
POST /v1/profiles/{name}/reseed	202; the profile starts no runner until its slots are free and slot 1 became its snapshot; 409 while it reseeds or a shell is open in one of its records
POST /v1/caches/{name}/clean	204; removes a slot’s volume or a shared cache, stopping the runners that wait for a job and use it, warm ones too; 409 naming the jobs that use it
GET /v1/config	the config file’s path, text and revision; notification targets’ url and header values read "<secret>"
POST /v1/config/check with TOML	what runnerd check-config says about it, each "<secret>" read as the value it stands for
PUT /v1/config?revision= with TOML	checks it, with each "<secret>" read as the value it stands for, writes it in place of the file and runs with it; invalid (422), not writable (500) or runnerd stopping (503) leaves the file and what runs as they were. With revision, from GET /v1/config, a config replaced since, as by auto sizing, or a file edited on disk since, is not overwritten (409); the reply’s revision is that of the text written, to send with the next change
GET /v1/doctor	runnerd doctor’s checks, for the config runnerd runs with
POST /v1/shutdown with {"force": false}	202
GET /v1/dashboard	the dashboard’s [web] open, as runnerd read it at start
POST /v1/dashboard/sign-out	204; signs every browser out of the dashboard and ends every login link and pairing code made before
POST /v1/dashboard/pairings?origin=	a 5-minute one-time login link for origin, one of [web] origins, or one for each of them; 409 when the dashboard is off or has no [web] origins
GET /v1/jobs?repository=&state=&conclusion=&workflow=&name=&since=&until=&before=&limit=	a page of jobs, newest first; next is the before of the next page
GET /v1/jobs/{id}	a job with its timeline, steps, metrics and log size
GET /v1/jobs/{id}/samples	its CPU, memory and network over time
GET /v1/jobs/{id}/logs?from=&follow=	its log as text; with follow=true, open until the job’s record is final
GET /v1/host/samples?since=	the slice’s CPU and memory, the last 24 hours at most
GET /v1/stats?repository=&since=&until=&bucket=day	finished jobs per repository and day (or hour), 744 buckets back at most: counts, queue and run time, CPU, memory, mirror bytes
GET /v1/activity?since=	every job runnerd’s runners started or finished from since (default 24 hours ago) until now, by when it started or ended rather than when it was queued: each start with how long the job waited, how many finished and failed, and how many of the failed their runner failed: lost with it, or killed for memory at its cap
GET /v1/sizing?since=	per configured profile and per job (repository, workflow, name): percentiles of the sampled memory and CPU peaks, waits, how busy the slots were, and the advice for memory and slots; whether auto sizing is on, the changes it made, and while on what its next look would change, which profiles a recent change holds, which cpus raises it cannot make and why, and why no look writes anything when the file on disk was edited since runnerd read it; the last 30 days by default
GET /v1/insights?days=	what the history says (Insights): slower, flaky and near-cap jobs, repositories whose downloads memory pressure throttled, waits that got longer, and each repository’s waits this week against the week before; 14 days by default, 90 at most
GET /v1/digest	the last 24 hours: jobs finished and failed, the insights new since the day before, the three longest waits, and the month’s cost estimate
GET /v1/savings	what this machine’s job minutes would have cost on GitHub-hosted runners, this month (UTC) and over the last 30 days, with the jobs left out and why
POST /v1/notify/test?target=	sends a test message to every [notify] target, or the one named, and says what each did; 404 without targets
GET /v1/caches, GET /v1/caches/history?since=	cache volume sizes and their limits, now and over time
GET /v1/records	every record of a failed job, oldest first, with the limits and the jobs whose record has a shell open
DELETE /v1/records/{job}, DELETE /v1/records?all=true, DELETE /v1/records?older_than=2d	the jobs whose records went and the disk they took; records with a shell open stay, and removing one by its job is refused with 409
POST /v1/records/{job}/shell?step=&cols=&rows= with Connection: Upgrade, Upgrade: runnerd-tty	101, then the connection is the shell’s terminal, raw bytes both ways; 409 when the budget is full
GET /v1/events?job=&logs=	server-sent events, below

curl --unix-socket "$XDG_RUNTIME_DIR/runnerd.sock" http://runnerd/v1/status
curl --unix-socket "$XDG_RUNTIME_DIR/runnerd.sock" -X POST http://runnerd/v1/repositories/halyard/ledger/pause
curl -N --unix-socket "$XDG_RUNTIME_DIR/runnerd.sock" http://runnerd/v1/events

Events

Event	Sent when
job.queued, job.started, job.step, job.finished	a job moves on
slot.changed	a runner comes or goes in a slot
profile.changed	a reload adds or changes a profile, or it starts or stops reseeding
profile.removed	a reload removed it and its last runner is gone
repository.changed	paused or resumed, its queue changed, or a reload added or changed it
repository.removed	a reload removed it
budget.changed, cache.reseeded	the budget changed; a profile has a new snapshot
record.created, record.deleted	a failed job’s runner was kept as a record; a record went, and why
sizing.changed	auto sizing wrote a profile’s new memory and slots; it carries the old ones
runner.failed	a runner could not be registered or started, and the repository gets none for a while; once until one of its runners takes a job again
job.sample, host.sample, log.line	live: a sample, or a log line with logs=true

Every event but the live ones has an ID: send the last one seen as Last-Event-ID and the stream resumes after it, across daemon restarts. Live events are not stored; a client that missed some reads the job’s samples or log.

slot.changed, profile.changed, budget.changed and repository.changed carry the state after a change, not each transition: a pause and a resume that land before the daemon publishes its status send nothing. Applying profile.* and slot.changed to a status’s profiles, and repository.* to its repositories, keeps them equal to what /v1/status returns.

Pausing

A forced pause cancels the run of each of the repository’s running jobs. GitHub cancels runs, not single jobs, so the run’s other jobs are cancelled too. The runner reports the job cancelled, which takes under a minute; one still running 90 s later is removed, and its registration is deleted once GitHub lets go of it.

A paused repository’s runners still waiting for a job, warm ones included, stop at once, so none takes a job queued before the pause. A pause lasts until a resume, across restarts: the job history records the paused repositories, and a restarted daemon starts no runner for them. If the history cannot record a pause or resume, the change holds until the daemon restarts, and the request fails with 500 saying so. A reload that removes a paused repository forgets its pause; added back, it starts active.

Reloading

A new config takes effect without a restart, and running jobs keep running. A removed profile, or slots past a lowered slots, stay until their runners finish their jobs, and no runner starts in a profile while it runs as many as its slots. A runner waiting for a job in a slot past a lowered slots is removed. A runner still waiting for a job, warm ones included, is removed when its repository is removed or moved to another profile, or when the repository’s labels or its profile’s image, memory or cpus changed, by hand or by auto sizing. For a new image, that waits until Docker has it, so the old runners serve jobs while it pulls. Runners started afterwards get the new limits, image and labels. history.database, caches.build.directory and [web] change only at the next start.