Skip to content

Will my build reuse the last one's work?

Yes, in three layers: the slot a job runs in keeps what the last job there left, a build cache per repository serves every slot, and actions/cache restores from this machine. runnerctl caches shows each volume, its size and the build cache’s hits:

$ runnerctl caches
measured 15m 01s ago
VOLUME HOLDS SIZE LIMIT
runnerd-jvm-base jvm snapshot 14G -
runnerd-jvm-slot-1 jvm slot 1 layer 1.76G -
runnerd-jvm-slot-2 jvm slot 2 layer 640M -
runnerd-node-base node snapshot 9G -
runnerd-node-slot-1 node slot 1 layer 950M -
runnerd-node-slot-2 node slot 2 layer 300M -
runnerd-pnpm pnpm store, shared 3.39G emptied above 5G
runnerd-maven Maven repository, shared 4.1G emptied above 5G
runnerd-mirror Docker Hub mirror 7G -
runnerd-build-cache build cache, shared 6.29G trimmed to 10G
build cache for Gradle, sccache, Bazel and Turborepo, and Nx where a workflow opts in
18412 entries; 81% of 51085 lookups hit in 7 days, 38G served, 5G stored, 1204 uploads refused
GitHub Actions' cache: served from the build cache on this host, in place of GitHub's, and counted with it; [caches.build] actions_cache = false turns it off
action archives
23 action archives, 41M, refreshed 4m 01s ago
The Caches page: 47.38G in 10 volumes. Each volume is a row with its size, its bound where it has one, drawn as a fill against it, its size over 30 days and one labelled button: Reseed from slot 1 for a profile's snapshot, Empty for the rest. Under them, the build cache's week as figures: entries, lookups hit, served, stored and uploads refused.The Caches page: 47.38G in 10 volumes. Each volume is a row with its size, its bound where it has one, drawn as a fill against it, its size over 30 days and one labelled button: Reseed from slot 1 for a profile's snapshot, Empty for the rest. Under them, the build cache's week as figures: entries, lookups hit, served, stored and uploads refused.

Each job gets a new container, but its slot keeps a volume over the profile’s read-only snapshot, and a job goes to the slot that last ran it when that slot is free.

The warm caches of one profileEach slot of the jvm profile has its own read-write volume, runnerd-jvm-slot-1 and runnerd-jvm-slot-2, holding its Docker Engine's data and its changes to ~/.cache, the tool cache and the Android SDK. Under both slots is the profile's read-only snapshot, runnerd-jvm-base, and under that the runner image. When the profile is idle and slot 1 changed by reseed_at, slot 1's layers become the snapshot. Every slot of every profile shares the pnpm store, runnerd-pnpm, the Maven repository, runnerd-maven, the Docker Hub mirror, runnerd-mirror, and the build cache the daemon keeps per repository, which pushes to the default branch write and every job reads.jvm 1's containerjvm 2's containerslot 1, read-writeits Docker images,~/.cache, tool cacheslot 2, read-writeits Docker images,~/.cache, tool cachejvm snapshot, read-onlyrunnerd-jvm-basethe profile's runner imageread-only, as pulledreseeded from slot 1when the profile is idleshared by every slotpnpm storeemptied abovemax_sizeMaven repoemptied abovemax_sizeDocker Hubmirrorwhile runners runbuild cachemain writes,all readrunnerd-pnpm, runnerd-maven, runnerd-mirror, and thedaemon's build cache, one per repository
Two slots of the jvm profile. A slot's changes stay in its own volume; slot 1's become the snapshot when the profile is idle.
Cache Kept
the slot’s Docker images, ~/.cache, tool cache, Android SDK per slot, between jobs, for every repository and branch
the slot’s Docker build cache per slot. Build contexts go first, then layers, least recently used first, to about half the slot’s share; cache mounts (RUN --mount=type=cache) stay up to three quarters. Slots together get a quarter of the free disk plus what their volumes hold, 1 GB to 20 GB each. When Docker’s root is a file system in a file, the free disk is the lesser of its own and the free space under the file
cargo credentials, Gradle’s gradle.properties and init scripts, Maven’s settings*.xml, Hugging Face tokens never kept. Anything else a job leaves, jobs of other repositories see (trust boundary)
the profile’s snapshot replaced by slot 1’s changes once they reach reseed_at and the profile is idle, which drops every slot’s own changes. One that would pass snapshot_max starts the tool cache over from the image, then the Android SDK, and all of it when that is not enough, so Gradle and npm keep their caches when they can. Python environments made with a tool cache’s interpreter, as pipx’s and Poetry’s, go with it
pnpm store, Maven repository, Docker Hub mirror shared by all slots; emptied above their max_size when no runner runs
build cache per repository; least recently used entries go above build.max_size

docker/setup-buildx-action starts a new BuildKit container per job, which keeps none of this; set driver: docker to build on the slot’s Docker Engine, as docker build does.

Runners skip fsync, fdatasync, sync, sync_file_range and syncfs, which keep their caches whole across a power loss and cost the disk writes. Only runners of an image that carries the filter skip them; runnerd reads the image’s runnerd.skip-syncs label before each start, and a runner of an older image keeps its syncs. Their layers are mounted with overlayfs’s volatile option too, so a runner that stops does not make the disk commit what its layers wrote; a kernel that refuses the option gets the plain mount, which the runner’s output says. The Docker Engine inside a runner and its builds keep the sync their own overlays make when unmounted: Docker mounts a container’s files more than once, which volatile forbids. A container that stops or is killed loses nothing to either, and neither does an orderly shutdown or reboot, which writes everything before it powers off. A host crash or power loss can tear what runners wrote in about the last minute: a file comes back short, empty, zero-filled in places, or part old and part new, and a Docker Engine can start on a store whose databases are torn.

So runnerd empties what runners write after such a crash, before any runner starts:

After the host boots again
with a stop since that left no runner running nothing is emptied
after an orderly shutdown or reboot, runners busy or not nothing is emptied: the journal shows the shutdown’s final sync
after a crash, or a journal that does not say every slot’s volume Docker has (layers and Docker store), of profiles and slots the config no longer lists too, and the pnpm and Maven volumes are emptied, after runnerd’s stopped containers on them; jobs start cold
a volume Docker does not remove no runner starts until a later tick removes it. If a running container, or one that is not runnerd’s, mounts it, runners start on it after 5 ticks, a slot’s layers deleted first and its Docker store as it is, and the log names it and the docker volume rm to run

runnerd reads the journal of the boot that ended. A host whose journal is not kept across boots (Storage=volatile, or no /var/log/journal), or whose runnerd user cannot read it, cannot tell a reboot from a crash and empties the caches after every boot once runners skipped their syncs; runnerd doctor says so and how to fix it.

Snapshots, the mirror and the build cache keep their syncs and are not emptied. A record keeps its files as they are.

The calls also return 0 on a file descriptor the kernel would refuse, so a test that expects fsync on a closed descriptor to fail fails here. skip_syncs = false under [host] keeps the syncs, for runners started after the reload.

One store per repository, kept in ~/.cache/runnerd/build-cache, that every runner reaches: a task built in slot 2 is a hit in slot 3, and a pull request’s build gets what main built.

Tool Uses the cache when
Gradle the build enables caching (org.gradle.caching=true or --build-cache) and sets no remote cache
sccache the workflow runs it (RUSTC_WRAPPER=sccache) with no other backend
Bazel .bazelrc sets no --remote_cache
Turborepo always; a TURBO_API, TURBO_TOKEN or TURBO_TEAM the workflow sets wins
Nx the workflow opts in (below) and uses no Nx Cloud

Nx fails the run when its cache does not answer, so a step before Nx opts in:

- run: '[ -z "$RUNNERD_BUILD_CACHE_URL" ] || echo "NX_SELF_HOSTED_REMOTE_CACHE_SERVER=$RUNNERD_BUILD_CACHE_URL/nx" >> "$GITHUB_ENV"'

A job writes only when GitHub says it runs a push, schedule or manual run (workflow_dispatch) of the default branch or of one of build.branches. With build_cache_writers = "pull-requests" (Configuration), a pull_request event from the repository’s own branch that a person, not a bot, opened, reopened or pushed to also writes. Anyone who can push a branch can then change what main builds load, so enable it only where that is you. runnerd asks GitHub about each job’s runner, trigger and commit; nothing a job writes to its log or environment counts. Every other job, fork and bot pull requests included, reads only; its uploads are answered as stored and dropped.

sccache uses the shared cache instead of its disk cache in the slot, not as well as it: a pull request’s own compilations are not kept, so a second push compiles again what the first did. A Rust repository with many pull requests may do better with its own backend or build_cache = false.

Containers a job starts itself, such as container: jobs, do not reach the cache. build_cache = false under a repository keeps it out; a workflow turns off Gradle’s part with RUNNERD_BUILD_CACHE_URL: "" and Turborepo’s with TURBO_CACHE: local:rw. runnerctl caches --clean runnerd-build-cache empties it.

The build cache also serves actions/cache, the cache: input of setup-node, setup-java, setup-python and setup-gradle, and Docker’s type=gha cache, with no workflow change, in place of GitHub’s. Entries count within build.max_size. build.actions_cache = false under [caches], or a repository’s build_cache = false, leaves jobs on GitHub’s cache; runnerd doctor and runnerctl caches say which.

Entries are scoped as on GitHub, and only the jobs that may write save. A default-branch or build.branches job saves into its branch and restores from it, then the default branch. An opted-in pull request saves into its own scope and restores from it, then its base branch, then the default branch. Every other job restores from the default branch, and its saves report success and are dropped. Artifacts (upload-artifact, download-artifact) still go to GitHub through runnerd and fail while it restarts; the actions retry for about 25 seconds. container: jobs, docker:// actions and containers a job starts use GitHub’s cache.

Before a job’s first step, the runner downloads every action the job uses from GitHub. runnerd keeps those archives on the host, and the runner copies one from there instead of downloading it. The runner still asks GitHub which commit each action is at, so an outage of GitHub’s API still fails a job’s setup; a failing download from GitHub no longer does for an action runnerd keeps.

Every 10 minutes at most, runnerd reads the workflows and action files on each served repository’s default branch. It keeps the public actions their steps pin by commit (uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955), and the actions those pin in turn, nine levels deep. An action pinned by tag or branch, a local action, a docker:// image, a reusable workflow, a private action and an action whose archive holds a symbolic link are downloaded by the runner as before. Job logs never decide what is downloaded, and runnerd downloads archives without its token.

Each archive is checked before a runner can read it: it must be the tar git archive writes, unpack within size limits under one directory, with nothing leading out of it, and name the commit it was asked for. At start and at each refresh, runnerd checks every kept archive against what it verified. A changed one moves to quarantine/ and is downloaded again. A runner mounts the archives read-only, and only once that check found them good: a runner that starts while a check runs gets none. After a restart, runners that kept running with the archives mounted are counted until the first check, and named in the attention list if it moved a file they may have read. An archive stays for actions.keep after the last time a repository pinned it, and goes at once if its repository is no longer public. While a repository’s workflows cannot be read in full, what it last pinned stays, and the caches page says so. The archives take at most 8G.

When a job’s log shows the runner downloading an action as an immutable package, runnerd removes that repository’s archives and never keeps it again, since the runner would read the archive in the package’s place.

runnerctl caches and the dashboard’s Caches page show how many archives are kept, their size, when they were last refreshed, and what failed. A failure also shows in the attention list. actions.enabled = false turns them off.