haylan/godot-ci
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
godot-ci/docs/research/build-caching.md
T
haylan 08b0fedf6f docs(research): investigate Docker build caching options 
Ranked recommendation: fix the docker system prune -af --volumes step
first (it deletes BuildKit's build cache before every build, so no
Dockerfile change can help until that stops), then split the
apt-get install layer from the Godot/Blender download layer so the
~470s apt layer becomes cache-eligible across nightly runs. Registry
cache and Gitea Actions' actions/cache don't fit this repo's single
persistent runner / multi-GB nightly-changing layers.

Research only, no code changes.
2026-08-07 00:52:49 +02:00

18 KiB
Raw Permalink Blame History

Can this repo's Docker build be sped up, and does caching exist for it?

Research triggered by the real-run timing gap: ~5 min locally vs ~16 min on the actual runner (run 1493), dominated by the single apt-get install layer (~470s, build-essential, mingw-w64, X11/audio/udev dev headers) and the Godot export templates + Blender downloads.

Verdict: yes — but the fix is "stop deleting the cache", not "add a cache"

The runner (runs-on: ubuntu-latest in .gitea/workflows/publish.yml) is a self-hosted, persistent act_runner host, not an ephemeral GitHub-style VM — the workflow's own "Free up runner disk" step and its comment only make sense on a persistent box ("the runner's disk fills up over successive runs until a push … is what finally fails"; an ephemeral runner starts clean every time and would never accumulate anything to prune). That persistence is exactly what makes plain Docker build-cache (no registry, no extra tooling) viable here — and it's also exactly what today's docker system prune -af --volumes step throws away before every single build, per Docker's own docs: system prune's default removal set explicitly includes "Unused build cache", and -a widens that to "all unused build cache," not just dangling entries (docs.docker.com/reference/cli/docker/system/prune). So right now, local BuildKit cache cannot survive from one nightly run to the next no matter how the Dockerfile is written — the prune step deletes it every time, before the build that could have reused it even starts.

Ranked recommendations:

  1. Stop nuking the build cache in the prune step. Replace docker system prune -af --volumes with targeted pruning that skips builder cache: docker container prune -f && docker image prune -af && docker volume prune -f, plus a separate, bounded docker builder prune -f --filter until=24h (or --keep-storage=<N>GB) to cap build-cache growth without wiping cache that's still same-day reusable. This alone is what unlocks everything below — no Dockerfile change helps while the current step runs first. Near-zero risk: it still frees the disk that motivated the step, just not by deleting reusable build cache along with it.
  2. Split the apt-get install layer from the Godot/Blender download layer in the Dockerfile, apt first. Right now both live in one giant RUN chained with && (Dockerfile lines 19–62), so a single-byte change anywhere in that chain — including the Godot/Blender URLs that do change nightly — invalidates the apt install too. BuildKit's cache is a linear, per-instruction, content-addressed chain: it reuses a layer only if every instruction up to and including it is byte-identical to a cached run (Docker's build-cache docs: cache is invalidated "from that point" the instant one instruction's content changes, docs.docker.com/build/cache). The apt package list changes essentially never; GODOT_VERSION/ BLENDER_URL change most nights. Splitting them means the ~470s apt layer becomes a cache hit on every night that doesn't touch the Dockerfile itself, while the Godot/Blender layer keeps rebuilding (it has to — new content every time a version bumps). Combined with #1, this is the actual fix for the dominant cost in the real run.
  3. Add RUN --mount=type=cache for apt's own directories (/var/cache/apt, /var/lib/apt, sharing=locked) as a smaller supplement to #2 — it persists apt's downloaded .deb cache independent of the Dockerfile-instruction cache chain, so even a Dockerfile edit that invalidates the layer doesn't force re-downloading every .deb from Debian's mirrors, only re-running dpkg against the still-local cache (docs.docker.com/reference/dockerfile/#run---mounttypecache). Worth doing alongside #2 since it's a few lines, but it's a hedge against Dockerfile churn, not the main win — #2 already gets full cache hits on the common case (no Dockerfile change, only build-args change).
  4. Skip registry-based cache (--cache-to/--cache-from type=registry). Not worth it here. Registry cache exists to share a build cache across machines that don't share disk — ephemeral GitHub-hosted runners, fleets of self-hosted runners, etc. (docs.docker.com/build/cache/backends/registry). This repo has exactly one persistent runner with its own disk, so #1–#3 already give it that same cache locally, for free, with no push/pull round-trip. Pushing a separate cache image on top would add push/pull time on every run for a benefit local cache already provides, and it's a bad fit for this specific image's shape: the Godot/Blender layers are multi-GB and, by design, different nearly every night (new versions), so mode=max would repeatedly re-push multi-GB "cache" blobs that are very unlikely to ever be pulled back down. There's also an open question of whether it would even work: Gitea's container registry only gained general OCI-artifact support in 1.24 (go-gitea/gitea#25846, closed via PR #34666) — before that it rejected non-image manifests with "unsupported: Schema version is not supported". This repo's Gitea instance version wasn't confirmed as part of this research, so pushing a buildx cache manifest (which uses custom OCI media types, distinct from a normal image manifest) is not guaranteed to work without checking that first.
  5. Gitea Actions' actions/cache action is the wrong tool for this, noted for completeness. Gitea Actions does support a GitHub-compatible actions/cache@v3/v4 via a cache server that ships enabled by default in act_runner (about.gitea.com tutorial; docs.gitea.com/usage/actions/overview), plus a separate RUNNER_TOOL_CACHE mechanism. Both are key/value or directory caches meant for things like node_modules/pip wheels between workflow steps — not a Docker layer-cache mechanism, and not something docker build/docker push (what this workflow actually calls, see below) has any hook into. It doesn't apply to this problem; #1–#3 do.

1. What the workflow and Dockerfile actually do today

.gitea/workflows/publish.yml runs plain docker build and docker push — not docker/build-push-action or docker buildx build with any cache flags (lines 86–89, 96/105/109). runs-on: ubuntu-latest is a Gitea-side runner label, matched against whatever labels this instance's act_runner was registered with — Gitea's own runner docs describe labels as exactly this kind of local mapping (ubuntu-latest:docker://... style entries a runner operator configures, not a GitHub-hosted VM type) (search result summary of docs.gitea.com/usage/actions, act_runner label docs; the specific main_runner name mentioned as a possible identifier for this instance's runner doesn't appear anywhere in this repo's tracked files — it would only show up in the Gitea Actions run UI, not the workflow YAML). The disk-fill problem the "Free up runner disk" step's own comment describes ("nothing here ever pruned old ones, so the runner's disk fills up over successive runs") is only possible on a runner whose disk persists between runs — corroborated independently by act_runner's own architecture: its Docker executor keeps a long-lived host Docker daemon and reuses the image cache across job runs by default (act_runner documentation on the docker/dind executor flavors and idle-cleanup behavior — runner cleans up stale workspaces on an interval rather than starting from a fresh disk each job).

The Dockerfile's expensive work is one RUN chain (lines 19–62): a single apt-get install covering build-essential, scons, mingw-w64, and the X11/audio/udev dev headers, immediately followed — in the same layer, via && — by the curl+unzip of Godot's editor and export templates and the curl+tar of the Blender tarball. Because it's one RUN, it's one cache entry: any change anywhere in it (including the Godot/Blender URLs, which change on essentially every nightly build since the workflow tracks newest-stable) invalidates the whole thing, apt install included.

2. Docker/BuildKit's own caching mechanisms (docs.docker.com)

  • Layer cache (default, local, free): BuildKit caches each instruction's result keyed on that instruction's content plus everything before it in the Dockerfile; a build on the same machine reuses it automatically with no extra flags, provided the cache wasn't evicted (docs.docker.com/build/cache). This is what #1+#2 above unlock for this repo — it needs nothing but (a) not deleting it and (b) ordering the Dockerfile so the part that doesn't change (apt) comes before the part that does (downloads).
  • RUN --mount=type=cache: a build cache mount, separate from the layer cache — a directory that survives across builder invocations without itself being part of the cached layer, intended for package- manager caches like apt's /var/cache/apt//var/lib/apt. Needs sharing=locked for apt specifically, since apt needs exclusive access to its own state (docs.docker.com/reference/dockerfile/#run---mounttypecache). Persists even when the layer cache above gets invalidated by a Dockerfile edit — a smaller, complementary win, not a replacement for #2's layer split.
  • --cache-to/--cache-from type=registry: exports/imports build cache to/from an OCI registry, as a separate artifact from the final image, specifically to let machines that don't share local disk share a cache (mode=max caches every stage, not just the final one — bigger push, more reuse potential) (docs.docker.com/build/cache/backends/registry). Not needed for a single persistent runner (see verdict §4).
  • BUILDKIT_INLINE_CACHE/type=inline: embeds cache metadata directly in the pushed image manifest instead of a separate cache artifact — Docker's own CI guidance is "in most cases you want to use the inline cache exporter" for simple cases, but it only supports mode=min (final stage only) versus registry cache's min/max (docs.docker.com/build/ci/github-actions/cache). Same applicability caveat as registry cache: solves a multi-machine problem this repo doesn't have.
  • Docker Engine ≥23 makes BuildKit (via buildx) the default builder for plain docker build — "Set Buildx and BuildKit as the default builder on Linux. Alias docker build to docker buildx build" (docs.docker.com/engine/release-notes/23.0). So the workflow's existing plain docker build call (no buildx in the command) should already be getting BuildKit's layer cache today — it's the prune step deleting it that matters, not the build command needing to change to buildx.

3. docker system prune -af --volumes vs docker builder prune

Confirmed directly against Docker's CLI reference:

  • docker system prune's default removal set is stopped containers, unused networks, dangling images, and unused build cache; -a/--all widens image removal to all unused images (not just dangling), and --volumes additionally removes unattached anonymous volumes (docs.docker.com/reference/cli/docker/system/prune). There's no flag on system prune to keep build cache while still pruning everything else — build cache removal isn't optional once you call it.
  • docker builder prune is the narrower, cache-only equivalent: --all to remove all unused cache (not just dangling), --filter until=<duration> to only remove cache older than a given age, --keep-storage=<size> to cap total cache size instead of clearing it outright (docs.docker.com/reference/cli/docker/builder/prune).

This is the concrete lever for recommendation #1: keep freeing the disk the workflow's own comment says filled up (old images, old anonymous volumes), but do it with container prune/image prune -af/volume prune instead of system prune, and bound the build cache separately with docker builder prune -f --filter until=24h (or a --keep-storage cap) rather than deleting all of it unconditionally before every build. A same- day rerun (e.g. workflow_dispatch shortly after the nightly cron) would then still get a cache hit on the apt layer; a multi-day-old apt cache entry still gets swept on the next age-filtered prune, so disk doesn't grow unbounded either.

4. Layer-order restructuring compatibility

Splitting Dockerfile's one RUN into "apt install" then "Godot/Blender downloads" is a pure reorder/split — no semantic change, no new dependencies, and it's exactly what BuildKit's per-instruction cache keying is designed to reward (§2). Its payoff is entirely contingent on §3: with docker system prune -af --volumes still running first, splitting the layer changes nothing, because the reusable apt-layer cache entry is deleted moments before the build that would've hit it starts. The two recommendations are a pair, not independent options.

5. Registry cache realism for this specific image

Weighed directly against this repo's shape:

  • Storage cost: mode=max registry cache stores every stage as its own cache layer set in the registry, separate from the pushed image (docs.docker.com/build/cache/backends/registry) — for an image whose expensive layers are multi-GB Godot export templates and a Blender tarball that both change nearly nightly, that's another near-full copy of those multi-GB layers pushed to Gitea's package registry on top of the image push that already happens, for cache entries with a short effective lifetime (superseded the next time Godot/Blender bump).
  • Gitea registry support: Gitea's container registry is OCI-compliant for standard image/Helm-chart manifests (docs.gitea.com/usage/packages/container), but general OCI-artifact support (the category buildx's cache manifest falls into, using non-image media types) only landed in Gitea 1.24 via go-gitea/gitea#25846 — prior to that, pushes of non-standard-image manifests failed with 501 unsupported: Schema version is not supported. This repo's actual Gitea server version wasn't checked as part of this research, so this is a real "verify before use" gate, not just a style preference.
  • Time saved: none of it addresses the local-persistent-runner case this repo actually has — §2/§4's plain layer-cache fix gets the same or better result (apt-layer cache hits) for free, with no push/pull latency and no registry storage growth.

Net: registry cache isn't wrong in general, it's solving a problem (cache sharing across machines without shared disk) this repo doesn't have, at a storage and reliability cost this repo's layer shapes make worse than typical.

Sources

Reference in New Issue View Git Blame Copy Permalink