2026-08-06 23:00:32 +00:00
1 changed files with 282 additions and 0 deletions
@@ -0,0 +1,282 @@
+# Can this repo's Docker build be sped up, and does caching exist for it?
+
+Research triggered by the real-run timing gap: ~5 min locally vs **~16 min** on
+the actual runner
+(run [1493](https://git.arthurerlich.de/haylan/godot-ci/actions/runs/1493)),
+dominated by the single `apt-get install` layer (~470s, `build-essential`,
+`mingw-w64`, X11/audio/udev dev headers) and the Godot export
+templates + Blender downloads.
+
+## Verdict: **yes — but the fix is "stop deleting the cache", not "add a cache"**
+
+The runner (`runs-on: ubuntu-latest` in
+[`.gitea/workflows/publish.yml`](../../.gitea/workflows/publish.yml)) is a
+**self-hosted, persistent** [act_runner](https://gitea.com/gitea/act_runner)
+host, not an ephemeral GitHub-style VM — the workflow's own "Free up runner
+disk" step and its comment only make sense on a persistent box ("*the
+runner's disk fills up over successive runs until a push … is what finally
+fails*"; an ephemeral runner starts clean every time and would never
+accumulate anything to prune). That persistence is exactly what makes plain
+Docker build-cache (no registry, no extra tooling) viable here — and it's
+also exactly what today's `docker system prune -af --volumes` step throws
+away before every single build, per Docker's own docs: `system prune`'s
+default removal set explicitly includes *"Unused build cache"*, and `-a`
+widens that to "all unused build cache," not just dangling entries
+([docs.docker.com/reference/cli/docker/system/prune](https://docs.docker.com/reference/cli/docker/system/prune/)).
+So right now, **local BuildKit cache cannot survive from one nightly run to
+the next no matter how the Dockerfile is written** — the prune step deletes
+it every time, before the build that could have reused it even starts.
+
+Ranked recommendations:
+
+1. **Stop nuking the build cache in the prune step.** Replace
+   `docker system prune -af --volumes` with targeted pruning that skips
+   builder cache: `docker container prune -f && docker image prune -af &&
+   docker volume prune -f`, plus a separate, bounded
+   `docker builder prune -f --filter until=24h` (or `--keep-storage=<N>GB`)
+   to cap build-cache growth without wiping cache that's still same-day
+   reusable. This alone is what unlocks everything below — no Dockerfile
+   change helps while the current step runs first. Near-zero risk: it still
+   frees the disk that motivated the step, just not by deleting reusable
+   build cache along with it.
+2. **Split the `apt-get install` layer from the Godot/Blender download
+   layer in the Dockerfile**, apt first. Right now both live in one giant
+   `RUN` chained with `&&` (`Dockerfile` lines 19–62), so a single-byte
+   change anywhere in that chain — including the Godot/Blender URLs that
+   *do* change nightly — invalidates the apt install too. BuildKit's cache
+   is a linear, per-instruction, content-addressed chain: it reuses a layer
+   only if every instruction up to and including it is byte-identical to a
+   cached run (Docker's build-cache docs: cache is invalidated "from that
+   point" the instant one instruction's content changes,
+   [docs.docker.com/build/cache](https://docs.docker.com/build/cache/)).
+   The apt package list changes essentially never; `GODOT_VERSION`/
+   `BLENDER_URL` change most nights. Splitting them means the ~470s apt
+   layer becomes a cache hit on every night that doesn't touch the
+   Dockerfile itself, while the Godot/Blender layer keeps rebuilding (it has
+   to — new content every time a version bumps). Combined with #1, this is
+   the actual fix for the dominant cost in the real run.
+3. **Add `RUN --mount=type=cache` for apt's own directories**
+   (`/var/cache/apt`, `/var/lib/apt`, `sharing=locked`) as a smaller
+   supplement to #2 — it persists apt's downloaded `.deb` cache independent
+   of the Dockerfile-instruction cache chain, so even a Dockerfile edit that
+   invalidates the layer doesn't force re-downloading every `.deb` from
+   Debian's mirrors, only re-running `dpkg` against the still-local cache
+   ([docs.docker.com/reference/dockerfile/#run---mounttypecache](https://docs.docker.com/reference/dockerfile/#run---mounttypecache)).
+   Worth doing alongside #2 since it's a few lines, but it's a hedge against
+   Dockerfile churn, not the main win — #2 already gets full cache hits on
+   the common case (no Dockerfile change, only build-args change).
+4. **Skip registry-based cache (`--cache-to/--cache-from type=registry`).**
+   Not worth it here. Registry cache exists to share a build cache *across
+   machines that don't share disk* — ephemeral GitHub-hosted runners, fleets
+   of self-hosted runners, etc.
+   ([docs.docker.com/build/cache/backends/registry](https://docs.docker.com/build/cache/backends/registry/)).
+   This repo has exactly one persistent runner with its own disk, so #1–#3
+   already give it that same cache locally, for free, with no push/pull
+   round-trip. Pushing a *separate* cache image on top would add push/pull
+   time on every run for a benefit local cache already provides, and it's a
+   bad fit for this specific image's shape: the Godot/Blender layers are
+   multi-GB and, by design, different nearly every night (new versions),
+   so `mode=max` would repeatedly re-push multi-GB "cache" blobs that are
+   very unlikely to ever be pulled back down. There's also an open question
+   of whether it would even work: Gitea's container registry only gained
+   general OCI-artifact support in 1.24
+   ([go-gitea/gitea#25846](https://github.com/go-gitea/gitea/issues/25846),
+   closed via PR #34666) — before that it rejected non-image manifests with
+   `"unsupported: Schema version is not supported"`. This repo's Gitea
+   instance version wasn't confirmed as part of this research, so pushing a
+   buildx cache manifest (which uses custom OCI media types, distinct from
+   a normal image manifest) is not guaranteed to work without checking that
+   first.
+5. **Gitea Actions' `actions/cache` action is the wrong tool for this,
+   noted for completeness.** Gitea Actions does support a GitHub-compatible
+   `actions/cache@v3`/`v4` via a cache server that ships enabled by default
+   in `act_runner`
+   ([about.gitea.com tutorial](https://about.gitea.com/resources/tutorials/enable-gitea-actions-cache-to-accelerate-cicd/);
+   [docs.gitea.com/usage/actions/overview](https://docs.gitea.com/usage/actions/overview/)),
+   plus a separate `RUNNER_TOOL_CACHE` mechanism. Both are key/value or
+   directory caches meant for things like `node_modules`/pip wheels between
+   workflow steps — not a Docker layer-cache mechanism, and not something
+   `docker build`/`docker push` (what this workflow actually calls, see
+   below) has any hook into. It doesn't apply to this problem; #1–#3 do.
+
+## 1. What the workflow and Dockerfile actually do today
+
+`.gitea/workflows/publish.yml` runs plain `docker build` and `docker push` —
+not `docker/build-push-action` or `docker buildx build` with any cache flags
+(lines 86–89, 96/105/109). `runs-on: ubuntu-latest` is a Gitea-side runner
+*label*, matched against whatever labels this instance's `act_runner` was
+registered with — Gitea's own runner docs describe labels as exactly this
+kind of local mapping (`ubuntu-latest:docker://...` style entries a runner
+operator configures, not a GitHub-hosted VM type)
+([search result summary of docs.gitea.com/usage/actions, act_runner label
+docs](https://docs.gitea.com/1.23/usage/actions/act-runner/); the specific
+`main_runner` name mentioned as a possible identifier for this instance's
+runner doesn't appear anywhere in this repo's tracked files — it would only
+show up in the Gitea Actions run UI, not the workflow YAML). The disk-fill
+problem the "Free up runner disk" step's own comment describes (*"nothing
+here ever pruned old ones, so the runner's disk fills up over successive
+runs"*) is only possible on a runner whose disk persists between runs —
+corroborated independently by act_runner's own architecture: its Docker
+executor keeps a long-lived host Docker daemon and reuses the image cache
+across job runs by default (act_runner documentation on the docker/dind
+executor flavors and idle-cleanup behavior — runner cleans up stale
+workspaces on an interval rather than starting from a fresh disk each job).
+
+The `Dockerfile`'s expensive work is one `RUN` chain (lines 19–62): a single
+`apt-get install` covering `build-essential`, `scons`, `mingw-w64`, and the
+X11/audio/udev dev headers, immediately followed — in the same layer, via
+`&&` — by the `curl`+`unzip` of Godot's editor and export templates and the
+`curl`+`tar` of the Blender tarball. Because it's one `RUN`, it's one cache
+entry: any change anywhere in it (including the Godot/Blender URLs, which
+change on essentially every nightly build since the workflow tracks
+newest-stable) invalidates the whole thing, apt install included.
+
+## 2. Docker/BuildKit's own caching mechanisms (docs.docker.com)
+
+- **Layer cache (default, local, free)**: BuildKit caches each instruction's
+  result keyed on that instruction's content plus everything before it in
+  the Dockerfile; a build on the same machine reuses it automatically with
+  no extra flags, provided the cache wasn't evicted
+  ([docs.docker.com/build/cache](https://docs.docker.com/build/cache/)).
+  This is what #1+#2 above unlock for this repo — it needs nothing but (a)
+  not deleting it and (b) ordering the Dockerfile so the part that doesn't
+  change (apt) comes before the part that does (downloads).
+- **`RUN --mount=type=cache`**: a *build cache mount*, separate from the
+  layer cache — a directory that survives across builder invocations
+  without itself being part of the cached layer, intended for package-
+  manager caches like apt's `/var/cache/apt`/`/var/lib/apt`. Needs
+  `sharing=locked` for apt specifically, since apt needs exclusive access to
+  its own state
+  ([docs.docker.com/reference/dockerfile/#run---mounttypecache](https://docs.docker.com/reference/dockerfile/#run---mounttypecache)).
+  Persists even when the layer cache above gets invalidated by a Dockerfile
+  edit — a smaller, complementary win, not a replacement for #2's layer
+  split.
+- **`--cache-to`/`--cache-from type=registry`**: exports/imports build
+  cache to/from an OCI registry, as a separate artifact from the final
+  image, specifically to let machines that don't share local disk share a
+  cache (`mode=max` caches every stage, not just the final one — bigger
+  push, more reuse potential)
+  ([docs.docker.com/build/cache/backends/registry](https://docs.docker.com/build/cache/backends/registry/)).
+  Not needed for a single persistent runner (see verdict §4).
+- **`BUILDKIT_INLINE_CACHE`/`type=inline`**: embeds cache metadata directly
+  in the pushed image manifest instead of a separate cache artifact —
+  Docker's own CI guidance is "in most cases you want to use the inline
+  cache exporter" for simple cases, but it only supports `mode=min` (final
+  stage only) versus registry cache's `min`/`max`
+  ([docs.docker.com/build/ci/github-actions/cache](https://docs.docker.com/build/ci/github-actions/cache/)).
+  Same applicability caveat as registry cache: solves a multi-machine
+  problem this repo doesn't have.
+- Docker Engine ≥23 makes BuildKit (via buildx) the default builder for
+  plain `docker build` — *"Set Buildx and BuildKit as the default builder on
+  Linux. Alias `docker build` to `docker buildx build`"*
+  ([docs.docker.com/engine/release-notes/23.0](https://docs.docker.com/engine/release-notes/23.0/)).
+  So the workflow's existing plain `docker build` call (no `buildx` in the
+  command) should already be getting BuildKit's layer cache today — it's
+  the prune step deleting it that matters, not the build command needing to
+  change to buildx.
+
+## 3. `docker system prune -af --volumes` vs `docker builder prune`
+
+Confirmed directly against Docker's CLI reference:
+
+- `docker system prune`'s default removal set is stopped containers, unused
+  networks, dangling images, and **unused build cache**; `-a`/`--all`
+  widens image removal to *all* unused images (not just dangling), and
+  `--volumes` additionally removes unattached anonymous volumes
+  ([docs.docker.com/reference/cli/docker/system/prune](https://docs.docker.com/reference/cli/docker/system/prune/)).
+  There's no flag on `system prune` to keep build cache while still pruning
+  everything else — build cache removal isn't optional once you call it.
+- `docker builder prune` is the narrower, cache-only equivalent: `--all` to
+  remove all unused cache (not just dangling), `--filter until=<duration>`
+  to only remove cache older than a given age, `--keep-storage=<size>` to
+  cap total cache size instead of clearing it outright
+  ([docs.docker.com/reference/cli/docker/builder/prune](https://docs.docker.com/reference/cli/docker/builder/prune/)).
+
+This is the concrete lever for recommendation #1: keep freeing the disk the
+workflow's own comment says filled up (old images, old anonymous volumes),
+but do it with `container prune`/`image prune -af`/`volume prune` instead of
+`system prune`, and bound the build cache separately with
+`docker builder prune -f --filter until=24h` (or a `--keep-storage` cap)
+rather than deleting all of it unconditionally before every build. A same-
+day rerun (e.g. `workflow_dispatch` shortly after the nightly cron) would
+then still get a cache hit on the apt layer; a multi-day-old apt cache entry
+still gets swept on the next age-filtered prune, so disk doesn't grow
+unbounded either.
+
+## 4. Layer-order restructuring compatibility
+
+Splitting `Dockerfile`'s one `RUN` into "apt install" then "Godot/Blender
+downloads" is a pure reorder/split — no semantic change, no new
+dependencies, and it's exactly what BuildKit's per-instruction cache keying
+is designed to reward (§2). Its payoff is entirely contingent on §3: with
+`docker system prune -af --volumes` still running first, splitting the
+layer changes nothing, because the reusable apt-layer cache entry is deleted
+moments before the build that would've hit it starts. The two
+recommendations are a pair, not independent options.
+
+## 5. Registry cache realism for this specific image
+
+Weighed directly against this repo's shape:
+
+- **Storage cost**: `mode=max` registry cache stores every stage as its own
+  cache layer set in the registry, separate from the pushed image
+  ([docs.docker.com/build/cache/backends/registry](https://docs.docker.com/build/cache/backends/registry/)) —
+  for an image whose expensive layers are multi-GB Godot export templates
+  and a Blender tarball that both change nearly nightly, that's another
+  near-full copy of those multi-GB layers pushed to Gitea's package registry
+  on top of the image push that already happens, for cache entries with a
+  short effective lifetime (superseded the next time Godot/Blender bump).
+- **Gitea registry support**: Gitea's container registry is OCI-compliant
+  for standard image/Helm-chart manifests
+  ([docs.gitea.com/usage/packages/container](https://docs.gitea.com/usage/packages/container/)),
+  but general OCI-*artifact* support (the category buildx's cache manifest
+  falls into, using non-image media types) only landed in Gitea 1.24 via
+  [go-gitea/gitea#25846](https://github.com/go-gitea/gitea/issues/25846) —
+  prior to that, pushes of non-standard-image manifests failed with
+  `501 unsupported: Schema version is not supported`. This repo's actual
+  Gitea server version wasn't checked as part of this research, so this is
+  a real "verify before use" gate, not just a style preference.
+- **Time saved**: none of it addresses the local-persistent-runner case
+  this repo actually has — §2/§4's plain layer-cache fix gets the same or
+  better result (apt-layer cache hits) for free, with no push/pull latency
+  and no registry storage growth.
+
+Net: registry cache isn't wrong in general, it's solving a problem (cache
+sharing across machines without shared disk) this repo doesn't have, at a
+storage and reliability cost this repo's layer shapes make worse than
+typical.
+
+## Sources
+
+- `Dockerfile` — this repo's build, current single `RUN` chain (apt install
+  + Godot download/unpack + Blender download/unpack), fetched at HEAD
+- `.gitea/workflows/publish.yml` — `runs-on: ubuntu-latest`, plain
+  `docker build`/`docker push` (no buildx/cache flags), the "Free up runner
+  disk" step and its comment explaining *why* it prunes before every build
+- https://docs.docker.com/build/cache/ — BuildKit layer cache mechanics,
+  invalidation model
+- https://docs.docker.com/reference/dockerfile/#run---mounttypecache —
+  `RUN --mount=type=cache`, apt cache-mount example, `sharing=locked`
+- https://docs.docker.com/build/cache/backends/registry/ — `--cache-to`/
+  `--cache-from type=registry`, `mode=min`/`max`
+- https://docs.docker.com/build/ci/github-actions/cache/ —
+  `BUILDKIT_INLINE_CACHE`/`type=inline` vs registry cache tradeoffs
+- https://docs.docker.com/engine/release-notes/23.0/ — BuildKit/buildx as
+  default `docker build` builder since Engine 23.0
+- https://docs.docker.com/reference/cli/docker/system/prune/ — `docker
+  system prune` default removal set (incl. build cache), `-a`/`--volumes`
+- https://docs.docker.com/reference/cli/docker/builder/prune/ — `docker
+  builder prune`, `--filter until=`, `--keep-storage`
+- https://docs.gitea.com/usage/actions/act-runner and
+  https://docs.gitea.com/1.23/usage/actions/act-runner/ — act_runner
+  labels, docker/dind executor persistence and idle cleanup behavior
+- https://about.gitea.com/resources/tutorials/enable-gitea-actions-cache-to-accelerate-cicd/
+  and https://docs.gitea.com/usage/actions/overview/ — Gitea Actions'
+  `actions/cache` support and default-enabled cache server
+- https://docs.gitea.com/usage/packages/container/ — Gitea's OCI-compliant
+  container registry
+- https://github.com/go-gitea/gitea/issues/25846 (closed via PR #34666) —
+  Gitea container registry's OCI-artifact (non-image-manifest) support
+  landing in 1.24, and the `501 unsupported: Schema version is not
+  supported` error prior to that
+- https://gitea.com/gitea/act_runner — act_runner project, docker executor