From adbc3cd6e78b03cc88811f9be9272f0d48319703 Mon Sep 17 00:00:00 2001 From: ArthurErlich Date: Fri, 7 Aug 2026 00:57:16 +0200 Subject: [PATCH] docs(research): act_runner Docker volume accumulation and cleanup Investigates why git.arthurerlich.de's act_runner/Docker CI host accumulates Docker volumes, grounded in act_runner's own config docs, Gitea's Actions docs, Docker's official prune/rootless docs, and a maintainer-confirmed analogous root cause from GitHub's actions/runner. Co-Authored-By: Claude Sonnet 5 --- docs/research/act-runner-volume-cleanup.md | 318 +++++++++++++++++++++ 1 file changed, 318 insertions(+) create mode 100644 docs/research/act-runner-volume-cleanup.md diff --git a/docs/research/act-runner-volume-cleanup.md b/docs/research/act-runner-volume-cleanup.md new file mode 100644 index 0000000..546ffb1 --- /dev/null +++ b/docs/research/act-runner-volume-cleanup.md @@ -0,0 +1,318 @@ +# Why act_runner's Docker volumes accumulate, and how to clean them up + +> **Scope note:** this is about the **host/runner infrastructure** for +> git.arthurerlich.de's Gitea Actions (`act_runner` + the Docker daemon it +> drives, in the docker-compose setup for that Gitea instance) — **not** +> about this repo's own `Dockerfile`/published image. This repo's +> `publish.yml` already runs `docker system prune -af --volumes` before its +> build for the same underlying reason described here (unbounded volume/image +> growth on the CI host), which is what prompted this research. No access to +> the actual Gitea docker-compose file was available or used; every claim +> below is grounded in act_runner's/Gitea's/Docker's own docs and source, or +> flagged as inferred/secondary where it isn't. + +## Recommendations (ranked) + +1. **Schedule `docker system prune -af --volumes` on the runner host on a + recurring basis (cron or a scheduled Gitea Action), the same pattern this + repo's own `publish.yml` already uses before its build.** — **Proven + mechanism, standard mitigation.** No built-in act_runner/Gitea equivalent + exists (see §2); this is the documented, generic Docker-level fix and is + what community guidance for this exact runner-volume-leak class of + problem converges on. Prefer a filtered form if the host also runs + same-day rebuilds you don't want swept up: + `docker system prune -af --volumes --filter "until=24h"` (or + `docker volume prune --filter "until=24h"` for a volumes-only, less + aggressive sweep) — see §2 for the exact filter semantics. +2. **Confirm/enable act_runner's `container.valid_volumes` allowlist and + review `container.options`/`container.network`, but do not expect them to + fix the leak** — **Proven from act_runner's own config reference (§1.2): + these control what containers are *allowed* to mount, not automatic + cleanup after a job ends.** Leaving `valid_volumes` wide open + (`- '**'`) doesn't itself cause more leakage than default, but tightening + it is good hygiene independent of the disk problem. +3. **If DinD (Docker-in-Docker) mode is in use, that is very likely where + the bulk of the leaked volumes come from** — **inferred, but from a + documented, closely-analogous mechanism (§1), not confirmed against this + host's actual config** since no docker-compose file was available. DinD + needs its own persistent Docker data directory (commonly volume-backed), + and any job container that declares a `VOLUME` in its Dockerfile gets an + anonymous volume that Docker does not remove on `docker rm` unless the + container was started with `--rm` — a mechanism GitHub's own + `actions/runner` maintainers confirmed by design for the conceptually + identical GitHub Actions Docker executor (§1.3). act_runner's job/step + containers are created, started, stopped, and removed as separate Docker + API calls (not a single `docker run --rm`), which is the same shape of + gap. +4. **A dedicated Linux user for act_runner (and rootless Docker specifically) + is a security/isolation improvement, not a fix for volume accumulation.** + — **Proven boundary, directly stated in act_runner's own docs (§3).** + Do it for blast-radius/`docker.sock`-exposure reasons; don't expect it to + change disk usage. Running the Docker daemon itself in **rootless mode** + changes *where* volumes live (per-user data root under + `$XDG_RUNTIME_DIR`/`~/.local/share/docker` instead of `/var/lib/docker`) + but Docker's own rootless docs describe no change to prune/cleanup + semantics — unused volumes still only go away via an explicit prune, + run as that user against that daemon. + +**Net: there is no act_runner/Gitea-side switch that stops the leak.** The +fix is operational — a scheduled prune — same shape as the one already +present in this repo's own `publish.yml`. + +--- + +## 1. Why this happens + +### 1.1 act_runner's job execution model + +Per Gitea's own Act Runner documentation, act_runner supports two execution +modes: **Docker container mode** ("it is recommended to run jobs in a +docker container", requiring a running Docker daemon) and **host mode** +(jobs run directly on the machine act_runner is on, no Docker involved). +Docker mode is the default/recommended mode and is what a normal +docker-compose Gitea+act_runner setup uses. +([docs.gitea.com/1.23/usage/actions/act-runner](https://docs.gitea.com/1.23/usage/actions/act-runner/)) + +In Docker mode, act_runner creates a container per job (and per +service-container / step container as declared in the workflow), runs it, +and removes it — as separate Docker API calls, not a single atomic +`docker run --rm ...`. This matters because Docker's own volume-removal +rule is tied to *how* a container is removed, not just that it's gone (see +§1.3). + +### 1.2 act_runner's `container.*` config settings + +From act_runner's own `config.yaml` reference (fetched from +`gitea.com/gitea/act_runner`'s example config, +[gitea.com/gitea/act_runner — config.example.yaml](https://gitea.com/gitea/act_runner/raw/branch/main/internal/pkg/config/config.example.yaml)): + +- **`container.docker_host`** — overrides the Docker daemon act_runner talks + to. Empty (default) auto-detects; `-` auto-detects without mounting the + socket into job containers. Controls *which* daemon runs jobs, not + cleanup. +- **`container.network`** — `host`, `bridge`, or a custom network name; + empty auto-creates a per-job network. Unrelated to volumes. +- **`container.privileged`** — "Privileged mode is required for + Docker-in-Docker." This is the flag that turns on DinD-style job + execution. +- **`container.options`** — free-form extra flags passed to container + creation (e.g. `--add-host=...`, or extra `--volume` mounts). Whatever a + workflow or admin puts here can itself add more bind mounts/volumes per + job, but is opt-in, not a default source of leakage. +- **`container.valid_volumes`** — an **allowlist** (glob syntax) of + volumes/bind mounts job containers are permitted to mount; empty means + none permitted, `'**'` means any. This is an authorization control, not a + lifecycle/cleanup control — it limits what *can* be mounted, and doesn't + cause or prevent leftover anonymous volumes from images that declare + `VOLUME` internally. +- **`container.bind_workdir`** — relevant specifically to DinD: "required + for Docker-in-Docker (DinD) setups when jobs use docker compose with bind + mounts (e.g. `.:/app`), as volume-based workspaces are not accessible + from the DinD daemon's filesystem." Confirms DinD's workspace handling is + a distinct code path from plain Docker mode. +- **`workdir_cleanup_age` / `idle_cleanup_interval`** — these exist, but per + the config reference they clean up **act_runner's own workspace + directories** (stale bind-mounted task-workdir folders on the host + filesystem, and orphaned host-mode scratch dirs) — **not Docker volumes**. + There is no equivalent `container`-level setting for pruning Docker + volumes. + +None of act_runner's documented settings amount to "clean up Docker volumes +after a job." The closest things (`workdir_cleanup_age`, +`idle_cleanup_interval`) are scoped to filesystem workdirs, a different +mechanism from Docker volume objects. + +### 1.3 Is DinD specifically the source of the leak, vs plain Docker mode? + +Two independent, additive causes are documented: + +1. **Any Docker execution mode (plain or DinD): anonymous volumes from + job/service containers leak by Docker's own removal semantics.** This is + not act_runner-specific — it's how Docker containers work. GitHub's own + `actions/runner` maintainers documented the identical failure mode for + GitHub Actions' (conceptually equivalent) Docker executor: *"When using + service containers with a `VOLUME` declaration in their Dockerfile, an + anonymous volume is automatically created with the container. At the end + of the workflow, the container is stopped and removed, but the anonymous + volumes it created stay around... The runner creates, starts, stops and + removes containers with separate commands, so passing `--rm` to `docker + create` has no effect on `docker remove`."* + ([actions/runner#1885](https://github.com/actions/runner/issues/1885) — + maintainer-confirmed root cause, not community speculation). act_runner's + job execution is architecturally the same shape (separate create/start/ + stop/remove calls against the Docker API), so the same leak mechanism + applies to it by design, absent extra config — this is the strongest + available evidence, though it is evidence from GitHub's runner, applied + by analogy to act_runner rather than a direct act_runner maintainer + statement (no act_runner-specific issue making this exact claim was + found in this pass). +2. **DinD adds its own persistent data directory on top.** A Docker-in-Docker + daemon needs its own `/var/lib/docker`-equivalent to store the images/ + layers/volumes *of the jobs it runs*, and community DinD setups + routinely back that directory with a Docker volume so it survives + restarts of the DinD container itself (e.g. "`/var/lib/docker` cannot be + on AUFS, so it needs to be made a volume" — a long-standing operational + note about DinD from general Docker-in-Docker tooling, not Gitea-specific, + flagged here as secondary/community evidence). This is additive to (1): + DinD setups accumulate both the generic per-job anonymous-volume leak + *and* whatever grows inside DinD's own backing store (images, build + cache, and volumes belonging to the containers it spawns), which is a + second, larger accumulation surface than plain Docker mode. + +**Conclusion:** the generic anonymous-volume leak happens in plain Docker +mode too (it's a Docker container-removal semantics issue, not a DinD-only +bug), but DinD mode is very likely the bigger contributor on a real host, +because it adds a whole second Docker data root that grows independently. +Without the actual docker-compose file, it isn't possible to confirm from +this pass alone whether git.arthurerlich.de's act_runner is configured for +DinD (`container.privileged: true` set) or plain Docker mode. + +--- + +## 2. How to clean up + +### 2.1 `docker volume prune` / `docker system prune` semantics (Docker's own docs) + +Per Docker's official pruning reference +([docs.docker.com/engine/manage-resources/pruning](https://docs.docker.com/engine/manage-resources/pruning/), +[docs.docker.com/reference/cli/docker/volume/prune](https://docs.docker.com/reference/cli/docker/volume/prune/), +[docs.docker.com/reference/cli/docker/system/prune](https://docs.docker.com/reference/cli/docker/system/prune/)): + +- **`docker system prune`** (no flags) removes stopped containers, networks + not used by at least one container, dangling images, and unused build + cache. **Volumes are excluded by default.** +- **`docker system prune --volumes`** (or `-a --volumes` to also sweep + non-dangling unused images) additionally removes "all volumes not used by + at least one container." +- **`docker volume prune`** removes unused local volumes; by default this + means **anonymous** volumes not referenced by any container. `--all` + extends this to unused **named** volumes too. +- **`docker volume prune --filter "until="`** narrows the sweep to + volumes older than the given age (still only unused ones) — a way to keep + volumes from a job that finished minutes ago while still cleaning up + older leftovers. + +**Safety on a host mid-job:** all of the above operate only on resources +**not referenced by any container, running or stopped**. A volume mounted +into a currently-running job container is never touched. This makes it safe +to run on a schedule against a live runner host without coordinating with +in-flight jobs — the documented behavior is conservative by construction, +not merely "usually fine in practice." + +### 2.2 Does act_runner or Gitea ship any built-in cleanup for this? + +No. Searched act_runner's config reference, Gitea's Actions/runner +documentation, and the two most relevant issue threads found +([go-gitea/gitea#31457](https://github.com/go-gitea/gitea/issues/31457), +about `act_runner` manageability generally, and +[gitea.com/gitea/runner#167](https://gitea.com/gitea/runner/issues/167) / +[go-gitea/gitea#24438](https://github.com/go-gitea/gitea/issues/24438), +about `docker.sock` exposure) — none describe a built-in Docker volume +cleanup mechanism, cron-style prune config, or maintainer statement +addressing this exact problem. The only cleanup knobs act_runner documents +(`workdir_cleanup_age`, `idle_cleanup_interval`) are scoped to its own +host-filesystem workdirs, not Docker volume objects (§1.2). This absence is +itself informative: there is nothing to "turn on" here — a scheduled +external prune is the only available lever. + +### 2.3 Is a scheduled `docker system prune -af --volumes` the standard approach? + +Yes, and it is corroborated by both this repo's own precedent and general +community guidance: + +- This repo's `publish.yml` already runs `docker system prune -af --volumes` + before its build, for the same underlying class of problem (disk + exhaustion from accumulated Docker state on a CI runner) — direct + precedent already in this codebase. +- General community guidance for self-hosted CI runners converges on the + same shape of fix: a scheduled/cron `docker system prune` (optionally + filtered by age) run on the runner host. This is consistent, secondary + evidence (blog posts / community write-ups, not a Docker or Gitea + standards document) but aligns with what Docker's own docs make safe + (§2.1). + +**A more targeted alternative** — `docker volume prune --filter "until=24h"` +run on a schedule (e.g. nightly cron) — trades some aggressiveness for +narrower scope: it only ever removes volumes, leaves stopped +containers/networks/images alone, and skips anything from the last 24h in +case a long-running or recently-finished job's volume is still wanted for +debugging. This is a reasonable middle ground if a full `system prune -af` +feels too broad for the runner host (e.g. if other non-CI Docker workloads +share the machine); `docker system prune -af --volumes` is simpler and is +what this repo already does elsewhere. + +--- + +## 3. Dedicated user for the runner: security concern, not a cleanup mechanism + +### 3.1 act_runner's own guidance + +Gitea's Act Runner docs (systemd install instructions) recommend creating a +dedicated unprivileged `act_runner` user, and note that adding it to the +`docker` group (needed for Docker mode) **"effectively gives act_runner +root access to the system"** — framed entirely as a privilege/blast-radius +concern. +([docs.gitea.com/1.23/usage/actions/act-runner](https://docs.gitea.com/1.23/usage/actions/act-runner/)) + +This is echoed more sharply in Gitea's own tracker: +[gitea.com/gitea/runner#167](https://gitea.com/gitea/runner/issues/167) / +mirrored as [go-gitea/gitea#24438](https://github.com/go-gitea/gitea/issues/24438), +titled *"Gitea Actions is HIGHLY insecure due to binding of docker.sock into +all containers (= root on host)"* — the concern raised (and act_runner's own +`examples/vm/rootless-docker.md` walkthrough, built specifically in +response to this class of concern) is **container escape / arbitrary root +on the host via `docker.sock` exposure to untrusted job code**, not disk +usage. + +### 3.2 Rootless Docker / user-namespace remapping (Docker's own docs) + +Per Docker's official rootless-mode docs +([docs.docker.com/engine/security/rootless](https://docs.docker.com/engine/security/rootless/)): +rootless mode "lets you run the Docker daemon and containers as a non-root +user to mitigate potential vulnerabilities in the daemon and the container +runtime" — again, a security-isolation feature. The docs describe no +special relationship to volume/disk accumulation or cleanup behavior. +act_runner's own `rootless-docker.md` walkthrough +([gitea.com/gitea/act_runner — examples/vm/rootless-docker.md](https://gitea.com/gitea/act_runner/raw/branch/main/examples/vm/rootless-docker.md)) +is, in its own words, exclusively about security/isolation setup (dedicated +user, rootless daemon, systemd units) and makes no mention of volume +cleanup or storage management at any point. + +Practically, rootless mode does shift *where* Docker's data root (and +therefore its volumes) lives — per-user, typically under +`$XDG_RUNTIME_DIR` / `~/.local/share/docker` rather than the system-wide +`/var/lib/docker` — but this only changes the location being filled up, not +whether it fills up. The same `docker volume prune` semantics (§2.1) still +apply, run as that user against that user's daemon. + +### 3.3 Conclusion + +- **(a) Dedicated Linux user for the act_runner service itself:** purely a + blast-radius/permissions control (who can touch what if the runner or a + job container is compromised). No documented bearing on volume + accumulation. +- **(b) Rootless Docker / userns-remap:** also purely a security-isolation + mechanism per Docker's own docs. It relocates where volumes are stored + (per-user vs system-wide) but does not change prune semantics or provide + any automatic cleanup — the volume-growth problem is orthogonal to this + and is not solved by adopting either (a) or (b). + +Both are worth doing for `docker.sock`-exposure/security reasons +independent of this investigation, but neither should be expected to move +the disk-usage needle — only a scheduled prune (§1 recommendation #1, §2.3) +addresses that. + +--- + +## Sources + +- act_runner config reference — [gitea.com/gitea/act_runner: config.example.yaml](https://gitea.com/gitea/act_runner/raw/branch/main/internal/pkg/config/config.example.yaml) +- Gitea Act Runner docs — [docs.gitea.com/1.23/usage/actions/act-runner](https://docs.gitea.com/1.23/usage/actions/act-runner/) +- act_runner rootless Docker walkthrough — [gitea.com/gitea/act_runner: examples/vm/rootless-docker.md](https://gitea.com/gitea/act_runner/raw/branch/main/examples/vm/rootless-docker.md) +- Gitea Actions `docker.sock` security issue — [gitea.com/gitea/runner#167](https://gitea.com/gitea/runner/issues/167) / mirrored [go-gitea/gitea#24438](https://github.com/go-gitea/gitea/issues/24438) +- act_runner manageability issue — [go-gitea/gitea#31457](https://github.com/go-gitea/gitea/issues/31457) (no maintainer disk/volume-cleanup discussion found in this thread) +- Anonymous volume leak, maintainer-confirmed root cause (GitHub's `actions/runner`, conceptually equivalent Docker executor) — [actions/runner#1885](https://github.com/actions/runner/issues/1885) +- Docker pruning reference — [docs.docker.com/engine/manage-resources/pruning](https://docs.docker.com/engine/manage-resources/pruning/) +- `docker volume prune` reference — [docs.docker.com/reference/cli/docker/volume/prune](https://docs.docker.com/reference/cli/docker/volume/prune/) +- `docker system prune` reference — [docs.docker.com/reference/cli/docker/system/prune](https://docs.docker.com/reference/cli/docker/system/prune/) +- Docker rootless mode — [docs.docker.com/engine/security/rootless](https://docs.docker.com/engine/security/rootless/) -- 2.54.0