2026-08-06 23:00:40 +00:00
1 changed files with 318 additions and 0 deletions
@@ -0,0 +1,318 @@
 # Why act_runner's Docker volumes accumulate, and how to clean them up
 > **Scope note:** this is about the **host/runner infrastructure** for
 > git.arthurerlich.de's Gitea Actions (`act_runner` + the Docker daemon it
 > drives, in the docker-compose setup for that Gitea instance) — **not**
 > about this repo's own `Dockerfile`/published image. This repo's
 > `publish.yml` already runs `docker system prune -af --volumes` before its
 > build for the same underlying reason described here (unbounded volume/image
 > growth on the CI host), which is what prompted this research. No access to
 > the actual Gitea docker-compose file was available or used; every claim
 > below is grounded in act_runner's/Gitea's/Docker's own docs and source, or
 > flagged as inferred/secondary where it isn't.
 ## Recommendations (ranked)
 1. **Schedule `docker system prune -af --volumes` on the runner host on a
   recurring basis (cron or a scheduled Gitea Action), the same pattern this
   repo's own `publish.yml` already uses before its build.** — **Proven
   mechanism, standard mitigation.** No built-in act_runner/Gitea equivalent
   exists (see §2); this is the documented, generic Docker-level fix and is
   what community guidance for this exact runner-volume-leak class of
   problem converges on. Prefer a filtered form if the host also runs
   same-day rebuilds you don't want swept up:
   `docker system prune -af --volumes --filter "until=24h"` (or
   `docker volume prune --filter "until=24h"` for a volumes-only, less
   aggressive sweep) — see §2 for the exact filter semantics.
 2. **Confirm/enable act_runner's `container.valid_volumes` allowlist and
   review `container.options`/`container.network`, but do not expect them to
   fix the leak** — **Proven from act_runner's own config reference (§1.2):
   these control what containers are *allowed* to mount, not automatic
   cleanup after a job ends.** Leaving `valid_volumes` wide open
   (`- '**'`) doesn't itself cause more leakage than default, but tightening
   it is good hygiene independent of the disk problem.
 3. **If DinD (Docker-in-Docker) mode is in use, that is very likely where
   the bulk of the leaked volumes come from** — **inferred, but from a
   documented, closely-analogous mechanism (§1), not confirmed against this
   host's actual config** since no docker-compose file was available. DinD
   needs its own persistent Docker data directory (commonly volume-backed),
   and any job container that declares a `VOLUME` in its Dockerfile gets an
   anonymous volume that Docker does not remove on `docker rm` unless the
   container was started with `--rm` — a mechanism GitHub's own
   `actions/runner` maintainers confirmed by design for the conceptually
   identical GitHub Actions Docker executor (§1.3). act_runner's job/step
   containers are created, started, stopped, and removed as separate Docker
   API calls (not a single `docker run --rm`), which is the same shape of
   gap.
 4. **A dedicated Linux user for act_runner (and rootless Docker specifically)
   is a security/isolation improvement, not a fix for volume accumulation.**
   — **Proven boundary, directly stated in act_runner's own docs (§3).**
   Do it for blast-radius/`docker.sock`-exposure reasons; don't expect it to
   change disk usage. Running the Docker daemon itself in **rootless mode**
   changes *where* volumes live (per-user data root under
   `$XDG_RUNTIME_DIR`/`~/.local/share/docker` instead of `/var/lib/docker`)
   but Docker's own rootless docs describe no change to prune/cleanup
   semantics — unused volumes still only go away via an explicit prune,
   run as that user against that daemon.
 **Net: there is no act_runner/Gitea-side switch that stops the leak.** The
 fix is operational — a scheduled prune — same shape as the one already
 present in this repo's own `publish.yml`.
 ---
 ## 1. Why this happens
 ### 1.1 act_runner's job execution model
 Per Gitea's own Act Runner documentation, act_runner supports two execution
 modes: **Docker container mode** ("it is recommended to run jobs in a
 docker container", requiring a running Docker daemon) and **host mode**
 (jobs run directly on the machine act_runner is on, no Docker involved).
 Docker mode is the default/recommended mode and is what a normal
 docker-compose Gitea+act_runner setup uses.
 ([docs.gitea.com/1.23/usage/actions/act-runner](https://docs.gitea.com/1.23/usage/actions/act-runner/))
 In Docker mode, act_runner creates a container per job (and per
 service-container / step container as declared in the workflow), runs it,
 and removes it — as separate Docker API calls, not a single atomic
 `docker run --rm ...`. This matters because Docker's own volume-removal
 rule is tied to *how* a container is removed, not just that it's gone (see
 §1.3).
 ### 1.2 act_runner's `container.*` config settings
 From act_runner's own `config.yaml` reference (fetched from
 `gitea.com/gitea/act_runner`'s example config,
 [gitea.com/gitea/act_runner — config.example.yaml](https://gitea.com/gitea/act_runner/raw/branch/main/internal/pkg/config/config.example.yaml)):
 - **`container.docker_host`** — overrides the Docker daemon act_runner talks
  to. Empty (default) auto-detects; `-` auto-detects without mounting the
  socket into job containers. Controls *which* daemon runs jobs, not
  cleanup.
 - **`container.network`** — `host`, `bridge`, or a custom network name;
  empty auto-creates a per-job network. Unrelated to volumes.
 - **`container.privileged`** — "Privileged mode is required for
  Docker-in-Docker." This is the flag that turns on DinD-style job
  execution.
 - **`container.options`** — free-form extra flags passed to container
  creation (e.g. `--add-host=...`, or extra `--volume` mounts). Whatever a
  workflow or admin puts here can itself add more bind mounts/volumes per
  job, but is opt-in, not a default source of leakage.
 - **`container.valid_volumes`** — an **allowlist** (glob syntax) of
  volumes/bind mounts job containers are permitted to mount; empty means
  none permitted, `'**'` means any. This is an authorization control, not a
  lifecycle/cleanup control — it limits what *can* be mounted, and doesn't
  cause or prevent leftover anonymous volumes from images that declare
  `VOLUME` internally.
 - **`container.bind_workdir`** — relevant specifically to DinD: "required
  for Docker-in-Docker (DinD) setups when jobs use docker compose with bind
  mounts (e.g. `.:/app`), as volume-based workspaces are not accessible
  from the DinD daemon's filesystem." Confirms DinD's workspace handling is
  a distinct code path from plain Docker mode.
 - **`workdir_cleanup_age` / `idle_cleanup_interval`** — these exist, but per
  the config reference they clean up **act_runner's own workspace
  directories** (stale bind-mounted task-workdir folders on the host
  filesystem, and orphaned host-mode scratch dirs) — **not Docker volumes**.
  There is no equivalent `container`-level setting for pruning Docker
  volumes.
 None of act_runner's documented settings amount to "clean up Docker volumes
 after a job." The closest things (`workdir_cleanup_age`,
 `idle_cleanup_interval`) are scoped to filesystem workdirs, a different
 mechanism from Docker volume objects.
 ### 1.3 Is DinD specifically the source of the leak, vs plain Docker mode?
 Two independent, additive causes are documented:
 1. **Any Docker execution mode (plain or DinD): anonymous volumes from
   job/service containers leak by Docker's own removal semantics.** This is
   not act_runner-specific — it's how Docker containers work. GitHub's own
   `actions/runner` maintainers documented the identical failure mode for
   GitHub Actions' (conceptually equivalent) Docker executor: *"When using
   service containers with a `VOLUME` declaration in their Dockerfile, an
   anonymous volume is automatically created with the container. At the end
   of the workflow, the container is stopped and removed, but the anonymous
   volumes it created stay around... The runner creates, starts, stops and
   removes containers with separate commands, so passing `--rm` to `docker
   create` has no effect on `docker remove`."*
   ([actions/runner#1885](https://github.com/actions/runner/issues/1885) —
   maintainer-confirmed root cause, not community speculation). act_runner's
   job execution is architecturally the same shape (separate create/start/
   stop/remove calls against the Docker API), so the same leak mechanism
   applies to it by design, absent extra config — this is the strongest
   available evidence, though it is evidence from GitHub's runner, applied
   by analogy to act_runner rather than a direct act_runner maintainer
   statement (no act_runner-specific issue making this exact claim was
   found in this pass).
 2. **DinD adds its own persistent data directory on top.** A Docker-in-Docker
   daemon needs its own `/var/lib/docker`-equivalent to store the images/
   layers/volumes *of the jobs it runs*, and community DinD setups
   routinely back that directory with a Docker volume so it survives
   restarts of the DinD container itself (e.g. "`/var/lib/docker` cannot be
   on AUFS, so it needs to be made a volume" — a long-standing operational
   note about DinD from general Docker-in-Docker tooling, not Gitea-specific,
   flagged here as secondary/community evidence). This is additive to (1):
   DinD setups accumulate both the generic per-job anonymous-volume leak
   *and* whatever grows inside DinD's own backing store (images, build
   cache, and volumes belonging to the containers it spawns), which is a
   second, larger accumulation surface than plain Docker mode.
 **Conclusion:** the generic anonymous-volume leak happens in plain Docker
 mode too (it's a Docker container-removal semantics issue, not a DinD-only
 bug), but DinD mode is very likely the bigger contributor on a real host,
 because it adds a whole second Docker data root that grows independently.
 Without the actual docker-compose file, it isn't possible to confirm from
 this pass alone whether git.arthurerlich.de's act_runner is configured for
 DinD (`container.privileged: true` set) or plain Docker mode.
 ---
 ## 2. How to clean up
 ### 2.1 `docker volume prune` / `docker system prune` semantics (Docker's own docs)
 Per Docker's official pruning reference
 ([docs.docker.com/engine/manage-resources/pruning](https://docs.docker.com/engine/manage-resources/pruning/),
 [docs.docker.com/reference/cli/docker/volume/prune](https://docs.docker.com/reference/cli/docker/volume/prune/),
 [docs.docker.com/reference/cli/docker/system/prune](https://docs.docker.com/reference/cli/docker/system/prune/)):
 - **`docker system prune`** (no flags) removes stopped containers, networks
  not used by at least one container, dangling images, and unused build
  cache. **Volumes are excluded by default.**
 - **`docker system prune --volumes`** (or `-a --volumes` to also sweep
  non-dangling unused images) additionally removes "all volumes not used by
  at least one container."
 - **`docker volume prune`** removes unused local volumes; by default this
  means **anonymous** volumes not referenced by any container. `--all`
  extends this to unused **named** volumes too.
 - **`docker volume prune --filter "until=<duration>"`** narrows the sweep to
  volumes older than the given age (still only unused ones) — a way to keep
  volumes from a job that finished minutes ago while still cleaning up
  older leftovers.
 **Safety on a host mid-job:** all of the above operate only on resources
 **not referenced by any container, running or stopped**. A volume mounted
 into a currently-running job container is never touched. This makes it safe
 to run on a schedule against a live runner host without coordinating with
 in-flight jobs — the documented behavior is conservative by construction,
 not merely "usually fine in practice."
 ### 2.2 Does act_runner or Gitea ship any built-in cleanup for this?
 No. Searched act_runner's config reference, Gitea's Actions/runner
 documentation, and the two most relevant issue threads found
 ([go-gitea/gitea#31457](https://github.com/go-gitea/gitea/issues/31457),
 about `act_runner` manageability generally, and
 [gitea.com/gitea/runner#167](https://gitea.com/gitea/runner/issues/167) /
 [go-gitea/gitea#24438](https://github.com/go-gitea/gitea/issues/24438),
 about `docker.sock` exposure) — none describe a built-in Docker volume
 cleanup mechanism, cron-style prune config, or maintainer statement
 addressing this exact problem. The only cleanup knobs act_runner documents
 (`workdir_cleanup_age`, `idle_cleanup_interval`) are scoped to its own
 host-filesystem workdirs, not Docker volume objects (§1.2). This absence is
 itself informative: there is nothing to "turn on" here — a scheduled
 external prune is the only available lever.
 ### 2.3 Is a scheduled `docker system prune -af --volumes` the standard approach?
 Yes, and it is corroborated by both this repo's own precedent and general
 community guidance:
 - This repo's `publish.yml` already runs `docker system prune -af --volumes`
  before its build, for the same underlying class of problem (disk
  exhaustion from accumulated Docker state on a CI runner) — direct
  precedent already in this codebase.
 - General community guidance for self-hosted CI runners converges on the
  same shape of fix: a scheduled/cron `docker system prune` (optionally
  filtered by age) run on the runner host. This is consistent, secondary
  evidence (blog posts / community write-ups, not a Docker or Gitea
  standards document) but aligns with what Docker's own docs make safe
  (§2.1).
 **A more targeted alternative** — `docker volume prune --filter "until=24h"`
 run on a schedule (e.g. nightly cron) — trades some aggressiveness for
 narrower scope: it only ever removes volumes, leaves stopped
 containers/networks/images alone, and skips anything from the last 24h in
 case a long-running or recently-finished job's volume is still wanted for
 debugging. This is a reasonable middle ground if a full `system prune -af`
 feels too broad for the runner host (e.g. if other non-CI Docker workloads
 share the machine); `docker system prune -af --volumes` is simpler and is
 what this repo already does elsewhere.
 ---
 ## 3. Dedicated user for the runner: security concern, not a cleanup mechanism
 ### 3.1 act_runner's own guidance
 Gitea's Act Runner docs (systemd install instructions) recommend creating a
 dedicated unprivileged `act_runner` user, and note that adding it to the
 `docker` group (needed for Docker mode) **"effectively gives act_runner
 root access to the system"** — framed entirely as a privilege/blast-radius
 concern.
 ([docs.gitea.com/1.23/usage/actions/act-runner](https://docs.gitea.com/1.23/usage/actions/act-runner/))
 This is echoed more sharply in Gitea's own tracker:
 [gitea.com/gitea/runner#167](https://gitea.com/gitea/runner/issues/167) /
 mirrored as [go-gitea/gitea#24438](https://github.com/go-gitea/gitea/issues/24438),
 titled *"Gitea Actions is HIGHLY insecure due to binding of docker.sock into
 all containers (= root on host)"* — the concern raised (and act_runner's own
 `examples/vm/rootless-docker.md` walkthrough, built specifically in
 response to this class of concern) is **container escape / arbitrary root
 on the host via `docker.sock` exposure to untrusted job code**, not disk
 usage.
 ### 3.2 Rootless Docker / user-namespace remapping (Docker's own docs)
 Per Docker's official rootless-mode docs
 ([docs.docker.com/engine/security/rootless](https://docs.docker.com/engine/security/rootless/)):
 rootless mode "lets you run the Docker daemon and containers as a non-root
 user to mitigate potential vulnerabilities in the daemon and the container
 runtime" — again, a security-isolation feature. The docs describe no
 special relationship to volume/disk accumulation or cleanup behavior.
 act_runner's own `rootless-docker.md` walkthrough
 ([gitea.com/gitea/act_runner — examples/vm/rootless-docker.md](https://gitea.com/gitea/act_runner/raw/branch/main/examples/vm/rootless-docker.md))
 is, in its own words, exclusively about security/isolation setup (dedicated
 user, rootless daemon, systemd units) and makes no mention of volume
 cleanup or storage management at any point.
 Practically, rootless mode does shift *where* Docker's data root (and
 therefore its volumes) lives — per-user, typically under
 `$XDG_RUNTIME_DIR` / `~/.local/share/docker` rather than the system-wide
 `/var/lib/docker` — but this only changes the location being filled up, not
 whether it fills up. The same `docker volume prune` semantics (§2.1) still
 apply, run as that user against that user's daemon.
 ### 3.3 Conclusion
 - **(a) Dedicated Linux user for the act_runner service itself:** purely a
  blast-radius/permissions control (who can touch what if the runner or a
  job container is compromised). No documented bearing on volume
  accumulation.
 - **(b) Rootless Docker / userns-remap:** also purely a security-isolation
  mechanism per Docker's own docs. It relocates where volumes are stored
  (per-user vs system-wide) but does not change prune semantics or provide
  any automatic cleanup — the volume-growth problem is orthogonal to this
  and is not solved by adopting either (a) or (b).
 Both are worth doing for `docker.sock`-exposure/security reasons
 independent of this investigation, but neither should be expected to move
 the disk-usage needle — only a scheduled prune (§1 recommendation #1, §2.3)
 addresses that.
 ---
 ## Sources
 - act_runner config reference — [gitea.com/gitea/act_runner: config.example.yaml](https://gitea.com/gitea/act_runner/raw/branch/main/internal/pkg/config/config.example.yaml)
 - Gitea Act Runner docs — [docs.gitea.com/1.23/usage/actions/act-runner](https://docs.gitea.com/1.23/usage/actions/act-runner/)
 - act_runner rootless Docker walkthrough — [gitea.com/gitea/act_runner: examples/vm/rootless-docker.md](https://gitea.com/gitea/act_runner/raw/branch/main/examples/vm/rootless-docker.md)
 - Gitea Actions `docker.sock` security issue — [gitea.com/gitea/runner#167](https://gitea.com/gitea/runner/issues/167) / mirrored [go-gitea/gitea#24438](https://github.com/go-gitea/gitea/issues/24438)
 - act_runner manageability issue — [go-gitea/gitea#31457](https://github.com/go-gitea/gitea/issues/31457) (no maintainer disk/volume-cleanup discussion found in this thread)
 - Anonymous volume leak, maintainer-confirmed root cause (GitHub's `actions/runner`, conceptually equivalent Docker executor) — [actions/runner#1885](https://github.com/actions/runner/issues/1885)
 - Docker pruning reference — [docs.docker.com/engine/manage-resources/pruning](https://docs.docker.com/engine/manage-resources/pruning/)
 - `docker volume prune` reference — [docs.docker.com/reference/cli/docker/volume/prune](https://docs.docker.com/reference/cli/docker/volume/prune/)
 - `docker system prune` reference — [docs.docker.com/reference/cli/docker/system/prune](https://docs.docker.com/reference/cli/docker/system/prune/)
 - Docker rootless mode — [docs.docker.com/engine/security/rootless](https://docs.docker.com/engine/security/rootless/)