godot-ci/docs/research/act-runner-volume-cleanup.md

Why act_runner's Docker volumes accumulate, and how to clean them up

Scope note: this is about the host/runner infrastructure for git.arthurerlich.de's Gitea Actions (act_runner + the Docker daemon it drives, in the docker-compose setup for that Gitea instance) — not about this repo's own Dockerfile/published image. This repo's publish.yml already runs docker system prune -af --volumes before its build for the same underlying reason described here (unbounded volume/image growth on the CI host), which is what prompted this research. No access to the actual Gitea docker-compose file was available or used; every claim below is grounded in act_runner's/Gitea's/Docker's own docs and source, or flagged as inferred/secondary where it isn't.

Recommendations (ranked)

1. Schedule docker system prune -af --volumes on the runner host on a recurring basis (cron or a scheduled Gitea Action), the same pattern this repo's own publish.yml already uses before its build. — Proven mechanism, standard mitigation. No built-in act_runner/Gitea equivalent exists (see §2); this is the documented, generic Docker-level fix and is what community guidance for this exact runner-volume-leak class of problem converges on. Prefer a filtered form if the host also runs same-day rebuilds you don't want swept up: docker system prune -af --volumes --filter "until=24h" (or docker volume prune --filter "until=24h" for a volumes-only, less aggressive sweep) — see §2 for the exact filter semantics.
2. Confirm/enable act_runner's container.valid_volumes allowlist and review container.options/container.network, but do not expect them to fix the leak — Proven from act_runner's own config reference (§1.2): these control what containers are allowed to mount, not automatic cleanup after a job ends. Leaving valid_volumes wide open (- '**') doesn't itself cause more leakage than default, but tightening it is good hygiene independent of the disk problem.
3. If DinD (Docker-in-Docker) mode is in use, that is very likely where the bulk of the leaked volumes come from — inferred, but from a documented, closely-analogous mechanism (§1), not confirmed against this host's actual config since no docker-compose file was available. DinD needs its own persistent Docker data directory (commonly volume-backed), and any job container that declares a VOLUME in its Dockerfile gets an anonymous volume that Docker does not remove on docker rm unless the container was started with --rm — a mechanism GitHub's own actions/runner maintainers confirmed by design for the conceptually identical GitHub Actions Docker executor (§1.3). act_runner's job/step containers are created, started, stopped, and removed as separate Docker API calls (not a single docker run --rm), which is the same shape of gap.
4. A dedicated Linux user for act_runner (and rootless Docker specifically) is a security/isolation improvement, not a fix for volume accumulation. — Proven boundary, directly stated in act_runner's own docs (§3). Do it for blast-radius/docker.sock-exposure reasons; don't expect it to change disk usage. Running the Docker daemon itself in rootless mode changes where volumes live (per-user data root under $XDG_RUNTIME_DIR/~/.local/share/docker instead of /var/lib/docker) but Docker's own rootless docs describe no change to prune/cleanup semantics — unused volumes still only go away via an explicit prune, run as that user against that daemon.

Net: there is no act_runner/Gitea-side switch that stops the leak. The fix is operational — a scheduled prune — same shape as the one already present in this repo's own publish.yml.

---

1. Why this happens

1.1 act_runner's job execution model

Per Gitea's own Act Runner documentation, act_runner supports two execution modes: Docker container mode ("it is recommended to run jobs in a docker container", requiring a running Docker daemon) and host mode (jobs run directly on the machine act_runner is on, no Docker involved). Docker mode is the default/recommended mode and is what a normal docker-compose Gitea+act_runner setup uses. (docs.gitea.com/1.23/usage/actions/act-runner)

In Docker mode, act_runner creates a container per job (and per service-container / step container as declared in the workflow), runs it, and removes it — as separate Docker API calls, not a single atomic docker run --rm .... This matters because Docker's own volume-removal rule is tied to how a container is removed, not just that it's gone (see §1.3).

1.2 act_runner's container.* config settings

From act_runner's own config.yaml reference (fetched from gitea.com/gitea/act_runner's example config, gitea.com/gitea/act_runner — config.example.yaml):

• container.docker_host — overrides the Docker daemon act_runner talks to. Empty (default) auto-detects; - auto-detects without mounting the socket into job containers. Controls which daemon runs jobs, not cleanup.
• container.network — host, bridge, or a custom network name; empty auto-creates a per-job network. Unrelated to volumes.
• container.privileged — "Privileged mode is required for Docker-in-Docker." This is the flag that turns on DinD-style job execution.
• container.options — free-form extra flags passed to container creation (e.g. --add-host=..., or extra --volume mounts). Whatever a workflow or admin puts here can itself add more bind mounts/volumes per job, but is opt-in, not a default source of leakage.
• container.valid_volumes — an allowlist (glob syntax) of volumes/bind mounts job containers are permitted to mount; empty means none permitted, '**' means any. This is an authorization control, not a lifecycle/cleanup control — it limits what can be mounted, and doesn't cause or prevent leftover anonymous volumes from images that declare VOLUME internally.
• container.bind_workdir — relevant specifically to DinD: "required for Docker-in-Docker (DinD) setups when jobs use docker compose with bind mounts (e.g. .:/app), as volume-based workspaces are not accessible from the DinD daemon's filesystem." Confirms DinD's workspace handling is a distinct code path from plain Docker mode.
• workdir_cleanup_age / idle_cleanup_interval — these exist, but per the config reference they clean up act_runner's own workspace directories (stale bind-mounted task-workdir folders on the host filesystem, and orphaned host-mode scratch dirs) — not Docker volumes. There is no equivalent container-level setting for pruning Docker volumes.

None of act_runner's documented settings amount to "clean up Docker volumes after a job." The closest things (workdir_cleanup_age, idle_cleanup_interval) are scoped to filesystem workdirs, a different mechanism from Docker volume objects.

1.3 Is DinD specifically the source of the leak, vs plain Docker mode?

Two independent, additive causes are documented:

1. Any Docker execution mode (plain or DinD): anonymous volumes from job/service containers leak by Docker's own removal semantics. This is not act_runner-specific — it's how Docker containers work. GitHub's own actions/runner maintainers documented the identical failure mode for GitHub Actions' (conceptually equivalent) Docker executor: "When using service containers with a VOLUME declaration in their Dockerfile, an anonymous volume is automatically created with the container. At the end of the workflow, the container is stopped and removed, but the anonymous volumes it created stay around... The runner creates, starts, stops and removes containers with separate commands, so passing --rm to docker create has no effect on docker remove." (actions/runner#1885 — maintainer-confirmed root cause, not community speculation). act_runner's job execution is architecturally the same shape (separate create/start/ stop/remove calls against the Docker API), so the same leak mechanism applies to it by design, absent extra config — this is the strongest available evidence, though it is evidence from GitHub's runner, applied by analogy to act_runner rather than a direct act_runner maintainer statement (no act_runner-specific issue making this exact claim was found in this pass).
2. DinD adds its own persistent data directory on top. A Docker-in-Docker daemon needs its own /var/lib/docker-equivalent to store the images/ layers/volumes of the jobs it runs, and community DinD setups routinely back that directory with a Docker volume so it survives restarts of the DinD container itself (e.g. "/var/lib/docker cannot be on AUFS, so it needs to be made a volume" — a long-standing operational note about DinD from general Docker-in-Docker tooling, not Gitea-specific, flagged here as secondary/community evidence). This is additive to (1): DinD setups accumulate both the generic per-job anonymous-volume leak and whatever grows inside DinD's own backing store (images, build cache, and volumes belonging to the containers it spawns), which is a second, larger accumulation surface than plain Docker mode.

Conclusion: the generic anonymous-volume leak happens in plain Docker mode too (it's a Docker container-removal semantics issue, not a DinD-only bug), but DinD mode is very likely the bigger contributor on a real host, because it adds a whole second Docker data root that grows independently. Without the actual docker-compose file, it isn't possible to confirm from this pass alone whether git.arthurerlich.de's act_runner is configured for DinD (container.privileged: true set) or plain Docker mode.

---

2. How to clean up

2.1 docker volume prune / docker system prune semantics (Docker's own docs)

Per Docker's official pruning reference (docs.docker.com/engine/manage-resources/pruning, docs.docker.com/reference/cli/docker/volume/prune, docs.docker.com/reference/cli/docker/system/prune):

• docker system prune (no flags) removes stopped containers, networks not used by at least one container, dangling images, and unused build cache. Volumes are excluded by default.
• docker system prune --volumes (or -a --volumes to also sweep non-dangling unused images) additionally removes "all volumes not used by at least one container."
• docker volume prune removes unused local volumes; by default this means anonymous volumes not referenced by any container. --all extends this to unused named volumes too.
• docker volume prune --filter "until=<duration>" narrows the sweep to volumes older than the given age (still only unused ones) — a way to keep volumes from a job that finished minutes ago while still cleaning up older leftovers.

Safety on a host mid-job: all of the above operate only on resources not referenced by any container, running or stopped. A volume mounted into a currently-running job container is never touched. This makes it safe to run on a schedule against a live runner host without coordinating with in-flight jobs — the documented behavior is conservative by construction, not merely "usually fine in practice."

2.2 Does act_runner or Gitea ship any built-in cleanup for this?

No. Searched act_runner's config reference, Gitea's Actions/runner documentation, and the two most relevant issue threads found (go-gitea/gitea#31457, about act_runner manageability generally, and gitea.com/gitea/runner#167 / go-gitea/gitea#24438, about docker.sock exposure) — none describe a built-in Docker volume cleanup mechanism, cron-style prune config, or maintainer statement addressing this exact problem. The only cleanup knobs act_runner documents (workdir_cleanup_age, idle_cleanup_interval) are scoped to its own host-filesystem workdirs, not Docker volume objects (§1.2). This absence is itself informative: there is nothing to "turn on" here — a scheduled external prune is the only available lever.

2.3 Is a scheduled docker system prune -af --volumes the standard approach?

Yes, and it is corroborated by both this repo's own precedent and general community guidance:

• This repo's publish.yml already runs docker system prune -af --volumes before its build, for the same underlying class of problem (disk exhaustion from accumulated Docker state on a CI runner) — direct precedent already in this codebase.
• General community guidance for self-hosted CI runners converges on the same shape of fix: a scheduled/cron docker system prune (optionally filtered by age) run on the runner host. This is consistent, secondary evidence (blog posts / community write-ups, not a Docker or Gitea standards document) but aligns with what Docker's own docs make safe (§2.1).

A more targeted alternative — docker volume prune --filter "until=24h" run on a schedule (e.g. nightly cron) — trades some aggressiveness for narrower scope: it only ever removes volumes, leaves stopped containers/networks/images alone, and skips anything from the last 24h in case a long-running or recently-finished job's volume is still wanted for debugging. This is a reasonable middle ground if a full system prune -af feels too broad for the runner host (e.g. if other non-CI Docker workloads share the machine); docker system prune -af --volumes is simpler and is what this repo already does elsewhere.

---

3. Dedicated user for the runner: security concern, not a cleanup mechanism

3.1 act_runner's own guidance

Gitea's Act Runner docs (systemd install instructions) recommend creating a dedicated unprivileged act_runner user, and note that adding it to the docker group (needed for Docker mode) "effectively gives act_runner root access to the system" — framed entirely as a privilege/blast-radius concern. (docs.gitea.com/1.23/usage/actions/act-runner)

This is echoed more sharply in Gitea's own tracker: gitea.com/gitea/runner#167 / mirrored as go-gitea/gitea#24438, titled "Gitea Actions is HIGHLY insecure due to binding of docker.sock into all containers (= root on host)" — the concern raised (and act_runner's own examples/vm/rootless-docker.md walkthrough, built specifically in response to this class of concern) is container escape / arbitrary root on the host via docker.sock exposure to untrusted job code, not disk usage.

3.2 Rootless Docker / user-namespace remapping (Docker's own docs)

Per Docker's official rootless-mode docs (docs.docker.com/engine/security/rootless): rootless mode "lets you run the Docker daemon and containers as a non-root user to mitigate potential vulnerabilities in the daemon and the container runtime" — again, a security-isolation feature. The docs describe no special relationship to volume/disk accumulation or cleanup behavior. act_runner's own rootless-docker.md walkthrough (gitea.com/gitea/act_runner — examples/vm/rootless-docker.md) is, in its own words, exclusively about security/isolation setup (dedicated user, rootless daemon, systemd units) and makes no mention of volume cleanup or storage management at any point.

Practically, rootless mode does shift where Docker's data root (and therefore its volumes) lives — per-user, typically under $XDG_RUNTIME_DIR / ~/.local/share/docker rather than the system-wide /var/lib/docker — but this only changes the location being filled up, not whether it fills up. The same docker volume prune semantics (§2.1) still apply, run as that user against that user's daemon.

3.3 Conclusion

• (a) Dedicated Linux user for the act_runner service itself: purely a blast-radius/permissions control (who can touch what if the runner or a job container is compromised). No documented bearing on volume accumulation.
• (b) Rootless Docker / userns-remap: also purely a security-isolation mechanism per Docker's own docs. It relocates where volumes are stored (per-user vs system-wide) but does not change prune semantics or provide any automatic cleanup — the volume-growth problem is orthogonal to this and is not solved by adopting either (a) or (b).

Both are worth doing for docker.sock-exposure/security reasons independent of this investigation, but neither should be expected to move the disk-usage needle — only a scheduled prune (§1 recommendation #1, §2.3) addresses that.

---

Sources

• act_runner config reference — gitea.com/gitea/act_runner: config.example.yaml
• Gitea Act Runner docs — docs.gitea.com/1.23/usage/actions/act-runner
• act_runner rootless Docker walkthrough — gitea.com/gitea/act_runner: examples/vm/rootless-docker.md
• Gitea Actions docker.sock security issue — gitea.com/gitea/runner#167 / mirrored go-gitea/gitea#24438
• act_runner manageability issue — go-gitea/gitea#31457 (no maintainer disk/volume-cleanup discussion found in this thread)
• Anonymous volume leak, maintainer-confirmed root cause (GitHub's actions/runner, conceptually equivalent Docker executor) — actions/runner#1885
• Docker pruning reference — docs.docker.com/engine/manage-resources/pruning
• docker volume prune reference — docs.docker.com/reference/cli/docker/volume/prune
• docker system prune reference — docs.docker.com/reference/cli/docker/system/prune
• Docker rootless mode — docs.docker.com/engine/security/rootless