haylan/godot-ci
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(research): act_runner Docker volume accumulation and cleanup

Browse Source 
Investigates why git.arthurerlich.de's act_runner/Docker CI host
accumulates Docker volumes, grounded in act_runner's own config docs,
Gitea's Actions docs, Docker's official prune/rootless docs, and a
maintainer-confirmed analogous root cause from GitHub's actions/runner.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
haylan
2026-08-07 00:57:16 +02:00
co-authored by Claude-Bot
parent f0307c6558
commit adbc3cd6e7
1 changed files with 318 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/act-runner-volume-cleanup.md
+318
View File
@@ -0,0 +1,318 @@
# Why act_runner's Docker volumes accumulate, and how to clean them up
> **Scope note:** this is about the **host/runner infrastructure** for
> git.arthurerlich.de's Gitea Actions (`act_runner` + the Docker daemon it
> drives, in the docker-compose setup for that Gitea instance) — **not**
> about this repo's own `Dockerfile`/published image. This repo's
> `publish.yml` already runs `docker system prune -af --volumes` before its
> build for the same underlying reason described here (unbounded volume/image
> growth on the CI host), which is what prompted this research. No access to
> the actual Gitea docker-compose file was available or used; every claim
> below is grounded in act_runner's/Gitea's/Docker's own docs and source, or
> flagged as inferred/secondary where it isn't.
## Recommendations (ranked)
1. **Schedule `docker system prune -af --volumes` on the runner host on a
recurring basis (cron or a scheduled Gitea Action), the same pattern this
repo's own `publish.yml` already uses before its build.** — **Proven
mechanism, standard mitigation.** No built-in act_runner/Gitea equivalent
exists (see §2); this is the documented, generic Docker-level fix and is
what community guidance for this exact runner-volume-leak class of
problem converges on. Prefer a filtered form if the host also runs
same-day rebuilds you don't want swept up:
`docker system prune -af --volumes --filter "until=24h"` (or
`docker volume prune --filter "until=24h"` for a volumes-only, less
aggressive sweep) — see §2 for the exact filter semantics.
2. **Confirm/enable act_runner's `container.valid_volumes` allowlist and
review `container.options`/`container.network`, but do not expect them to
fix the leak** — **Proven from act_runner's own config reference (§1.2):
these control what containers are *allowed* to mount, not automatic
cleanup after a job ends.** Leaving `valid_volumes` wide open
(`- '**'`) doesn't itself cause more leakage than default, but tightening
it is good hygiene independent of the disk problem.
3. **If DinD (Docker-in-Docker) mode is in use, that is very likely where
the bulk of the leaked volumes come from** — **inferred, but from a
documented, closely-analogous mechanism (§1), not confirmed against this
host's actual config** since no docker-compose file was available. DinD
needs its own persistent Docker data directory (commonly volume-backed),
and any job container that declares a `VOLUME` in its Dockerfile gets an
anonymous volume that Docker does not remove on `docker rm` unless the
container was started with `--rm` — a mechanism GitHub's own
`actions/runner` maintainers confirmed by design for the conceptually
identical GitHub Actions Docker executor (§1.3). act_runner's job/step
containers are created, started, stopped, and removed as separate Docker
API calls (not a single `docker run --rm`), which is the same shape of
gap.
4. **A dedicated Linux user for act_runner (and rootless Docker specifically)
is a security/isolation improvement, not a fix for volume accumulation.**
— **Proven boundary, directly stated in act_runner's own docs (§3).**
Do it for blast-radius/`docker.sock`-exposure reasons; don't expect it to
change disk usage. Running the Docker daemon itself in **rootless mode**
changes *where* volumes live (per-user data root under
`$XDG_RUNTIME_DIR`/`~/.local/share/docker` instead of `/var/lib/docker`)
but Docker's own rootless docs describe no change to prune/cleanup
semantics — unused volumes still only go away via an explicit prune,
run as that user against that daemon.
**Net: there is no act_runner/Gitea-side switch that stops the leak.** The
fix is operational — a scheduled prune — same shape as the one already
present in this repo's own `publish.yml`.
---
## 1. Why this happens
### 1.1 act_runner's job execution model
Per Gitea's own Act Runner documentation, act_runner supports two execution
modes: **Docker container mode** ("it is recommended to run jobs in a
docker container", requiring a running Docker daemon) and **host mode**
(jobs run directly on the machine act_runner is on, no Docker involved).
Docker mode is the default/recommended mode and is what a normal
docker-compose Gitea+act_runner setup uses.
([docs.gitea.com/1.23/usage/actions/act-runner](https://docs.gitea.com/1.23/usage/actions/act-runner/))
In Docker mode, act_runner creates a container per job (and per
service-container / step container as declared in the workflow), runs it,
and removes it — as separate Docker API calls, not a single atomic
`docker run --rm ...`. This matters because Docker's own volume-removal
rule is tied to *how* a container is removed, not just that it's gone (see
§1.3).
### 1.2 act_runner's `container.*` config settings
From act_runner's own `config.yaml` reference (fetched from
`gitea.com/gitea/act_runner`'s example config,
[gitea.com/gitea/act_runner — config.example.yaml](https://gitea.com/gitea/act_runner/raw/branch/main/internal/pkg/config/config.example.yaml)):
- **`container.docker_host`** — overrides the Docker daemon act_runner talks
to. Empty (default) auto-detects; `-` auto-detects without mounting the
socket into job containers. Controls *which* daemon runs jobs, not
cleanup.
- **`container.network`** — `host`, `bridge`, or a custom network name;
empty auto-creates a per-job network. Unrelated to volumes.
- **`container.privileged`** — "Privileged mode is required for
Docker-in-Docker." This is the flag that turns on DinD-style job
execution.
- **`container.options`** — free-form extra flags passed to container
creation (e.g. `--add-host=...`, or extra `--volume` mounts). Whatever a
workflow or admin puts here can itself add more bind mounts/volumes per
job, but is opt-in, not a default source of leakage.
- **`container.valid_volumes`** — an **allowlist** (glob syntax) of
volumes/bind mounts job containers are permitted to mount; empty means
none permitted, `'**'` means any. This is an authorization control, not a
lifecycle/cleanup control — it limits what *can* be mounted, and doesn't
cause or prevent leftover anonymous volumes from images that declare
`VOLUME` internally.
- **`container.bind_workdir`** — relevant specifically to DinD: "required
for Docker-in-Docker (DinD) setups when jobs use docker compose with bind
mounts (e.g. `.:/app`), as volume-based workspaces are not accessible
from the DinD daemon's filesystem." Confirms DinD's workspace handling is
a distinct code path from plain Docker mode.
- **`workdir_cleanup_age` / `idle_cleanup_interval`** — these exist, but per
the config reference they clean up **act_runner's own workspace
directories** (stale bind-mounted task-workdir folders on the host
filesystem, and orphaned host-mode scratch dirs) — **not Docker volumes**.
There is no equivalent `container`-level setting for pruning Docker
volumes.
None of act_runner's documented settings amount to "clean up Docker volumes
after a job." The closest things (`workdir_cleanup_age`,
`idle_cleanup_interval`) are scoped to filesystem workdirs, a different
mechanism from Docker volume objects.
### 1.3 Is DinD specifically the source of the leak, vs plain Docker mode?
Two independent, additive causes are documented:
1. **Any Docker execution mode (plain or DinD): anonymous volumes from
job/service containers leak by Docker's own removal semantics.** This is
not act_runner-specific — it's how Docker containers work. GitHub's own
`actions/runner` maintainers documented the identical failure mode for
GitHub Actions' (conceptually equivalent) Docker executor: *"When using
service containers with a `VOLUME` declaration in their Dockerfile, an
anonymous volume is automatically created with the container. At the end
of the workflow, the container is stopped and removed, but the anonymous
volumes it created stay around... The runner creates, starts, stops and
removes containers with separate commands, so passing `--rm` to `docker
create` has no effect on `docker remove`."*
([actions/runner#1885](https://github.com/actions/runner/issues/1885) —
maintainer-confirmed root cause, not community speculation). act_runner's
job execution is architecturally the same shape (separate create/start/
stop/remove calls against the Docker API), so the same leak mechanism
applies to it by design, absent extra config — this is the strongest
available evidence, though it is evidence from GitHub's runner, applied
by analogy to act_runner rather than a direct act_runner maintainer
statement (no act_runner-specific issue making this exact claim was
found in this pass).
2. **DinD adds its own persistent data directory on top.** A Docker-in-Docker
daemon needs its own `/var/lib/docker`-equivalent to store the images/
layers/volumes *of the jobs it runs*, and community DinD setups
routinely back that directory with a Docker volume so it survives
restarts of the DinD container itself (e.g. "`/var/lib/docker` cannot be
on AUFS, so it needs to be made a volume" — a long-standing operational
note about DinD from general Docker-in-Docker tooling, not Gitea-specific,
flagged here as secondary/community evidence). This is additive to (1):
DinD setups accumulate both the generic per-job anonymous-volume leak
*and* whatever grows inside DinD's own backing store (images, build
cache, and volumes belonging to the containers it spawns), which is a
second, larger accumulation surface than plain Docker mode.
**Conclusion:** the generic anonymous-volume leak happens in plain Docker
mode too (it's a Docker container-removal semantics issue, not a DinD-only
bug), but DinD mode is very likely the bigger contributor on a real host,
because it adds a whole second Docker data root that grows independently.
Without the actual docker-compose file, it isn't possible to confirm from
this pass alone whether git.arthurerlich.de's act_runner is configured for
DinD (`container.privileged: true` set) or plain Docker mode.
---
## 2. How to clean up
### 2.1 `docker volume prune` / `docker system prune` semantics (Docker's own docs)
Per Docker's official pruning reference
([docs.docker.com/engine/manage-resources/pruning](https://docs.docker.com/engine/manage-resources/pruning/),
[docs.docker.com/reference/cli/docker/volume/prune](https://docs.docker.com/reference/cli/docker/volume/prune/),
[docs.docker.com/reference/cli/docker/system/prune](https://docs.docker.com/reference/cli/docker/system/prune/)):
- **`docker system prune`** (no flags) removes stopped containers, networks
not used by at least one container, dangling images, and unused build
cache. **Volumes are excluded by default.**
- **`docker system prune --volumes`** (or `-a --volumes` to also sweep
non-dangling unused images) additionally removes "all volumes not used by
at least one container."
- **`docker volume prune`** removes unused local volumes; by default this
means **anonymous** volumes not referenced by any container. `--all`
extends this to unused **named** volumes too.
- **`docker volume prune --filter "until=<duration>"`** narrows the sweep to
volumes older than the given age (still only unused ones) — a way to keep
volumes from a job that finished minutes ago while still cleaning up
older leftovers.
**Safety on a host mid-job:** all of the above operate only on resources
**not referenced by any container, running or stopped**. A volume mounted
into a currently-running job container is never touched. This makes it safe
to run on a schedule against a live runner host without coordinating with
in-flight jobs — the documented behavior is conservative by construction,
not merely "usually fine in practice."
### 2.2 Does act_runner or Gitea ship any built-in cleanup for this?
No. Searched act_runner's config reference, Gitea's Actions/runner
documentation, and the two most relevant issue threads found
([go-gitea/gitea#31457](https://github.com/go-gitea/gitea/issues/31457),
about `act_runner` manageability generally, and
[gitea.com/gitea/runner#167](https://gitea.com/gitea/runner/issues/167) /
[go-gitea/gitea#24438](https://github.com/go-gitea/gitea/issues/24438),
about `docker.sock` exposure) — none describe a built-in Docker volume
cleanup mechanism, cron-style prune config, or maintainer statement
addressing this exact problem. The only cleanup knobs act_runner documents
(`workdir_cleanup_age`, `idle_cleanup_interval`) are scoped to its own
host-filesystem workdirs, not Docker volume objects (§1.2). This absence is
itself informative: there is nothing to "turn on" here — a scheduled
external prune is the only available lever.
### 2.3 Is a scheduled `docker system prune -af --volumes` the standard approach?
Yes, and it is corroborated by both this repo's own precedent and general
community guidance:
- This repo's `publish.yml` already runs `docker system prune -af --volumes`
before its build, for the same underlying class of problem (disk
exhaustion from accumulated Docker state on a CI runner) — direct
precedent already in this codebase.
- General community guidance for self-hosted CI runners converges on the
same shape of fix: a scheduled/cron `docker system prune` (optionally
filtered by age) run on the runner host. This is consistent, secondary
evidence (blog posts / community write-ups, not a Docker or Gitea
standards document) but aligns with what Docker's own docs make safe
(§2.1).
**A more targeted alternative** — `docker volume prune --filter "until=24h"`
run on a schedule (e.g. nightly cron) — trades some aggressiveness for
narrower scope: it only ever removes volumes, leaves stopped
containers/networks/images alone, and skips anything from the last 24h in
case a long-running or recently-finished job's volume is still wanted for
debugging. This is a reasonable middle ground if a full `system prune -af`
feels too broad for the runner host (e.g. if other non-CI Docker workloads
share the machine); `docker system prune -af --volumes` is simpler and is
what this repo already does elsewhere.
---
## 3. Dedicated user for the runner: security concern, not a cleanup mechanism
### 3.1 act_runner's own guidance
Gitea's Act Runner docs (systemd install instructions) recommend creating a
dedicated unprivileged `act_runner` user, and note that adding it to the
`docker` group (needed for Docker mode) **"effectively gives act_runner
root access to the system"** — framed entirely as a privilege/blast-radius
concern.
([docs.gitea.com/1.23/usage/actions/act-runner](https://docs.gitea.com/1.23/usage/actions/act-runner/))
This is echoed more sharply in Gitea's own tracker:
[gitea.com/gitea/runner#167](https://gitea.com/gitea/runner/issues/167) /
mirrored as [go-gitea/gitea#24438](https://github.com/go-gitea/gitea/issues/24438),
titled *"Gitea Actions is HIGHLY insecure due to binding of docker.sock into
all containers (= root on host)"* — the concern raised (and act_runner's own
`examples/vm/rootless-docker.md` walkthrough, built specifically in
response to this class of concern) is **container escape / arbitrary root
on the host via `docker.sock` exposure to untrusted job code**, not disk
usage.
### 3.2 Rootless Docker / user-namespace remapping (Docker's own docs)
Per Docker's official rootless-mode docs
([docs.docker.com/engine/security/rootless](https://docs.docker.com/engine/security/rootless/)):
rootless mode "lets you run the Docker daemon and containers as a non-root
user to mitigate potential vulnerabilities in the daemon and the container
runtime" — again, a security-isolation feature. The docs describe no
special relationship to volume/disk accumulation or cleanup behavior.
act_runner's own `rootless-docker.md` walkthrough
([gitea.com/gitea/act_runner — examples/vm/rootless-docker.md](https://gitea.com/gitea/act_runner/raw/branch/main/examples/vm/rootless-docker.md))
is, in its own words, exclusively about security/isolation setup (dedicated
user, rootless daemon, systemd units) and makes no mention of volume
cleanup or storage management at any point.
Practically, rootless mode does shift *where* Docker's data root (and
therefore its volumes) lives — per-user, typically under
`$XDG_RUNTIME_DIR` / `~/.local/share/docker` rather than the system-wide
`/var/lib/docker` — but this only changes the location being filled up, not
whether it fills up. The same `docker volume prune` semantics (§2.1) still
apply, run as that user against that user's daemon.
### 3.3 Conclusion
- **(a) Dedicated Linux user for the act_runner service itself:** purely a
blast-radius/permissions control (who can touch what if the runner or a
job container is compromised). No documented bearing on volume
accumulation.
- **(b) Rootless Docker / userns-remap:** also purely a security-isolation
mechanism per Docker's own docs. It relocates where volumes are stored
(per-user vs system-wide) but does not change prune semantics or provide
any automatic cleanup — the volume-growth problem is orthogonal to this
and is not solved by adopting either (a) or (b).
Both are worth doing for `docker.sock`-exposure/security reasons
independent of this investigation, but neither should be expected to move
the disk-usage needle — only a scheduled prune (§1 recommendation #1, §2.3)
addresses that.
---
## Sources
- act_runner config reference — [gitea.com/gitea/act_runner: config.example.yaml](https://gitea.com/gitea/act_runner/raw/branch/main/internal/pkg/config/config.example.yaml)
- Gitea Act Runner docs — [docs.gitea.com/1.23/usage/actions/act-runner](https://docs.gitea.com/1.23/usage/actions/act-runner/)
- act_runner rootless Docker walkthrough — [gitea.com/gitea/act_runner: examples/vm/rootless-docker.md](https://gitea.com/gitea/act_runner/raw/branch/main/examples/vm/rootless-docker.md)
- Gitea Actions `docker.sock` security issue — [gitea.com/gitea/runner#167](https://gitea.com/gitea/runner/issues/167) / mirrored [go-gitea/gitea#24438](https://github.com/go-gitea/gitea/issues/24438)
- act_runner manageability issue — [go-gitea/gitea#31457](https://github.com/go-gitea/gitea/issues/31457) (no maintainer disk/volume-cleanup discussion found in this thread)
- Anonymous volume leak, maintainer-confirmed root cause (GitHub's `actions/runner`, conceptually equivalent Docker executor) — [actions/runner#1885](https://github.com/actions/runner/issues/1885)
- Docker pruning reference — [docs.docker.com/engine/manage-resources/pruning](https://docs.docker.com/engine/manage-resources/pruning/)
- `docker volume prune` reference — [docs.docker.com/reference/cli/docker/volume/prune](https://docs.docker.com/reference/cli/docker/volume/prune/)
- `docker system prune` reference — [docs.docker.com/reference/cli/docker/system/prune](https://docs.docker.com/reference/cli/docker/system/prune/)
- Docker rootless mode — [docs.docker.com/engine/security/rootless](https://docs.docker.com/engine/security/rootless/)