git-contribution-graph/TODO.md

# TODO: persist contribution history in SQLite + audit cleanup

Backend-only work. Follow Red → Green → Refactor per `CLAUDE.md` for every
step that adds behavior (store, retention, provider `$since` handling).
Order matters — later steps assume earlier ones are done.

## 1. Audit cleanup

- [x] **Remove `eightpoints/guzzle-bundle` + `idci/graphql-client-bundle`.**
      They exist only to build one near-static GitHub GraphQL query string
      (`GitHubProvider.php:59-74`); the bundle's own HTTP transport is already
      bypassed (comment at `GitHubProvider.php:76-77`).
  - [x] Replace
        `$graphqlClient->buildQuery(...)->getGraphQLQuery()` with a plain
        `sprintf`/heredoc query string. Delete the two packages from
        `composer.json`, `config/bundles.php`, and delete
        `config/packages/eight_points_guzzle.yaml` +
        `config/packages/idci_graphql_client.yaml`. Run `composer update` and
        commit the regenerated `composer.lock`.

- [x] **Dedupe `baseUrl` normalization.** `GitLabProvider.php` (lines 41, 53) and `GiteaProvider.php` (lines 42, 54) each call
      `rtrim($this->baseUrl..., '/')` twice — once in `ping()`, once in
      `fetch()`. Compute it once as a `private readonly string $baseUrl` in
      the constructor instead.

## 2. Fix stale `config/services.yaml` (found during exploration, blocks step 1 & 4)

- [x] `services.yaml` still binds `$username`/`$token`/`$baseUrl` to the
      pre-refactor FQCNs (`App\Service\GitHubProvider` etc.) and
      `_instanceof: App\Service\ProviderInterface`, left over from the
      `Service/` → `Provider/`+`Renderer/` namespace reorg. Update all of
      these to `App\Service\Provider\...`. Currently these bindings silently
      no-op, and scalar constructor args can't autowire without them — any
      edit to these constructors (steps 1, 4) needs this fixed first, or the
      container fails to compile.

## 3. `ContributionStore` (SQLite via native PDO)

> Docs: [PDO](https://www.php.net/manual/en/book.pdo.php) ·
> [PDO_SQLITE driver](https://www.php.net/manual/en/ref.pdo-sqlite.php) ·
> [PDO::prepare / prepared statements](https://www.php.net/manual/en/pdo.prepare.php) ·
> [SQLite `INSERT ... ON CONFLICT` upsert](https://www.sqlite.org/lang_upsert.html) ·
> [SQLite `STRICT` tables](https://www.sqlite.org/stricttables.html) ·
> [SQLite `WITHOUT ROWID` tables](https://www.sqlite.org/withoutrowid.html) ·
> [SQLite datatypes (why `date` is `INTEGER`/unixtime, not `TEXT`)](https://www.sqlite.org/datatype3.html)

- [x] New `src/Service/ContributionStore.php`. PDO SQLite, DB file at
      `%kernel.project_dir%/var/data/contributions.db` (configurable
      constructor arg), table created lazily. **Schema note:** `date` is
      stored as a Unix timestamp (`INTEGER`), not `TEXT` — matches the
      current implementation:
  ```sql
  CREATE TABLE IF NOT EXISTS contributions (
      provider TEXT NOT NULL,
      date     INTEGER NOT NULL,
      count    INTEGER NOT NULL CHECK (count >= 0),
      PRIMARY KEY (provider, date)
  ) WITHOUT ROWID, STRICT;
  ```
- [x] `add(string $provider, int $unixtime, int $count): void` — plain
      insert (done, needs test coverage — see below).
- [x] Fix `add()`: currently a plain `INSERT`, so re-adding an existing
      `(provider, date)` throws a unique-constraint violation instead of
      upserting. Switch to
      `INSERT ... ON CONFLICT(provider, date) DO UPDATE SET count = excluded.count`
      (see SQLite upsert doc above), or keep `add()` insert-only and add a
      separate `merge()` for the upsert case used by step 5.
- [x] `remove(string $provider, int $unixtime): void` — started, has a
      bug: `DELETE contributions WHERE ...` is invalid SQL, missing the
      `FROM` keyword (must be `DELETE FROM contributions WHERE ...`); also
      drop the `LIKE` on `provider` (exact match, use `=`) since it's an
      unindexed wildcard scan for what should be an exact key lookup.
- [x] **`Contribution` entity.** Small immutable value object
      (`src/Entity/Contribution.php`) wrapping one row: `provider` (string),
      `date` (unix timestamp int, or `\DateTimeImmutable` — pick one and
      use it consistently everywhere, including `add()`/`all()`), `count`
      (int). Gives `ContributionStore::all()` and the aggregator something
      typed to pass around instead of raw arrays/tuples.
- [x] **`ContributionCollection`.** Typed collection
      (`src/Entity/ContributionCollection.php`) wrapping
      `array<Contribution>` — implement `IteratorAggregate` + `Countable`
      at minimum ([`IteratorAggregate`](https://www.php.net/manual/en/class.iteratoraggregate.php),
      [`Countable`](https://www.php.net/manual/en/class.countable.php)) so
      it can be `foreach`'d and `count()`'d like a normal array. This is
      what `ContributionStore::all()` should return instead of a bare
      `date => count` array.
- [x] `latestDate(string $provider): ?int` — `SELECT MAX(date) WHERE provider = ?`
      (returns a unix timestamp, not a string, per the schema above).
- [x] `merge(string $provider, array $dateCounts): void` — upsert via
      `INSERT ... ON CONFLICT(provider, date) DO UPDATE SET count = excluded.count`;
      `$dateCounts` keyed by unix timestamp.
- [x] `all(string $provider, ?int $sinceDays = null): ContributionCollection` —
      fetch all rows for a provider, or since a given range. `null` returns
      full history (no arbitrary cutoff), a value filters to rows where
      `date >= (now - sinceDays * 86400)`. **Bug:** current stub in
      `ContributionStore.php:58` has invalid PHP (`$provider == null` instead
      of `?string $provider = null` — this won't parse) and returns `void`
      instead of `ContributionCollection`; fix the signature when implementing.
- [x] `prune(): void` — `DELETE FROM contributions WHERE date < ?` using
      the configured retention window (a unix timestamp cutoff); no-ops if
      retention is unset/0.
- [x] Constructor takes `?int $retentionDays` bound from new env var
      `CONTRIBUTIONS_RETENTION_DAYS` (empty/default = keep forever). _(constructor
      param wired; the env binding itself is step 6's job.)_
- [x] Tests: `tests/Unit/Service/ContributionStoreTest.php` — store &
      retrieve, upsert overwrites existing date, `sinceDays` filtering,
      `prune()` no-ops when retention unset, `prune()` deletes rows older
      than the window when set.
- [x] Tests: `tests/Unit/Entity/ContributionTest.php` and
      `ContributionCollectionTest.php` — construction, iteration, `count()`.

## 4. Wire `$since`/`$until` through providers

Root cause of the live `MaxExecutionTimeError` crash: every cache-miss
request re-fetches the full ~365-day history from every host, and
GitLab's pagination is unbounded and sequential. Bounding the fetch
window (step 5's 3-day trailing overlap) is what actually fixes it — a
PHP `max_execution_time` fatal can't be caught by `try/catch` at all, so
"catch it better" was never on the table.

- [x] `ProviderInterface::startFetch()` → `startFetch(?\DateTimeImmutable $since = null, ?\DateTimeImmutable $until = null): mixed`
      (interface already split into `startFetch`/`resolveFetch` for
      concurrency — this doc previously said `fetch()`, which predates that
      split).
- [x] `GitHubProvider::startFetch()` — already builds explicit `from`/`to`
      GraphQL args (`GitHubProvider.php:51-58`); swap the hardcoded
      `-365 days`/`now` for `$since ?? -365 days` / `$until ?? now`.
- [x] `GitLabProvider::startFetch()` — add a `before` query param alongside
      the existing `after` (`GitLabProvider.php:77-84`), fed by `$until`;
      `$since` already flows into `after` (this is what actually shrinks the
      pagination loop).
- [x] `GiteaProvider::resolveFetch()` — heatmap endpoint has no query
      params (always returns full history); add an `$until` upper-bound
      filter alongside the existing `$cutoff` lower bound.
- [x] Update `GitHubProviderTest.php`, `GitLabProviderTest.php`,
      `GiteaProviderTest.php` — add cases asserting a passed `$since`/`$until`
      narrows the request window/query params.

## 5. Wire `ContributionStore` into `ContributionAggregator`

- [x] Fix `ContributionStore` wiring in `config/services.yaml` first —
      it's currently dead code (nothing calls it). The constructor default
      `$dbPath` string (`%kernel.project_dir%/var/data/contributions.db`) is
      a plain PHP default, not a resolved container parameter; bind it
      explicitly (same pattern as the provider bindings):
  ```yaml
  App\Service\ContributionStore:
    arguments:
      $dbPath: "%kernel.project_dir%/var/data/contributions.db"
      $retentionDays: "%env(int:CONTRIBUTIONS_RETENTION_DAYS)%"
  ```
  Add `env(CONTRIBUTIONS_RETENTION_DAYS): ''` to `parameters:` (empty →
  casts to `0` → `prune()`'s existing `retentionDays === 0` check already
  treats that as "keep forever"). Document the var in `.env`.
- [x] Inject `ContributionStore` into `ContributionAggregator`.
- [x] Per configured provider: `$latest = $store->latestDate($name)` →
      `$since = $latest !== null ? (new \DateTimeImmutable('@' . $latest))->modify('-3 days') : null`
      (3-day overlap for late corrections — old stored days are immutable and
      never re-fetched, only this trailing window + anything new hits the
      network), else `null` (first run, provider's own default lookback).
      `$until = null` (always "up to now" on the normal request path).
- [x] `startFetch($since, $until)` / `resolveFetch()` as today,
      `$store->merge($name, $fresh)` on success, then read back
      `$store->all($name, sinceDays: 371)` for the render window
      (53 weeks × 7 days) — a `ContributionCollection` — and merge its
      contributions into the returned array (the store becomes the source
      of truth for what gets rendered, not the fresh fetch alone).
- [x] Call `$store->prune()` once per `aggregate()` call, after all
      providers have merged.
- [x] Keep the existing try/catch-and-log-per-provider behavior — a
      provider failure leaves its DB history stale, doesn't break the render.
- [x] Update `ContributionAggregatorTest.php` with store-interaction
      (`latestDate` consulted, `merge` called, `all()` feeds the result,
      `prune()` runs once) and a case confirming a provider failure leaves
      other providers' stored data intact.

## 6. Docker / env

- [x] `docker-compose.yml` — add a `data` named volume mounted at
      `/app/var/data` (same pattern as `cache`/`logs`), and pass through
      `CONTRIBUTIONS_RETENTION_DAYS: "${CONTRIBUTIONS_RETENTION_DAYS:-}"`.
- [x] `Dockerfile` — add `var/data` to the `mkdir -p` in the `final` stage
      alongside `var/cache/prod/pools var/log`, owned by `app`.
- [x] `.env` — document `CONTRIBUTIONS_RETENTION_DAYS` (empty by default),
      same style as the existing `ALLOWED_HOSTS` comment.

## 9. `graph:contributions:refetch` console command

Manual escape hatch for forcing a full or ranged re-fetch (e.g. after
adding a new host, or if the store needs rebuilding) — bypasses step 5's
incremental `latestDate()` window entirely for the providers/range given.

- [x] `src/Command/RefetchContributionsCommand.php`, `#[AsCommand]` +
      constructor-promoted readonly properties (matches this codebase's
      attribute-first style), using `SymfonyStyle`
      ([console styling guide](https://symfony.com/doc/current/console/style.html)).
      Inject `iterable $providers` (`#[AutowireIterator('app.provider')]`,
      same as `ContributionAggregator`) and `ContributionStore`.
- [x] Options: `--provider=github,gitlab` (repeatable/comma-split,
      restricts to named provider(s), default = all configured; unknown name
      → `$io->error()` + `Command::FAILURE`), `--from=YYYY-MM-DD` (default
      today − 365 days), `--to=YYYY-MM-DD` (default today), `--all` (shorthand
      for `--from` far enough back — e.g. 2005-01-01 — to mean "existing full
      history"; combinable with `--provider`).
- [x] **Batch by range**: split `[from, to]` into ≤365-day chunks
      (GitHub's GraphQL `contributionsCollection` rejects windows over a
      year; chunking also caps GitLab's pagination per call, so a big
      `--all` re-fetch can't reintroduce the original runaway-pagination
      timeout). Iterate chunks oldest-first.
- [x] Per provider, per chunk: `$io->section(...)`, `startFetch($chunkFrom, $chunkTo)`
      / `resolveFetch()`, `$store->merge($name, $result)` immediately (don't
      accumulate all chunks in memory), `ProgressBar` across chunks.
- [x] Catch per-provider/per-chunk `\Throwable` → `$io->warning(...)`,
      continue with remaining chunks/providers (same graceful-degradation
      philosophy as `ContributionAggregator`).
- [x] Finish with `$io->table(...)` summary (provider, days written,
      chunks fetched, any errors) and `$io->success()`/`$io->error()`; exit
      `Command::SUCCESS` if at least one provider fully succeeded, else
      `Command::FAILURE`.
- [x] Tests: `tests/Unit/Command/RefetchContributionsCommandTest.php`
      using `CommandTester` with fake `ProviderInterface` stubs and a
      `:memory:` `ContributionStore` — assert chunking count for a >365-day
      range, store ends up populated, unknown `--provider` name fails
      cleanly, a provider throwing doesn't abort the others.

## 10. PHPStan

- [x] `composer require --dev phpstan/phpstan` (plain PHPStan — no
      Symfony extension needed for this app's size).
- [x] Add `phpstan.neon`: `paths: [src, tests]`, `level: 8`.
- [x] Add composer script `"phpstan": "phpstan analyse"`.
- [x] Update README.md docs with new command and php stan static lintin.
- [x] Document in PHP-Stan-Errors.md whatever level-8 flags on first run.

## 11. `CLAUDE.md` update

- [x] Architecture section: add the `ContributionStore` (SQLite) tier
      between providers and the renderer; note the two-tier cache (1h SVG
      cache → SQLite raw-data store → provider APIs).
- [x] Environment variables table: add `CONTRIBUTIONS_RETENTION_DAYS`.
- [x] Document `app:contributions:refetch` (options, batching behavior).
- [x] Development section: add `composer phpstan` next to the existing
      `vendor/bin/phpunit` commands.
- [x] Remove any remaining mentions of the two deleted bundles. (none found —
      already clean)

## Verification

- `vendor/bin/phpunit --testdox` green after every step.
- `composer phpstan` clean at level 8.
- `composer install` succeeds with no dangling references to the removed bundles.
- `docker compose up -d --build`; clear the FS cache
  (`docker compose exec graph rm -rf var/cache/*`); hit
  `/graph.svg?theme=dark` twice; check `docker compose logs -f graph` shows
  a narrower `since` window on the second fetch.
- Set `CONTRIBUTIONS_RETENTION_DAYS=30`; confirm rows older than 30 days
  are gone after the next refresh
  (`sqlite3 var/data/contributions.db "SELECT date FROM contributions ORDER BY date LIMIT 1"`).
- `curl localhost:8080/health` still reports all configured providers healthy.
- `docker compose exec graph bin/console app:contributions:refetch --all`
  populates `var/data/contributions.db` from empty; re-run with no flags
  and confirm logs show only the trailing-window re-fetch.
- `docker compose exec graph bin/console app:contributions:refetch --provider=github --from=2020-01-01 --to=2023-01-01`
  exercises multi-chunk batching.
- Confirm the original crash scenario is gone: an account with heavy
  GitLab history no longer triggers `MaxExecutionTimeError` on a cold
  cache (bounded by the 3-day trailing window, not full history).