@@ -1,267 +0,0 @@
# TODO: persist contribution history in SQLite + audit cleanup
Backend-only work. Follow Red → Green → Refactor per `CLAUDE.md` for every
step that adds behavior (store, retention, provider `$since` handling).
Order matters — later steps assume earlier ones are done.
## 1. Audit cleanup
- [x] **Remove `eightpoints/guzzle-bundle` + `idci/graphql-client-bundle`. **
They exist only to build one near-static GitHub GraphQL query string
(`GitHubProvider.php:59-74` ); the bundle's own HTTP transport is already
bypassed (comment at `GitHubProvider.php:76-77` ).
- [x] Replace
`$graphqlClient->buildQuery(...)->getGraphQLQuery()` with a plain
`sprintf` /heredoc query string. Delete the two packages from
`composer.json` , `config/bundles.php` , and delete
`config/packages/eight_points_guzzle.yaml` +
`config/packages/idci_graphql_client.yaml` . Run `composer update` and
commit the regenerated `composer.lock` .
- [x] **Dedupe `baseUrl` normalization. ** `GitLabProvider.php` (lines 41, 53) and `GiteaProvider.php` (lines 42, 54) each call
`rtrim($this->baseUrl..., '/')` twice — once in `ping()` , once in
`fetch()` . Compute it once as a `private readonly string $baseUrl` in
the constructor instead.
## 2. Fix stale `config/services.yaml` (found during exploration, blocks step 1 & 4)
- [x] `services.yaml` still binds `$username` /`$token` /`$baseUrl` to the
pre-refactor FQCNs (`App\Service\GitHubProvider` etc.) and
`_instanceof: App\Service\ProviderInterface` , left over from the
`Service/` → `Provider/` +`Renderer/` namespace reorg. Update all of
these to `App\Service\Provider\...` . Currently these bindings silently
no-op, and scalar constructor args can't autowire without them — any
edit to these constructors (steps 1, 4) needs this fixed first, or the
container fails to compile.
## 3. `ContributionStore` (SQLite via native PDO)
> Docs: [PDO](https://www.php.net/manual/en/book.pdo.php) ·
> [PDO_SQLITE driver](https://www.php.net/manual/en/ref.pdo-sqlite.php) ·
> [PDO::prepare / prepared statements](https://www.php.net/manual/en/pdo.prepare.php) ·
> [SQLite `INSERT ... ON CONFLICT` upsert](https://www.sqlite.org/lang_upsert.html) ·
> [SQLite `STRICT` tables](https://www.sqlite.org/stricttables.html) ·
> [SQLite `WITHOUT ROWID` tables](https://www.sqlite.org/withoutrowid.html) ·
> [SQLite datatypes (why `date` is `INTEGER`/unixtime, not `TEXT`)](https://www.sqlite.org/datatype3.html)
- [x] New `src/Service/ContributionStore.php` . PDO SQLite, DB file at
`%kernel.project_dir%/var/data/contributions.db` (configurable
constructor arg), table created lazily. **Schema note: ** `date` is
stored as a Unix timestamp (`INTEGER` ), not `TEXT` — matches the
current implementation:
```sql
CREATE TABLE IF NOT EXISTS contributions (
provider TEXT NOT NULL,
date INTEGER NOT NULL,
count INTEGER NOT NULL CHECK (count >= 0),
PRIMARY KEY (provider, date)
) WITHOUT ROWID, STRICT;
` ``
- [x] ` add(string $provider, int $unixtime, int $count): void` — plain
insert (done, needs test coverage — see below).
- [x] Fix ` add()`: currently a plain ` INSERT`, so re-adding an existing
` (provider, date)` throws a unique-constraint violation instead of
upserting. Switch to
` INSERT ... ON CONFLICT(provider, date) DO UPDATE SET count = excluded.count`
(see SQLite upsert doc above), or keep ` add()` insert-only and add a
separate ` merge()` for the upsert case used by step 5.
- [x] ` remove(string $provider, int $unixtime): void` — started, has a
bug: ` DELETE contributions WHERE ...` is invalid SQL, missing the
` FROM` keyword (must be ` DELETE FROM contributions WHERE ...`); also
drop the ` LIKE` on ` provider` (exact match, use ` =`) since it's an
unindexed wildcard scan for what should be an exact key lookup.
- [x] **` Contribution` entity.** Small immutable value object
(` src/Entity/Contribution.php`) wrapping one row: ` provider` (string),
` date` (unix timestamp int, or ` \DateTimeImmutable` — pick one and
use it consistently everywhere, including ` add()`/` all()`), ` count`
(int). Gives ` ContributionStore::all()` and the aggregator something
typed to pass around instead of raw arrays/tuples.
- [x] **` ContributionCollection`.** Typed collection
(` src/Entity/ContributionCollection.php`) wrapping
` array<Contribution>` — implement ` IteratorAggregate` + ` Countable`
at minimum ([` IteratorAggregate`](https://www.php.net/manual/en/class.iteratoraggregate.php),
[` Countable`](https://www.php.net/manual/en/class.countable.php)) so
it can be ` foreach`'d and ` count()`'d like a normal array. This is
what ` ContributionStore::all()` should return instead of a bare
` date => count` array.
- [x] ` latestDate(string $provider): ?int` — ` SELECT MAX(date) WHERE provider = ?`
(returns a unix timestamp, not a string, per the schema above).
- [x] ` merge(string $provider, array $dateCounts): void` — upsert via
` INSERT ... ON CONFLICT(provider, date) DO UPDATE SET count = excluded.count`;
` $dateCounts` keyed by unix timestamp.
- [x] ` all(string $provider, ?int $sinceDays = null): ContributionCollection` —
fetch all rows for a provider, or since a given range. ` null` returns
full history (no arbitrary cutoff), a value filters to rows where
` date >= (now - sinceDays * 86400)`. * *Bug:** current stub in
`ContributionStore.php:58` has invalid PHP (`$provider == null` instead
of `?string $provider = null` — this won't parse) and returns `void`
instead of `ContributionCollection` ; fix the signature when implementing.
- [x] `prune(): void` — `DELETE FROM contributions WHERE date < ?` using
the configured retention window (a unix timestamp cutoff); no-ops if
retention is unset/0.
- [x] Constructor takes `?int $retentionDays` bound from new env var
`CONTRIBUTIONS_RETENTION_DAYS` (empty/default = keep forever). _(constructor
param wired; the env binding itself is step 6's job.)_
- [x] Tests: `tests/Unit/Service/ContributionStoreTest.php` — store &
retrieve, upsert overwrites existing date, `sinceDays` filtering,
`prune()` no-ops when retention unset, `prune()` deletes rows older
than the window when set.
- [x] Tests: `tests/Unit/Entity/ContributionTest.php` and
`ContributionCollectionTest.php` — construction, iteration, `count()` .
## 4. Wire `$since`/`$until` through providers
Root cause of the live `MaxExecutionTimeError` crash: every cache-miss
request re-fetches the full ~365-day history from every host, and
GitLab's pagination is unbounded and sequential. Bounding the fetch
window (step 5's 3-day trailing overlap) is what actually fixes it — a
PHP `max_execution_time` fatal can't be caught by `try/catch` at all, so
"catch it better" was never on the table.
- [x] `ProviderInterface::startFetch()` → `startFetch(?\DateTimeImmutable $since = null, ?\DateTimeImmutable $until = null): mixed`
(interface already split into `startFetch` /`resolveFetch` for
concurrency — this doc previously said `fetch()` , which predates that
split).
- [x] `GitHubProvider::startFetch()` — already builds explicit `from` /`to`
GraphQL args (`GitHubProvider.php:51-58` ); swap the hardcoded
`-365 days` /`now` for `$since ?? -365 days` / `$until ?? now` .
- [x] `GitLabProvider::startFetch()` — add a `before` query param alongside
the existing `after` (`GitLabProvider.php:77-84` ), fed by `$until` ;
`$since` already flows into `after` (this is what actually shrinks the
pagination loop).
- [x] `GiteaProvider::resolveFetch()` — heatmap endpoint has no query
params (always returns full history); add an `$until` upper-bound
filter alongside the existing `$cutoff` lower bound.
- [x] Update `GitHubProviderTest.php` , `GitLabProviderTest.php` ,
`GiteaProviderTest.php` — add cases asserting a passed `$since` /`$until`
narrows the request window/query params.
## 5. Wire `ContributionStore` into `ContributionAggregator`
- [x] Fix `ContributionStore` wiring in `config/services.yaml` first —
it's currently dead code (nothing calls it). The constructor default
`$dbPath` string (`%kernel.project_dir%/var/data/contributions.db` ) is
a plain PHP default, not a resolved container parameter; bind it
explicitly (same pattern as the provider bindings):
```yaml
App\Service\ContributionStore:
arguments:
$dbPath: "%kernel.project_dir%/var/data/contributions.db"
$retentionDays: "%env(int:CONTRIBUTIONS_RETENTION_DAYS)%"
` ``
Add ` env(CONTRIBUTIONS_RETENTION_DAYS): ''` to ` parameters:` (empty →
casts to ` 0` → ` prune()`'s existing ` retentionDays === 0` check already
treats that as "keep forever"). Document the var in ` .env`.
- [x] Inject ` ContributionStore` into ` ContributionAggregator`.
- [x] Per configured provider: ` $latest = $store->latestDate($name)` →
` $since = $latest !== null ? (new \DateTimeImmutable('@' . $latest))->modify('-3 days') : null`
(3-day overlap for late corrections — old stored days are immutable and
never re-fetched, only this trailing window + anything new hits the
network), else ` null` (first run, provider's own default lookback).
` $until = null` (always "up to now" on the normal request path).
- [x] ` startFetch($since, $until)` / ` resolveFetch()` as today,
` $store->merge($name, $fresh)` on success, then read back
` $store->all($name, sinceDays: 371)` for the render window
(53 weeks × ContributionCollection` — and merge its
contributions into the returned array (the store becomes the source
of truth for what gets rendered, not the fresh fetch alone).
- [x] Call ` $store->prune()` once per ` aggregate()` call, after all
providers have merged.
- [x] Keep the existing try/catch-and-log-per-provider behavior — a
provider failure leaves its DB history stale, doesn't break the render.
- [x] Update ` ContributionAggregatorTest.php` with store-interaction
(` latestDate` consulted, ` merge` called, ` all()` feeds the result,
` prune()` runs once) and a case confirming a provider failure leaves
other providers' stored data intact.
## 6. Docker / env
- [x] ` docker-compose.yml` — add a ` data` named volume mounted at
` /app/var/data` (same pattern as ` cache`/` logs`), and pass through
` CONTRIBUTIONS_RETENTION_DAYS: "${CONTRIBUTIONS_RETENTION_DAYS:-}"`.
- [x] ` Dockerfile` — add ` var/data` to the ` mkdir -p` in the ` final` stage
alongside ` var/cache/prod/pools var/log`, owned by ` app`.
- [x] ` .env` — document ` CONTRIBUTIONS_RETENTION_DAYS` (empty by default),
same style as the existing ` ALLOWED_HOSTS` comment.
## 9. ` graph:contributions:refetch` console command
Manual escape hatch for forcing a full or ranged re-fetch (e.g. after
adding a new host, or if the store needs rebuilding) — bypasses step 5's
incremental ` latestDate()` window entirely for the providers/range given.
- [x] ` src/Command/RefetchContributionsCommand.php`, ` #[AsCommand]` +
constructor-promoted readonly properties (matches this codebase's
attribute-first style), using ` SymfonyStyle`
([console styling guide](https://symfony.com/doc/current/console/style.html)).
Inject ` iterable $providers` (` #[AutowireIterator('app.provider')]`,
same as ` ContributionAggregator`) and ` ContributionStore`.
- [x] Options: ` --provider=github,gitlab` (repeatable/comma-split,
restricts to named provider(s), default = all configured; unknown name
→ ` $io->error()` + ` Command::FAILURE`), ` --from=YYYY-MM-DD` (default
today − --to=YYYY-MM-DD` (default today), ` --all` (shorthand
for ` --from` far enough back — e.g. 2005-01-01 — to mean "existing full
history"; combinable with ` --provider`).
- [x] **Batch by range**: split ` [from, to]` into ≤365-day chunks
(GitHub's GraphQL ` contributionsCollection` rejects windows over a
year; chunking also caps GitLab's pagination per call, so a big
` --all` re-fetch can't reintroduce the original runaway-pagination
timeout). Iterate chunks oldest-first.
- [x] Per provider, per chunk: ` $io->section(...)`, ` startFetch($chunkFrom, $chunkTo)`
/ ` resolveFetch()`, ` $store->merge($name, $result)` immediately (don't
accumulate all chunks in memory), ` ProgressBar` across chunks.
- [x] Catch per-provider/per-chunk ` \Throwable` → ` $io->warning(...)`,
continue with remaining chunks/providers (same graceful-degradation
philosophy as ` ContributionAggregator`).
- [x] Finish with ` $io->table(...)` summary (provider, days written,
chunks fetched, any errors) and ` $io->success()`/` $io->error()`; exit
` Command::SUCCESS` if at least one provider fully succeeded, else
` Command::FAILURE`.
- [x] Tests: ` tests/Unit/Command/RefetchContributionsCommandTest.php`
using ` CommandTester` with fake ` ProviderInterface` stubs and a
` :memory:` ` ContributionStore` — assert chunking count for a >365-day
range, store ends up populated, unknown ` --provider` name fails
cleanly, a provider throwing doesn't abort the others.
## 10. PHPStan
- [x] ` composer require --dev phpstan/phpstan` (plain PHPStan — no
Symfony extension needed for this app's size).
- [x] Add ` phpstan.neon`: ` paths: [src, tests]`, ` level: 8`.
- [x] Add composer script ` "phpstan": "phpstan analyse"`.
- [x] Update README.md docs with new command and php stan static lintin.
- [x] Document in PHP-Stan-Errors.md whatever level-8 flags on first run.
## 11. ` CLAUDE.md` update
- [x] Architecture section: add the ` ContributionStore` (SQLite) tier
between providers and the renderer; note the two-tier cache (1h SVG
cache → SQLite raw-data store → provider APIs).
- [x] Environment variables table: add ` CONTRIBUTIONS_RETENTION_DAYS`.
- [x] Document ` app:contributions:refetch` (options, batching behavior).
- [x] Development section: add ` composer phpstan` next to the existing
` vendor/bin/phpunit` commands.
- [x] Remove any remaining mentions of the two deleted bundles. (none found —
already clean)
## Verification
- ` vendor/bin/phpunit --testdox` green after every step.
- ` composer phpstan` clean at level 8.
- ` composer install` succeeds with no dangling references to the removed bundles.
- ` docker compose up -d --build`; clear the FS cache
(` docker compose exec graph rm -rf var/cache/*`); hit
` /graph.svg?theme=dark` twice; check ` docker compose logs -f graph` shows
a narrower ` since` window on the second fetch.
- Set ` CONTRIBUTIONS_RETENTION_DAYS=30`; confirm rows older than 30 days
are gone after the next refresh
(` sqlite3 var/data/contributions.db "SELECT date FROM contributions ORDER BY date LIMIT 1"`).
- ` curl localhost:8080/health` still reports all configured providers healthy.
- ` docker compose exec graph bin/console app:contributions:refetch --all`
populates ` var/data/contributions.db` from empty; re-run with no flags
and confirm logs show only the trailing-window re-fetch.
- ` docker compose exec graph bin/console app:contributions:refetch --provider=github --from=2020-01-01 --to=2023-01-01`
exercises multi-chunk batching.
- Confirm the original crash scenario is gone: an account with heavy
GitLab history no longer triggers ` MaxExecutionTimeError` on a cold
cache (bounded by the 3-day trailing window, not full history).
haylan