diff --git a/TODO.md b/TODO.md index 8ce8925..c007d5e 100644 --- a/TODO.md +++ b/TODO.md @@ -110,35 +110,70 @@ Order matters — later steps assume earlier ones are done. - [x] Tests: `tests/Unit/Entity/ContributionTest.php` and `ContributionCollectionTest.php` — construction, iteration, `count()`. -## 4. Wire `$since` through providers +## 4. Wire `$since`/`$until` through providers -- [ ] `ProviderInterface::fetch()` → `fetch(?\DateTimeImmutable $since = null): array`. -- [ ] `GitHubProvider::fetch()` — use `$since ?? new \DateTimeImmutable('-365 days')` - for the GraphQL `from` param. -- [ ] `GitLabProvider::fetch()` — same, feeds the `after` query param - (this is what actually shrinks the pagination loop). -- [ ] `GiteaProvider::fetch()` — same, feeds the `$cutoff` timestamp filter. -- [ ] Update `GitHubProviderTest.php` (drop `GraphQLApiClientRegistryInterface` - stub setup entirely per step 1), `GitLabProviderTest.php`, - `GiteaProviderTest.php` — add a case asserting a passed `$since` narrows - the request window. +Root cause of the live `MaxExecutionTimeError` crash: every cache-miss +request re-fetches the full ~365-day history from every host, and +GitLab's pagination is unbounded and sequential. Bounding the fetch +window (step 5's 3-day trailing overlap) is what actually fixes it — a +PHP `max_execution_time` fatal can't be caught by `try/catch` at all, so +"catch it better" was never on the table. + +- [ ] `ProviderInterface::startFetch()` → `startFetch(?\DateTimeImmutable $since = null, ?\DateTimeImmutable $until = null): mixed` + (interface already split into `startFetch`/`resolveFetch` for + concurrency — this doc previously said `fetch()`, which predates that + split). +- [ ] `GitHubProvider::startFetch()` — already builds explicit `from`/`to` + GraphQL args (`GitHubProvider.php:51-58`); swap the hardcoded + `-365 days`/`now` for `$since ?? -365 days` / `$until ?? now`. +- [ ] `GitLabProvider::startFetch()` — add a `before` query param alongside + the existing `after` (`GitLabProvider.php:77-84`), fed by `$until`; + `$since` already flows into `after` (this is what actually shrinks the + pagination loop). +- [ ] `GiteaProvider::resolveFetch()` — heatmap endpoint has no query + params (always returns full history); add an `$until` upper-bound + filter alongside the existing `$cutoff` lower bound. +- [ ] Update `GitHubProviderTest.php`, `GitLabProviderTest.php`, + `GiteaProviderTest.php` — add cases asserting a passed `$since`/`$until` + narrows the request window/query params. ## 5. Wire `ContributionStore` into `ContributionAggregator` +- [ ] Fix `ContributionStore` wiring in `config/services.yaml` first — + it's currently dead code (nothing calls it). The constructor default + `$dbPath` string (`%kernel.project_dir%/var/data/contributions.db`) is + a plain PHP default, not a resolved container parameter; bind it + explicitly (same pattern as the provider bindings): + ```yaml + App\Service\ContributionStore: + arguments: + $dbPath: '%kernel.project_dir%/var/data/contributions.db' + $retentionDays: '%env(int:CONTRIBUTIONS_RETENTION_DAYS)%' + ``` + Add `env(CONTRIBUTIONS_RETENTION_DAYS): ''` to `parameters:` (empty → + casts to `0` → `prune()`'s existing `retentionDays === 0` check already + treats that as "keep forever"). Document the var in `.env`. - [ ] Inject `ContributionStore` into `ContributionAggregator`. -- [ ] Per configured provider: `$since = $store->latestDate($name)` → if - set, `(new \DateTimeImmutable($latest))->modify('-3 days')` (3-day - overlap for late corrections), else `null` (first run). -- [ ] `$fresh = $provider->fetch($since)`, `$store->merge($name, $fresh)`, - then read back `$store->all($name, sinceDays: 371)` for the render - window (53 weeks × 7 days) — a `ContributionCollection` — and merge - its contributions into the returned array. +- [ ] Per configured provider: `$latest = $store->latestDate($name)` → + `$since = $latest !== null ? (new \DateTimeImmutable('@' . $latest))->modify('-3 days') : null` + (3-day overlap for late corrections — old stored days are immutable and + never re-fetched, only this trailing window + anything new hits the + network), else `null` (first run, provider's own default lookback). + `$until = null` (always "up to now" on the normal request path). +- [ ] `startFetch($since, $until)` / `resolveFetch()` as today, + `$store->merge($name, $fresh)` on success, then read back + `$store->all($name, sinceDays: 371)` for the render window + (53 weeks × 7 days) — a `ContributionCollection` — and merge its + contributions into the returned array (the store becomes the source + of truth for what gets rendered, not the fresh fetch alone). - [ ] Call `$store->prune()` once per `aggregate()` call, after all providers have merged. - [ ] Keep the existing try/catch-and-log-per-provider behavior — a provider failure leaves its DB history stale, doesn't break the render. -- [ ] Update `ContributionAggregatorTest.php` with store-interaction and - prune-call cases. +- [ ] Update `ContributionAggregatorTest.php` with store-interaction + (`latestDate` consulted, `merge` called, `all()` feeds the result, + `prune()` runs once) and a case confirming a provider failure leaves + other providers' stored data intact. ## 6. Docker / env @@ -150,7 +185,46 @@ Order matters — later steps assume earlier ones are done. - [ ] `.env` — document `CONTRIBUTIONS_RETENTION_DAYS` (empty by default), same style as the existing `ALLOWED_HOSTS` comment. -## 7. PHPStan +## 9. `app:contributions:refetch` console command + +Manual escape hatch for forcing a full or ranged re-fetch (e.g. after +adding a new host, or if the store needs rebuilding) — bypasses step 5's +incremental `latestDate()` window entirely for the providers/range given. + +- [ ] `src/Command/RefetchContributionsCommand.php`, `#[AsCommand]` + + constructor-promoted readonly properties (matches this codebase's + attribute-first style), using `SymfonyStyle` + ([console styling guide](https://symfony.com/doc/current/console/style.html)). + Inject `iterable $providers` (`#[AutowireIterator('app.provider')]`, + same as `ContributionAggregator`) and `ContributionStore`. +- [ ] Options: `--provider=github,gitlab` (repeatable/comma-split, + restricts to named provider(s), default = all configured; unknown name + → `$io->error()` + `Command::FAILURE`), `--from=YYYY-MM-DD` (default + today − 365 days), `--to=YYYY-MM-DD` (default today), `--all` (shorthand + for `--from` far enough back — e.g. 2005-01-01 — to mean "existing full + history"; combinable with `--provider`). +- [ ] **Batch by range**: split `[from, to]` into ≤365-day chunks + (GitHub's GraphQL `contributionsCollection` rejects windows over a + year; chunking also caps GitLab's pagination per call, so a big + `--all` re-fetch can't reintroduce the original runaway-pagination + timeout). Iterate chunks oldest-first. +- [ ] Per provider, per chunk: `$io->section(...)`, `startFetch($chunkFrom, $chunkTo)` + / `resolveFetch()`, `$store->merge($name, $result)` immediately (don't + accumulate all chunks in memory), `ProgressBar` across chunks. +- [ ] Catch per-provider/per-chunk `\Throwable` → `$io->warning(...)`, + continue with remaining chunks/providers (same graceful-degradation + philosophy as `ContributionAggregator`). +- [ ] Finish with `$io->table(...)` summary (provider, days written, + chunks fetched, any errors) and `$io->success()`/`$io->error()`; exit + `Command::SUCCESS` if at least one provider fully succeeded, else + `Command::FAILURE`. +- [ ] Tests: `tests/Unit/Command/RefetchContributionsCommandTest.php` + using `CommandTester` with fake `ProviderInterface` stubs and a + `:memory:` `ContributionStore` — assert chunking count for a >365-day + range, store ends up populated, unknown `--provider` name fails + cleanly, a provider throwing doesn't abort the others. + +## 10. PHPStan - [ ] `composer require --dev phpstan/phpstan` (plain PHPStan — no Symfony extension needed for this app's size). @@ -158,12 +232,13 @@ Order matters — later steps assume earlier ones are done. - [ ] Add composer script `"phpstan": "phpstan analyse"`. - [ ] Fix whatever level-8 flags on first run. -## 8. `CLAUDE.md` update +## 11. `CLAUDE.md` update - [ ] Architecture section: add the `ContributionStore` (SQLite) tier between providers and the renderer; note the two-tier cache (1h SVG cache → SQLite raw-data store → provider APIs). - [ ] Environment variables table: add `CONTRIBUTIONS_RETENTION_DAYS`. +- [ ] Document `app:contributions:refetch` (options, batching behavior). - [ ] Development section: add `composer phpstan` next to the existing `vendor/bin/phpunit` commands. - [ ] Remove any remaining mentions of the two deleted bundles. @@ -181,3 +256,11 @@ Order matters — later steps assume earlier ones are done. are gone after the next refresh (`sqlite3 var/data/contributions.db "SELECT date FROM contributions ORDER BY date LIMIT 1"`). - `curl localhost:8080/health` still reports all configured providers healthy. +- `docker compose exec graph bin/console app:contributions:refetch --all` + populates `var/data/contributions.db` from empty; re-run with no flags + and confirm logs show only the trailing-window re-fetch. +- `docker compose exec graph bin/console app:contributions:refetch --provider=github --from=2020-01-01 --to=2023-01-01` + exercises multi-chunk batching. +- Confirm the original crash scenario is gone: an account with heavy + GitLab history no longer triggers `MaxExecutionTimeError` on a cold + cache (bounded by the 3-day trailing window, not full history).