TODO.md

@@ -110,35 +110,70 @@ Order matters — later steps assume earlier ones are done.
 - [x] Tests: `tests/Unit/Entity/ContributionTest.php` and
   `ContributionCollectionTest.php` — construction, iteration, `count()`.
 
-## 4. Wire `$since` through providers
+## 4. Wire `$since`/`$until` through providers
 
-- [ ] `ProviderInterface::fetch()` → `fetch(?\DateTimeImmutable $since = null): array`.
+Root cause of the live `MaxExecutionTimeError` crash: every cache-miss
-- [ ] `GitHubProvider::fetch()` — use `$since ?? new \DateTimeImmutable('-365 days')`
+request re-fetches the full ~365-day history from every host, and
-  for the GraphQL `from` param.
+GitLab's pagination is unbounded and sequential. Bounding the fetch
-- [ ] `GitLabProvider::fetch()` — same, feeds the `after` query param
+window (step 5's 3-day trailing overlap) is what actually fixes it — a
-  (this is what actually shrinks the pagination loop).
+PHP `max_execution_time` fatal can't be caught by `try/catch` at all, so
-- [ ] `GiteaProvider::fetch()` — same, feeds the `$cutoff` timestamp filter.
+"catch it better" was never on the table.
-- [ ] Update `GitHubProviderTest.php` (drop `GraphQLApiClientRegistryInterface`
+
-  stub setup entirely per step 1), `GitLabProviderTest.php`,
+- [ ] `ProviderInterface::startFetch()` → `startFetch(?\DateTimeImmutable $since = null, ?\DateTimeImmutable $until = null): mixed`
-  `GiteaProviderTest.php` — add a case asserting a passed `$since` narrows
+  (interface already split into `startFetch`/`resolveFetch` for
-  the request window.
+  concurrency — this doc previously said `fetch()`, which predates that
+  split).
+- [ ] `GitHubProvider::startFetch()` — already builds explicit `from`/`to`
+  GraphQL args (`GitHubProvider.php:51-58`); swap the hardcoded
+  `-365 days`/`now` for `$since ?? -365 days` / `$until ?? now`.
+- [ ] `GitLabProvider::startFetch()` — add a `before` query param alongside
+  the existing `after` (`GitLabProvider.php:77-84`), fed by `$until`;
+  `$since` already flows into `after` (this is what actually shrinks the
+  pagination loop).
+- [ ] `GiteaProvider::resolveFetch()` — heatmap endpoint has no query
+  params (always returns full history); add an `$until` upper-bound
+  filter alongside the existing `$cutoff` lower bound.
+- [ ] Update `GitHubProviderTest.php`, `GitLabProviderTest.php`,
+  `GiteaProviderTest.php` — add cases asserting a passed `$since`/`$until`
+  narrows the request window/query params.
 
 ## 5. Wire `ContributionStore` into `ContributionAggregator`
 
+- [ ] Fix `ContributionStore` wiring in `config/services.yaml` first —
+  it's currently dead code (nothing calls it). The constructor default
+  `$dbPath` string (`%kernel.project_dir%/var/data/contributions.db`) is
+  a plain PHP default, not a resolved container parameter; bind it
+  explicitly (same pattern as the provider bindings):
+  ```yaml
+  App\Service\ContributionStore:
+      arguments:
+          $dbPath: '%kernel.project_dir%/var/data/contributions.db'
+          $retentionDays: '%env(int:CONTRIBUTIONS_RETENTION_DAYS)%'
+  ```
+  Add `env(CONTRIBUTIONS_RETENTION_DAYS): ''` to `parameters:` (empty →
+  casts to `0` → `prune()`'s existing `retentionDays === 0` check already
+  treats that as "keep forever"). Document the var in `.env`.
 - [ ] Inject `ContributionStore` into `ContributionAggregator`.
-- [ ] Per configured provider: `$since = $store->latestDate($name)` → if
+- [ ] Per configured provider: `$latest = $store->latestDate($name)` →
-  set, `(new \DateTimeImmutable($latest))->modify('-3 days')` (3-day
+  `$since = $latest !== null ? (new \DateTimeImmutable('@' . $latest))->modify('-3 days') : null`
-  overlap for late corrections), else `null` (first run).
+  (3-day overlap for late corrections — old stored days are immutable and
-- [ ] `$fresh = $provider->fetch($since)`, `$store->merge($name, $fresh)`,
+  never re-fetched, only this trailing window + anything new hits the
-  then read back `$store->all($name, sinceDays: 371)` for the render
+  network), else `null` (first run, provider's own default lookback).
-  window (53 weeks × 7 days) — a `ContributionCollection` — and merge
+  `$until = null` (always "up to now" on the normal request path).
-  its contributions into the returned array.
+- [ ] `startFetch($since, $until)` / `resolveFetch()` as today,
+  `$store->merge($name, $fresh)` on success, then read back
+  `$store->all($name, sinceDays: 371)` for the render window
+  (53 weeks × 7 days) — a `ContributionCollection` — and merge its
+  contributions into the returned array (the store becomes the source
+  of truth for what gets rendered, not the fresh fetch alone).
 - [ ] Call `$store->prune()` once per `aggregate()` call, after all
   providers have merged.
 - [ ] Keep the existing try/catch-and-log-per-provider behavior — a
   provider failure leaves its DB history stale, doesn't break the render.
-- [ ] Update `ContributionAggregatorTest.php` with store-interaction and
+- [ ] Update `ContributionAggregatorTest.php` with store-interaction
-  prune-call cases.
+  (`latestDate` consulted, `merge` called, `all()` feeds the result,
+  `prune()` runs once) and a case confirming a provider failure leaves
+  other providers' stored data intact.
 
 ## 6. Docker / env
 
@@ -150,7 +185,46 @@ Order matters — later steps assume earlier ones are done.
 - [ ] `.env` — document `CONTRIBUTIONS_RETENTION_DAYS` (empty by default),
   same style as the existing `ALLOWED_HOSTS` comment.
 
-## 7. PHPStan
+## 9. `app:contributions:refetch` console command
+
+Manual escape hatch for forcing a full or ranged re-fetch (e.g. after
+adding a new host, or if the store needs rebuilding) — bypasses step 5's
+incremental `latestDate()` window entirely for the providers/range given.
+
+- [ ] `src/Command/RefetchContributionsCommand.php`, `#[AsCommand]` +
+  constructor-promoted readonly properties (matches this codebase's
+  attribute-first style), using `SymfonyStyle`
+  ([console styling guide](https://symfony.com/doc/current/console/style.html)).
+  Inject `iterable $providers` (`#[AutowireIterator('app.provider')]`,
+  same as `ContributionAggregator`) and `ContributionStore`.
+- [ ] Options: `--provider=github,gitlab` (repeatable/comma-split,
+  restricts to named provider(s), default = all configured; unknown name
+  → `$io->error()` + `Command::FAILURE`), `--from=YYYY-MM-DD` (default
+  today − 365 days), `--to=YYYY-MM-DD` (default today), `--all` (shorthand
+  for `--from` far enough back — e.g. 2005-01-01 — to mean "existing full
+  history"; combinable with `--provider`).
+- [ ] **Batch by range**: split `[from, to]` into ≤365-day chunks
+  (GitHub's GraphQL `contributionsCollection` rejects windows over a
+  year; chunking also caps GitLab's pagination per call, so a big
+  `--all` re-fetch can't reintroduce the original runaway-pagination
+  timeout). Iterate chunks oldest-first.
+- [ ] Per provider, per chunk: `$io->section(...)`, `startFetch($chunkFrom, $chunkTo)`
+  / `resolveFetch()`, `$store->merge($name, $result)` immediately (don't
+  accumulate all chunks in memory), `ProgressBar` across chunks.
+- [ ] Catch per-provider/per-chunk `\Throwable` → `$io->warning(...)`,
+  continue with remaining chunks/providers (same graceful-degradation
+  philosophy as `ContributionAggregator`).
+- [ ] Finish with `$io->table(...)` summary (provider, days written,
+  chunks fetched, any errors) and `$io->success()`/`$io->error()`; exit
+  `Command::SUCCESS` if at least one provider fully succeeded, else
+  `Command::FAILURE`.
+- [ ] Tests: `tests/Unit/Command/RefetchContributionsCommandTest.php`
+  using `CommandTester` with fake `ProviderInterface` stubs and a
+  `:memory:` `ContributionStore` — assert chunking count for a >365-day
+  range, store ends up populated, unknown `--provider` name fails
+  cleanly, a provider throwing doesn't abort the others.
+
+## 10. PHPStan
 
 - [ ] `composer require --dev phpstan/phpstan` (plain PHPStan — no
   Symfony extension needed for this app's size).
@@ -158,12 +232,13 @@ Order matters — later steps assume earlier ones are done.
 - [ ] Add composer script `"phpstan": "phpstan analyse"`.
 - [ ] Fix whatever level-8 flags on first run.
 
-## 8. `CLAUDE.md` update
+## 11. `CLAUDE.md` update
 
 - [ ] Architecture section: add the `ContributionStore` (SQLite) tier
   between providers and the renderer; note the two-tier cache (1h SVG
   cache → SQLite raw-data store → provider APIs).
 - [ ] Environment variables table: add `CONTRIBUTIONS_RETENTION_DAYS`.
+- [ ] Document `app:contributions:refetch` (options, batching behavior).
 - [ ] Development section: add `composer phpstan` next to the existing
   `vendor/bin/phpunit` commands.
 - [ ] Remove any remaining mentions of the two deleted bundles.
@@ -181,3 +256,11 @@ Order matters — later steps assume earlier ones are done.
   are gone after the next refresh
   (`sqlite3 var/data/contributions.db "SELECT date FROM contributions ORDER BY date LIMIT 1"`).
 - `curl localhost:8080/health` still reports all configured providers healthy.
+- `docker compose exec graph bin/console app:contributions:refetch --all`
+  populates `var/data/contributions.db` from empty; re-run with no flags
+  and confirm logs show only the trailing-window re-fetch.
+- `docker compose exec graph bin/console app:contributions:refetch --provider=github --from=2020-01-01 --to=2023-01-01`
+  exercises multi-chunk batching.
+- Confirm the original crash scenario is gone: an account with heavy
+  GitLab history no longer triggers `MaxExecutionTimeError` on a cold
+  cache (bounded by the 3-day trailing window, not full history).