Save to database #13
@@ -118,27 +118,27 @@ window (step 5's 3-day trailing overlap) is what actually fixes it — a
|
|||||||
PHP `max_execution_time` fatal can't be caught by `try/catch` at all, so
|
PHP `max_execution_time` fatal can't be caught by `try/catch` at all, so
|
||||||
"catch it better" was never on the table.
|
"catch it better" was never on the table.
|
||||||
|
|
||||||
- [ ] `ProviderInterface::startFetch()` → `startFetch(?\DateTimeImmutable $since = null, ?\DateTimeImmutable $until = null): mixed`
|
- [x] `ProviderInterface::startFetch()` → `startFetch(?\DateTimeImmutable $since = null, ?\DateTimeImmutable $until = null): mixed`
|
||||||
(interface already split into `startFetch`/`resolveFetch` for
|
(interface already split into `startFetch`/`resolveFetch` for
|
||||||
concurrency — this doc previously said `fetch()`, which predates that
|
concurrency — this doc previously said `fetch()`, which predates that
|
||||||
split).
|
split).
|
||||||
- [ ] `GitHubProvider::startFetch()` — already builds explicit `from`/`to`
|
- [x] `GitHubProvider::startFetch()` — already builds explicit `from`/`to`
|
||||||
GraphQL args (`GitHubProvider.php:51-58`); swap the hardcoded
|
GraphQL args (`GitHubProvider.php:51-58`); swap the hardcoded
|
||||||
`-365 days`/`now` for `$since ?? -365 days` / `$until ?? now`.
|
`-365 days`/`now` for `$since ?? -365 days` / `$until ?? now`.
|
||||||
- [ ] `GitLabProvider::startFetch()` — add a `before` query param alongside
|
- [x] `GitLabProvider::startFetch()` — add a `before` query param alongside
|
||||||
the existing `after` (`GitLabProvider.php:77-84`), fed by `$until`;
|
the existing `after` (`GitLabProvider.php:77-84`), fed by `$until`;
|
||||||
`$since` already flows into `after` (this is what actually shrinks the
|
`$since` already flows into `after` (this is what actually shrinks the
|
||||||
pagination loop).
|
pagination loop).
|
||||||
- [ ] `GiteaProvider::resolveFetch()` — heatmap endpoint has no query
|
- [x] `GiteaProvider::resolveFetch()` — heatmap endpoint has no query
|
||||||
params (always returns full history); add an `$until` upper-bound
|
params (always returns full history); add an `$until` upper-bound
|
||||||
filter alongside the existing `$cutoff` lower bound.
|
filter alongside the existing `$cutoff` lower bound.
|
||||||
- [ ] Update `GitHubProviderTest.php`, `GitLabProviderTest.php`,
|
- [x] Update `GitHubProviderTest.php`, `GitLabProviderTest.php`,
|
||||||
`GiteaProviderTest.php` — add cases asserting a passed `$since`/`$until`
|
`GiteaProviderTest.php` — add cases asserting a passed `$since`/`$until`
|
||||||
narrows the request window/query params.
|
narrows the request window/query params.
|
||||||
|
|
||||||
## 5. Wire `ContributionStore` into `ContributionAggregator`
|
## 5. Wire `ContributionStore` into `ContributionAggregator`
|
||||||
|
|
||||||
- [ ] Fix `ContributionStore` wiring in `config/services.yaml` first —
|
- [x] Fix `ContributionStore` wiring in `config/services.yaml` first —
|
||||||
it's currently dead code (nothing calls it). The constructor default
|
it's currently dead code (nothing calls it). The constructor default
|
||||||
`$dbPath` string (`%kernel.project_dir%/var/data/contributions.db`) is
|
`$dbPath` string (`%kernel.project_dir%/var/data/contributions.db`) is
|
||||||
a plain PHP default, not a resolved container parameter; bind it
|
a plain PHP default, not a resolved container parameter; bind it
|
||||||
@@ -152,36 +152,36 @@ PHP `max_execution_time` fatal can't be caught by `try/catch` at all, so
|
|||||||
Add `env(CONTRIBUTIONS_RETENTION_DAYS): ''` to `parameters:` (empty →
|
Add `env(CONTRIBUTIONS_RETENTION_DAYS): ''` to `parameters:` (empty →
|
||||||
casts to `0` → `prune()`'s existing `retentionDays === 0` check already
|
casts to `0` → `prune()`'s existing `retentionDays === 0` check already
|
||||||
treats that as "keep forever"). Document the var in `.env`.
|
treats that as "keep forever"). Document the var in `.env`.
|
||||||
- [ ] Inject `ContributionStore` into `ContributionAggregator`.
|
- [x] Inject `ContributionStore` into `ContributionAggregator`.
|
||||||
- [ ] Per configured provider: `$latest = $store->latestDate($name)` →
|
- [x] Per configured provider: `$latest = $store->latestDate($name)` →
|
||||||
`$since = $latest !== null ? (new \DateTimeImmutable('@' . $latest))->modify('-3 days') : null`
|
`$since = $latest !== null ? (new \DateTimeImmutable('@' . $latest))->modify('-3 days') : null`
|
||||||
(3-day overlap for late corrections — old stored days are immutable and
|
(3-day overlap for late corrections — old stored days are immutable and
|
||||||
never re-fetched, only this trailing window + anything new hits the
|
never re-fetched, only this trailing window + anything new hits the
|
||||||
network), else `null` (first run, provider's own default lookback).
|
network), else `null` (first run, provider's own default lookback).
|
||||||
`$until = null` (always "up to now" on the normal request path).
|
`$until = null` (always "up to now" on the normal request path).
|
||||||
- [ ] `startFetch($since, $until)` / `resolveFetch()` as today,
|
- [x] `startFetch($since, $until)` / `resolveFetch()` as today,
|
||||||
`$store->merge($name, $fresh)` on success, then read back
|
`$store->merge($name, $fresh)` on success, then read back
|
||||||
`$store->all($name, sinceDays: 371)` for the render window
|
`$store->all($name, sinceDays: 371)` for the render window
|
||||||
(53 weeks × 7 days) — a `ContributionCollection` — and merge its
|
(53 weeks × 7 days) — a `ContributionCollection` — and merge its
|
||||||
contributions into the returned array (the store becomes the source
|
contributions into the returned array (the store becomes the source
|
||||||
of truth for what gets rendered, not the fresh fetch alone).
|
of truth for what gets rendered, not the fresh fetch alone).
|
||||||
- [ ] Call `$store->prune()` once per `aggregate()` call, after all
|
- [x] Call `$store->prune()` once per `aggregate()` call, after all
|
||||||
providers have merged.
|
providers have merged.
|
||||||
- [ ] Keep the existing try/catch-and-log-per-provider behavior — a
|
- [x] Keep the existing try/catch-and-log-per-provider behavior — a
|
||||||
provider failure leaves its DB history stale, doesn't break the render.
|
provider failure leaves its DB history stale, doesn't break the render.
|
||||||
- [ ] Update `ContributionAggregatorTest.php` with store-interaction
|
- [x] Update `ContributionAggregatorTest.php` with store-interaction
|
||||||
(`latestDate` consulted, `merge` called, `all()` feeds the result,
|
(`latestDate` consulted, `merge` called, `all()` feeds the result,
|
||||||
`prune()` runs once) and a case confirming a provider failure leaves
|
`prune()` runs once) and a case confirming a provider failure leaves
|
||||||
other providers' stored data intact.
|
other providers' stored data intact.
|
||||||
|
|
||||||
## 6. Docker / env
|
## 6. Docker / env
|
||||||
|
|
||||||
- [ ] `docker-compose.yml` — add a `data` named volume mounted at
|
- [x] `docker-compose.yml` — add a `data` named volume mounted at
|
||||||
`/app/var/data` (same pattern as `cache`/`logs`), and pass through
|
`/app/var/data` (same pattern as `cache`/`logs`), and pass through
|
||||||
`CONTRIBUTIONS_RETENTION_DAYS: "${CONTRIBUTIONS_RETENTION_DAYS:-}"`.
|
`CONTRIBUTIONS_RETENTION_DAYS: "${CONTRIBUTIONS_RETENTION_DAYS:-}"`.
|
||||||
- [ ] `Dockerfile` — add `var/data` to the `mkdir -p` in the `final` stage
|
- [x] `Dockerfile` — add `var/data` to the `mkdir -p` in the `final` stage
|
||||||
alongside `var/cache/prod/pools var/log`, owned by `app`.
|
alongside `var/cache/prod/pools var/log`, owned by `app`.
|
||||||
- [ ] `.env` — document `CONTRIBUTIONS_RETENTION_DAYS` (empty by default),
|
- [x] `.env` — document `CONTRIBUTIONS_RETENTION_DAYS` (empty by default),
|
||||||
same style as the existing `ALLOWED_HOSTS` comment.
|
same style as the existing `ALLOWED_HOSTS` comment.
|
||||||
|
|
||||||
## 9. `graph:contributions:refetch` console command
|
## 9. `graph:contributions:refetch` console command
|
||||||
@@ -190,34 +190,34 @@ Manual escape hatch for forcing a full or ranged re-fetch (e.g. after
|
|||||||
adding a new host, or if the store needs rebuilding) — bypasses step 5's
|
adding a new host, or if the store needs rebuilding) — bypasses step 5's
|
||||||
incremental `latestDate()` window entirely for the providers/range given.
|
incremental `latestDate()` window entirely for the providers/range given.
|
||||||
|
|
||||||
- [ ] `src/Command/RefetchContributionsCommand.php`, `#[AsCommand]` +
|
- [x] `src/Command/RefetchContributionsCommand.php`, `#[AsCommand]` +
|
||||||
constructor-promoted readonly properties (matches this codebase's
|
constructor-promoted readonly properties (matches this codebase's
|
||||||
attribute-first style), using `SymfonyStyle`
|
attribute-first style), using `SymfonyStyle`
|
||||||
([console styling guide](https://symfony.com/doc/current/console/style.html)).
|
([console styling guide](https://symfony.com/doc/current/console/style.html)).
|
||||||
Inject `iterable $providers` (`#[AutowireIterator('app.provider')]`,
|
Inject `iterable $providers` (`#[AutowireIterator('app.provider')]`,
|
||||||
same as `ContributionAggregator`) and `ContributionStore`.
|
same as `ContributionAggregator`) and `ContributionStore`.
|
||||||
- [ ] Options: `--provider=github,gitlab` (repeatable/comma-split,
|
- [x] Options: `--provider=github,gitlab` (repeatable/comma-split,
|
||||||
restricts to named provider(s), default = all configured; unknown name
|
restricts to named provider(s), default = all configured; unknown name
|
||||||
→ `$io->error()` + `Command::FAILURE`), `--from=YYYY-MM-DD` (default
|
→ `$io->error()` + `Command::FAILURE`), `--from=YYYY-MM-DD` (default
|
||||||
today − 365 days), `--to=YYYY-MM-DD` (default today), `--all` (shorthand
|
today − 365 days), `--to=YYYY-MM-DD` (default today), `--all` (shorthand
|
||||||
for `--from` far enough back — e.g. 2005-01-01 — to mean "existing full
|
for `--from` far enough back — e.g. 2005-01-01 — to mean "existing full
|
||||||
history"; combinable with `--provider`).
|
history"; combinable with `--provider`).
|
||||||
- [ ] **Batch by range**: split `[from, to]` into ≤365-day chunks
|
- [x] **Batch by range**: split `[from, to]` into ≤365-day chunks
|
||||||
(GitHub's GraphQL `contributionsCollection` rejects windows over a
|
(GitHub's GraphQL `contributionsCollection` rejects windows over a
|
||||||
year; chunking also caps GitLab's pagination per call, so a big
|
year; chunking also caps GitLab's pagination per call, so a big
|
||||||
`--all` re-fetch can't reintroduce the original runaway-pagination
|
`--all` re-fetch can't reintroduce the original runaway-pagination
|
||||||
timeout). Iterate chunks oldest-first.
|
timeout). Iterate chunks oldest-first.
|
||||||
- [ ] Per provider, per chunk: `$io->section(...)`, `startFetch($chunkFrom, $chunkTo)`
|
- [x] Per provider, per chunk: `$io->section(...)`, `startFetch($chunkFrom, $chunkTo)`
|
||||||
/ `resolveFetch()`, `$store->merge($name, $result)` immediately (don't
|
/ `resolveFetch()`, `$store->merge($name, $result)` immediately (don't
|
||||||
accumulate all chunks in memory), `ProgressBar` across chunks.
|
accumulate all chunks in memory), `ProgressBar` across chunks.
|
||||||
- [ ] Catch per-provider/per-chunk `\Throwable` → `$io->warning(...)`,
|
- [x] Catch per-provider/per-chunk `\Throwable` → `$io->warning(...)`,
|
||||||
continue with remaining chunks/providers (same graceful-degradation
|
continue with remaining chunks/providers (same graceful-degradation
|
||||||
philosophy as `ContributionAggregator`).
|
philosophy as `ContributionAggregator`).
|
||||||
- [ ] Finish with `$io->table(...)` summary (provider, days written,
|
- [x] Finish with `$io->table(...)` summary (provider, days written,
|
||||||
chunks fetched, any errors) and `$io->success()`/`$io->error()`; exit
|
chunks fetched, any errors) and `$io->success()`/`$io->error()`; exit
|
||||||
`Command::SUCCESS` if at least one provider fully succeeded, else
|
`Command::SUCCESS` if at least one provider fully succeeded, else
|
||||||
`Command::FAILURE`.
|
`Command::FAILURE`.
|
||||||
- [ ] Tests: `tests/Unit/Command/RefetchContributionsCommandTest.php`
|
- [x] Tests: `tests/Unit/Command/RefetchContributionsCommandTest.php`
|
||||||
using `CommandTester` with fake `ProviderInterface` stubs and a
|
using `CommandTester` with fake `ProviderInterface` stubs and a
|
||||||
`:memory:` `ContributionStore` — assert chunking count for a >365-day
|
`:memory:` `ContributionStore` — assert chunking count for a >365-day
|
||||||
range, store ends up populated, unknown `--provider` name fails
|
range, store ends up populated, unknown `--provider` name fails
|
||||||
@@ -228,20 +228,21 @@ incremental `latestDate()` window entirely for the providers/range given.
|
|||||||
- [x] `composer require --dev phpstan/phpstan` (plain PHPStan — no
|
- [x] `composer require --dev phpstan/phpstan` (plain PHPStan — no
|
||||||
Symfony extension needed for this app's size).
|
Symfony extension needed for this app's size).
|
||||||
- [x] Add `phpstan.neon`: `paths: [src, tests]`, `level: 8`.
|
- [x] Add `phpstan.neon`: `paths: [src, tests]`, `level: 8`.
|
||||||
- [ ] Add composer script `"phpstan": "phpstan analyse"`.
|
- [x] Add composer script `"phpstan": "phpstan analyse"`.
|
||||||
- [ ] Update README.md docs with new command and php stan static lintin.
|
- [x] Update README.md docs with new command and php stan static lintin.
|
||||||
- [ ] Document in PHP-Stan-Errors.md whatever level-8 flags on first run.
|
- [x] Document in PHP-Stan-Errors.md whatever level-8 flags on first run.
|
||||||
|
|
||||||
## 11. `CLAUDE.md` update
|
## 11. `CLAUDE.md` update
|
||||||
|
|
||||||
- [ ] Architecture section: add the `ContributionStore` (SQLite) tier
|
- [x] Architecture section: add the `ContributionStore` (SQLite) tier
|
||||||
between providers and the renderer; note the two-tier cache (1h SVG
|
between providers and the renderer; note the two-tier cache (1h SVG
|
||||||
cache → SQLite raw-data store → provider APIs).
|
cache → SQLite raw-data store → provider APIs).
|
||||||
- [ ] Environment variables table: add `CONTRIBUTIONS_RETENTION_DAYS`.
|
- [x] Environment variables table: add `CONTRIBUTIONS_RETENTION_DAYS`.
|
||||||
- [ ] Document `app:contributions:refetch` (options, batching behavior).
|
- [x] Document `app:contributions:refetch` (options, batching behavior).
|
||||||
- [ ] Development section: add `composer phpstan` next to the existing
|
- [x] Development section: add `composer phpstan` next to the existing
|
||||||
`vendor/bin/phpunit` commands.
|
`vendor/bin/phpunit` commands.
|
||||||
- [ ] Remove any remaining mentions of the two deleted bundles.
|
- [x] Remove any remaining mentions of the two deleted bundles. (none found —
|
||||||
|
already clean)
|
||||||
|
|
||||||
## Verification
|
## Verification
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user