fix(vuln): drop vestigial UNIQUE on unified-vuln tables to prevent refresh freeze #51

Merged

jonas merged 1 commit from fix/unified-vuln-indexes-nonunique into main

2026-07-08 22:53:36 +02:00

jonas commented

2026-07-08 22:23:09 +02:00

Owner

Problem

Worker logs show duplicate-key failures identical in shape to the cluster_summary freeze:

view_unified_repositories_vulnerabilities: duplicate key value violates unique constraint "idx_view_unified_repos_vulns_unique"
view_unified_image_vulnerabilities:        duplicate key value violates unique constraint "idx_view_unified_image_vulns_unique"

Why it freezes

Both objects are plain tables (converted from matviews in 20260430), maintained by DELETE + INSERT … SELECT (RebuildDerivedTables / ExtractDerivedOnArrival in internal/db/derived_tables.go) — no ON CONFLICT anywhere. Their UNIQUE indexes were only ever required by the old REFRESH MATERIALIZED VIEW CONCURRENTLY, which no longer runs, so they're vestigial.

They are not harmless, though: any duplicate row the feeding SELECT emits raises a 23505 that aborts the whole INSERT, the extract errors, recordMaterializedViewRefresh is skipped, the <table>:wm watermark never advances, and the derived vuln tables freeze fleet-wide until a manual rebuild. This is the same abort-and-freeze class as the cluster_summary cluster_id collision (PR #49 / 20260710).

Note: on the current branch the feeding SELECT DISTINCT ON (…) keys are already aligned with these indexes, so it can't emit a colliding row — which means if prod is currently throwing these errors, prod is likely still on the older view/matview definitions where the full-key dedup was absent, and simply shipping the derived-tables branch already resolves the immediate errors. This PR removes the underlying fragility so a future duplicate (new scanner source, a DISTINCT-ON regression, a manual backfill) can never re-freeze the family.

Fix

Recreate both indexes as plain (non-unique). Safe because every read path re-aggregates these tables down to canonical_id (internal/vulnmetrics/metrics.go, the vuln_canonical_* / asset_risk views), so end-result counts are unchanged (the 20260430 conversion comment says exactly this). The plain index preserves the (repo_id|image_id, vuln_id, …) lookup/filter performance, and the DISTINCT ON in the *_scoped functions remains as belt-and-suspenders dedup.

Verification

go vet ./cmd/server/... — clean.
Migration is DROP INDEX IF EXISTS + CREATE INDEX IF NOT EXISTS, safe to replay.

PR #49 — the same fix for cluster_summary.
A separate correctness bug in the image incremental path (cross-scanner finding deletion) is fixed in its own PR.

🤖 Generated with Claude Code

## Problem Worker logs show duplicate-key failures identical in *shape* to the cluster_summary freeze: ``` view_unified_repositories_vulnerabilities: duplicate key value violates unique constraint "idx_view_unified_repos_vulns_unique" view_unified_image_vulnerabilities: duplicate key value violates unique constraint "idx_view_unified_image_vulns_unique" ``` ## Why it freezes Both objects are plain **tables** (converted from matviews in `20260430`), maintained by `DELETE + INSERT … SELECT` (`RebuildDerivedTables` / `ExtractDerivedOnArrival` in `internal/db/derived_tables.go`) — **no `ON CONFLICT` anywhere**. Their `UNIQUE` indexes were only ever required by the old `REFRESH MATERIALIZED VIEW CONCURRENTLY`, which no longer runs, so they're **vestigial**. They are not harmless, though: any duplicate row the feeding `SELECT` emits raises a `23505` that aborts the whole `INSERT`, the extract errors, `recordMaterializedViewRefresh` is skipped, the `<table>:wm` watermark never advances, and the derived vuln tables **freeze fleet-wide** until a manual rebuild. This is the same abort-and-freeze class as the cluster_summary `cluster_id` collision (PR #49 / `20260710`). > Note: on the current branch the feeding `SELECT DISTINCT ON (…)` keys are already aligned with these indexes, so it can't emit a colliding row — which means if prod is *currently* throwing these errors, prod is likely still on the older view/matview definitions where the full-key dedup was absent, and simply shipping the derived-tables branch already resolves the immediate errors. This PR removes the underlying fragility so a future duplicate (new scanner source, a DISTINCT-ON regression, a manual backfill) can never re-freeze the family. ## Fix Recreate both indexes as **plain** (non-unique). Safe because every read path re-aggregates these tables down to `canonical_id` (`internal/vulnmetrics/metrics.go`, the `vuln_canonical_*` / `asset_risk` views), so end-result counts are unchanged (the `20260430` conversion comment says exactly this). The plain index preserves the `(repo_id|image_id, vuln_id, …)` lookup/filter performance, and the `DISTINCT ON` in the `*_scoped` functions remains as belt-and-suspenders dedup. ## Verification - `go vet ./cmd/server/...` — clean. - Migration is `DROP INDEX IF EXISTS` + `CREATE INDEX IF NOT EXISTS`, safe to replay. ## Related - PR #49 — the same fix for cluster_summary. - A separate correctness bug in the *image* incremental path (cross-scanner finding deletion) is fixed in its own PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jonas added 1 commit

2026-07-08 22:23:11 +02:00

fix(vuln): drop vestigial UNIQUE on unified-vuln tables to prevent refresh freeze

Build and Deploy / build-sbom-scanner (push) Successful in 1m43s

Details

Build and Deploy / build-repo-runner (push) Successful in 1m49s

Details

Build and Deploy / build-image-scanner (push) Successful in 1m47s

Details

Build and Deploy / build-app (push) Successful in 5m2s

Details

Build and Deploy / deploy-helm (push) Successful in 26s

Details

addee49f18

view_unified_repositories_vulnerabilities and view_unified_image_vulnerabilities
are plain tables (since 20260430) maintained by DELETE + INSERT … SELECT with no
ON CONFLICT. Their UNIQUE indexes were only ever needed by the old REFRESH
MATERIALIZED VIEW CONCURRENTLY and are now vestigial — but not harmless: any
duplicate row the feeding SELECT emits raises a 23505 that aborts the whole
INSERT, so the extract errors, the watermark never advances, and the derived
vuln tables freeze fleet-wide until a manual rebuild. Same abort-and-freeze
class as the cluster_summary cluster_id collision (20260710).

Recreate both indexes as plain (non-unique). Safe because every read path
re-aggregates to canonical_id, so counts are unchanged, and the DISTINCT ON in
the *_scoped functions stays as the actual dedup. Removes the freeze fragility
while preserving lookup performance.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

jonas referenced this pull request

2026-07-08 22:26:26 +02:00

fix(vuln): stop image-vuln extract dropping other scanners' findings #52

jonas referenced this pull request from a commit

2026-07-08 22:42:04 +02:00

fix(vuln): stop image-vuln extract dropping other scanners' findings (#52)

jonas force-pushed fix/unified-vuln-indexes-nonunique from addee49f18

Build and Deploy / build-sbom-scanner (push) Successful in 1m43s

Details

Build and Deploy / build-repo-runner (push) Successful in 1m49s

Details

Build and Deploy / build-image-scanner (push) Successful in 1m47s