perf(db): replace materialized views with maintained tables (cluster + sbom/vuln) #12
No reviewers
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Motstandskraft/spam!12
Loading…
Reference in a new issue
No description provided.
Delete branch "chore/remove-material-views"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Replaces the materialized-view families that were the dominant DB CPU cost
(wholesale
REFRESHre-computing the whole fleet on every ingest trigger,around the clock — pinning prod at ~95% node CPU) with regular tables that
are maintained cheaply.
cluster_summary / cluster_image_inventory
Incrementally maintained: each trigger recomputes only the clusters /
records changed since a
max(last_change_at)watermark and UPSERTs them.sbom/vuln family → tables fed by
_srcviewssbom_component_view,view_unified_repositories_vulnerabilities,view_unified_image_vulnerabilities) — extract-on-arrival: re-extractonly the entities whose source rows landed since a per-table watermark,
debounced from the existing SBOM/scan ingest hooks. Fresh as data lands,
cost proportional to what changed.
vuln_canonical_assets/summary,asset_risk) —daily
REBUILD_DERIVED_TABLESjob (03:00 Oslo) + manual "Rebuild now"via
/admin/views/refresh. They fold in KEV/EPSS feeds and cluster/exposure state that change without a scan, so per-entity extraction
doesn't fit them.
REFRESHpaths are neutralized;*Populatedgatesread
materialized_view_refreshes; cold-start populates once at boot.Migration cutover: type-guarded DROP +
CREATE TABLE AS ... WITH NO DATA(columns inferred from the body);
matviewsExist()now treats a table as"present" so converted views aren't re-asserted as matviews on later boots.
Verify before merge (SQL is untested; Go vets)
asset_risk+ canonicaltables land at their final definitions; no "relation does not exist"
loop on a second boot.
EXPLAINan extract-on-arrival re-INSERT for the two unified tables:the entity predicate must push into the source scans (one scan
re-extracts one repo/image, not the fleet).
sbom_component_viewisalready safe.
the per-finding surfaces while the rolled-up dashboard/triage update
on rebuild.
Note: freshness trade-off — the rolled-up vuln dashboard + triage scores
reflect the last rebuild (daily/manual), not the latest scan.
🤖 Generated with Claude Code
The image detail endpoint counted vuln_severity from raw findings in the latest image_scan_runs row, while /api/images/{id}/vulnerabilities counts canonical advisories from vuln_canonical_assets. The two disagreed: scanner variants (CVE/GHSA/BIT for one advisory) were collapsed in the list but not the chip, NEGLIGIBLE folded into Low vs Unknown, and the grain was latest-scan vs the MV's cross-scan snapshot. Add vulnmetrics.ImageSeverityCounts, which counts per-severity canonical advisories for one image from the same source LoadListPage reads, with the identical readiness tiering (canonical MV, unified-view bootstrap fallback, or empty when MVs aren't populated). The detail handler now uses it, so the chip totals always match the list — and both move together on an MV refresh instead of drifting apart. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Three more read paths counted vulnerabilities from live tables instead of the canonical MV, so their numbers disagreed with the vuln list and the image detail chip: - RepoImagesHandler (/api/repos/{repo_id}/images) - RepoWorkloadsHandler (/api/repos/{repo_id}/workloads) - AppSummaryHandler "Recent SBOMs" card (/api/app/summary) The image-side counts were COUNT(*) over image_vuln_findings with no scan_run_id filter — summing every historical scan run of a digest and never collapsing scanner variants, so a rescanned image was inflated several-fold. The app-summary repo side summed raw sbom_scan_results severity columns, double-counting an advisory seen by multiple scanners. All now count vuln_canonical_assets (one row per asset+canonical advisory), matching the list / detail / dashboard. Add vulnmetrics.CanonicalAssetsReady so these larger raw queries fall back to 0 during the pre-first-refresh window instead of failing on SQLSTATE 55000 against the unpopulated MV, and a shared imageVulnCountExpr helper for the two image-list subqueries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Per-image vulnerability views now count at finding grain — one row per (vuln_id, package, installed_version) — instead of canonical one-row-per-CVE. The canonical dedup collapsed a CVE affecting two packages (e.g. libcrypto3 + libssl3) into a single row, so SPAM read lower than `grype` run against the same image and eroded trust. It also caused a real search bug: the canonical row kept only one representative pkg_name, so searching the dropped package returned nothing — a false negative on a vuln that does affect that component. Changes (per-asset surfaces → per-finding; fleet dashboard stays canonical): - vulnmetrics.LoadImageVulnerabilities: new per-finding list for /api/images/{id}/vulnerabilities. Collapses scanner duplicates, keeps the scanner-reported vuln_id, enriches KEV/EPSS/year off the canonical id so GHSA-only findings still get their CVE's signals. Searches the real per-finding package, fixing the component-search false negative. - vulnmetrics.ImageSeverityCounts: count findings (vuln × package), same grain as the list, so the detail chip equals the list and grype. - RepoImagesHandler / RepoWorkloadsHandler vuln_count + AppSummary "Recent SBOMs" card: count findings from the unified views, matching the detail pages. - Add vulnmetrics.UnifiedViewsReady; drop now-unused CanonicalAssetsReady. The fleet dashboard (/api/vuln/list admin + summary) keeps canonical unique-CVE grain. Fixing its package search needs a searchable package column on the canonical MV — separate follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Per-finding vuln grain means a CVE can appear on multiple packages, so keying the vuln {#each} on vuln_id alone threw each_key_duplicate and froze the page on its loading spinner. Key rows on vuln_id+pkg+version; route expand state through the same key while caching advisory detail per-CVE. Also: drop the duplicate /api/auth/me fetch (layout did a raw fetch then loadSession) and the duplicate vulnerabilities fetch (effect depended on `image`, refiring when the profile resolved), and remove the eager per-row advisory prefetch that pulled large payloads on open — rows lazy-load detail on expand. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Replace the two divergent vuln surfaces with dedicated expandable tables (no shared generic — the views differ too much). Both drop the Signals column in favour of first-class KEV and EPSS columns. - ImageVulnTable: extracts the image page's per-package table; expand still lazy-loads advisory detail with the skeleton placeholder. - ClusterVulnTable: new CVE-grouped table for the cluster page; each row expands to the advisory text plus the affected *running* images in the cluster, lazily fetched per CVE. - Cluster Vulnerabilities tab pill no longer shows a count. Backend: add ClusterVulnImagesHandler (GET /api/cluster/{id}/vulnerabilities/{vuln_id}/images) returning the running images affected by one canonical CVE — digest, registry/repo, worst severity, fix availability/version, package count — ACL-gated and namespace-filtered like the sibling endpoints. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Converting cluster_summary / cluster_image_inventory from materialized views to tables left the old materialized_view_refreshes rows behind. The new ClusterSummaryViewPopulated / ClusterImageInventoryPopulated checks treat "a refresh row exists" as populated, so EnsureFirstPopulate saw the stale rows, skipped the cold-start full populate, and the tables filled only incrementally from the stale watermark — i.e. just the clusters that changed since the old MV's last refresh. Symptom: /clusters shows a subset that trickles in as clusters send changes. Delete the leftover rows in each conversion migration (mirrors what 20260624 does for the canonical tables) so the first refresh does a full -infinity populate of every cluster. On an already-converted prod, the one-time manual equivalent is: DELETE FROM materialized_view_refreshes WHERE name IN ('cluster_summary','cluster_image_inventory'); the next ingest then full-populates within seconds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>The endpoint returned only per-cluster rows, so any consumer that wanted a fleet total summed the rows — which over-counts images: the per-cluster images value is DISTINCT registry/image@digest within one cluster, and a shared base/sidecar image runs in many clusters, so the sum fans each shared image out by its cluster count (the 28k-vs-2k inflation). Response is now {"clusters": [...], "summary": {...}}. The summary rolls up the returned clusters: - images: COUNT(DISTINCT digest) over cluster_image_inventory, scoped to the same cluster-id set the rows came from (so ACL + liveness + search are already applied). Keyed on bare digest so the same image pulled via a mirror or under two names counts once — the true unique-image count. - containers / namespaces / ingress_count: sums of the per-cluster rows, which is correct since each is keyed on an entity unique to one cluster. Deny and cold-start paths emit the same shape with an empty cluster list and a zero summary, so consumers never special-case a bare array. SearchPalette (the only in-repo consumer) reads data.clusters with an array fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>The /clusters page was the real consumer of /api/clusters/summary (via loadUnfiltered/loadMain), not just the search palette. The previous commit changed the response from a bare array to { clusters, summary }, which broke the page: both loaders parsed it as `Array.isArray(body) ? body : []` and so got an empty cluster list. - Add parseClusterSummary() to read { clusters, summary } (tolerating a bare array from an older/cached backend), and use it in both loaders. - Drive the page-level "Images" card from summary.images (COUNT(DISTINCT digest), de-duplicated across clusters) instead of summing clustersAll.images across rows, which fanned each shared image out by its cluster count — the 28k-vs-2k inflation. The card's "Unique by digest" subtitle is now accurate. - Containers fall back to a row reduce if the summary is absent; that sum is already correct since containers are unique per cluster. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>gitleaks emits a bare `null` (not `[]`) when a repo has no findings. json.Unmarshal("null", &slice) succeeds with a nil slice and no error, so the raw bytes were stored as findings = 'null'::jsonb. COALESCE(findings, '[]') only catches SQL NULL, so that JSON null reached jsonb_array_elements and threw "cannot extract elements from a scalar" — which 500s the secrets dashboard endpoints AND fails asset_risk_src, rolling back the entire derived-tables rebuild (asset_risk is in the rebuild order). - runner: normalise the parsed findings slice (nil -> []) and re-marshal before persisting, so the column is always an array. Stops new bad rows at the source. - secrets dashboard: guard all four jsonb_array_elements sites with CASE WHEN jsonb_typeof(findings) = 'array' ... ELSE '[]', which also survives scalar/object values, and log the real error on the three foreground 500 paths (they were swallowing it). Existing malformed rows still need a one-time cleanup: UPDATE run_secrets SET findings = '[]'::jsonb, finding_count = 0 WHERE findings IS NOT NULL AND jsonb_typeof(findings) <> 'array'; Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>