fix(triage): stop cluster-summary refresh freezing on cutover cluster_id collision #49

Merged

jonas merged 1 commit from fix/cluster-summary-refresh-freeze into main 2026-07-08 22:19:59 +02:00

Conversation 0 Commits 1 Files changed 4 +97 -13

jonas commented 2026-07-08 21:57:04 +02:00

Owner

Copy link

Problem

A live agent pushes snapshots fine and lands in cluster_record, but is invisible in /api/clusters/summary — along with every other cluster registered after a certain date. The rollup tables cluster_summary and cluster_image_inventory had been stale for weeks.

Root cause

The incremental refresh (clusterSummaryUpsertSQL) upserts with ON CONFLICT (cluster_key), where cluster_key prefers the ROR slug. But cluster_summary also carried a separate UNIQUE index on cluster_id.

During an ROR identity cutover, one physical cluster can transiently produce two cluster_key rows — the pre-cutover kube-UID key and the post-cutover slug key — that resolve to the same derived cluster_id (the kube-system UID). Postgres allows only one conflict arbiter per statement, so the second row raises 23505 on idx_cluster_summary_cluster_id, which ON CONFLICT (cluster_key) cannot catch.

The whole INSERT ... SELECT aborts → the refresh returns an error → recordMaterializedViewRefresh never runs → the watermark never advances → both rollup tables freeze fleet-wide until manually truncated. A single colliding cluster starves every cluster's rollup, and it fails silently (error logged, but no alert).

Fix

• Drop the UNIQUE constraint on cluster_summary.cluster_id, keeping it as a plain index (migration 20260710). The refresh can no longer abort on a cluster_id collision, so a cutover can never freeze the rollups again.
• Dedup at read time in ClusterSummaryHandler with DISTINCT ON (cluster_id), preferring the ROR-slug-bearing (post-cutover) row, then most-recent activity. The transient double-row collapses to one, and per-fleet totals count each cluster once.
• Staleness WARN: log when a refresh runs against long-stale rollups (age > 15m vs the ~2m interval), so a future stalled watermark surfaces instead of hiding.

Why not "TRUNCATE before INSERT" / "DELETE + upsert"

Those don't fix it: the SELECT still emits two rows with the same cluster_id in one statement, so the cluster_id-index collision recurs regardless of pre-existing table state. Removing the failure mode (non-unique cluster_id + read-side dedup) is the robust fix. A manual TRUNCATE only resets it temporarily — the next cutover would re-freeze it.

Verification

• go vet ./internal/db/... ./internal/scam/... ./cmd/server/... — clean.
• Migration is a plain index swap, DROP INDEX IF EXISTS + CREATE INDEX IF NOT EXISTS, safe to replay.

🤖 Generated with Claude Code

## Problem A live agent pushes snapshots fine and lands in `cluster_record`, but is **invisible** in `/api/clusters/summary` — along with every other cluster registered after a certain date. The rollup tables `cluster_summary` and `cluster_image_inventory` had been stale for weeks. ## Root cause The incremental refresh (`clusterSummaryUpsertSQL`) upserts with `ON CONFLICT (cluster_key)`, where `cluster_key` prefers the ROR slug. But `cluster_summary` also carried a **separate `UNIQUE` index on `cluster_id`**. During an ROR identity cutover, one physical cluster can transiently produce **two `cluster_key` rows** — the pre-cutover kube-UID key and the post-cutover slug key — that resolve to the **same derived `cluster_id`** (the kube-system UID). Postgres allows only one conflict arbiter per statement, so the second row raises `23505` on `idx_cluster_summary_cluster_id`, which `ON CONFLICT (cluster_key)` cannot catch. The whole `INSERT ... SELECT` aborts → the refresh returns an error → `recordMaterializedViewRefresh` never runs → the watermark never advances → **both rollup tables freeze fleet-wide** until manually truncated. A single colliding cluster starves every cluster's rollup, and it fails silently (error logged, but no alert). ## Fix - **Drop the `UNIQUE` constraint** on `cluster_summary.cluster_id`, keeping it as a plain index (migration `20260710`). The refresh can no longer abort on a `cluster_id` collision, so a cutover can never freeze the rollups again. - **Dedup at read time** in `ClusterSummaryHandler` with `DISTINCT ON (cluster_id)`, preferring the ROR-slug-bearing (post-cutover) row, then most-recent activity. The transient double-row collapses to one, and per-fleet totals count each cluster once. - **Staleness WARN**: log when a refresh runs against long-stale rollups (age > 15m vs the ~2m interval), so a future stalled watermark surfaces instead of hiding. ## Why not "TRUNCATE before INSERT" / "DELETE + upsert" Those don't fix it: the `SELECT` still emits two rows with the same `cluster_id` in one statement, so the `cluster_id`-index collision recurs regardless of pre-existing table state. Removing the failure mode (non-unique `cluster_id` + read-side dedup) is the robust fix. A manual TRUNCATE only resets it temporarily — the next cutover would re-freeze it. ## Verification - `go vet ./internal/db/... ./internal/scam/... ./cmd/server/...` — clean. - Migration is a plain index swap, `DROP INDEX IF EXISTS` + `CREATE INDEX IF NOT EXISTS`, safe to replay. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jonas added 3 commits 2026-07-08 21:57:05 +02:00

feat(triage): cluster-scoped postpone/suppress for attack paths (#22) ... 57d5ff568b

Adds operational triage decisions — postpone (time-boxed) and suppress
(permanent) — attached to an asset's attack path for a set of clusters.

- attackpath package: a banded weakness snapshot (three likelihood bands)
  built from pluggable contributors (CVEs + secrets today; archived/
  abandoned deps later) with escalation-only resurface — EPSS jitter
  within a band and off-path churn never break a decision
- cluster-scoped ACL: never suppress a cluster you cannot read; all or a
  subset of your accessible clusters; team-shared and actor-attributed
- read-time dashboard filtering: fully-suppressed paths move to a new
  Suppressed section, partial ones stay in-tier with a badge
- suppress/revoke/list API (audited) + migration with a partial unique
  index (one active decision per asset+cluster, soft-revoked for history)
- frontend suppress dialog, inline history with revoke, header badges

Coexists with component_vex and secret_dismissals; does not replace them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(triage): extend attack-path suppression to repos + hide parked items ... fcdc43a4d8

- Repos are now suppressible (secrets and/or vulns), alongside images. The
  weakness contributors branch on asset type; repos have no cluster
  dimension, so a repo decision is a single asset-wide row.
- Repo authorization flows through one seam (canSuppressRepo) so a future
  repo ACL — write-grants, scope/prefix, external RBAC — slots in without
  touching the handlers.
- Suppressed/acknowledged paths are now off the to-do: fully-suppressed
  rows leave the tiers and counts and move into a collapsed
  "Postponed / Suppressed" disclosure (expand to review, Revoke to bring
  back); partial ones stay in-tier with an N/M badge.
- Harden the triage page against a null/partial wire shape (normalize the
  response so no tier array is ever dereferenced null).
- Add an opt-in dev demo seed for attack-path triage data.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(triage): stop cluster-summary refresh freezing on cutover cluster_id collision ... 370f81dd22

The incremental cluster rollup refresh upserts with ON CONFLICT
(cluster_key), but cluster_summary also carried a separate UNIQUE index on
cluster_id. During an ROR identity cutover one physical cluster can hold two
cluster_key rows (pre-cutover kube-UID key + post-cutover slug key) that
resolve to the same derived cluster_id. Postgres allows one conflict arbiter
per statement, so the second row raises 23505 on the cluster_id index, which
ON CONFLICT (cluster_key) cannot catch. The whole INSERT ... SELECT aborts,
the refresh errors, and the watermark never advances — freezing both
cluster_summary and cluster_image_inventory fleet-wide until manually
truncated. One colliding cluster starved every cluster's rollup, hiding all
newly-registered agents from /api/clusters/summary.

- Drop the UNIQUE constraint on cluster_summary.cluster_id, keep it as a
  plain index (migration 20260710). The refresh can no longer abort on a
  cluster_id collision.
- Dedup at read time in ClusterSummaryHandler with DISTINCT ON (cluster_id),
  preferring the ROR-slug-bearing (post-cutover) row, so the transient
  double-row collapses to one and per-fleet totals count each cluster once.
- Log a WARN when a refresh runs against long-stale rollups, so a future
  stalled watermark surfaces instead of failing silently.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

jonas force-pushed fix/cluster-summary-refresh-freeze from 370f81dd22 to 7797e211f8 2026-07-08 22:02:42 +02:00 Compare

jonas merged commit b89ab3bb76 into main 2026-07-08 22:19:59 +02:00

jonas deleted branch fix/cluster-summary-refresh-freeze 2026-07-08 22:20:00 +02:00

jonas referenced this pull request from a commit 2026-07-08 22:20:00 +02:00

fix(triage): stop cluster-summary refresh freezing on cutover cluster_id collision (#49)

jonas referenced this pull request 2026-07-08 22:23:10 +02:00

fix(vuln): drop vestigial UNIQUE on unified-vuln tables to prevent refresh freeze #51

jonas referenced this pull request from a commit 2026-07-08 22:53:37 +02:00

fix(vuln): drop vestigial UNIQUE on unified-vuln tables to prevent refresh freeze (#51)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/api

Public/UI API surface

  

area/cluster

SCAM cluster data

  

area/llm

LLM advisory / agents

  

area/notifications

Notifications & actions

  

area/triage

Vulnerability triage

  

epic

Large multi-PR feature

  

feature

New functionality

  

needs-design

Open design questions to resolve

No labels

area/api

area/cluster

area/llm

area/notifications

area/triage

epic

feature

needs-design

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Motstandskraft/spam!49

No description provided.