LLM-agent repo audit: lint/types/security/deps/perf/a11y breakdowns + dependency intelligence #26

New issue

Open

opened 2026-06-30 13:51:20 +02:00 by jonas · 0 comments

jonas commented

2026-06-30 13:51:20 +02:00

Owner

Goal

Expand the repo check into an LLM-agent-driven, multi-dimension code & dependency audit. For a given repo/commit, an agent runs (and reasons over) checks across several breakdowns and produces a structured, scored report — combining classic tooling output (ESLint, tsc, linters, dependency advisories) with LLM analysis, cross-referenced with the CVE/KEV/EPSS results SPAM already has.

Breakdowns (inspired by dagmawibabi/projectlens): security, dependencies, performance, environment, documentation, types, lint, tests, accessibility, network.

Includes a potential new dependency-intelligence feature: an LLM agent that, for all dependencies, does security analysis and finds dead / abandoned / unused dependencies (beyond what version-bump metadata shows).

What already exists (reuse)

CREATE_RUN job + sandboxed K8s runner (internal/runner, internal/jobs): already clones a repo at a commit and runs a suite of checks (secret probe, etc.) in an isolated pod with SecurityContext + NetworkPolicy. This is where the agent / linters run — never in the API process.
LLM plumbing: internal/llmadvisory (OpenAI-compatible, pluggable Open WebUI / vLLM, encrypted keys, streaming, caching keyed by a signals hash). Extend with new use cases per breakdown, or one repo_audit use case with structured output.
Existing signals to cross-reference:
- CVE/KEV/EPSS per asset: asset_risk, vuln_canonical_summary, unified vuln views.
- Dependency health already exists: internal/dephealth (activity, deprecation, stars, ArchivedDepCount, DeprecatedDepCount, MaxMajorBehind, WorstDepHealthScore) — the "dead dependency" feature should extend this, not duplicate it.
- Manifest parsing: internal/manifests (lockfiles, package.json, etc.).
- Triage tiers + asset_risk scoring for the security breakdown.

Proposed design

Audit run

New job type REPO_AUDIT (or extend CREATE_RUN) producing a repo_audit record per (repo_commit, breakdown) with a score + findings + LLM narrative.
In the runner pod: run the deterministic tools first (ESLint, tsc --noEmit, framework linters, npm audit/equivalent, test runner if cheap), capture raw output as artifacts (like ImageScanArtifact blobs).
Feed tool output + repo structure + SPAM's existing CVE/dep-health signals into the LLM agent, which produces the structured per-breakdown report (severity-ranked findings, remediation hints). Cache by a signals hash (same pattern as advisory) so re-runs only happen when inputs change.

Breakdown model

repo_audit_finding {audit_id, breakdown, severity, title, detail, file?, line?, source: tool|llm, remediation}. Breakdowns: security / dependencies / performance / environment / documentation / types / lint / tests / accessibility / network. UI shows a per-repo scorecard (one card per breakdown) + drill-in.

Dependency intelligence (sub-feature, can be its own issue)

LLM agent over the full dependency graph (manifests + dephealth):

Dead / unused deps: static import-graph analysis (tool) + LLM confirmation that a declared dep is never imported / behind dead code.
Abandoned / risky: combine dephealth (archived/deprecated/last-release age) with LLM read of repo + advisories.
Security analysis: map deps to known CVEs (already have) + LLM reasoning about transitive risk and maintainer trust.

Edge cases / decisions

Determinism vs. LLM drift. Tool output is ground truth; LLM is for synthesis/prioritization/explanation. Don't let the LLM invent findings — require it to cite tool output or file evidence. (Important given the AI-security-audit angle.)
Language/ecosystem coverage. ESLint/tsc are JS/TS-specific. Define which stacks v1 supports; make linter selection per-detected-ecosystem (from manifests).
Cost & runtime. Full audit is expensive — gate behind on-demand trigger + caching; don't run on every commit by default.
Sandbox & egress. Audit pod must not exfiltrate source; network egress locked down (existing runner NetworkPolicy). npm audit etc. need controlled registry access.
False positives in "dead dependency". Dynamic imports / build-time-only / type-only deps → conservative defaults, mark confidence, let users confirm/suppress (ties into suppression epic).
Result schema should be expandable and feed the public scan API (issue: expandable scan results) and notifications (audit.completed).
Auth/ACL & PAT: repo clone uses existing providerconfig git PATs (encrypted).

Tasks

Define breakdown taxonomy + repo_audit / repo_audit_finding models (+ migration)
REPO_AUDIT job: run linters/tsc/test/audit tools in runner pod, capture artifacts
LLM synthesis use case(s) in llmadvisory with structured output + evidence-citation guardrails
Cross-reference existing CVE/KEV/EPSS + dephealth into the security & dependencies breakdowns
Dependency-intelligence agent: dead/unused + abandoned + security (extends dephealth) — consider splitting into its own issue
Web: per-repo audit scorecard (one card per breakdown) + drill-in + re-run trigger
Caching + on-demand trigger; audit.completed event for notifications

Relates to: notifications epic; public scan API (expandable results); suppression epic (suppress audit findings).

## Goal Expand the **repo check** into an **LLM-agent-driven, multi-dimension code & dependency audit**. For a given repo/commit, an agent runs (and reasons over) checks across several breakdowns and produces a structured, scored report — combining classic tooling output (ESLint, tsc, linters, dependency advisories) with LLM analysis, **cross-referenced with the CVE/KEV/EPSS results SPAM already has**. Breakdowns (inspired by `dagmawibabi/projectlens`): **security, dependencies, performance, environment, documentation, types, lint, tests, accessibility, network.** Includes a potential **new dependency-intelligence feature**: an LLM agent that, for *all* dependencies, does security analysis and finds **dead / abandoned / unused dependencies** (beyond what version-bump metadata shows). ## What already exists (reuse) - **`CREATE_RUN` job + sandboxed K8s runner** (`internal/runner`, `internal/jobs`): already clones a repo at a commit and runs a suite of checks (secret probe, etc.) in an isolated pod with SecurityContext + NetworkPolicy. **This is where the agent / linters run** — never in the API process. - **LLM plumbing:** `internal/llmadvisory` (OpenAI-compatible, pluggable Open WebUI / vLLM, encrypted keys, streaming, caching keyed by a signals hash). Extend with new use cases per breakdown, or one `repo_audit` use case with structured output. - **Existing signals to cross-reference:** - CVE/KEV/EPSS per asset: `asset_risk`, `vuln_canonical_summary`, unified vuln views. - **Dependency health already exists:** `internal/dephealth` (activity, deprecation, stars, `ArchivedDepCount`, `DeprecatedDepCount`, `MaxMajorBehind`, `WorstDepHealthScore`) — the "dead dependency" feature should *extend* this, not duplicate it. - Manifest parsing: `internal/manifests` (lockfiles, package.json, etc.). - Triage tiers + `asset_risk` scoring for the security breakdown. ## Proposed design ### Audit run - New job type `REPO_AUDIT` (or extend `CREATE_RUN`) producing a `repo_audit` record per (repo_commit, breakdown) with a score + findings + LLM narrative. - In the runner pod: run the deterministic tools first (ESLint, `tsc --noEmit`, framework linters, `npm audit`/equivalent, test runner if cheap), capture raw output as artifacts (like `ImageScanArtifact` blobs). - Feed tool output + repo structure + SPAM's existing CVE/dep-health signals into the LLM agent, which produces the **structured per-breakdown report** (severity-ranked findings, remediation hints). Cache by a signals hash (same pattern as advisory) so re-runs only happen when inputs change. ### Breakdown model `repo_audit_finding` `{audit_id, breakdown, severity, title, detail, file?, line?, source: tool|llm, remediation}`. Breakdowns: security / dependencies / performance / environment / documentation / types / lint / tests / accessibility / network. UI shows a per-repo scorecard (one card per breakdown) + drill-in. ### Dependency intelligence (sub-feature, can be its own issue) LLM agent over the full dependency graph (`manifests` + `dephealth`): - **Dead / unused deps:** static import-graph analysis (tool) + LLM confirmation that a declared dep is never imported / behind dead code. - **Abandoned / risky:** combine `dephealth` (archived/deprecated/last-release age) with LLM read of repo + advisories. - **Security analysis:** map deps to known CVEs (already have) + LLM reasoning about transitive risk and maintainer trust. ## Edge cases / decisions 1. **Determinism vs. LLM drift.** Tool output is ground truth; LLM is for synthesis/prioritization/explanation. Don't let the LLM *invent* findings — require it to cite tool output or file evidence. (Important given the AI-security-audit angle.) 2. **Language/ecosystem coverage.** ESLint/tsc are JS/TS-specific. Define which stacks v1 supports; make linter selection per-detected-ecosystem (from `manifests`). 3. **Cost & runtime.** Full audit is expensive — gate behind on-demand trigger + caching; don't run on every commit by default. 4. **Sandbox & egress.** Audit pod must not exfiltrate source; network egress locked down (existing runner NetworkPolicy). `npm audit` etc. need controlled registry access. 5. **False positives in "dead dependency".** Dynamic imports / build-time-only / type-only deps → conservative defaults, mark confidence, let users confirm/suppress (ties into suppression epic). 6. **Result schema** should be expandable and feed the public scan API (issue: expandable scan results) and notifications (`audit.completed`). 7. **Auth/ACL & PAT:** repo clone uses existing `providerconfig` git PATs (encrypted). ## Tasks - [ ] Define breakdown taxonomy + `repo_audit` / `repo_audit_finding` models (+ migration) - [ ] `REPO_AUDIT` job: run linters/tsc/test/audit tools in runner pod, capture artifacts - [ ] LLM synthesis use case(s) in `llmadvisory` with structured output + evidence-citation guardrails - [ ] Cross-reference existing CVE/KEV/EPSS + `dephealth` into the security & dependencies breakdowns - [ ] Dependency-intelligence agent: dead/unused + abandoned + security (extends `dephealth`) — consider splitting into its own issue - [ ] Web: per-repo audit scorecard (one card per breakdown) + drill-in + re-run trigger - [ ] Caching + on-demand trigger; `audit.completed` event for notifications **Relates to:** notifications epic; public scan API (expandable results); suppression epic (suppress audit findings).

jonas added the

labels

2026-06-30 13:51:20 +02:00

jonas referenced this issue

2026-06-30 15:53:42 +02:00

Dependency maintenance intelligence: detect stale/abandoned libs, per-library threat view, deepsearch harness #28