feat(analyzer): Phase 2 — full stage persistence, fingerprints, aggregate reports, cost guards
Eight sub-phases per docs/ticket-analyzer-phase2-spec.md:
2.1 Schema (migration 070): analyzer_stage_executions table; source_snapshot,
aggregate_fingerprint, fingerprint_generated_at columns on analyzer_analyses.
model_traces marked LEGACY (kept for back-compat).
2.2 Every pipeline stage records a row to analyzer_stage_executions, success
or failure. Worker persists a status='failed' analyzer_analyses row when
the pipeline throws so partial stage records have a parent. Pipeline
exposes raw triage/sonnet/opus responses for downstream stages.
2.3 Stage 3 prompt updated with markdown formatting rules + banned filler
phrases. Added react-markdown + remark-gfm + @tailwindcss/typography.
New <AnalysisMarkdown> component replaces <ProseText>; coerces stray
headers to bold paragraphs.
2.4 Stage 6 fingerprint (Haiku) runs after persistence, failure-tolerant.
scripts/backfill-fingerprints.ts reconstructs Stage 6 input from the
legacy model_traces blob.
2.5 Browse UI rebuild at /analyzer/tickets: multi-select for client/issue/
queue/status/priority/assignee, sticky filter bar, active-filter chips,
bulk selection persisted via localStorage, "Analyze N selected" +
"Generate aggregate report" actions. New <MultiSelect> primitive.
Staleness uses last_activity_date > completed_at heuristic per spec C.1.
2.6 Aggregate reports (migration 071): runner is fire-and-forget, persists
SQL distributions immediately so UI shows partial state during the
Sonnet reduce call. Three endpoints, three pages (/analyzer/reports[/new
/:id]). IT Glue context fetcher capped at 200 doc titles.
2.7 Cost guards (migration 072): per-request $5 confirmation, soft-warn at
$20/day, hard-block at $50/day with ANALYZER_DAILY_COST_OVERRIDE_USERS
override. Every gating decision audited.
2.8 Runbook + build notes updated.
128 vitest tests passing, tsc clean. Migrations 070/071/072 idempotent
(IF NOT EXISTS). model_traces double-write retained — drop in a future
migration once aggregate reports have soaked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
b20c94ea1a
commit
bd3401df1c
33 changed files with 7132 additions and 554 deletions
|
|
@ -506,3 +506,291 @@ analyze without typing the URL.
|
|||
| 7 | 128 | clean | share email via existing SMTP transport |
|
||||
| 8 | 128 | clean | operator runbook + README link |
|
||||
| 9 | 128 | clean | browse page + analysis-view formatting + nav entry |
|
||||
|
||||
---
|
||||
|
||||
# Phase 2 (cross-ticket analysis)
|
||||
|
||||
Spec: `docs/ticket-analyzer-phase2-spec.md`. Eight sub-phases delivered as
|
||||
one Phase 2 push.
|
||||
|
||||
## 2.1 — Schema additions
|
||||
|
||||
**Delivered**
|
||||
|
||||
- Migration 070: `analyzer_stage_executions` table (per-stage I/O for
|
||||
every analyzer run, including failed attempts) + three columns on
|
||||
`analyzer_analyses`: `source_snapshot`, `aggregate_fingerprint`,
|
||||
`fingerprint_generated_at`.
|
||||
- `model_traces` column annotated with a `LEGACY` `COMMENT ON COLUMN`
|
||||
for the SQL side and a `// LEGACY` doc comment in TS — kept for
|
||||
back-compat until aggregate reports have soaked.
|
||||
- Zod schemas: `AggregateFingerprint`, `StageName`, `StageExecution`
|
||||
(read-back), `StageExecutionRecord` (write-time interface).
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- `analyzer_stage_executions.analysis_id` starts NOT NULL in 070.
|
||||
Migration 071 (Phase 2.6) relaxes it and adds the
|
||||
mutually-exclusive CHECK with `aggregate_report_id`.
|
||||
- 070 is idempotent (`IF NOT EXISTS` on every object) so re-running
|
||||
against an already-applied DB is safe.
|
||||
|
||||
## 2.2 — Pipeline writes to stage executions
|
||||
|
||||
**Delivered**
|
||||
|
||||
- `recordedStage(meta, fn, callbacks, outputSelector)` helper in
|
||||
`pipeline.ts` wraps each stage call, emits a `StageExecutionRecord`
|
||||
on success or failure (re-throws after recording). Pipeline wires
|
||||
it into Stage 1 (triage), Stage 3 (analyze), Stage 4 (deep_review).
|
||||
Stage 0 (preprocess) and Stage 2 (itglue) are recorded inline since
|
||||
they're not LLM calls.
|
||||
- Pipeline result now includes `triage_response`, `sonnet_response`,
|
||||
`opus_response` for downstream stages (Stage 6 fingerprint).
|
||||
- Worker's `runJob` collects records via `onStageRecord` callback,
|
||||
bulk-inserts them after `insertAnalysis` succeeds. On pipeline
|
||||
throw, worker captures the preprocessed bundle via `onPreprocessed`
|
||||
callback, persists a `status='failed'` analyzer_analyses row with
|
||||
source_snapshot intact, and bulk-inserts the partial stage records
|
||||
linked to it.
|
||||
- New persistence functions: `bulkInsertStageExecutions`,
|
||||
`insertFailedAnalysis`, `updateAnalysisFingerprint`.
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Failure-tolerant audit**: spec says "Every stage that runs MUST
|
||||
insert a row, including stages that fail." We persist a failed
|
||||
analyzer_analyses row even on pipeline crash so the partial stage
|
||||
records have a parent. Without this the FK would be orphaned.
|
||||
- **Single bulk insert**: ~5–6 stage rows per pipeline run. One
|
||||
multi-VALUES INSERT is fast enough; no need for COPY.
|
||||
- **`model_traces` double-write retained**: the legacy column still
|
||||
receives the old payload. Drop it in a future migration once
|
||||
aggregate reports have soaked through prod.
|
||||
|
||||
## 2.3 — Prose formatting
|
||||
|
||||
**Delivered**
|
||||
|
||||
- Stage 3 system prompt updated with the markdown formatting rules
|
||||
from the spec verbatim (banned filler phrases, action-verb
|
||||
emphasis, no headers).
|
||||
- `react-markdown@10`, `remark-gfm@4`, `@tailwindcss/typography@0.5`
|
||||
added. Tailwind 4 plugin registered via `@plugin
|
||||
"@tailwindcss/typography"` in `app/globals.css`.
|
||||
- `<AnalysisMarkdown>` component at `components/analyzer/analysis-markdown.tsx`
|
||||
renders prose with `prose prose-sm dark:prose-invert
|
||||
max-w-none prose-p:leading-7`. Coerces stray model headers into
|
||||
bold paragraphs (the prompt forbids them but defense-in-depth).
|
||||
- `<ProseText>` removed. Summary, Recommended Next Step,
|
||||
next_step_rationale, and post_resolution_analysis all use
|
||||
`<AnalysisMarkdown>`.
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- Tailwind 4 syntax: `@plugin "@tailwindcss/typography"` in CSS, no
|
||||
JS config needed.
|
||||
- Stage 3 prompt change is back-compatible — old analyses with
|
||||
plain-text summaries still render fine through ReactMarkdown.
|
||||
|
||||
## 2.4 — Stage 6 fingerprint + backfill CLI
|
||||
|
||||
**Delivered**
|
||||
|
||||
- `lib/services/analyzer/stages/stage6-fingerprint.ts` — Haiku call
|
||||
with the spec's verbatim system prompt. Server-overrides
|
||||
`generated_by_model` and `generated_at` after parse so the model's
|
||||
guess for those fields can't drift.
|
||||
- Worker integration: after `insertAnalysis` succeeds, run
|
||||
fingerprint with try/catch. Failure logs a warn, fingerprint
|
||||
stays NULL on the row, but the analysis is still complete and
|
||||
usable. The fingerprint stage record is added to the bulk insert
|
||||
whether it succeeded or failed.
|
||||
- `scripts/backfill-fingerprints.ts` — idempotent CLI. Reads
|
||||
`analyzer_analyses.model_traces.{triage_response, sonnet_response,
|
||||
opus_response}` (which Phase 1 was already storing), runs Stage 6,
|
||||
writes `aggregate_fingerprint`. Supports `--dry-run` and
|
||||
`--limit=N`. Skips analyses where model_traces is incomplete.
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- Stage 6 input is just the analysis content (triage + sonnet +
|
||||
optional opus). No `pre` payload needed — fingerprinting is about
|
||||
the produced *analysis*, not the source ticket.
|
||||
- Backfill processes oldest-first (triggered_at ASC). Lets us
|
||||
observe a few rounds before chewing through hundreds.
|
||||
- Stage 6 failure is non-fatal. Spec: "If fingerprinting fails, do
|
||||
NOT fail the overall analysis."
|
||||
|
||||
## 2.5 — Browse / filter UI rebuild
|
||||
|
||||
**Delivered**
|
||||
|
||||
- New endpoint: `GET /api/analyzer/tickets` (replaces the simpler
|
||||
`GET /api/analyzer/tickets/list` from Phase 1.9). Multi-select
|
||||
CSV-style query params (clientId, issueType, queue, status,
|
||||
priority, assignedTo); analyzed segmented filter (any/yes/no/stale);
|
||||
needsReview toggle; search; sort.
|
||||
- `/api/analyzer/tickets/filter-options` extended with queues,
|
||||
statuses, priorities, resources (joined to "has at least one
|
||||
ticket" so the dropdowns aren't padded).
|
||||
- New `<MultiSelect>` component at `components/ui/multi-select.tsx`
|
||||
— Popover + checkbox list with optional search box (auto-shown
|
||||
above 8 options). One trigger + one popover, no shadcn Command
|
||||
dependency.
|
||||
- `/analyzer/tickets` page rebuilt:
|
||||
- Sticky filter bar with period pills, multi-selects, search,
|
||||
analyzed segmented, needs-review checkbox, sort
|
||||
- Active-filter chips (click to clear individual filter)
|
||||
- Bulk selection persisted via localStorage (key
|
||||
`analyzer:ticket-selection:v1`) — survives pagination
|
||||
- "Analyze N selected" — sequential job queue, forces re-analyze
|
||||
on `stale` rows
|
||||
- "Generate aggregate report" — routes to `/analyzer/reports/new`;
|
||||
only enabled when all selected are `current`
|
||||
- Top nav reorganized: Browse Tickets / Aggregate Reports / Needs
|
||||
Review under "Analyzer".
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Staleness via `last_activity_date > completed_at`**, not
|
||||
content-hash compare. The spec lets either; the date heuristic is
|
||||
good enough and avoids per-row preprocessing on 50-row paginated
|
||||
responses.
|
||||
- **`MultiSelect` is a one-popover-per-instance design** — multiple
|
||||
popovers can be open across the bar. Acceptable; matches how
|
||||
Linear / Vercel's table filters behave.
|
||||
- **No "Select all matching filters" semantic**. Selection is an
|
||||
explicit per-row action stored as ticket numbers in localStorage.
|
||||
Filter-level selection adds significant complexity (server has to
|
||||
resolve filter→IDs, two-modes everywhere). Skipped for V1; the
|
||||
spec's intent (don't lose selection on pagination) is met.
|
||||
- **Bulk Analyze is sequential, not parallel.** N concurrent calls
|
||||
would all hit `claimQueuedJob` and the worker would process them
|
||||
one at a time anyway (single in-process worker). Sequential POSTs
|
||||
are more honest about that.
|
||||
|
||||
## 2.6 — Aggregate reports
|
||||
|
||||
**Delivered**
|
||||
|
||||
- Migration 071: `analyzer_aggregate_reports` table + ALTER on
|
||||
`analyzer_stage_executions` to drop NOT NULL on `analysis_id`,
|
||||
add `aggregate_report_id` FK, add the
|
||||
`analyzer_stage_executions_parent_check` CHECK constraint
|
||||
(`(analysis_id IS NOT NULL) <> (aggregate_report_id IS NOT NULL)`).
|
||||
- `lib/services/analyzer/stages/aggregate-reduce.ts` — Sonnet (Opus
|
||||
opt-in) reduce stage with the spec's verbatim system prompt and
|
||||
`AggregateReduceResponse` Zod schema.
|
||||
- `lib/services/analyzer/aggregate-persistence.ts` — `createAggregateReport`,
|
||||
`getAggregateReport`, `listAggregateReports`, `runAggregateReport`,
|
||||
`bulkInsertReportStageExecutions`. The runner is fire-and-forget
|
||||
(called via `void runAggregateReport(id)` from the POST endpoint);
|
||||
it persists distributions immediately so the UI can show partial
|
||||
results during the LLM call.
|
||||
- IT Glue context fetcher: per-client `findOrganizationByName` +
|
||||
`getFlexibleAssets`, capped at 200 doc titles total per spec.
|
||||
Failure tolerant — per-client errors don't fail the report.
|
||||
- API endpoints:
|
||||
- `POST /api/analyzer/aggregate-reports` — validates (≤100,
|
||||
fingerprint exists, not stale), creates pending row, fires runner
|
||||
- `GET /api/analyzer/aggregate-reports/:id` — full report row
|
||||
(UI polls this every 3s while pending/running)
|
||||
- `GET /api/analyzer/aggregate-reports` — paginated list
|
||||
- Pages:
|
||||
- `/analyzer/reports/new?ids=T...,T...` — pre-flight: shows
|
||||
selected tickets, options (title, IT Glue context toggle),
|
||||
Generate button
|
||||
- `/analyzer/reports/[id]` — pending → distributions → completed.
|
||||
Sections: header, executive summary, four distribution mini-bar
|
||||
cards, documentation gaps (with `itglue_check` color tone),
|
||||
process gaps (severity tone), recurrence clusters, recommended
|
||||
actions (sorted by priority), narrative summary.
|
||||
- `/analyzer/reports` — table list of past reports
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Fire-and-forget runner**, no separate worker module. The POST
|
||||
endpoint kicks `void runAggregateReport(id)`; updates land in the
|
||||
row when the LLM call completes. Frontend polls. Avoids adding a
|
||||
second polling worker alongside `analyzerWorker`.
|
||||
- **Stage names reused for aggregate sub-stages.** The CHECK constraint
|
||||
on `analyzer_stage_executions.stage` enumerates the per-analysis
|
||||
stage names. Aggregate sub-stages (SQL aggregation, IT Glue context,
|
||||
reduce LLM) are recorded with `stage='analyze'` / `'itglue'` plus
|
||||
`aggregate_report_id` set. A future migration could add
|
||||
`aggregate_sql` / `aggregate_reduce` to the enum and re-emit those
|
||||
rows; for now the existing names are good enough for forensics.
|
||||
- **`generated_by_user_id` is TEXT nullable**, not `uuid NOT NULL`
|
||||
per spec. Better Auth's `user.id` is text, and we want the report
|
||||
to remain readable if the generating user is later deleted —
|
||||
matches the pattern from `analyzer_analyses.triggered_by_user_id`.
|
||||
- **Distributions persist before LLM call** so partial-state UI
|
||||
doesn't have to wait the full 30–90s for anything to render.
|
||||
|
||||
## 2.7 — Cost guards
|
||||
|
||||
**Delivered**
|
||||
|
||||
- Migration 072: `analyzer_cost_audit` table.
|
||||
- `lib/services/analyzer/cost-guard.ts`:
|
||||
`estimateAggregateReportCost`, `getUserDailySpend`, `evaluateCost`,
|
||||
`recordCostAuditDecision`. Thresholds: `REQUIRES_CONFIRMATION_USD = 5`,
|
||||
`SOFT_WARN_DAILY_USD = 20`, `HARD_BLOCK_DAILY_USD = 50`.
|
||||
- `evaluateCost` produces a four-state decision (`approved` /
|
||||
`requires_confirmation` / `blocked` / `overridden`) plus boolean
|
||||
`softWarn`/`hardBlocked`/`requiresConfirmation`/`isOverride`
|
||||
fields the API can return for UX.
|
||||
- POST `/api/analyzer/aggregate-reports` enforces:
|
||||
- `requires_confirmation` → 400 with `requiresConfirmation:true,
|
||||
estimatedCost, dailySpendBefore` so the frontend can show
|
||||
`confirm()` and re-POST with `confirmedCost:true`.
|
||||
- `blocked` → 403 with the daily spend in the body.
|
||||
- Every decision (including `approved`) writes a row to
|
||||
`analyzer_cost_audit`.
|
||||
- Override env var `ANALYZER_DAILY_COST_OVERRIDE_USERS`
|
||||
(comma-separated user ids).
|
||||
- Frontend new-report page: catches `requiresConfirmation`,
|
||||
shows `window.confirm()` with the dollar figure, retries with
|
||||
`confirmedCost: true`.
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Cost estimate is char/4 → tokens × Sonnet pricing.** Crude but
|
||||
pessimistic in the right direction. At 100 tickets the estimate
|
||||
comes in under $0.50 — far below the $5 threshold — so the
|
||||
confirmation modal almost never fires in practice. Ceiling exists
|
||||
to catch payload bloat / Opus-opt-in scenarios.
|
||||
- **Daily window is trailing 24h, not "today UTC".** Avoids
|
||||
midnight-edge-of-day reset gaming; rolling window is what the
|
||||
spec calls "$X/day" naturally.
|
||||
- **Soft warn at $20/day is informational only.** Fields exposed
|
||||
in the cost evaluation; UI can choose to surface, but the API
|
||||
doesn't refuse to proceed. Hard block at $50/day is the only
|
||||
enforcement.
|
||||
|
||||
## 2.8 — Documentation
|
||||
|
||||
**Delivered**
|
||||
|
||||
- `docs/wulf-pulse-ticket-analyzer-runbook.md` — added Phase 2
|
||||
sections covering stage execution forensics, fingerprint backfill,
|
||||
aggregate report flow + SQL queries, cost guard configuration +
|
||||
override env var, and the rebuilt browse UI behavior.
|
||||
- This file — the per-sub-phase notes above.
|
||||
|
||||
---
|
||||
|
||||
## Status after Phase 2
|
||||
|
||||
| Phase | Tests | tsc | Notes |
|
||||
|---|---|---|---|
|
||||
| 2.1 | 128 | clean | schema (070) |
|
||||
| 2.2 | 128 | clean | stage_executions writes + failure-tolerant persistence |
|
||||
| 2.3 | 128 | clean | markdown rendering + Stage 3 prompt |
|
||||
| 2.4 | 128 | clean | Stage 6 fingerprint + backfill CLI |
|
||||
| 2.5 | 128 | clean | browse UI rebuild |
|
||||
| 2.6 | 128 | clean | aggregate reports (071, runner, 3 endpoints, 3 pages) |
|
||||
| 2.7 | 128 | clean | cost guards (072, audit log, threshold gating) |
|
||||
| 2.8 | 128 | clean | runbook + build notes |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue