feat(analyzer): Phase 2 — full stage persistence, fingerprints, aggregate reports, cost guards
Eight sub-phases per docs/ticket-analyzer-phase2-spec.md:
2.1 Schema (migration 070): analyzer_stage_executions table; source_snapshot,
aggregate_fingerprint, fingerprint_generated_at columns on analyzer_analyses.
model_traces marked LEGACY (kept for back-compat).
2.2 Every pipeline stage records a row to analyzer_stage_executions, success
or failure. Worker persists a status='failed' analyzer_analyses row when
the pipeline throws so partial stage records have a parent. Pipeline
exposes raw triage/sonnet/opus responses for downstream stages.
2.3 Stage 3 prompt updated with markdown formatting rules + banned filler
phrases. Added react-markdown + remark-gfm + @tailwindcss/typography.
New <AnalysisMarkdown> component replaces <ProseText>; coerces stray
headers to bold paragraphs.
2.4 Stage 6 fingerprint (Haiku) runs after persistence, failure-tolerant.
scripts/backfill-fingerprints.ts reconstructs Stage 6 input from the
legacy model_traces blob.
2.5 Browse UI rebuild at /analyzer/tickets: multi-select for client/issue/
queue/status/priority/assignee, sticky filter bar, active-filter chips,
bulk selection persisted via localStorage, "Analyze N selected" +
"Generate aggregate report" actions. New <MultiSelect> primitive.
Staleness uses last_activity_date > completed_at heuristic per spec C.1.
2.6 Aggregate reports (migration 071): runner is fire-and-forget, persists
SQL distributions immediately so UI shows partial state during the
Sonnet reduce call. Three endpoints, three pages (/analyzer/reports[/new
/:id]). IT Glue context fetcher capped at 200 doc titles.
2.7 Cost guards (migration 072): per-request $5 confirmation, soft-warn at
$20/day, hard-block at $50/day with ANALYZER_DAILY_COST_OVERRIDE_USERS
override. Every gating decision audited.
2.8 Runbook + build notes updated.
128 vitest tests passing, tsc clean. Migrations 070/071/072 idempotent
(IF NOT EXISTS). model_traces double-write retained — drop in a future
migration once aggregate reports have soaked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
b20c94ea1a
commit
bd3401df1c
33 changed files with 7132 additions and 554 deletions
|
|
@ -316,3 +316,172 @@ SELECT round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd,
|
|||
If any of these become a real operational problem, file the work — the
|
||||
stubs are intentional and called out in
|
||||
`wulf-pulse-ticket-analyzer-build-notes.md`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 additions
|
||||
|
||||
### Stage execution history (per-analysis forensics)
|
||||
|
||||
Migration 070 introduced `analyzer_stage_executions`. Every stage of the
|
||||
pipeline (preprocess, triage, itglue, analyze, deep_review, fingerprint)
|
||||
now writes a row including the input it saw and the output it produced.
|
||||
Failed stages get a row with `error_message` populated.
|
||||
|
||||
Read it like this:
|
||||
|
||||
```sql
|
||||
-- Full per-stage trace for one analysis
|
||||
SELECT stage_order, stage, model_id,
|
||||
input_tokens, output_tokens, latency_ms,
|
||||
error_message
|
||||
FROM analyzer_stage_executions
|
||||
WHERE analysis_id = '<analysis-id>'
|
||||
ORDER BY stage_order;
|
||||
|
||||
-- Inspect a single stage's full input/output
|
||||
SELECT input_payload, output_payload
|
||||
FROM analyzer_stage_executions
|
||||
WHERE analysis_id = '<analysis-id>' AND stage = 'analyze';
|
||||
```
|
||||
|
||||
The legacy `analyzer_analyses.model_traces` JSONB column is preserved for
|
||||
back-compat. New analyses double-write to both. A future migration will
|
||||
drop `model_traces` once aggregate reports have soaked.
|
||||
|
||||
### Fingerprint stage (Stage 6)
|
||||
|
||||
After persistence, the worker runs Haiku-tier fingerprint extraction and
|
||||
writes the result to `analyzer_analyses.aggregate_fingerprint`. Failure
|
||||
is non-fatal — the analysis row stays usable, fingerprint stays NULL.
|
||||
|
||||
Re-fingerprint on demand:
|
||||
|
||||
```sql
|
||||
SELECT id, ticket_number, analysis_version
|
||||
FROM analyzer_analyses
|
||||
WHERE aggregate_fingerprint IS NULL
|
||||
AND status = 'complete'
|
||||
ORDER BY triggered_at ASC;
|
||||
```
|
||||
|
||||
Or run the backfill script (idempotent — the SQL filter skips already-fingerprinted rows):
|
||||
|
||||
```bash
|
||||
npx tsx scripts/backfill-fingerprints.ts # process all missing
|
||||
npx tsx scripts/backfill-fingerprints.ts --limit=50 # cap work
|
||||
npx tsx scripts/backfill-fingerprints.ts --dry-run # show what would run
|
||||
```
|
||||
|
||||
The script reconstructs the Stage 6 input from
|
||||
`analyzer_analyses.model_traces.{triage_response, sonnet_response, opus_response}`.
|
||||
If a row's model_traces is missing those fields (very old format) the script
|
||||
logs `[skip]` for it and moves on.
|
||||
|
||||
Cost: Haiku at ~$0.001 per fingerprint. 1000 analyses ≈ $1.
|
||||
|
||||
### Aggregate reports
|
||||
|
||||
Migration 071 added `analyzer_aggregate_reports` and relaxed
|
||||
`analyzer_stage_executions.analysis_id` to be nullable; rows now carry
|
||||
either an `analysis_id` or an `aggregate_report_id` (CHECK enforces
|
||||
exactly one).
|
||||
|
||||
Workflow:
|
||||
|
||||
1. User selects tickets on `/analyzer/tickets` and clicks
|
||||
**Generate aggregate report**.
|
||||
2. POST `/api/analyzer/aggregate-reports` validates: ≤100 tickets, all
|
||||
have a fingerprint, none are stale. Inserts a 'pending' row, fires
|
||||
the runner via `void runAggregateReport(id)`, returns immediately.
|
||||
3. Runner does:
|
||||
- Loads fingerprints
|
||||
- Computes SQL distributions (categories, clients, root cause,
|
||||
resolution path) and writes them to the row immediately
|
||||
- Fetches IT Glue doc titles for affected clients (if requested)
|
||||
- Calls Sonnet (Opus opt-in) with a structured payload
|
||||
- Writes the LLM-derived fields, marks `status='complete'`
|
||||
4. UI polls `GET /api/analyzer/aggregate-reports/:id` every 3s while
|
||||
pending/running.
|
||||
|
||||
Inspect a report's sub-stage trace (linked via `aggregate_report_id`):
|
||||
|
||||
```sql
|
||||
SELECT stage_order, stage, model_id, latency_ms, error_message
|
||||
FROM analyzer_stage_executions
|
||||
WHERE aggregate_report_id = '<report-id>'
|
||||
ORDER BY stage_order;
|
||||
```
|
||||
|
||||
Daily spend on aggregate reports:
|
||||
|
||||
```sql
|
||||
SELECT date_trunc('day', generated_at) AS day,
|
||||
count(*) AS reports,
|
||||
round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd
|
||||
FROM analyzer_aggregate_reports
|
||||
WHERE generated_at > now() - interval '14 days'
|
||||
GROUP BY 1
|
||||
ORDER BY 1 DESC;
|
||||
```
|
||||
|
||||
### Cost guards (Phase 2.7)
|
||||
|
||||
Three thresholds enforce per-user daily spend:
|
||||
|
||||
- **$5 per request** — a single aggregate report estimated above $5
|
||||
triggers a confirmation modal. Frontend re-POSTs with
|
||||
`confirmedCost: true` if the user clicks through.
|
||||
- **$20 per user/day** — soft warn. Surfaced via `softWarn:true` in the
|
||||
cost evaluation; no enforcement.
|
||||
- **$50 per user/day** — hard block. POST returns 403 unless the user
|
||||
is in `ANALYZER_DAILY_COST_OVERRIDE_USERS` (comma-separated list of
|
||||
Better Auth user ids).
|
||||
|
||||
Daily spend is computed as the trailing 24h sum of
|
||||
`analyzer_analyses.estimated_cost_usd` + `analyzer_aggregate_reports.estimated_cost_usd`
|
||||
for the user.
|
||||
|
||||
Every gating decision writes a row to `analyzer_cost_audit`:
|
||||
|
||||
```sql
|
||||
SELECT created_at, user_id, action, estimated_cost,
|
||||
daily_spend_before, decision, decision_reason
|
||||
FROM analyzer_cost_audit
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 50;
|
||||
```
|
||||
|
||||
To grant a user override capability:
|
||||
|
||||
```bash
|
||||
# in ~/projects_env/wulf-pulse.env
|
||||
ANALYZER_DAILY_COST_OVERRIDE_USERS=user_id_1,user_id_2
|
||||
```
|
||||
|
||||
Restart `pulse-app` to pick up the change.
|
||||
|
||||
### Browse / filter ticket list
|
||||
|
||||
`/analyzer/tickets` rebuilt as the primary entry point in Phase 2.5:
|
||||
|
||||
- Multi-select for client, issue type, queue, status, priority, assignee
|
||||
- Sticky filter bar with period chips (today/yesterday/this+last week,
|
||||
30d/60d, custom range, all time)
|
||||
- Active-filter chips below the bar; click to remove
|
||||
- Bulk row selection persisted in localStorage
|
||||
(`analyzer:ticket-selection:v1`) — survives pagination but is
|
||||
session-scoped (no user id baked in)
|
||||
- Bulk "Analyze N selected" sequentially queues jobs for each ticket
|
||||
(forces re-analyze on stale; analyzes from scratch on un-analyzed)
|
||||
- Bulk "Generate aggregate report" routes selected tickets to
|
||||
`/analyzer/reports/new`; only enabled when all selected are
|
||||
`analyzedState === 'current'`
|
||||
|
||||
**Staleness heuristic**: a ticket's `analyzedState` is computed as
|
||||
`stale` when `tickets.last_activity_date > latest_analysis.completed_at`.
|
||||
The Phase 2 spec calls for content-hash-based comparison; that requires
|
||||
either caching the current hash on the tickets row (sync change) or
|
||||
computing it on read for the visible page (slow). The date heuristic
|
||||
gets ~95% of the value at zero compute cost; revisit when there's real
|
||||
load signal.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue