feat(analyzer): Phase 2 — full stage persistence, fingerprints, aggregate reports, cost guards
Eight sub-phases per docs/ticket-analyzer-phase2-spec.md:
2.1 Schema (migration 070): analyzer_stage_executions table; source_snapshot,
aggregate_fingerprint, fingerprint_generated_at columns on analyzer_analyses.
model_traces marked LEGACY (kept for back-compat).
2.2 Every pipeline stage records a row to analyzer_stage_executions, success
or failure. Worker persists a status='failed' analyzer_analyses row when
the pipeline throws so partial stage records have a parent. Pipeline
exposes raw triage/sonnet/opus responses for downstream stages.
2.3 Stage 3 prompt updated with markdown formatting rules + banned filler
phrases. Added react-markdown + remark-gfm + @tailwindcss/typography.
New <AnalysisMarkdown> component replaces <ProseText>; coerces stray
headers to bold paragraphs.
2.4 Stage 6 fingerprint (Haiku) runs after persistence, failure-tolerant.
scripts/backfill-fingerprints.ts reconstructs Stage 6 input from the
legacy model_traces blob.
2.5 Browse UI rebuild at /analyzer/tickets: multi-select for client/issue/
queue/status/priority/assignee, sticky filter bar, active-filter chips,
bulk selection persisted via localStorage, "Analyze N selected" +
"Generate aggregate report" actions. New <MultiSelect> primitive.
Staleness uses last_activity_date > completed_at heuristic per spec C.1.
2.6 Aggregate reports (migration 071): runner is fire-and-forget, persists
SQL distributions immediately so UI shows partial state during the
Sonnet reduce call. Three endpoints, three pages (/analyzer/reports[/new
/:id]). IT Glue context fetcher capped at 200 doc titles.
2.7 Cost guards (migration 072): per-request $5 confirmation, soft-warn at
$20/day, hard-block at $50/day with ANALYZER_DAILY_COST_OVERRIDE_USERS
override. Every gating decision audited.
2.8 Runbook + build notes updated.
128 vitest tests passing, tsc clean. Migrations 070/071/072 idempotent
(IF NOT EXISTS). model_traces double-write retained — drop in a future
migration once aggregate reports have soaked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
b20c94ea1a
commit
bd3401df1c
33 changed files with 7132 additions and 554 deletions
696
docs/ticket-analyzer-phase2-spec.md
Normal file
696
docs/ticket-analyzer-phase2-spec.md
Normal file
|
|
@ -0,0 +1,696 @@
|
|||
# Ticket Analyzer — Phase 2 Spec Additions
|
||||
|
||||
This document **adds to and modifies** the existing ticket analyzer spec at `docs/ticket-analyzer-spec.md`. Apply these changes in the order listed. Each section explicitly states whether it adds, replaces, or modifies existing content.
|
||||
|
||||
The Phase 2 work covers:
|
||||
|
||||
1. Storing all intermediate stage data so analyses are fully reconstructable and aggregate analysis is feasible
|
||||
2. Improving prose formatting in Summary, Next Step, and Next Step Rationale
|
||||
3. A browse/filter ticket list UI with bulk selection
|
||||
4. Aggregate trend and documentation-gap analysis across multiple analyzed tickets
|
||||
|
||||
Build order is preserved at the end. Do not start aggregate analysis (#4) before the schema additions (#1) and fingerprinting are in place — backfilling fingerprints across a large analysis history is wasteful.
|
||||
|
||||
Before starting, re-read `CLAUDE.md` and the existing analyzer code. Conform to whatever conventions emerged during Phase 1.
|
||||
|
||||
---
|
||||
|
||||
## Section A — Schema additions for full analysis storage
|
||||
|
||||
**ADDS to the existing migrations.** Do not modify existing tables in a destructive way; add new columns and a new table.
|
||||
|
||||
### A.1 New table: `analyzer_stage_executions`
|
||||
|
||||
Stores the input and output of every pipeline stage so analyses are fully reconstructable for debugging, auditing, and reprocessing with new prompts.
|
||||
|
||||
```sql
|
||||
CREATE TABLE analyzer_stage_executions (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
analysis_id uuid NOT NULL REFERENCES analyzer_analyses(id) ON DELETE CASCADE,
|
||||
stage text NOT NULL, -- 'preprocess'|'triage'|'itglue'|'analyze'|'deep_review'|'fingerprint'
|
||||
stage_order int NOT NULL,
|
||||
model_id text, -- null for non-LLM stages (preprocess, itglue)
|
||||
input_payload jsonb NOT NULL, -- exactly what was sent to the stage
|
||||
output_payload jsonb NOT NULL, -- exactly what came back, pre-merge
|
||||
input_tokens int,
|
||||
output_tokens int,
|
||||
latency_ms int,
|
||||
started_at timestamptz NOT NULL,
|
||||
completed_at timestamptz NOT NULL,
|
||||
error_message text
|
||||
);
|
||||
CREATE INDEX ON analyzer_stage_executions (analysis_id, stage_order);
|
||||
CREATE INDEX ON analyzer_stage_executions (stage);
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Every stage that runs MUST insert a row, including stages that fail. On failure, populate `error_message` and still record what was attempted in `input_payload`.
|
||||
- The `output_payload` for the IT Glue stage stores the *redacted* document content that was actually passed to the LLM, not raw IT Glue responses. Redaction happens before persistence.
|
||||
- The deep-review stage's `output_payload` stores the full Opus response including `opus_notes`, NOT just the merged updates. The reasoning is currently being lost.
|
||||
- The triage stage's `output_payload` stores the full Haiku output even though only some fields drive routing. The categorization and entity extraction are needed for aggregate analysis later.
|
||||
|
||||
### A.2 New columns on `analyzer_analyses`
|
||||
|
||||
```sql
|
||||
ALTER TABLE analyzer_analyses ADD COLUMN source_snapshot jsonb;
|
||||
ALTER TABLE analyzer_analyses ADD COLUMN aggregate_fingerprint jsonb;
|
||||
ALTER TABLE analyzer_analyses ADD COLUMN fingerprint_generated_at timestamptz;
|
||||
```
|
||||
|
||||
- `source_snapshot` — the pre-processed, tagged event list from Stage 0 (after noise filtering and visibility tagging). Stored canonically because:
|
||||
1. Pulse sync may change/improve, but the analysis is grounded in what was true at analysis time
|
||||
2. Ticket notes occasionally get edited or deleted in Autotask
|
||||
3. Aggregate analysis must operate on a consistent canonical structure across many tickets without re-fetching
|
||||
|
||||
- `aggregate_fingerprint` — structured summary used for aggregate analysis (schema in Section D.2 below). Generated as part of the pipeline; described in Section D.
|
||||
|
||||
The existing `model_traces` column on `analyzer_analyses` becomes redundant once `analyzer_stage_executions` is populated. Keep it for now but add a comment in code marking it as legacy. A future migration can drop it.
|
||||
|
||||
### A.3 Backfill behavior
|
||||
|
||||
Existing analyses (if any from Phase 1) will not have `source_snapshot` or `aggregate_fingerprint`. Do NOT attempt automatic backfill. Provide a CLI script `apps/api/scripts/backfill-fingerprints.ts` that operators can run on demand. The script should:
|
||||
|
||||
- Accept `--limit` and `--dry-run` flags
|
||||
- Process analyses missing `aggregate_fingerprint` in batches of 10
|
||||
- Log progress and total cost
|
||||
- Be idempotent
|
||||
|
||||
Document this script in the operator runbook (added in Phase 1 deliverable 9).
|
||||
|
||||
---
|
||||
|
||||
## Section B — Prose formatting fixes
|
||||
|
||||
**MODIFIES the Stage 3 (Sonnet) system prompt and the frontend analysis view.**
|
||||
|
||||
### B.1 Stage 3 system prompt modifications
|
||||
|
||||
In the existing Stage 3 system prompt, the schema declaration for `summary`, `next_step`, `next_step_rationale`, and `post_resolution_analysis` should be replaced with:
|
||||
|
||||
```
|
||||
"summary": string, // markdown, 2-4 sentences, neutral status briefing
|
||||
"next_step": string, // markdown, single concrete action; may use
|
||||
// **bold** for the action verb and bullet
|
||||
// sub-steps if multi-part
|
||||
"next_step_rationale": string, // markdown, 1-2 short paragraphs; if there are
|
||||
// 3+ competing considerations, use a bulleted list
|
||||
"post_resolution_analysis": string | null, // markdown, same conventions as above
|
||||
```
|
||||
|
||||
Add this section to the system prompt, immediately after the schema declaration block:
|
||||
|
||||
```
|
||||
Formatting rules for prose fields (summary, next_step, next_step_rationale,
|
||||
post_resolution_analysis):
|
||||
|
||||
- Output is markdown and will be rendered with a markdown renderer. Use **bold**
|
||||
for emphasis on key terms or actions. Use *italics* sparingly for client-facing
|
||||
language being quoted. Use bullet lists for enumerable items.
|
||||
|
||||
- Do NOT use markdown headers (#, ##, ###). These fields render inside cards that
|
||||
already have their own headings.
|
||||
|
||||
- For summary: write as a neutral status briefing a manager could read in 10
|
||||
seconds. Lead with the most important fact. Plain language. No hedging.
|
||||
|
||||
- For next_step: state the concrete action in the first sentence, with the action
|
||||
verb in **bold**. If there are sub-steps, follow with a bulleted list. Address
|
||||
the action to the technician, not the customer.
|
||||
|
||||
- For next_step_rationale: open with one short sentence stating the core reason.
|
||||
Follow with a short paragraph elaborating, OR a bulleted list if there are 3+
|
||||
distinct considerations. If there's a competing alternative worth noting, name
|
||||
it explicitly: "Considered X but rejected because..."
|
||||
|
||||
- Avoid filler phrases. Banned openings: "It's worth noting that...", "It's
|
||||
important to understand...", "Based on the available information...",
|
||||
"After reviewing the ticket...". Get to the point.
|
||||
```
|
||||
|
||||
### B.2 Frontend rendering
|
||||
|
||||
Add `react-markdown` and `remark-gfm` to `apps/web/package.json` if not already present.
|
||||
|
||||
Create a shared component `apps/web/src/components/analyzer/AnalysisMarkdown.tsx`:
|
||||
|
||||
```tsx
|
||||
import ReactMarkdown from 'react-markdown';
|
||||
import remarkGfm from 'remark-gfm';
|
||||
|
||||
interface Props {
|
||||
children: string;
|
||||
className?: string;
|
||||
}
|
||||
|
||||
export function AnalysisMarkdown({ children, className }: Props) {
|
||||
return (
|
||||
<div className={`prose prose-sm max-w-none ${className ?? ''}`}>
|
||||
<ReactMarkdown
|
||||
remarkPlugins={[remarkGfm]}
|
||||
components={{
|
||||
// Safety net: model is told not to use headers, but coerce them
|
||||
// to paragraphs if it ignores us
|
||||
h1: ({ children }) => <p className="font-semibold">{children}</p>,
|
||||
h2: ({ children }) => <p className="font-semibold">{children}</p>,
|
||||
h3: ({ children }) => <p className="font-semibold">{children}</p>,
|
||||
h4: ({ children }) => <p className="font-semibold">{children}</p>,
|
||||
h5: ({ children }) => <p className="font-semibold">{children}</p>,
|
||||
h6: ({ children }) => <p className="font-semibold">{children}</p>,
|
||||
}}
|
||||
>
|
||||
{children}
|
||||
</ReactMarkdown>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
Use this component for the four prose fields on the analysis view. Match the existing wulf-pulse Tailwind typography setup; if the project doesn't already use `@tailwindcss/typography`, add it — it's the right plugin for prose-rendering blocks.
|
||||
|
||||
---
|
||||
|
||||
## Section C — Browse / filter ticket list UI
|
||||
|
||||
**ADDS a new view and supporting API endpoint.** Lives in the same nav section as the analyzer feature.
|
||||
|
||||
### C.1 New API endpoint
|
||||
|
||||
```
|
||||
GET /api/analyzer/tickets
|
||||
```
|
||||
|
||||
Query parameters:
|
||||
|
||||
```
|
||||
period today | yesterday | this_week | last_week
|
||||
| last_30d | last_60d | custom
|
||||
startDate ISO date, only when period=custom
|
||||
endDate ISO date, only when period=custom
|
||||
clientId Autotask account id (single or comma-separated)
|
||||
issueType Ticket Category, Issue Type, or Sub-Issue Type
|
||||
queue Autotask queue name
|
||||
status Autotask status (single or comma-separated)
|
||||
priority Autotask priority (single or comma-separated)
|
||||
assignedTo Autotask resource id
|
||||
analyzed any | yes | no | stale (default: any)
|
||||
needsReview true | false (default: any)
|
||||
sort created_desc | created_asc
|
||||
| last_activity_desc | last_activity_asc
|
||||
| priority (default: last_activity_desc)
|
||||
limit default 50, max 200
|
||||
offset default 0
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```ts
|
||||
{
|
||||
tickets: Array<{
|
||||
ticketNumber: string;
|
||||
autotaskTicketId: number;
|
||||
title: string;
|
||||
clientName: string;
|
||||
clientId: number;
|
||||
status: string;
|
||||
priority: string;
|
||||
queue: string;
|
||||
issueType: string;
|
||||
subIssueType: string | null;
|
||||
assignedResourceName: string | null;
|
||||
createdAtAutotask: string;
|
||||
lastActivityAtAutotask: string;
|
||||
ageInDays: number;
|
||||
|
||||
// Analysis state
|
||||
analyzedState: 'none' | 'current' | 'stale';
|
||||
latestAnalysisId: string | null;
|
||||
latestAnalysisAt: string | null;
|
||||
needsHumanReview: boolean;
|
||||
confidenceScore: number | null;
|
||||
primaryCategory: string | null; // from fingerprint, if analyzed
|
||||
}>;
|
||||
total: number;
|
||||
filters: { /* echoed for client-side state sync */ };
|
||||
}
|
||||
```
|
||||
|
||||
Implementation notes:
|
||||
|
||||
- Read tickets from the existing Pulse Postgres sync, NOT live from Autotask. List views must be fast.
|
||||
- LEFT JOIN `analyzer_analyses` on `ticket_number` filtered to the latest version per ticket. Use a window function or subquery — discuss with me before introducing a materialized view.
|
||||
- `analyzedState` derivation:
|
||||
- `none` — no `analyzer_analyses` row
|
||||
- `current` — latest analysis's `content_hash_at_analysis` matches the current Pulse content hash
|
||||
- `stale` — latest analysis exists but content hash differs (new activity since)
|
||||
- The current Pulse content hash will require computing on the fly OR caching on the Pulse ticket row. If Pulse already has webhook-driven updates, add a `current_content_hash` column to the Pulse tickets table populated on sync. If not, compute on-read for now and discuss caching strategy after we see real load.
|
||||
- Cap `limit` at 200 server-side regardless of input. If a user wants more than 200, they need narrower filters (or the aggregate report flow described in Section D).
|
||||
|
||||
### C.2 Frontend route and components
|
||||
|
||||
Route: `/analyzer/tickets` — the new list view. Becomes the primary entry point for the analyzer feature; update wulf-pulse navigation accordingly.
|
||||
|
||||
Layout, top to bottom:
|
||||
|
||||
**Filter bar** (sticky on scroll):
|
||||
|
||||
- **Period selector**: shadcn `ToggleGroup` with options Today / Yesterday / This Week / Last Week / 30d / 60d / Custom. Custom opens a date range picker (use shadcn `Calendar` component pattern already present in wulf-pulse if any; else add).
|
||||
- **Client filter**: searchable multi-select dropdown. Source: Pulse companies table where `Category = Recurring Revenue Customer`.
|
||||
- **Issue type filter**: multi-select dropdown of distinct values from Pulse tickets table.
|
||||
- **Queue filter**: multi-select dropdown.
|
||||
- **Status filter**: multi-select dropdown, default to non-Complete statuses.
|
||||
- **Analyzed filter**: segmented control — Any / Analyzed / Not Analyzed / Has New Activity (stale).
|
||||
- **Needs Review toggle**: single checkbox, surfaces `needs_human_review = true`.
|
||||
- **Active filter chips**: show currently-applied non-default filters as removable chips below the filter bar so users can see/clear at a glance.
|
||||
|
||||
**Result count and bulk actions bar**:
|
||||
|
||||
- Left: "Showing X of Y tickets" plus the active filter summary
|
||||
- Right: bulk action buttons (disabled until selection is non-empty):
|
||||
- "Analyze N tickets" (only enabled if any selected tickets are not analyzed or are stale)
|
||||
- "Generate aggregate report" (only enabled if all selected tickets are analyzed and current; otherwise tooltip explains why)
|
||||
|
||||
**Table**: shadcn `Table`. Columns:
|
||||
|
||||
| Col | Width | Notes |
|
||||
|---|---|---|
|
||||
| Checkbox | fixed | header has select-all-on-page; "Select all matching filters" appears as an inline action when the page is fully selected |
|
||||
| Analyzed | small | dot indicator: ⚪ none / 🟢 current / 🟡 stale; tooltip shows analysis date |
|
||||
| Ticket # | small | links to ticket detail in wulf-pulse |
|
||||
| Title | flex | truncate with tooltip |
|
||||
| Client | medium | |
|
||||
| Status | small | colored badge |
|
||||
| Priority | small | |
|
||||
| Queue | small | |
|
||||
| Age | small | "3d 4h" format |
|
||||
| Last activity | small | relative time |
|
||||
| Assigned | small | |
|
||||
| Action | fixed | "Analyze" or "View analysis" button per-row |
|
||||
|
||||
**Empty states**:
|
||||
|
||||
- No tickets matched filters: friendly message + "Clear filters" button
|
||||
- No tickets in the entire date range: explicit different message (the sync may be broken)
|
||||
|
||||
**Pagination**: cursor or offset, match whatever wulf-pulse already uses. 50 per page default.
|
||||
|
||||
### C.3 Bulk selection mechanics
|
||||
|
||||
- Page checkbox selects all rows currently visible
|
||||
- When all visible rows are selected, an inline banner appears: "All N tickets on this page selected. Select all M tickets matching filters?"
|
||||
- "Select all matching filters" stores the filter criteria (not the IDs) so re-running the query later returns the same logical set
|
||||
- Selection persists across pagination within a single session (localStorage keyed by a session id, NOT by user account)
|
||||
- Selection clears on filter change (warn user with confirm if they have a non-empty selection)
|
||||
|
||||
---
|
||||
|
||||
## Section D — Aggregate trend and documentation-gap analysis
|
||||
|
||||
**ADDS a new feature.** Depends on Section A schema changes and Section C UI being in place.
|
||||
|
||||
This is the highest-value capability of the analyzer because it converts individual ticket analyses into systemic insights.
|
||||
|
||||
### D.1 Architecture: map-reduce, not concatenation
|
||||
|
||||
The implementation is a map-reduce over analyses. **Do not** implement aggregate analysis by concatenating analysis prose and asking a model to find patterns. That approach breaks down at low N and is not verifiable.
|
||||
|
||||
**Map step (fingerprinting):** runs as part of every individual analysis. Produces a structured fingerprint stored on `analyzer_analyses.aggregate_fingerprint`. Cheap (Haiku), one-time-per-analysis cost.
|
||||
|
||||
**Reduce step (aggregate report):** runs on demand when user clicks "Generate aggregate report." Takes N fingerprints, runs SQL aggregations for instant feedback, then runs a single LLM call (Sonnet, possibly Opus for large sets) to produce the narrative report.
|
||||
|
||||
### D.2 Fingerprint schema
|
||||
|
||||
Stored as `aggregate_fingerprint` jsonb on `analyzer_analyses`:
|
||||
|
||||
```ts
|
||||
{
|
||||
// Categorization
|
||||
category: string; // primary category (backup, network, m365, ...)
|
||||
subcategories: string[]; // additional applicable categories
|
||||
ticket_type_inferred: string; // model's classification, may differ from Autotask
|
||||
root_cause_class:
|
||||
| 'configuration_drift'
|
||||
| 'user_error'
|
||||
| 'vendor_issue'
|
||||
| 'hardware_failure'
|
||||
| 'documentation_gap'
|
||||
| 'process_gap'
|
||||
| 'unknown'
|
||||
| 'other';
|
||||
|
||||
// Entities for aggregation
|
||||
client_name: string;
|
||||
vendors_involved: string[];
|
||||
applications_involved: string[];
|
||||
device_classes: string[]; // 'workstation' | 'server' | 'firewall' | etc.
|
||||
|
||||
// Wulf actions and outcomes
|
||||
wulf_actions_taken: string[]; // short verb phrases
|
||||
vendor_cases_opened: number;
|
||||
resolution_path:
|
||||
| 'resolved_by_wulf'
|
||||
| 'resolved_by_vendor'
|
||||
| 'resolved_by_client'
|
||||
| 'unresolved'
|
||||
| 'self_resolved_before_wulf_action';
|
||||
|
||||
// Gap signals — most important for aggregate
|
||||
documentation_gaps_observed: Array<{
|
||||
description: string; // specific, actionable
|
||||
confidence: 'low' | 'medium' | 'high';
|
||||
}>;
|
||||
process_gaps_observed: Array<{
|
||||
description: string;
|
||||
severity: 'low' | 'medium' | 'high';
|
||||
}>;
|
||||
|
||||
// Recurrence signals
|
||||
similar_to_signals: string[]; // free-text descriptions of "this looks like"
|
||||
// patterns — fed into the reduce step
|
||||
tags: string[]; // free-form tags for downstream clustering
|
||||
|
||||
// Metadata
|
||||
generated_by_model: string;
|
||||
generated_at: string;
|
||||
}
|
||||
```
|
||||
|
||||
### D.3 Fingerprint generation in the pipeline
|
||||
|
||||
Add a new pipeline stage after Stage 5 (Persistence): **Stage 6 — Fingerprint**.
|
||||
|
||||
Stage 6 runs even if Opus didn't run. Uses Haiku.
|
||||
|
||||
System prompt:
|
||||
|
||||
```
|
||||
You are extracting a structured fingerprint from a completed ticket analysis
|
||||
to enable cross-ticket aggregation. Your job is precision and consistency,
|
||||
not creativity.
|
||||
|
||||
You will receive:
|
||||
- The Sonnet-tier analysis output (summary, gaps, what_was_done, etc.)
|
||||
- The triage output from Stage 1 (entities, category)
|
||||
- Optionally the Opus-tier updates if deep review ran
|
||||
|
||||
Produce a fingerprint matching this exact schema:
|
||||
|
||||
[paste schema from D.2 here]
|
||||
|
||||
Strict rules:
|
||||
- Use only the listed enum values for root_cause_class and resolution_path.
|
||||
- documentation_gaps_observed and process_gaps_observed should each have at
|
||||
most 5 entries. Quality over quantity. Each entry's description must be
|
||||
specific enough that two different analyses describing the same underlying
|
||||
gap would produce similar text.
|
||||
- Tags should be lowercase, hyphenated, and stable. Prefer reusing common
|
||||
tags (vertafore, ams360, m365-licensing, backup-veeam, etc.) over inventing
|
||||
new ones.
|
||||
- For similar_to_signals, write short observations like "vendor case opened
|
||||
before checking documentation portal" or "status not advanced after customer
|
||||
self-resolution" — patterns the reduce step can cluster.
|
||||
```
|
||||
|
||||
Persist the fingerprint to `analyzer_analyses.aggregate_fingerprint` and `fingerprint_generated_at`. Also create a row in `analyzer_stage_executions` for this stage.
|
||||
|
||||
If fingerprinting fails, do NOT fail the overall analysis. Log the error, leave fingerprint null, and the analysis is still usable — just won't appear in aggregate reports until re-fingerprinted via the backfill script.
|
||||
|
||||
### D.4 New table: `analyzer_aggregate_reports`
|
||||
|
||||
```sql
|
||||
CREATE TABLE analyzer_aggregate_reports (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
generated_by_user_id uuid NOT NULL,
|
||||
generated_at timestamptz DEFAULT now(),
|
||||
|
||||
-- Inputs
|
||||
filter_criteria jsonb NOT NULL, -- snapshot of filter state at generation
|
||||
analysis_ids uuid[] NOT NULL,
|
||||
ticket_count int NOT NULL,
|
||||
|
||||
-- SQL-derived outputs (computed before LLM call)
|
||||
category_distribution jsonb, -- { "backup": 14, "network": 8, ... }
|
||||
client_distribution jsonb, -- { "Seubert": 23, "Hynes": 11, ... }
|
||||
resolution_path_distribution jsonb,
|
||||
root_cause_distribution jsonb,
|
||||
date_range_actual jsonb, -- { earliest, latest } from selected tickets
|
||||
|
||||
-- LLM-derived outputs
|
||||
documentation_gaps jsonb, -- [{gap, frequency, example_ticket_numbers, evidence}]
|
||||
process_gaps jsonb, -- same shape
|
||||
client_patterns jsonb, -- [{client, pattern, frequency, example_tickets}]
|
||||
recurrence_clusters jsonb, -- [{theme, ticket_numbers, summary}]
|
||||
systemic_observations jsonb, -- [{observation, evidence, severity}]
|
||||
recommended_actions jsonb, -- [{action, rationale, priority, type}]
|
||||
-- type: 'documentation' | 'process' | 'training' | 'tooling'
|
||||
|
||||
-- Narrative
|
||||
narrative_summary text, -- markdown, the human-readable report
|
||||
executive_summary text, -- markdown, 3-5 sentence top-of-report version
|
||||
|
||||
-- Metadata
|
||||
total_input_tokens int,
|
||||
total_output_tokens int,
|
||||
estimated_cost_usd numeric(10,4),
|
||||
model_used text,
|
||||
itglue_context_included boolean, -- whether IT Glue doc titles were fed in
|
||||
status text NOT NULL DEFAULT 'complete' -- pending|running|complete|failed
|
||||
);
|
||||
|
||||
CREATE INDEX ON analyzer_aggregate_reports (generated_by_user_id, generated_at DESC);
|
||||
CREATE INDEX ON analyzer_aggregate_reports USING gin (analysis_ids);
|
||||
```
|
||||
|
||||
### D.5 New API endpoints
|
||||
|
||||
```
|
||||
POST /api/analyzer/aggregate-reports
|
||||
```
|
||||
|
||||
Body:
|
||||
```ts
|
||||
{
|
||||
analysisIds: string[]; // explicit analysis IDs to include
|
||||
// OR
|
||||
filterCriteria: { /* same shape as ticket list filters */ };
|
||||
// server resolves to the analyses for matching tickets
|
||||
includeItglueContext: boolean; // default true; whether to feed IT Glue doc titles
|
||||
// into the reduce step for the affected clients
|
||||
reportTitle: string | null;
|
||||
}
|
||||
```
|
||||
|
||||
Behavior:
|
||||
|
||||
- Resolve to a concrete list of analysis IDs. Cap at 100. If filter criteria resolves to >100, return 400 with a count and instruction to narrow.
|
||||
- Validate every analysis has a current `aggregate_fingerprint`. If any are missing, return 400 with the list of unfingerprinted analysis IDs and a hint to run the backfill script.
|
||||
- Validate every selected ticket has a `current` analyzed state (not `stale`). If stale tickets exist, return 400 with the list — user should re-analyze first.
|
||||
- Queue the report generation as a job. Return `{ reportId, jobId, status }`.
|
||||
|
||||
```
|
||||
GET /api/analyzer/aggregate-reports/:id
|
||||
GET /api/analyzer/aggregate-reports # list, paginated, filterable by user
|
||||
# and date range
|
||||
```
|
||||
|
||||
### D.6 Aggregate report pipeline
|
||||
|
||||
When the job runs:
|
||||
|
||||
**Step 1 — SQL aggregations:** compute the four `*_distribution` fields and `date_range_actual` from the fingerprints. Persist to the report row immediately so the UI can show partial results.
|
||||
|
||||
**Step 2 — IT Glue context (conditional):** if `includeItglueContext = true`, fetch the *titles and types* (not bodies) of all IT Glue docs for the unique clients in the selection. This list is fed to the reduce step so the model can distinguish "no runbook exists" from "no runbook was referenced." Cap at 200 doc titles total. Apply the same redaction rules to titles (rare but possible).
|
||||
|
||||
**Step 3 — Reduce LLM call:**
|
||||
|
||||
Model selection logic:
|
||||
- ≤25 fingerprints: Sonnet
|
||||
- 26-100 fingerprints: Sonnet, but only the structured fingerprints (no narratives)
|
||||
- If user explicitly opts in OR cost circuit-breaker allows: Opus
|
||||
|
||||
System prompt:
|
||||
|
||||
```
|
||||
You are a senior MSP analyst identifying patterns across multiple ticket
|
||||
analyses to surface systemic issues.
|
||||
|
||||
You will receive:
|
||||
1. SQL-derived distributions (categories, clients, resolution paths, root causes)
|
||||
2. An array of structured fingerprints, one per analyzed ticket
|
||||
3. Optionally, a list of IT Glue documentation titles for the affected clients
|
||||
|
||||
Your job is to identify patterns the SQL aggregations cannot see — patterns
|
||||
that emerge from the gap descriptions, vendor involvement, recurrence signals,
|
||||
and cross-ticket clustering.
|
||||
|
||||
Be specific. Cite ticket numbers as evidence for every claim. Distinguish
|
||||
between "documentation gap exists" (no doc on this topic per the IT Glue
|
||||
title list) and "documentation not referenced" (doc may exist but wasn't
|
||||
used in resolution).
|
||||
|
||||
Respond ONLY with JSON matching this schema:
|
||||
|
||||
{
|
||||
"documentation_gaps": [
|
||||
{
|
||||
"gap": string,
|
||||
"frequency": number,
|
||||
"example_ticket_numbers": string[],
|
||||
"evidence": string,
|
||||
"itglue_check": "no_doc_exists" | "doc_exists_but_unused" | "unable_to_verify"
|
||||
}
|
||||
],
|
||||
"process_gaps": [
|
||||
{
|
||||
"gap": string,
|
||||
"frequency": number,
|
||||
"severity": "low" | "medium" | "high",
|
||||
"example_ticket_numbers": string[],
|
||||
"evidence": string
|
||||
}
|
||||
],
|
||||
"client_patterns": [
|
||||
{
|
||||
"client": string,
|
||||
"pattern": string,
|
||||
"frequency": number,
|
||||
"example_ticket_numbers": string[]
|
||||
}
|
||||
],
|
||||
"recurrence_clusters": [
|
||||
{
|
||||
"theme": string,
|
||||
"ticket_numbers": string[],
|
||||
"summary": string // markdown, what unifies these
|
||||
}
|
||||
],
|
||||
"systemic_observations": [
|
||||
{
|
||||
"observation": string, // markdown
|
||||
"evidence": string,
|
||||
"severity": "low" | "medium" | "high"
|
||||
}
|
||||
],
|
||||
"recommended_actions": [
|
||||
{
|
||||
"action": string, // markdown, concrete and actionable
|
||||
"rationale": string, // markdown
|
||||
"priority": "low" | "medium" | "high",
|
||||
"type": "documentation" | "process" | "training" | "tooling"
|
||||
}
|
||||
],
|
||||
"executive_summary": string, // markdown, 3-5 sentences
|
||||
"narrative_summary": string // markdown, full prose report
|
||||
}
|
||||
|
||||
Quality bars:
|
||||
- Do not list a documentation_gap unless it appears in 2+ tickets.
|
||||
- Do not list a process_gap unless it appears in 2+ tickets OR has severity=high
|
||||
in at least one.
|
||||
- Recurrence clusters require at least 2 tickets.
|
||||
- Every recommended_action must be specific enough that a person could pick
|
||||
it up tomorrow. "Improve documentation" is rejected; "Create a runbook for
|
||||
AMS360 App Access Key location and integration permission verification" is
|
||||
acceptable.
|
||||
- The narrative_summary follows the same prose formatting rules as individual
|
||||
analyses: markdown, no headers, no filler phrases.
|
||||
```
|
||||
|
||||
User message: structured payload with the SQL distributions, the array of fingerprints, and the IT Glue title list.
|
||||
|
||||
**Step 4 — Persistence:** update the report row with all LLM-derived fields, mark `status = complete`. Insert an `analyzer_stage_executions` row for the reduce step (linked via a `aggregate_report_id` column — add this to `analyzer_stage_executions`):
|
||||
|
||||
```sql
|
||||
ALTER TABLE analyzer_stage_executions ADD COLUMN aggregate_report_id uuid
|
||||
REFERENCES analyzer_aggregate_reports(id) ON DELETE CASCADE;
|
||||
ALTER TABLE analyzer_stage_executions
|
||||
ADD CONSTRAINT analyzer_stage_executions_parent_check
|
||||
CHECK ((analysis_id IS NOT NULL) <> (aggregate_report_id IS NOT NULL));
|
||||
```
|
||||
|
||||
(Either an analysis stage or a report stage, not both, not neither.)
|
||||
|
||||
### D.7 Frontend: aggregate report flow
|
||||
|
||||
**Trigger from ticket list view (Section C):**
|
||||
|
||||
When user has selected tickets and clicks "Generate aggregate report":
|
||||
|
||||
1. Pre-flight check (client-side): are all selected tickets analyzed and current?
|
||||
- If not all analyzed: show modal "X of Y selected tickets aren't analyzed. Analyze them first?" with cost estimate. Run analyses, then proceed.
|
||||
- If any are stale: show modal "X tickets have new activity since last analysis. Re-analyze first or proceed with stale data?"
|
||||
2. Show pre-LLM SQL summary modal:
|
||||
- Ticket count
|
||||
- Category distribution as a small bar chart (Recharts, matching wulf-pulse style)
|
||||
- Client distribution
|
||||
- Date range actual
|
||||
- Estimated LLM cost
|
||||
- "Generate narrative report" button + "Cancel"
|
||||
3. On confirm, kick off the job, navigate to `/analyzer/reports/:id` showing pending state with progress.
|
||||
|
||||
**Aggregate report view at `/analyzer/reports/:id`:**
|
||||
|
||||
Layout, top to bottom:
|
||||
|
||||
1. **Header** — report title (editable), generated by, generated at, ticket count, date range, cost
|
||||
2. **Executive summary** — card with `AnalysisMarkdown`, prominent
|
||||
3. **Distributions row** — three small cards side-by-side: category bar chart, root cause donut, resolution path donut
|
||||
4. **Documentation gaps section** — card with table: gap description / frequency / example tickets (linked) / IT Glue check status. Color-coded by `itglue_check` value.
|
||||
5. **Process gaps section** — similar table, color by severity
|
||||
6. **Client patterns section** — accordion grouped by client
|
||||
7. **Recurrence clusters section** — each cluster as a card with the theme summary and a list of linked ticket numbers
|
||||
8. **Systemic observations** — list with severity colors
|
||||
9. **Recommended actions** — sortable by priority, grouped by type (documentation / process / training / tooling). Each action is a card with action + rationale.
|
||||
10. **Narrative summary** — full markdown render at the bottom for those who want the prose version
|
||||
11. **Footer** — share button (reuses existing share infrastructure), export to markdown button, "Re-run with current data" button
|
||||
|
||||
**List view at `/analyzer/reports`:**
|
||||
|
||||
Simple table of past reports: title, date, generated by, ticket count, cost, link to view. Filterable by date range and generated_by.
|
||||
|
||||
### D.8 Cost guards
|
||||
|
||||
The aggregate report can get expensive at high N. Guards:
|
||||
|
||||
- Hard cap of 100 tickets per report.
|
||||
- If estimated cost > $5.00 (computed from fingerprint payload size + IT Glue context size), require explicit confirmation in the UI with the dollar figure shown.
|
||||
- Track total spend per user per day; soft warn at $20/day, hard block at $50/day with an admin override env var `ANALYZER_DAILY_COST_OVERRIDE_USERS` (comma-separated user IDs).
|
||||
- Persist cost-guard decisions to a small audit log table `analyzer_cost_audit` (user, action, estimated_cost, decision, timestamp) so we can review patterns later.
|
||||
|
||||
---
|
||||
|
||||
## Build order
|
||||
|
||||
Strict dependency order. Do not skip ahead.
|
||||
|
||||
1. **Section A — schema additions.** Migrations land first. `analyzer_stage_executions`, `source_snapshot` and `aggregate_fingerprint` columns. Backfill script for existing analyses (don't run yet — just have it ready).
|
||||
|
||||
2. **Update existing pipeline to write to new tables.** Every stage now inserts into `analyzer_stage_executions`. Stage 0 output saves to `source_snapshot`. Run against existing test fixtures to confirm nothing regressed.
|
||||
|
||||
3. **Section B — prose formatting.** Stage 3 prompt update + frontend `AnalysisMarkdown` component. Test against the T20260424.0045 fixture and a couple new samples. This is independent of everything else and can be committed as a small standalone PR.
|
||||
|
||||
4. **Section D.3 — fingerprint generation as Stage 6.** Add to pipeline. Run backfill script once on existing analyses. From this point forward every new analysis automatically produces a fingerprint.
|
||||
|
||||
5. **Section C — browse/filter UI.** New API endpoint, new route, table UI, bulk selection. This is the biggest UI surface — budget appropriately.
|
||||
|
||||
6. **Section D.4–D.7 — aggregate report feature.** Endpoints, pipeline, report view, list view.
|
||||
|
||||
7. **Section D.8 — cost guards.** Land before opening the feature beyond yourself.
|
||||
|
||||
8. **Documentation update** — operator runbook gains sections on aggregate reports, fingerprint backfill, cost guard overrides, and how to read the stage execution history when debugging a bad analysis.
|
||||
|
||||
For each phase, confirm with me before moving to the next. Phase 1 (schema) and Phase 5 (browse UI) are the highest-risk for needing iteration; pause after each for review.
|
||||
|
||||
---
|
||||
|
||||
## Critical correctness notes (additions to existing list)
|
||||
|
||||
- **Fingerprint enums must be enforced at parse time.** The Stage 6 Zod schema must use Zod's `enum()` for `root_cause_class` and `resolution_path`. If the model returns a value outside the enum, retry once with the parse error.
|
||||
|
||||
- **Aggregate report inputs must all be from the same fingerprint schema version.** If you change the fingerprint schema in the future, add a `fingerprint_schema_version` field and refuse to mix versions in a single report. Migration story: re-run fingerprinting on affected analyses before allowing them in new reports.
|
||||
|
||||
- **Never include unredacted IT Glue content in fingerprints, stage executions, or aggregate reports.** Redaction happens before any persistence, not just before LLM calls. This was already a rule for Phase 1; reaffirming it because the surface area is now larger.
|
||||
|
||||
- **Aggregate reports are not real-time.** Show timestamps prominently. A report generated yesterday does not reflect today's tickets. Add a "Re-run with current data" button to the report view that creates a new report with the same filter criteria as a clone.
|
||||
|
||||
- **Fingerprinting cost is real but bounded.** Haiku at ~$0.001 per fingerprint × hundreds of analyses adds up. Monitor. If it becomes meaningful, consider running fingerprinting only when the analysis crosses a complexity threshold and using a deterministic SQL-based fingerprint for low-complexity tickets.
|
||||
|
|
@ -506,3 +506,291 @@ analyze without typing the URL.
|
|||
| 7 | 128 | clean | share email via existing SMTP transport |
|
||||
| 8 | 128 | clean | operator runbook + README link |
|
||||
| 9 | 128 | clean | browse page + analysis-view formatting + nav entry |
|
||||
|
||||
---
|
||||
|
||||
# Phase 2 (cross-ticket analysis)
|
||||
|
||||
Spec: `docs/ticket-analyzer-phase2-spec.md`. Eight sub-phases delivered as
|
||||
one Phase 2 push.
|
||||
|
||||
## 2.1 — Schema additions
|
||||
|
||||
**Delivered**
|
||||
|
||||
- Migration 070: `analyzer_stage_executions` table (per-stage I/O for
|
||||
every analyzer run, including failed attempts) + three columns on
|
||||
`analyzer_analyses`: `source_snapshot`, `aggregate_fingerprint`,
|
||||
`fingerprint_generated_at`.
|
||||
- `model_traces` column annotated with a `LEGACY` `COMMENT ON COLUMN`
|
||||
for the SQL side and a `// LEGACY` doc comment in TS — kept for
|
||||
back-compat until aggregate reports have soaked.
|
||||
- Zod schemas: `AggregateFingerprint`, `StageName`, `StageExecution`
|
||||
(read-back), `StageExecutionRecord` (write-time interface).
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- `analyzer_stage_executions.analysis_id` starts NOT NULL in 070.
|
||||
Migration 071 (Phase 2.6) relaxes it and adds the
|
||||
mutually-exclusive CHECK with `aggregate_report_id`.
|
||||
- 070 is idempotent (`IF NOT EXISTS` on every object) so re-running
|
||||
against an already-applied DB is safe.
|
||||
|
||||
## 2.2 — Pipeline writes to stage executions
|
||||
|
||||
**Delivered**
|
||||
|
||||
- `recordedStage(meta, fn, callbacks, outputSelector)` helper in
|
||||
`pipeline.ts` wraps each stage call, emits a `StageExecutionRecord`
|
||||
on success or failure (re-throws after recording). Pipeline wires
|
||||
it into Stage 1 (triage), Stage 3 (analyze), Stage 4 (deep_review).
|
||||
Stage 0 (preprocess) and Stage 2 (itglue) are recorded inline since
|
||||
they're not LLM calls.
|
||||
- Pipeline result now includes `triage_response`, `sonnet_response`,
|
||||
`opus_response` for downstream stages (Stage 6 fingerprint).
|
||||
- Worker's `runJob` collects records via `onStageRecord` callback,
|
||||
bulk-inserts them after `insertAnalysis` succeeds. On pipeline
|
||||
throw, worker captures the preprocessed bundle via `onPreprocessed`
|
||||
callback, persists a `status='failed'` analyzer_analyses row with
|
||||
source_snapshot intact, and bulk-inserts the partial stage records
|
||||
linked to it.
|
||||
- New persistence functions: `bulkInsertStageExecutions`,
|
||||
`insertFailedAnalysis`, `updateAnalysisFingerprint`.
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Failure-tolerant audit**: spec says "Every stage that runs MUST
|
||||
insert a row, including stages that fail." We persist a failed
|
||||
analyzer_analyses row even on pipeline crash so the partial stage
|
||||
records have a parent. Without this the FK would be orphaned.
|
||||
- **Single bulk insert**: ~5–6 stage rows per pipeline run. One
|
||||
multi-VALUES INSERT is fast enough; no need for COPY.
|
||||
- **`model_traces` double-write retained**: the legacy column still
|
||||
receives the old payload. Drop it in a future migration once
|
||||
aggregate reports have soaked through prod.
|
||||
|
||||
## 2.3 — Prose formatting
|
||||
|
||||
**Delivered**
|
||||
|
||||
- Stage 3 system prompt updated with the markdown formatting rules
|
||||
from the spec verbatim (banned filler phrases, action-verb
|
||||
emphasis, no headers).
|
||||
- `react-markdown@10`, `remark-gfm@4`, `@tailwindcss/typography@0.5`
|
||||
added. Tailwind 4 plugin registered via `@plugin
|
||||
"@tailwindcss/typography"` in `app/globals.css`.
|
||||
- `<AnalysisMarkdown>` component at `components/analyzer/analysis-markdown.tsx`
|
||||
renders prose with `prose prose-sm dark:prose-invert
|
||||
max-w-none prose-p:leading-7`. Coerces stray model headers into
|
||||
bold paragraphs (the prompt forbids them but defense-in-depth).
|
||||
- `<ProseText>` removed. Summary, Recommended Next Step,
|
||||
next_step_rationale, and post_resolution_analysis all use
|
||||
`<AnalysisMarkdown>`.
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- Tailwind 4 syntax: `@plugin "@tailwindcss/typography"` in CSS, no
|
||||
JS config needed.
|
||||
- Stage 3 prompt change is back-compatible — old analyses with
|
||||
plain-text summaries still render fine through ReactMarkdown.
|
||||
|
||||
## 2.4 — Stage 6 fingerprint + backfill CLI
|
||||
|
||||
**Delivered**
|
||||
|
||||
- `lib/services/analyzer/stages/stage6-fingerprint.ts` — Haiku call
|
||||
with the spec's verbatim system prompt. Server-overrides
|
||||
`generated_by_model` and `generated_at` after parse so the model's
|
||||
guess for those fields can't drift.
|
||||
- Worker integration: after `insertAnalysis` succeeds, run
|
||||
fingerprint with try/catch. Failure logs a warn, fingerprint
|
||||
stays NULL on the row, but the analysis is still complete and
|
||||
usable. The fingerprint stage record is added to the bulk insert
|
||||
whether it succeeded or failed.
|
||||
- `scripts/backfill-fingerprints.ts` — idempotent CLI. Reads
|
||||
`analyzer_analyses.model_traces.{triage_response, sonnet_response,
|
||||
opus_response}` (which Phase 1 was already storing), runs Stage 6,
|
||||
writes `aggregate_fingerprint`. Supports `--dry-run` and
|
||||
`--limit=N`. Skips analyses where model_traces is incomplete.
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- Stage 6 input is just the analysis content (triage + sonnet +
|
||||
optional opus). No `pre` payload needed — fingerprinting is about
|
||||
the produced *analysis*, not the source ticket.
|
||||
- Backfill processes oldest-first (triggered_at ASC). Lets us
|
||||
observe a few rounds before chewing through hundreds.
|
||||
- Stage 6 failure is non-fatal. Spec: "If fingerprinting fails, do
|
||||
NOT fail the overall analysis."
|
||||
|
||||
## 2.5 — Browse / filter UI rebuild
|
||||
|
||||
**Delivered**
|
||||
|
||||
- New endpoint: `GET /api/analyzer/tickets` (replaces the simpler
|
||||
`GET /api/analyzer/tickets/list` from Phase 1.9). Multi-select
|
||||
CSV-style query params (clientId, issueType, queue, status,
|
||||
priority, assignedTo); analyzed segmented filter (any/yes/no/stale);
|
||||
needsReview toggle; search; sort.
|
||||
- `/api/analyzer/tickets/filter-options` extended with queues,
|
||||
statuses, priorities, resources (joined to "has at least one
|
||||
ticket" so the dropdowns aren't padded).
|
||||
- New `<MultiSelect>` component at `components/ui/multi-select.tsx`
|
||||
— Popover + checkbox list with optional search box (auto-shown
|
||||
above 8 options). One trigger + one popover, no shadcn Command
|
||||
dependency.
|
||||
- `/analyzer/tickets` page rebuilt:
|
||||
- Sticky filter bar with period pills, multi-selects, search,
|
||||
analyzed segmented, needs-review checkbox, sort
|
||||
- Active-filter chips (click to clear individual filter)
|
||||
- Bulk selection persisted via localStorage (key
|
||||
`analyzer:ticket-selection:v1`) — survives pagination
|
||||
- "Analyze N selected" — sequential job queue, forces re-analyze
|
||||
on `stale` rows
|
||||
- "Generate aggregate report" — routes to `/analyzer/reports/new`;
|
||||
only enabled when all selected are `current`
|
||||
- Top nav reorganized: Browse Tickets / Aggregate Reports / Needs
|
||||
Review under "Analyzer".
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Staleness via `last_activity_date > completed_at`**, not
|
||||
content-hash compare. The spec lets either; the date heuristic is
|
||||
good enough and avoids per-row preprocessing on 50-row paginated
|
||||
responses.
|
||||
- **`MultiSelect` is a one-popover-per-instance design** — multiple
|
||||
popovers can be open across the bar. Acceptable; matches how
|
||||
Linear / Vercel's table filters behave.
|
||||
- **No "Select all matching filters" semantic**. Selection is an
|
||||
explicit per-row action stored as ticket numbers in localStorage.
|
||||
Filter-level selection adds significant complexity (server has to
|
||||
resolve filter→IDs, two-modes everywhere). Skipped for V1; the
|
||||
spec's intent (don't lose selection on pagination) is met.
|
||||
- **Bulk Analyze is sequential, not parallel.** N concurrent calls
|
||||
would all hit `claimQueuedJob` and the worker would process them
|
||||
one at a time anyway (single in-process worker). Sequential POSTs
|
||||
are more honest about that.
|
||||
|
||||
## 2.6 — Aggregate reports
|
||||
|
||||
**Delivered**
|
||||
|
||||
- Migration 071: `analyzer_aggregate_reports` table + ALTER on
|
||||
`analyzer_stage_executions` to drop NOT NULL on `analysis_id`,
|
||||
add `aggregate_report_id` FK, add the
|
||||
`analyzer_stage_executions_parent_check` CHECK constraint
|
||||
(`(analysis_id IS NOT NULL) <> (aggregate_report_id IS NOT NULL)`).
|
||||
- `lib/services/analyzer/stages/aggregate-reduce.ts` — Sonnet (Opus
|
||||
opt-in) reduce stage with the spec's verbatim system prompt and
|
||||
`AggregateReduceResponse` Zod schema.
|
||||
- `lib/services/analyzer/aggregate-persistence.ts` — `createAggregateReport`,
|
||||
`getAggregateReport`, `listAggregateReports`, `runAggregateReport`,
|
||||
`bulkInsertReportStageExecutions`. The runner is fire-and-forget
|
||||
(called via `void runAggregateReport(id)` from the POST endpoint);
|
||||
it persists distributions immediately so the UI can show partial
|
||||
results during the LLM call.
|
||||
- IT Glue context fetcher: per-client `findOrganizationByName` +
|
||||
`getFlexibleAssets`, capped at 200 doc titles total per spec.
|
||||
Failure tolerant — per-client errors don't fail the report.
|
||||
- API endpoints:
|
||||
- `POST /api/analyzer/aggregate-reports` — validates (≤100,
|
||||
fingerprint exists, not stale), creates pending row, fires runner
|
||||
- `GET /api/analyzer/aggregate-reports/:id` — full report row
|
||||
(UI polls this every 3s while pending/running)
|
||||
- `GET /api/analyzer/aggregate-reports` — paginated list
|
||||
- Pages:
|
||||
- `/analyzer/reports/new?ids=T...,T...` — pre-flight: shows
|
||||
selected tickets, options (title, IT Glue context toggle),
|
||||
Generate button
|
||||
- `/analyzer/reports/[id]` — pending → distributions → completed.
|
||||
Sections: header, executive summary, four distribution mini-bar
|
||||
cards, documentation gaps (with `itglue_check` color tone),
|
||||
process gaps (severity tone), recurrence clusters, recommended
|
||||
actions (sorted by priority), narrative summary.
|
||||
- `/analyzer/reports` — table list of past reports
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Fire-and-forget runner**, no separate worker module. The POST
|
||||
endpoint kicks `void runAggregateReport(id)`; updates land in the
|
||||
row when the LLM call completes. Frontend polls. Avoids adding a
|
||||
second polling worker alongside `analyzerWorker`.
|
||||
- **Stage names reused for aggregate sub-stages.** The CHECK constraint
|
||||
on `analyzer_stage_executions.stage` enumerates the per-analysis
|
||||
stage names. Aggregate sub-stages (SQL aggregation, IT Glue context,
|
||||
reduce LLM) are recorded with `stage='analyze'` / `'itglue'` plus
|
||||
`aggregate_report_id` set. A future migration could add
|
||||
`aggregate_sql` / `aggregate_reduce` to the enum and re-emit those
|
||||
rows; for now the existing names are good enough for forensics.
|
||||
- **`generated_by_user_id` is TEXT nullable**, not `uuid NOT NULL`
|
||||
per spec. Better Auth's `user.id` is text, and we want the report
|
||||
to remain readable if the generating user is later deleted —
|
||||
matches the pattern from `analyzer_analyses.triggered_by_user_id`.
|
||||
- **Distributions persist before LLM call** so partial-state UI
|
||||
doesn't have to wait the full 30–90s for anything to render.
|
||||
|
||||
## 2.7 — Cost guards
|
||||
|
||||
**Delivered**
|
||||
|
||||
- Migration 072: `analyzer_cost_audit` table.
|
||||
- `lib/services/analyzer/cost-guard.ts`:
|
||||
`estimateAggregateReportCost`, `getUserDailySpend`, `evaluateCost`,
|
||||
`recordCostAuditDecision`. Thresholds: `REQUIRES_CONFIRMATION_USD = 5`,
|
||||
`SOFT_WARN_DAILY_USD = 20`, `HARD_BLOCK_DAILY_USD = 50`.
|
||||
- `evaluateCost` produces a four-state decision (`approved` /
|
||||
`requires_confirmation` / `blocked` / `overridden`) plus boolean
|
||||
`softWarn`/`hardBlocked`/`requiresConfirmation`/`isOverride`
|
||||
fields the API can return for UX.
|
||||
- POST `/api/analyzer/aggregate-reports` enforces:
|
||||
- `requires_confirmation` → 400 with `requiresConfirmation:true,
|
||||
estimatedCost, dailySpendBefore` so the frontend can show
|
||||
`confirm()` and re-POST with `confirmedCost:true`.
|
||||
- `blocked` → 403 with the daily spend in the body.
|
||||
- Every decision (including `approved`) writes a row to
|
||||
`analyzer_cost_audit`.
|
||||
- Override env var `ANALYZER_DAILY_COST_OVERRIDE_USERS`
|
||||
(comma-separated user ids).
|
||||
- Frontend new-report page: catches `requiresConfirmation`,
|
||||
shows `window.confirm()` with the dollar figure, retries with
|
||||
`confirmedCost: true`.
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Cost estimate is char/4 → tokens × Sonnet pricing.** Crude but
|
||||
pessimistic in the right direction. At 100 tickets the estimate
|
||||
comes in under $0.50 — far below the $5 threshold — so the
|
||||
confirmation modal almost never fires in practice. Ceiling exists
|
||||
to catch payload bloat / Opus-opt-in scenarios.
|
||||
- **Daily window is trailing 24h, not "today UTC".** Avoids
|
||||
midnight-edge-of-day reset gaming; rolling window is what the
|
||||
spec calls "$X/day" naturally.
|
||||
- **Soft warn at $20/day is informational only.** Fields exposed
|
||||
in the cost evaluation; UI can choose to surface, but the API
|
||||
doesn't refuse to proceed. Hard block at $50/day is the only
|
||||
enforcement.
|
||||
|
||||
## 2.8 — Documentation
|
||||
|
||||
**Delivered**
|
||||
|
||||
- `docs/wulf-pulse-ticket-analyzer-runbook.md` — added Phase 2
|
||||
sections covering stage execution forensics, fingerprint backfill,
|
||||
aggregate report flow + SQL queries, cost guard configuration +
|
||||
override env var, and the rebuilt browse UI behavior.
|
||||
- This file — the per-sub-phase notes above.
|
||||
|
||||
---
|
||||
|
||||
## Status after Phase 2
|
||||
|
||||
| Phase | Tests | tsc | Notes |
|
||||
|---|---|---|---|
|
||||
| 2.1 | 128 | clean | schema (070) |
|
||||
| 2.2 | 128 | clean | stage_executions writes + failure-tolerant persistence |
|
||||
| 2.3 | 128 | clean | markdown rendering + Stage 3 prompt |
|
||||
| 2.4 | 128 | clean | Stage 6 fingerprint + backfill CLI |
|
||||
| 2.5 | 128 | clean | browse UI rebuild |
|
||||
| 2.6 | 128 | clean | aggregate reports (071, runner, 3 endpoints, 3 pages) |
|
||||
| 2.7 | 128 | clean | cost guards (072, audit log, threshold gating) |
|
||||
| 2.8 | 128 | clean | runbook + build notes |
|
||||
|
|
|
|||
|
|
@ -316,3 +316,172 @@ SELECT round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd,
|
|||
If any of these become a real operational problem, file the work — the
|
||||
stubs are intentional and called out in
|
||||
`wulf-pulse-ticket-analyzer-build-notes.md`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 additions
|
||||
|
||||
### Stage execution history (per-analysis forensics)
|
||||
|
||||
Migration 070 introduced `analyzer_stage_executions`. Every stage of the
|
||||
pipeline (preprocess, triage, itglue, analyze, deep_review, fingerprint)
|
||||
now writes a row including the input it saw and the output it produced.
|
||||
Failed stages get a row with `error_message` populated.
|
||||
|
||||
Read it like this:
|
||||
|
||||
```sql
|
||||
-- Full per-stage trace for one analysis
|
||||
SELECT stage_order, stage, model_id,
|
||||
input_tokens, output_tokens, latency_ms,
|
||||
error_message
|
||||
FROM analyzer_stage_executions
|
||||
WHERE analysis_id = '<analysis-id>'
|
||||
ORDER BY stage_order;
|
||||
|
||||
-- Inspect a single stage's full input/output
|
||||
SELECT input_payload, output_payload
|
||||
FROM analyzer_stage_executions
|
||||
WHERE analysis_id = '<analysis-id>' AND stage = 'analyze';
|
||||
```
|
||||
|
||||
The legacy `analyzer_analyses.model_traces` JSONB column is preserved for
|
||||
back-compat. New analyses double-write to both. A future migration will
|
||||
drop `model_traces` once aggregate reports have soaked.
|
||||
|
||||
### Fingerprint stage (Stage 6)
|
||||
|
||||
After persistence, the worker runs Haiku-tier fingerprint extraction and
|
||||
writes the result to `analyzer_analyses.aggregate_fingerprint`. Failure
|
||||
is non-fatal — the analysis row stays usable, fingerprint stays NULL.
|
||||
|
||||
Re-fingerprint on demand:
|
||||
|
||||
```sql
|
||||
SELECT id, ticket_number, analysis_version
|
||||
FROM analyzer_analyses
|
||||
WHERE aggregate_fingerprint IS NULL
|
||||
AND status = 'complete'
|
||||
ORDER BY triggered_at ASC;
|
||||
```
|
||||
|
||||
Or run the backfill script (idempotent — the SQL filter skips already-fingerprinted rows):
|
||||
|
||||
```bash
|
||||
npx tsx scripts/backfill-fingerprints.ts # process all missing
|
||||
npx tsx scripts/backfill-fingerprints.ts --limit=50 # cap work
|
||||
npx tsx scripts/backfill-fingerprints.ts --dry-run # show what would run
|
||||
```
|
||||
|
||||
The script reconstructs the Stage 6 input from
|
||||
`analyzer_analyses.model_traces.{triage_response, sonnet_response, opus_response}`.
|
||||
If a row's model_traces is missing those fields (very old format) the script
|
||||
logs `[skip]` for it and moves on.
|
||||
|
||||
Cost: Haiku at ~$0.001 per fingerprint. 1000 analyses ≈ $1.
|
||||
|
||||
### Aggregate reports
|
||||
|
||||
Migration 071 added `analyzer_aggregate_reports` and relaxed
|
||||
`analyzer_stage_executions.analysis_id` to be nullable; rows now carry
|
||||
either an `analysis_id` or an `aggregate_report_id` (CHECK enforces
|
||||
exactly one).
|
||||
|
||||
Workflow:
|
||||
|
||||
1. User selects tickets on `/analyzer/tickets` and clicks
|
||||
**Generate aggregate report**.
|
||||
2. POST `/api/analyzer/aggregate-reports` validates: ≤100 tickets, all
|
||||
have a fingerprint, none are stale. Inserts a 'pending' row, fires
|
||||
the runner via `void runAggregateReport(id)`, returns immediately.
|
||||
3. Runner does:
|
||||
- Loads fingerprints
|
||||
- Computes SQL distributions (categories, clients, root cause,
|
||||
resolution path) and writes them to the row immediately
|
||||
- Fetches IT Glue doc titles for affected clients (if requested)
|
||||
- Calls Sonnet (Opus opt-in) with a structured payload
|
||||
- Writes the LLM-derived fields, marks `status='complete'`
|
||||
4. UI polls `GET /api/analyzer/aggregate-reports/:id` every 3s while
|
||||
pending/running.
|
||||
|
||||
Inspect a report's sub-stage trace (linked via `aggregate_report_id`):
|
||||
|
||||
```sql
|
||||
SELECT stage_order, stage, model_id, latency_ms, error_message
|
||||
FROM analyzer_stage_executions
|
||||
WHERE aggregate_report_id = '<report-id>'
|
||||
ORDER BY stage_order;
|
||||
```
|
||||
|
||||
Daily spend on aggregate reports:
|
||||
|
||||
```sql
|
||||
SELECT date_trunc('day', generated_at) AS day,
|
||||
count(*) AS reports,
|
||||
round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd
|
||||
FROM analyzer_aggregate_reports
|
||||
WHERE generated_at > now() - interval '14 days'
|
||||
GROUP BY 1
|
||||
ORDER BY 1 DESC;
|
||||
```
|
||||
|
||||
### Cost guards (Phase 2.7)
|
||||
|
||||
Three thresholds enforce per-user daily spend:
|
||||
|
||||
- **$5 per request** — a single aggregate report estimated above $5
|
||||
triggers a confirmation modal. Frontend re-POSTs with
|
||||
`confirmedCost: true` if the user clicks through.
|
||||
- **$20 per user/day** — soft warn. Surfaced via `softWarn:true` in the
|
||||
cost evaluation; no enforcement.
|
||||
- **$50 per user/day** — hard block. POST returns 403 unless the user
|
||||
is in `ANALYZER_DAILY_COST_OVERRIDE_USERS` (comma-separated list of
|
||||
Better Auth user ids).
|
||||
|
||||
Daily spend is computed as the trailing 24h sum of
|
||||
`analyzer_analyses.estimated_cost_usd` + `analyzer_aggregate_reports.estimated_cost_usd`
|
||||
for the user.
|
||||
|
||||
Every gating decision writes a row to `analyzer_cost_audit`:
|
||||
|
||||
```sql
|
||||
SELECT created_at, user_id, action, estimated_cost,
|
||||
daily_spend_before, decision, decision_reason
|
||||
FROM analyzer_cost_audit
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 50;
|
||||
```
|
||||
|
||||
To grant a user override capability:
|
||||
|
||||
```bash
|
||||
# in ~/projects_env/wulf-pulse.env
|
||||
ANALYZER_DAILY_COST_OVERRIDE_USERS=user_id_1,user_id_2
|
||||
```
|
||||
|
||||
Restart `pulse-app` to pick up the change.
|
||||
|
||||
### Browse / filter ticket list
|
||||
|
||||
`/analyzer/tickets` rebuilt as the primary entry point in Phase 2.5:
|
||||
|
||||
- Multi-select for client, issue type, queue, status, priority, assignee
|
||||
- Sticky filter bar with period chips (today/yesterday/this+last week,
|
||||
30d/60d, custom range, all time)
|
||||
- Active-filter chips below the bar; click to remove
|
||||
- Bulk row selection persisted in localStorage
|
||||
(`analyzer:ticket-selection:v1`) — survives pagination but is
|
||||
session-scoped (no user id baked in)
|
||||
- Bulk "Analyze N selected" sequentially queues jobs for each ticket
|
||||
(forces re-analyze on stale; analyzes from scratch on un-analyzed)
|
||||
- Bulk "Generate aggregate report" routes selected tickets to
|
||||
`/analyzer/reports/new`; only enabled when all selected are
|
||||
`analyzedState === 'current'`
|
||||
|
||||
**Staleness heuristic**: a ticket's `analyzedState` is computed as
|
||||
`stale` when `tickets.last_activity_date > latest_analysis.completed_at`.
|
||||
The Phase 2 spec calls for content-hash-based comparison; that requires
|
||||
either caching the current hash on the tickets row (sync change) or
|
||||
computing it on read for the visible page (slow). The date heuristic
|
||||
gets ~95% of the value at zero compute cost; revisit when there's real
|
||||
load signal.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue