wulf-pulse/docs/ticket-analyzer-phase2-spec.md
lorentz bd3401df1c feat(analyzer): Phase 2 — full stage persistence, fingerprints, aggregate reports, cost guards
Eight sub-phases per docs/ticket-analyzer-phase2-spec.md:

2.1 Schema (migration 070): analyzer_stage_executions table; source_snapshot,
    aggregate_fingerprint, fingerprint_generated_at columns on analyzer_analyses.
    model_traces marked LEGACY (kept for back-compat).
2.2 Every pipeline stage records a row to analyzer_stage_executions, success
    or failure. Worker persists a status='failed' analyzer_analyses row when
    the pipeline throws so partial stage records have a parent. Pipeline
    exposes raw triage/sonnet/opus responses for downstream stages.
2.3 Stage 3 prompt updated with markdown formatting rules + banned filler
    phrases. Added react-markdown + remark-gfm + @tailwindcss/typography.
    New <AnalysisMarkdown> component replaces <ProseText>; coerces stray
    headers to bold paragraphs.
2.4 Stage 6 fingerprint (Haiku) runs after persistence, failure-tolerant.
    scripts/backfill-fingerprints.ts reconstructs Stage 6 input from the
    legacy model_traces blob.
2.5 Browse UI rebuild at /analyzer/tickets: multi-select for client/issue/
    queue/status/priority/assignee, sticky filter bar, active-filter chips,
    bulk selection persisted via localStorage, "Analyze N selected" +
    "Generate aggregate report" actions. New <MultiSelect> primitive.
    Staleness uses last_activity_date > completed_at heuristic per spec C.1.
2.6 Aggregate reports (migration 071): runner is fire-and-forget, persists
    SQL distributions immediately so UI shows partial state during the
    Sonnet reduce call. Three endpoints, three pages (/analyzer/reports[/new
    /:id]). IT Glue context fetcher capped at 200 doc titles.
2.7 Cost guards (migration 072): per-request $5 confirmation, soft-warn at
    $20/day, hard-block at $50/day with ANALYZER_DAILY_COST_OVERRIDE_USERS
    override. Every gating decision audited.
2.8 Runbook + build notes updated.

128 vitest tests passing, tsc clean. Migrations 070/071/072 idempotent
(IF NOT EXISTS). model_traces double-write retained — drop in a future
migration once aggregate reports have soaked.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 14:00:22 -04:00

696 lines
32 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Ticket Analyzer — Phase 2 Spec Additions
This document **adds to and modifies** the existing ticket analyzer spec at `docs/ticket-analyzer-spec.md`. Apply these changes in the order listed. Each section explicitly states whether it adds, replaces, or modifies existing content.
The Phase 2 work covers:
1. Storing all intermediate stage data so analyses are fully reconstructable and aggregate analysis is feasible
2. Improving prose formatting in Summary, Next Step, and Next Step Rationale
3. A browse/filter ticket list UI with bulk selection
4. Aggregate trend and documentation-gap analysis across multiple analyzed tickets
Build order is preserved at the end. Do not start aggregate analysis (#4) before the schema additions (#1) and fingerprinting are in place — backfilling fingerprints across a large analysis history is wasteful.
Before starting, re-read `CLAUDE.md` and the existing analyzer code. Conform to whatever conventions emerged during Phase 1.
---
## Section A — Schema additions for full analysis storage
**ADDS to the existing migrations.** Do not modify existing tables in a destructive way; add new columns and a new table.
### A.1 New table: `analyzer_stage_executions`
Stores the input and output of every pipeline stage so analyses are fully reconstructable for debugging, auditing, and reprocessing with new prompts.
```sql
CREATE TABLE analyzer_stage_executions (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
analysis_id uuid NOT NULL REFERENCES analyzer_analyses(id) ON DELETE CASCADE,
stage text NOT NULL, -- 'preprocess'|'triage'|'itglue'|'analyze'|'deep_review'|'fingerprint'
stage_order int NOT NULL,
model_id text, -- null for non-LLM stages (preprocess, itglue)
input_payload jsonb NOT NULL, -- exactly what was sent to the stage
output_payload jsonb NOT NULL, -- exactly what came back, pre-merge
input_tokens int,
output_tokens int,
latency_ms int,
started_at timestamptz NOT NULL,
completed_at timestamptz NOT NULL,
error_message text
);
CREATE INDEX ON analyzer_stage_executions (analysis_id, stage_order);
CREATE INDEX ON analyzer_stage_executions (stage);
```
Rules:
- Every stage that runs MUST insert a row, including stages that fail. On failure, populate `error_message` and still record what was attempted in `input_payload`.
- The `output_payload` for the IT Glue stage stores the *redacted* document content that was actually passed to the LLM, not raw IT Glue responses. Redaction happens before persistence.
- The deep-review stage's `output_payload` stores the full Opus response including `opus_notes`, NOT just the merged updates. The reasoning is currently being lost.
- The triage stage's `output_payload` stores the full Haiku output even though only some fields drive routing. The categorization and entity extraction are needed for aggregate analysis later.
### A.2 New columns on `analyzer_analyses`
```sql
ALTER TABLE analyzer_analyses ADD COLUMN source_snapshot jsonb;
ALTER TABLE analyzer_analyses ADD COLUMN aggregate_fingerprint jsonb;
ALTER TABLE analyzer_analyses ADD COLUMN fingerprint_generated_at timestamptz;
```
- `source_snapshot` — the pre-processed, tagged event list from Stage 0 (after noise filtering and visibility tagging). Stored canonically because:
1. Pulse sync may change/improve, but the analysis is grounded in what was true at analysis time
2. Ticket notes occasionally get edited or deleted in Autotask
3. Aggregate analysis must operate on a consistent canonical structure across many tickets without re-fetching
- `aggregate_fingerprint` — structured summary used for aggregate analysis (schema in Section D.2 below). Generated as part of the pipeline; described in Section D.
The existing `model_traces` column on `analyzer_analyses` becomes redundant once `analyzer_stage_executions` is populated. Keep it for now but add a comment in code marking it as legacy. A future migration can drop it.
### A.3 Backfill behavior
Existing analyses (if any from Phase 1) will not have `source_snapshot` or `aggregate_fingerprint`. Do NOT attempt automatic backfill. Provide a CLI script `apps/api/scripts/backfill-fingerprints.ts` that operators can run on demand. The script should:
- Accept `--limit` and `--dry-run` flags
- Process analyses missing `aggregate_fingerprint` in batches of 10
- Log progress and total cost
- Be idempotent
Document this script in the operator runbook (added in Phase 1 deliverable 9).
---
## Section B — Prose formatting fixes
**MODIFIES the Stage 3 (Sonnet) system prompt and the frontend analysis view.**
### B.1 Stage 3 system prompt modifications
In the existing Stage 3 system prompt, the schema declaration for `summary`, `next_step`, `next_step_rationale`, and `post_resolution_analysis` should be replaced with:
```
"summary": string, // markdown, 2-4 sentences, neutral status briefing
"next_step": string, // markdown, single concrete action; may use
// **bold** for the action verb and bullet
// sub-steps if multi-part
"next_step_rationale": string, // markdown, 1-2 short paragraphs; if there are
// 3+ competing considerations, use a bulleted list
"post_resolution_analysis": string | null, // markdown, same conventions as above
```
Add this section to the system prompt, immediately after the schema declaration block:
```
Formatting rules for prose fields (summary, next_step, next_step_rationale,
post_resolution_analysis):
- Output is markdown and will be rendered with a markdown renderer. Use **bold**
for emphasis on key terms or actions. Use *italics* sparingly for client-facing
language being quoted. Use bullet lists for enumerable items.
- Do NOT use markdown headers (#, ##, ###). These fields render inside cards that
already have their own headings.
- For summary: write as a neutral status briefing a manager could read in 10
seconds. Lead with the most important fact. Plain language. No hedging.
- For next_step: state the concrete action in the first sentence, with the action
verb in **bold**. If there are sub-steps, follow with a bulleted list. Address
the action to the technician, not the customer.
- For next_step_rationale: open with one short sentence stating the core reason.
Follow with a short paragraph elaborating, OR a bulleted list if there are 3+
distinct considerations. If there's a competing alternative worth noting, name
it explicitly: "Considered X but rejected because..."
- Avoid filler phrases. Banned openings: "It's worth noting that...", "It's
important to understand...", "Based on the available information...",
"After reviewing the ticket...". Get to the point.
```
### B.2 Frontend rendering
Add `react-markdown` and `remark-gfm` to `apps/web/package.json` if not already present.
Create a shared component `apps/web/src/components/analyzer/AnalysisMarkdown.tsx`:
```tsx
import ReactMarkdown from 'react-markdown';
import remarkGfm from 'remark-gfm';
interface Props {
children: string;
className?: string;
}
export function AnalysisMarkdown({ children, className }: Props) {
return (
<div className={`prose prose-sm max-w-none ${className ?? ''}`}>
<ReactMarkdown
remarkPlugins={[remarkGfm]}
components={{
// Safety net: model is told not to use headers, but coerce them
// to paragraphs if it ignores us
h1: ({ children }) => <p className="font-semibold">{children}</p>,
h2: ({ children }) => <p className="font-semibold">{children}</p>,
h3: ({ children }) => <p className="font-semibold">{children}</p>,
h4: ({ children }) => <p className="font-semibold">{children}</p>,
h5: ({ children }) => <p className="font-semibold">{children}</p>,
h6: ({ children }) => <p className="font-semibold">{children}</p>,
}}
>
{children}
</ReactMarkdown>
</div>
);
}
```
Use this component for the four prose fields on the analysis view. Match the existing wulf-pulse Tailwind typography setup; if the project doesn't already use `@tailwindcss/typography`, add it — it's the right plugin for prose-rendering blocks.
---
## Section C — Browse / filter ticket list UI
**ADDS a new view and supporting API endpoint.** Lives in the same nav section as the analyzer feature.
### C.1 New API endpoint
```
GET /api/analyzer/tickets
```
Query parameters:
```
period today | yesterday | this_week | last_week
| last_30d | last_60d | custom
startDate ISO date, only when period=custom
endDate ISO date, only when period=custom
clientId Autotask account id (single or comma-separated)
issueType Ticket Category, Issue Type, or Sub-Issue Type
queue Autotask queue name
status Autotask status (single or comma-separated)
priority Autotask priority (single or comma-separated)
assignedTo Autotask resource id
analyzed any | yes | no | stale (default: any)
needsReview true | false (default: any)
sort created_desc | created_asc
| last_activity_desc | last_activity_asc
| priority (default: last_activity_desc)
limit default 50, max 200
offset default 0
```
Response:
```ts
{
tickets: Array<{
ticketNumber: string;
autotaskTicketId: number;
title: string;
clientName: string;
clientId: number;
status: string;
priority: string;
queue: string;
issueType: string;
subIssueType: string | null;
assignedResourceName: string | null;
createdAtAutotask: string;
lastActivityAtAutotask: string;
ageInDays: number;
// Analysis state
analyzedState: 'none' | 'current' | 'stale';
latestAnalysisId: string | null;
latestAnalysisAt: string | null;
needsHumanReview: boolean;
confidenceScore: number | null;
primaryCategory: string | null; // from fingerprint, if analyzed
}>;
total: number;
filters: { /* echoed for client-side state sync */ };
}
```
Implementation notes:
- Read tickets from the existing Pulse Postgres sync, NOT live from Autotask. List views must be fast.
- LEFT JOIN `analyzer_analyses` on `ticket_number` filtered to the latest version per ticket. Use a window function or subquery — discuss with me before introducing a materialized view.
- `analyzedState` derivation:
- `none` — no `analyzer_analyses` row
- `current` — latest analysis's `content_hash_at_analysis` matches the current Pulse content hash
- `stale` — latest analysis exists but content hash differs (new activity since)
- The current Pulse content hash will require computing on the fly OR caching on the Pulse ticket row. If Pulse already has webhook-driven updates, add a `current_content_hash` column to the Pulse tickets table populated on sync. If not, compute on-read for now and discuss caching strategy after we see real load.
- Cap `limit` at 200 server-side regardless of input. If a user wants more than 200, they need narrower filters (or the aggregate report flow described in Section D).
### C.2 Frontend route and components
Route: `/analyzer/tickets` — the new list view. Becomes the primary entry point for the analyzer feature; update wulf-pulse navigation accordingly.
Layout, top to bottom:
**Filter bar** (sticky on scroll):
- **Period selector**: shadcn `ToggleGroup` with options Today / Yesterday / This Week / Last Week / 30d / 60d / Custom. Custom opens a date range picker (use shadcn `Calendar` component pattern already present in wulf-pulse if any; else add).
- **Client filter**: searchable multi-select dropdown. Source: Pulse companies table where `Category = Recurring Revenue Customer`.
- **Issue type filter**: multi-select dropdown of distinct values from Pulse tickets table.
- **Queue filter**: multi-select dropdown.
- **Status filter**: multi-select dropdown, default to non-Complete statuses.
- **Analyzed filter**: segmented control — Any / Analyzed / Not Analyzed / Has New Activity (stale).
- **Needs Review toggle**: single checkbox, surfaces `needs_human_review = true`.
- **Active filter chips**: show currently-applied non-default filters as removable chips below the filter bar so users can see/clear at a glance.
**Result count and bulk actions bar**:
- Left: "Showing X of Y tickets" plus the active filter summary
- Right: bulk action buttons (disabled until selection is non-empty):
- "Analyze N tickets" (only enabled if any selected tickets are not analyzed or are stale)
- "Generate aggregate report" (only enabled if all selected tickets are analyzed and current; otherwise tooltip explains why)
**Table**: shadcn `Table`. Columns:
| Col | Width | Notes |
|---|---|---|
| Checkbox | fixed | header has select-all-on-page; "Select all matching filters" appears as an inline action when the page is fully selected |
| Analyzed | small | dot indicator: ⚪ none / 🟢 current / 🟡 stale; tooltip shows analysis date |
| Ticket # | small | links to ticket detail in wulf-pulse |
| Title | flex | truncate with tooltip |
| Client | medium | |
| Status | small | colored badge |
| Priority | small | |
| Queue | small | |
| Age | small | "3d 4h" format |
| Last activity | small | relative time |
| Assigned | small | |
| Action | fixed | "Analyze" or "View analysis" button per-row |
**Empty states**:
- No tickets matched filters: friendly message + "Clear filters" button
- No tickets in the entire date range: explicit different message (the sync may be broken)
**Pagination**: cursor or offset, match whatever wulf-pulse already uses. 50 per page default.
### C.3 Bulk selection mechanics
- Page checkbox selects all rows currently visible
- When all visible rows are selected, an inline banner appears: "All N tickets on this page selected. Select all M tickets matching filters?"
- "Select all matching filters" stores the filter criteria (not the IDs) so re-running the query later returns the same logical set
- Selection persists across pagination within a single session (localStorage keyed by a session id, NOT by user account)
- Selection clears on filter change (warn user with confirm if they have a non-empty selection)
---
## Section D — Aggregate trend and documentation-gap analysis
**ADDS a new feature.** Depends on Section A schema changes and Section C UI being in place.
This is the highest-value capability of the analyzer because it converts individual ticket analyses into systemic insights.
### D.1 Architecture: map-reduce, not concatenation
The implementation is a map-reduce over analyses. **Do not** implement aggregate analysis by concatenating analysis prose and asking a model to find patterns. That approach breaks down at low N and is not verifiable.
**Map step (fingerprinting):** runs as part of every individual analysis. Produces a structured fingerprint stored on `analyzer_analyses.aggregate_fingerprint`. Cheap (Haiku), one-time-per-analysis cost.
**Reduce step (aggregate report):** runs on demand when user clicks "Generate aggregate report." Takes N fingerprints, runs SQL aggregations for instant feedback, then runs a single LLM call (Sonnet, possibly Opus for large sets) to produce the narrative report.
### D.2 Fingerprint schema
Stored as `aggregate_fingerprint` jsonb on `analyzer_analyses`:
```ts
{
// Categorization
category: string; // primary category (backup, network, m365, ...)
subcategories: string[]; // additional applicable categories
ticket_type_inferred: string; // model's classification, may differ from Autotask
root_cause_class:
| 'configuration_drift'
| 'user_error'
| 'vendor_issue'
| 'hardware_failure'
| 'documentation_gap'
| 'process_gap'
| 'unknown'
| 'other';
// Entities for aggregation
client_name: string;
vendors_involved: string[];
applications_involved: string[];
device_classes: string[]; // 'workstation' | 'server' | 'firewall' | etc.
// Wulf actions and outcomes
wulf_actions_taken: string[]; // short verb phrases
vendor_cases_opened: number;
resolution_path:
| 'resolved_by_wulf'
| 'resolved_by_vendor'
| 'resolved_by_client'
| 'unresolved'
| 'self_resolved_before_wulf_action';
// Gap signals — most important for aggregate
documentation_gaps_observed: Array<{
description: string; // specific, actionable
confidence: 'low' | 'medium' | 'high';
}>;
process_gaps_observed: Array<{
description: string;
severity: 'low' | 'medium' | 'high';
}>;
// Recurrence signals
similar_to_signals: string[]; // free-text descriptions of "this looks like"
// patterns — fed into the reduce step
tags: string[]; // free-form tags for downstream clustering
// Metadata
generated_by_model: string;
generated_at: string;
}
```
### D.3 Fingerprint generation in the pipeline
Add a new pipeline stage after Stage 5 (Persistence): **Stage 6 — Fingerprint**.
Stage 6 runs even if Opus didn't run. Uses Haiku.
System prompt:
```
You are extracting a structured fingerprint from a completed ticket analysis
to enable cross-ticket aggregation. Your job is precision and consistency,
not creativity.
You will receive:
- The Sonnet-tier analysis output (summary, gaps, what_was_done, etc.)
- The triage output from Stage 1 (entities, category)
- Optionally the Opus-tier updates if deep review ran
Produce a fingerprint matching this exact schema:
[paste schema from D.2 here]
Strict rules:
- Use only the listed enum values for root_cause_class and resolution_path.
- documentation_gaps_observed and process_gaps_observed should each have at
most 5 entries. Quality over quantity. Each entry's description must be
specific enough that two different analyses describing the same underlying
gap would produce similar text.
- Tags should be lowercase, hyphenated, and stable. Prefer reusing common
tags (vertafore, ams360, m365-licensing, backup-veeam, etc.) over inventing
new ones.
- For similar_to_signals, write short observations like "vendor case opened
before checking documentation portal" or "status not advanced after customer
self-resolution" — patterns the reduce step can cluster.
```
Persist the fingerprint to `analyzer_analyses.aggregate_fingerprint` and `fingerprint_generated_at`. Also create a row in `analyzer_stage_executions` for this stage.
If fingerprinting fails, do NOT fail the overall analysis. Log the error, leave fingerprint null, and the analysis is still usable — just won't appear in aggregate reports until re-fingerprinted via the backfill script.
### D.4 New table: `analyzer_aggregate_reports`
```sql
CREATE TABLE analyzer_aggregate_reports (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
generated_by_user_id uuid NOT NULL,
generated_at timestamptz DEFAULT now(),
-- Inputs
filter_criteria jsonb NOT NULL, -- snapshot of filter state at generation
analysis_ids uuid[] NOT NULL,
ticket_count int NOT NULL,
-- SQL-derived outputs (computed before LLM call)
category_distribution jsonb, -- { "backup": 14, "network": 8, ... }
client_distribution jsonb, -- { "Seubert": 23, "Hynes": 11, ... }
resolution_path_distribution jsonb,
root_cause_distribution jsonb,
date_range_actual jsonb, -- { earliest, latest } from selected tickets
-- LLM-derived outputs
documentation_gaps jsonb, -- [{gap, frequency, example_ticket_numbers, evidence}]
process_gaps jsonb, -- same shape
client_patterns jsonb, -- [{client, pattern, frequency, example_tickets}]
recurrence_clusters jsonb, -- [{theme, ticket_numbers, summary}]
systemic_observations jsonb, -- [{observation, evidence, severity}]
recommended_actions jsonb, -- [{action, rationale, priority, type}]
-- type: 'documentation' | 'process' | 'training' | 'tooling'
-- Narrative
narrative_summary text, -- markdown, the human-readable report
executive_summary text, -- markdown, 3-5 sentence top-of-report version
-- Metadata
total_input_tokens int,
total_output_tokens int,
estimated_cost_usd numeric(10,4),
model_used text,
itglue_context_included boolean, -- whether IT Glue doc titles were fed in
status text NOT NULL DEFAULT 'complete' -- pending|running|complete|failed
);
CREATE INDEX ON analyzer_aggregate_reports (generated_by_user_id, generated_at DESC);
CREATE INDEX ON analyzer_aggregate_reports USING gin (analysis_ids);
```
### D.5 New API endpoints
```
POST /api/analyzer/aggregate-reports
```
Body:
```ts
{
analysisIds: string[]; // explicit analysis IDs to include
// OR
filterCriteria: { /* same shape as ticket list filters */ };
// server resolves to the analyses for matching tickets
includeItglueContext: boolean; // default true; whether to feed IT Glue doc titles
// into the reduce step for the affected clients
reportTitle: string | null;
}
```
Behavior:
- Resolve to a concrete list of analysis IDs. Cap at 100. If filter criteria resolves to >100, return 400 with a count and instruction to narrow.
- Validate every analysis has a current `aggregate_fingerprint`. If any are missing, return 400 with the list of unfingerprinted analysis IDs and a hint to run the backfill script.
- Validate every selected ticket has a `current` analyzed state (not `stale`). If stale tickets exist, return 400 with the list — user should re-analyze first.
- Queue the report generation as a job. Return `{ reportId, jobId, status }`.
```
GET /api/analyzer/aggregate-reports/:id
GET /api/analyzer/aggregate-reports # list, paginated, filterable by user
# and date range
```
### D.6 Aggregate report pipeline
When the job runs:
**Step 1 — SQL aggregations:** compute the four `*_distribution` fields and `date_range_actual` from the fingerprints. Persist to the report row immediately so the UI can show partial results.
**Step 2 — IT Glue context (conditional):** if `includeItglueContext = true`, fetch the *titles and types* (not bodies) of all IT Glue docs for the unique clients in the selection. This list is fed to the reduce step so the model can distinguish "no runbook exists" from "no runbook was referenced." Cap at 200 doc titles total. Apply the same redaction rules to titles (rare but possible).
**Step 3 — Reduce LLM call:**
Model selection logic:
- ≤25 fingerprints: Sonnet
- 26-100 fingerprints: Sonnet, but only the structured fingerprints (no narratives)
- If user explicitly opts in OR cost circuit-breaker allows: Opus
System prompt:
```
You are a senior MSP analyst identifying patterns across multiple ticket
analyses to surface systemic issues.
You will receive:
1. SQL-derived distributions (categories, clients, resolution paths, root causes)
2. An array of structured fingerprints, one per analyzed ticket
3. Optionally, a list of IT Glue documentation titles for the affected clients
Your job is to identify patterns the SQL aggregations cannot see — patterns
that emerge from the gap descriptions, vendor involvement, recurrence signals,
and cross-ticket clustering.
Be specific. Cite ticket numbers as evidence for every claim. Distinguish
between "documentation gap exists" (no doc on this topic per the IT Glue
title list) and "documentation not referenced" (doc may exist but wasn't
used in resolution).
Respond ONLY with JSON matching this schema:
{
"documentation_gaps": [
{
"gap": string,
"frequency": number,
"example_ticket_numbers": string[],
"evidence": string,
"itglue_check": "no_doc_exists" | "doc_exists_but_unused" | "unable_to_verify"
}
],
"process_gaps": [
{
"gap": string,
"frequency": number,
"severity": "low" | "medium" | "high",
"example_ticket_numbers": string[],
"evidence": string
}
],
"client_patterns": [
{
"client": string,
"pattern": string,
"frequency": number,
"example_ticket_numbers": string[]
}
],
"recurrence_clusters": [
{
"theme": string,
"ticket_numbers": string[],
"summary": string // markdown, what unifies these
}
],
"systemic_observations": [
{
"observation": string, // markdown
"evidence": string,
"severity": "low" | "medium" | "high"
}
],
"recommended_actions": [
{
"action": string, // markdown, concrete and actionable
"rationale": string, // markdown
"priority": "low" | "medium" | "high",
"type": "documentation" | "process" | "training" | "tooling"
}
],
"executive_summary": string, // markdown, 3-5 sentences
"narrative_summary": string // markdown, full prose report
}
Quality bars:
- Do not list a documentation_gap unless it appears in 2+ tickets.
- Do not list a process_gap unless it appears in 2+ tickets OR has severity=high
in at least one.
- Recurrence clusters require at least 2 tickets.
- Every recommended_action must be specific enough that a person could pick
it up tomorrow. "Improve documentation" is rejected; "Create a runbook for
AMS360 App Access Key location and integration permission verification" is
acceptable.
- The narrative_summary follows the same prose formatting rules as individual
analyses: markdown, no headers, no filler phrases.
```
User message: structured payload with the SQL distributions, the array of fingerprints, and the IT Glue title list.
**Step 4 — Persistence:** update the report row with all LLM-derived fields, mark `status = complete`. Insert an `analyzer_stage_executions` row for the reduce step (linked via a `aggregate_report_id` column — add this to `analyzer_stage_executions`):
```sql
ALTER TABLE analyzer_stage_executions ADD COLUMN aggregate_report_id uuid
REFERENCES analyzer_aggregate_reports(id) ON DELETE CASCADE;
ALTER TABLE analyzer_stage_executions
ADD CONSTRAINT analyzer_stage_executions_parent_check
CHECK ((analysis_id IS NOT NULL) <> (aggregate_report_id IS NOT NULL));
```
(Either an analysis stage or a report stage, not both, not neither.)
### D.7 Frontend: aggregate report flow
**Trigger from ticket list view (Section C):**
When user has selected tickets and clicks "Generate aggregate report":
1. Pre-flight check (client-side): are all selected tickets analyzed and current?
- If not all analyzed: show modal "X of Y selected tickets aren't analyzed. Analyze them first?" with cost estimate. Run analyses, then proceed.
- If any are stale: show modal "X tickets have new activity since last analysis. Re-analyze first or proceed with stale data?"
2. Show pre-LLM SQL summary modal:
- Ticket count
- Category distribution as a small bar chart (Recharts, matching wulf-pulse style)
- Client distribution
- Date range actual
- Estimated LLM cost
- "Generate narrative report" button + "Cancel"
3. On confirm, kick off the job, navigate to `/analyzer/reports/:id` showing pending state with progress.
**Aggregate report view at `/analyzer/reports/:id`:**
Layout, top to bottom:
1. **Header** — report title (editable), generated by, generated at, ticket count, date range, cost
2. **Executive summary** — card with `AnalysisMarkdown`, prominent
3. **Distributions row** — three small cards side-by-side: category bar chart, root cause donut, resolution path donut
4. **Documentation gaps section** — card with table: gap description / frequency / example tickets (linked) / IT Glue check status. Color-coded by `itglue_check` value.
5. **Process gaps section** — similar table, color by severity
6. **Client patterns section** — accordion grouped by client
7. **Recurrence clusters section** — each cluster as a card with the theme summary and a list of linked ticket numbers
8. **Systemic observations** — list with severity colors
9. **Recommended actions** — sortable by priority, grouped by type (documentation / process / training / tooling). Each action is a card with action + rationale.
10. **Narrative summary** — full markdown render at the bottom for those who want the prose version
11. **Footer** — share button (reuses existing share infrastructure), export to markdown button, "Re-run with current data" button
**List view at `/analyzer/reports`:**
Simple table of past reports: title, date, generated by, ticket count, cost, link to view. Filterable by date range and generated_by.
### D.8 Cost guards
The aggregate report can get expensive at high N. Guards:
- Hard cap of 100 tickets per report.
- If estimated cost > $5.00 (computed from fingerprint payload size + IT Glue context size), require explicit confirmation in the UI with the dollar figure shown.
- Track total spend per user per day; soft warn at $20/day, hard block at $50/day with an admin override env var `ANALYZER_DAILY_COST_OVERRIDE_USERS` (comma-separated user IDs).
- Persist cost-guard decisions to a small audit log table `analyzer_cost_audit` (user, action, estimated_cost, decision, timestamp) so we can review patterns later.
---
## Build order
Strict dependency order. Do not skip ahead.
1. **Section A — schema additions.** Migrations land first. `analyzer_stage_executions`, `source_snapshot` and `aggregate_fingerprint` columns. Backfill script for existing analyses (don't run yet — just have it ready).
2. **Update existing pipeline to write to new tables.** Every stage now inserts into `analyzer_stage_executions`. Stage 0 output saves to `source_snapshot`. Run against existing test fixtures to confirm nothing regressed.
3. **Section B — prose formatting.** Stage 3 prompt update + frontend `AnalysisMarkdown` component. Test against the T20260424.0045 fixture and a couple new samples. This is independent of everything else and can be committed as a small standalone PR.
4. **Section D.3 — fingerprint generation as Stage 6.** Add to pipeline. Run backfill script once on existing analyses. From this point forward every new analysis automatically produces a fingerprint.
5. **Section C — browse/filter UI.** New API endpoint, new route, table UI, bulk selection. This is the biggest UI surface — budget appropriately.
6. **Section D.4D.7 — aggregate report feature.** Endpoints, pipeline, report view, list view.
7. **Section D.8 — cost guards.** Land before opening the feature beyond yourself.
8. **Documentation update** — operator runbook gains sections on aggregate reports, fingerprint backfill, cost guard overrides, and how to read the stage execution history when debugging a bad analysis.
For each phase, confirm with me before moving to the next. Phase 1 (schema) and Phase 5 (browse UI) are the highest-risk for needing iteration; pause after each for review.
---
## Critical correctness notes (additions to existing list)
- **Fingerprint enums must be enforced at parse time.** The Stage 6 Zod schema must use Zod's `enum()` for `root_cause_class` and `resolution_path`. If the model returns a value outside the enum, retry once with the parse error.
- **Aggregate report inputs must all be from the same fingerprint schema version.** If you change the fingerprint schema in the future, add a `fingerprint_schema_version` field and refuse to mix versions in a single report. Migration story: re-run fingerprinting on affected analyses before allowing them in new reports.
- **Never include unredacted IT Glue content in fingerprints, stage executions, or aggregate reports.** Redaction happens before any persistence, not just before LLM calls. This was already a rule for Phase 1; reaffirming it because the surface area is now larger.
- **Aggregate reports are not real-time.** Show timestamps prominently. A report generated yesterday does not reflect today's tickets. Add a "Re-run with current data" button to the report view that creates a new report with the same filter criteria as a clone.
- **Fingerprinting cost is real but bounded.** Haiku at ~$0.001 per fingerprint × hundreds of analyses adds up. Monitor. If it becomes meaningful, consider running fingerprinting only when the analysis crosses a complexity threshold and using a deterministic SQL-based fingerprint for low-complexity tickets.