wulf-pulse/docs/wulf-pulse-ticket-analyzer-build-notes.md
lorentz bd3401df1c feat(analyzer): Phase 2 — full stage persistence, fingerprints, aggregate reports, cost guards
Eight sub-phases per docs/ticket-analyzer-phase2-spec.md:

2.1 Schema (migration 070): analyzer_stage_executions table; source_snapshot,
    aggregate_fingerprint, fingerprint_generated_at columns on analyzer_analyses.
    model_traces marked LEGACY (kept for back-compat).
2.2 Every pipeline stage records a row to analyzer_stage_executions, success
    or failure. Worker persists a status='failed' analyzer_analyses row when
    the pipeline throws so partial stage records have a parent. Pipeline
    exposes raw triage/sonnet/opus responses for downstream stages.
2.3 Stage 3 prompt updated with markdown formatting rules + banned filler
    phrases. Added react-markdown + remark-gfm + @tailwindcss/typography.
    New <AnalysisMarkdown> component replaces <ProseText>; coerces stray
    headers to bold paragraphs.
2.4 Stage 6 fingerprint (Haiku) runs after persistence, failure-tolerant.
    scripts/backfill-fingerprints.ts reconstructs Stage 6 input from the
    legacy model_traces blob.
2.5 Browse UI rebuild at /analyzer/tickets: multi-select for client/issue/
    queue/status/priority/assignee, sticky filter bar, active-filter chips,
    bulk selection persisted via localStorage, "Analyze N selected" +
    "Generate aggregate report" actions. New <MultiSelect> primitive.
    Staleness uses last_activity_date > completed_at heuristic per spec C.1.
2.6 Aggregate reports (migration 071): runner is fire-and-forget, persists
    SQL distributions immediately so UI shows partial state during the
    Sonnet reduce call. Three endpoints, three pages (/analyzer/reports[/new
    /:id]). IT Glue context fetcher capped at 200 doc titles.
2.7 Cost guards (migration 072): per-request $5 confirmation, soft-warn at
    $20/day, hard-block at $50/day with ANALYZER_DAILY_COST_OVERRIDE_USERS
    override. Every gating decision audited.
2.8 Runbook + build notes updated.

128 vitest tests passing, tsc clean. Migrations 070/071/072 idempotent
(IF NOT EXISTS). model_traces double-write retained — drop in a future
migration once aggregate reports have soaked.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 14:00:22 -04:00

38 KiB
Raw Blame History

AI Ticket Analyzer — Build Notes

A running journal of the multi-phase build for the AI Ticket Analyzer feature. Captures what was delivered each phase, design decisions worth flagging, and what was deliberately left out. Spec lives in wulf-pulse-ticket-analyzer-prompt.md.

This file is updated after each phase ships.


Phase 1 — Migration + Zod schemas

Delivered

  • migrations/069_create_analyzer_tables.sql — three tables (analyzer_analyses, analyzer_shares, analyzer_jobs) with pgcrypto extension guard, indexes per spec, status CHECK constraints, and FK types corrected to TEXT (not UUID) to match Better Auth's user.id.
  • lib/types/analyzer.ts — Zod schemas for every LLM-stage parsed JSON (TaggedEvent, TriageResponse, DeepAnalysisResponse, OpusResponse), persisted row shapes, job status, API request bodies, and the internal PreprocessedTicket payload that flows through the pipeline.

Decisions worth flagging

  • triggered_by_user_id is nullable + ON DELETE SET NULL (not NOT NULL). An analysis should still be readable in history if the triggering user is later deleted. analyzer_shares.shared_by_user_id is NOT NULL + ON DELETE CASCADE (audit-log style — share rows go with the user).
  • evidence_timestamps typed as string().datetime({offset: true}) (ISO timestamps), not numeric indices. More robust to model hallucination and reads better in the UI.
  • TaggedEvent does NOT support a "two separate events" form for time entries with both Summary + Internal Notes — single event with both fields, per the spec's preference for a cleaner timeline.

Deliberately left out

  • Migration is committed but not applied to any running DB. Postgres only re-applies migrations on first init of a fresh volume; the existing DB needs a manual run of this migration when phase 5 is exercised.

Phase 2 — Stage 0 pre-processor + IT Glue redaction

Delivered

  • lib/services/analyzer/itglue-redact.ts — recursive walk; matches keys against /password|secret|key|token|credential|api[_-]?key/i; replaces values with [REDACTED]. Subtree redaction (key matching auth → no leaves leak), defensive copy, cycle guard ([CIRCULAR] marker).
  • lib/services/analyzer/preprocessor.ts — filters workflow noise + Service Desk Notification rows, tags ticket_create/notes/time entries with actor / actor_type / source / visibility / summary_notes / internal_notes / hours, sorts chronologically, computes sha256 content hash over canonical JSON of (events, status, priority, queue).
  • vitest@^4.1.5 added as devDep with vitest.config.ts setting up the @/ alias. Two test files at this phase: 32 redaction tests + 36 preprocessor tests including the regression run against the T20260424.0045 fixture.
  • Type-aliasing fix in lib/types/analyzer.ts: added export type X = z.infer<typeof X> for ActorType, EventSource, Visibility, Severity, ComplexityTier, TicketType — the Zod-enum const exports alone don't produce a usable TypeScript type.

Decisions worth flagging

  • actor_type is classified by email domain, not author text, per the spec. lorentz@wulfconsulting.com is wulf_tech regardless of how the message reads. Sonnet handles the role nuance at Stage 3.
  • Time entries with no narrative content (no summary, no internal notes) are dropped — a purely numeric entry adds nothing.
  • Vendor-domain allowlist is intentionally short and conservative (Vertafore, Datto, Microsoft, etc.). Misclassifying a customer domain as "vendor" is worse than the default client_contact.

Deliberately left out

  • No real protection for secrets embedded in free text (e.g. a notes field containing the literal string "the password is hunter2"). The redaction guarantee is on field keys, not values. The IT Glue search test documents this contract explicitly so it isn't "fixed" without thought.

Phase 3 — Anthropic SDK setup + Stage 1 (Haiku triage)

Delivered

  • lib/services/llm/{models,pricing,client,call}.ts — model ID constants, per-model rate table (Haiku $1/$5, Sonnet $3/$15, Opus $5/$25 per 1M tokens
    • cache read/write tiers), lazy SDK singleton, generic callLLMStage<T>({model, system, user, schema, maxTokens, client?}) helper that:
    • Marks the system prompt with cache_control: {type: "ephemeral"}
    • Sends NO temperature/top_p/top_k (Opus 4.7 would 400)
    • Strips a single ```json fence before parsing
    • On parse failure, retries once with prior-attempt + error in a follow-up user turn
    • Returns {data, usage, estimated_cost_usd, attempts, raw_response}
  • lib/services/analyzer/stages/stage1-triage.ts — Haiku caller with the spec's verbatim system prompt and a 50KB user-payload cap that drops oldest internal-only events first when oversized.
  • lib/services/analyzer/itglue-search.ts — search facade that runs every result through redact() before returning. Snippets capped at 2000 chars, doc count capped at 10. itglue-aliases.json skeleton for known fuzzy org-name mappings.
  • 32 new tests across pricing/call/Stage 1/IT Glue search.

Decisions worth flagging

  • Manual JSON.parse + Zod validate, not output_config.format / client.messages.parse(). The spec said retry-once on Zod parse failure, and Zod 4 ↔ JSON Schema conversion has edge cases I didn't want to depend on (e.g. .datetime({offset: true}) → JSON Schema format). Manual parse is what the spec asks for and is more transparent.
  • Prompt caching probably won't fire on these stages. Our system prompts are ~1-2 KB (~250-500 tokens); minimum cacheable prefix is 2048 (Sonnet) or 4096 (Haiku/Opus) tokens. The cache_control marker is a no-op below threshold and incurs no cost — left in defensively, but caching is not a meaningful lever for this workload.
  • Redaction is on KEY names, not free text — re-stated for emphasis. The IT Glue search test asserts this contract.
  • itglue-aliases.json is a skeleton with _comment / _example keys documenting the format. Real org-id entries get added when phase 5 wires this into the pipeline.

Deliberately left out

  • No live API integration test. All Stage 1 tests use a mocked Anthropic client. A real-API smoke test belongs in phase 5+ when we run end-to-end.
  • Did not wrap lib/services/itglue-client.ts in a redacting search facade for non-LLM callers. The redaction primitive is ready; non-LLM callers (sync service, data browsers) intentionally have full data — they are not the path that needs protection.

Phase 4 — Pipeline + Job worker (Stages 3, 4, 5)

Delivered

  • lib/services/analyzer/stages/stage3-deep-analysis.ts — Sonnet caller, spec's verbatim prompt, 80KB payload cap.
  • lib/services/analyzer/stages/stage4-deep-reasoning.ts — Opus caller plus pure helpers shouldRunDeepReasoning() and applyOpusUpdates().
  • lib/services/analyzer/data-access.tsloadTicketBundle(ticketNumber) joins tickets / statuses / priorities / queues / companies / contacts / resources / ticket_notes / time_entries and returns the exact RawTicketBundle shape the preprocessor expects. Throws typed TicketNotFoundError.
  • lib/services/analyzer/persistence.tsgetNextAnalysisVersion, findExistingAnalysisByContentHash, insertAnalysis, plus job claimQueuedJob / updateJobStatus / completeJob / failJob / queueJob / getJob. Job claim uses FOR UPDATE SKIP LOCKED so multiple Next.js workers can poll safely.
  • lib/services/analyzer/pipeline.ts — composes Stage 0 → idempotency check → Stage 1 → (Stage 2 if itglue_lookup_needed) → Stage 3 → (Stage 4 if trigger fires AND cost ceiling not reached) → result. Cost circuit breaker trips at $2.00 before Opus, sets needs_human_review=true with a reason. Returns full model_traces for debugging.
  • lib/services/analyzer/worker.ts — singleton with 2-second poll loop; auto-starts in production; opt-in in dev via ANALYZER_WORKER_AUTOSTART=1; skipped under vitest. runJob() exposed for tests + manual triggers. TicketNotFoundError produces a user-facing job error message that fingers the sync as the culprit.
  • 28 new tests bringing the total to 128.

Decisions worth flagging

  • Worker auto-start is more conservative than sync-scheduler.ts. That one auto-starts on any non-browser import (including tests). I gated this one because the worker hits the database AND runs LLM calls — much higher blast radius. To run locally, set ANALYZER_WORKER_AUTOSTART=1.
  • Cost circuit breaker only fires before Opus. Sonnet runs unconditionally even if it would push past $2. Spec wording matches; if the team wants stricter control, the natural place is next to COST_CEILING_USD in pipeline.ts.
  • Idempotency check matches status='complete' only. Failed runs don't poison the cache.
  • IT Glue failures are tolerated. Search throws → analysis continues without context, doesn't fail the run.
  • worker.test.ts casts mockImplementationOnce to as never because vitest's overload resolution fights us when the mocked function has multiple call signatures. Functional, just ugly.

Deliberately left out

  • No live API integration test (still). Phase 5+ is the natural place for an end-to-end smoke test.
  • Stage 3 doesn't have a fixture-driven happy-path test like Stage 1; it's exercised at the orchestration layer via pipeline.test.ts only. Worth adding direct Stage 3 tests later.
  • Migration 069 still not applied — same as phase 1.

Phase 5 — API routes

Delivered

  • Persistence read paths added to lib/services/analyzer/persistence.ts: getAnalysisById, listAnalysesByTicketNumber, listNeedsReview, createShare, plus a shared rowToPersistedAnalysis row mapper.
  • 6 routes under app/api/analyzer/:
    Route Method Returns
    tickets/[ticketNumber]/analyze POST {status, jobId?, existingAnalysisId?}
    tickets/[ticketNumber]/analyses GET {analyses: PersistedAnalysis[]}
    jobs/[jobId] GET {job}
    analyses/[id] GET {analysis}
    analyses/[id]/share POST {share}
    needs-review?limit=&offset= GET {analyses}

Decisions worth flagging

  • Analyze runs preprocess inline. The route does loadTicketBundle → preprocessTicket → findExistingAnalysisByContentHash synchronously to support the spec's existingAnalysisId? immediate response. The worker also runs this — duplicated work, but preprocess is fast (deterministic, one DB load) and the alternative (always queue, frontend polls to discover the short-circuit) is worse UX.
  • Email send deferred to phase 8 per the spec's delivery order. The share route persists the audit row and validates the recipient domain against ALLOWED_SHARE_DOMAINS. Until phase 8, share rows have viewed_at = null indefinitely.
  • requireAuth() everywhere — not requireAdmin(). Any authenticated user can analyze a ticket they have access to. If /needs-review should be admin-only later, swap that one to requirePermission('analyzer', 'review') once the permission map is decided.
  • No new permission entries added to lib/permissions.ts. Adding scoped permissions for a feature still under build risks getting them wrong.
  • Routes are NOT in middleware.ts publicRoutes — they require a session.

Deliberately left out

  • No API route tests. The repo has zero app/api/**/*.test.ts files; the routes are thin orchestration on top of already-tested persistence. Adding integration tests means setting up a test harness for the auth helpers + Postgres, which is a separate effort.

Phase 6 — Frontend pages

Delivered

  • components/analyzer/analyze-button.tsx<AnalyzeButton> with the full state machine: POSTs to the analyze endpoint, navigates straight to an existing analysis if existingAnalysisId came back, otherwise polls /api/analyzer/jobs/:jobId every 2s and renders stage labels (Queued → Fetching → Triaging → Searching IT Glue → Analyzing → Deep review → Done). 5-minute hard timeout. Failures surface as toast errors. Supports force for explicit re-run.
  • components/analyzer/share-modal.tsx<ShareModal> with a shadcn Dialog
    • email field + optional note (max 2000 chars). POSTs to the share endpoint and surfaces server-side validation errors (domain not allowed → toast).
  • components/analyzer/analysis-view.tsx — the full 10-section analysis layout per spec. Header with model-tier badges (Haiku/Sonnet/Opus pills), confidence score, total cost, Share + Re-analyze buttons. Summary, Next Step (with rationale collapsed), Timeline (vertical list with 🟢 / 🔒 / 🔄 markers, click to expand), What Was Done / Should Have Been Done side-by-side on wide screens, Gaps colored by severity with "Evidence:" links that scroll-and-expand the matching timeline event, Post-Resolution (only if present), Human Review Flags (only if needed), IT Glue References.
  • app/analyzer/ticket/[ticketNumber]/page.tsx — ticket detail with the Analyze button and a list of historical versions; latest is badged.
  • app/analyzer/analysis/[id]/page.tsx — fetches one analysis and renders it via <AnalysisView>.
  • app/analyzer/queue/page.tsx — needs-review queue, shows ticket number, version, summary preview, top reasons, confidence badge, and a high gap badge if any gap is high severity.

Decisions worth flagging

  • Imperative useState + useEffect + fetch, no SWR / react-query — matches CLAUDE.md and the rest of the repo. Don't refactor to a global cache layer for these three pages; if it becomes a real pain, that's a whole-app concern.
  • 'use client' everywhere. The pages use use(params) (the React hook) to unwrap Next.js 16's params: Promise<...> shape on the client. Server components weren't appropriate here — every page does interactive state (analyze flow, expand events, share modal).
  • Stage labels render directly from JobStatus enum values, not a separate label list, so any new statuses added to the enum auto-render with their default name.
  • Re-analyze banner not implemented yet. The spec calls for "New activity since last analysis · Re-analyze" when the live content_hash drifts from the persisted one. That requires a live-preprocess endpoint (or running preprocess on the page render). Skipped for now — the user can always click Re-analyze. Worth adding once we have real-world signal on whether activity drift is common.
  • No autocomplete on the share-recipient field. Spec says "autocomplete from existing wulf-pulse user list if available". Skipped — the existing user list is in Better Auth's user table; exposing it requires a small API endpoint. Easy follow-up.
  • No navigation entry yet. components/navigation/app-navigation.tsx doesn't yet have an "Analyzer" link. Adding that is a one-line edit; I left it for the operator to opt in once the feature is staged.
  • Printable analysis view — the spec mentions "printable" for the analysis page. The current layout is print-friendly by accident (no fixed sidebars, sectioned cards), but no explicit @media print styles yet. Add when someone asks.

Deliberately left out

  • No frontend tests. vitest is configured for lib/**/*.test.ts only; the pages and components are visually verified. Component tests with testing-library would be a separate setup decision.
  • Analysis view doesn't auto-refresh while a job is running on a different version. If a user navigates to an old version while a new one is in progress, they don't see the in-progress state. Acceptable — the queue view + ticket history give that signal.

Phase 7 — Share-via-email integration

Delivered

  • sendAnalysisShareEmail() added to lib/services/email.ts — re-uses the existing nodemailer SMTP transport that already serves magic-link and invitation mail. Subject Pulse analysis · <ticket> v<n> — <summary excerpt>, gradient-header HTML body matching the other Pulse emails, plain text fallback, replyTo set to the sharer so a recipient reply lands with the right person.
  • Share route app/api/analyzer/analyses/[id]/share/route.ts now resolves the sharer's session, builds the analysis URL from BETTER_AUTH_URL (falling back to NEXT_PUBLIC_BETTER_AUTH_URL, then localhost:3100), persists the audit row, then attempts the email send.
  • Share-modal frontend handles the new emailSent/emailError fields: success → green toast; row-saved-but-send-failed → orange toast.warning carrying the SMTP error.

Decisions worth flagging

  • Audit row persists even if email send fails. The analyzer_shares row is the audit log, not just a delivery receipt. SMTP outages should not erase the record that the user attempted a share.
  • Response is HTTP 200 on email failure. The route returns {share, emailSent: false, emailError}. A non-2xx would imply the share itself failed; surfacing emailSent: false is the more honest signal.
  • No email_sent_at column added. Adding it requires a migration and a way to retry — neither is asked for by the spec. Send state lives only in the response and the server log line [ANALYZER-SHARE] email send failed for share <id>. If retries become a real need, a dedicated analyzer_share_email_attempts table is the natural shape.
  • Used existing nodemailer SMTP, not Graph sendMail. Spec said "via M365 Graph using existing wulf-pulse mail integration if one exists; otherwise use a new module" — email.ts is the existing module. The Graph integration in this codebase is read-only (mailbox search, user reports), not send-capable.
  • HTML escaping is hand-rolled. Five-replace function for amp/lt/gt/quote/apos. The user-controlled fields (sender name, note, analysis summary, next step) all flow through it. The repo has no HTML escaper utility and the magic-link/invitation emails don't need one because their inputs are URLs and admin-set names.

Deliberately left out

  • No retries. A failed send is logged once and surfaced to the user. They can re-share if they want — that creates a fresh audit row, which is correct behavior.
  • No tests. lib/services/email.ts has no existing tests, mocking nodemailer fully would add a non-trivial test scaffold for one function, and the share route is route-handler thin. Consistent with phases 5/6.
  • No viewed_at tracking yet. The migration has the column but nothing writes to it. A /share/:id/viewed endpoint with an unguessable token would be the smallest addition; spec didn't ask, so skipped.

Phase 8 — Operator runbook

Delivered

  • docs/wulf-pulse-ticket-analyzer-runbook.md — cost monitoring queries (daily spend, top expensive analyses, multi-run tickets), the cost ceiling explanation (COST_CEILING_USD = 2.00 in pipeline.ts), the IT Glue alias workflow (where to find org IDs, how to verify via model_traces.itglue.alias_used), failure triage tables (failed jobs vs. needs_human_review flagged completes, with the actual error strings the worker emits), and a manual-ops section for queueing jobs from psql.
  • README.md "Documentation" section links the runbook so operators finding the project for the first time discover it.

Decisions worth flagging

  • Runbook lives in docs/, not the README. Per CLAUDE.md, long-form per-feature docs go in docs/ and the README points to them. Matches the existing pattern (workflow-editor-guide, sync guides, etc.).
  • Manual migration step is documented. Postgres only re-applies migrations on first init — operators applying analyzer to an existing DB need to run psql -f migrations/069_*.sql. The runbook leads with this.
  • Listed what's not implemented. Retries, viewed_at, share email_sent_at, pre-Sonnet cost gate, share-recipient autocomplete. Better to advertise the gaps than have an operator trip over them.

Deliberately left out

  • No production-monitoring dashboard (Grafana etc.) — Pulse doesn't have one for any other feature, and the SQL queries cover the same ground.
  • No README expansion for the analyzer feature itself. Pulse's README is intentionally short; the runbook + spec + build notes are the deep docs.

Phase 9 — Ticket browser + analysis-view formatting

Reactive to first-use feedback: the analysis page Summary / Next Step text was bare and dense, and there was no way to discover tickets to analyze without typing the URL.

Delivered

  • <ProseText> helper inside analysis-view.tsx — splits text on blank lines, renders each chunk as a separate <p> with leading-7 and whitespace-pre-line. Applied to Summary, Next Step, Next Step rationale, and Post-Resolution Analysis. Single-paragraph text still renders cleanly.
  • Summary + Post-Resolution headers got an uppercase tracking-wide treatment to act as section dividers, and body text bumped to text-base text-foreground so it reads as a finding rather than a caption.
  • Next Step card now has a subtle bg-primary/5 tint, an ArrowRight icon next to "Recommended Next Step", a stronger separator before the rationale collapsible, and the rationale itself renders in a bordered indented block.
  • New /analyzer/tickets browse page — pill-style period chips (Today, Yesterday, This week, Last week, Last 30/60 days, All time), a client (company) Select, an issue-type Select, and a debounced free-text search across ticket_number/title. Compact table with per-row Analyze/Re-analyze button (reusing <AnalyzeButton>) and a "View" button shortcut to the existing analysis when one is recorded. Active-filter count + clear-all in the filter card header.
  • New API GET /api/analyzer/tickets/list — filters by period (computed in Postgres against last_activity_date), companyId, issueType, search. Returns 50 rows + total via COUNT(*) OVER (), plus latestAnalysisId from a LATERAL join into analyzer_analyses.
  • New API GET /api/analyzer/tickets/filter-options — companies that have at least one non-deleted ticket (drops dormant accounts) + active issue types ordered by sort_order, label.
  • Top-level "Analyzer" nav menu added to app-navigation.tsx, with "Browse Tickets" + "Needs Review". Earlier phases left this off intentionally; this phase opts in.

Decisions worth flagging

  • Period filters on last_activity_date, not create_date. "Today" surfaces tickets that had activity today (new tickets, re-opened, status churn) — much more useful for an analyzer-driven triage flow than tickets created today. A new ticket created today also has activity today, so we don't lose those.
  • Period math runs in Postgres via date_trunc('day', NOW()) etc. Database server-clock = app-process clock for an internal Docker stack, so naive timestamps and naive NOW() agree. If users complain about edge-of-day drift, swap to NOW() AT TIME ZONE 'America/New_York' — Pulse's primary user base.
  • Default period is last_30d. "All time" pulls many thousands of rows; defaulting wide-open hurts first-page latency. 30 days hits ~7K rows in our DB, paginates cleanly.
  • Per-row analyze button reuses <AnalyzeButton> directly. Each row gets its own component instance — no shared state, the running state lives per-button. The button navigates on completion, which feels right: click Analyze, watch the stages, land on the analysis page.
  • force=true is set automatically on tickets that already have an analysis. Re-analyze should re-run, not short-circuit to the cached row. The "View" button covers the cached path.
  • No Linear/JIRA-style multi-select filters. Single-value Selects are simpler and match the rest of Pulse.

Deliberately left out

  • No saved views. A filter URL is shareable, but there's no bookmark / saved-view UX. Add when someone asks.
  • No latest_analysis_status exposure. A failed analysis doesn't show up — the LATERAL join filters by status='complete'. So a ticket whose only analysis failed looks like an un-analyzed ticket. Acceptable: re-running is the intended action there anyway.
  • Search is ILIKE '%...%'. No tsvector / trigram index. 7K-row scans are sub-100ms in this DB; if the corpus grows past low six digits, swap in pg_trgm.

Status after each phase

Phase Tests tsc Notes
1 0 clean migration + types only
2 68 clean redaction + preprocessor regression
3 100 clean + LLM scaffolding + Stage 1
4 128 clean + pipeline + worker
5 128 clean API routes (no route tests)
6 128 clean frontend (no FE tests)
7 128 clean share email via existing SMTP transport
8 128 clean operator runbook + README link
9 128 clean browse page + analysis-view formatting + nav entry

Phase 2 (cross-ticket analysis)

Spec: docs/ticket-analyzer-phase2-spec.md. Eight sub-phases delivered as one Phase 2 push.

2.1 — Schema additions

Delivered

  • Migration 070: analyzer_stage_executions table (per-stage I/O for every analyzer run, including failed attempts) + three columns on analyzer_analyses: source_snapshot, aggregate_fingerprint, fingerprint_generated_at.
  • model_traces column annotated with a LEGACY COMMENT ON COLUMN for the SQL side and a // LEGACY doc comment in TS — kept for back-compat until aggregate reports have soaked.
  • Zod schemas: AggregateFingerprint, StageName, StageExecution (read-back), StageExecutionRecord (write-time interface).

Decisions worth flagging

  • analyzer_stage_executions.analysis_id starts NOT NULL in 070. Migration 071 (Phase 2.6) relaxes it and adds the mutually-exclusive CHECK with aggregate_report_id.
  • 070 is idempotent (IF NOT EXISTS on every object) so re-running against an already-applied DB is safe.

2.2 — Pipeline writes to stage executions

Delivered

  • recordedStage(meta, fn, callbacks, outputSelector) helper in pipeline.ts wraps each stage call, emits a StageExecutionRecord on success or failure (re-throws after recording). Pipeline wires it into Stage 1 (triage), Stage 3 (analyze), Stage 4 (deep_review). Stage 0 (preprocess) and Stage 2 (itglue) are recorded inline since they're not LLM calls.
  • Pipeline result now includes triage_response, sonnet_response, opus_response for downstream stages (Stage 6 fingerprint).
  • Worker's runJob collects records via onStageRecord callback, bulk-inserts them after insertAnalysis succeeds. On pipeline throw, worker captures the preprocessed bundle via onPreprocessed callback, persists a status='failed' analyzer_analyses row with source_snapshot intact, and bulk-inserts the partial stage records linked to it.
  • New persistence functions: bulkInsertStageExecutions, insertFailedAnalysis, updateAnalysisFingerprint.

Decisions worth flagging

  • Failure-tolerant audit: spec says "Every stage that runs MUST insert a row, including stages that fail." We persist a failed analyzer_analyses row even on pipeline crash so the partial stage records have a parent. Without this the FK would be orphaned.
  • Single bulk insert: ~56 stage rows per pipeline run. One multi-VALUES INSERT is fast enough; no need for COPY.
  • model_traces double-write retained: the legacy column still receives the old payload. Drop it in a future migration once aggregate reports have soaked through prod.

2.3 — Prose formatting

Delivered

  • Stage 3 system prompt updated with the markdown formatting rules from the spec verbatim (banned filler phrases, action-verb emphasis, no headers).
  • react-markdown@10, remark-gfm@4, @tailwindcss/typography@0.5 added. Tailwind 4 plugin registered via @plugin "@tailwindcss/typography" in app/globals.css.
  • <AnalysisMarkdown> component at components/analyzer/analysis-markdown.tsx renders prose with prose prose-sm dark:prose-invert max-w-none prose-p:leading-7. Coerces stray model headers into bold paragraphs (the prompt forbids them but defense-in-depth).
  • <ProseText> removed. Summary, Recommended Next Step, next_step_rationale, and post_resolution_analysis all use <AnalysisMarkdown>.

Decisions worth flagging

  • Tailwind 4 syntax: @plugin "@tailwindcss/typography" in CSS, no JS config needed.
  • Stage 3 prompt change is back-compatible — old analyses with plain-text summaries still render fine through ReactMarkdown.

2.4 — Stage 6 fingerprint + backfill CLI

Delivered

  • lib/services/analyzer/stages/stage6-fingerprint.ts — Haiku call with the spec's verbatim system prompt. Server-overrides generated_by_model and generated_at after parse so the model's guess for those fields can't drift.
  • Worker integration: after insertAnalysis succeeds, run fingerprint with try/catch. Failure logs a warn, fingerprint stays NULL on the row, but the analysis is still complete and usable. The fingerprint stage record is added to the bulk insert whether it succeeded or failed.
  • scripts/backfill-fingerprints.ts — idempotent CLI. Reads analyzer_analyses.model_traces.{triage_response, sonnet_response, opus_response} (which Phase 1 was already storing), runs Stage 6, writes aggregate_fingerprint. Supports --dry-run and --limit=N. Skips analyses where model_traces is incomplete.

Decisions worth flagging

  • Stage 6 input is just the analysis content (triage + sonnet + optional opus). No pre payload needed — fingerprinting is about the produced analysis, not the source ticket.
  • Backfill processes oldest-first (triggered_at ASC). Lets us observe a few rounds before chewing through hundreds.
  • Stage 6 failure is non-fatal. Spec: "If fingerprinting fails, do NOT fail the overall analysis."

2.5 — Browse / filter UI rebuild

Delivered

  • New endpoint: GET /api/analyzer/tickets (replaces the simpler GET /api/analyzer/tickets/list from Phase 1.9). Multi-select CSV-style query params (clientId, issueType, queue, status, priority, assignedTo); analyzed segmented filter (any/yes/no/stale); needsReview toggle; search; sort.
  • /api/analyzer/tickets/filter-options extended with queues, statuses, priorities, resources (joined to "has at least one ticket" so the dropdowns aren't padded).
  • New <MultiSelect> component at components/ui/multi-select.tsx — Popover + checkbox list with optional search box (auto-shown above 8 options). One trigger + one popover, no shadcn Command dependency.
  • /analyzer/tickets page rebuilt:
    • Sticky filter bar with period pills, multi-selects, search, analyzed segmented, needs-review checkbox, sort
    • Active-filter chips (click to clear individual filter)
    • Bulk selection persisted via localStorage (key analyzer:ticket-selection:v1) — survives pagination
    • "Analyze N selected" — sequential job queue, forces re-analyze on stale rows
    • "Generate aggregate report" — routes to /analyzer/reports/new; only enabled when all selected are current
  • Top nav reorganized: Browse Tickets / Aggregate Reports / Needs Review under "Analyzer".

Decisions worth flagging

  • Staleness via last_activity_date > completed_at, not content-hash compare. The spec lets either; the date heuristic is good enough and avoids per-row preprocessing on 50-row paginated responses.
  • MultiSelect is a one-popover-per-instance design — multiple popovers can be open across the bar. Acceptable; matches how Linear / Vercel's table filters behave.
  • No "Select all matching filters" semantic. Selection is an explicit per-row action stored as ticket numbers in localStorage. Filter-level selection adds significant complexity (server has to resolve filter→IDs, two-modes everywhere). Skipped for V1; the spec's intent (don't lose selection on pagination) is met.
  • Bulk Analyze is sequential, not parallel. N concurrent calls would all hit claimQueuedJob and the worker would process them one at a time anyway (single in-process worker). Sequential POSTs are more honest about that.

2.6 — Aggregate reports

Delivered

  • Migration 071: analyzer_aggregate_reports table + ALTER on analyzer_stage_executions to drop NOT NULL on analysis_id, add aggregate_report_id FK, add the analyzer_stage_executions_parent_check CHECK constraint ((analysis_id IS NOT NULL) <> (aggregate_report_id IS NOT NULL)).
  • lib/services/analyzer/stages/aggregate-reduce.ts — Sonnet (Opus opt-in) reduce stage with the spec's verbatim system prompt and AggregateReduceResponse Zod schema.
  • lib/services/analyzer/aggregate-persistence.tscreateAggregateReport, getAggregateReport, listAggregateReports, runAggregateReport, bulkInsertReportStageExecutions. The runner is fire-and-forget (called via void runAggregateReport(id) from the POST endpoint); it persists distributions immediately so the UI can show partial results during the LLM call.
  • IT Glue context fetcher: per-client findOrganizationByName + getFlexibleAssets, capped at 200 doc titles total per spec. Failure tolerant — per-client errors don't fail the report.
  • API endpoints:
    • POST /api/analyzer/aggregate-reports — validates (≤100, fingerprint exists, not stale), creates pending row, fires runner
    • GET /api/analyzer/aggregate-reports/:id — full report row (UI polls this every 3s while pending/running)
    • GET /api/analyzer/aggregate-reports — paginated list
  • Pages:
    • /analyzer/reports/new?ids=T...,T... — pre-flight: shows selected tickets, options (title, IT Glue context toggle), Generate button
    • /analyzer/reports/[id] — pending → distributions → completed. Sections: header, executive summary, four distribution mini-bar cards, documentation gaps (with itglue_check color tone), process gaps (severity tone), recurrence clusters, recommended actions (sorted by priority), narrative summary.
    • /analyzer/reports — table list of past reports

Decisions worth flagging

  • Fire-and-forget runner, no separate worker module. The POST endpoint kicks void runAggregateReport(id); updates land in the row when the LLM call completes. Frontend polls. Avoids adding a second polling worker alongside analyzerWorker.
  • Stage names reused for aggregate sub-stages. The CHECK constraint on analyzer_stage_executions.stage enumerates the per-analysis stage names. Aggregate sub-stages (SQL aggregation, IT Glue context, reduce LLM) are recorded with stage='analyze' / 'itglue' plus aggregate_report_id set. A future migration could add aggregate_sql / aggregate_reduce to the enum and re-emit those rows; for now the existing names are good enough for forensics.
  • generated_by_user_id is TEXT nullable, not uuid NOT NULL per spec. Better Auth's user.id is text, and we want the report to remain readable if the generating user is later deleted — matches the pattern from analyzer_analyses.triggered_by_user_id.
  • Distributions persist before LLM call so partial-state UI doesn't have to wait the full 3090s for anything to render.

2.7 — Cost guards

Delivered

  • Migration 072: analyzer_cost_audit table.
  • lib/services/analyzer/cost-guard.ts: estimateAggregateReportCost, getUserDailySpend, evaluateCost, recordCostAuditDecision. Thresholds: REQUIRES_CONFIRMATION_USD = 5, SOFT_WARN_DAILY_USD = 20, HARD_BLOCK_DAILY_USD = 50.
  • evaluateCost produces a four-state decision (approved / requires_confirmation / blocked / overridden) plus boolean softWarn/hardBlocked/requiresConfirmation/isOverride fields the API can return for UX.
  • POST /api/analyzer/aggregate-reports enforces:
    • requires_confirmation → 400 with requiresConfirmation:true, estimatedCost, dailySpendBefore so the frontend can show confirm() and re-POST with confirmedCost:true.
    • blocked → 403 with the daily spend in the body.
    • Every decision (including approved) writes a row to analyzer_cost_audit.
  • Override env var ANALYZER_DAILY_COST_OVERRIDE_USERS (comma-separated user ids).
  • Frontend new-report page: catches requiresConfirmation, shows window.confirm() with the dollar figure, retries with confirmedCost: true.

Decisions worth flagging

  • Cost estimate is char/4 → tokens × Sonnet pricing. Crude but pessimistic in the right direction. At 100 tickets the estimate comes in under $0.50 — far below the $5 threshold — so the confirmation modal almost never fires in practice. Ceiling exists to catch payload bloat / Opus-opt-in scenarios.
  • Daily window is trailing 24h, not "today UTC". Avoids midnight-edge-of-day reset gaming; rolling window is what the spec calls "$X/day" naturally.
  • Soft warn at $20/day is informational only. Fields exposed in the cost evaluation; UI can choose to surface, but the API doesn't refuse to proceed. Hard block at $50/day is the only enforcement.

2.8 — Documentation

Delivered

  • docs/wulf-pulse-ticket-analyzer-runbook.md — added Phase 2 sections covering stage execution forensics, fingerprint backfill, aggregate report flow + SQL queries, cost guard configuration + override env var, and the rebuilt browse UI behavior.
  • This file — the per-sub-phase notes above.

Status after Phase 2

Phase Tests tsc Notes
2.1 128 clean schema (070)
2.2 128 clean stage_executions writes + failure-tolerant persistence
2.3 128 clean markdown rendering + Stage 3 prompt
2.4 128 clean Stage 6 fingerprint + backfill CLI
2.5 128 clean browse UI rebuild
2.6 128 clean aggregate reports (071, runner, 3 endpoints, 3 pages)
2.7 128 clean cost guards (072, audit log, threshold gating)
2.8 128 clean runbook + build notes