Eight sub-phases per docs/ticket-analyzer-phase2-spec.md:
2.1 Schema (migration 070): analyzer_stage_executions table; source_snapshot,
aggregate_fingerprint, fingerprint_generated_at columns on analyzer_analyses.
model_traces marked LEGACY (kept for back-compat).
2.2 Every pipeline stage records a row to analyzer_stage_executions, success
or failure. Worker persists a status='failed' analyzer_analyses row when
the pipeline throws so partial stage records have a parent. Pipeline
exposes raw triage/sonnet/opus responses for downstream stages.
2.3 Stage 3 prompt updated with markdown formatting rules + banned filler
phrases. Added react-markdown + remark-gfm + @tailwindcss/typography.
New <AnalysisMarkdown> component replaces <ProseText>; coerces stray
headers to bold paragraphs.
2.4 Stage 6 fingerprint (Haiku) runs after persistence, failure-tolerant.
scripts/backfill-fingerprints.ts reconstructs Stage 6 input from the
legacy model_traces blob.
2.5 Browse UI rebuild at /analyzer/tickets: multi-select for client/issue/
queue/status/priority/assignee, sticky filter bar, active-filter chips,
bulk selection persisted via localStorage, "Analyze N selected" +
"Generate aggregate report" actions. New <MultiSelect> primitive.
Staleness uses last_activity_date > completed_at heuristic per spec C.1.
2.6 Aggregate reports (migration 071): runner is fire-and-forget, persists
SQL distributions immediately so UI shows partial state during the
Sonnet reduce call. Three endpoints, three pages (/analyzer/reports[/new
/:id]). IT Glue context fetcher capped at 200 doc titles.
2.7 Cost guards (migration 072): per-request $5 confirmation, soft-warn at
$20/day, hard-block at $50/day with ANALYZER_DAILY_COST_OVERRIDE_USERS
override. Every gating decision audited.
2.8 Runbook + build notes updated.
128 vitest tests passing, tsc clean. Migrations 070/071/072 idempotent
(IF NOT EXISTS). model_traces double-write retained — drop in a future
migration once aggregate reports have soaked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
38 KiB
AI Ticket Analyzer — Build Notes
A running journal of the multi-phase build for the AI Ticket Analyzer feature.
Captures what was delivered each phase, design decisions worth flagging, and
what was deliberately left out. Spec lives in
wulf-pulse-ticket-analyzer-prompt.md.
This file is updated after each phase ships.
Phase 1 — Migration + Zod schemas
Delivered
migrations/069_create_analyzer_tables.sql— three tables (analyzer_analyses,analyzer_shares,analyzer_jobs) withpgcryptoextension guard, indexes per spec, statusCHECKconstraints, and FK types corrected toTEXT(notUUID) to match Better Auth'suser.id.lib/types/analyzer.ts— Zod schemas for every LLM-stage parsed JSON (TaggedEvent,TriageResponse,DeepAnalysisResponse,OpusResponse), persisted row shapes, job status, API request bodies, and the internalPreprocessedTicketpayload that flows through the pipeline.
Decisions worth flagging
triggered_by_user_idis nullable +ON DELETE SET NULL(notNOT NULL). An analysis should still be readable in history if the triggering user is later deleted.analyzer_shares.shared_by_user_idisNOT NULL + ON DELETE CASCADE(audit-log style — share rows go with the user).evidence_timestampstyped asstring().datetime({offset: true})(ISO timestamps), not numeric indices. More robust to model hallucination and reads better in the UI.TaggedEventdoes NOT support a "two separate events" form for time entries with both Summary + Internal Notes — single event with both fields, per the spec's preference for a cleaner timeline.
Deliberately left out
- Migration is committed but not applied to any running DB. Postgres only re-applies migrations on first init of a fresh volume; the existing DB needs a manual run of this migration when phase 5 is exercised.
Phase 2 — Stage 0 pre-processor + IT Glue redaction
Delivered
lib/services/analyzer/itglue-redact.ts— recursive walk; matches keys against/password|secret|key|token|credential|api[_-]?key/i; replaces values with[REDACTED]. Subtree redaction (key matchingauth→ no leaves leak), defensive copy, cycle guard ([CIRCULAR]marker).lib/services/analyzer/preprocessor.ts— filters workflow noise + Service Desk Notification rows, tags ticket_create/notes/time entries withactor / actor_type / source / visibility / summary_notes / internal_notes / hours, sorts chronologically, computessha256content hash over canonical JSON of(events, status, priority, queue).vitest@^4.1.5added as devDep withvitest.config.tssetting up the@/alias. Two test files at this phase: 32 redaction tests + 36 preprocessor tests including the regression run against theT20260424.0045fixture.- Type-aliasing fix in
lib/types/analyzer.ts: addedexport type X = z.infer<typeof X>forActorType,EventSource,Visibility,Severity,ComplexityTier,TicketType— the Zod-enum const exports alone don't produce a usable TypeScript type.
Decisions worth flagging
actor_typeis classified by email domain, not author text, per the spec.lorentz@wulfconsulting.comiswulf_techregardless of how the message reads. Sonnet handles the role nuance at Stage 3.- Time entries with no narrative content (no summary, no internal notes) are dropped — a purely numeric entry adds nothing.
- Vendor-domain allowlist is intentionally short and conservative (Vertafore,
Datto, Microsoft, etc.). Misclassifying a customer domain as "vendor" is
worse than the default
client_contact.
Deliberately left out
- No real protection for secrets embedded in free text (e.g. a notes field containing the literal string "the password is hunter2"). The redaction guarantee is on field keys, not values. The IT Glue search test documents this contract explicitly so it isn't "fixed" without thought.
Phase 3 — Anthropic SDK setup + Stage 1 (Haiku triage)
Delivered
lib/services/llm/{models,pricing,client,call}.ts— model ID constants, per-model rate table (Haiku $1/$5, Sonnet $3/$15, Opus $5/$25 per 1M tokens- cache read/write tiers), lazy SDK singleton, generic
callLLMStage<T>({model, system, user, schema, maxTokens, client?})helper that:
- Marks the system prompt with
cache_control: {type: "ephemeral"} - Sends NO
temperature/top_p/top_k(Opus 4.7 would 400) - Strips a single
```jsonfence before parsing - On parse failure, retries once with prior-attempt + error in a follow-up user turn
- Returns
{data, usage, estimated_cost_usd, attempts, raw_response}
- cache read/write tiers), lazy SDK singleton, generic
lib/services/analyzer/stages/stage1-triage.ts— Haiku caller with the spec's verbatim system prompt and a 50KB user-payload cap that drops oldest internal-only events first when oversized.lib/services/analyzer/itglue-search.ts— search facade that runs every result throughredact()before returning. Snippets capped at 2000 chars, doc count capped at 10.itglue-aliases.jsonskeleton for known fuzzy org-name mappings.- 32 new tests across pricing/call/Stage 1/IT Glue search.
Decisions worth flagging
- Manual JSON.parse + Zod validate, not
output_config.format/client.messages.parse(). The spec said retry-once on Zod parse failure, and Zod 4 ↔ JSON Schema conversion has edge cases I didn't want to depend on (e.g..datetime({offset: true})→ JSON Schemaformat). Manual parse is what the spec asks for and is more transparent. - Prompt caching probably won't fire on these stages. Our system prompts
are ~1-2 KB (~250-500 tokens); minimum cacheable prefix is 2048 (Sonnet) or
4096 (Haiku/Opus) tokens. The
cache_controlmarker is a no-op below threshold and incurs no cost — left in defensively, but caching is not a meaningful lever for this workload. - Redaction is on KEY names, not free text — re-stated for emphasis. The IT Glue search test asserts this contract.
itglue-aliases.jsonis a skeleton with_comment/_examplekeys documenting the format. Real org-id entries get added when phase 5 wires this into the pipeline.
Deliberately left out
- No live API integration test. All Stage 1 tests use a mocked
Anthropicclient. A real-API smoke test belongs in phase 5+ when we run end-to-end. - Did not wrap
lib/services/itglue-client.tsin a redacting search facade for non-LLM callers. The redaction primitive is ready; non-LLM callers (sync service, data browsers) intentionally have full data — they are not the path that needs protection.
Phase 4 — Pipeline + Job worker (Stages 3, 4, 5)
Delivered
lib/services/analyzer/stages/stage3-deep-analysis.ts— Sonnet caller, spec's verbatim prompt, 80KB payload cap.lib/services/analyzer/stages/stage4-deep-reasoning.ts— Opus caller plus pure helpersshouldRunDeepReasoning()andapplyOpusUpdates().lib/services/analyzer/data-access.ts—loadTicketBundle(ticketNumber)joinstickets/statuses/priorities/queues/companies/contacts/resources/ticket_notes/time_entriesand returns the exactRawTicketBundleshape the preprocessor expects. Throws typedTicketNotFoundError.lib/services/analyzer/persistence.ts—getNextAnalysisVersion,findExistingAnalysisByContentHash,insertAnalysis, plus jobclaimQueuedJob/updateJobStatus/completeJob/failJob/queueJob/getJob. Job claim usesFOR UPDATE SKIP LOCKEDso multiple Next.js workers can poll safely.lib/services/analyzer/pipeline.ts— composes Stage 0 → idempotency check → Stage 1 → (Stage 2 ifitglue_lookup_needed) → Stage 3 → (Stage 4 if trigger fires AND cost ceiling not reached) → result. Cost circuit breaker trips at $2.00 before Opus, setsneeds_human_review=truewith a reason. Returns fullmodel_tracesfor debugging.lib/services/analyzer/worker.ts— singleton with 2-second poll loop; auto-starts in production; opt-in in dev viaANALYZER_WORKER_AUTOSTART=1; skipped under vitest.runJob()exposed for tests + manual triggers.TicketNotFoundErrorproduces a user-facing job error message that fingers the sync as the culprit.- 28 new tests bringing the total to 128.
Decisions worth flagging
- Worker auto-start is more conservative than
sync-scheduler.ts. That one auto-starts on any non-browser import (including tests). I gated this one because the worker hits the database AND runs LLM calls — much higher blast radius. To run locally, setANALYZER_WORKER_AUTOSTART=1. - Cost circuit breaker only fires before Opus. Sonnet runs unconditionally
even if it would push past $2. Spec wording matches; if the team wants
stricter control, the natural place is next to
COST_CEILING_USDinpipeline.ts. - Idempotency check matches
status='complete'only. Failed runs don't poison the cache. - IT Glue failures are tolerated. Search throws → analysis continues without context, doesn't fail the run.
worker.test.tscastsmockImplementationOncetoas neverbecause vitest's overload resolution fights us when the mocked function has multiple call signatures. Functional, just ugly.
Deliberately left out
- No live API integration test (still). Phase 5+ is the natural place for an end-to-end smoke test.
- Stage 3 doesn't have a fixture-driven happy-path test like Stage 1; it's
exercised at the orchestration layer via
pipeline.test.tsonly. Worth adding direct Stage 3 tests later. - Migration
069still not applied — same as phase 1.
Phase 5 — API routes
Delivered
- Persistence read paths added to
lib/services/analyzer/persistence.ts:getAnalysisById,listAnalysesByTicketNumber,listNeedsReview,createShare, plus a sharedrowToPersistedAnalysisrow mapper. - 6 routes under
app/api/analyzer/:Route Method Returns tickets/[ticketNumber]/analyzePOST {status, jobId?, existingAnalysisId?}tickets/[ticketNumber]/analysesGET {analyses: PersistedAnalysis[]}jobs/[jobId]GET {job}analyses/[id]GET {analysis}analyses/[id]/sharePOST {share}needs-review?limit=&offset=GET {analyses}
Decisions worth flagging
- Analyze runs preprocess inline. The route does
loadTicketBundle → preprocessTicket → findExistingAnalysisByContentHashsynchronously to support the spec'sexistingAnalysisId?immediate response. The worker also runs this — duplicated work, but preprocess is fast (deterministic, one DB load) and the alternative (always queue, frontend polls to discover the short-circuit) is worse UX. - Email send deferred to phase 8 per the spec's delivery order. The share
route persists the audit row and validates the recipient domain against
ALLOWED_SHARE_DOMAINS. Until phase 8, share rows haveviewed_at = nullindefinitely. requireAuth()everywhere — notrequireAdmin(). Any authenticated user can analyze a ticket they have access to. If/needs-reviewshould be admin-only later, swap that one torequirePermission('analyzer', 'review')once the permission map is decided.- No new permission entries added to
lib/permissions.ts. Adding scoped permissions for a feature still under build risks getting them wrong. - Routes are NOT in
middleware.tspublicRoutes— they require a session.
Deliberately left out
- No API route tests. The repo has zero
app/api/**/*.test.tsfiles; the routes are thin orchestration on top of already-tested persistence. Adding integration tests means setting up a test harness for the auth helpers + Postgres, which is a separate effort.
Phase 6 — Frontend pages
Delivered
components/analyzer/analyze-button.tsx—<AnalyzeButton>with the full state machine: POSTs to the analyze endpoint, navigates straight to an existing analysis ifexistingAnalysisIdcame back, otherwise polls/api/analyzer/jobs/:jobIdevery 2s and renders stage labels (Queued → Fetching → Triaging → Searching IT Glue → Analyzing → Deep review → Done). 5-minute hard timeout. Failures surface as toast errors. Supportsforcefor explicit re-run.components/analyzer/share-modal.tsx—<ShareModal>with a shadcn Dialog- email field + optional note (max 2000 chars). POSTs to the share endpoint and surfaces server-side validation errors (domain not allowed → toast).
components/analyzer/analysis-view.tsx— the full 10-section analysis layout per spec. Header with model-tier badges (Haiku/Sonnet/Opus pills), confidence score, total cost, Share + Re-analyze buttons. Summary, Next Step (with rationale collapsed), Timeline (vertical list with 🟢 / 🔒 / 🔄 markers, click to expand), What Was Done / Should Have Been Done side-by-side on wide screens, Gaps colored by severity with "Evidence:" links that scroll-and-expand the matching timeline event, Post-Resolution (only if present), Human Review Flags (only if needed), IT Glue References.app/analyzer/ticket/[ticketNumber]/page.tsx— ticket detail with the Analyze button and a list of historical versions; latest is badged.app/analyzer/analysis/[id]/page.tsx— fetches one analysis and renders it via<AnalysisView>.app/analyzer/queue/page.tsx— needs-review queue, shows ticket number, version, summary preview, top reasons, confidence badge, and ahigh gapbadge if any gap is high severity.
Decisions worth flagging
- Imperative
useState+useEffect+fetch, no SWR / react-query — matches CLAUDE.md and the rest of the repo. Don't refactor to a global cache layer for these three pages; if it becomes a real pain, that's a whole-app concern. 'use client'everywhere. The pages useuse(params)(the React hook) to unwrap Next.js 16'sparams: Promise<...>shape on the client. Server components weren't appropriate here — every page does interactive state (analyze flow, expand events, share modal).- Stage labels render directly from
JobStatusenum values, not a separate label list, so any new statuses added to the enum auto-render with their default name. - Re-analyze banner not implemented yet. The spec calls for "New activity since last analysis · Re-analyze" when the live content_hash drifts from the persisted one. That requires a live-preprocess endpoint (or running preprocess on the page render). Skipped for now — the user can always click Re-analyze. Worth adding once we have real-world signal on whether activity drift is common.
- No autocomplete on the share-recipient field. Spec says "autocomplete
from existing wulf-pulse user list if available". Skipped — the existing
user list is in Better Auth's
usertable; exposing it requires a small API endpoint. Easy follow-up. - No navigation entry yet.
components/navigation/app-navigation.tsxdoesn't yet have an "Analyzer" link. Adding that is a one-line edit; I left it for the operator to opt in once the feature is staged. - Printable analysis view — the spec mentions "printable" for the
analysis page. The current layout is print-friendly by accident (no
fixed sidebars, sectioned cards), but no explicit
@media printstyles yet. Add when someone asks.
Deliberately left out
- No frontend tests. vitest is configured for
lib/**/*.test.tsonly; the pages and components are visually verified. Component tests with testing-library would be a separate setup decision. - Analysis view doesn't auto-refresh while a job is running on a different version. If a user navigates to an old version while a new one is in progress, they don't see the in-progress state. Acceptable — the queue view + ticket history give that signal.
Phase 7 — Share-via-email integration
Delivered
sendAnalysisShareEmail()added tolib/services/email.ts— re-uses the existing nodemailer SMTP transport that already serves magic-link and invitation mail. SubjectPulse analysis · <ticket> v<n> — <summary excerpt>, gradient-header HTML body matching the other Pulse emails, plain text fallback,replyToset to the sharer so a recipient reply lands with the right person.- Share route
app/api/analyzer/analyses/[id]/share/route.tsnow resolves the sharer's session, builds the analysis URL fromBETTER_AUTH_URL(falling back toNEXT_PUBLIC_BETTER_AUTH_URL, thenlocalhost:3100), persists the audit row, then attempts the email send. - Share-modal frontend handles the new
emailSent/emailErrorfields: success → green toast; row-saved-but-send-failed → orangetoast.warningcarrying the SMTP error.
Decisions worth flagging
- Audit row persists even if email send fails. The
analyzer_sharesrow is the audit log, not just a delivery receipt. SMTP outages should not erase the record that the user attempted a share. - Response is HTTP 200 on email failure. The route returns
{share, emailSent: false, emailError}. A non-2xx would imply the share itself failed; surfacingemailSent: falseis the more honest signal. - No
email_sent_atcolumn added. Adding it requires a migration and a way to retry — neither is asked for by the spec. Send state lives only in the response and the server log line[ANALYZER-SHARE] email send failed for share <id>. If retries become a real need, a dedicatedanalyzer_share_email_attemptstable is the natural shape. - Used existing nodemailer SMTP, not Graph sendMail. Spec said "via
M365 Graph using existing wulf-pulse mail integration if one exists;
otherwise use a new module" —
email.tsis the existing module. The Graph integration in this codebase is read-only (mailbox search, user reports), not send-capable. - HTML escaping is hand-rolled. Five-replace function for amp/lt/gt/quote/apos. The user-controlled fields (sender name, note, analysis summary, next step) all flow through it. The repo has no HTML escaper utility and the magic-link/invitation emails don't need one because their inputs are URLs and admin-set names.
Deliberately left out
- No retries. A failed send is logged once and surfaced to the user. They can re-share if they want — that creates a fresh audit row, which is correct behavior.
- No tests.
lib/services/email.tshas no existing tests, mocking nodemailer fully would add a non-trivial test scaffold for one function, and the share route is route-handler thin. Consistent with phases 5/6. - No
viewed_attracking yet. The migration has the column but nothing writes to it. A/share/:id/viewedendpoint with an unguessable token would be the smallest addition; spec didn't ask, so skipped.
Phase 8 — Operator runbook
Delivered
docs/wulf-pulse-ticket-analyzer-runbook.md— cost monitoring queries (daily spend, top expensive analyses, multi-run tickets), the cost ceiling explanation (COST_CEILING_USD = 2.00inpipeline.ts), the IT Glue alias workflow (where to find org IDs, how to verify viamodel_traces.itglue.alias_used), failure triage tables (failed jobs vs.needs_human_reviewflagged completes, with the actual error strings the worker emits), and a manual-ops section for queueing jobs from psql.- README.md "Documentation" section links the runbook so operators finding the project for the first time discover it.
Decisions worth flagging
- Runbook lives in
docs/, not the README. Per CLAUDE.md, long-form per-feature docs go indocs/and the README points to them. Matches the existing pattern (workflow-editor-guide, sync guides, etc.). - Manual migration step is documented. Postgres only re-applies
migrations on first init — operators applying analyzer to an existing
DB need to run
psql -f migrations/069_*.sql. The runbook leads with this. - Listed what's not implemented. Retries,
viewed_at, shareemail_sent_at, pre-Sonnet cost gate, share-recipient autocomplete. Better to advertise the gaps than have an operator trip over them.
Deliberately left out
- No production-monitoring dashboard (Grafana etc.) — Pulse doesn't have one for any other feature, and the SQL queries cover the same ground.
- No README expansion for the analyzer feature itself. Pulse's README is intentionally short; the runbook + spec + build notes are the deep docs.
Phase 9 — Ticket browser + analysis-view formatting
Reactive to first-use feedback: the analysis page Summary / Next Step text was bare and dense, and there was no way to discover tickets to analyze without typing the URL.
Delivered
<ProseText>helper insideanalysis-view.tsx— splits text on blank lines, renders each chunk as a separate<p>withleading-7andwhitespace-pre-line. Applied to Summary, Next Step, Next Step rationale, and Post-Resolution Analysis. Single-paragraph text still renders cleanly.- Summary + Post-Resolution headers got an uppercase tracking-wide
treatment to act as section dividers, and body text bumped to
text-base text-foregroundso it reads as a finding rather than a caption. - Next Step card now has a subtle
bg-primary/5tint, anArrowRighticon next to "Recommended Next Step", a stronger separator before the rationale collapsible, and the rationale itself renders in a bordered indented block. - New
/analyzer/ticketsbrowse page — pill-style period chips (Today, Yesterday, This week, Last week, Last 30/60 days, All time), a client (company) Select, an issue-type Select, and a debounced free-text search across ticket_number/title. Compact table with per-row Analyze/Re-analyze button (reusing<AnalyzeButton>) and a "View" button shortcut to the existing analysis when one is recorded. Active-filter count + clear-all in the filter card header. - New API
GET /api/analyzer/tickets/list— filters byperiod(computed in Postgres againstlast_activity_date),companyId,issueType,search. Returns 50 rows + total viaCOUNT(*) OVER (), pluslatestAnalysisIdfrom a LATERAL join intoanalyzer_analyses. - New API
GET /api/analyzer/tickets/filter-options— companies that have at least one non-deleted ticket (drops dormant accounts) + active issue types ordered bysort_order, label. - Top-level "Analyzer" nav menu added to
app-navigation.tsx, with "Browse Tickets" + "Needs Review". Earlier phases left this off intentionally; this phase opts in.
Decisions worth flagging
- Period filters on
last_activity_date, notcreate_date. "Today" surfaces tickets that had activity today (new tickets, re-opened, status churn) — much more useful for an analyzer-driven triage flow than tickets created today. A new ticket created today also has activity today, so we don't lose those. - Period math runs in Postgres via
date_trunc('day', NOW())etc. Database server-clock = app-process clock for an internal Docker stack, so naive timestamps and naiveNOW()agree. If users complain about edge-of-day drift, swap toNOW() AT TIME ZONE 'America/New_York'— Pulse's primary user base. - Default period is
last_30d. "All time" pulls many thousands of rows; defaulting wide-open hurts first-page latency. 30 days hits ~7K rows in our DB, paginates cleanly. - Per-row analyze button reuses
<AnalyzeButton>directly. Each row gets its own component instance — no shared state, the running state lives per-button. The button navigates on completion, which feels right: click Analyze, watch the stages, land on the analysis page. force=trueis set automatically on tickets that already have an analysis. Re-analyze should re-run, not short-circuit to the cached row. The "View" button covers the cached path.- No Linear/JIRA-style multi-select filters. Single-value Selects are simpler and match the rest of Pulse.
Deliberately left out
- No saved views. A filter URL is shareable, but there's no bookmark / saved-view UX. Add when someone asks.
- No
latest_analysis_statusexposure. A failed analysis doesn't show up — the LATERAL join filters bystatus='complete'. So a ticket whose only analysis failed looks like an un-analyzed ticket. Acceptable: re-running is the intended action there anyway. - Search is
ILIKE '%...%'. No tsvector / trigram index. 7K-row scans are sub-100ms in this DB; if the corpus grows past low six digits, swap inpg_trgm.
Status after each phase
| Phase | Tests | tsc | Notes |
|---|---|---|---|
| 1 | 0 | clean | migration + types only |
| 2 | 68 | clean | redaction + preprocessor regression |
| 3 | 100 | clean | + LLM scaffolding + Stage 1 |
| 4 | 128 | clean | + pipeline + worker |
| 5 | 128 | clean | API routes (no route tests) |
| 6 | 128 | clean | frontend (no FE tests) |
| 7 | 128 | clean | share email via existing SMTP transport |
| 8 | 128 | clean | operator runbook + README link |
| 9 | 128 | clean | browse page + analysis-view formatting + nav entry |
Phase 2 (cross-ticket analysis)
Spec: docs/ticket-analyzer-phase2-spec.md. Eight sub-phases delivered as
one Phase 2 push.
2.1 — Schema additions
Delivered
- Migration 070:
analyzer_stage_executionstable (per-stage I/O for every analyzer run, including failed attempts) + three columns onanalyzer_analyses:source_snapshot,aggregate_fingerprint,fingerprint_generated_at. model_tracescolumn annotated with aLEGACYCOMMENT ON COLUMNfor the SQL side and a// LEGACYdoc comment in TS — kept for back-compat until aggregate reports have soaked.- Zod schemas:
AggregateFingerprint,StageName,StageExecution(read-back),StageExecutionRecord(write-time interface).
Decisions worth flagging
analyzer_stage_executions.analysis_idstarts NOT NULL in 070. Migration 071 (Phase 2.6) relaxes it and adds the mutually-exclusive CHECK withaggregate_report_id.- 070 is idempotent (
IF NOT EXISTSon every object) so re-running against an already-applied DB is safe.
2.2 — Pipeline writes to stage executions
Delivered
recordedStage(meta, fn, callbacks, outputSelector)helper inpipeline.tswraps each stage call, emits aStageExecutionRecordon success or failure (re-throws after recording). Pipeline wires it into Stage 1 (triage), Stage 3 (analyze), Stage 4 (deep_review). Stage 0 (preprocess) and Stage 2 (itglue) are recorded inline since they're not LLM calls.- Pipeline result now includes
triage_response,sonnet_response,opus_responsefor downstream stages (Stage 6 fingerprint). - Worker's
runJobcollects records viaonStageRecordcallback, bulk-inserts them afterinsertAnalysissucceeds. On pipeline throw, worker captures the preprocessed bundle viaonPreprocessedcallback, persists astatus='failed'analyzer_analyses row with source_snapshot intact, and bulk-inserts the partial stage records linked to it. - New persistence functions:
bulkInsertStageExecutions,insertFailedAnalysis,updateAnalysisFingerprint.
Decisions worth flagging
- Failure-tolerant audit: spec says "Every stage that runs MUST insert a row, including stages that fail." We persist a failed analyzer_analyses row even on pipeline crash so the partial stage records have a parent. Without this the FK would be orphaned.
- Single bulk insert: ~5–6 stage rows per pipeline run. One multi-VALUES INSERT is fast enough; no need for COPY.
model_tracesdouble-write retained: the legacy column still receives the old payload. Drop it in a future migration once aggregate reports have soaked through prod.
2.3 — Prose formatting
Delivered
- Stage 3 system prompt updated with the markdown formatting rules from the spec verbatim (banned filler phrases, action-verb emphasis, no headers).
react-markdown@10,remark-gfm@4,@tailwindcss/typography@0.5added. Tailwind 4 plugin registered via@plugin "@tailwindcss/typography"inapp/globals.css.<AnalysisMarkdown>component atcomponents/analyzer/analysis-markdown.tsxrenders prose withprose prose-sm dark:prose-invert max-w-none prose-p:leading-7. Coerces stray model headers into bold paragraphs (the prompt forbids them but defense-in-depth).<ProseText>removed. Summary, Recommended Next Step, next_step_rationale, and post_resolution_analysis all use<AnalysisMarkdown>.
Decisions worth flagging
- Tailwind 4 syntax:
@plugin "@tailwindcss/typography"in CSS, no JS config needed. - Stage 3 prompt change is back-compatible — old analyses with plain-text summaries still render fine through ReactMarkdown.
2.4 — Stage 6 fingerprint + backfill CLI
Delivered
lib/services/analyzer/stages/stage6-fingerprint.ts— Haiku call with the spec's verbatim system prompt. Server-overridesgenerated_by_modelandgenerated_atafter parse so the model's guess for those fields can't drift.- Worker integration: after
insertAnalysissucceeds, run fingerprint with try/catch. Failure logs a warn, fingerprint stays NULL on the row, but the analysis is still complete and usable. The fingerprint stage record is added to the bulk insert whether it succeeded or failed. scripts/backfill-fingerprints.ts— idempotent CLI. Readsanalyzer_analyses.model_traces.{triage_response, sonnet_response, opus_response}(which Phase 1 was already storing), runs Stage 6, writesaggregate_fingerprint. Supports--dry-runand--limit=N. Skips analyses where model_traces is incomplete.
Decisions worth flagging
- Stage 6 input is just the analysis content (triage + sonnet +
optional opus). No
prepayload needed — fingerprinting is about the produced analysis, not the source ticket. - Backfill processes oldest-first (triggered_at ASC). Lets us observe a few rounds before chewing through hundreds.
- Stage 6 failure is non-fatal. Spec: "If fingerprinting fails, do NOT fail the overall analysis."
2.5 — Browse / filter UI rebuild
Delivered
- New endpoint:
GET /api/analyzer/tickets(replaces the simplerGET /api/analyzer/tickets/listfrom Phase 1.9). Multi-select CSV-style query params (clientId, issueType, queue, status, priority, assignedTo); analyzed segmented filter (any/yes/no/stale); needsReview toggle; search; sort. /api/analyzer/tickets/filter-optionsextended with queues, statuses, priorities, resources (joined to "has at least one ticket" so the dropdowns aren't padded).- New
<MultiSelect>component atcomponents/ui/multi-select.tsx— Popover + checkbox list with optional search box (auto-shown above 8 options). One trigger + one popover, no shadcn Command dependency. /analyzer/ticketspage rebuilt:- Sticky filter bar with period pills, multi-selects, search, analyzed segmented, needs-review checkbox, sort
- Active-filter chips (click to clear individual filter)
- Bulk selection persisted via localStorage (key
analyzer:ticket-selection:v1) — survives pagination - "Analyze N selected" — sequential job queue, forces re-analyze
on
stalerows - "Generate aggregate report" — routes to
/analyzer/reports/new; only enabled when all selected arecurrent
- Top nav reorganized: Browse Tickets / Aggregate Reports / Needs Review under "Analyzer".
Decisions worth flagging
- Staleness via
last_activity_date > completed_at, not content-hash compare. The spec lets either; the date heuristic is good enough and avoids per-row preprocessing on 50-row paginated responses. MultiSelectis a one-popover-per-instance design — multiple popovers can be open across the bar. Acceptable; matches how Linear / Vercel's table filters behave.- No "Select all matching filters" semantic. Selection is an explicit per-row action stored as ticket numbers in localStorage. Filter-level selection adds significant complexity (server has to resolve filter→IDs, two-modes everywhere). Skipped for V1; the spec's intent (don't lose selection on pagination) is met.
- Bulk Analyze is sequential, not parallel. N concurrent calls
would all hit
claimQueuedJoband the worker would process them one at a time anyway (single in-process worker). Sequential POSTs are more honest about that.
2.6 — Aggregate reports
Delivered
- Migration 071:
analyzer_aggregate_reportstable + ALTER onanalyzer_stage_executionsto drop NOT NULL onanalysis_id, addaggregate_report_idFK, add theanalyzer_stage_executions_parent_checkCHECK constraint ((analysis_id IS NOT NULL) <> (aggregate_report_id IS NOT NULL)). lib/services/analyzer/stages/aggregate-reduce.ts— Sonnet (Opus opt-in) reduce stage with the spec's verbatim system prompt andAggregateReduceResponseZod schema.lib/services/analyzer/aggregate-persistence.ts—createAggregateReport,getAggregateReport,listAggregateReports,runAggregateReport,bulkInsertReportStageExecutions. The runner is fire-and-forget (called viavoid runAggregateReport(id)from the POST endpoint); it persists distributions immediately so the UI can show partial results during the LLM call.- IT Glue context fetcher: per-client
findOrganizationByName+getFlexibleAssets, capped at 200 doc titles total per spec. Failure tolerant — per-client errors don't fail the report. - API endpoints:
POST /api/analyzer/aggregate-reports— validates (≤100, fingerprint exists, not stale), creates pending row, fires runnerGET /api/analyzer/aggregate-reports/:id— full report row (UI polls this every 3s while pending/running)GET /api/analyzer/aggregate-reports— paginated list
- Pages:
/analyzer/reports/new?ids=T...,T...— pre-flight: shows selected tickets, options (title, IT Glue context toggle), Generate button/analyzer/reports/[id]— pending → distributions → completed. Sections: header, executive summary, four distribution mini-bar cards, documentation gaps (withitglue_checkcolor tone), process gaps (severity tone), recurrence clusters, recommended actions (sorted by priority), narrative summary./analyzer/reports— table list of past reports
Decisions worth flagging
- Fire-and-forget runner, no separate worker module. The POST
endpoint kicks
void runAggregateReport(id); updates land in the row when the LLM call completes. Frontend polls. Avoids adding a second polling worker alongsideanalyzerWorker. - Stage names reused for aggregate sub-stages. The CHECK constraint
on
analyzer_stage_executions.stageenumerates the per-analysis stage names. Aggregate sub-stages (SQL aggregation, IT Glue context, reduce LLM) are recorded withstage='analyze'/'itglue'plusaggregate_report_idset. A future migration could addaggregate_sql/aggregate_reduceto the enum and re-emit those rows; for now the existing names are good enough for forensics. generated_by_user_idis TEXT nullable, notuuid NOT NULLper spec. Better Auth'suser.idis text, and we want the report to remain readable if the generating user is later deleted — matches the pattern fromanalyzer_analyses.triggered_by_user_id.- Distributions persist before LLM call so partial-state UI doesn't have to wait the full 30–90s for anything to render.
2.7 — Cost guards
Delivered
- Migration 072:
analyzer_cost_audittable. lib/services/analyzer/cost-guard.ts:estimateAggregateReportCost,getUserDailySpend,evaluateCost,recordCostAuditDecision. Thresholds:REQUIRES_CONFIRMATION_USD = 5,SOFT_WARN_DAILY_USD = 20,HARD_BLOCK_DAILY_USD = 50.evaluateCostproduces a four-state decision (approved/requires_confirmation/blocked/overridden) plus booleansoftWarn/hardBlocked/requiresConfirmation/isOverridefields the API can return for UX.- POST
/api/analyzer/aggregate-reportsenforces:requires_confirmation→ 400 withrequiresConfirmation:true, estimatedCost, dailySpendBeforeso the frontend can showconfirm()and re-POST withconfirmedCost:true.blocked→ 403 with the daily spend in the body.- Every decision (including
approved) writes a row toanalyzer_cost_audit.
- Override env var
ANALYZER_DAILY_COST_OVERRIDE_USERS(comma-separated user ids). - Frontend new-report page: catches
requiresConfirmation, showswindow.confirm()with the dollar figure, retries withconfirmedCost: true.
Decisions worth flagging
- Cost estimate is char/4 → tokens × Sonnet pricing. Crude but pessimistic in the right direction. At 100 tickets the estimate comes in under $0.50 — far below the $5 threshold — so the confirmation modal almost never fires in practice. Ceiling exists to catch payload bloat / Opus-opt-in scenarios.
- Daily window is trailing 24h, not "today UTC". Avoids midnight-edge-of-day reset gaming; rolling window is what the spec calls "$X/day" naturally.
- Soft warn at $20/day is informational only. Fields exposed in the cost evaluation; UI can choose to surface, but the API doesn't refuse to proceed. Hard block at $50/day is the only enforcement.
2.8 — Documentation
Delivered
docs/wulf-pulse-ticket-analyzer-runbook.md— added Phase 2 sections covering stage execution forensics, fingerprint backfill, aggregate report flow + SQL queries, cost guard configuration + override env var, and the rebuilt browse UI behavior.- This file — the per-sub-phase notes above.
Status after Phase 2
| Phase | Tests | tsc | Notes |
|---|---|---|---|
| 2.1 | 128 | clean | schema (070) |
| 2.2 | 128 | clean | stage_executions writes + failure-tolerant persistence |
| 2.3 | 128 | clean | markdown rendering + Stage 3 prompt |
| 2.4 | 128 | clean | Stage 6 fingerprint + backfill CLI |
| 2.5 | 128 | clean | browse UI rebuild |
| 2.6 | 128 | clean | aggregate reports (071, runner, 3 endpoints, 3 pages) |
| 2.7 | 128 | clean | cost guards (072, audit log, threshold gating) |
| 2.8 | 128 | clean | runbook + build notes |