# AI Ticket Analyzer — Build Notes A running journal of the multi-phase build for the AI Ticket Analyzer feature. Captures what was delivered each phase, design decisions worth flagging, and what was deliberately left out. Spec lives in `wulf-pulse-ticket-analyzer-prompt.md`. This file is updated after each phase ships. --- ## Phase 1 — Migration + Zod schemas **Delivered** - `migrations/069_create_analyzer_tables.sql` — three tables (`analyzer_analyses`, `analyzer_shares`, `analyzer_jobs`) with `pgcrypto` extension guard, indexes per spec, status `CHECK` constraints, and FK types corrected to `TEXT` (not `UUID`) to match Better Auth's `user.id`. - `lib/types/analyzer.ts` — Zod schemas for every LLM-stage parsed JSON (`TaggedEvent`, `TriageResponse`, `DeepAnalysisResponse`, `OpusResponse`), persisted row shapes, job status, API request bodies, and the internal `PreprocessedTicket` payload that flows through the pipeline. **Decisions worth flagging** - `triggered_by_user_id` is **nullable + `ON DELETE SET NULL`** (not `NOT NULL`). An analysis should still be readable in history if the triggering user is later deleted. `analyzer_shares.shared_by_user_id` is `NOT NULL + ON DELETE CASCADE` (audit-log style — share rows go with the user). - `evidence_timestamps` typed as `string().datetime({offset: true})` (ISO timestamps), not numeric indices. More robust to model hallucination and reads better in the UI. - `TaggedEvent` does NOT support a "two separate events" form for time entries with both Summary + Internal Notes — single event with both fields, per the spec's preference for a cleaner timeline. **Deliberately left out** - Migration is committed but **not applied** to any running DB. Postgres only re-applies migrations on first init of a fresh volume; the existing DB needs a manual run of this migration when phase 5 is exercised. --- ## Phase 2 — Stage 0 pre-processor + IT Glue redaction **Delivered** - `lib/services/analyzer/itglue-redact.ts` — recursive walk; matches keys against `/password|secret|key|token|credential|api[_-]?key/i`; replaces values with `[REDACTED]`. Subtree redaction (key matching `auth` → no leaves leak), defensive copy, cycle guard (`[CIRCULAR]` marker). - `lib/services/analyzer/preprocessor.ts` — filters workflow noise + Service Desk Notification rows, tags ticket_create/notes/time entries with `actor / actor_type / source / visibility / summary_notes / internal_notes / hours`, sorts chronologically, computes `sha256` content hash over canonical JSON of `(events, status, priority, queue)`. - `vitest@^4.1.5` added as devDep with `vitest.config.ts` setting up the `@/` alias. Two test files at this phase: 32 redaction tests + 36 preprocessor tests including the regression run against the `T20260424.0045` fixture. - Type-aliasing fix in `lib/types/analyzer.ts`: added `export type X = z.infer` for `ActorType`, `EventSource`, `Visibility`, `Severity`, `ComplexityTier`, `TicketType` — the Zod-enum const exports alone don't produce a usable TypeScript type. **Decisions worth flagging** - `actor_type` is classified by **email domain, not author text**, per the spec. `lorentz@wulfconsulting.com` is `wulf_tech` regardless of how the message reads. Sonnet handles the role nuance at Stage 3. - Time entries with no narrative content (no summary, no internal notes) are dropped — a purely numeric entry adds nothing. - Vendor-domain allowlist is intentionally short and conservative (Vertafore, Datto, Microsoft, etc.). Misclassifying a customer domain as "vendor" is worse than the default `client_contact`. **Deliberately left out** - No real protection for secrets embedded in **free text** (e.g. a notes field containing the literal string "the password is hunter2"). The redaction guarantee is on field **keys**, not values. The IT Glue search test documents this contract explicitly so it isn't "fixed" without thought. --- ## Phase 3 — Anthropic SDK setup + Stage 1 (Haiku triage) **Delivered** - `lib/services/llm/{models,pricing,client,call}.ts` — model ID constants, per-model rate table (Haiku $1/$5, Sonnet $3/$15, Opus $5/$25 per 1M tokens + cache read/write tiers), lazy SDK singleton, generic `callLLMStage({model, system, user, schema, maxTokens, client?})` helper that: - Marks the system prompt with `cache_control: {type: "ephemeral"}` - Sends NO `temperature`/`top_p`/`top_k` (Opus 4.7 would 400) - Strips a single ` ```json ` fence before parsing - On parse failure, retries **once** with prior-attempt + error in a follow-up user turn - Returns `{data, usage, estimated_cost_usd, attempts, raw_response}` - `lib/services/analyzer/stages/stage1-triage.ts` — Haiku caller with the spec's verbatim system prompt and a 50KB user-payload cap that drops oldest internal-only events first when oversized. - `lib/services/analyzer/itglue-search.ts` — search facade that runs every result through `redact()` before returning. Snippets capped at 2000 chars, doc count capped at 10. `itglue-aliases.json` skeleton for known fuzzy org-name mappings. - 32 new tests across pricing/call/Stage 1/IT Glue search. **Decisions worth flagging** - **Manual JSON.parse + Zod validate, not `output_config.format` / `client.messages.parse()`.** The spec said retry-once on Zod parse failure, and Zod 4 ↔ JSON Schema conversion has edge cases I didn't want to depend on (e.g. `.datetime({offset: true})` → JSON Schema `format`). Manual parse is what the spec asks for and is more transparent. - **Prompt caching probably won't fire on these stages.** Our system prompts are ~1-2 KB (~250-500 tokens); minimum cacheable prefix is 2048 (Sonnet) or 4096 (Haiku/Opus) tokens. The `cache_control` marker is a no-op below threshold and incurs no cost — left in defensively, but caching is not a meaningful lever for this workload. - **Redaction is on KEY names, not free text** — re-stated for emphasis. The IT Glue search test asserts this contract. - **`itglue-aliases.json` is a skeleton** with `_comment` / `_example` keys documenting the format. Real org-id entries get added when phase 5 wires this into the pipeline. **Deliberately left out** - No live API integration test. All Stage 1 tests use a mocked `Anthropic` client. A real-API smoke test belongs in phase 5+ when we run end-to-end. - Did **not** wrap `lib/services/itglue-client.ts` in a redacting search facade for non-LLM callers. The redaction primitive is ready; non-LLM callers (sync service, data browsers) intentionally have full data — they are not the path that needs protection. --- ## Phase 4 — Pipeline + Job worker (Stages 3, 4, 5) **Delivered** - `lib/services/analyzer/stages/stage3-deep-analysis.ts` — Sonnet caller, spec's verbatim prompt, 80KB payload cap. - `lib/services/analyzer/stages/stage4-deep-reasoning.ts` — Opus caller plus pure helpers `shouldRunDeepReasoning()` and `applyOpusUpdates()`. - `lib/services/analyzer/data-access.ts` — `loadTicketBundle(ticketNumber)` joins `tickets` / `statuses` / `priorities` / `queues` / `companies` / `contacts` / `resources` / `ticket_notes` / `time_entries` and returns the exact `RawTicketBundle` shape the preprocessor expects. Throws typed `TicketNotFoundError`. - `lib/services/analyzer/persistence.ts` — `getNextAnalysisVersion`, `findExistingAnalysisByContentHash`, `insertAnalysis`, plus job `claimQueuedJob` / `updateJobStatus` / `completeJob` / `failJob` / `queueJob` / `getJob`. Job claim uses `FOR UPDATE SKIP LOCKED` so multiple Next.js workers can poll safely. - `lib/services/analyzer/pipeline.ts` — composes Stage 0 → idempotency check → Stage 1 → (Stage 2 if `itglue_lookup_needed`) → Stage 3 → (Stage 4 if trigger fires AND cost ceiling not reached) → result. Cost circuit breaker trips at $2.00 before Opus, sets `needs_human_review=true` with a reason. Returns full `model_traces` for debugging. - `lib/services/analyzer/worker.ts` — singleton with 2-second poll loop; auto-starts in production; opt-in in dev via `ANALYZER_WORKER_AUTOSTART=1`; **skipped** under vitest. `runJob()` exposed for tests + manual triggers. `TicketNotFoundError` produces a user-facing job error message that fingers the sync as the culprit. - 28 new tests bringing the total to 128. **Decisions worth flagging** - **Worker auto-start is more conservative than `sync-scheduler.ts`.** That one auto-starts on any non-browser import (including tests). I gated this one because the worker hits the database AND runs LLM calls — much higher blast radius. To run locally, set `ANALYZER_WORKER_AUTOSTART=1`. - **Cost circuit breaker only fires before Opus.** Sonnet runs unconditionally even if it would push past $2. Spec wording matches; if the team wants stricter control, the natural place is next to `COST_CEILING_USD` in `pipeline.ts`. - **Idempotency check matches `status='complete'` only.** Failed runs don't poison the cache. - **IT Glue failures are tolerated.** Search throws → analysis continues without context, doesn't fail the run. - `worker.test.ts` casts `mockImplementationOnce` to `as never` because vitest's overload resolution fights us when the mocked function has multiple call signatures. Functional, just ugly. **Deliberately left out** - No live API integration test (still). Phase 5+ is the natural place for an end-to-end smoke test. - Stage 3 doesn't have a fixture-driven happy-path test like Stage 1; it's exercised at the orchestration layer via `pipeline.test.ts` only. Worth adding direct Stage 3 tests later. - Migration `069` still not applied — same as phase 1. --- ## Phase 5 — API routes **Delivered** - Persistence read paths added to `lib/services/analyzer/persistence.ts`: `getAnalysisById`, `listAnalysesByTicketNumber`, `listNeedsReview`, `createShare`, plus a shared `rowToPersistedAnalysis` row mapper. - 6 routes under `app/api/analyzer/`: | Route | Method | Returns | |---|---|---| | `tickets/[ticketNumber]/analyze` | POST | `{status, jobId?, existingAnalysisId?}` | | `tickets/[ticketNumber]/analyses` | GET | `{analyses: PersistedAnalysis[]}` | | `jobs/[jobId]` | GET | `{job}` | | `analyses/[id]` | GET | `{analysis}` | | `analyses/[id]/share` | POST | `{share}` | | `needs-review?limit=&offset=` | GET | `{analyses}` | **Decisions worth flagging** - **Analyze runs preprocess inline.** The route does `loadTicketBundle → preprocessTicket → findExistingAnalysisByContentHash` synchronously to support the spec's `existingAnalysisId?` immediate response. The worker also runs this — duplicated work, but preprocess is fast (deterministic, one DB load) and the alternative (always queue, frontend polls to discover the short-circuit) is worse UX. - **Email send deferred to phase 8** per the spec's delivery order. The share route persists the audit row and validates the recipient domain against `ALLOWED_SHARE_DOMAINS`. Until phase 8, share rows have `viewed_at = null` indefinitely. - **`requireAuth()` everywhere — not `requireAdmin()`.** Any authenticated user can analyze a ticket they have access to. If `/needs-review` should be admin-only later, swap that one to `requirePermission('analyzer', 'review')` once the permission map is decided. - **No new permission entries added to `lib/permissions.ts`.** Adding scoped permissions for a feature still under build risks getting them wrong. - Routes are NOT in `middleware.ts` `publicRoutes` — they require a session. **Deliberately left out** - No API route tests. The repo has zero `app/api/**/*.test.ts` files; the routes are thin orchestration on top of already-tested persistence. Adding integration tests means setting up a test harness for the auth helpers + Postgres, which is a separate effort. --- ## Phase 6 — Frontend pages **Delivered** - `components/analyzer/analyze-button.tsx` — `` with the full state machine: POSTs to the analyze endpoint, navigates straight to an existing analysis if `existingAnalysisId` came back, otherwise polls `/api/analyzer/jobs/:jobId` every 2s and renders stage labels (Queued → Fetching → Triaging → Searching IT Glue → Analyzing → Deep review → Done). 5-minute hard timeout. Failures surface as toast errors. Supports `force` for explicit re-run. - `components/analyzer/share-modal.tsx` — `` with a shadcn Dialog + email field + optional note (max 2000 chars). POSTs to the share endpoint and surfaces server-side validation errors (domain not allowed → toast). - `components/analyzer/analysis-view.tsx` — the full 10-section analysis layout per spec. Header with model-tier badges (Haiku/Sonnet/Opus pills), confidence score, total cost, Share + Re-analyze buttons. Summary, Next Step (with rationale collapsed), Timeline (vertical list with 🟢 / 🔒 / 🔄 markers, click to expand), What Was Done / Should Have Been Done side-by-side on wide screens, Gaps colored by severity with "Evidence:" links that scroll-and-expand the matching timeline event, Post-Resolution (only if present), Human Review Flags (only if needed), IT Glue References. - `app/analyzer/ticket/[ticketNumber]/page.tsx` — ticket detail with the Analyze button and a list of historical versions; latest is badged. - `app/analyzer/analysis/[id]/page.tsx` — fetches one analysis and renders it via ``. - `app/analyzer/queue/page.tsx` — needs-review queue, shows ticket number, version, summary preview, top reasons, confidence badge, and a `high gap` badge if any gap is high severity. **Decisions worth flagging** - **Imperative `useState` + `useEffect` + `fetch`**, no SWR / react-query — matches CLAUDE.md and the rest of the repo. Don't refactor to a global cache layer for these three pages; if it becomes a real pain, that's a whole-app concern. - **`'use client'` everywhere.** The pages use `use(params)` (the React hook) to unwrap Next.js 16's `params: Promise<...>` shape on the client. Server components weren't appropriate here — every page does interactive state (analyze flow, expand events, share modal). - **Stage labels render directly from `JobStatus` enum values**, not a separate label list, so any new statuses added to the enum auto-render with their default name. - **Re-analyze banner not implemented yet.** The spec calls for "New activity since last analysis · Re-analyze" when the live content_hash drifts from the persisted one. That requires a live-preprocess endpoint (or running preprocess on the page render). Skipped for now — the user can always click Re-analyze. Worth adding once we have real-world signal on whether activity drift is common. - **No autocomplete on the share-recipient field.** Spec says "autocomplete from existing wulf-pulse user list if available". Skipped — the existing user list is in Better Auth's `user` table; exposing it requires a small API endpoint. Easy follow-up. - **No navigation entry yet.** `components/navigation/app-navigation.tsx` doesn't yet have an "Analyzer" link. Adding that is a one-line edit; I left it for the operator to opt in once the feature is staged. - **Printable analysis view** — the spec mentions "printable" for the analysis page. The current layout is print-friendly by accident (no fixed sidebars, sectioned cards), but no explicit `@media print` styles yet. Add when someone asks. **Deliberately left out** - No frontend tests. vitest is configured for `lib/**/*.test.ts` only; the pages and components are visually verified. Component tests with testing-library would be a separate setup decision. - Analysis view doesn't auto-refresh while a job is running on a *different* version. If a user navigates to an old version while a new one is in progress, they don't see the in-progress state. Acceptable — the queue view + ticket history give that signal. --- ## Phase 7 — Share-via-email integration **Delivered** - `sendAnalysisShareEmail()` added to `lib/services/email.ts` — re-uses the existing nodemailer SMTP transport that already serves magic-link and invitation mail. Subject `Pulse analysis · v — `, gradient-header HTML body matching the other Pulse emails, plain text fallback, `replyTo` set to the sharer so a recipient reply lands with the right person. - Share route `app/api/analyzer/analyses/[id]/share/route.ts` now resolves the sharer's session, builds the analysis URL from `BETTER_AUTH_URL` (falling back to `NEXT_PUBLIC_BETTER_AUTH_URL`, then `localhost:3100`), persists the audit row, then attempts the email send. - Share-modal frontend handles the new `emailSent`/`emailError` fields: success → green toast; row-saved-but-send-failed → orange `toast.warning` carrying the SMTP error. **Decisions worth flagging** - **Audit row persists even if email send fails.** The `analyzer_shares` row is the audit log, not just a delivery receipt. SMTP outages should not erase the record that the user attempted a share. - **Response is HTTP 200 on email failure.** The route returns `{share, emailSent: false, emailError}`. A non-2xx would imply the share itself failed; surfacing `emailSent: false` is the more honest signal. - **No `email_sent_at` column added.** Adding it requires a migration and a way to retry — neither is asked for by the spec. Send state lives only in the response and the server log line `[ANALYZER-SHARE] email send failed for share `. If retries become a real need, a dedicated `analyzer_share_email_attempts` table is the natural shape. - **Used existing nodemailer SMTP, not Graph sendMail.** Spec said "via M365 Graph using existing wulf-pulse mail integration if one exists; otherwise use a new module" — `email.ts` *is* the existing module. The Graph integration in this codebase is read-only (mailbox search, user reports), not send-capable. - **HTML escaping is hand-rolled.** Five-replace function for amp/lt/gt/quote/apos. The user-controlled fields (sender name, note, analysis summary, next step) all flow through it. The repo has no HTML escaper utility and the magic-link/invitation emails don't need one because their inputs are URLs and admin-set names. **Deliberately left out** - **No retries.** A failed send is logged once and surfaced to the user. They can re-share if they want — that creates a fresh audit row, which is correct behavior. - **No tests.** `lib/services/email.ts` has no existing tests, mocking nodemailer fully would add a non-trivial test scaffold for one function, and the share route is route-handler thin. Consistent with phases 5/6. - **No `viewed_at` tracking yet.** The migration has the column but nothing writes to it. A `/share/:id/viewed` endpoint with an unguessable token would be the smallest addition; spec didn't ask, so skipped. --- ## Phase 8 — Operator runbook **Delivered** - `docs/wulf-pulse-ticket-analyzer-runbook.md` — cost monitoring queries (daily spend, top expensive analyses, multi-run tickets), the cost ceiling explanation (`COST_CEILING_USD = 2.00` in `pipeline.ts`), the IT Glue alias workflow (where to find org IDs, how to verify via `model_traces.itglue.alias_used`), failure triage tables (failed jobs vs. `needs_human_review` flagged completes, with the actual error strings the worker emits), and a manual-ops section for queueing jobs from psql. - README.md "Documentation" section links the runbook so operators finding the project for the first time discover it. **Decisions worth flagging** - **Runbook lives in `docs/`, not the README.** Per CLAUDE.md, long-form per-feature docs go in `docs/` and the README points to them. Matches the existing pattern (workflow-editor-guide, sync guides, etc.). - **Manual migration step is documented.** Postgres only re-applies migrations on first init — operators applying analyzer to an existing DB need to run `psql -f migrations/069_*.sql`. The runbook leads with this. - **Listed what's *not* implemented.** Retries, `viewed_at`, share `email_sent_at`, pre-Sonnet cost gate, share-recipient autocomplete. Better to advertise the gaps than have an operator trip over them. **Deliberately left out** - No production-monitoring dashboard (Grafana etc.) — Pulse doesn't have one for any other feature, and the SQL queries cover the same ground. - No README expansion for the analyzer feature itself. Pulse's README is intentionally short; the runbook + spec + build notes are the deep docs. --- ## Phase 9 — Ticket browser + analysis-view formatting Reactive to first-use feedback: the analysis page Summary / Next Step text was bare and dense, and there was no way to discover tickets to analyze without typing the URL. **Delivered** - `` helper inside `analysis-view.tsx` — splits text on blank lines, renders each chunk as a separate `

` with `leading-7` and `whitespace-pre-line`. Applied to Summary, Next Step, Next Step rationale, and Post-Resolution Analysis. Single-paragraph text still renders cleanly. - Summary + Post-Resolution headers got an uppercase tracking-wide treatment to act as section dividers, and body text bumped to `text-base text-foreground` so it reads as a finding rather than a caption. - Next Step card now has a subtle `bg-primary/5` tint, an `ArrowRight` icon next to "Recommended Next Step", a stronger separator before the rationale collapsible, and the rationale itself renders in a bordered indented block. - New `/analyzer/tickets` browse page — pill-style period chips (Today, Yesterday, This week, Last week, Last 30/60 days, All time), a client (company) Select, an issue-type Select, and a debounced free-text search across ticket_number/title. Compact table with per-row Analyze/Re-analyze button (reusing ``) and a "View" button shortcut to the existing analysis when one is recorded. Active-filter count + clear-all in the filter card header. - New API `GET /api/analyzer/tickets/list` — filters by `period` (computed in Postgres against `last_activity_date`), `companyId`, `issueType`, `search`. Returns 50 rows + total via `COUNT(*) OVER ()`, plus `latestAnalysisId` from a LATERAL join into `analyzer_analyses`. - New API `GET /api/analyzer/tickets/filter-options` — companies that have at least one non-deleted ticket (drops dormant accounts) + active issue types ordered by `sort_order, label`. - Top-level "Analyzer" nav menu added to `app-navigation.tsx`, with "Browse Tickets" + "Needs Review". Earlier phases left this off intentionally; this phase opts in. **Decisions worth flagging** - **Period filters on `last_activity_date`, not `create_date`.** "Today" surfaces tickets that had activity today (new tickets, re-opened, status churn) — much more useful for an analyzer-driven triage flow than tickets created today. A new ticket created today also has activity today, so we don't lose those. - **Period math runs in Postgres via `date_trunc('day', NOW())` etc.** Database server-clock = app-process clock for an internal Docker stack, so naive timestamps and naive `NOW()` agree. If users complain about edge-of-day drift, swap to `NOW() AT TIME ZONE 'America/New_York'` — Pulse's primary user base. - **Default period is `last_30d`.** "All time" pulls many thousands of rows; defaulting wide-open hurts first-page latency. 30 days hits ~7K rows in our DB, paginates cleanly. - **Per-row analyze button reuses `` directly.** Each row gets its own component instance — no shared state, the running state lives per-button. The button navigates on completion, which feels right: click Analyze, watch the stages, land on the analysis page. - **`force=true` is set automatically on tickets that already have an analysis.** Re-analyze should re-run, not short-circuit to the cached row. The "View" button covers the cached path. - **No Linear/JIRA-style multi-select filters.** Single-value Selects are simpler and match the rest of Pulse. **Deliberately left out** - **No saved views.** A filter URL is shareable, but there's no bookmark / saved-view UX. Add when someone asks. - **No `latest_analysis_status` exposure.** A failed analysis doesn't show up — the LATERAL join filters by `status='complete'`. So a ticket whose only analysis failed looks like an un-analyzed ticket. Acceptable: re-running is the intended action there anyway. - **Search is `ILIKE '%...%'`.** No tsvector / trigram index. 7K-row scans are sub-100ms in this DB; if the corpus grows past low six digits, swap in `pg_trgm`. --- ## Status after each phase | Phase | Tests | tsc | Notes | |---|---|---|---| | 1 | 0 | clean | migration + types only | | 2 | 68 | clean | redaction + preprocessor regression | | 3 | 100 | clean | + LLM scaffolding + Stage 1 | | 4 | 128 | clean | + pipeline + worker | | 5 | 128 | clean | API routes (no route tests) | | 6 | 128 | clean | frontend (no FE tests) | | 7 | 128 | clean | share email via existing SMTP transport | | 8 | 128 | clean | operator runbook + README link | | 9 | 128 | clean | browse page + analysis-view formatting + nav entry | --- # Phase 2 (cross-ticket analysis) Spec: `docs/ticket-analyzer-phase2-spec.md`. Eight sub-phases delivered as one Phase 2 push. ## 2.1 — Schema additions **Delivered** - Migration 070: `analyzer_stage_executions` table (per-stage I/O for every analyzer run, including failed attempts) + three columns on `analyzer_analyses`: `source_snapshot`, `aggregate_fingerprint`, `fingerprint_generated_at`. - `model_traces` column annotated with a `LEGACY` `COMMENT ON COLUMN` for the SQL side and a `// LEGACY` doc comment in TS — kept for back-compat until aggregate reports have soaked. - Zod schemas: `AggregateFingerprint`, `StageName`, `StageExecution` (read-back), `StageExecutionRecord` (write-time interface). **Decisions worth flagging** - `analyzer_stage_executions.analysis_id` starts NOT NULL in 070. Migration 071 (Phase 2.6) relaxes it and adds the mutually-exclusive CHECK with `aggregate_report_id`. - 070 is idempotent (`IF NOT EXISTS` on every object) so re-running against an already-applied DB is safe. ## 2.2 — Pipeline writes to stage executions **Delivered** - `recordedStage(meta, fn, callbacks, outputSelector)` helper in `pipeline.ts` wraps each stage call, emits a `StageExecutionRecord` on success or failure (re-throws after recording). Pipeline wires it into Stage 1 (triage), Stage 3 (analyze), Stage 4 (deep_review). Stage 0 (preprocess) and Stage 2 (itglue) are recorded inline since they're not LLM calls. - Pipeline result now includes `triage_response`, `sonnet_response`, `opus_response` for downstream stages (Stage 6 fingerprint). - Worker's `runJob` collects records via `onStageRecord` callback, bulk-inserts them after `insertAnalysis` succeeds. On pipeline throw, worker captures the preprocessed bundle via `onPreprocessed` callback, persists a `status='failed'` analyzer_analyses row with source_snapshot intact, and bulk-inserts the partial stage records linked to it. - New persistence functions: `bulkInsertStageExecutions`, `insertFailedAnalysis`, `updateAnalysisFingerprint`. **Decisions worth flagging** - **Failure-tolerant audit**: spec says "Every stage that runs MUST insert a row, including stages that fail." We persist a failed analyzer_analyses row even on pipeline crash so the partial stage records have a parent. Without this the FK would be orphaned. - **Single bulk insert**: ~5–6 stage rows per pipeline run. One multi-VALUES INSERT is fast enough; no need for COPY. - **`model_traces` double-write retained**: the legacy column still receives the old payload. Drop it in a future migration once aggregate reports have soaked through prod. ## 2.3 — Prose formatting **Delivered** - Stage 3 system prompt updated with the markdown formatting rules from the spec verbatim (banned filler phrases, action-verb emphasis, no headers). - `react-markdown@10`, `remark-gfm@4`, `@tailwindcss/typography@0.5` added. Tailwind 4 plugin registered via `@plugin "@tailwindcss/typography"` in `app/globals.css`. - `` component at `components/analyzer/analysis-markdown.tsx` renders prose with `prose prose-sm dark:prose-invert max-w-none prose-p:leading-7`. Coerces stray model headers into bold paragraphs (the prompt forbids them but defense-in-depth). - `` removed. Summary, Recommended Next Step, next_step_rationale, and post_resolution_analysis all use ``. **Decisions worth flagging** - Tailwind 4 syntax: `@plugin "@tailwindcss/typography"` in CSS, no JS config needed. - Stage 3 prompt change is back-compatible — old analyses with plain-text summaries still render fine through ReactMarkdown. ## 2.4 — Stage 6 fingerprint + backfill CLI **Delivered** - `lib/services/analyzer/stages/stage6-fingerprint.ts` — Haiku call with the spec's verbatim system prompt. Server-overrides `generated_by_model` and `generated_at` after parse so the model's guess for those fields can't drift. - Worker integration: after `insertAnalysis` succeeds, run fingerprint with try/catch. Failure logs a warn, fingerprint stays NULL on the row, but the analysis is still complete and usable. The fingerprint stage record is added to the bulk insert whether it succeeded or failed. - `scripts/backfill-fingerprints.ts` — idempotent CLI. Reads `analyzer_analyses.model_traces.{triage_response, sonnet_response, opus_response}` (which Phase 1 was already storing), runs Stage 6, writes `aggregate_fingerprint`. Supports `--dry-run` and `--limit=N`. Skips analyses where model_traces is incomplete. **Decisions worth flagging** - Stage 6 input is just the analysis content (triage + sonnet + optional opus). No `pre` payload needed — fingerprinting is about the produced *analysis*, not the source ticket. - Backfill processes oldest-first (triggered_at ASC). Lets us observe a few rounds before chewing through hundreds. - Stage 6 failure is non-fatal. Spec: "If fingerprinting fails, do NOT fail the overall analysis." ## 2.5 — Browse / filter UI rebuild **Delivered** - New endpoint: `GET /api/analyzer/tickets` (replaces the simpler `GET /api/analyzer/tickets/list` from Phase 1.9). Multi-select CSV-style query params (clientId, issueType, queue, status, priority, assignedTo); analyzed segmented filter (any/yes/no/stale); needsReview toggle; search; sort. - `/api/analyzer/tickets/filter-options` extended with queues, statuses, priorities, resources (joined to "has at least one ticket" so the dropdowns aren't padded). - New `` component at `components/ui/multi-select.tsx` — Popover + checkbox list with optional search box (auto-shown above 8 options). One trigger + one popover, no shadcn Command dependency. - `/analyzer/tickets` page rebuilt: - Sticky filter bar with period pills, multi-selects, search, analyzed segmented, needs-review checkbox, sort - Active-filter chips (click to clear individual filter) - Bulk selection persisted via localStorage (key `analyzer:ticket-selection:v1`) — survives pagination - "Analyze N selected" — sequential job queue, forces re-analyze on `stale` rows - "Generate aggregate report" — routes to `/analyzer/reports/new`; only enabled when all selected are `current` - Top nav reorganized: Browse Tickets / Aggregate Reports / Needs Review under "Analyzer". **Decisions worth flagging** - **Staleness via `last_activity_date > completed_at`**, not content-hash compare. The spec lets either; the date heuristic is good enough and avoids per-row preprocessing on 50-row paginated responses. - **`MultiSelect` is a one-popover-per-instance design** — multiple popovers can be open across the bar. Acceptable; matches how Linear / Vercel's table filters behave. - **No "Select all matching filters" semantic**. Selection is an explicit per-row action stored as ticket numbers in localStorage. Filter-level selection adds significant complexity (server has to resolve filter→IDs, two-modes everywhere). Skipped for V1; the spec's intent (don't lose selection on pagination) is met. - **Bulk Analyze is sequential, not parallel.** N concurrent calls would all hit `claimQueuedJob` and the worker would process them one at a time anyway (single in-process worker). Sequential POSTs are more honest about that. ## 2.6 — Aggregate reports **Delivered** - Migration 071: `analyzer_aggregate_reports` table + ALTER on `analyzer_stage_executions` to drop NOT NULL on `analysis_id`, add `aggregate_report_id` FK, add the `analyzer_stage_executions_parent_check` CHECK constraint (`(analysis_id IS NOT NULL) <> (aggregate_report_id IS NOT NULL)`). - `lib/services/analyzer/stages/aggregate-reduce.ts` — Sonnet (Opus opt-in) reduce stage with the spec's verbatim system prompt and `AggregateReduceResponse` Zod schema. - `lib/services/analyzer/aggregate-persistence.ts` — `createAggregateReport`, `getAggregateReport`, `listAggregateReports`, `runAggregateReport`, `bulkInsertReportStageExecutions`. The runner is fire-and-forget (called via `void runAggregateReport(id)` from the POST endpoint); it persists distributions immediately so the UI can show partial results during the LLM call. - IT Glue context fetcher: per-client `findOrganizationByName` + `getFlexibleAssets`, capped at 200 doc titles total per spec. Failure tolerant — per-client errors don't fail the report. - API endpoints: - `POST /api/analyzer/aggregate-reports` — validates (≤100, fingerprint exists, not stale), creates pending row, fires runner - `GET /api/analyzer/aggregate-reports/:id` — full report row (UI polls this every 3s while pending/running) - `GET /api/analyzer/aggregate-reports` — paginated list - Pages: - `/analyzer/reports/new?ids=T...,T...` — pre-flight: shows selected tickets, options (title, IT Glue context toggle), Generate button - `/analyzer/reports/[id]` — pending → distributions → completed. Sections: header, executive summary, four distribution mini-bar cards, documentation gaps (with `itglue_check` color tone), process gaps (severity tone), recurrence clusters, recommended actions (sorted by priority), narrative summary. - `/analyzer/reports` — table list of past reports **Decisions worth flagging** - **Fire-and-forget runner**, no separate worker module. The POST endpoint kicks `void runAggregateReport(id)`; updates land in the row when the LLM call completes. Frontend polls. Avoids adding a second polling worker alongside `analyzerWorker`. - **Stage names reused for aggregate sub-stages.** The CHECK constraint on `analyzer_stage_executions.stage` enumerates the per-analysis stage names. Aggregate sub-stages (SQL aggregation, IT Glue context, reduce LLM) are recorded with `stage='analyze'` / `'itglue'` plus `aggregate_report_id` set. A future migration could add `aggregate_sql` / `aggregate_reduce` to the enum and re-emit those rows; for now the existing names are good enough for forensics. - **`generated_by_user_id` is TEXT nullable**, not `uuid NOT NULL` per spec. Better Auth's `user.id` is text, and we want the report to remain readable if the generating user is later deleted — matches the pattern from `analyzer_analyses.triggered_by_user_id`. - **Distributions persist before LLM call** so partial-state UI doesn't have to wait the full 30–90s for anything to render. ## 2.7 — Cost guards **Delivered** - Migration 072: `analyzer_cost_audit` table. - `lib/services/analyzer/cost-guard.ts`: `estimateAggregateReportCost`, `getUserDailySpend`, `evaluateCost`, `recordCostAuditDecision`. Thresholds: `REQUIRES_CONFIRMATION_USD = 5`, `SOFT_WARN_DAILY_USD = 20`, `HARD_BLOCK_DAILY_USD = 50`. - `evaluateCost` produces a four-state decision (`approved` / `requires_confirmation` / `blocked` / `overridden`) plus boolean `softWarn`/`hardBlocked`/`requiresConfirmation`/`isOverride` fields the API can return for UX. - POST `/api/analyzer/aggregate-reports` enforces: - `requires_confirmation` → 400 with `requiresConfirmation:true, estimatedCost, dailySpendBefore` so the frontend can show `confirm()` and re-POST with `confirmedCost:true`. - `blocked` → 403 with the daily spend in the body. - Every decision (including `approved`) writes a row to `analyzer_cost_audit`. - Override env var `ANALYZER_DAILY_COST_OVERRIDE_USERS` (comma-separated user ids). - Frontend new-report page: catches `requiresConfirmation`, shows `window.confirm()` with the dollar figure, retries with `confirmedCost: true`. **Decisions worth flagging** - **Cost estimate is char/4 → tokens × Sonnet pricing.** Crude but pessimistic in the right direction. At 100 tickets the estimate comes in under $0.50 — far below the $5 threshold — so the confirmation modal almost never fires in practice. Ceiling exists to catch payload bloat / Opus-opt-in scenarios. - **Daily window is trailing 24h, not "today UTC".** Avoids midnight-edge-of-day reset gaming; rolling window is what the spec calls "$X/day" naturally. - **Soft warn at $20/day is informational only.** Fields exposed in the cost evaluation; UI can choose to surface, but the API doesn't refuse to proceed. Hard block at $50/day is the only enforcement. ## 2.8 — Documentation **Delivered** - `docs/wulf-pulse-ticket-analyzer-runbook.md` — added Phase 2 sections covering stage execution forensics, fingerprint backfill, aggregate report flow + SQL queries, cost guard configuration + override env var, and the rebuilt browse UI behavior. - This file — the per-sub-phase notes above. --- ## Status after Phase 2 | Phase | Tests | tsc | Notes | |---|---|---|---| | 2.1 | 128 | clean | schema (070) | | 2.2 | 128 | clean | stage_executions writes + failure-tolerant persistence | | 2.3 | 128 | clean | markdown rendering + Stage 3 prompt | | 2.4 | 128 | clean | Stage 6 fingerprint + backfill CLI | | 2.5 | 128 | clean | browse UI rebuild | | 2.6 | 128 | clean | aggregate reports (071, runner, 3 endpoints, 3 pages) | | 2.7 | 128 | clean | cost guards (072, audit log, threshold gating) | | 2.8 | 128 | clean | runbook + build notes | --- ## Phase 3 — Link-aware bundle analysis **Why** Single-ticket analysis misses the bigger picture for master/problem tickets, which are explicitly aggregator records — a "Master problem ticket" with a `RELATED TICKETS:` block in its description naming the constituent incidents. Aggregate reports already existed (Phase 2.6) but required the user to pre-analyze every constituent and hand-pick them on `/analyzer/reports/new`. Phase 3 closes the gap: one click on a problem ticket fans out individual analyses for each linked ticket and chains them into an aggregate report. **Delivered** - `migrations/073_analyzer_link_aware_bundles.sql`: - `analyzer_aggregate_reports.expected_ticket_numbers TEXT[]` — the full set of ticket numbers a bundle is waiting on. - `analyzer_aggregate_reports.triggered_by_ticket_number TEXT` — the master ticket the bundle was launched from. - Status check extended to include `'pending_analyses'` (waiting for individual analyses) before transitioning to `'pending'` (ready for aggregate-reduce). - Replaced the partial pending-status index to cover the new state; added a GIN index on `expected_ticket_numbers` filtered to `pending_analyses` for the worker chain-trigger lookup. - `lib/services/analyzer/link-discovery.ts`: - **Explicit arm** (no LLM, deterministic): regex scan over the ticket description and each retained note for `T\d{8}\.\d{4}` references, detection of the structured `RELATED TICKETS:` block (refs inside it flagged `confidence: 'high'`), and resolution of `tickets.problem_ticket_id` to a ticket number. Self-references and refs not present in the local mirror are dropped silently. Capped at `MAX_EXPLICIT_LINKS = 15`. - **Suggested arm** (Haiku, opt-in): one LLM pass over recent same-company tickets (±30 days, capped at 50 candidates). Returns up to 5 candidates with one-sentence reasons. Hallucination-guarded — drops any number not in the candidate list. - `detectProblemTicket()` returns boolean + signal list; UI uses signals to decide whether to highlight the bundle CTA as the primary action. - `app/api/analyzer/tickets/[ticketNumber]/links/route.ts`: - `GET` returns the explicit arm only (cheap, called on page load). - `POST { includeSuggested: true }` runs both arms. - `app/api/analyzer/tickets/[ticketNumber]/analyze-bundle/route.ts`: - Validates master + linked tickets exist locally (single SQL roundtrip). - Runs per-ticket idempotency: existing complete analyses with matching content hash short-circuit; missing tickets are queued via the existing `queueJob()` helper. - Cost guard runs against the **new work only** — already-complete analyses don't add cost. Per-ticket estimate is a flat $0.15 (Sonnet-tier pessimistic) plus `estimateAggregateReportCost()` for the reduce step. - Creates one `analyzer_aggregate_reports` row in `'pending_analyses'` (or straight to `'pending'` and fires `runAggregateReport()` if everything was already complete). - Bundle cap: `MAX_BUNDLE_SIZE = 25`. - `lib/services/analyzer/aggregate-persistence.ts`: - `createAggregateReport` accepts `expectedTicketNumbers` and `triggeredByTicketNumber`. When set, status starts as `'pending_analyses'`. - `chainTriggerForCompletedAnalysis(ticketNumber, analysisId)` — called by the worker after each successful job. Atomically appends the analysis_id to every pending_analyses bundle expecting that ticket (deduped via `analysis_ids @> ARRAY[…]` guard) and re-checks whether the full set is now satisfied. Returns `readyReportIds` for the worker to fire `runAggregateReport()` on. - `lib/services/analyzer/worker.ts` — chain-trigger fires from both the success branch and the idempotent-short-circuit branch (the bundle endpoint's idempotency check happens at submit time, but a parallel analysis can complete between then and when the worker picks the job up). Failures here are logged but never fail the underlying job. - `components/analyzer/related-tickets-panel.tsx` — renders above the existing `` on `/analyzer/ticket/[ticketNumber]`: - Cheap GET on mount populates the panel only when refs exist or the ticket looks like a problem ticket — otherwise the component renders nothing. - Pre-checked checkboxes for explicit refs; Switch toggle to load AI-suggested refs (additive, suggestions show with a badge, unchecked by default). - Primary CTA is bundle ("Analyze with N linked tickets") highlighted when `isProblemTicket=true`. Single-ticket flow is preserved untouched on the existing AnalyzeButton in the parent header. - Polls `GET /api/analyzer/aggregate-reports/:id` every 3s after submit; routes to `/analyzer/reports/:id` on completion. - `lib/services/analyzer/link-discovery.test.ts` — 18 new tests covering the regex, RELATED TICKETS section bounds, problem-ticket signal detection, dedup/self-skip, mirror filtering, `MAX_EXPLICIT_LINKS` cap, and confidence-based sorting. **Decisions worth flagging** - **Bundle is opt-in via the panel, not auto.** A ticket that mentions another ticket once in passing (e.g. "see T20260101.0001 for context") shouldn't quietly trigger 2× the LLM cost on every analysis. The panel is the consent surface — pre-checked when explicit refs exist, but the user explicitly picks the CTA. - **`RELATED TICKETS:` is a strong signal, not a parser-required format.** The regex catches T-numbers anywhere; the structured block just promotes them to high confidence and acts as a problem-ticket signal. No new format is imposed on whoever writes the master ticket. - **Suggested arm uses Haiku, not Sonnet.** ~$0.005 per call against the $5 per-request confirmation threshold — never trips the modal. We never fail the whole call if the suggestion arm throws (logged-and-suppressed via try/catch in `discoverLinks`). - **Per-ticket cost estimate is flat $0.15.** We could compute it from the preprocessed event count, but at the bundle's typical size (3-10 tickets) that's $0.45 – $1.50 — far below the $5 confirmation threshold. Worth revisiting if we see real false-positive blocks. - **`pending_analyses` is the new status, distinct from `'pending'`.** Explicit two-step state lets the runner stay simple — it never has to ask "are all my analyses ready?" — that gate is the chain-trigger's job. Existing manual-multi-select reports continue to start at `'pending'`; their flow is untouched. - **`expected_ticket_numbers` matches via array containment, not a separate join table.** Postgres GIN gives us O(log n) lookup and the data lives where it's used — no new table, no foreign-key cascade decisions to make. - **Self-references and unknown tickets are dropped silently.** The user isn't asked to pick from a list; they get a clean "you have N linked tickets" panel. A ghost reference (T-number that doesn't exist in the mirror) is a sync gap, not a bundle decision. - **Fixture migration: `T20260424.0045.input.json` updated** to include `problem_ticket_id: null` so the type-strict load through the preprocessor still parses. The column is nullable in the data-access query and on the row type. **Deliberately left out** - No retroactive linking for already-completed analyses. If a master ticket got a single-ticket analysis before this shipped, the user re-runs from the panel to bundle. - No editing the bundle composition after submit. Re-run with a different selection if you want a different scope. - No time-window auto-correlation arm (e.g. "all tickets at this client in the last 6 hours"). At Hynes Industries on 2026-05-01 we observed 84 tickets in one day — auto-grouping by time would have been useless noise. Same-company time-window ranking is what the Haiku suggested arm is for. - No UI for the `/analyzer/reports/[id]` page to flag itself as a "bundle" vs a manual report. The new fields are surfaced in the API response but the page renders the same regardless. ## Status after Phase 3 | Phase | Tests | tsc | Notes | |---|---|---|---| | 3 | 146 | clean | link discovery, bundle endpoint, chain-trigger, panel (073) | --- ## Phase 4 — IT Glue asset audit + documentation write-back **Why** Single-ticket analysis already extracts `documentation_gaps_observed` per ticket (in `aggregate_fingerprint`), but nothing acts on them. The companion direction — audit IT Glue records *against* ticket history — converts a passive output into an actionable backlog and offers direct write-back. Concrete proof: T20260502.0033 (Hynes — tags not printing) was resolved by identifying a stopped Windows service on MISYS-SQL processing BarTender scan-folder text files. The IT Glue record `MISYS 6.3` (asset 17096940) had 12/17 fields empty — including Wulf Application Champion and Vendor Maintenance/Support — so the next tech with this issue would re-discover everything. **Delivered** - `migrations/075_itglue_audit.sql` — two new tables: - `itglue_asset_audits` — one row per audit run with full LLM context snapshot (asset traits at audit time, redacted), the gaps/promotions/ contradictions output, and cost. - `itglue_writes` — one row per PATCH attempt; before/after diff, who, when, status pending → committed | failed | reverted, audit_id provenance, raw IT Glue API response, source_evidence (tickets that prompted the gap). - `lib/services/itglue-client.ts`: - `updateFlexibleAsset(id, traits)` — PATCH /flexible_assets/:id with JSON:API body. - `refreshFlexibleAsset(id)` thin wrapper around getFlexibleAsset. - `getRawSingle(path, params)` — for callers that need raw attributes (created-at/updated-at) for upserts. - `isITGlueConfigured()` helper. - Internal `patch(path, body)` mirrors the existing `request` pattern. - `lib/services/itglue-sync-service.ts`: - `refreshFlexibleAssetById(id)` per-record sync helper. Avoids running the full 27-entity `fullSync()` after every write. - `lib/permissions.ts`: - New `itglue: ['read', 'write']` permission. Admin + super-admin get write; user gets read-only. - `lib/services/analyzer/asset-audit/`: - `data-builder.ts` — collects all six context arms: asset snapshot, field schema with hints, peer exemplars same-client (top 5 by trait fill count), peer exemplars across all clients (top 3), per-field fill-rate stats (per-client + global), recent ticket fingerprints matching the asset's name. Redacts asset traits + every peer. - `prompt.ts` — single Sonnet/V4 Pro call; system prompt categorizes findings into field_gaps / notes_promotions / contradictions; refuses to suggest credential-shaped fields. Payload cap 80KB; trims peer_global first, then oldest tickets. - `runner.ts` — `runAssetAudit({assetId, generatedByUserId, provider})`. Provider-aware via `stageModelsFor(provider).deep_analysis`. Persists `itglue_asset_audits` row (or `failed` row on throw). - `persistence.ts` — typed read/write of both tables; `fieldNameToTraitKey()` helper matches IT Glue's `lower-hyphen-strip` convention. - `runner.test.ts` — 12 unit tests covering fillCount semantics, trait-key conversion, AssetAuditResponse Zod validation, payload trimming. - API routes: - `GET /api/analyzer/itglue/applications` — list of all Application records joined to latest audit, ordered worst-score-first. - `GET /api/analyzer/itglue/applications/[id]` — asset detail with field schema (the renderer uses field order + populated state). - `GET /api/analyzer/itglue/applications/[id]/audit?history=1` — latest audit + history. - `POST /api/analyzer/itglue/applications/[id]/audit` — runs a fresh audit; cost-guard via `recordCostAuditDecision({action:'itglue_audit'})`. - `POST /api/analyzer/itglue/applications/[id]/apply` — admin-only; inserts pending row, calls IT Glue PATCH, marks committed/failed, refreshes mirror, writes generic audit_log row. - `POST /api/analyzer/itglue/applications/[id]/revert/[writeId]` — admin-only; inserts a new write row with reversed before/after, applies, marks original status='reverted'. - `GET /api/analyzer/itglue/applications/[id]/writes` — auth-only per-asset history. - `GET /api/analyzer/itglue/writes` — admin-only cross-asset write log. - UI: - `/analyzer/itglue/applications` — list with score badges + filter input. - `/analyzer/itglue/applications/[id]` — asset header (with link to IT Glue), audit panel (gaps with severity-toned cards, notes promotions, contradictions), current fields rendered with hints for empty ones, write history with revert button, audit history with score timeline. `` reused from Phase 3. - `/admin/itglue-writes` — admin-only global write log with status filters. - Navigation: new "IT Glue audit" entry under the Analyzer dropdown. **Audit-trail design** Three layers of trail, all permanent: 1. `itglue_asset_audits` — every audit run with full LLM context. 2. `itglue_writes` — every write attempt. Before/after, who, when, status, audit provenance. Reverts produce a new row with reversed diff; original row → `status='reverted'`. The chain is always traceable. 3. Generic `audit_log` (existing migration 014) — written in parallel via `audit.log()`. Action `itglue.write` / `itglue.revert`, resource `flexible_asset`, resourceId = asset_id, details = `{ field_name, before, after, audit_id }`. Surfaces in `/admin/audit-log` next to every other admin action. **Decisions worth flagging** - **Two domain-specific tables, not one generic events table.** Audits carry full LLM context (heavy, infrequent). Writes are atomic per-field decisions with hard-typed before/after diffs. The generic `audit_log`'s free-form JSONB doesn't model the diff cleanly — but we still write to it so admins see a unified feed. - **Admin-direct write, no two-step approval.** Per the user's call. The audit log is the safety net. We left `approval_requests` as a possible v2 if mistakes start happening. - **Per-record sync helper instead of `fullSync()` after every write.** `refreshFlexibleAssetById()` does one GET + one upsert. Keeping `fullSync()` available for ops + scheduler; bypassing it on the write path keeps Apply latency under a second after the IT Glue PATCH lands. - **Credential refusal at three layers.** Prompt instructs the LLM not to suggest password/secret/key/token/credential fields. The Apply endpoint also regex-blocks any field name matching that pattern. The `redact()` utility from `itglue-redact.ts` strips matching keys from any payload flowing into the LLM in the first place. - **Trait-key derivation in code, not hard-coded.** IT Glue's convention is field-name lowercased, non-alphanum → single hyphen, stripped. We compute this from each field's `name` (verified against the live Hynes MISYS 6.3 trait map). If a future field doesn't match the rule, it'd surface as an unmatched fill-rate / value-not-applied — easy to spot. - **Fill-rate computation in JS, not SQL.** Avoids one query per field. At the example data volume (~92 active Hynes assets, ~few-thousand globally per type) this stays sub-100ms; can revisit if it grows. - **Peer-exemplar ranking by populated-key count.** Cheap proxy for "well-documented." Doesn't penalize asset types whose fields are legitimately optional. If the LLM starts producing strange suggestions we can refine to require/expected-field weighting. - **Asset detail page renders from local mirror, not IT Glue API.** Means a user could see a stale value for a few seconds between Apply and the per-record sync landing. Acceptable for a read view; the API response from Apply returns the freshly-PATCH'd asset so the UI can immediately reflect the new state. **Deliberately left out** - v1 covers Applications only (`flexible_asset_type_id = 3790`). The schema generalizes (`asset_type` is a column), but Configurations / Procedures / Domains have different shapes and prompts. One type at a time. - No bulk-apply. Admin clicks each suggestion. If a single audit produces 10 gaps that's 10 clicks — fine for v1; bulk-apply is an easy follow-on. - No two-step approval workflow. - No auto-create of new asset records — Apply only updates existing. - No inline editor for suggested values — admin sees the LLM's suggestion verbatim and clicks Apply, then edits in IT Glue if they want to tweak. - Passwords / Secrets / Keys / Tokens / Credentials: never written via this surface, ever. Refused at prompt + endpoint + redaction layers. ## Status after Phase 4 | Phase | Tests | tsc | Notes | |---|---|---|---| | 4 | 158 | clean | IT Glue asset audit (075), runner + 6 endpoints + 3 pages, write-back with revert | --- ## Phase 4.1 — Ticket-first capture + Configurations + cross-reference index **Why** Phase 4 was asset-first (admin browses worst-scoring records). Phase 4.1 flips perspective: every time we analyze a ticket, learn whether *this ticket* taught us something documentable. Plus extends write-back to IT Glue **Configurations** (servers, workstations, devices) — the prior flexible-asset-only scope missed records like MISYS-SQL where the T20260502.0033 root cause actually lived. Plus a cross-reference table so both perspectives become one-query lookups, and the future RAG automation has its lookup index. **Delivered** - `migrations/076_itglue_ticket_xrefs.sql`: - `triggered_by_ticket_number` + `triggered_by_analysis_id` on `itglue_asset_audits`; `triggered_by_ticket_number` on `itglue_writes` (denormalized per user's call so "every write a ticket drove" is a direct query). - asset_type CHECK extended to include `'configuration'` on both audit + write tables. - New `itglue_ticket_xrefs` table — ticket↔asset linkage with relationship type (`referenced` | `updated` | `should_have_referenced`), source (`analyzer_referenced` | `audit_write` | `manual`), unique constraint preventing dup ingestion. - `lib/services/itglue-client.ts`: - `updateConfiguration(id, attributes)` — PATCH /configurations/:id with flat JSON:API attributes (no traits blob). - `refreshConfiguration(id)` thin wrapper. - `lib/services/itglue-sync-service.ts`: - `refreshConfigurationById(id)` per-record refresh (mirrors the bulk syncConfigurations 33-column upsert). - `lib/services/analyzer/asset-audit/`: - `data-builder.ts` generalized: dispatches on `assetType`, supports `ticketScopeAnalysisId` for ticket-first audits. Configurations get a hand-curated 16-field schema with hints (since IT Glue Configurations don't have a `_fields` table). Synthesizes a "traits" map from flat columns so the prompt stays type-agnostic. - `prompt.ts` — two system prompts (Application-flavored vs Configuration-flavored, the latter focused on hostname/FQDN, OS currency, named services, IP/MAC, contact ownership). Ticket-scoped suffix when an analysis is the source so the LLM frames findings as "what did this ticket teach us?" - `runner.ts` accepts `assetType`, `ticketScopeAnalysisId`; persists `triggered_by_*` columns. - `persistence.ts` — `asset_type` union extended; new `getLatestTicketScopedAudit` helper for the analysis-page panel; `createPendingWrite` accepts asset_type + triggered_by_ticket_number. - `xrefs.ts` (new) — bulk-insert helpers: `insertReferencedXrefsFromAnalysis` (post-analysis hook), `insertUpdatedXref` (post-apply hook), with listXrefsForAsset / listXrefsForTicket queries. - `asset-matcher.ts` (new) — given an analysis_id, returns matched flexible_assets + configurations for the ticket's client based on fingerprint terms (applications_involved, device_classes, vendors_involved). Score = exact (3) > word-boundary (2) > substring (1); top 5 per kind. - `lib/services/analyzer/worker.ts` — post-analysis hook calls `insertReferencedXrefsFromAnalysis` for every doc the LLM cited; best-effort, never fails the job. - API routes: - `GET /api/analyzer/analyses/[id]/itglue-suggestions` — match assets + return any existing ticket-scoped audits keyed by (assetType, assetId). - `POST /api/analyzer/analyses/[id]/itglue-suggestions` — body `{ assetType, assetId, provider }`, runs a ticket-scoped audit. - Full Configurations route tree mirroring Applications: list, detail, audit (GET/POST), apply (admin), revert (admin), writes, xrefs. - `GET /api/analyzer/applications/[id]/xrefs` (new) and `GET /api/analyzer/tickets/[ticketNumber]/itglue-xrefs` (new). - UI: - `` rendered on the analysis detail page. Opt-in trigger ("Check IT Glue documentation"); shows matched Applications + Configurations grouped, per-asset ticket-scoped audit buttons, inline gap cards with Apply (admin-only), score badges. Reuses `` from Phase 3. - `/analyzer/itglue/configurations` — list mirroring Applications. - `/analyzer/itglue/configurations/[id]` — detail mirroring Applications, plus the new "Tickets that touched this configuration" section. - Application detail page — added the same xref section. - Navigation split into "IT Glue — Applications" + "IT Glue — Configurations". **Decisions worth flagging** - **Two separate Configuration write methods + two route trees instead of one polymorphic surface.** The codebase has no other `[assetType]`-style polymorphism; existing patterns favor parallel resource paths. Added ~80 LOC duplication on the page side, but each surface is independently testable + obvious in URL routing. - **Configuration field schema is hand-curated**, not loaded from IT Glue. IT Glue exposes flexible-asset field metadata via `/flexible_asset_fields` but Configurations have no equivalent endpoint. The 16 hand-written hints (in `data-builder.ts`) are what the LLM sees as field documentation. Versioned in code; PR review is the change control. - **Apply on Configurations is column-allowlisted.** Even with the audit pipeline picking `field_name`, the apply route refuses anything outside `name | hostname | primary_ip | mac_address | serial_number | asset_tag | position | notes | operating_system_notes`. Stops the LLM from suggesting edits to read-only/derived fields like `manufacturer_id` (which is an FK resolved by IT Glue, not a free-text field). - **xref ingestion happens in the worker after a successful analysis**, not as a separate batch job. Best-effort wrap means a transient DB hiccup never fails the analysis itself. The unique index on the xref table ensures retries are idempotent. - **ticketScopeAnalysisId narrows ticket evidence to one row.** This is the key prompt-shaping decision for Phase 4.1: the LLM sees just the one analysis the user clicked from, plus the asset state + schema + peer exemplars. Findings frame as "what *this ticket* revealed" rather than all-time history. - **No backfill of existing analyses.** Per user's call. The xref table fills forward; backfill is a future opt-in script if needed. - **Asset matching is loose** — substring + word-boundary. A ticket mentioning "MISYS" matches both `MISYS 6.3` (the Application) and `MISYS-SQL` (the Configuration), and the user picks per-asset which to audit. Less false-negative-y than strict matching; user controls confirmation. - **Configuration audits don't write to manufacturer/model/OS-name/contact/location** — those are FK fields IT Glue resolves by id, not free-text. The audit prompt can suggest changes but Apply blocks them. Future iteration could resolve names → ids via the IT Glue manufacturers/models endpoints. **Deliberately left out** - **Datto RMM script execution** (Phase 4.2 — separate plan). Ability to run PowerShell via Datto RMM Overshell on Wulf Nurse endpoints to gather fresh evidence (DHCP scopes, DNS zones, AD info, named services) and feed it into the audit pipeline. Decisions logged: generic Overshell component + Pulse-managed scripts; admin-direct with audit log; new `rmm.execute` permission. - No backfill of the xref table. - No bulk-apply across multiple gaps; admin clicks each one. - No two-step approval workflow. Audit log + role gating remain the safety net. - No Configuration write for FK-shaped fields (manufacturer, model, OS, contact, location) — only flat editable columns. ## Status after Phase 4.1 | Phase | Tests | tsc | Notes | |---|---|---|---| | 4.1 | 160 | clean | xref table (076), Configurations parity, ticket-first capture, analysis-page panel | --- ## Phase 4.2 — Datto RMM Overshell evidence pipeline **Why** Phase 4.1 wires ticket history + IT Glue field schemas into LLM-driven documentation suggestions. The next leverage point is **fresh evidence from the live environment** — service lists, AD health, DHCP scopes, DNS zones, event logs — that ticket history can't surface. Without it, audits flag "the named service that processes BarTender scan-folder text files isn't documented" but can't suggest the actual service name. With it, we suggest the literal value pulled from the running server seconds ago. The proven test case: openclaw produced an AD health summary at Hynes around 2026-04-25 (IP conflicts, ZR006 missing trust account, DNS forwarders timing out, Hendricks site missing site-links) by orchestrating Datto RMM Overshell. Phase 4.2 lets Pulse produce the same intel directly from a button on the asset page, store it, and feed it back into audits. **Delivered** - `migrations/077_rmm_overshell.sql`: - `rmm_settings` singleton — caches the discovered Overshell `component_uid`, `component_name`, `variable_name` (default `CommandLine`). - `rmm_executions` — full lifecycle row per dispatch: queued → running → complete | failed | timeout. Captures `target_device_uid`, `target_hostname`, `target_company_id`, optional audit/asset linkage, `job_uid`, raw stdout/stderr (redacted), `parsed_evidence`, exit code, `timeout_at`. 8 indexes covering all query paths the audit pipeline + UI need. - `lib/services/rmm/scripts/`: - 7 v1 scripts, all read-only. Each is a typed `RmmScript` exporting `body` (PowerShell), `target_type`, `parseOutput`, `expected_runtime_seconds`, `version`. Bodies end with `ConvertTo-Json -Depth … -Compress` so the parser is just `JSON.parse`. Registry validates uniqueness at load. - asset_self: `get-services`, `get-installed-software`, `get-event-log-recent`. - site_anchor: `get-ad-health` (mirrors the openclaw test case), `get-dhcp-scopes`, `get-dns-zones`, `get-network-discovery` (catches the IP-conflict pattern from the proven test case). - `lib/services/rmm/target-resolver.ts`: - `resolveSiteAnchorTarget(companyId)` — looks up `datto_rmm_sites` where `autotask_company_id = $1`, finds devices matching `^[A-Z]{3}[A-Z]{3}WNP\d{2}$`, picks online + lowest numeric suffix. - `resolveAssetSelfTarget(deviceUid)` — direct lookup. - `resolveDeviceByHostname(hostname)` — fallback when an IT Glue Configuration's `rmm_id` doesn't resolve cleanly. - `lib/services/rmm/settings.ts`: - `discoverOvershellComponent()` — calls `client.findOvershellComponent(/overshell/i)` and persists the uid. - `resolveOvershellComponent()` — read-cache-or-discover; throws if nothing matches. - `lib/services/rmm/executor.ts`: - `queueExecution({ scriptId, target, performedByUserId, triggeredByAuditId? })` — validates registry, resolves target, runs cost-guard rate limit (50/user/24h, decision logged to `analyzer_cost_audit`), inserts pending row, calls `runQuickJob`, captures `jobUid`, flips to `running`. Generic `audit.log` entry on success. - `lib/services/rmm/worker.ts`: - 5-second poll loop, self-init pattern matching `analyzerWorker`. - Sweeps timed-out rows first (status → `timeout`). - Polls `running` rows via `client.getJobResults` per `target_device_uid`. On terminal status: redacts stdout/stderr, runs script's `parseOutput`, persists. Parse errors are non-fatal — raw output still kept. - `lib/services/rmm/persistence.ts`: - Typed `RmmExecutionRow` + status helpers, plus the audit-pipeline queries `listLatestEvidenceForCompany(companyId, days)` and `listLatestEvidenceForAsset(assetType, assetId)`. - `lib/services/datto-rmm-client.ts` — added `findOvershellComponent(pattern)`. - `lib/services/analyzer/asset-audit/data-builder.ts`: - 7th LLM context arm `rmm_evidence` populated from `listLatestEvidenceForCompany` (site-anchored, last 7 days) + `listLatestEvidenceForAsset` (asset-self, all-time). - Joins via `companies → itg_organizations` on case-insensitive `company_name` match (same join the ticket-evidence loader uses). - `lib/services/analyzer/asset-audit/prompt.ts`: - New `=== LIVE RMM EVIDENCE ===` section emits when `ctx.rmm_evidence.length > 0`. Trim path drops it last (highest-value section). - `LIVE_EVIDENCE_NOTE` injected into the system prompt: *"Treat parsed contents as authoritative current state … Cite execution_id alongside ticket numbers."* - `lib/permissions.ts` — new `rmm: ['read','execute']`. Admin + super-admin get both; user gets read. - API routes: - `GET/PATCH /api/admin/rmm/settings` — admin-only, view + edit variable name. - `POST /api/admin/rmm/settings/discover` — admin-only, force component scan. - `GET /api/rmm/scripts` — auth, library catalog (no bodies). - `GET/POST /api/rmm/executions` — list (auth) + queue (`rmm.execute`). - `GET /api/rmm/executions/[id]` — auth, polls one execution. - `GET /api/analyzer/itglue/sites/[companyId]` — site-discovery summary. - UI: - `` — popover listing applicable scripts, dispatches on click, disables for non-admins. - `` — polls every 3s, shows status + parsed evidence + raw stdout (collapsible). - `/admin/rmm-overshell` — settings + recent execution log. - `/analyzer/itglue/sites/[companyId]` — site-discovery view. - Embedded picker on Application detail (site-anchor with parent client) and Configuration detail (asset-self if `rmm_id` resolves to a Datto device). - Nav entry: Admin → "RMM Overshell". **Decisions worth flagging** - **Component discovery is automatic and cached.** Pulse scans for any component matching `/overshell/i` on first dispatch, persists the uid, and never re-scans unless an admin clicks "Re-discover". The variable name defaults to `CommandLine` (Datto's "Run Command" component). If Wulf's Overshell uses a different variable, admin sets it once via `/admin/rmm-overshell`. - **Script bodies live in code, not the DB.** Three reasons: PR review is the change-control mechanism; nothing in the database is treated as executable PowerShell; the 7 scripts are already curated and we don't need (or want) ad-hoc paste-a-script UX. - **WNP-only target resolution.** Site-anchored scripts hit the Wulf Nurse Production endpoint (`LLLCCCWNPNN`); PowerShell uses native AD cmdlets to reach the DC over the network. Direct-to-DC role detection is a fast-follow. - **5-minute hard timeout + 50/user/24h rate limit.** Both enforced server-side in the executor. The cost-guard rows in `analyzer_cost_audit` give admins a unified view of LLM and RMM activity per user. - **Output is redacted before persistence.** Same `redact()` from the IT Glue redaction module; strips any password/secret/key/token/credential keyed values from stdout/stderr before the parser sees them. - **Live RMM evidence trims last.** When the audit prompt overflows the 80KB cap, peer_global → ticket_evidence → rmm_evidence (in that order). Live evidence is the most novel signal; it's worth keeping. - **Worker is in-process, not a separate service.** Same auto-start pattern as `analyzerWorker`. `RMM_WORKER_AUTOSTART=1` opt-in for dev. Multiple Next.js workers are safe — each row's `jobUid` is set once and the poll loop is idempotent. - **`getJobResults` response shape is variable across Datto tenants.** The worker handles both top-level and `results[*]` payloads, picks the per-device result when present, and falls back to the first array entry. **Deliberately left out** - **No backfill of existing Overshell jobs.** Per user's call. Pulse starts capturing from the first dispatch. - **No DC-role detection.** WNP-only. Add later if AD scripts that need native DC execution become important. - **No ad-hoc PowerShell paste-in.** Only registry-listed scripts run. - **No openclaw integration.** Phase 4.2 talks directly to Datto RMM. - **No audit-driven auto-execution.** v1 is admin-clicks-button. The audit panel will gain a "Run Get-Services to fill this gap?" prompt in a fast-follow once we trust the safety layers. - **No Overshell write operations.** All scripts are read-only / discovery. Configuration changes happen via IT Glue (Phase 4) or manually. - **No per-script per-user permissions.** Anyone with `rmm.execute` can run any script. Per-script gating is a fast-follow if needed. - **No credential output ever.** Three-layer refusal: 1. Script library has no credential-handling scripts; tests verify bodies don't reference `$plaintext` password patterns. 2. `redact()` strips matching keys from output before persistence. 3. The audit prompt's existing credential refusal applies to anything that does sneak through. ## Status after Phase 4.2 | Phase | Tests | tsc | Notes | |---|---|---|---| | 4.2 | 174 | clean | RMM Overshell pipeline (077), 7 scripts, executor + worker, audit-context arm | ## Phase 4.3 — LogLift event-log ingestion ### Why Overshell stdout caps around ~50KB practical — fine for service lists or installed-software dumps, too small for full Windows event logs across critical/error/warning levels. Wulf already runs a richer collector via n8n: PowerShell on each endpoint gathers logs + system context, gzips it, uploads to a Backblaze B2 bucket (`wulf-audits` / `us-west-002`), then POSTs metadata. n8n decompresses, runs an LLM analysis, and posts a Telegram summary. Phase 4.3 makes Pulse the receiver instead of n8n so: - LogLift evidence lands in the same `rmm_executions` table 4.2 introduced. - The audit pipeline's `rmm_evidence` arm picks it up automatically. - Admins can dispatch a LogLift run directly from the Configuration page. - Successful uploads matched to a unique IT Glue Configuration auto-fire an asset-first audit so documentation suggestions surface immediately. ### Shape `migrations/078_loglift_uploads.sql` adds three columns to `rmm_executions` (`transport`, `evidence_object_key`, `run_id` — with a unique index on `run_id`), and three to `rmm_settings` (`loglift_component_uid`, `loglift_component_name`, `loglift_discovered_at`). The transport column has a CHECK constraint restricting it to `overshell_stdout` | `b2_upload`. `lib/services/b2/client.ts` is a from-scratch SigV4 implementation ported from `docs/LogLift Review.json`: presigned GET + PUT (different expiries), 25MB hard download cap, path-traversal-safe object key regex, and a `B2NotConfiguredError` when env vars are missing. 8 tests cover the regex + signature stability + signing-key derivation. `lib/services/rmm/scripts/loglift-eventlogs.ts` registers the script: `target_type='asset_self'`, `transport='b2_upload'`, empty body (the collector PowerShell lives in the Datto-registered LogLift component, not in Pulse). The registry's body-length sanity test skips `b2_upload` scripts. ### Dispatch path `executor.ts` forks on `script.transport`: - `overshell_stdout` (default) — unchanged 4.2 path: resolve Overshell component, dispatch with `{Variable: body}`, worker polls for stdout. - `b2_upload` — new fork. Resolves the Datto site uid from the device, resolves the LogLift component (discover-on-demand), generates a `runId` (`pulse__`), inserts an `rmm_executions` row with `transport='b2_upload'`, dispatches the Quick Job with variables `RunId`, `ClientId`, `WebhookUrl`, `WebhookSecret`. The persisted `variables` column strips `WebhookSecret` so admins can read the row without exposing the OPENCLAW key. ### Receive path `POST /api/rmm/loglift/upload` (public per `middleware.ts`, `x-openclaw-key` validated): 1. Zod validate body + object-key regex. 2. Resolve `clientId` (Datto site uid) → `datto_rmm_sites.id` → `autotask_company_id` (FK or name fallback — same as 4.2 multi-site work). 3. Resolve `computerName` → Datto device uid (case-insensitive). 4. Resolve `computerName` + company → `itg_configurations.id`. Two-pass (count + fetch) sets `single_match=true` only when exactly one Configuration matches. 5. Correlate to a Pulse-dispatched execution by `run_id`. If no match (out-of-band collector), insert a fresh `running` row. 6. Download from B2 (25MB cap), gunzip with zip-bomb guard (refuse > 100MB inflated, checked via gzip ISIZE before decompression and again after). 7. Slim: keep `system_context` + `summary` + top 100 events sorted by severity (Critical → Error → Warning → Info), then recency. Drop the raw `events` array; the full gzip stays in B2 forever. 8. `redact()` the slim object, persist with `markExecutionFromB2Upload`. 9. Auto-audit hook: if Configuration matched single, fire `runAssetAudit({assetType:'configuration', assetId})` synchronously (still in the webhook handler — the LLM call is the bottleneck but the agent doesn't care about webhook latency past ~30s). On failure, log + continue — webhook still 200s. `audit_log` actions: `rmm.loglift.dispatched`, `rmm.loglift.received`, `rmm.loglift.matched`, `rmm.loglift.audit_triggered`. ### Worker change `worker.ts` filters `transport='b2_upload'` rows out of the running poll list — no stdout to fetch. The 5-minute timeout sweep still applies; stuck rows get marked `timeout`. ### Prompt update `LIVE_EVIDENCE_NOTE` extended to teach the LLM about the LogLift slim shape: cite events as `event:` or `execution:`, `event_count_total` is the original count (top 100 only in the prompt), and `system_context` is authoritative for OS / hardware / disk / memory facts on the matched Configuration. ### UI surfaces - `/admin/rmm-overshell` gets a second "LogLift component" block with a "Re-discover LogLift" button next to the existing Overshell discovery. `discoverLogliftComponent()` matches `/loglift|eventlog/i`. - Configuration page picker (filtered by `target_type='asset_self'`) surfaces the LogLift entry automatically — Phase 4.2's executor + UI scaffolding handles it through the new dispatch fork. ### Files **New:** `migrations/078_loglift_uploads.sql`, `lib/services/b2/client.ts` (+ test), `lib/services/rmm/scripts/loglift-eventlogs.ts`, `lib/services/rmm/loglift-matcher.ts`, `lib/services/rmm/loglift-receiver.ts`, `app/api/rmm/loglift/upload/route.ts`, `app/api/admin/rmm/settings/discover-loglift/route.ts`, `docs/loglift-eventlog-pipeline-spec.md`. **Modified:** `lib/services/datto-rmm-client.ts` (generalized `findOvershellComponent` → `findComponentByName`), `lib/services/rmm/settings.ts` (LogLift discover/resolve), `lib/services/rmm/persistence.ts` (transport + run_id + new `findExecutionByRunId`, `createOutOfBandUploadExecution`, `markExecutionFromB2Upload`), `lib/services/rmm/executor.ts` (b2_upload fork), `lib/services/rmm/worker.ts` (skip b2_upload poll), `lib/services/rmm/scripts/index.ts` (register), `…/scripts/types.ts` (transport field), `…/scripts/registry.test.ts` (8-script expectation + b2_upload body skip), `lib/services/analyzer/asset-audit/prompt.ts` (LIVE_EVIDENCE_NOTE), `app/admin/rmm-overshell/page.tsx` (LogLift block + button), `middleware.ts` (`/api/rmm/loglift` public). ### Refusals + guards 1. Object-key regex (`^[A-Za-z0-9_-]+/[A-Za-z0-9_.-]+/eventlogs_[0-9_]+\.json\.gz$`). 2. B2 25MB download cap. 3. Decompress 100MB cap (gzip ISIZE pre-check + post-inflate re-check). 4. `redact()` on slim payload before persistence. 5. Auto-audit only on single-match Configurations — multiple matches logged + skipped. 6. Webhook secret stripped from persisted `variables` column. ### Status after Phase 4.3 | Phase | Tests | tsc | Notes | |---|---|---|---| | 4.3 | TBD (target ~182) | TBD | LogLift pipeline (078), B2 SigV4, b2_upload transport, slim + auto-audit |