wulf-pulse/docs/wulf-pulse-ticket-analyzer-runbook.md

AI Ticket Analyzer — Operator Runbook

How to keep the analyzer healthy in production. Audience is whoever is on call for Pulse. The feature spec is in wulf-pulse-ticket-analyzer-prompt.md; the implementation log is in wulf-pulse-ticket-analyzer-build-notes.md.

---

At a glance

• Pipeline: Haiku triage → (optional) IT Glue retrieval → Sonnet deep analysis → (optional) Opus deep reasoning → persistence.
• Trigger: user clicks "Analyze" on a ticket page; that POSTs to /api/analyzer/tickets/:ticketNumber/analyze. The route preprocesses inline, short-circuits to the cached row when content hasn't changed, otherwise queues a job for the worker.
• Worker: analyzerWorker polls analyzer_jobs every 2 s in-process. Auto-starts in production (NODE_ENV=production). Opt in for dev with ANALYZER_WORKER_AUTOSTART=1.
• Cost ceiling: hard-coded COST_CEILING_USD = 2.00 in lib/services/analyzer/pipeline.ts. Trips before Opus only — Sonnet runs unconditionally. When tripped, the analysis is flagged needs_human_review = true with reason "cost ceiling reached".

---

Required environment

Add to ~/projects_env/wulf-pulse.env (do not commit values):

ANTHROPIC_API_KEY=
ITGLUE_API_KEY=
ITGLUE_API_BASE=https://api.itglue.com
ALLOWED_SHARE_DOMAINS=wulfconsulting.com
# Optional in dev:
ANALYZER_WORKER_AUTOSTART=1

SMTP_* and BETTER_AUTH_URL are already set elsewhere in Pulse — share emails reuse the existing magic-link transport.

---

First-time setup on an existing database

Migration 069_create_analyzer_tables.sql creates analyzer_analyses, analyzer_shares, and analyzer_jobs. Postgres only re-applies migrations on a fresh data volume — for an existing DB, run the migration directly:

psql "$DATABASE_URL" -f migrations/069_create_analyzer_tables.sql

The script is idempotent (IF NOT EXISTS on every object).

---

Monitoring cost

Every completed analysis stores estimated_cost_usd, total_input_tokens, total_output_tokens, and per-stage tier flags (haiku_used / sonnet_used / opus_used). The number is a forward estimate from lib/services/llm/pricing.ts and will drift if Anthropic changes published rates — re-check that file before believing big spend numbers.

Daily spend (last 14 days)

SELECT date_trunc('day', triggered_at) AS day,
       count(*) FILTER (WHERE status = 'complete')        AS analyses,
       count(*) FILTER (WHERE opus_used)                  AS opus_runs,
       count(*) FILTER (WHERE needs_human_review)         AS needs_review,
       round(sum(estimated_cost_usd)::numeric, 2)         AS spend_usd
  FROM analyzer_analyses
 WHERE triggered_at > now() - interval '14 days'
 GROUP BY 1
 ORDER BY 1 DESC;

Top 20 most expensive analyses (last 30 days)

SELECT id, ticket_number, analysis_version, estimated_cost_usd,
       opus_used, needs_human_review, triggered_at
  FROM analyzer_analyses
 WHERE triggered_at > now() - interval '30 days'
 ORDER BY estimated_cost_usd DESC
 LIMIT 20;

Heaviest tickets (multiple re-runs against the same ticket)

SELECT ticket_number,
       count(*)                              AS analyses,
       round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd,
       max(triggered_at)                     AS last_run
  FROM analyzer_analyses
 GROUP BY 1
HAVING count(*) > 1
 ORDER BY spend_usd DESC
 LIMIT 20;

Cost ceiling — what to do when it trips

needs_human_review = true with human_review_reasons containing "cost ceiling reached" means Sonnet ran but Opus was skipped. The Sonnet output is still persisted and is usually fine. To re-run with Opus forced on, trigger a new analysis (the route accepts force=true); the cost ceiling will reapply, so check the prior estimated_cost_usd before committing. To raise the ceiling, edit COST_CEILING_USD in lib/services/analyzer/pipeline.ts and redeploy.

To reduce spend without code changes: discourage operators from clicking Re-analyze unless content has changed (the idempotency short-circuit already prevents charged re-runs against the same content_hash).

---

Adding IT Glue org aliases

Stage 2 resolves an Autotask company name to an IT Glue organization in two steps:

1. Alias map — lib/services/analyzer/itglue-aliases.json. Keys are normalized company names (lowercase, single-spaced); values are IT Glue organization IDs as strings. Exact-match wins immediately.
2. Live IT Glue search — fallback when the alias map misses.

When the live search misses or returns the wrong org, add an alias.

Finding the IT Glue organization ID

In the IT Glue UI, navigate to the organization. The numeric segment of the URL (/organizations/12345) is the ID. Or via API:

curl -s -H "x-api-key: $ITGLUE_API_KEY" \
  "$ITGLUE_API_BASE/organizations?filter[name]=Seubert%20and%20Associates" \
  | jq '.data[] | {id, name: .attributes.name}'

Updating the alias file

Edit lib/services/analyzer/itglue-aliases.json. Keys must be already normalized — lowercase, single-spaced — because the matcher applies the same normalization to the Autotask company name and looks up an exact key.

{
  "seubert": "1234567",
  "seubert and associates": "1234567",
  "acme insurance group": "7654321"
}

Strip the _comment / _example keys when adding the first real entry, or leave them — they're ignored at match time (no real org name normalizes to a leading underscore).

Changes require a redeploy. The file is bundled into the build by Next's import — there is no runtime reload.

Verifying an alias works

After deploy, run an analysis on a ticket from that company. Check the model_traces.itglue field on the resulting row:

SELECT model_traces->'itglue'
  FROM analyzer_analyses
 WHERE id = '<analysis-id>';

alias_used: true confirms the alias hit. org_id should be the value you put in the JSON.

---

Triaging failed analyses

Two failure surfaces to know about:

1. Failed jobs (analyzer_jobs.status = 'failed')

The job hit an exception before the analysis row was written. The error message is in analyzer_jobs.error_message. Common causes:

Error message	Cause	Fix
Ticket T... not found in local mirror — confirm sync is current.	Autotask sync hasn't pulled the ticket yet.	Wait for the next sync, or trigger a manual sync from /admin. The webhook usually fills the gap; if not, see scripts/backfill-ticket-notes-gap.ts for the reconciliation pattern.
LLM stage on claude-... failed twice	Two Anthropic call attempts both failed (network / parse / 5xx).	Check the API key is valid (curl Anthropic), and check Anthropic status. Re-queue by clicking Re-analyze.
Anthropic API error: 429 ...	Rate limit.	Wait and re-queue.
IT Glue search failed: ... should NOT appear here — the pipeline tolerates IT Glue failures and continues without context.	If you see one, the failure was outside the catch block and is a bug.

-- Recent failures
SELECT id, ticket_number, error_message, queued_at, finished_at
  FROM analyzer_jobs
 WHERE status = 'failed'
   AND queued_at > now() - interval '7 days'
 ORDER BY queued_at DESC;

To retry a failed job, queue a fresh one — there is no in-place retry. Either click Re-analyze in the UI or:

INSERT INTO analyzer_jobs (ticket_number, queued_by_user_id)
SELECT ticket_number, queued_by_user_id
  FROM analyzer_jobs
 WHERE id = '<failed-job-id>';

2. Completed but flagged for human review (needs_human_review = true)

The pipeline finished and produced an analysis, but the model itself reported low confidence or the cost circuit breaker tripped before Opus. human_review_reasons is a JSONB array of short strings — read it.

The UI exposes this queue at /analyzer/queue. SQL equivalent:

SELECT id, ticket_number, analysis_version,
       confidence_score, estimated_cost_usd,
       human_review_reasons
  FROM analyzer_analyses
 WHERE needs_human_review = true
   AND status = 'complete'
 ORDER BY triggered_at DESC
 LIMIT 50;

Reasons you'll see:

• "cost ceiling reached" — Opus skipped to save money. Sonnet output is what you have. Re-run with force=true to override (and accept the spend).
• "low confidence: <score>" — the model itself reports uncertainty. The Sonnet output may still be useful as a starting point; treat as guidance, not verdict.
• A model-emitted reason from Stage 3 or Stage 4. The full string is in the JSONB — don't truncate before reading.

3. Worker not picking up jobs

If analyzer_jobs has rows in status='queued' for more than ~30s, the worker is wedged or not running.

• Check the application log for [ANALYZER-WORKER] starting at boot. If missing in production, NODE_ENV may not be set to production — bring up ANALYZER_WORKER_AUTOSTART=1 as a workaround.
• The worker is in-process; restarting the Pulse app restarts it.
• Concurrency: multiple replicas all run a worker, but claimQueuedJob uses FOR UPDATE SKIP LOCKED so each job is processed exactly once.

---

Manual ops

Trigger an analysis from psql

INSERT INTO analyzer_jobs (ticket_number, queued_by_user_id)
VALUES ('T20260424.0045', NULL);

The worker will pick it up within 2 seconds. queued_by_user_id may be null (ON DELETE SET NULL).

Force a re-run despite a cached analysis

The force=true flag on POST /api/analyzer/tickets/:ticketNumber/analyze bypasses the idempotency short-circuit. The Re-analyze button in the UI sends this flag.

Inspect raw stage outputs for a debugging session

SELECT model_traces
  FROM analyzer_analyses
 WHERE id = '<analysis-id>';

Per-stage payload includes input/output tokens, attempt count, full parsed model responses, and IT Glue resolution metadata. Don't paste this into external systems — it can include redacted-but-still-sensitive snippets of customer data.

Ad hoc: how much have we spent this month?

SELECT round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd,
       count(*)                                   AS analyses
  FROM analyzer_analyses
 WHERE triggered_at >= date_trunc('month', now());

---

Things that are not implemented yet

• No automated retries on failed jobs.
• No viewed_at tracking on share rows (column exists, nothing writes to it).
• No email_sent_at column — share email send state lives only in the HTTP response and the server log.
• No pre-Sonnet cost gate. The circuit breaker only protects against Opus.
• No autocomplete on the share-recipient field.

If any of these become a real operational problem, file the work — the stubs are intentional and called out in wulf-pulse-ticket-analyzer-build-notes.md.

---

Phase 2 additions

Stage execution history (per-analysis forensics)

Migration 070 introduced analyzer_stage_executions. Every stage of the pipeline (preprocess, triage, itglue, analyze, deep_review, fingerprint) now writes a row including the input it saw and the output it produced. Failed stages get a row with error_message populated.

Read it like this:

-- Full per-stage trace for one analysis
SELECT stage_order, stage, model_id,
       input_tokens, output_tokens, latency_ms,
       error_message
  FROM analyzer_stage_executions
 WHERE analysis_id = '<analysis-id>'
 ORDER BY stage_order;

-- Inspect a single stage's full input/output
SELECT input_payload, output_payload
  FROM analyzer_stage_executions
 WHERE analysis_id = '<analysis-id>' AND stage = 'analyze';

The legacy analyzer_analyses.model_traces JSONB column is preserved for back-compat. New analyses double-write to both. A future migration will drop model_traces once aggregate reports have soaked.

Fingerprint stage (Stage 6)

After persistence, the worker runs Haiku-tier fingerprint extraction and writes the result to analyzer_analyses.aggregate_fingerprint. Failure is non-fatal — the analysis row stays usable, fingerprint stays NULL.

Re-fingerprint on demand:

SELECT id, ticket_number, analysis_version
  FROM analyzer_analyses
 WHERE aggregate_fingerprint IS NULL
   AND status = 'complete'
 ORDER BY triggered_at ASC;

Or run the backfill script (idempotent — the SQL filter skips already-fingerprinted rows):

npx tsx scripts/backfill-fingerprints.ts                # process all missing
npx tsx scripts/backfill-fingerprints.ts --limit=50     # cap work
npx tsx scripts/backfill-fingerprints.ts --dry-run      # show what would run

The script reconstructs the Stage 6 input from analyzer_analyses.model_traces.{triage_response, sonnet_response, opus_response}. If a row's model_traces is missing those fields (very old format) the script logs [skip] for it and moves on.

Cost: Haiku at ~$0.001 per fingerprint. 1000 analyses ≈ $1.

Aggregate reports

Migration 071 added analyzer_aggregate_reports and relaxed analyzer_stage_executions.analysis_id to be nullable; rows now carry either an analysis_id or an aggregate_report_id (CHECK enforces exactly one).

Workflow:

1. User selects tickets on /analyzer/tickets and clicks Generate aggregate report.
2. POST /api/analyzer/aggregate-reports validates: ≤100 tickets, all have a fingerprint, none are stale. Inserts a 'pending' row, fires the runner via void runAggregateReport(id), returns immediately.
3. Runner does:
   • Loads fingerprints
   • Computes SQL distributions (categories, clients, root cause, resolution path) and writes them to the row immediately
   • Fetches IT Glue doc titles for affected clients (if requested)
   • Calls Sonnet (Opus opt-in) with a structured payload
   • Writes the LLM-derived fields, marks status='complete'
4. UI polls GET /api/analyzer/aggregate-reports/:id every 3s while pending/running.

Inspect a report's sub-stage trace (linked via aggregate_report_id):

SELECT stage_order, stage, model_id, latency_ms, error_message
  FROM analyzer_stage_executions
 WHERE aggregate_report_id = '<report-id>'
 ORDER BY stage_order;

Daily spend on aggregate reports:

SELECT date_trunc('day', generated_at) AS day,
       count(*) AS reports,
       round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd
  FROM analyzer_aggregate_reports
 WHERE generated_at > now() - interval '14 days'
 GROUP BY 1
 ORDER BY 1 DESC;

Cost guards (Phase 2.7)

Three thresholds enforce per-user daily spend:

• $5 per request — a single aggregate report estimated above $5 triggers a confirmation modal. Frontend re-POSTs with confirmedCost: true if the user clicks through.
• $20 per user/day — soft warn. Surfaced via softWarn:true in the cost evaluation; no enforcement.
• $50 per user/day — hard block. POST returns 403 unless the user is in ANALYZER_DAILY_COST_OVERRIDE_USERS (comma-separated list of Better Auth user ids).

Daily spend is computed as the trailing 24h sum of analyzer_analyses.estimated_cost_usd + analyzer_aggregate_reports.estimated_cost_usd for the user.

Every gating decision writes a row to analyzer_cost_audit:

SELECT created_at, user_id, action, estimated_cost,
       daily_spend_before, decision, decision_reason
  FROM analyzer_cost_audit
 ORDER BY created_at DESC
 LIMIT 50;

To grant a user override capability:

# in ~/projects_env/wulf-pulse.env
ANALYZER_DAILY_COST_OVERRIDE_USERS=user_id_1,user_id_2

Restart pulse-app to pick up the change.

Browse / filter ticket list

/analyzer/tickets rebuilt as the primary entry point in Phase 2.5:

• Multi-select for client, issue type, queue, status, priority, assignee
• Sticky filter bar with period chips (today/yesterday/this+last week, 30d/60d, custom range, all time)
• Active-filter chips below the bar; click to remove
• Bulk row selection persisted in localStorage (analyzer:ticket-selection:v1) — survives pagination but is session-scoped (no user id baked in)
• Bulk "Analyze N selected" sequentially queues jobs for each ticket (forces re-analyze on stale; analyzes from scratch on un-analyzed)
• Bulk "Generate aggregate report" routes selected tickets to /analyzer/reports/new; only enabled when all selected are analyzedState === 'current'

Staleness heuristic: a ticket's analyzedState is computed as stale when tickets.last_activity_date > latest_analysis.completed_at. The Phase 2 spec calls for content-hash-based comparison; that requires either caching the current hash on the tickets row (sync change) or computing it on read for the visible page (slow). The date heuristic gets ~95% of the value at zero compute cost; revisit when there's real load signal.