wulf-pulse/docs/wulf-pulse-ticket-analyzer-runbook.md

AI Ticket Analyzer — Operator Runbook

How to keep the analyzer healthy in production. Audience is whoever is on call for Pulse. The feature spec is in wulf-pulse-ticket-analyzer-prompt.md; the implementation log is in wulf-pulse-ticket-analyzer-build-notes.md.

---

At a glance

• Pipeline: Haiku triage → (optional) IT Glue retrieval → Sonnet deep analysis → (optional) Opus deep reasoning → persistence.
• Trigger: user clicks "Analyze" on a ticket page; that POSTs to /api/analyzer/tickets/:ticketNumber/analyze. The route preprocesses inline, short-circuits to the cached row when content hasn't changed, otherwise queues a job for the worker.
• Worker: analyzerWorker polls analyzer_jobs every 2 s in-process. Auto-starts in production (NODE_ENV=production). Opt in for dev with ANALYZER_WORKER_AUTOSTART=1.
• Cost ceiling: hard-coded COST_CEILING_USD = 2.00 in lib/services/analyzer/pipeline.ts. Trips before Opus only — Sonnet runs unconditionally. When tripped, the analysis is flagged needs_human_review = true with reason "cost ceiling reached".

---

Required environment

Add to ~/projects_env/wulf-pulse.env (do not commit values):

ANTHROPIC_API_KEY=
ITGLUE_API_KEY=
ITGLUE_API_BASE=https://api.itglue.com
ALLOWED_SHARE_DOMAINS=wulfconsulting.com
# Optional in dev:
ANALYZER_WORKER_AUTOSTART=1

SMTP_* and BETTER_AUTH_URL are already set elsewhere in Pulse — share emails reuse the existing magic-link transport.

---

First-time setup on an existing database

Migration 069_create_analyzer_tables.sql creates analyzer_analyses, analyzer_shares, and analyzer_jobs. Postgres only re-applies migrations on a fresh data volume — for an existing DB, run the migration directly:

psql "$DATABASE_URL" -f migrations/069_create_analyzer_tables.sql

The script is idempotent (IF NOT EXISTS on every object).

---

Monitoring cost

Every completed analysis stores estimated_cost_usd, total_input_tokens, total_output_tokens, and per-stage tier flags (haiku_used / sonnet_used / opus_used). The number is a forward estimate from lib/services/llm/pricing.ts and will drift if Anthropic changes published rates — re-check that file before believing big spend numbers.

Daily spend (last 14 days)

SELECT date_trunc('day', triggered_at) AS day,
       count(*) FILTER (WHERE status = 'complete')        AS analyses,
       count(*) FILTER (WHERE opus_used)                  AS opus_runs,
       count(*) FILTER (WHERE needs_human_review)         AS needs_review,
       round(sum(estimated_cost_usd)::numeric, 2)         AS spend_usd
  FROM analyzer_analyses
 WHERE triggered_at > now() - interval '14 days'
 GROUP BY 1
 ORDER BY 1 DESC;

Top 20 most expensive analyses (last 30 days)

SELECT id, ticket_number, analysis_version, estimated_cost_usd,
       opus_used, needs_human_review, triggered_at
  FROM analyzer_analyses
 WHERE triggered_at > now() - interval '30 days'
 ORDER BY estimated_cost_usd DESC
 LIMIT 20;

Heaviest tickets (multiple re-runs against the same ticket)

SELECT ticket_number,
       count(*)                              AS analyses,
       round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd,
       max(triggered_at)                     AS last_run
  FROM analyzer_analyses
 GROUP BY 1
HAVING count(*) > 1
 ORDER BY spend_usd DESC
 LIMIT 20;

Cost ceiling — what to do when it trips

needs_human_review = true with human_review_reasons containing "cost ceiling reached" means Sonnet ran but Opus was skipped. The Sonnet output is still persisted and is usually fine. To re-run with Opus forced on, trigger a new analysis (the route accepts force=true); the cost ceiling will reapply, so check the prior estimated_cost_usd before committing. To raise the ceiling, edit COST_CEILING_USD in lib/services/analyzer/pipeline.ts and redeploy.

To reduce spend without code changes: discourage operators from clicking Re-analyze unless content has changed (the idempotency short-circuit already prevents charged re-runs against the same content_hash).

---

Adding IT Glue org aliases

Stage 2 resolves an Autotask company name to an IT Glue organization in two steps:

1. Alias map — lib/services/analyzer/itglue-aliases.json. Keys are normalized company names (lowercase, single-spaced); values are IT Glue organization IDs as strings. Exact-match wins immediately.
2. Live IT Glue search — fallback when the alias map misses.

When the live search misses or returns the wrong org, add an alias.

Finding the IT Glue organization ID

In the IT Glue UI, navigate to the organization. The numeric segment of the URL (/organizations/12345) is the ID. Or via API:

curl -s -H "x-api-key: $ITGLUE_API_KEY" \
  "$ITGLUE_API_BASE/organizations?filter[name]=Seubert%20and%20Associates" \
  | jq '.data[] | {id, name: .attributes.name}'

Updating the alias file

Edit lib/services/analyzer/itglue-aliases.json. Keys must be already normalized — lowercase, single-spaced — because the matcher applies the same normalization to the Autotask company name and looks up an exact key.

{
  "seubert": "1234567",
  "seubert and associates": "1234567",
  "acme insurance group": "7654321"
}

Strip the _comment / _example keys when adding the first real entry, or leave them — they're ignored at match time (no real org name normalizes to a leading underscore).

Changes require a redeploy. The file is bundled into the build by Next's import — there is no runtime reload.

Verifying an alias works

After deploy, run an analysis on a ticket from that company. Check the model_traces.itglue field on the resulting row:

SELECT model_traces->'itglue'
  FROM analyzer_analyses
 WHERE id = '<analysis-id>';

alias_used: true confirms the alias hit. org_id should be the value you put in the JSON.

---

Triaging failed analyses

Two failure surfaces to know about:

1. Failed jobs (analyzer_jobs.status = 'failed')

The job hit an exception before the analysis row was written. The error message is in analyzer_jobs.error_message. Common causes:

Error message	Cause	Fix
Ticket T... not found in local mirror — confirm sync is current.	Autotask sync hasn't pulled the ticket yet.	Wait for the next sync, or trigger a manual sync from /admin. The webhook usually fills the gap; if not, see scripts/backfill-ticket-notes-gap.ts for the reconciliation pattern.
LLM stage on claude-... failed twice	Two Anthropic call attempts both failed (network / parse / 5xx).	Check the API key is valid (curl Anthropic), and check Anthropic status. Re-queue by clicking Re-analyze.
Anthropic API error: 429 ...	Rate limit.	Wait and re-queue.
IT Glue search failed: ... should NOT appear here — the pipeline tolerates IT Glue failures and continues without context.	If you see one, the failure was outside the catch block and is a bug.

-- Recent failures
SELECT id, ticket_number, error_message, queued_at, finished_at
  FROM analyzer_jobs
 WHERE status = 'failed'
   AND queued_at > now() - interval '7 days'
 ORDER BY queued_at DESC;

To retry a failed job, queue a fresh one — there is no in-place retry. Either click Re-analyze in the UI or:

INSERT INTO analyzer_jobs (ticket_number, queued_by_user_id)
SELECT ticket_number, queued_by_user_id
  FROM analyzer_jobs
 WHERE id = '<failed-job-id>';

2. Completed but flagged for human review (needs_human_review = true)

The pipeline finished and produced an analysis, but the model itself reported low confidence or the cost circuit breaker tripped before Opus. human_review_reasons is a JSONB array of short strings — read it.

The UI exposes this queue at /analyzer/queue. SQL equivalent:

SELECT id, ticket_number, analysis_version,
       confidence_score, estimated_cost_usd,
       human_review_reasons
  FROM analyzer_analyses
 WHERE needs_human_review = true
   AND status = 'complete'
 ORDER BY triggered_at DESC
 LIMIT 50;

Reasons you'll see:

• "cost ceiling reached" — Opus skipped to save money. Sonnet output is what you have. Re-run with force=true to override (and accept the spend).
• "low confidence: <score>" — the model itself reports uncertainty. The Sonnet output may still be useful as a starting point; treat as guidance, not verdict.
• A model-emitted reason from Stage 3 or Stage 4. The full string is in the JSONB — don't truncate before reading.

3. Worker not picking up jobs

If analyzer_jobs has rows in status='queued' for more than ~30s, the worker is wedged or not running.

• Check the application log for [ANALYZER-WORKER] starting at boot. If missing in production, NODE_ENV may not be set to production — bring up ANALYZER_WORKER_AUTOSTART=1 as a workaround.
• The worker is in-process; restarting the Pulse app restarts it.
• Concurrency: multiple replicas all run a worker, but claimQueuedJob uses FOR UPDATE SKIP LOCKED so each job is processed exactly once.

---

Manual ops

Trigger an analysis from psql

INSERT INTO analyzer_jobs (ticket_number, queued_by_user_id)
VALUES ('T20260424.0045', NULL);

The worker will pick it up within 2 seconds. queued_by_user_id may be null (ON DELETE SET NULL).

Force a re-run despite a cached analysis

The force=true flag on POST /api/analyzer/tickets/:ticketNumber/analyze bypasses the idempotency short-circuit. The Re-analyze button in the UI sends this flag.

Inspect raw stage outputs for a debugging session

SELECT model_traces
  FROM analyzer_analyses
 WHERE id = '<analysis-id>';

Per-stage payload includes input/output tokens, attempt count, full parsed model responses, and IT Glue resolution metadata. Don't paste this into external systems — it can include redacted-but-still-sensitive snippets of customer data.

Ad hoc: how much have we spent this month?

SELECT round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd,
       count(*)                                   AS analyses
  FROM analyzer_analyses
 WHERE triggered_at >= date_trunc('month', now());

---

Things that are not implemented yet

• No automated retries on failed jobs.
• No viewed_at tracking on share rows (column exists, nothing writes to it).
• No email_sent_at column — share email send state lives only in the HTTP response and the server log.
• No pre-Sonnet cost gate. The circuit breaker only protects against Opus.
• No autocomplete on the share-recipient field.

If any of these become a real operational problem, file the work — the stubs are intentional and called out in wulf-pulse-ticket-analyzer-build-notes.md.