wulf-pulse/docs/wulf-pulse-ticket-analyzer-runbook.md

# AI Ticket Analyzer — Operator Runbook

How to keep the analyzer healthy in production. Audience is whoever is on
call for Pulse. The feature spec is in
`wulf-pulse-ticket-analyzer-prompt.md`; the implementation log is in
`wulf-pulse-ticket-analyzer-build-notes.md`.

---

## At a glance

- **Pipeline**: Haiku triage → (optional) IT Glue retrieval → Sonnet deep
  analysis → (optional) Opus deep reasoning → persistence.
- **Trigger**: user clicks "Analyze" on a ticket page; that POSTs to
  `/api/analyzer/tickets/:ticketNumber/analyze`. The route preprocesses
  inline, short-circuits to the cached row when content hasn't changed,
  otherwise queues a job for the worker.
- **Worker**: `analyzerWorker` polls `analyzer_jobs` every 2 s in-process.
  Auto-starts in production (`NODE_ENV=production`). Opt in for dev with
  `ANALYZER_WORKER_AUTOSTART=1`.
- **Cost ceiling**: hard-coded `COST_CEILING_USD = 2.00` in
  `lib/services/analyzer/pipeline.ts`. Trips before Opus only — Sonnet runs
  unconditionally. When tripped, the analysis is flagged
  `needs_human_review = true` with reason "cost ceiling reached".

---

## Required environment

Add to `~/projects_env/wulf-pulse.env` (do not commit values):

```
ANTHROPIC_API_KEY=
ITGLUE_API_KEY=
ITGLUE_API_BASE=https://api.itglue.com
ALLOWED_SHARE_DOMAINS=wulfconsulting.com
# Optional in dev:
ANALYZER_WORKER_AUTOSTART=1
```

`SMTP_*` and `BETTER_AUTH_URL` are already set elsewhere in Pulse — share
emails reuse the existing magic-link transport.

---

## First-time setup on an existing database

Migration `069_create_analyzer_tables.sql` creates `analyzer_analyses`,
`analyzer_shares`, and `analyzer_jobs`. Postgres only re-applies migrations
on a fresh data volume — for an existing DB, run the migration directly:

```bash
psql "$DATABASE_URL" -f migrations/069_create_analyzer_tables.sql
```

The script is idempotent (`IF NOT EXISTS` on every object).

---

## Monitoring cost

Every completed analysis stores `estimated_cost_usd`, `total_input_tokens`,
`total_output_tokens`, and per-stage tier flags
(`haiku_used` / `sonnet_used` / `opus_used`). The number is a forward
estimate from `lib/services/llm/pricing.ts` and will drift if Anthropic
changes published rates — re-check that file before believing big spend
numbers.

### Daily spend (last 14 days)

```sql
SELECT date_trunc('day', triggered_at) AS day,
       count(*) FILTER (WHERE status = 'complete')        AS analyses,
       count(*) FILTER (WHERE opus_used)                  AS opus_runs,
       count(*) FILTER (WHERE needs_human_review)         AS needs_review,
       round(sum(estimated_cost_usd)::numeric, 2)         AS spend_usd
  FROM analyzer_analyses
 WHERE triggered_at > now() - interval '14 days'
 GROUP BY 1
 ORDER BY 1 DESC;
```

### Top 20 most expensive analyses (last 30 days)

```sql
SELECT id, ticket_number, analysis_version, estimated_cost_usd,
       opus_used, needs_human_review, triggered_at
  FROM analyzer_analyses
 WHERE triggered_at > now() - interval '30 days'
 ORDER BY estimated_cost_usd DESC
 LIMIT 20;
```

### Heaviest tickets (multiple re-runs against the same ticket)

```sql
SELECT ticket_number,
       count(*)                              AS analyses,
       round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd,
       max(triggered_at)                     AS last_run
  FROM analyzer_analyses
 GROUP BY 1
HAVING count(*) > 1
 ORDER BY spend_usd DESC
 LIMIT 20;
```

### Cost ceiling — what to do when it trips

`needs_human_review = true` with `human_review_reasons` containing "cost
ceiling reached" means Sonnet ran but Opus was skipped. The Sonnet output
is still persisted and is usually fine. To re-run with Opus forced on,
trigger a new analysis (the route accepts `force=true`); the cost ceiling
will reapply, so check the prior `estimated_cost_usd` before committing.
To raise the ceiling, edit `COST_CEILING_USD` in
`lib/services/analyzer/pipeline.ts` and redeploy.

To reduce spend without code changes: discourage operators from clicking
Re-analyze unless content has changed (the idempotency short-circuit
already prevents charged re-runs against the same `content_hash`).

---

## Adding IT Glue org aliases

Stage 2 resolves an Autotask company name to an IT Glue organization in
two steps:

1. **Alias map** — `lib/services/analyzer/itglue-aliases.json`. Keys are
   normalized company names (lowercase, single-spaced); values are IT Glue
   organization IDs as **strings**. Exact-match wins immediately.
2. **Live IT Glue search** — fallback when the alias map misses.

When the live search misses or returns the wrong org, add an alias.

### Finding the IT Glue organization ID

In the IT Glue UI, navigate to the organization. The numeric segment of
the URL (`/organizations/12345`) is the ID. Or via API:

```bash
curl -s -H "x-api-key: $ITGLUE_API_KEY" \
  "$ITGLUE_API_BASE/organizations?filter[name]=Seubert%20and%20Associates" \
  | jq '.data[] | {id, name: .attributes.name}'
```

### Updating the alias file

Edit `lib/services/analyzer/itglue-aliases.json`. Keys must be **already
normalized** — lowercase, single-spaced — because the matcher applies the
same normalization to the Autotask company name and looks up an exact key.

```json
{
  "seubert": "1234567",
  "seubert and associates": "1234567",
  "acme insurance group": "7654321"
}
```

Strip the `_comment` / `_example` keys when adding the first real entry,
or leave them — they're ignored at match time (no real org name normalizes
to a leading underscore).

Changes require a redeploy. The file is bundled into the build by Next's
`import` — there is no runtime reload.

### Verifying an alias works

After deploy, run an analysis on a ticket from that company. Check the
`model_traces.itglue` field on the resulting row:

```sql
SELECT model_traces->'itglue'
  FROM analyzer_analyses
 WHERE id = '<analysis-id>';
```

`alias_used: true` confirms the alias hit. `org_id` should be the value
you put in the JSON.

---

## Triaging failed analyses

Two failure surfaces to know about:

### 1. Failed jobs (`analyzer_jobs.status = 'failed'`)

The job hit an exception before the analysis row was written. The error
message is in `analyzer_jobs.error_message`. Common causes:

| Error message | Cause | Fix |
|---|---|---|
| `Ticket T... not found in local mirror — confirm sync is current.` | Autotask sync hasn't pulled the ticket yet. | Wait for the next sync, or trigger a manual sync from `/admin`. The webhook usually fills the gap; if not, see `scripts/backfill-ticket-notes-gap.ts` for the reconciliation pattern. |
| `LLM stage on claude-... failed twice` | Two Anthropic call attempts both failed (network / parse / 5xx). | Check the API key is valid (`curl` Anthropic), and check Anthropic status. Re-queue by clicking Re-analyze. |
| `Anthropic API error: 429 ...` | Rate limit. | Wait and re-queue. |
| `IT Glue search failed: ...` should NOT appear here — the pipeline tolerates IT Glue failures and continues without context. | If you see one, the failure was outside the catch block and is a bug. |

```sql
-- Recent failures
SELECT id, ticket_number, error_message, queued_at, finished_at
  FROM analyzer_jobs
 WHERE status = 'failed'
   AND queued_at > now() - interval '7 days'
 ORDER BY queued_at DESC;
```

To retry a failed job, queue a fresh one — there is no in-place retry.
Either click Re-analyze in the UI or:

```sql
INSERT INTO analyzer_jobs (ticket_number, queued_by_user_id)
SELECT ticket_number, queued_by_user_id
  FROM analyzer_jobs
 WHERE id = '<failed-job-id>';
```

### 2. Completed but flagged for human review (`needs_human_review = true`)

The pipeline finished and produced an analysis, but the model itself
reported low confidence or the cost circuit breaker tripped before Opus.
`human_review_reasons` is a JSONB array of short strings — read it.

The UI exposes this queue at `/analyzer/queue`. SQL equivalent:

```sql
SELECT id, ticket_number, analysis_version,
       confidence_score, estimated_cost_usd,
       human_review_reasons
  FROM analyzer_analyses
 WHERE needs_human_review = true
   AND status = 'complete'
 ORDER BY triggered_at DESC
 LIMIT 50;
```

Reasons you'll see:

- `"cost ceiling reached"` — Opus skipped to save money. Sonnet output is
  what you have. Re-run with `force=true` to override (and accept the
  spend).
- `"low confidence: <score>"` — the model itself reports uncertainty. The
  Sonnet output may still be useful as a starting point; treat as
  guidance, not verdict.
- A model-emitted reason from Stage 3 or Stage 4. The full string is in
  the JSONB — don't truncate before reading.

### 3. Worker not picking up jobs

If `analyzer_jobs` has rows in `status='queued'` for more than ~30s, the
worker is wedged or not running.

- Check the application log for `[ANALYZER-WORKER] starting` at boot. If
  missing in production, `NODE_ENV` may not be set to `production` —
  bring up `ANALYZER_WORKER_AUTOSTART=1` as a workaround.
- The worker is in-process; restarting the Pulse app restarts it.
- Concurrency: multiple replicas all run a worker, but `claimQueuedJob`
  uses `FOR UPDATE SKIP LOCKED` so each job is processed exactly once.

---

## Manual ops

### Trigger an analysis from psql

```sql
INSERT INTO analyzer_jobs (ticket_number, queued_by_user_id)
VALUES ('T20260424.0045', NULL);
```

The worker will pick it up within 2 seconds. `queued_by_user_id` may be
null (`ON DELETE SET NULL`).

### Force a re-run despite a cached analysis

The `force=true` flag on `POST /api/analyzer/tickets/:ticketNumber/analyze`
bypasses the idempotency short-circuit. The Re-analyze button in the UI
sends this flag.

### Inspect raw stage outputs for a debugging session

```sql
SELECT model_traces
  FROM analyzer_analyses
 WHERE id = '<analysis-id>';
```

Per-stage payload includes input/output tokens, attempt count, full parsed
model responses, and IT Glue resolution metadata. Don't paste this into
external systems — it can include redacted-but-still-sensitive snippets
of customer data.

### Ad hoc: how much have we spent this month?

```sql
SELECT round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd,
       count(*)                                   AS analyses
  FROM analyzer_analyses
 WHERE triggered_at >= date_trunc('month', now());
```

---

## Things that are not implemented yet

- No automated retries on failed jobs.
- No `viewed_at` tracking on share rows (column exists, nothing writes
  to it).
- No `email_sent_at` column — share email send state lives only in the
  HTTP response and the server log.
- No pre-Sonnet cost gate. The circuit breaker only protects against
  Opus.
- No autocomplete on the share-recipient field.

If any of these become a real operational problem, file the work — the
stubs are intentional and called out in
`wulf-pulse-ticket-analyzer-build-notes.md`.

---

## Phase 2 additions

### Stage execution history (per-analysis forensics)

Migration 070 introduced `analyzer_stage_executions`. Every stage of the
pipeline (preprocess, triage, itglue, analyze, deep_review, fingerprint)
now writes a row including the input it saw and the output it produced.
Failed stages get a row with `error_message` populated.

Read it like this:

```sql
-- Full per-stage trace for one analysis
SELECT stage_order, stage, model_id,
       input_tokens, output_tokens, latency_ms,
       error_message
  FROM analyzer_stage_executions
 WHERE analysis_id = '<analysis-id>'
 ORDER BY stage_order;

-- Inspect a single stage's full input/output
SELECT input_payload, output_payload
  FROM analyzer_stage_executions
 WHERE analysis_id = '<analysis-id>' AND stage = 'analyze';
```

The legacy `analyzer_analyses.model_traces` JSONB column is preserved for
back-compat. New analyses double-write to both. A future migration will
drop `model_traces` once aggregate reports have soaked.

### Fingerprint stage (Stage 6)

After persistence, the worker runs Haiku-tier fingerprint extraction and
writes the result to `analyzer_analyses.aggregate_fingerprint`. Failure
is non-fatal — the analysis row stays usable, fingerprint stays NULL.

Re-fingerprint on demand:

```sql
SELECT id, ticket_number, analysis_version
  FROM analyzer_analyses
 WHERE aggregate_fingerprint IS NULL
   AND status = 'complete'
 ORDER BY triggered_at ASC;
```

Or run the backfill script (idempotent — the SQL filter skips already-fingerprinted rows):

```bash
npx tsx scripts/backfill-fingerprints.ts                # process all missing
npx tsx scripts/backfill-fingerprints.ts --limit=50     # cap work
npx tsx scripts/backfill-fingerprints.ts --dry-run      # show what would run
```

The script reconstructs the Stage 6 input from
`analyzer_analyses.model_traces.{triage_response, sonnet_response, opus_response}`.
If a row's model_traces is missing those fields (very old format) the script
logs `[skip]` for it and moves on.

Cost: Haiku at ~$0.001 per fingerprint. 1000 analyses ≈ $1.

### Aggregate reports

Migration 071 added `analyzer_aggregate_reports` and relaxed
`analyzer_stage_executions.analysis_id` to be nullable; rows now carry
either an `analysis_id` or an `aggregate_report_id` (CHECK enforces
exactly one).

Workflow:

1. User selects tickets on `/analyzer/tickets` and clicks
   **Generate aggregate report**.
2. POST `/api/analyzer/aggregate-reports` validates: ≤100 tickets, all
   have a fingerprint, none are stale. Inserts a 'pending' row, fires
   the runner via `void runAggregateReport(id)`, returns immediately.
3. Runner does:
   - Loads fingerprints
   - Computes SQL distributions (categories, clients, root cause,
     resolution path) and writes them to the row immediately
   - Fetches IT Glue doc titles for affected clients (if requested)
   - Calls Sonnet (Opus opt-in) with a structured payload
   - Writes the LLM-derived fields, marks `status='complete'`
4. UI polls `GET /api/analyzer/aggregate-reports/:id` every 3s while
   pending/running.

Inspect a report's sub-stage trace (linked via `aggregate_report_id`):

```sql
SELECT stage_order, stage, model_id, latency_ms, error_message
  FROM analyzer_stage_executions
 WHERE aggregate_report_id = '<report-id>'
 ORDER BY stage_order;
```

Daily spend on aggregate reports:

```sql
SELECT date_trunc('day', generated_at) AS day,
       count(*) AS reports,
       round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd
  FROM analyzer_aggregate_reports
 WHERE generated_at > now() - interval '14 days'
 GROUP BY 1
 ORDER BY 1 DESC;
```

### Cost guards (Phase 2.7)

Three thresholds enforce per-user daily spend:

- **$5 per request** — a single aggregate report estimated above $5
  triggers a confirmation modal. Frontend re-POSTs with
  `confirmedCost: true` if the user clicks through.
- **$20 per user/day** — soft warn. Surfaced via `softWarn:true` in the
  cost evaluation; no enforcement.
- **$50 per user/day** — hard block. POST returns 403 unless the user
  is in `ANALYZER_DAILY_COST_OVERRIDE_USERS` (comma-separated list of
  Better Auth user ids).

Daily spend is computed as the trailing 24h sum of
`analyzer_analyses.estimated_cost_usd` + `analyzer_aggregate_reports.estimated_cost_usd`
for the user.

Every gating decision writes a row to `analyzer_cost_audit`:

```sql
SELECT created_at, user_id, action, estimated_cost,
       daily_spend_before, decision, decision_reason
  FROM analyzer_cost_audit
 ORDER BY created_at DESC
 LIMIT 50;
```

To grant a user override capability:

```bash
# in ~/projects_env/wulf-pulse.env
ANALYZER_DAILY_COST_OVERRIDE_USERS=user_id_1,user_id_2
```

Restart `pulse-app` to pick up the change.

### Browse / filter ticket list

`/analyzer/tickets` rebuilt as the primary entry point in Phase 2.5:

- Multi-select for client, issue type, queue, status, priority, assignee
- Sticky filter bar with period chips (today/yesterday/this+last week,
  30d/60d, custom range, all time)
- Active-filter chips below the bar; click to remove
- Bulk row selection persisted in localStorage
  (`analyzer:ticket-selection:v1`) — survives pagination but is
  session-scoped (no user id baked in)
- Bulk "Analyze N selected" sequentially queues jobs for each ticket
  (forces re-analyze on stale; analyzes from scratch on un-analyzed)
- Bulk "Generate aggregate report" routes selected tickets to
  `/analyzer/reports/new`; only enabled when all selected are
  `analyzedState === 'current'`

**Staleness heuristic**: a ticket's `analyzedState` is computed as
`stale` when `tickets.last_activity_date > latest_analysis.completed_at`.
The Phase 2 spec calls for content-hash-based comparison; that requires
either caching the current hash on the tickets row (sync change) or
computing it on read for the visible page (slow). The date heuristic
gets ~95% of the value at zero compute cost; revisit when there's real
load signal.