wulf-pulse/docs/wulf-pulse-ticket-analyzer-runbook.md

# AI Ticket Analyzer — Operator Runbook

How to keep the analyzer healthy in production. Audience is whoever is on
call for Pulse. The feature spec is in
`wulf-pulse-ticket-analyzer-prompt.md`; the implementation log is in
`wulf-pulse-ticket-analyzer-build-notes.md`.

---

## At a glance

- **Pipeline**: Haiku triage → (optional) IT Glue retrieval → Sonnet deep
  analysis → (optional) Opus deep reasoning → persistence.
- **Trigger**: user clicks "Analyze" on a ticket page; that POSTs to
  `/api/analyzer/tickets/:ticketNumber/analyze`. The route preprocesses
  inline, short-circuits to the cached row when content hasn't changed,
  otherwise queues a job for the worker.
- **Worker**: `analyzerWorker` polls `analyzer_jobs` every 2 s in-process.
  Auto-starts in production (`NODE_ENV=production`). Opt in for dev with
  `ANALYZER_WORKER_AUTOSTART=1`.
- **Cost ceiling**: hard-coded `COST_CEILING_USD = 2.00` in
  `lib/services/analyzer/pipeline.ts`. Trips before Opus only — Sonnet runs
  unconditionally. When tripped, the analysis is flagged
  `needs_human_review = true` with reason "cost ceiling reached".

---

## Required environment

Add to `~/projects_env/wulf-pulse.env` (do not commit values):

```
ANTHROPIC_API_KEY=
ITGLUE_API_KEY=
ITGLUE_API_BASE=https://api.itglue.com
ALLOWED_SHARE_DOMAINS=wulfconsulting.com
# Optional in dev:
ANALYZER_WORKER_AUTOSTART=1
```

`SMTP_*` and `BETTER_AUTH_URL` are already set elsewhere in Pulse — share
emails reuse the existing magic-link transport.

---

## First-time setup on an existing database

Migration `069_create_analyzer_tables.sql` creates `analyzer_analyses`,
`analyzer_shares`, and `analyzer_jobs`. Postgres only re-applies migrations
on a fresh data volume — for an existing DB, run the migration directly:

```bash
psql "$DATABASE_URL" -f migrations/069_create_analyzer_tables.sql
```

The script is idempotent (`IF NOT EXISTS` on every object).

---

## Monitoring cost

Every completed analysis stores `estimated_cost_usd`, `total_input_tokens`,
`total_output_tokens`, and per-stage tier flags
(`haiku_used` / `sonnet_used` / `opus_used`). The number is a forward
estimate from `lib/services/llm/pricing.ts` and will drift if Anthropic
changes published rates — re-check that file before believing big spend
numbers.

### Daily spend (last 14 days)

```sql
SELECT date_trunc('day', triggered_at) AS day,
       count(*) FILTER (WHERE status = 'complete')        AS analyses,
       count(*) FILTER (WHERE opus_used)                  AS opus_runs,
       count(*) FILTER (WHERE needs_human_review)         AS needs_review,
       round(sum(estimated_cost_usd)::numeric, 2)         AS spend_usd
  FROM analyzer_analyses
 WHERE triggered_at > now() - interval '14 days'
 GROUP BY 1
 ORDER BY 1 DESC;
```

### Top 20 most expensive analyses (last 30 days)

```sql
SELECT id, ticket_number, analysis_version, estimated_cost_usd,
       opus_used, needs_human_review, triggered_at
  FROM analyzer_analyses
 WHERE triggered_at > now() - interval '30 days'
 ORDER BY estimated_cost_usd DESC
 LIMIT 20;
```

### Heaviest tickets (multiple re-runs against the same ticket)

```sql
SELECT ticket_number,
       count(*)                              AS analyses,
       round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd,
       max(triggered_at)                     AS last_run
  FROM analyzer_analyses
 GROUP BY 1
HAVING count(*) > 1
 ORDER BY spend_usd DESC
 LIMIT 20;
```

### Cost ceiling — what to do when it trips

`needs_human_review = true` with `human_review_reasons` containing "cost
ceiling reached" means Sonnet ran but Opus was skipped. The Sonnet output
is still persisted and is usually fine. To re-run with Opus forced on,
trigger a new analysis (the route accepts `force=true`); the cost ceiling
will reapply, so check the prior `estimated_cost_usd` before committing.
To raise the ceiling, edit `COST_CEILING_USD` in
`lib/services/analyzer/pipeline.ts` and redeploy.

To reduce spend without code changes: discourage operators from clicking
Re-analyze unless content has changed (the idempotency short-circuit
already prevents charged re-runs against the same `content_hash`).

---

## Adding IT Glue org aliases

Stage 2 resolves an Autotask company name to an IT Glue organization in
two steps:

1. **Alias map** — `lib/services/analyzer/itglue-aliases.json`. Keys are
   normalized company names (lowercase, single-spaced); values are IT Glue
   organization IDs as **strings**. Exact-match wins immediately.
2. **Live IT Glue search** — fallback when the alias map misses.

When the live search misses or returns the wrong org, add an alias.

### Finding the IT Glue organization ID

In the IT Glue UI, navigate to the organization. The numeric segment of
the URL (`/organizations/12345`) is the ID. Or via API:

```bash
curl -s -H "x-api-key: $ITGLUE_API_KEY" \
  "$ITGLUE_API_BASE/organizations?filter[name]=Seubert%20and%20Associates" \
  | jq '.data[] | {id, name: .attributes.name}'
```

### Updating the alias file

Edit `lib/services/analyzer/itglue-aliases.json`. Keys must be **already
normalized** — lowercase, single-spaced — because the matcher applies the
same normalization to the Autotask company name and looks up an exact key.

```json
{
  "seubert": "1234567",
  "seubert and associates": "1234567",
  "acme insurance group": "7654321"
}
```

Strip the `_comment` / `_example` keys when adding the first real entry,
or leave them — they're ignored at match time (no real org name normalizes
to a leading underscore).

Changes require a redeploy. The file is bundled into the build by Next's
`import` — there is no runtime reload.

### Verifying an alias works

After deploy, run an analysis on a ticket from that company. Check the
`model_traces.itglue` field on the resulting row:

```sql
SELECT model_traces->'itglue'
  FROM analyzer_analyses
 WHERE id = '<analysis-id>';
```

`alias_used: true` confirms the alias hit. `org_id` should be the value
you put in the JSON.

---

## Triaging failed analyses

Two failure surfaces to know about:

### 1. Failed jobs (`analyzer_jobs.status = 'failed'`)

The job hit an exception before the analysis row was written. The error
message is in `analyzer_jobs.error_message`. Common causes:

| Error message | Cause | Fix |
|---|---|---|
| `Ticket T... not found in local mirror — confirm sync is current.` | Autotask sync hasn't pulled the ticket yet. | Wait for the next sync, or trigger a manual sync from `/admin`. The webhook usually fills the gap; if not, see `scripts/backfill-ticket-notes-gap.ts` for the reconciliation pattern. |
| `LLM stage on claude-... failed twice` | Two Anthropic call attempts both failed (network / parse / 5xx). | Check the API key is valid (`curl` Anthropic), and check Anthropic status. Re-queue by clicking Re-analyze. |
| `Anthropic API error: 429 ...` | Rate limit. | Wait and re-queue. |
| `IT Glue search failed: ...` should NOT appear here — the pipeline tolerates IT Glue failures and continues without context. | If you see one, the failure was outside the catch block and is a bug. |

```sql
-- Recent failures
SELECT id, ticket_number, error_message, queued_at, finished_at
  FROM analyzer_jobs
 WHERE status = 'failed'
   AND queued_at > now() - interval '7 days'
 ORDER BY queued_at DESC;
```

To retry a failed job, queue a fresh one — there is no in-place retry.
Either click Re-analyze in the UI or:

```sql
INSERT INTO analyzer_jobs (ticket_number, queued_by_user_id)
SELECT ticket_number, queued_by_user_id
  FROM analyzer_jobs
 WHERE id = '<failed-job-id>';
```

### 2. Completed but flagged for human review (`needs_human_review = true`)

The pipeline finished and produced an analysis, but the model itself
reported low confidence or the cost circuit breaker tripped before Opus.
`human_review_reasons` is a JSONB array of short strings — read it.

The UI exposes this queue at `/analyzer/queue`. SQL equivalent:

```sql
SELECT id, ticket_number, analysis_version,
       confidence_score, estimated_cost_usd,
       human_review_reasons
  FROM analyzer_analyses
 WHERE needs_human_review = true
   AND status = 'complete'
 ORDER BY triggered_at DESC
 LIMIT 50;
```

Reasons you'll see:

- `"cost ceiling reached"` — Opus skipped to save money. Sonnet output is
  what you have. Re-run with `force=true` to override (and accept the
  spend).
- `"low confidence: <score>"` — the model itself reports uncertainty. The
  Sonnet output may still be useful as a starting point; treat as
  guidance, not verdict.
- A model-emitted reason from Stage 3 or Stage 4. The full string is in
  the JSONB — don't truncate before reading.

### 3. Worker not picking up jobs

If `analyzer_jobs` has rows in `status='queued'` for more than ~30s, the
worker is wedged or not running.

- Check the application log for `[ANALYZER-WORKER] starting` at boot. If
  missing in production, `NODE_ENV` may not be set to `production` —
  bring up `ANALYZER_WORKER_AUTOSTART=1` as a workaround.
- The worker is in-process; restarting the Pulse app restarts it.
- Concurrency: multiple replicas all run a worker, but `claimQueuedJob`
  uses `FOR UPDATE SKIP LOCKED` so each job is processed exactly once.

---

## Manual ops

### Trigger an analysis from psql

```sql
INSERT INTO analyzer_jobs (ticket_number, queued_by_user_id)
VALUES ('T20260424.0045', NULL);
```

The worker will pick it up within 2 seconds. `queued_by_user_id` may be
null (`ON DELETE SET NULL`).

### Force a re-run despite a cached analysis

The `force=true` flag on `POST /api/analyzer/tickets/:ticketNumber/analyze`
bypasses the idempotency short-circuit. The Re-analyze button in the UI
sends this flag.

### Inspect raw stage outputs for a debugging session

```sql
SELECT model_traces
  FROM analyzer_analyses
 WHERE id = '<analysis-id>';
```

Per-stage payload includes input/output tokens, attempt count, full parsed
model responses, and IT Glue resolution metadata. Don't paste this into
external systems — it can include redacted-but-still-sensitive snippets
of customer data.

### Ad hoc: how much have we spent this month?

```sql
SELECT round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd,
       count(*)                                   AS analyses
  FROM analyzer_analyses
 WHERE triggered_at >= date_trunc('month', now());
```

---

## Things that are not implemented yet

- No automated retries on failed jobs.
- No `viewed_at` tracking on share rows (column exists, nothing writes
  to it).
- No `email_sent_at` column — share email send state lives only in the
  HTTP response and the server log.
- No pre-Sonnet cost gate. The circuit breaker only protects against
  Opus.
- No autocomplete on the share-recipient field.

If any of these become a real operational problem, file the work — the
stubs are intentional and called out in
`wulf-pulse-ticket-analyzer-build-notes.md`.