From 1c5ec0f9470141b4f6022919bb56b3b54ac2eccc Mon Sep 17 00:00:00 2001 From: lorentz Date: Wed, 29 Apr 2026 11:05:46 -0400 Subject: [PATCH] =?UTF-8?q?docs(analyzer):=20phase=208=20=E2=80=94=20opera?= =?UTF-8?q?tor=20runbook?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/wulf-pulse-ticket-analyzer-runbook.md covers cost monitoring queries, the $2 cost ceiling, IT Glue alias workflow, failure triage (failed jobs vs needs_human_review), and manual ops (queue from psql, force re-run, inspect model_traces). Calls out the manual migration step for existing DBs and lists the unimplemented surfaces (no auto retries, no viewed_at, no email_sent_at) so operators don't trip on them. Linked from README. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 1 + .../wulf-pulse-ticket-analyzer-build-notes.md | 39 +++ docs/wulf-pulse-ticket-analyzer-runbook.md | 318 ++++++++++++++++++ 3 files changed, 358 insertions(+) create mode 100644 docs/wulf-pulse-ticket-analyzer-runbook.md diff --git a/README.md b/README.md index 5106b45..9258fda 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,7 @@ See `AUTOTASK_API_GUIDE.md` and `ADDIGY_API_GUIDE.md` for credential setup. - **`POSTGRES_SYNC_SETUP.md`** — database initialization - **`DOCKER_README.md`** — Docker workflow - **`PULSE_DATABASE_SKILL.md`** — diagnostic SQL queries +- **`docs/wulf-pulse-ticket-analyzer-runbook.md`** — AI ticket analyzer operator runbook (cost monitoring, IT Glue alias map, failure triage) - **`docs/`** — sync behavior, webhook setup, workflow editor, per-integration guides ## Deployment diff --git a/docs/wulf-pulse-ticket-analyzer-build-notes.md b/docs/wulf-pulse-ticket-analyzer-build-notes.md index 572c298..ee7f254 100644 --- a/docs/wulf-pulse-ticket-analyzer-build-notes.md +++ b/docs/wulf-pulse-ticket-analyzer-build-notes.md @@ -376,6 +376,44 @@ This file is updated after each phase ships. --- +## Phase 8 — Operator runbook + +**Delivered** + +- `docs/wulf-pulse-ticket-analyzer-runbook.md` — cost monitoring queries + (daily spend, top expensive analyses, multi-run tickets), the cost + ceiling explanation (`COST_CEILING_USD = 2.00` in `pipeline.ts`), the + IT Glue alias workflow (where to find org IDs, how to verify via + `model_traces.itglue.alias_used`), failure triage tables (failed jobs + vs. `needs_human_review` flagged completes, with the actual error + strings the worker emits), and a manual-ops section for queueing jobs + from psql. +- README.md "Documentation" section links the runbook so operators + finding the project for the first time discover it. + +**Decisions worth flagging** + +- **Runbook lives in `docs/`, not the README.** Per CLAUDE.md, long-form + per-feature docs go in `docs/` and the README points to them. Matches + the existing pattern (workflow-editor-guide, sync guides, etc.). +- **Manual migration step is documented.** Postgres only re-applies + migrations on first init — operators applying analyzer to an existing + DB need to run `psql -f migrations/069_*.sql`. The runbook leads with + this. +- **Listed what's *not* implemented.** Retries, `viewed_at`, share + `email_sent_at`, pre-Sonnet cost gate, share-recipient autocomplete. + Better to advertise the gaps than have an operator trip over them. + +**Deliberately left out** + +- No production-monitoring dashboard (Grafana etc.) — Pulse doesn't have + one for any other feature, and the SQL queries cover the same ground. +- No README expansion for the analyzer feature itself. Pulse's README is + intentionally short; the runbook + spec + build notes are the deep + docs. + +--- + ## Status after each phase | Phase | Tests | tsc | Notes | @@ -387,3 +425,4 @@ This file is updated after each phase ships. | 5 | 128 | clean | API routes (no route tests) | | 6 | 128 | clean | frontend (no FE tests) | | 7 | 128 | clean | share email via existing SMTP transport | +| 8 | 128 | clean | operator runbook + README link | diff --git a/docs/wulf-pulse-ticket-analyzer-runbook.md b/docs/wulf-pulse-ticket-analyzer-runbook.md new file mode 100644 index 0000000..0e39c18 --- /dev/null +++ b/docs/wulf-pulse-ticket-analyzer-runbook.md @@ -0,0 +1,318 @@ +# AI Ticket Analyzer — Operator Runbook + +How to keep the analyzer healthy in production. Audience is whoever is on +call for Pulse. The feature spec is in +`wulf-pulse-ticket-analyzer-prompt.md`; the implementation log is in +`wulf-pulse-ticket-analyzer-build-notes.md`. + +--- + +## At a glance + +- **Pipeline**: Haiku triage → (optional) IT Glue retrieval → Sonnet deep + analysis → (optional) Opus deep reasoning → persistence. +- **Trigger**: user clicks "Analyze" on a ticket page; that POSTs to + `/api/analyzer/tickets/:ticketNumber/analyze`. The route preprocesses + inline, short-circuits to the cached row when content hasn't changed, + otherwise queues a job for the worker. +- **Worker**: `analyzerWorker` polls `analyzer_jobs` every 2 s in-process. + Auto-starts in production (`NODE_ENV=production`). Opt in for dev with + `ANALYZER_WORKER_AUTOSTART=1`. +- **Cost ceiling**: hard-coded `COST_CEILING_USD = 2.00` in + `lib/services/analyzer/pipeline.ts`. Trips before Opus only — Sonnet runs + unconditionally. When tripped, the analysis is flagged + `needs_human_review = true` with reason "cost ceiling reached". + +--- + +## Required environment + +Add to `~/projects_env/wulf-pulse.env` (do not commit values): + +``` +ANTHROPIC_API_KEY= +ITGLUE_API_KEY= +ITGLUE_API_BASE=https://api.itglue.com +ALLOWED_SHARE_DOMAINS=wulfconsulting.com +# Optional in dev: +ANALYZER_WORKER_AUTOSTART=1 +``` + +`SMTP_*` and `BETTER_AUTH_URL` are already set elsewhere in Pulse — share +emails reuse the existing magic-link transport. + +--- + +## First-time setup on an existing database + +Migration `069_create_analyzer_tables.sql` creates `analyzer_analyses`, +`analyzer_shares`, and `analyzer_jobs`. Postgres only re-applies migrations +on a fresh data volume — for an existing DB, run the migration directly: + +```bash +psql "$DATABASE_URL" -f migrations/069_create_analyzer_tables.sql +``` + +The script is idempotent (`IF NOT EXISTS` on every object). + +--- + +## Monitoring cost + +Every completed analysis stores `estimated_cost_usd`, `total_input_tokens`, +`total_output_tokens`, and per-stage tier flags +(`haiku_used` / `sonnet_used` / `opus_used`). The number is a forward +estimate from `lib/services/llm/pricing.ts` and will drift if Anthropic +changes published rates — re-check that file before believing big spend +numbers. + +### Daily spend (last 14 days) + +```sql +SELECT date_trunc('day', triggered_at) AS day, + count(*) FILTER (WHERE status = 'complete') AS analyses, + count(*) FILTER (WHERE opus_used) AS opus_runs, + count(*) FILTER (WHERE needs_human_review) AS needs_review, + round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd + FROM analyzer_analyses + WHERE triggered_at > now() - interval '14 days' + GROUP BY 1 + ORDER BY 1 DESC; +``` + +### Top 20 most expensive analyses (last 30 days) + +```sql +SELECT id, ticket_number, analysis_version, estimated_cost_usd, + opus_used, needs_human_review, triggered_at + FROM analyzer_analyses + WHERE triggered_at > now() - interval '30 days' + ORDER BY estimated_cost_usd DESC + LIMIT 20; +``` + +### Heaviest tickets (multiple re-runs against the same ticket) + +```sql +SELECT ticket_number, + count(*) AS analyses, + round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd, + max(triggered_at) AS last_run + FROM analyzer_analyses + GROUP BY 1 +HAVING count(*) > 1 + ORDER BY spend_usd DESC + LIMIT 20; +``` + +### Cost ceiling — what to do when it trips + +`needs_human_review = true` with `human_review_reasons` containing "cost +ceiling reached" means Sonnet ran but Opus was skipped. The Sonnet output +is still persisted and is usually fine. To re-run with Opus forced on, +trigger a new analysis (the route accepts `force=true`); the cost ceiling +will reapply, so check the prior `estimated_cost_usd` before committing. +To raise the ceiling, edit `COST_CEILING_USD` in +`lib/services/analyzer/pipeline.ts` and redeploy. + +To reduce spend without code changes: discourage operators from clicking +Re-analyze unless content has changed (the idempotency short-circuit +already prevents charged re-runs against the same `content_hash`). + +--- + +## Adding IT Glue org aliases + +Stage 2 resolves an Autotask company name to an IT Glue organization in +two steps: + +1. **Alias map** — `lib/services/analyzer/itglue-aliases.json`. Keys are + normalized company names (lowercase, single-spaced); values are IT Glue + organization IDs as **strings**. Exact-match wins immediately. +2. **Live IT Glue search** — fallback when the alias map misses. + +When the live search misses or returns the wrong org, add an alias. + +### Finding the IT Glue organization ID + +In the IT Glue UI, navigate to the organization. The numeric segment of +the URL (`/organizations/12345`) is the ID. Or via API: + +```bash +curl -s -H "x-api-key: $ITGLUE_API_KEY" \ + "$ITGLUE_API_BASE/organizations?filter[name]=Seubert%20and%20Associates" \ + | jq '.data[] | {id, name: .attributes.name}' +``` + +### Updating the alias file + +Edit `lib/services/analyzer/itglue-aliases.json`. Keys must be **already +normalized** — lowercase, single-spaced — because the matcher applies the +same normalization to the Autotask company name and looks up an exact key. + +```json +{ + "seubert": "1234567", + "seubert and associates": "1234567", + "acme insurance group": "7654321" +} +``` + +Strip the `_comment` / `_example` keys when adding the first real entry, +or leave them — they're ignored at match time (no real org name normalizes +to a leading underscore). + +Changes require a redeploy. The file is bundled into the build by Next's +`import` — there is no runtime reload. + +### Verifying an alias works + +After deploy, run an analysis on a ticket from that company. Check the +`model_traces.itglue` field on the resulting row: + +```sql +SELECT model_traces->'itglue' + FROM analyzer_analyses + WHERE id = ''; +``` + +`alias_used: true` confirms the alias hit. `org_id` should be the value +you put in the JSON. + +--- + +## Triaging failed analyses + +Two failure surfaces to know about: + +### 1. Failed jobs (`analyzer_jobs.status = 'failed'`) + +The job hit an exception before the analysis row was written. The error +message is in `analyzer_jobs.error_message`. Common causes: + +| Error message | Cause | Fix | +|---|---|---| +| `Ticket T... not found in local mirror — confirm sync is current.` | Autotask sync hasn't pulled the ticket yet. | Wait for the next sync, or trigger a manual sync from `/admin`. The webhook usually fills the gap; if not, see `scripts/backfill-ticket-notes-gap.ts` for the reconciliation pattern. | +| `LLM stage on claude-... failed twice` | Two Anthropic call attempts both failed (network / parse / 5xx). | Check the API key is valid (`curl` Anthropic), and check Anthropic status. Re-queue by clicking Re-analyze. | +| `Anthropic API error: 429 ...` | Rate limit. | Wait and re-queue. | +| `IT Glue search failed: ...` should NOT appear here — the pipeline tolerates IT Glue failures and continues without context. | If you see one, the failure was outside the catch block and is a bug. | + +```sql +-- Recent failures +SELECT id, ticket_number, error_message, queued_at, finished_at + FROM analyzer_jobs + WHERE status = 'failed' + AND queued_at > now() - interval '7 days' + ORDER BY queued_at DESC; +``` + +To retry a failed job, queue a fresh one — there is no in-place retry. +Either click Re-analyze in the UI or: + +```sql +INSERT INTO analyzer_jobs (ticket_number, queued_by_user_id) +SELECT ticket_number, queued_by_user_id + FROM analyzer_jobs + WHERE id = ''; +``` + +### 2. Completed but flagged for human review (`needs_human_review = true`) + +The pipeline finished and produced an analysis, but the model itself +reported low confidence or the cost circuit breaker tripped before Opus. +`human_review_reasons` is a JSONB array of short strings — read it. + +The UI exposes this queue at `/analyzer/queue`. SQL equivalent: + +```sql +SELECT id, ticket_number, analysis_version, + confidence_score, estimated_cost_usd, + human_review_reasons + FROM analyzer_analyses + WHERE needs_human_review = true + AND status = 'complete' + ORDER BY triggered_at DESC + LIMIT 50; +``` + +Reasons you'll see: + +- `"cost ceiling reached"` — Opus skipped to save money. Sonnet output is + what you have. Re-run with `force=true` to override (and accept the + spend). +- `"low confidence: "` — the model itself reports uncertainty. The + Sonnet output may still be useful as a starting point; treat as + guidance, not verdict. +- A model-emitted reason from Stage 3 or Stage 4. The full string is in + the JSONB — don't truncate before reading. + +### 3. Worker not picking up jobs + +If `analyzer_jobs` has rows in `status='queued'` for more than ~30s, the +worker is wedged or not running. + +- Check the application log for `[ANALYZER-WORKER] starting` at boot. If + missing in production, `NODE_ENV` may not be set to `production` — + bring up `ANALYZER_WORKER_AUTOSTART=1` as a workaround. +- The worker is in-process; restarting the Pulse app restarts it. +- Concurrency: multiple replicas all run a worker, but `claimQueuedJob` + uses `FOR UPDATE SKIP LOCKED` so each job is processed exactly once. + +--- + +## Manual ops + +### Trigger an analysis from psql + +```sql +INSERT INTO analyzer_jobs (ticket_number, queued_by_user_id) +VALUES ('T20260424.0045', NULL); +``` + +The worker will pick it up within 2 seconds. `queued_by_user_id` may be +null (`ON DELETE SET NULL`). + +### Force a re-run despite a cached analysis + +The `force=true` flag on `POST /api/analyzer/tickets/:ticketNumber/analyze` +bypasses the idempotency short-circuit. The Re-analyze button in the UI +sends this flag. + +### Inspect raw stage outputs for a debugging session + +```sql +SELECT model_traces + FROM analyzer_analyses + WHERE id = ''; +``` + +Per-stage payload includes input/output tokens, attempt count, full parsed +model responses, and IT Glue resolution metadata. Don't paste this into +external systems — it can include redacted-but-still-sensitive snippets +of customer data. + +### Ad hoc: how much have we spent this month? + +```sql +SELECT round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd, + count(*) AS analyses + FROM analyzer_analyses + WHERE triggered_at >= date_trunc('month', now()); +``` + +--- + +## Things that are not implemented yet + +- No automated retries on failed jobs. +- No `viewed_at` tracking on share rows (column exists, nothing writes + to it). +- No `email_sent_at` column — share email send state lives only in the + HTTP response and the server log. +- No pre-Sonnet cost gate. The circuit breaker only protects against + Opus. +- No autocomplete on the share-recipient field. + +If any of these become a real operational problem, file the work — the +stubs are intentional and called out in +`wulf-pulse-ticket-analyzer-build-notes.md`.