docs(analyzer): phase 8 — operator runbook
docs/wulf-pulse-ticket-analyzer-runbook.md covers cost monitoring queries, the $2 cost ceiling, IT Glue alias workflow, failure triage (failed jobs vs needs_human_review), and manual ops (queue from psql, force re-run, inspect model_traces). Calls out the manual migration step for existing DBs and lists the unimplemented surfaces (no auto retries, no viewed_at, no email_sent_at) so operators don't trip on them. Linked from README. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
ed3b363d02
commit
1c5ec0f947
3 changed files with 358 additions and 0 deletions
|
|
@ -74,6 +74,7 @@ See `AUTOTASK_API_GUIDE.md` and `ADDIGY_API_GUIDE.md` for credential setup.
|
|||
- **`POSTGRES_SYNC_SETUP.md`** — database initialization
|
||||
- **`DOCKER_README.md`** — Docker workflow
|
||||
- **`PULSE_DATABASE_SKILL.md`** — diagnostic SQL queries
|
||||
- **`docs/wulf-pulse-ticket-analyzer-runbook.md`** — AI ticket analyzer operator runbook (cost monitoring, IT Glue alias map, failure triage)
|
||||
- **`docs/`** — sync behavior, webhook setup, workflow editor, per-integration guides
|
||||
|
||||
## Deployment
|
||||
|
|
|
|||
|
|
@ -376,6 +376,44 @@ This file is updated after each phase ships.
|
|||
|
||||
---
|
||||
|
||||
## Phase 8 — Operator runbook
|
||||
|
||||
**Delivered**
|
||||
|
||||
- `docs/wulf-pulse-ticket-analyzer-runbook.md` — cost monitoring queries
|
||||
(daily spend, top expensive analyses, multi-run tickets), the cost
|
||||
ceiling explanation (`COST_CEILING_USD = 2.00` in `pipeline.ts`), the
|
||||
IT Glue alias workflow (where to find org IDs, how to verify via
|
||||
`model_traces.itglue.alias_used`), failure triage tables (failed jobs
|
||||
vs. `needs_human_review` flagged completes, with the actual error
|
||||
strings the worker emits), and a manual-ops section for queueing jobs
|
||||
from psql.
|
||||
- README.md "Documentation" section links the runbook so operators
|
||||
finding the project for the first time discover it.
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Runbook lives in `docs/`, not the README.** Per CLAUDE.md, long-form
|
||||
per-feature docs go in `docs/` and the README points to them. Matches
|
||||
the existing pattern (workflow-editor-guide, sync guides, etc.).
|
||||
- **Manual migration step is documented.** Postgres only re-applies
|
||||
migrations on first init — operators applying analyzer to an existing
|
||||
DB need to run `psql -f migrations/069_*.sql`. The runbook leads with
|
||||
this.
|
||||
- **Listed what's *not* implemented.** Retries, `viewed_at`, share
|
||||
`email_sent_at`, pre-Sonnet cost gate, share-recipient autocomplete.
|
||||
Better to advertise the gaps than have an operator trip over them.
|
||||
|
||||
**Deliberately left out**
|
||||
|
||||
- No production-monitoring dashboard (Grafana etc.) — Pulse doesn't have
|
||||
one for any other feature, and the SQL queries cover the same ground.
|
||||
- No README expansion for the analyzer feature itself. Pulse's README is
|
||||
intentionally short; the runbook + spec + build notes are the deep
|
||||
docs.
|
||||
|
||||
---
|
||||
|
||||
## Status after each phase
|
||||
|
||||
| Phase | Tests | tsc | Notes |
|
||||
|
|
@ -387,3 +425,4 @@ This file is updated after each phase ships.
|
|||
| 5 | 128 | clean | API routes (no route tests) |
|
||||
| 6 | 128 | clean | frontend (no FE tests) |
|
||||
| 7 | 128 | clean | share email via existing SMTP transport |
|
||||
| 8 | 128 | clean | operator runbook + README link |
|
||||
|
|
|
|||
318
docs/wulf-pulse-ticket-analyzer-runbook.md
Normal file
318
docs/wulf-pulse-ticket-analyzer-runbook.md
Normal file
|
|
@ -0,0 +1,318 @@
|
|||
# AI Ticket Analyzer — Operator Runbook
|
||||
|
||||
How to keep the analyzer healthy in production. Audience is whoever is on
|
||||
call for Pulse. The feature spec is in
|
||||
`wulf-pulse-ticket-analyzer-prompt.md`; the implementation log is in
|
||||
`wulf-pulse-ticket-analyzer-build-notes.md`.
|
||||
|
||||
---
|
||||
|
||||
## At a glance
|
||||
|
||||
- **Pipeline**: Haiku triage → (optional) IT Glue retrieval → Sonnet deep
|
||||
analysis → (optional) Opus deep reasoning → persistence.
|
||||
- **Trigger**: user clicks "Analyze" on a ticket page; that POSTs to
|
||||
`/api/analyzer/tickets/:ticketNumber/analyze`. The route preprocesses
|
||||
inline, short-circuits to the cached row when content hasn't changed,
|
||||
otherwise queues a job for the worker.
|
||||
- **Worker**: `analyzerWorker` polls `analyzer_jobs` every 2 s in-process.
|
||||
Auto-starts in production (`NODE_ENV=production`). Opt in for dev with
|
||||
`ANALYZER_WORKER_AUTOSTART=1`.
|
||||
- **Cost ceiling**: hard-coded `COST_CEILING_USD = 2.00` in
|
||||
`lib/services/analyzer/pipeline.ts`. Trips before Opus only — Sonnet runs
|
||||
unconditionally. When tripped, the analysis is flagged
|
||||
`needs_human_review = true` with reason "cost ceiling reached".
|
||||
|
||||
---
|
||||
|
||||
## Required environment
|
||||
|
||||
Add to `~/projects_env/wulf-pulse.env` (do not commit values):
|
||||
|
||||
```
|
||||
ANTHROPIC_API_KEY=
|
||||
ITGLUE_API_KEY=
|
||||
ITGLUE_API_BASE=https://api.itglue.com
|
||||
ALLOWED_SHARE_DOMAINS=wulfconsulting.com
|
||||
# Optional in dev:
|
||||
ANALYZER_WORKER_AUTOSTART=1
|
||||
```
|
||||
|
||||
`SMTP_*` and `BETTER_AUTH_URL` are already set elsewhere in Pulse — share
|
||||
emails reuse the existing magic-link transport.
|
||||
|
||||
---
|
||||
|
||||
## First-time setup on an existing database
|
||||
|
||||
Migration `069_create_analyzer_tables.sql` creates `analyzer_analyses`,
|
||||
`analyzer_shares`, and `analyzer_jobs`. Postgres only re-applies migrations
|
||||
on a fresh data volume — for an existing DB, run the migration directly:
|
||||
|
||||
```bash
|
||||
psql "$DATABASE_URL" -f migrations/069_create_analyzer_tables.sql
|
||||
```
|
||||
|
||||
The script is idempotent (`IF NOT EXISTS` on every object).
|
||||
|
||||
---
|
||||
|
||||
## Monitoring cost
|
||||
|
||||
Every completed analysis stores `estimated_cost_usd`, `total_input_tokens`,
|
||||
`total_output_tokens`, and per-stage tier flags
|
||||
(`haiku_used` / `sonnet_used` / `opus_used`). The number is a forward
|
||||
estimate from `lib/services/llm/pricing.ts` and will drift if Anthropic
|
||||
changes published rates — re-check that file before believing big spend
|
||||
numbers.
|
||||
|
||||
### Daily spend (last 14 days)
|
||||
|
||||
```sql
|
||||
SELECT date_trunc('day', triggered_at) AS day,
|
||||
count(*) FILTER (WHERE status = 'complete') AS analyses,
|
||||
count(*) FILTER (WHERE opus_used) AS opus_runs,
|
||||
count(*) FILTER (WHERE needs_human_review) AS needs_review,
|
||||
round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd
|
||||
FROM analyzer_analyses
|
||||
WHERE triggered_at > now() - interval '14 days'
|
||||
GROUP BY 1
|
||||
ORDER BY 1 DESC;
|
||||
```
|
||||
|
||||
### Top 20 most expensive analyses (last 30 days)
|
||||
|
||||
```sql
|
||||
SELECT id, ticket_number, analysis_version, estimated_cost_usd,
|
||||
opus_used, needs_human_review, triggered_at
|
||||
FROM analyzer_analyses
|
||||
WHERE triggered_at > now() - interval '30 days'
|
||||
ORDER BY estimated_cost_usd DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
### Heaviest tickets (multiple re-runs against the same ticket)
|
||||
|
||||
```sql
|
||||
SELECT ticket_number,
|
||||
count(*) AS analyses,
|
||||
round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd,
|
||||
max(triggered_at) AS last_run
|
||||
FROM analyzer_analyses
|
||||
GROUP BY 1
|
||||
HAVING count(*) > 1
|
||||
ORDER BY spend_usd DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
### Cost ceiling — what to do when it trips
|
||||
|
||||
`needs_human_review = true` with `human_review_reasons` containing "cost
|
||||
ceiling reached" means Sonnet ran but Opus was skipped. The Sonnet output
|
||||
is still persisted and is usually fine. To re-run with Opus forced on,
|
||||
trigger a new analysis (the route accepts `force=true`); the cost ceiling
|
||||
will reapply, so check the prior `estimated_cost_usd` before committing.
|
||||
To raise the ceiling, edit `COST_CEILING_USD` in
|
||||
`lib/services/analyzer/pipeline.ts` and redeploy.
|
||||
|
||||
To reduce spend without code changes: discourage operators from clicking
|
||||
Re-analyze unless content has changed (the idempotency short-circuit
|
||||
already prevents charged re-runs against the same `content_hash`).
|
||||
|
||||
---
|
||||
|
||||
## Adding IT Glue org aliases
|
||||
|
||||
Stage 2 resolves an Autotask company name to an IT Glue organization in
|
||||
two steps:
|
||||
|
||||
1. **Alias map** — `lib/services/analyzer/itglue-aliases.json`. Keys are
|
||||
normalized company names (lowercase, single-spaced); values are IT Glue
|
||||
organization IDs as **strings**. Exact-match wins immediately.
|
||||
2. **Live IT Glue search** — fallback when the alias map misses.
|
||||
|
||||
When the live search misses or returns the wrong org, add an alias.
|
||||
|
||||
### Finding the IT Glue organization ID
|
||||
|
||||
In the IT Glue UI, navigate to the organization. The numeric segment of
|
||||
the URL (`/organizations/12345`) is the ID. Or via API:
|
||||
|
||||
```bash
|
||||
curl -s -H "x-api-key: $ITGLUE_API_KEY" \
|
||||
"$ITGLUE_API_BASE/organizations?filter[name]=Seubert%20and%20Associates" \
|
||||
| jq '.data[] | {id, name: .attributes.name}'
|
||||
```
|
||||
|
||||
### Updating the alias file
|
||||
|
||||
Edit `lib/services/analyzer/itglue-aliases.json`. Keys must be **already
|
||||
normalized** — lowercase, single-spaced — because the matcher applies the
|
||||
same normalization to the Autotask company name and looks up an exact key.
|
||||
|
||||
```json
|
||||
{
|
||||
"seubert": "1234567",
|
||||
"seubert and associates": "1234567",
|
||||
"acme insurance group": "7654321"
|
||||
}
|
||||
```
|
||||
|
||||
Strip the `_comment` / `_example` keys when adding the first real entry,
|
||||
or leave them — they're ignored at match time (no real org name normalizes
|
||||
to a leading underscore).
|
||||
|
||||
Changes require a redeploy. The file is bundled into the build by Next's
|
||||
`import` — there is no runtime reload.
|
||||
|
||||
### Verifying an alias works
|
||||
|
||||
After deploy, run an analysis on a ticket from that company. Check the
|
||||
`model_traces.itglue` field on the resulting row:
|
||||
|
||||
```sql
|
||||
SELECT model_traces->'itglue'
|
||||
FROM analyzer_analyses
|
||||
WHERE id = '<analysis-id>';
|
||||
```
|
||||
|
||||
`alias_used: true` confirms the alias hit. `org_id` should be the value
|
||||
you put in the JSON.
|
||||
|
||||
---
|
||||
|
||||
## Triaging failed analyses
|
||||
|
||||
Two failure surfaces to know about:
|
||||
|
||||
### 1. Failed jobs (`analyzer_jobs.status = 'failed'`)
|
||||
|
||||
The job hit an exception before the analysis row was written. The error
|
||||
message is in `analyzer_jobs.error_message`. Common causes:
|
||||
|
||||
| Error message | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `Ticket T... not found in local mirror — confirm sync is current.` | Autotask sync hasn't pulled the ticket yet. | Wait for the next sync, or trigger a manual sync from `/admin`. The webhook usually fills the gap; if not, see `scripts/backfill-ticket-notes-gap.ts` for the reconciliation pattern. |
|
||||
| `LLM stage on claude-... failed twice` | Two Anthropic call attempts both failed (network / parse / 5xx). | Check the API key is valid (`curl` Anthropic), and check Anthropic status. Re-queue by clicking Re-analyze. |
|
||||
| `Anthropic API error: 429 ...` | Rate limit. | Wait and re-queue. |
|
||||
| `IT Glue search failed: ...` should NOT appear here — the pipeline tolerates IT Glue failures and continues without context. | If you see one, the failure was outside the catch block and is a bug. |
|
||||
|
||||
```sql
|
||||
-- Recent failures
|
||||
SELECT id, ticket_number, error_message, queued_at, finished_at
|
||||
FROM analyzer_jobs
|
||||
WHERE status = 'failed'
|
||||
AND queued_at > now() - interval '7 days'
|
||||
ORDER BY queued_at DESC;
|
||||
```
|
||||
|
||||
To retry a failed job, queue a fresh one — there is no in-place retry.
|
||||
Either click Re-analyze in the UI or:
|
||||
|
||||
```sql
|
||||
INSERT INTO analyzer_jobs (ticket_number, queued_by_user_id)
|
||||
SELECT ticket_number, queued_by_user_id
|
||||
FROM analyzer_jobs
|
||||
WHERE id = '<failed-job-id>';
|
||||
```
|
||||
|
||||
### 2. Completed but flagged for human review (`needs_human_review = true`)
|
||||
|
||||
The pipeline finished and produced an analysis, but the model itself
|
||||
reported low confidence or the cost circuit breaker tripped before Opus.
|
||||
`human_review_reasons` is a JSONB array of short strings — read it.
|
||||
|
||||
The UI exposes this queue at `/analyzer/queue`. SQL equivalent:
|
||||
|
||||
```sql
|
||||
SELECT id, ticket_number, analysis_version,
|
||||
confidence_score, estimated_cost_usd,
|
||||
human_review_reasons
|
||||
FROM analyzer_analyses
|
||||
WHERE needs_human_review = true
|
||||
AND status = 'complete'
|
||||
ORDER BY triggered_at DESC
|
||||
LIMIT 50;
|
||||
```
|
||||
|
||||
Reasons you'll see:
|
||||
|
||||
- `"cost ceiling reached"` — Opus skipped to save money. Sonnet output is
|
||||
what you have. Re-run with `force=true` to override (and accept the
|
||||
spend).
|
||||
- `"low confidence: <score>"` — the model itself reports uncertainty. The
|
||||
Sonnet output may still be useful as a starting point; treat as
|
||||
guidance, not verdict.
|
||||
- A model-emitted reason from Stage 3 or Stage 4. The full string is in
|
||||
the JSONB — don't truncate before reading.
|
||||
|
||||
### 3. Worker not picking up jobs
|
||||
|
||||
If `analyzer_jobs` has rows in `status='queued'` for more than ~30s, the
|
||||
worker is wedged or not running.
|
||||
|
||||
- Check the application log for `[ANALYZER-WORKER] starting` at boot. If
|
||||
missing in production, `NODE_ENV` may not be set to `production` —
|
||||
bring up `ANALYZER_WORKER_AUTOSTART=1` as a workaround.
|
||||
- The worker is in-process; restarting the Pulse app restarts it.
|
||||
- Concurrency: multiple replicas all run a worker, but `claimQueuedJob`
|
||||
uses `FOR UPDATE SKIP LOCKED` so each job is processed exactly once.
|
||||
|
||||
---
|
||||
|
||||
## Manual ops
|
||||
|
||||
### Trigger an analysis from psql
|
||||
|
||||
```sql
|
||||
INSERT INTO analyzer_jobs (ticket_number, queued_by_user_id)
|
||||
VALUES ('T20260424.0045', NULL);
|
||||
```
|
||||
|
||||
The worker will pick it up within 2 seconds. `queued_by_user_id` may be
|
||||
null (`ON DELETE SET NULL`).
|
||||
|
||||
### Force a re-run despite a cached analysis
|
||||
|
||||
The `force=true` flag on `POST /api/analyzer/tickets/:ticketNumber/analyze`
|
||||
bypasses the idempotency short-circuit. The Re-analyze button in the UI
|
||||
sends this flag.
|
||||
|
||||
### Inspect raw stage outputs for a debugging session
|
||||
|
||||
```sql
|
||||
SELECT model_traces
|
||||
FROM analyzer_analyses
|
||||
WHERE id = '<analysis-id>';
|
||||
```
|
||||
|
||||
Per-stage payload includes input/output tokens, attempt count, full parsed
|
||||
model responses, and IT Glue resolution metadata. Don't paste this into
|
||||
external systems — it can include redacted-but-still-sensitive snippets
|
||||
of customer data.
|
||||
|
||||
### Ad hoc: how much have we spent this month?
|
||||
|
||||
```sql
|
||||
SELECT round(sum(estimated_cost_usd)::numeric, 2) AS spend_usd,
|
||||
count(*) AS analyses
|
||||
FROM analyzer_analyses
|
||||
WHERE triggered_at >= date_trunc('month', now());
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Things that are not implemented yet
|
||||
|
||||
- No automated retries on failed jobs.
|
||||
- No `viewed_at` tracking on share rows (column exists, nothing writes
|
||||
to it).
|
||||
- No `email_sent_at` column — share email send state lives only in the
|
||||
HTTP response and the server log.
|
||||
- No pre-Sonnet cost gate. The circuit breaker only protects against
|
||||
Opus.
|
||||
- No autocomplete on the share-recipient field.
|
||||
|
||||
If any of these become a real operational problem, file the work — the
|
||||
stubs are intentional and called out in
|
||||
`wulf-pulse-ticket-analyzer-build-notes.md`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue