wulf-pulse/ARCHITECTURE.md

# Pulse — Architecture

Pulse is a single Next.js 16 app (`output: 'standalone'`) that backs Wulf
Consulting's PSA workflows. It pulls data from Autotask, Datto RMM, IT Glue, MS
Graph, Veeam, and ~10 other systems into Postgres, runs background workers for
sync / AI analysis / RMM execution, and serves dashboards + admin tooling on
port **3100**.

This file is the load-bearing reference for *how the system is wired*. Per-
feature deep dives live in `docs/`. UI/visual conventions live in `DESIGN.md`.

## 1. Runtime topology

One Node process, one Postgres, one Redis. Background work runs **in-process**
inside the Next server — there is no external job queue.

```
┌─────────────────────────────────────────────────────────┐
│ Next.js 16 (port 3100, output: 'standalone')            │
│                                                         │
│  HTTP routes ──► API handlers ──► Postgres / Redis      │
│                                                         │
│  Side-effect imports auto-start three workers:          │
│   • SyncScheduler     (node-cron)                       │
│   • AnalyzerWorker    (poll analyzer_jobs every 2s)     │
│   • RmmOvershellWorker (poll rmm_executions every 5s)   │
└─────────────────────────────────────────────────────────┘
        │                  │                    │
        ▼                  ▼                    ▼
   Postgres 16         Redis 7            External APIs
   (state)             (cache only)       (Autotask, RMM, …)
```

**Workers are side-effect imports.** Touching one of these modules from a
server-side import path starts the loop:

| Worker | File | Trigger |
|---|---|---|
| Sync scheduler | `lib/services/sync-scheduler.ts` | Self-init at module bottom |
| Analyzer | `lib/services/analyzer/worker.ts` | Auto-starts when `NODE_ENV=production` or `ANALYZER_WORKER_AUTOSTART=1` |
| RMM Overshell | `lib/services/rmm/worker.ts` | Same gate as analyzer |

Routes that need a worker running deliberately import its module — e.g.
`app/api/analyzer/tickets/[ticketNumber]/analyze/route.ts` imports
`lib/services/analyzer/worker.ts` purely so the loop starts on first request.
**Don't eager-import these from hot paths or shared utilities.**

There is **no out-of-process queue**. Scaling out to multiple Next instances
means each instance runs duplicate pollers. The analyzer worker uses
`SELECT … FOR UPDATE SKIP LOCKED` so jobs run exactly once across instances,
but the sync scheduler does not — run it on a single dedicated instance, or
gate it behind an env flag on replicas.

## 2. Data flow

### Autotask (primary system of record)
- **Webhook** — `POST /api/webhooks/autotask`, public per `middleware.ts`,
  HMAC-verified inside the handler. Returns 200 even on processing errors so
  Autotask doesn't deactivate the subscription. Async handlers may enqueue
  analyzer jobs.
- **Periodic sync** — `lib/services/entity-sync.ts` runs per scheduled task.
  Incremental via `lastTrackedModificationDateTime` when supported; full upsert
  otherwise. Targets `tickets`, `companies`, `contacts`, `resources`, `tasks`,
  `time_entries`, etc.

### Datto RMM
- **Sync** — devices, sites, alerts → `datto_rmm_*` tables.
- **Overshell executor** — `lib/services/rmm/executor.ts` validates a
  registered script ID, resolves the target device, enforces a per-user rate
  limit (50 / 24h), inserts a `pending` row in `rmm_executions`, and calls
  `client.runQuickJob()`. The worker polls Datto for the result and parses
  output via the script's `parseOutput()` method. Scripts are **code-
  registered** (`lib/services/rmm/scripts/`) — adding one is a TS change, not a
  DB change.
- **LogLift** — `POST /api/rmm/loglift/upload` (public, `x-openclaw-key`
  header). Receives a B2 object key, downloads + decompresses the gzipped JSON
  (capped at 100 MB), stores a slim summary in `loglift_uploads` and the full
  payload in B2 (`lib/services/b2/client.ts`). Resolves the device → Autotask
  company → IT Glue configuration; if a unique IT Glue match is found, fires
  an asset-first audit automatically.

### IT Glue
- **Sync** — `lib/services/itglue-sync-service.ts` pulls org types, configs,
  contacts, locations, flexible assets, etc. → `itg_*` tables. Note: flexible
  assets must be listed per type (API 422 otherwise — see commit `a0a6e7f`).
- **Search (analyzer)** — `lib/services/analyzer/itglue-search.ts` returns
  redacted documents only. Credentials/PII pass through `redact()` before any
  LLM sees them. Callers must not bypass redaction; the raw client is for non-
  LLM use.
- **Audit + write-back** — `lib/services/analyzer/asset-audit/` runs LLM-driven
  audits against IT Glue configurations or flexible assets, writes results to
  `itglue_audit_logs`, and links tickets via `itglue_ticket_xrefs`. Reverts go
  through `…/revert/[writeId]`.

### MS Graph (Engagement)
- App-only auth, **specific tenant ID** (not `common`). Reports API returns
  CSV; parsed inline. Joins to Autotask hours via `resources.email =
  graph_users.email`. See `lib/services/engagement-sync-service.ts`.

### Other integrations
Each has a factory + `is<Name>Configured()` helper in `lib/services/`. All
credentials come from env; clients throw if missing.

| System | Role | Files |
|---|---|---|
| Veeam VSPC | Backup status, RPO, ticket analysis | `veeam-*-service.ts` |
| Auvik | Network monitoring; tenant mappings | `auvik-client.ts` |
| Addigy | Apple endpoints; org mappings | `addigy-factory.ts` |
| Mimecast | Mail security | `mimecast-sync-service.ts` |
| SentinelOne | EDR | `sentinelone-sync-service.ts` |
| Duo | MFA | `duo-sync-service.ts` |
| Zoom | Meetings | `zoom-sync-service.ts` |
| QuickBooks | Billing reconciliation | `qbo-sync-service.ts` |
| Zabbix | WAN monitoring; webhook at `/api/zabbix/webhook` | — |
| Salesbldr | Sales pipeline | — |

## 3. Analyzer pipeline

`lib/services/analyzer/pipeline.ts` orchestrates seven stages. Provider is
chosen per request (`anthropic` default, `openrouter` opt-in); models are
mapped per stage in `lib/services/llm/models.ts`.

| Stage | Model (Anthropic) | Purpose |
|---|---|---|
| 0 — Preprocess | — | Filter workflow noise, tag entities, compute `content_hash` (idempotency key) |
| 1 — Triage | Haiku | Categorize, extract entities, initial priority |
| 2 — IT Glue retrieval | — | Redacted doc lookup, skipped if IT Glue not configured |
| 3 — Deep analysis | Sonnet | Summary, gaps, what-was-done, what-should-have-been-done |
| 4 — Deep reasoning | Opus (optional) | Apply corrections, propose IT Glue updates, re-rank |
| 5 — Persist | — | Write `analyzer_analyses` row + per-stage execution rows |
| 6 — Fingerprint | Haiku | Structured fingerprint for cross-ticket aggregation |

**Idempotency.** If `content_hash` already exists for the ticket and
`force=false`, the pipeline returns the existing analysis. The hash is
provider-scoped — Claude and DeepSeek analyses of the same ticket are separate
rows.

**Cost ceiling.** Stage 4 is skipped if estimated total cost exceeds **$2.00**;
the analysis is flagged for human review. All LLM/RMM activity is logged to
`analyzer_cost_audit`.

**Link-aware bundles.** `lib/services/analyzer/link-discovery.ts` resolves
related tickets two ways: (1) explicit — regex T-numbers in descriptions/notes,
"RELATED TICKETS:" blocks, the `problem_ticket_id` column; (2) suggested
(opt-in) — Haiku ranks recent same-company tickets by semantic similarity. When
a bundle is analyzed, members get `pending_analyses` rows; once all complete,
an aggregate report fires.

**Aggregate reports.** `stages/aggregate-reduce.ts` pairs SQL distributions
(category, client, resolution path, root cause) with a Sonnet pass that
identifies documentation gaps, process gaps, client patterns, recurrence
clusters. Persisted to `analyzer_aggregate_reports`.

**Asset audit (Phase 4).** `lib/services/analyzer/asset-audit/runner.ts` runs
post-analysis. Two modes: all-time evidence across every analysis linked to
the asset, or ticket-first (Phase 4.1) narrowed to a single analysis ID.

## 4. Background jobs

| Worker | Cadence | Scope | Concurrency model |
|---|---|---|---|
| `AnalyzerWorker` | 2s poll | Claims `analyzer_jobs.status='queued'` | `FOR UPDATE SKIP LOCKED`, exactly-once across instances |
| `RmmOvershellWorker` | 5s poll | Polls in-flight `rmm_executions`; advances state by querying Datto | Single in-process loop |
| `SyncScheduler` | node-cron | Per-schedule rows in DB (Autotask, IT Glue, Veeam, Engagement, Zoom, Duo, …) | **Not** safe for multi-instance — overlaps possible |
| `integration-health-alerts` | Cron-fired from sync scheduler | Detects stale syncs, publishes alerts | Single in-process |

Stale in-flight analyzer jobs are reset on worker boot (commit `378e68a`) so a
crashed pod doesn't leave jobs orphaned.

## 5. Auth & permissions

**Better Auth 1.4** with magic link + TOTP 2FA + Microsoft OAuth. Sessions
live in Postgres (no Redis session store). Account-linking is enabled for
Microsoft so admin-invited users join their MS account in one click.

**Roles.** `user`, `admin`, `super-admin`. Default admin bootstrapped from
`DEFAULT_ADMIN_EMAIL` via `lib/bootstrap.ts`.

**Resources** (`lib/permissions.ts`) — `tickets`, `configItems`, `admin`,
`users`, `roles`, `auditLog`, `settings`, `itglue`, `rmm`. The `itglue` and
`rmm` resources were added with the Overshell + IT Glue write-back work; user
role gets read-only on both.

**API auth pattern.** Every route handler calls one of:
```ts
const { session, error } = await requireAuth();
const { session, error } = await requireAdmin();
const { session, error } = await requireSuperAdmin();
const { session, error } = await requirePermission('itglue', 'write');
if (error) return error;
```
`middleware.ts` only checks for a session cookie — role/permission checks live
in the route handler.

**Public routes** (hardcoded in `middleware.ts`): `/api/auth/*`,
`/api/webhooks/*`, `/api/sync/*`, `/api/health`, `/api/zabbix/webhook`,
`/api/rmm/loglift`, `/api/mobile/*`, `/api/openclaw/*`, `/legal`,
`/api/kiosk`, `/api/qbo/*`. **Add to that list whenever you introduce a new
public endpoint.**

## 6. Database

89 numbered migrations (`migrations/NNN_*.sql`), applied in **alphabetical**
order on Postgres init only. Existing volumes do not re-run them — for
schema changes against an existing DB, use `scripts/apply-migrations` (verify
behavior first; varies by age of script).

Topical groupings:

| Range | Topic |
|---|---|
| 001–014 | Core schema, auth tables, admin settings |
| 015–026 | Ticket fields, queues, RMM site mappings, integration health |
| 027–032 | Datto RMM, Veeam agents/alarms, priorities, ticket categories, RMM webhooks |
| 037–044 | IT Glue (large), Veeam RPO, contract services, engagement |
| 045–055 | Zoom, Teams, morning summary, ping suppression, ticket digest, Zabbix WAN, QBO, Mimecast |
| 056–068 | UDFs, Autotask tags, Duo, project phases, recurring revenue, Veeam ticket analysis |
| 069–074 | Analyzer (jobs, analyses, stage executions, aggregate reports, cost audit, link-aware bundles, provider) |
| 075–076 | IT Glue audit + ticket xrefs |
| 077–078 | RMM Overshell, LogLift uploads |
| 079–080 | Endpoint data model, device-xref `company_id` |

**Watch out for:**
- Duplicate numbers exist (002, 004, 009). Apply order is filesystem-sort
  alphabetical, not numeric. Don't introduce more.
- Conventions: `IF NOT EXISTS` for tables/indexes, `ON CONFLICT DO NOTHING`
  for seed data, audit columns `created_at` / `updated_at` / `synced_at` /
  `is_deleted` / `deleted_at`.
- `priorities` has no `is_deleted` column — caught the hard way (`9acf48e`).
- Columns are **`snake_case`**; API responses are **`camelCase`**. Handlers
  transform manually. No ORM.

DB access is the singleton at `lib/services/postgres-client.ts` —
`postgresClient.query()`, `.transaction()`, `.upsert()`, `.bulkUpsert()`.

## 7. Deployment

**Docker Compose** at the repo root.
- `postgres` — Postgres 16, port 5432. Migrations volume mounted at
  `/docker-entrypoint-initdb.d/`. Init runs once per volume.
- `redis` — Redis 7, port 6380 (host) / 6379 (container). Cache only.
- `app` — built from `Dockerfile` (turbopack, `output: 'standalone'`); runs
  `node server.js`. Port 3100. `.env.local` mounted read-only. Traefik labels
  for `pulse.wulfconsulting.cloud` (HTTPS via Cloudflare cert).

`npm run build` uses turbopack (Next 16 default). `npm run dev` for local;
`npx tsc --noEmit --pretty` for type check; `npm test` (vitest) for the
analyzer / RMM / B2 / link-discovery unit tests. **No CI**; type-check is the
only safety net for code that doesn't have unit tests.

## 8. Invariants & gotchas

1. **Worker side-effect imports.** Importing `sync-scheduler.ts`,
   `analyzer/worker.ts`, or `rmm/worker.ts` from a hot path starts the loop.
2. **No external queue.** Multiple instances duplicate pollers. Analyzer is
   safe via row locking; sync scheduler is not — pin to one instance.
3. **IT Glue redaction is mandatory** for any LLM-bound query. Use
   `itglue-search.ts`, never the raw client.
4. **Provider-scoped idempotency.** `force=false` only short-circuits if the
   same provider produced the existing analysis.
5. **Cost ceiling at $2.00** before Stage 4. Above that, Opus is skipped and
   the analysis is flagged.
6. **Webhook handlers return 200 on failure** (Autotask) to avoid
   deactivation. Errors are logged, not surfaced.
7. **Postgres init runs migrations once.** Existing volumes won't re-run them.
8. **Duplicate migration numbers.** Apply order is alphabetic.
9. **`.env` is committed.** Treat the values as potentially real production
   secrets; don't log or echo them.
10. **RMM script registry is in code.** `lib/services/rmm/scripts/` —
    unregistered scripts can't execute.
11. **LogLift zip-bomb guard** caps inflated payloads at 100 MB.
12. **Stale analyzer jobs reset on worker boot.** Don't rely on `in_flight`
    state surviving restarts.

## 9. Where to look

| Concern | Start here |
|---|---|
| HTTP routes | `app/api/**/route.ts` |
| Pages | `app/**/page.tsx` |
| Worker boot | `lib/services/{sync-scheduler,analyzer/worker,rmm/worker}.ts` |
| Postgres access | `lib/services/postgres-client.ts` |
| Auth wiring | `lib/auth.ts`, `lib/auth-utils.ts`, `lib/permissions.ts`, `middleware.ts` |
| Analyzer pipeline | `lib/services/analyzer/pipeline.ts` + `stages/` |
| LLM dispatch | `lib/services/llm/{call,models,pricing}.ts` |
| RMM executor | `lib/services/rmm/{executor,worker,target-resolver}.ts` |
| IT Glue write-back | `lib/services/analyzer/asset-audit/` |
| Per-feature notes | `docs/` (one file per system) |