# Architecture

**Analysis Date:** 2026-05-03

## Pattern Overview

**Overall:** Single-instance Next.js 16 backend with in-process background workers (no external job queue).

**Key Characteristics:**
- App Router pages (`'use client'`) fetch data via API routes using `fetch()`
- Three auto-starting background workers via side-effect imports (sync-scheduler, analyzer, RMM overshell)
- Service layer in `lib/services/` with factories + configuration helpers
- Postgres 16 as system of record (snake_case columns), Redis for caching only
- Manual transformation from snake_case DB columns to camelCase API responses
- Better Auth 1.4 for sessions + magic link + TOTP 2FA + Microsoft OAuth

## Layers

**Route Layer (HTTP entry):**
- Purpose: Accept HTTP requests, validate auth, delegate to services, return JSON responses
- Location: `app/api/*/route.ts`, `app/*/page.tsx`
- Contains: Next.js route handlers (GET/POST/PATCH/DELETE), page components
- Depends on: Auth via `lib/auth-utils.ts`, services via `lib/services/`
- Used by: Client-side fetch calls from UI components

**Service Layer (business logic):**
- Purpose: Sync data from external APIs, run background jobs, execute workflows, store results in Postgres
- Location: `lib/services/*.ts` (~50 files) plus subdirectories (`analyzer/`, `rmm/`, `llm/`, `b2/`)
- Contains: Integration clients (autotask, datto-rmm, itglue, veeam, msgraph, etc.), sync logic, job executors, pipeline orchestration
- Depends on: Postgres client, external API clients, environment configuration
- Used by: API routes (sync endpoints, webhook handlers) and background workers

**Data Access Layer (Postgres):**
- Purpose: Query, upsert, and manage state in Postgres
- Location: `lib/services/postgres-client.ts` (singleton) + `migrations/NNN_*.sql`
- Contains: Connection pool, query builder methods (`query()`, `transaction()`, `upsert()`, `bulkUpsert()`), migration definitions
- Depends on: PostgreSQL 16 connection string from env
- Used by: All services that read/write data

**Background Workers (long-running processes):**
- Purpose: Poll for work and execute sync/analysis/RMM tasks without blocking HTTP requests
- Location: `lib/services/sync-scheduler.ts`, `lib/services/analyzer/worker.ts`, `lib/services/rmm/worker.ts`
- Contains: node-cron scheduler, polling loops with exponential backoff, state machine handlers
- Depends on: Postgres client, service layer integrations
- Used by: Auto-start on module import (side effects); HTTP routes that need them running will import the module to start them

**Component Layer (UI):**
- Purpose: Render pages, dialogs, tables, charts, and handle client-side state
- Location: `components/ui/` (shadcn primitives), `components/*/` (feature-specific), `app/*/page.tsx`
- Contains: React components, hooks for fetch + state, sonner toasts, recharts visualizations
- Depends on: API routes via `fetch()`, client auth via Better Auth SDK
- Used by: Next.js pages and other components

**Types & Schema (contracts):**
- Purpose: Define TypeScript interfaces and database schema
- Location: `lib/types/<domain>.ts`, `migrations/NNN_*.sql`
- Contains: Entity types (ticket, company, analysis, rmm_execution, etc.), sync request shapes, API response envelopes
- Depends on: (none — they define contracts)
- Used by: Services, API routes, components

## Data Flow

**Autotask Webhook → Analyzer Pipeline:**

1. **Webhook ingress** (`POST /api/webhooks/autotask`) — Public endpoint (no auth), HMAC-verified inside handler via `lib/services/webhook-service.ts`
2. **Enqueue job** — If ticket.created, fires webhook handler in `lib/services/webhook-service.ts` which inserts `analyzer_jobs` row with `status='queued'`
3. **Worker poll** — `lib/services/analyzer/worker.ts` auto-starts in production; every 2s polls for `analyzer_jobs.status='queued'`, claims one with `FOR UPDATE SKIP LOCKED`
4. **Pipeline orchestration** — `lib/services/analyzer/pipeline.ts` runs 7 stages:
   - Stage 0: Preprocess (filter noise, compute `content_hash` for idempotency)
   - Stage 1: Triage (Haiku — categorize, extract entities)
   - Stage 2: IT Glue retrieval (if configured, redacted docs only via `itglue-search.ts`)
   - Stage 3: Deep analysis (Sonnet — summary, gaps, root cause)
   - Stage 4: Deep reasoning (Opus — optional, skipped above $2.00 cost ceiling)
   - Stage 5: Persist to `analyzer_analyses` + stage execution rows
   - Stage 6: Fingerprint (Haiku — cross-ticket aggregation data)
5. **Link-aware bundles** — `lib/services/analyzer/link-discovery.ts` resolves related tickets; members get `pending_analyses` rows
6. **Aggregate reports** (optional) — Once all bundle members analyzed, `stages/aggregate-reduce.ts` fires

**Periodic Sync Cadence:**

1. **Cron trigger** — `lib/services/sync-scheduler.ts` (node-cron singleton) reads `sync_schedules` table; fires at configured times
2. **Entity sync** — `lib/services/entity-sync.ts` per entity type (tickets, companies, resources, etc.), incremental via `lastTrackedModificationDateTime` when supported
3. **Postgres upsert** — `postgresClient.bulkUpsert()` writes batches to DB tables (`tickets`, `companies`, etc.)
4. **Integration-specific syncs** — Datto RMM devices/alerts, IT Glue configs/contacts, Veeam agents/alarms, Engagement data, Zoom, Duo, etc.

**RMM Overshell Execution:**

1. **User trigger** (`POST /api/rmm/execute`) — Admin user picks registered script + device
2. **Validation & rate limit** — `lib/services/rmm/executor.ts` validates script ID, resolves device, enforces per-user limit (50 / 24h)
3. **Queue insertion** — Insert `pending` row in `rmm_executions` table, call Datto `client.runQuickJob()`
4. **Worker poll** — `lib/services/rmm/worker.ts` (5s cadence) polls in-flight executions, queries Datto for result status
5. **Output parsing** — Script's `parseOutput()` method transforms Datto output; result stored in `rmm_executions`

**LogLift Evidence Ingest:**

1. **Webhook** (`POST /api/rmm/loglift/upload`) — Public, `x-openclaw-key` header auth
2. **Decompression** — Download gzipped JSON from B2, decompress, cap at 100 MB (zip-bomb guard)
3. **Storage** — Slim summary to `loglift_uploads` table, full payload to B2 via `lib/services/b2/client.ts`
4. **Auto-audit** — Resolve device → Autotask company → IT Glue config; if unique match, fire asset-first audit

**IT Glue Write-back:**

1. **Audit runner** (`lib/services/analyzer/asset-audit/runner.ts`) — Post-analysis, runs LLM audit against IT Glue config/flexible asset
2. **Results** — Persisted to `itglue_audit_logs`, linked via `itglue_ticket_xrefs`
3. **Revert** — Patch `/api/analyzer/itglue/configurations/[id]/revert/[writeId]` rolls back changes

**State Management:**

- **HTTP requests:** Stateless; session from Better Auth cookie
- **Background jobs:** State in Postgres (status columns: `queued` → `in_flight` → `complete` / `failed`)
- **Caching:** Redis (optional, used for integration health checks, cache duration varies)
- **Authorization:** Checked in route handlers via `requireAuth()` / `requireAdmin()` / `requirePermission()` from `lib/auth-utils.ts`

## Key Abstractions

**Factory Pattern (Integration Clients):**
- Purpose: Lazy-load integration clients with configured credentials; provide `is<Name>Configured()` helper to check env vars
- Examples: `lib/services/autotask-factory.ts`, `lib/services/datto-rmm-factory.ts`, `lib/services/msgraph-factory.ts`
- Pattern: Export `getAutotaskClient()`, `getDattoRmmClient()`, `getMsgraphClient()` with caching; throw if credentials missing; bundled with `is<Name>Configured()` for upstream checks
- Why: Decouples client initialization from route handlers; allows conditional feature gates per env

**PostgresClient Singleton:**
- Purpose: Single connection pool for all Postgres queries; auto-lazy-initializes; provides ORM-like query builder
- Examples: `postgresClient.query()`, `postgresClient.transaction()`, `postgresClient.upsert()`, `postgresClient.bulkUpsert()`
- Pattern: Private constructor, static `getInstance()`, pool initialized on first use
- Why: Prevents connection leaks; provides consistent interface across ~50 services

**Sync Service (Entity-Agnostic):**
- Purpose: Incremental/full sync of any entity type from external API → Postgres
- Examples: `lib/services/entity-sync.ts` (Autotask), `lib/services/itglue-sync-service.ts`, `lib/services/veeam-sync-service.ts`
- Pattern: Reads `lastTrackedModificationDateTime` from last_sync table; queries external API; batches upsert via `bulkUpsert()`
- Why: Codifies the "last sync timestamp + incremental pull + batch insert" pattern across integrations

**Analyzer Pipeline (7 Stages):**
- Purpose: Orchestrate multi-stage LLM analysis with fallbacks, cost guards, and persistence
- Examples: `lib/services/analyzer/pipeline.ts`, `lib/services/analyzer/stages/*.ts`
- Pattern: Each stage returns structured output (gap analysis, category, cost estimate); cost guard checks ceiling before Stage 4; all stages persisted to `analyzer_stage_executions`
- Why: Allows cost control (skip expensive Opus above $2.00), idempotency (content_hash), and debugging (inspect each stage's output)

**Background Worker Polling Loop:**
- Purpose: Auto-start in production, poll for work, claim rows with `FOR UPDATE SKIP LOCKED`, execute, persist result
- Examples: `lib/services/analyzer/worker.ts`, `lib/services/rmm/worker.ts`, `lib/services/sync-scheduler.ts`
- Pattern: Side-effect import auto-starts on module load; exponential backoff if no work; row locking for multi-instance safety (analyzer), or single-instance gate (sync scheduler)
- Why: Keeps background work out of HTTP request path; analyzer is safe to scale (row locking); sync scheduler should run on one instance only

**Better Auth Roles (RBAC):**
- Purpose: Define three roles (`user`, `admin`, `super-admin`) with per-resource permissions
- Examples: `lib/auth.ts` (auth config), `lib/permissions.ts` (permission matrix), `lib/auth-utils.ts` (runtime checks)
- Pattern: Route handler calls `requireAdmin()` / `requirePermission()`, which decode session and check role; middleware only verifies session cookie exists
- Why: Separates auth (middleware) from authorization (route handler); role-based gates are checked at the point of use

## Entry Points

**HTTP Pages (Authenticated):**
- Location: `app/*/page.tsx`
- Triggers: Browser navigation to any route except public paths
- Responsibilities: Render page shell with `AppNavigation` + `PageHeader`, fetch data from API, render client-side components, handle toast/dialog interactions

**HTTP API Routes (Public & Authenticated):**
- Location: `app/api/*/route.ts`
- Triggers: `fetch()` from client, external webhooks (Autotask, Zabbix, RMM), scheduler HTTP calls
- Responsibilities: Validate auth/webhook signature, delegate to service layer, return JSON response with appropriate status (200, 401, 403, 503, 500)

**Webhook Handlers (Public):**
- Location: `app/api/webhooks/autotask`, `app/api/zabbix/webhook`, `app/api/rmm/loglift`
- Triggers: External systems (Autotask, Zabbix, OpenClaw) POST events
- Responsibilities: Verify HMAC or custom header, parse event, enqueue jobs or upsert data, return 200 (even on error so Autotask doesn't deactivate)

**Sync Endpoints (Public, called by scheduler):**
- Location: `app/api/sync/*`, `app/api/datto-rmm/sync`, `app/api/itglue/sync`, `app/api/veeam/sync`, etc.
- Triggers: `lib/services/sync-scheduler.ts` fires HTTP POST at configured times
- Responsibilities: Call sync service, update `last_sync` timestamp, return 200 on success or error message

**Background Workers (Auto-starting, in-process):**
- Location: `lib/services/sync-scheduler.ts` (cron), `lib/services/analyzer/worker.ts` (2s poll), `lib/services/rmm/worker.ts` (5s poll)
- Triggers: Auto-starts as side effect of module import; runs indefinitely in production
- Responsibilities: Poll for work from DB, claim row, execute, persist result, handle errors + logging

## Error Handling

**Strategy:** Defensive; assume external APIs can fail, return 200 on webhook failures (so Autotask doesn't deactivate), log all errors, flag analyses for human review if cost ceiling exceeded.

**Patterns:**

- **Webhook handlers** — Return 200 even if processing fails; log error so ops can investigate via audit log
- **API route handlers** — `try/catch`, return `NextResponse.json({ error, message }, { status })` with conventions:
  - 401: Session missing or invalid
  - 403: Authenticated but lacks permission
  - 503: Missing/bad integration config (e.g., Autotask API key not set)
  - 500: Runtime error (query failed, external API timeout, etc.)
- **Analyzer pipeline** — Cost ceiling guard at $2.00; above that, Stage 4 (Opus) skipped, analysis flagged with `needs_review=true`
- **Sync services** — Incremental sync errors log + re-trigger on next schedule; full sync errors persist `last_error` to `sync_schedules` table
- **Background workers** — Stale jobs reset on worker boot (no recovery); exponential backoff on empty polls; errors logged with job ID for manual inspection

## Cross-Cutting Concerns

**Logging:** Console (stdout) in all services; elevated to syslog or Datadog in production. Analyzer logs all LLM calls + cost to `analyzer_cost_audit` table for billing reconciliation.

**Validation:** Explicit checks in route handlers where it matters (e.g., device ID exists before RMM execute); no centralized validation framework. Zod used for auth/admin forms only.

**Authentication:** Better Auth session stored in Postgres; cookie `Auth` + `Auth.Secure` sent on all requests. `middleware.ts` verifies session cookie exists for authenticated pages. Role checks happen in route handlers.

**Authorization:** Per-resource permissions defined in `lib/permissions.ts` (tickets, configItems, admin, users, roles, auditLog, settings, itglue, rmm). API routes call `requirePermission(resource, action)` to enforce. UI hides links based on `session.user.role`.

**Rate Limiting:** RMM execute endpoint has per-user limit (50 scripts / 24h via `lib/services/rate-limiter.ts`). No global rate limiter.

**Integration Health:** `lib/services/integration-health.ts` polls each integration's health (e.g., Autotask token expiry, last sync age); stores status in `integration_health` table; fires alerts if integration is down or sync is stale.

---

*Architecture analysis: 2026-05-03*