wulf-pulse/.planning/codebase/ARCHITECTURE.md

Architecture

Analysis Date: 2026-05-03

Pattern Overview

Overall: Single-instance Next.js 16 backend with in-process background workers (no external job queue).

Key Characteristics:

• App Router pages ('use client') fetch data via API routes using fetch()
• Three auto-starting background workers via side-effect imports (sync-scheduler, analyzer, RMM overshell)
• Service layer in lib/services/ with factories + configuration helpers
• Postgres 16 as system of record (snake_case columns), Redis for caching only
• Manual transformation from snake_case DB columns to camelCase API responses
• Better Auth 1.4 for sessions + magic link + TOTP 2FA + Microsoft OAuth

Layers

Route Layer (HTTP entry):

• Purpose: Accept HTTP requests, validate auth, delegate to services, return JSON responses
• Location: app/api/*/route.ts, app/*/page.tsx
• Contains: Next.js route handlers (GET/POST/PATCH/DELETE), page components
• Depends on: Auth via lib/auth-utils.ts, services via lib/services/
• Used by: Client-side fetch calls from UI components

Service Layer (business logic):

• Purpose: Sync data from external APIs, run background jobs, execute workflows, store results in Postgres
• Location: lib/services/*.ts (~50 files) plus subdirectories (analyzer/, rmm/, llm/, b2/)
• Contains: Integration clients (autotask, datto-rmm, itglue, veeam, msgraph, etc.), sync logic, job executors, pipeline orchestration
• Depends on: Postgres client, external API clients, environment configuration
• Used by: API routes (sync endpoints, webhook handlers) and background workers

Data Access Layer (Postgres):

• Purpose: Query, upsert, and manage state in Postgres
• Location: lib/services/postgres-client.ts (singleton) + migrations/NNN_*.sql
• Contains: Connection pool, query builder methods (query(), transaction(), upsert(), bulkUpsert()), migration definitions
• Depends on: PostgreSQL 16 connection string from env
• Used by: All services that read/write data

Background Workers (long-running processes):

• Purpose: Poll for work and execute sync/analysis/RMM tasks without blocking HTTP requests
• Location: lib/services/sync-scheduler.ts, lib/services/analyzer/worker.ts, lib/services/rmm/worker.ts
• Contains: node-cron scheduler, polling loops with exponential backoff, state machine handlers
• Depends on: Postgres client, service layer integrations
• Used by: Auto-start on module import (side effects); HTTP routes that need them running will import the module to start them

Component Layer (UI):

• Purpose: Render pages, dialogs, tables, charts, and handle client-side state
• Location: components/ui/ (shadcn primitives), components/*/ (feature-specific), app/*/page.tsx
• Contains: React components, hooks for fetch + state, sonner toasts, recharts visualizations
• Depends on: API routes via fetch(), client auth via Better Auth SDK
• Used by: Next.js pages and other components

Types & Schema (contracts):

• Purpose: Define TypeScript interfaces and database schema
• Location: lib/types/<domain>.ts, migrations/NNN_*.sql
• Contains: Entity types (ticket, company, analysis, rmm_execution, etc.), sync request shapes, API response envelopes
• Depends on: (none — they define contracts)
• Used by: Services, API routes, components

Data Flow

Autotask Webhook → Analyzer Pipeline:

1. Webhook ingress (POST /api/webhooks/autotask) — Public endpoint (no auth), HMAC-verified inside handler via lib/services/webhook-service.ts
2. Enqueue job — If ticket.created, fires webhook handler in lib/services/webhook-service.ts which inserts analyzer_jobs row with status='queued'
3. Worker poll — lib/services/analyzer/worker.ts auto-starts in production; every 2s polls for analyzer_jobs.status='queued', claims one with FOR UPDATE SKIP LOCKED
4. Pipeline orchestration — lib/services/analyzer/pipeline.ts runs 7 stages:
   • Stage 0: Preprocess (filter noise, compute content_hash for idempotency)
   • Stage 1: Triage (Haiku — categorize, extract entities)
   • Stage 2: IT Glue retrieval (if configured, redacted docs only via itglue-search.ts)
   • Stage 3: Deep analysis (Sonnet — summary, gaps, root cause)
   • Stage 4: Deep reasoning (Opus — optional, skipped above $2.00 cost ceiling)
   • Stage 5: Persist to analyzer_analyses + stage execution rows
   • Stage 6: Fingerprint (Haiku — cross-ticket aggregation data)
5. Link-aware bundles — lib/services/analyzer/link-discovery.ts resolves related tickets; members get pending_analyses rows
6. Aggregate reports (optional) — Once all bundle members analyzed, stages/aggregate-reduce.ts fires

Periodic Sync Cadence:

1. Cron trigger — lib/services/sync-scheduler.ts (node-cron singleton) reads sync_schedules table; fires at configured times
2. Entity sync — lib/services/entity-sync.ts per entity type (tickets, companies, resources, etc.), incremental via lastTrackedModificationDateTime when supported
3. Postgres upsert — postgresClient.bulkUpsert() writes batches to DB tables (tickets, companies, etc.)
4. Integration-specific syncs — Datto RMM devices/alerts, IT Glue configs/contacts, Veeam agents/alarms, Engagement data, Zoom, Duo, etc.

RMM Overshell Execution:

1. User trigger (POST /api/rmm/execute) — Admin user picks registered script + device
2. Validation & rate limit — lib/services/rmm/executor.ts validates script ID, resolves device, enforces per-user limit (50 / 24h)
3. Queue insertion — Insert pending row in rmm_executions table, call Datto client.runQuickJob()
4. Worker poll — lib/services/rmm/worker.ts (5s cadence) polls in-flight executions, queries Datto for result status
5. Output parsing — Script's parseOutput() method transforms Datto output; result stored in rmm_executions

LogLift Evidence Ingest:

1. Webhook (POST /api/rmm/loglift/upload) — Public, x-openclaw-key header auth
2. Decompression — Download gzipped JSON from B2, decompress, cap at 100 MB (zip-bomb guard)
3. Storage — Slim summary to loglift_uploads table, full payload to B2 via lib/services/b2/client.ts
4. Auto-audit — Resolve device → Autotask company → IT Glue config; if unique match, fire asset-first audit

IT Glue Write-back:

1. Audit runner (lib/services/analyzer/asset-audit/runner.ts) — Post-analysis, runs LLM audit against IT Glue config/flexible asset
2. Results — Persisted to itglue_audit_logs, linked via itglue_ticket_xrefs
3. Revert — Patch /api/analyzer/itglue/configurations/[id]/revert/[writeId] rolls back changes

State Management:

• HTTP requests: Stateless; session from Better Auth cookie
• Background jobs: State in Postgres (status columns: queued → in_flight → complete / failed)
• Caching: Redis (optional, used for integration health checks, cache duration varies)
• Authorization: Checked in route handlers via requireAuth() / requireAdmin() / requirePermission() from lib/auth-utils.ts

Key Abstractions

Factory Pattern (Integration Clients):

• Purpose: Lazy-load integration clients with configured credentials; provide is<Name>Configured() helper to check env vars
• Examples: lib/services/autotask-factory.ts, lib/services/datto-rmm-factory.ts, lib/services/msgraph-factory.ts
• Pattern: Export getAutotaskClient(), getDattoRmmClient(), getMsgraphClient() with caching; throw if credentials missing; bundled with is<Name>Configured() for upstream checks
• Why: Decouples client initialization from route handlers; allows conditional feature gates per env

PostgresClient Singleton:

• Purpose: Single connection pool for all Postgres queries; auto-lazy-initializes; provides ORM-like query builder
• Examples: postgresClient.query(), postgresClient.transaction(), postgresClient.upsert(), postgresClient.bulkUpsert()
• Pattern: Private constructor, static getInstance(), pool initialized on first use
• Why: Prevents connection leaks; provides consistent interface across ~50 services

Sync Service (Entity-Agnostic):

• Purpose: Incremental/full sync of any entity type from external API → Postgres
• Examples: lib/services/entity-sync.ts (Autotask), lib/services/itglue-sync-service.ts, lib/services/veeam-sync-service.ts
• Pattern: Reads lastTrackedModificationDateTime from last_sync table; queries external API; batches upsert via bulkUpsert()
• Why: Codifies the "last sync timestamp + incremental pull + batch insert" pattern across integrations

Analyzer Pipeline (7 Stages):

• Purpose: Orchestrate multi-stage LLM analysis with fallbacks, cost guards, and persistence
• Examples: lib/services/analyzer/pipeline.ts, lib/services/analyzer/stages/*.ts
• Pattern: Each stage returns structured output (gap analysis, category, cost estimate); cost guard checks ceiling before Stage 4; all stages persisted to analyzer_stage_executions
• Why: Allows cost control (skip expensive Opus above $2.00), idempotency (content_hash), and debugging (inspect each stage's output)

Background Worker Polling Loop:

• Purpose: Auto-start in production, poll for work, claim rows with FOR UPDATE SKIP LOCKED, execute, persist result
• Examples: lib/services/analyzer/worker.ts, lib/services/rmm/worker.ts, lib/services/sync-scheduler.ts
• Pattern: Side-effect import auto-starts on module load; exponential backoff if no work; row locking for multi-instance safety (analyzer), or single-instance gate (sync scheduler)
• Why: Keeps background work out of HTTP request path; analyzer is safe to scale (row locking); sync scheduler should run on one instance only

Better Auth Roles (RBAC):

• Purpose: Define three roles (user, admin, super-admin) with per-resource permissions
• Examples: lib/auth.ts (auth config), lib/permissions.ts (permission matrix), lib/auth-utils.ts (runtime checks)
• Pattern: Route handler calls requireAdmin() / requirePermission(), which decode session and check role; middleware only verifies session cookie exists
• Why: Separates auth (middleware) from authorization (route handler); role-based gates are checked at the point of use

Entry Points

HTTP Pages (Authenticated):

• Location: app/*/page.tsx
• Triggers: Browser navigation to any route except public paths
• Responsibilities: Render page shell with AppNavigation + PageHeader, fetch data from API, render client-side components, handle toast/dialog interactions

HTTP API Routes (Public & Authenticated):

• Location: app/api/*/route.ts
• Triggers: fetch() from client, external webhooks (Autotask, Zabbix, RMM), scheduler HTTP calls
• Responsibilities: Validate auth/webhook signature, delegate to service layer, return JSON response with appropriate status (200, 401, 403, 503, 500)

Webhook Handlers (Public):

• Location: app/api/webhooks/autotask, app/api/zabbix/webhook, app/api/rmm/loglift
• Triggers: External systems (Autotask, Zabbix, OpenClaw) POST events
• Responsibilities: Verify HMAC or custom header, parse event, enqueue jobs or upsert data, return 200 (even on error so Autotask doesn't deactivate)

Sync Endpoints (Public, called by scheduler):

• Location: app/api/sync/*, app/api/datto-rmm/sync, app/api/itglue/sync, app/api/veeam/sync, etc.
• Triggers: lib/services/sync-scheduler.ts fires HTTP POST at configured times
• Responsibilities: Call sync service, update last_sync timestamp, return 200 on success or error message

Background Workers (Auto-starting, in-process):

• Location: lib/services/sync-scheduler.ts (cron), lib/services/analyzer/worker.ts (2s poll), lib/services/rmm/worker.ts (5s poll)
• Triggers: Auto-starts as side effect of module import; runs indefinitely in production
• Responsibilities: Poll for work from DB, claim row, execute, persist result, handle errors + logging

Error Handling

Strategy: Defensive; assume external APIs can fail, return 200 on webhook failures (so Autotask doesn't deactivate), log all errors, flag analyses for human review if cost ceiling exceeded.

Patterns:

• Webhook handlers — Return 200 even if processing fails; log error so ops can investigate via audit log
• API route handlers — try/catch, return NextResponse.json({ error, message }, { status }) with conventions:
  • 401: Session missing or invalid
  • 403: Authenticated but lacks permission
  • 503: Missing/bad integration config (e.g., Autotask API key not set)
  • 500: Runtime error (query failed, external API timeout, etc.)
• Analyzer pipeline — Cost ceiling guard at $2.00; above that, Stage 4 (Opus) skipped, analysis flagged with needs_review=true
• Sync services — Incremental sync errors log + re-trigger on next schedule; full sync errors persist last_error to sync_schedules table
• Background workers — Stale jobs reset on worker boot (no recovery); exponential backoff on empty polls; errors logged with job ID for manual inspection

Cross-Cutting Concerns

Logging: Console (stdout) in all services; elevated to syslog or Datadog in production. Analyzer logs all LLM calls + cost to analyzer_cost_audit table for billing reconciliation.

Validation: Explicit checks in route handlers where it matters (e.g., device ID exists before RMM execute); no centralized validation framework. Zod used for auth/admin forms only.

Authentication: Better Auth session stored in Postgres; cookie Auth + Auth.Secure sent on all requests. middleware.ts verifies session cookie exists for authenticated pages. Role checks happen in route handlers.

Authorization: Per-resource permissions defined in lib/permissions.ts (tickets, configItems, admin, users, roles, auditLog, settings, itglue, rmm). API routes call requirePermission(resource, action) to enforce. UI hides links based on session.user.role.

Rate Limiting: RMM execute endpoint has per-user limit (50 scripts / 24h via lib/services/rate-limiter.ts). No global rate limiter.

Integration Health: lib/services/integration-health.ts polls each integration's health (e.g., Autotask token expiry, last sync age); stores status in integration_health table; fires alerts if integration is down or sync is stale.

---

Architecture analysis: 2026-05-03