diff --git a/.planning/phases/15-data-model-detection-ticket-evidence/15-PATTERNS.md b/.planning/phases/15-data-model-detection-ticket-evidence/15-PATTERNS.md new file mode 100644 index 0000000..501f0ee --- /dev/null +++ b/.planning/phases/15-data-model-detection-ticket-evidence/15-PATTERNS.md @@ -0,0 +1,352 @@ +# Phase 15: Data Model, Detection & Ticket Evidence - Pattern Map + +**Mapped:** 2026-07-15 +**Files analyzed:** 5 (1 migration, 1 detector service, 1 webhook hook-in, 1 scheduler registration, 1 migration for the schedule seed) +**Analogs found:** 5 / 5 + +## File Classification + +| New/Modified File | Role | Data Flow | Closest Analog | Match Quality | +|-------------------|------|-----------|-----------------|---------------| +| `migrations/097_phishing_triage_schema.sql` | migration | batch (DDL) | `migrations/091_pax8_tables.sql` | exact (multi-table schema-only migration, `raw_payload`/`synced_at`/soft-delete convention) | +| `lib/services/phishing-detector.ts` (new) | service | event-driven + batch (shared core, two callers) | `lib/services/robotic-classifier.ts` (matching logic) + `lib/services/ticket-reconciliation-service.ts` (sweep/report-shape) | role-match (classifier = pattern-match core; reconciliation-service = sweep/report shape) | +| `lib/services/webhook-service.ts` (modified) | service / event hook | event-driven | same file, `triggerWorkflowEngine` fire-and-forget block (lines 112-117, 398-420) | exact (in-file precedent to copy verbatim) | +| `lib/services/sync-scheduler.ts` (modified) | service / scheduler registration | batch (cron) | `pax8-daily` branch (lines 464-479) + `tickets-reconcile` branch (lines 458-463) + `ScheduleConfig.sync_type` union (line 25) + `defaultSchedules` array entries (lines 294-301) | exact | +| `migrations/098_phishing_sweep_schedule.sql` (new, seed row) | migration | batch (DDL seed) | `migrations/096_pax8_daily_schedule.sql` and `migrations/090_ticket_reconcile_schedule.sql` | exact | +| `lib/services/analyzer/preprocessor.ts` (`computeContentHash`, read-only reference) | utility | transform | N/A — this *is* the analog, not a file being modified | exact (content_hash pattern to mirror, not touch) | + +## Pattern Assignments + +### `migrations/097_phishing_triage_schema.sql` (migration, batch) + +**Analog:** `migrations/091_pax8_tables.sql` (schema-only, multi-table, "lays down full schema, later phases populate" precedent) + `migrations/025_*` `ticket_notes` table (audit-column convention) + `migrations/001_create_tables.sql` `tickets` table (columns available for matching: `title`, `description`, `company_id`, `last_activity_date`). + +**Header comment pattern** (`migrations/091_pax8_tables.sql` lines 1-18): +```sql +-- PAX8 integration — Postgres schema. +-- +-- Lays down the full PAX8 schema Phases 11-14 will populate and consume. +-- Schema-only migration (Phase 10) — no sync logic, no matching logic yet. +-- +-- • Client companies -> pax8_companies +-- • Active seat/license subscriptions -> pax8_subscriptions +-- ... +-- NOTE on pax8_orders / pax8_order_items: table names stay "orders" per the +-- D-01 header/line-item design decision, but the column shapes below carry +-- ... +``` +Copy this "here's the full future schema, only some populated now" framing verbatim for the 7 phishing tables (`campaigns`, `reports`, `messages`, `indicators`, `classifications`, `remediation_actions`, `audit_events`) — call out explicitly in the migration header which columns Phase 15 actually populates (`reports` + FKs) vs. which are stubs for Phases 16-21. + +**Table DDL shape** (`migrations/091_pax8_tables.sql` lines 27-40): +```sql +CREATE TABLE IF NOT EXISTS pax8_companies ( + id UUID PRIMARY KEY, + name TEXT NOT NULL, + external_id TEXT, + ... + raw_payload JSONB, + synced_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + is_deleted BOOLEAN NOT NULL DEFAULT false, + deleted_at TIMESTAMPTZ +); + +CREATE INDEX IF NOT EXISTS idx_pax8_companies_is_deleted ON pax8_companies(is_deleted); +``` +Apply the same `IF NOT EXISTS` + trailing `CREATE INDEX IF NOT EXISTS` convention per table. For `reports`, follow CLAUDE.md's snake_case + audit-column convention (`created_at`, `updated_at`, `synced_at`, `is_deleted`, `deleted_at` where relevant) — this schema is **new** data (not synced from Autotask), so `synced_at`/`is_deleted` may not all apply; use `created_at`/`updated_at` at minimum, and a `content_hash` column (TEXT) + `ticket_id BIGINT NOT NULL REFERENCES tickets(id)` FK for D-04's idempotency key, mirroring `analyzer_analyses.content_hash_at_analysis` (see below). + +**`tickets` table columns available for matching** (`migrations/001_create_tables.sql` lines 170-176): +```sql +CREATE TABLE IF NOT EXISTS tickets ( + id BIGINT PRIMARY KEY, + company_id BIGINT NOT NULL, + ticket_number VARCHAR(100), + title VARCHAR(255), + description TEXT, + status INTEGER, + ... + last_activity_date TIMESTAMP, +``` +No new columns needed on `tickets` per CONTEXT.md D-03 — the matcher reads `title` + `description` directly. + +**FK dependency shape** — mirror `lib/types/sync.ts` line 179: `[EntityType.TICKET_NOTES]: [EntityType.TICKETS]`. The new `reports` table's FK to `tickets(id)` (and `companies` transitively via `tickets.company_id`) is the same dependency shape, though this is a Postgres FK, not a sync-order graph entry (no `EntityType` needed since `reports` isn't Autotask-synced). + +--- + +### `lib/services/phishing-detector.ts` (new service — detector core, event-driven + batch) + +**Analog A (pattern-matching engine):** `lib/services/robotic-classifier.ts` +**Analog B (sweep/report shape + idempotent re-fetch loop):** `lib/services/ticket-reconciliation-service.ts` +**Analog C (content-hash idempotency):** `lib/services/analyzer/preprocessor.ts` (`computeContentHash`) + `lib/services/analyzer/persistence.ts` (`findExistingAnalysisByContentHash`) + +**Core matching pattern** — the locked substring/pattern list from DETECT-01 is closer to `robotic-classifier.ts`'s `evaluateContains` than a DB-driven rule table (no admin UI or `classification_rules`-style table is in scope for Phase 15). Copy the *shape* of case-insensitive substring matching, not the DB-rules-cache infrastructure: + +```typescript +// lib/services/robotic-classifier.ts lines 206-219 +private evaluateContains( + fieldValue: string | number, + matchValue: any, + caseSensitive: boolean +): boolean { + const text = String(fieldValue); + const searchText = caseSensitive ? text : text.toLowerCase(); + const patterns = Array.isArray(matchValue) ? matchValue : [matchValue]; + + return patterns.some((pattern: string) => { + const searchPattern = caseSensitive ? String(pattern) : String(pattern).toLowerCase(); + return searchText.includes(searchPattern); + }); +} +``` +And the combined-field target pattern (lines 151-153): +```typescript +case 'title_or_description': + // Return combined text for pattern matching + return [ticket.title, ticket.description].filter(Boolean).join(' ') || null; +``` +Use a `const KNOWN_PHISHING_PATTERNS = [...]` module-level constant (all 8 locked strings from DETECT-01/CONTEXT.md specifics section), matched case-insensitively against `title + ' ' + description`. + +**Content-hash idempotency (D-04)** — mirror `computeContentHash` shape exactly, but hash only `title`+`description` (not the full analyzer event list): + +```typescript +// lib/services/analyzer/preprocessor.ts lines 233-266 +function canonicalize(value: unknown): unknown { /* sorted-key JSON for determinism */ } + +export function computeContentHash( + events: TaggedEvent[], + ticketStatus: number, + ticketPriority: number, + queueId: number | null +): string { + const canonical = JSON.stringify(canonicalize({ events, status: ticketStatus, priority: ticketPriority, queue: queueId })); + return createHash('sha256').update(canonical).digest('hex'); +} +``` +Adapt to: `computeContentHash(title: string, description: string | null): string` — sha256 over `JSON.stringify({ title, description })` (or simpler: `createHash('sha256').update(`${title}\n${description ?? ''}`).digest('hex')`). Store on the `reports` row (e.g. `content_hash TEXT NOT NULL`). + +**Idempotency check** — mirror `findExistingAnalysisByContentHash` (`lib/services/analyzer/persistence.ts` lines 83-102): +```typescript +export async function findExistingAnalysisByContentHash( + ticketNumber: string, + contentHash: string, + provider: 'anthropic' | 'openrouter' = 'anthropic' +): Promise<{ id: string; analysis_version: number } | null> { + const res = await postgresClient.query<{ id: string; analysis_version: string }>( + `SELECT id::text AS id, analysis_version::text AS analysis_version + FROM analyzer_analyses + WHERE ticket_number = $1 AND content_hash_at_analysis = $2 AND provider = $3 AND status = 'complete' + ORDER BY analysis_version DESC LIMIT 1`, + [ticketNumber, contentHash, provider] + ); + if (res.rowCount === 0) return null; + return { id: res.rows[0].id, analysis_version: Number(res.rows[0].analysis_version) }; +} +``` +Adapt to a `SELECT id FROM reports WHERE ticket_id = $1 AND content_hash = $2` check before insert — skip reprocessing (per D-04, `last_activity_date` bumps alone must NOT trigger reprocessing since they don't change `content_hash`). + +**Sweep-loop shape (for the cron reconciliation path)** — mirror `reconcileStaleTickets`'s scan → per-row try/catch → aggregate-result shape (`lib/services/ticket-reconciliation-service.ts` lines 22-151): +```typescript +export interface ReconcileResult { + scanned: number; + updated: number; + statusFlippedToComplete: number; + softDeleted: number; + errors: number; +} + +export async function reconcileStaleTickets(): Promise { + const logger = createSyncLogger({ component: 'TicketReconciliation' }); + ... + const stale = await postgresClient.query<{ id: string; status: number | null }>(staleQuery, [COMPLETE_STATUS, SCAN_LIMIT]); + for (const row of stale.rows) { + try { + ... + result.updated += 1; + } catch (err) { + result.errors += 1; + logger.warn(`Reconcile failed for ticket ${ticketId}`, { ticketId }, err instanceof Error ? err : new Error(String(err))); + } + } + logger.info(`Reconciliation complete: scanned=... updated=...`, { duration: Date.now() - startedAt }); + return result; +} +``` +Adapt: query recently-modified tickets (`WHERE last_activity_date > NOW() - INTERVAL '...'` or similar bounded window) instead of stale-open ones, and call the **same shared detector function** used by the webhook path per row (CONTEXT.md discretion note: "no duplicated pattern-matching logic"). Return a `{ scanned, flagged, skippedUnchanged, errors }`-shaped result object for the scheduler's `console.log` summary line (matches the `tickets-reconcile` log line shape at `sync-scheduler.ts` lines 461-463). + +**Recommended shared-core shape**: one exported `detectPhishingTicket(ticket: { id, ticket_number, title, description, company_id }): Promise<{ flagged: boolean; reportId?: string }>` called both from `webhook-service.ts` (single ticket, fire-and-forget) and from a new `phishing-sweep-service.ts`-style sweep function (loop calling the same core), consistent with `roboticClassifier.classify()` being a single entry point reused across callers. + +**EVID-01 evidence capture** — for notes/time-entries, follow the existing per-ticket join pattern used by the analyzer's `RawTicketBundle` (bundles `ticket`, `notes`, `time_entries` — see `preprocessTicket(bundle: RawTicketBundle)` at `lib/services/analyzer/preprocessor.ts` line 272) and the plain SQL joins already available: `time_entries.ticket_id`, `ticket_notes.ticket_id` (both existing FKs, migration `006` and `025`). Attachment metadata: call the existing `AutotaskClient.getAttachments('Tickets', ticketId)` (`lib/services/autotask-client.ts` lines 424-436) which returns `Attachment[]` — persist metadata fields only (`fullPath`, `title`, `contentType`), not `data` (per CONTEXT.md, base64 content fetch is Phase 16's concern). + +--- + +### `lib/services/webhook-service.ts` (modified — event hook, event-driven) + +**Analog:** same file's existing `ticket.created` → workflow-engine trigger (this is an in-file precedent, not a separate file). + +**Fire-and-forget hook point** (lines 112-117): +```typescript +// Trigger workflow engine for new tickets (fire-and-forget) +if (payload.entityType === WebhookEntityType.TICKETS && payload.eventType === WebhookEventType.CREATE) { + this.triggerWorkflowEngine(payload).catch(err => + console.error('[WEBHOOK] Workflow engine error:', err) + ); +} +``` +Add a second, identically-shaped fire-and-forget block immediately after (or folded into the same `if`), calling the new detector, e.g.: +```typescript +if (payload.entityType === WebhookEntityType.TICKETS && payload.eventType === WebhookEventType.CREATE) { + this.triggerWorkflowEngine(payload).catch(err => console.error('[WEBHOOK] Workflow engine error:', err)); + triggerPhishingDetection(payload).catch(err => console.error('[WEBHOOK] Phishing detection error:', err)); +} +``` + +**Building typed ticket data from the webhook payload** (`triggerWorkflowEngine`, lines 398-420): +```typescript +private async triggerWorkflowEngine(payload: AutotaskWebhookPayload): Promise { + const event: WorkflowEvent = { + trigger_event: 'ticket.created', + entity_type: 'ticket', + entity_id: payload.entityId, + ticket_number: payload.fields?.ticketNumber || undefined, + }; + + // If the webhook payload includes the full entity, build TicketData from it + if (payload.entity) { + event.ticket_data = { + id: payload.entityId, + ticket_number: payload.entity.ticketNumber || null, + title: payload.entity.title || '', + description: payload.entity.description || null, + ... +``` +Copy this "prefer the inline `payload.entity` if present, else re-fetch by id" shape for constructing the detector's input (title/description/company_id) — don't assume the webhook always carries the full entity. + +**Import convention** — add the detector import alongside the existing ones at the top of the file (lines 13-16): +```typescript +import { workflowEngine } from './workflow-engine'; +import { ticketWorkflowEngine } from './ticket-workflow-engine'; +import '../services/workflow-steps'; // Register all workflow step executors +import { WorkflowEvent, TicketData } from '../types/workflow'; +``` + +--- + +### `lib/services/sync-scheduler.ts` (modified — cron registration, batch) + +**Analog:** `pax8-daily` schedule (registration array entry, dispatch branch) + `tickets-reconcile` schedule (disabled-by-default, simple dispatch, no external-config gate). + +**`sync_type` union extension** (line 25): +```typescript +sync_type: 'incremental' | 'full' | ... | 'tickets-reconcile' | 'pax8-daily'; +``` +Add `| 'phishing-sweep'` (or CLAUDE.md-consistent name per Claude's Discretion in CONTEXT.md) to this union. + +**Default schedule entry** (`tickets-reconcile`, lines 294-301 — closest shape: no external integration config needed, disabled by default): +```typescript +{ + id: 'tickets-reconcile', + name: 'Tickets Reconciliation', + description: 'STOPGAP backstop: re-fetches stale-open tickets (>7d since last sync) from Autotask and reconciles status / soft-deletes missing rows. Runs daily at 4:30 AM. Capped at 500 tickets per run.', + cron_expression: '30 4 * * *', + sync_type: 'tickets-reconcile', + is_enabled: false, +}, +``` +Copy this shape for the phishing sweep entry (new `id`, description mentioning the reconciliation purpose, `is_enabled: false` initially unless CONTEXT.md/planner decides otherwise — precedent leans disabled-by-default for new schedules pending admin opt-in). + +**Dispatch branch** (`tickets-reconcile`, lines 458-463 — simplest analog, no config/feature-flag gate needed): +```typescript +} else if (config.sync_type === 'tickets-reconcile') { + const { reconcileStaleTickets } = await import('@/lib/services/ticket-reconciliation-service'); + const result = await reconcileStaleTickets(); + console.log( + `[SCHEDULER] tickets-reconcile: scanned=${result.scanned} updated=${result.updated} flippedComplete=${result.statusFlippedToComplete} softDeleted=${result.softDeleted} errors=${result.errors}` + ); +} +``` +Add an `else if (config.sync_type === 'phishing-sweep')` branch with the same dynamic-import + result-log shape. Dynamic `await import(...)` inside the dispatch branch is the established convention (avoids eager side-effect imports per CLAUDE.md's "don't eager-import workers/schedulers from hot paths" warning) — every branch in this switch uses it (`pax8-daily`, `appgate-*`, `tickets-reconcile`, `integration-health` all do `await import('@/lib/services/...')` inline). + +**Config-gated variant** (`pax8-daily`, lines 464-479 — only needed if the detector should be admin-disableable via `/admin/integrations`; likely NOT needed here since there's no external integration to gate, but shown in case the planner wants an `is_enabled` toggle beyond the schedule row itself): +```typescript +} else if (config.sync_type === 'pax8-daily') { + const { isPax8Configured } = await import('@/lib/services/pax8-factory'); + if (!isPax8Configured()) { + console.log('[SCHEDULER] Skipping pax8-daily — PAX8 not configured'); + } else { + const disabledRes = await postgresClient.query<{ disabled: boolean }>( + `SELECT disabled FROM integration_settings WHERE key = 'pax8'` + ); + ... + } +} +``` +Not recommended as the primary analog (no external credential to check for phishing detection — it's pure local pattern-matching over already-synced tickets) — use the simpler `tickets-reconcile` shape instead. + +--- + +### `migrations/098_phishing_sweep_schedule.sql` (new — schedule seed row) + +**Analog:** `migrations/096_pax8_daily_schedule.sql` (NOT EXISTS-by-name guard) and `migrations/090_ticket_reconcile_schedule.sql` (ON CONFLICT (id) guard) — both cover the "existing installs won't get `createDefaultSchedules()`'s seed" gap. + +```sql +-- Migration 096 pattern (NOT EXISTS by name): +INSERT INTO sync_schedules (id, name, description, cron_expression, sync_type, is_enabled) +SELECT 'pax8-daily', 'PAX8 Daily Sync', '...', '0 4 * * *', 'pax8-daily', false + WHERE NOT EXISTS (SELECT 1 FROM sync_schedules WHERE name = 'PAX8 Daily Sync'); +``` +```sql +-- Migration 090 pattern (ON CONFLICT (id) — needs a unique/PK constraint on id, confirm sync_schedules.id is PK): +INSERT INTO sync_schedules (id, name, description, cron_expression, sync_type, years_back, is_enabled) +VALUES ('tickets-reconcile', 'Tickets Reconciliation', '...', '30 4 * * *', 'tickets-reconcile', NULL, false) +ON CONFLICT (id) DO NOTHING; +``` +Either guard style is acceptable per precedent (both exist in the codebase); prefer `ON CONFLICT (id) DO NOTHING` (migration 090's style) since `id` is the natural conflict target and is simpler than the `NOT EXISTS`-by-name workaround migration 096 needed. + +--- + +## Shared Patterns + +### Fire-and-forget background trigger from a synchronous handler +**Source:** `lib/services/webhook-service.ts` lines 112-117 +**Apply to:** the new webhook-hook-in for phishing detection +```typescript +this.triggerWorkflowEngine(payload).catch(err => + console.error('[WEBHOOK] Workflow engine error:', err) +); +``` +Never `await` the detector inline in the webhook request path — matches the existing workflow-engine precedent exactly (webhook response returns before detection completes). + +### Content-hash idempotency +**Source:** `lib/services/analyzer/preprocessor.ts` (`computeContentHash`, lines 251-266) + `lib/services/analyzer/persistence.ts` (`findExistingAnalysisByContentHash`, lines 83-102) +**Apply to:** `phishing-detector.ts`'s reprocessing guard (D-04) +- Hash only the fields relevant to re-triggering (title + description) — do NOT include `last_activity_date`, `status`, or other bump-prone fields in the hash input. +- Check-before-insert against the stored hash on `reports`, skip if unchanged. + +### Dynamic import inside scheduler dispatch branches +**Source:** `lib/services/sync-scheduler.ts` (every `else if (config.sync_type === ...)` branch, e.g. lines 458-479) +**Apply to:** the new `phishing-sweep` dispatch branch +```typescript +const { someFn } = await import('@/lib/services/some-service'); +``` +Required per CLAUDE.md's "don't eager-import workers/schedulers from hot paths" — the scheduler file itself is imported eagerly by `sync-scheduler.ts`'s own self-init side effect, so each service it dispatches to is imported lazily inside the branch, not at module top. + +### Schema-only migration lands ahead of full population +**Source:** `migrations/091_pax8_tables.sql` header comment +**Apply to:** `migrations/097_phishing_triage_schema.sql` +State explicitly in the migration header which tables/columns Phase 15 populates (`reports`, minimally) vs. which are stubs for later phases (`campaigns`, `messages`, `indicators`, `classifications`, `remediation_actions`, `audit_events`) — this is the established way this codebase documents "future phases populate this" intent inline in SQL comments. + +### Migration seed row for existing installs (schedule tables only apply defaults on a virgin table) +**Source:** `migrations/090_ticket_reconcile_schedule.sql`, `migrations/096_pax8_daily_schedule.sql` +**Apply to:** `migrations/098_phishing_sweep_schedule.sql` +`sync_scheduler.createDefaultSchedules()` (`lib/services/sync-scheduler.ts` lines 168-173) only seeds when `sync_schedules` is empty — any new schedule row needs both (a) an entry in the `defaultSchedules` array for fresh installs, AND (b) a guarded `INSERT` migration for existing installs, per this established two-part pattern. + +## No Analog Found + +| File | Role | Data Flow | Reason | +|------|------|-----------|--------| +| N/A | — | — | All files this phase touches have a strong existing analog in the codebase (workflow-engine trigger, PAX8/tickets-reconcile schedule pattern, robotic-classifier pattern matching, analyzer content_hash). No gap requiring RESEARCH.md-only guidance. | + +## Metadata + +**Analog search scope:** `lib/services/`, `lib/services/analyzer/`, `migrations/`, `lib/types/` +**Files scanned:** `lib/services/entity-sync.ts`, `lib/services/webhook-service.ts`, `lib/services/sync-scheduler.ts`, `lib/services/ticket-reconciliation-service.ts`, `lib/services/robotic-classifier.ts`, `lib/services/analyzer/preprocessor.ts`, `lib/services/analyzer/persistence.ts`, `lib/services/autotask-client.ts`, `lib/types/workflow.ts`, `lib/types/sync.ts`, `migrations/001_create_tables.sql`, `migrations/006_add_time_entries_table.sql`, `migrations/025_*` (ticket_notes), `migrations/090_ticket_reconcile_schedule.sql`, `migrations/091_pax8_tables.sql`, `migrations/096_pax8_daily_schedule.sql` +**Pattern extraction date:** 2026-07-15