docs(15): add pattern map

This commit is contained in:
lorentz 2026-07-15 08:23:34 -04:00
parent c33f52b435
commit d7b8c6b72d

View file

@ -0,0 +1,352 @@
# Phase 15: Data Model, Detection & Ticket Evidence - Pattern Map
**Mapped:** 2026-07-15
**Files analyzed:** 5 (1 migration, 1 detector service, 1 webhook hook-in, 1 scheduler registration, 1 migration for the schedule seed)
**Analogs found:** 5 / 5
## File Classification
| New/Modified File | Role | Data Flow | Closest Analog | Match Quality |
|-------------------|------|-----------|-----------------|---------------|
| `migrations/097_phishing_triage_schema.sql` | migration | batch (DDL) | `migrations/091_pax8_tables.sql` | exact (multi-table schema-only migration, `raw_payload`/`synced_at`/soft-delete convention) |
| `lib/services/phishing-detector.ts` (new) | service | event-driven + batch (shared core, two callers) | `lib/services/robotic-classifier.ts` (matching logic) + `lib/services/ticket-reconciliation-service.ts` (sweep/report-shape) | role-match (classifier = pattern-match core; reconciliation-service = sweep/report shape) |
| `lib/services/webhook-service.ts` (modified) | service / event hook | event-driven | same file, `triggerWorkflowEngine` fire-and-forget block (lines 112-117, 398-420) | exact (in-file precedent to copy verbatim) |
| `lib/services/sync-scheduler.ts` (modified) | service / scheduler registration | batch (cron) | `pax8-daily` branch (lines 464-479) + `tickets-reconcile` branch (lines 458-463) + `ScheduleConfig.sync_type` union (line 25) + `defaultSchedules` array entries (lines 294-301) | exact |
| `migrations/098_phishing_sweep_schedule.sql` (new, seed row) | migration | batch (DDL seed) | `migrations/096_pax8_daily_schedule.sql` and `migrations/090_ticket_reconcile_schedule.sql` | exact |
| `lib/services/analyzer/preprocessor.ts` (`computeContentHash`, read-only reference) | utility | transform | N/A — this *is* the analog, not a file being modified | exact (content_hash pattern to mirror, not touch) |
## Pattern Assignments
### `migrations/097_phishing_triage_schema.sql` (migration, batch)
**Analog:** `migrations/091_pax8_tables.sql` (schema-only, multi-table, "lays down full schema, later phases populate" precedent) + `migrations/025_*` `ticket_notes` table (audit-column convention) + `migrations/001_create_tables.sql` `tickets` table (columns available for matching: `title`, `description`, `company_id`, `last_activity_date`).
**Header comment pattern** (`migrations/091_pax8_tables.sql` lines 1-18):
```sql
-- PAX8 integration — Postgres schema.
--
-- Lays down the full PAX8 schema Phases 11-14 will populate and consume.
-- Schema-only migration (Phase 10) — no sync logic, no matching logic yet.
--
-- • Client companies -> pax8_companies
-- • Active seat/license subscriptions -> pax8_subscriptions
-- ...
-- NOTE on pax8_orders / pax8_order_items: table names stay "orders" per the
-- D-01 header/line-item design decision, but the column shapes below carry
-- ...
```
Copy this "here's the full future schema, only some populated now" framing verbatim for the 7 phishing tables (`campaigns`, `reports`, `messages`, `indicators`, `classifications`, `remediation_actions`, `audit_events`) — call out explicitly in the migration header which columns Phase 15 actually populates (`reports` + FKs) vs. which are stubs for Phases 16-21.
**Table DDL shape** (`migrations/091_pax8_tables.sql` lines 27-40):
```sql
CREATE TABLE IF NOT EXISTS pax8_companies (
id UUID PRIMARY KEY,
name TEXT NOT NULL,
external_id TEXT,
...
raw_payload JSONB,
synced_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
is_deleted BOOLEAN NOT NULL DEFAULT false,
deleted_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_pax8_companies_is_deleted ON pax8_companies(is_deleted);
```
Apply the same `IF NOT EXISTS` + trailing `CREATE INDEX IF NOT EXISTS` convention per table. For `reports`, follow CLAUDE.md's snake_case + audit-column convention (`created_at`, `updated_at`, `synced_at`, `is_deleted`, `deleted_at` where relevant) — this schema is **new** data (not synced from Autotask), so `synced_at`/`is_deleted` may not all apply; use `created_at`/`updated_at` at minimum, and a `content_hash` column (TEXT) + `ticket_id BIGINT NOT NULL REFERENCES tickets(id)` FK for D-04's idempotency key, mirroring `analyzer_analyses.content_hash_at_analysis` (see below).
**`tickets` table columns available for matching** (`migrations/001_create_tables.sql` lines 170-176):
```sql
CREATE TABLE IF NOT EXISTS tickets (
id BIGINT PRIMARY KEY,
company_id BIGINT NOT NULL,
ticket_number VARCHAR(100),
title VARCHAR(255),
description TEXT,
status INTEGER,
...
last_activity_date TIMESTAMP,
```
No new columns needed on `tickets` per CONTEXT.md D-03 — the matcher reads `title` + `description` directly.
**FK dependency shape** — mirror `lib/types/sync.ts` line 179: `[EntityType.TICKET_NOTES]: [EntityType.TICKETS]`. The new `reports` table's FK to `tickets(id)` (and `companies` transitively via `tickets.company_id`) is the same dependency shape, though this is a Postgres FK, not a sync-order graph entry (no `EntityType` needed since `reports` isn't Autotask-synced).
---
### `lib/services/phishing-detector.ts` (new service — detector core, event-driven + batch)
**Analog A (pattern-matching engine):** `lib/services/robotic-classifier.ts`
**Analog B (sweep/report shape + idempotent re-fetch loop):** `lib/services/ticket-reconciliation-service.ts`
**Analog C (content-hash idempotency):** `lib/services/analyzer/preprocessor.ts` (`computeContentHash`) + `lib/services/analyzer/persistence.ts` (`findExistingAnalysisByContentHash`)
**Core matching pattern** — the locked substring/pattern list from DETECT-01 is closer to `robotic-classifier.ts`'s `evaluateContains` than a DB-driven rule table (no admin UI or `classification_rules`-style table is in scope for Phase 15). Copy the *shape* of case-insensitive substring matching, not the DB-rules-cache infrastructure:
```typescript
// lib/services/robotic-classifier.ts lines 206-219
private evaluateContains(
fieldValue: string | number,
matchValue: any,
caseSensitive: boolean
): boolean {
const text = String(fieldValue);
const searchText = caseSensitive ? text : text.toLowerCase();
const patterns = Array.isArray(matchValue) ? matchValue : [matchValue];
return patterns.some((pattern: string) => {
const searchPattern = caseSensitive ? String(pattern) : String(pattern).toLowerCase();
return searchText.includes(searchPattern);
});
}
```
And the combined-field target pattern (lines 151-153):
```typescript
case 'title_or_description':
// Return combined text for pattern matching
return [ticket.title, ticket.description].filter(Boolean).join(' ') || null;
```
Use a `const KNOWN_PHISHING_PATTERNS = [...]` module-level constant (all 8 locked strings from DETECT-01/CONTEXT.md specifics section), matched case-insensitively against `title + ' ' + description`.
**Content-hash idempotency (D-04)** — mirror `computeContentHash` shape exactly, but hash only `title`+`description` (not the full analyzer event list):
```typescript
// lib/services/analyzer/preprocessor.ts lines 233-266
function canonicalize(value: unknown): unknown { /* sorted-key JSON for determinism */ }
export function computeContentHash(
events: TaggedEvent[],
ticketStatus: number,
ticketPriority: number,
queueId: number | null
): string {
const canonical = JSON.stringify(canonicalize({ events, status: ticketStatus, priority: ticketPriority, queue: queueId }));
return createHash('sha256').update(canonical).digest('hex');
}
```
Adapt to: `computeContentHash(title: string, description: string | null): string` — sha256 over `JSON.stringify({ title, description })` (or simpler: `createHash('sha256').update(`${title}\n${description ?? ''}`).digest('hex')`). Store on the `reports` row (e.g. `content_hash TEXT NOT NULL`).
**Idempotency check** — mirror `findExistingAnalysisByContentHash` (`lib/services/analyzer/persistence.ts` lines 83-102):
```typescript
export async function findExistingAnalysisByContentHash(
ticketNumber: string,
contentHash: string,
provider: 'anthropic' | 'openrouter' = 'anthropic'
): Promise<{ id: string; analysis_version: number } | null> {
const res = await postgresClient.query<{ id: string; analysis_version: string }>(
`SELECT id::text AS id, analysis_version::text AS analysis_version
FROM analyzer_analyses
WHERE ticket_number = $1 AND content_hash_at_analysis = $2 AND provider = $3 AND status = 'complete'
ORDER BY analysis_version DESC LIMIT 1`,
[ticketNumber, contentHash, provider]
);
if (res.rowCount === 0) return null;
return { id: res.rows[0].id, analysis_version: Number(res.rows[0].analysis_version) };
}
```
Adapt to a `SELECT id FROM reports WHERE ticket_id = $1 AND content_hash = $2` check before insert — skip reprocessing (per D-04, `last_activity_date` bumps alone must NOT trigger reprocessing since they don't change `content_hash`).
**Sweep-loop shape (for the cron reconciliation path)** — mirror `reconcileStaleTickets`'s scan → per-row try/catch → aggregate-result shape (`lib/services/ticket-reconciliation-service.ts` lines 22-151):
```typescript
export interface ReconcileResult {
scanned: number;
updated: number;
statusFlippedToComplete: number;
softDeleted: number;
errors: number;
}
export async function reconcileStaleTickets(): Promise<ReconcileResult> {
const logger = createSyncLogger({ component: 'TicketReconciliation' });
...
const stale = await postgresClient.query<{ id: string; status: number | null }>(staleQuery, [COMPLETE_STATUS, SCAN_LIMIT]);
for (const row of stale.rows) {
try {
...
result.updated += 1;
} catch (err) {
result.errors += 1;
logger.warn(`Reconcile failed for ticket ${ticketId}`, { ticketId }, err instanceof Error ? err : new Error(String(err)));
}
}
logger.info(`Reconciliation complete: scanned=... updated=...`, { duration: Date.now() - startedAt });
return result;
}
```
Adapt: query recently-modified tickets (`WHERE last_activity_date > NOW() - INTERVAL '...'` or similar bounded window) instead of stale-open ones, and call the **same shared detector function** used by the webhook path per row (CONTEXT.md discretion note: "no duplicated pattern-matching logic"). Return a `{ scanned, flagged, skippedUnchanged, errors }`-shaped result object for the scheduler's `console.log` summary line (matches the `tickets-reconcile` log line shape at `sync-scheduler.ts` lines 461-463).
**Recommended shared-core shape**: one exported `detectPhishingTicket(ticket: { id, ticket_number, title, description, company_id }): Promise<{ flagged: boolean; reportId?: string }>` called both from `webhook-service.ts` (single ticket, fire-and-forget) and from a new `phishing-sweep-service.ts`-style sweep function (loop calling the same core), consistent with `roboticClassifier.classify()` being a single entry point reused across callers.
**EVID-01 evidence capture** — for notes/time-entries, follow the existing per-ticket join pattern used by the analyzer's `RawTicketBundle` (bundles `ticket`, `notes`, `time_entries` — see `preprocessTicket(bundle: RawTicketBundle)` at `lib/services/analyzer/preprocessor.ts` line 272) and the plain SQL joins already available: `time_entries.ticket_id`, `ticket_notes.ticket_id` (both existing FKs, migration `006` and `025`). Attachment metadata: call the existing `AutotaskClient.getAttachments('Tickets', ticketId)` (`lib/services/autotask-client.ts` lines 424-436) which returns `Attachment[]` — persist metadata fields only (`fullPath`, `title`, `contentType`), not `data` (per CONTEXT.md, base64 content fetch is Phase 16's concern).
---
### `lib/services/webhook-service.ts` (modified — event hook, event-driven)
**Analog:** same file's existing `ticket.created` → workflow-engine trigger (this is an in-file precedent, not a separate file).
**Fire-and-forget hook point** (lines 112-117):
```typescript
// Trigger workflow engine for new tickets (fire-and-forget)
if (payload.entityType === WebhookEntityType.TICKETS && payload.eventType === WebhookEventType.CREATE) {
this.triggerWorkflowEngine(payload).catch(err =>
console.error('[WEBHOOK] Workflow engine error:', err)
);
}
```
Add a second, identically-shaped fire-and-forget block immediately after (or folded into the same `if`), calling the new detector, e.g.:
```typescript
if (payload.entityType === WebhookEntityType.TICKETS && payload.eventType === WebhookEventType.CREATE) {
this.triggerWorkflowEngine(payload).catch(err => console.error('[WEBHOOK] Workflow engine error:', err));
triggerPhishingDetection(payload).catch(err => console.error('[WEBHOOK] Phishing detection error:', err));
}
```
**Building typed ticket data from the webhook payload** (`triggerWorkflowEngine`, lines 398-420):
```typescript
private async triggerWorkflowEngine(payload: AutotaskWebhookPayload): Promise<void> {
const event: WorkflowEvent = {
trigger_event: 'ticket.created',
entity_type: 'ticket',
entity_id: payload.entityId,
ticket_number: payload.fields?.ticketNumber || undefined,
};
// If the webhook payload includes the full entity, build TicketData from it
if (payload.entity) {
event.ticket_data = {
id: payload.entityId,
ticket_number: payload.entity.ticketNumber || null,
title: payload.entity.title || '',
description: payload.entity.description || null,
...
```
Copy this "prefer the inline `payload.entity` if present, else re-fetch by id" shape for constructing the detector's input (title/description/company_id) — don't assume the webhook always carries the full entity.
**Import convention** — add the detector import alongside the existing ones at the top of the file (lines 13-16):
```typescript
import { workflowEngine } from './workflow-engine';
import { ticketWorkflowEngine } from './ticket-workflow-engine';
import '../services/workflow-steps'; // Register all workflow step executors
import { WorkflowEvent, TicketData } from '../types/workflow';
```
---
### `lib/services/sync-scheduler.ts` (modified — cron registration, batch)
**Analog:** `pax8-daily` schedule (registration array entry, dispatch branch) + `tickets-reconcile` schedule (disabled-by-default, simple dispatch, no external-config gate).
**`sync_type` union extension** (line 25):
```typescript
sync_type: 'incremental' | 'full' | ... | 'tickets-reconcile' | 'pax8-daily';
```
Add `| 'phishing-sweep'` (or CLAUDE.md-consistent name per Claude's Discretion in CONTEXT.md) to this union.
**Default schedule entry** (`tickets-reconcile`, lines 294-301 — closest shape: no external integration config needed, disabled by default):
```typescript
{
id: 'tickets-reconcile',
name: 'Tickets Reconciliation',
description: 'STOPGAP backstop: re-fetches stale-open tickets (>7d since last sync) from Autotask and reconciles status / soft-deletes missing rows. Runs daily at 4:30 AM. Capped at 500 tickets per run.',
cron_expression: '30 4 * * *',
sync_type: 'tickets-reconcile',
is_enabled: false,
},
```
Copy this shape for the phishing sweep entry (new `id`, description mentioning the reconciliation purpose, `is_enabled: false` initially unless CONTEXT.md/planner decides otherwise — precedent leans disabled-by-default for new schedules pending admin opt-in).
**Dispatch branch** (`tickets-reconcile`, lines 458-463 — simplest analog, no config/feature-flag gate needed):
```typescript
} else if (config.sync_type === 'tickets-reconcile') {
const { reconcileStaleTickets } = await import('@/lib/services/ticket-reconciliation-service');
const result = await reconcileStaleTickets();
console.log(
`[SCHEDULER] tickets-reconcile: scanned=${result.scanned} updated=${result.updated} flippedComplete=${result.statusFlippedToComplete} softDeleted=${result.softDeleted} errors=${result.errors}`
);
}
```
Add an `else if (config.sync_type === 'phishing-sweep')` branch with the same dynamic-import + result-log shape. Dynamic `await import(...)` inside the dispatch branch is the established convention (avoids eager side-effect imports per CLAUDE.md's "don't eager-import workers/schedulers from hot paths" warning) — every branch in this switch uses it (`pax8-daily`, `appgate-*`, `tickets-reconcile`, `integration-health` all do `await import('@/lib/services/...')` inline).
**Config-gated variant** (`pax8-daily`, lines 464-479 — only needed if the detector should be admin-disableable via `/admin/integrations`; likely NOT needed here since there's no external integration to gate, but shown in case the planner wants an `is_enabled` toggle beyond the schedule row itself):
```typescript
} else if (config.sync_type === 'pax8-daily') {
const { isPax8Configured } = await import('@/lib/services/pax8-factory');
if (!isPax8Configured()) {
console.log('[SCHEDULER] Skipping pax8-daily — PAX8 not configured');
} else {
const disabledRes = await postgresClient.query<{ disabled: boolean }>(
`SELECT disabled FROM integration_settings WHERE key = 'pax8'`
);
...
}
}
```
Not recommended as the primary analog (no external credential to check for phishing detection — it's pure local pattern-matching over already-synced tickets) — use the simpler `tickets-reconcile` shape instead.
---
### `migrations/098_phishing_sweep_schedule.sql` (new — schedule seed row)
**Analog:** `migrations/096_pax8_daily_schedule.sql` (NOT EXISTS-by-name guard) and `migrations/090_ticket_reconcile_schedule.sql` (ON CONFLICT (id) guard) — both cover the "existing installs won't get `createDefaultSchedules()`'s seed" gap.
```sql
-- Migration 096 pattern (NOT EXISTS by name):
INSERT INTO sync_schedules (id, name, description, cron_expression, sync_type, is_enabled)
SELECT 'pax8-daily', 'PAX8 Daily Sync', '...', '0 4 * * *', 'pax8-daily', false
WHERE NOT EXISTS (SELECT 1 FROM sync_schedules WHERE name = 'PAX8 Daily Sync');
```
```sql
-- Migration 090 pattern (ON CONFLICT (id) — needs a unique/PK constraint on id, confirm sync_schedules.id is PK):
INSERT INTO sync_schedules (id, name, description, cron_expression, sync_type, years_back, is_enabled)
VALUES ('tickets-reconcile', 'Tickets Reconciliation', '...', '30 4 * * *', 'tickets-reconcile', NULL, false)
ON CONFLICT (id) DO NOTHING;
```
Either guard style is acceptable per precedent (both exist in the codebase); prefer `ON CONFLICT (id) DO NOTHING` (migration 090's style) since `id` is the natural conflict target and is simpler than the `NOT EXISTS`-by-name workaround migration 096 needed.
---
## Shared Patterns
### Fire-and-forget background trigger from a synchronous handler
**Source:** `lib/services/webhook-service.ts` lines 112-117
**Apply to:** the new webhook-hook-in for phishing detection
```typescript
this.triggerWorkflowEngine(payload).catch(err =>
console.error('[WEBHOOK] Workflow engine error:', err)
);
```
Never `await` the detector inline in the webhook request path — matches the existing workflow-engine precedent exactly (webhook response returns before detection completes).
### Content-hash idempotency
**Source:** `lib/services/analyzer/preprocessor.ts` (`computeContentHash`, lines 251-266) + `lib/services/analyzer/persistence.ts` (`findExistingAnalysisByContentHash`, lines 83-102)
**Apply to:** `phishing-detector.ts`'s reprocessing guard (D-04)
- Hash only the fields relevant to re-triggering (title + description) — do NOT include `last_activity_date`, `status`, or other bump-prone fields in the hash input.
- Check-before-insert against the stored hash on `reports`, skip if unchanged.
### Dynamic import inside scheduler dispatch branches
**Source:** `lib/services/sync-scheduler.ts` (every `else if (config.sync_type === ...)` branch, e.g. lines 458-479)
**Apply to:** the new `phishing-sweep` dispatch branch
```typescript
const { someFn } = await import('@/lib/services/some-service');
```
Required per CLAUDE.md's "don't eager-import workers/schedulers from hot paths" — the scheduler file itself is imported eagerly by `sync-scheduler.ts`'s own self-init side effect, so each service it dispatches to is imported lazily inside the branch, not at module top.
### Schema-only migration lands ahead of full population
**Source:** `migrations/091_pax8_tables.sql` header comment
**Apply to:** `migrations/097_phishing_triage_schema.sql`
State explicitly in the migration header which tables/columns Phase 15 populates (`reports`, minimally) vs. which are stubs for later phases (`campaigns`, `messages`, `indicators`, `classifications`, `remediation_actions`, `audit_events`) — this is the established way this codebase documents "future phases populate this" intent inline in SQL comments.
### Migration seed row for existing installs (schedule tables only apply defaults on a virgin table)
**Source:** `migrations/090_ticket_reconcile_schedule.sql`, `migrations/096_pax8_daily_schedule.sql`
**Apply to:** `migrations/098_phishing_sweep_schedule.sql`
`sync_scheduler.createDefaultSchedules()` (`lib/services/sync-scheduler.ts` lines 168-173) only seeds when `sync_schedules` is empty — any new schedule row needs both (a) an entry in the `defaultSchedules` array for fresh installs, AND (b) a guarded `INSERT` migration for existing installs, per this established two-part pattern.
## No Analog Found
| File | Role | Data Flow | Reason |
|------|------|-----------|--------|
| N/A | — | — | All files this phase touches have a strong existing analog in the codebase (workflow-engine trigger, PAX8/tickets-reconcile schedule pattern, robotic-classifier pattern matching, analyzer content_hash). No gap requiring RESEARCH.md-only guidance. |
## Metadata
**Analog search scope:** `lib/services/`, `lib/services/analyzer/`, `migrations/`, `lib/types/`
**Files scanned:** `lib/services/entity-sync.ts`, `lib/services/webhook-service.ts`, `lib/services/sync-scheduler.ts`, `lib/services/ticket-reconciliation-service.ts`, `lib/services/robotic-classifier.ts`, `lib/services/analyzer/preprocessor.ts`, `lib/services/analyzer/persistence.ts`, `lib/services/autotask-client.ts`, `lib/types/workflow.ts`, `lib/types/sync.ts`, `migrations/001_create_tables.sql`, `migrations/006_add_time_entries_table.sql`, `migrations/025_*` (ticket_notes), `migrations/090_ticket_reconcile_schedule.sql`, `migrations/091_pax8_tables.sql`, `migrations/096_pax8_daily_schedule.sql`
**Pattern extraction date:** 2026-07-15