diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 5c9f1e6..4a29b57 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -374,7 +374,10 @@ summarizes classification, blast radius, and recommended/approved remediation st 2. Running the ticket scanner against Autotask/Pulse tickets flags candidates matching the known title/body patterns ("Phishing Report", "Spam Alert", "Phishing Alert - Email Security Report", "KnowBe4 Phish Alert Report", "Source: KnowBe4 Phish Alert Button", "userSubmissionsReportMessage", "reported message destinations", "Microsoft directly") and persists a `reports` row per candidate 3. Re-scanning tickets that haven't changed since last processed does not reprocess or duplicate their `reports` rows; a ticket whose Autotask data changed since last processed IS reprocessed (idempotent on ticket state, not just ticket ID) 4. Each flagged ticket's stored evidence includes ticket ID/number, company, requester/reporter, title, description, notes, relevant time entries, and attachment metadata (EVID-01) -**Plans**: TBD +**Plans**: 3 plans +- [ ] 15-01-PLAN.md — Migration 097: 7-table phishing-triage schema (reports fully designed, others stubbed) (DETECT-01, DETECT-02, EVID-01) +- [ ] 15-02-PLAN.md — phishing-detector.ts core: pattern matcher + content-hash idempotency + EVID-01 evidence capture + reports upsert (DETECT-01, DETECT-02, EVID-01) +- [ ] 15-03-PLAN.md — Wiring: webhook fire-and-forget hook + bounded cron sweep service + scheduler branch + migration 098 seed (DETECT-01, DETECT-02) **UI hint**: no ### Phase 16: EML/MIME Evidence Parser @@ -474,7 +477,7 @@ Phases execute in numeric order. v1.0 (Phases 1-9.1) shipped 2026-07-10. v2.0 (P | 12. Orders/Invoices & Company Matching | v2.0 | 5/5 | Complete | 2026-07-11 | | 13. Scheduler & Admin Toggle | v2.0 | 3/3 | Complete | 2026-07-11 | | 14. /pax8 UI Surface | v2.0 | 6/6 | Complete | 2026-07-12 | -| 15. Data Model, Detection & Ticket Evidence | v3.0 | 0/TBD | Not started | - | +| 15. Data Model, Detection & Ticket Evidence | v3.0 | 0/3 | Not started | - | | 16. EML/MIME Evidence Parser | v3.0 | 0/TBD | Not started | - | | 17. Mimecast Blast Radius Lookup | v3.0 | 0/TBD | Not started | - | | 18. Campaign Grouping & Phishing Analysis API | v3.0 | 0/TBD | Not started | - | diff --git a/.planning/phases/15-data-model-detection-ticket-evidence/15-01-PLAN.md b/.planning/phases/15-data-model-detection-ticket-evidence/15-01-PLAN.md new file mode 100644 index 0000000..2331e8a --- /dev/null +++ b/.planning/phases/15-data-model-detection-ticket-evidence/15-01-PLAN.md @@ -0,0 +1,209 @@ +--- +phase: 15-data-model-detection-ticket-evidence +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - migrations/097_phishing_triage_schema.sql +autonomous: true +requirements: [DETECT-01, DETECT-02, EVID-01] +must_haves: + truths: + - "The 7 phishing-triage tables exist in Postgres and can be queried" + - "A reports row can store a ticket's content_hash, matched patterns, and EVID-01 evidence" + - "Re-running the migration is a no-op (IF NOT EXISTS on every table/index)" + artifacts: + - path: "migrations/097_phishing_triage_schema.sql" + provides: "campaigns, reports, messages, indicators, classifications, remediation_actions, audit_events tables" + contains: "CREATE TABLE IF NOT EXISTS reports" + key_links: + - from: "reports.ticket_id" + to: "tickets.id" + via: "foreign key" + pattern: "REFERENCES tickets\\(id\\)" + - from: "reports.content_hash" + to: "phishing-detector idempotency (Plan 02)" + via: "unique key on ticket_id + stored hash" + pattern: "content_hash" +--- + + +Create migration `097_phishing_triage_schema.sql` — the durable phishing-triage +schema (7 tables) that every v3.0 phase reads and writes. Phase 15 only populates +`reports` (via the Plan 02 detector); `campaigns`, `messages`, `indicators`, +`classifications`, `remediation_actions`, and `audit_events` are laid down now as +stubs so later phases (16-21) have their schema ready and never need a second +foundation migration. + +Purpose: Land the schema before any service writes to it (STATE.md decision: +"durable schema lands in Phase 15, before any service that writes to it"). +Output: One numbered, idempotent, schema-only migration file. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/15-data-model-detection-ticket-evidence/15-CONTEXT.md +@.planning/phases/15-data-model-detection-ticket-evidence/15-PATTERNS.md + + + + +tickets (migrations/001_initial_schema.sql): id BIGINT PK, company_id BIGINT NOT NULL, +ticket_number VARCHAR(100), title VARCHAR(255), description TEXT, contact_id BIGINT, +created_by_contact_id BIGINT, assigned_resource_id BIGINT, last_activity_date TIMESTAMP. + +companies (migrations/001_initial_schema.sql): id BIGINT PK, company_name VARCHAR(255). + +Reference migration for header framing + table/index conventions: +migrations/091_pax8_tables.sql (schema-only, multi-table, IF NOT EXISTS + trailing +CREATE INDEX IF NOT EXISTS per table, "future phases populate this" header comment). + +Postgres 16 has built-in gen_random_uuid() — no extension needed for UUID PKs. + + + + + + + Task 1: Write migration 097 — 7-table phishing triage schema + migrations/097_phishing_triage_schema.sql + + - migrations/091_pax8_tables.sql (header-comment framing + IF NOT EXISTS + per-table CREATE INDEX convention to copy) + - migrations/090_ticket_reconcile_schedule.sql (ON CONFLICT idempotency convention used elsewhere) + - migrations/001_initial_schema.sql (confirm exact tickets/companies column names the FKs reference) + - .planning/phases/15-data-model-detection-ticket-evidence/15-PATTERNS.md (section on migrations/097 — reports column requirements) + - CLAUDE.md (Database + Migrations sections: snake_case columns, audit-column convention, IF NOT EXISTS, next sequential number) + + + Create `migrations/097_phishing_triage_schema.sql`. Open with a header comment + (mirror migrations/091_pax8_tables.sql lines 1-18) stating this is a schema-only + migration for v3.0 phishing triage, that Phase 15 populates only `reports`, and + that `campaigns`/`messages`/`indicators`/`classifications`/`remediation_actions`/ + `audit_events` are stubs populated by Phases 16-21. + + Create tables in this order so FKs resolve (every table `CREATE TABLE IF NOT EXISTS`, + every UUID PK `id UUID PRIMARY KEY DEFAULT gen_random_uuid()`, snake_case columns): + + 1. campaigns — id, campaign_key TEXT, group_method TEXT, first_seen_at TIMESTAMPTZ, + last_seen_at TIMESTAMPTZ, report_count INTEGER NOT NULL DEFAULT 0, status TEXT NOT NULL DEFAULT 'open', + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(). + + 2. reports (the only table Phase 15 populates — design fully): id; + ticket_id BIGINT NOT NULL REFERENCES tickets(id); ticket_number VARCHAR(100); + company_id BIGINT; company_name VARCHAR(255); requester_contact_id BIGINT; + created_by_contact_id BIGINT; title VARCHAR(255); description TEXT; + matched_patterns JSONB NOT NULL DEFAULT '[]'::jsonb (the DETECT-01 pattern strings that matched); + content_hash TEXT NOT NULL (D-04 idempotency key over title+description); + evidence JSONB NOT NULL DEFAULT '{}'::jsonb (EVID-01 notes/time_entries/attachments capture); + campaign_id UUID REFERENCES campaigns(id) (nullable — Phase 18 links it); + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(); updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(). + Add `CONSTRAINT uq_reports_ticket_id UNIQUE (ticket_id)` so the Plan 02 detector + can upsert one report per ticket (ON CONFLICT (ticket_id)) and enforce D-04. + + 3. messages (stub for Phase 16) — id, report_id UUID REFERENCES reports(id), + message_id TEXT, headers JSONB, urls JSONB, attachments JSONB, body_preview TEXT, + raw_ref TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(). + + 4. indicators (stub for Phase 16) — id, message_id UUID REFERENCES messages(id), + indicator_type TEXT NOT NULL, value TEXT NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(). + + 5. classifications (stub for Phase 19) — id, campaign_id UUID REFERENCES campaigns(id), + verdict TEXT, confidence NUMERIC, summary TEXT, reasons JSONB, recommended_actions JSONB, + requires_approval BOOLEAN NOT NULL DEFAULT false, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(). + + 6. remediation_actions (stub for Phase 20) — id, campaign_id UUID REFERENCES campaigns(id), + action_type TEXT, status TEXT NOT NULL DEFAULT 'proposed', params JSONB, + approved_by TEXT, approved_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(). + + 7. audit_events (stub for Phase 20) — id, campaign_id UUID, actor TEXT, event_type TEXT NOT NULL, + payload JSONB, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(). + + After each CREATE TABLE add `CREATE INDEX IF NOT EXISTS` statements: reports on + (ticket_id), (content_hash), (campaign_id); messages on (report_id), (message_id); + indicators on (message_id); classifications on (campaign_id); remediation_actions on + (campaign_id); campaigns on (campaign_key). Match the trailing-index style in + migrations/091_pax8_tables.sql. Do NOT edit any committed migration; this is a new file only. + + + test -f migrations/097_phishing_triage_schema.sql && grep -c "CREATE TABLE IF NOT EXISTS" migrations/097_phishing_triage_schema.sql | grep -qx 7 && echo "7 tables OK" + + + - `migrations/097_phishing_triage_schema.sql` exists + - File contains exactly 7 `CREATE TABLE IF NOT EXISTS` statements: campaigns, reports, messages, indicators, classifications, remediation_actions, audit_events + - `reports` table declares `ticket_id BIGINT NOT NULL REFERENCES tickets(id)`, `content_hash TEXT NOT NULL`, `matched_patterns JSONB`, `evidence JSONB`, and `CONSTRAINT uq_reports_ticket_id UNIQUE (ticket_id)` + - Every index statement uses `CREATE INDEX IF NOT EXISTS` + - No `DROP`, `ALTER ... DROP`, or `TRUNCATE` appears anywhere in the file (grep returns no matches) + + The migration file lays down all 7 tables idempotently with the reports schema needed by Plan 02, and contains no destructive statements. + + + + Task 2: Apply migration 097 to the dev database and verify tables exist + migrations/097_phishing_triage_schema.sql + + - docker-compose.yml (Postgres container name + POSTGRES_USER/POSTGRES_DB values for the psql apply command) + - CLAUDE.md (Migrations section: "Postgres init applies migrations on first boot only — for an existing DB, run via scripts/apply-migrations; check first, behavior varies") + - MEMORY note: existing DB volumes do NOT auto-apply new migrations; apply manually via `docker exec pulse-postgres psql ...` + + + Apply migration 097 to the running dev Postgres. First check for a project apply + helper (`ls scripts/apply-migrations*`); if one exists and matches convention, use it. + Otherwise pipe the file into the container's psql: read the Postgres container name + and POSTGRES_USER / POSTGRES_DB from docker-compose.yml, then run the migration with + `docker exec -i psql -U -d ` fed from + migrations/097_phishing_triage_schema.sql. Because the migration is fully IF NOT EXISTS, + re-applying it must be safe. Do NOT drop or recreate any existing table. + + + docker exec -i "$(grep -oiE 'container_name:\s*\S*postgres\S*' docker-compose.yml | head -1 | awk '{print $2}')" psql -U "${POSTGRES_USER:-postgres}" -d "${POSTGRES_DB:-pulse}" -tAc "SELECT count(*) FROM information_schema.tables WHERE table_name IN ('campaigns','reports','messages','indicators','classifications','remediation_actions','audit_events');" | grep -qx 7 && echo "all 7 tables present" + + + - Querying `information_schema.tables` returns all 7 phishing-triage table names present in the dev DB + - `\d reports` (or an information_schema.columns query) shows columns `ticket_id`, `content_hash`, `matched_patterns`, `evidence`, and the unique constraint on `ticket_id` + - Re-running the migration produces no error (IF NOT EXISTS makes it idempotent) + + All 7 tables exist in the dev database; the reports table has the columns and unique constraint Plan 02 depends on. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| migration → Postgres DDL | Schema definition applied to the system-of-record DB | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-15-01 | Tampering | migration DDL against existing tables | mitigate | Schema-only, additive, every statement IF NOT EXISTS; no DROP/ALTER-DROP/TRUNCATE (enforced by acceptance criterion grep). New FKs reference existing tickets/companies only | +| T-15-02 | Denial of Service | re-applying migration on an existing volume | mitigate | Idempotent IF NOT EXISTS on tables and indexes — safe to re-run, no data loss | +| T-15-03 | Information Disclosure | reports/evidence tables hold ticket content | accept | No new external access path in this phase; standard DB access controls apply; the /api/phishing/* auth gate (ACCESS-01) lands in Phase 18 before any read surface exists | +| T-15-SC | Tampering | package installs | accept | No npm/pip/cargo installs in this plan — SQL-only migration, no new dependencies | + + + +- `grep -c "CREATE TABLE IF NOT EXISTS" migrations/097_phishing_triage_schema.sql` returns 7 +- No destructive DDL: `grep -iE "DROP|TRUNCATE" migrations/097_phishing_triage_schema.sql` returns nothing +- Dev DB shows all 7 tables + reports unique constraint on ticket_id + + + +Migration 097 creates all 7 phishing-triage tables with IF NOT EXISTS (ROADMAP SC#1), +the reports table is fully designed to store content_hash + matched patterns + EVID-01 +evidence, and the migration is applied and verified in the dev database. + + + +Create `.planning/phases/15-data-model-detection-ticket-evidence/15-01-SUMMARY.md` when done. + diff --git a/.planning/phases/15-data-model-detection-ticket-evidence/15-02-PLAN.md b/.planning/phases/15-data-model-detection-ticket-evidence/15-02-PLAN.md new file mode 100644 index 0000000..75358bd --- /dev/null +++ b/.planning/phases/15-data-model-detection-ticket-evidence/15-02-PLAN.md @@ -0,0 +1,253 @@ +--- +phase: 15-data-model-detection-ticket-evidence +plan: 02 +type: execute +wave: 2 +depends_on: ["15-01"] +files_modified: + - lib/services/phishing-detector.ts + - lib/services/phishing-detector.test.ts +autonomous: true +requirements: [DETECT-01, DETECT-02, EVID-01] +must_haves: + truths: + - "Given a ticket whose title/description contains any of the 8 known patterns, the matcher flags it and reports which patterns matched" + - "Given a ticket with none of the patterns, the matcher does not flag it" + - "The content hash is stable for identical title+description and changes when either changes" + - "Detecting a flagged ticket persists exactly one reports row with EVID-01 evidence" + - "Re-detecting an unchanged ticket does not create or duplicate a reports row; a ticket whose title/description changed IS reprocessed" + artifacts: + - path: "lib/services/phishing-detector.ts" + provides: "KNOWN_PHISHING_PATTERNS, matchesPhishingPatterns, computePhishingContentHash, gatherTicketEvidence, detectPhishingTicket" + min_lines: 120 + - path: "lib/services/phishing-detector.test.ts" + provides: "unit tests for matcher + content-hash covering all 8 patterns, negative case, and hash stability/change" + key_links: + - from: "detectPhishingTicket" + to: "reports table (Plan 01)" + via: "ON CONFLICT (ticket_id) upsert guarded by content_hash comparison" + pattern: "ON CONFLICT \\(ticket_id\\)" + - from: "computePhishingContentHash" + to: "reports.content_hash" + via: "sha256 over title+description (D-04)" + pattern: "createHash\\('sha256'\\)" +--- + + +Build `lib/services/phishing-detector.ts` — the shared detection core called by both +the webhook path and the cron sweep (wired in Plan 03). It matches a ticket's +title+description against the 8 locked DETECT-01 patterns, computes a content hash for +D-04 idempotency, gathers EVID-01 evidence, and upserts a single `reports` row per +candidate ticket — reprocessing only when the content hash changed. + +Purpose: One deterministic, testable detector with no duplicated matching logic +(CONTEXT.md discretion: "both call the same underlying logic"). +Output: The detector service + a unit test file for its pure logic. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/15-data-model-detection-ticket-evidence/15-CONTEXT.md +@.planning/phases/15-data-model-detection-ticket-evidence/15-PATTERNS.md +@.planning/phases/15-data-model-detection-ticket-evidence/15-01-SUMMARY.md + + + + +reports table (Plan 01): id UUID, ticket_id BIGINT NOT NULL UNIQUE, ticket_number, +company_id, company_name, requester_contact_id, created_by_contact_id, title, description, +matched_patterns JSONB, content_hash TEXT NOT NULL, evidence JSONB, campaign_id UUID (null here), +created_at, updated_at. Upsert target: ON CONFLICT (ticket_id). + +postgresClient singleton (lib/services/postgres-client.ts): postgresClient.query(sql, params) +returns { rows, rowCount }. Always use parameterized $1/$2 placeholders. + +Evidence source tables (already synced, join by ticket_id): +- ticket_notes (migration 025): id, ticket_id BIGINT, title, description, note_type INTEGER, creator_resource_id, created_at +- time_entries (migration 006): id, resource_id, ticket_id, entry_date, hours_worked, start_date_time, end_date_time +- companies: id, company_name (join tickets.company_id → companies.id for company_name) + +AutotaskClient.getAttachments(entityName, entityId): Promise +(lib/services/autotask-client.ts:424) — Attachment metadata fields: fullPath, title, contentType. +Persist metadata only (fullPath, title, contentType) — NOT `data` (base64); content fetch is Phase 16. +Instantiate the client the same way ticket-reconciliation-service.ts:34-45 does (env-var config, +lazy singleton), or reuse an existing factory if one is present. + +Pattern-matching analog: lib/services/robotic-classifier.ts:206-219 (evaluateContains — +case-insensitive substring via .toLowerCase() + .includes(); NO regex, NO eval). +Content-hash analog: lib/services/analyzer/preprocessor.ts:251-266 (computeContentHash — +createHash('sha256') over canonical JSON). Import `createHash` from node 'crypto'. + +The 8 locked DETECT-01 patterns (case-insensitive substrings — all must be covered): +"Phishing Report", "Spam Alert", "Phishing Alert - Email Security Report", +"KnowBe4 Phish Alert Report", "Source: KnowBe4 Phish Alert Button", +"userSubmissionsReportMessage", "reported message destinations", "Microsoft directly". + + + + + + + Task 1: Pure detection logic — pattern matcher + content hash (TDD) + lib/services/phishing-detector.ts, lib/services/phishing-detector.test.ts + + - matchesPhishingPatterns("Fwd: Phishing Report", null) → { flagged: true, matched: ["Phishing Report"] } + - matchesPhishingPatterns("Re: order confirmation", "please review invoice") → { flagged: false, matched: [] } + - Matching is case-insensitive: matchesPhishingPatterns("SPAM ALERT from user", null).flagged === true + - Each of the 8 locked patterns individually triggers a flag when present in title OR description + - matched[] contains exactly the patterns that were present (subset of KNOWN_PHISHING_PATTERNS) + - computePhishingContentHash("t","d") === computePhishingContentHash("t","d") (stable) + - computePhishingContentHash("t","d") !== computePhishingContentHash("t2","d") (title change) + - computePhishingContentHash("t","d") !== computePhishingContentHash("t","d2") (description change) + - computePhishingContentHash("t", null) is defined and stable (null description normalizes) + + + - lib/services/robotic-classifier.ts (evaluateContains, lines 206-219 — case-insensitive substring shape to copy) + - lib/services/analyzer/preprocessor.ts (computeContentHash, lines 233-266 — sha256 shape to mirror, hashing only title+description) + - vitest.config.ts (confirm include glob `lib/**/*.test.ts` picks up the new test file) + - .planning/phases/15-data-model-detection-ticket-evidence/15-PATTERNS.md (detector section — matcher + content-hash requirements) + + + Write the tests in `lib/services/phishing-detector.test.ts` FIRST (RED), covering + every behavior listed above — one assertion per locked pattern (all 8), the negative + case, case-insensitivity, and hash stability/change/null-normalization. Run vitest and + confirm the suite fails because the functions don't exist yet. + + Then create `lib/services/phishing-detector.ts` and implement (GREEN): + - `export const KNOWN_PHISHING_PATTERNS: readonly string[]` — the 8 locked strings + exactly as given in the interfaces block above. + - `export function matchesPhishingPatterns(title: string | null, description: string | null): + { flagged: boolean; matched: string[] }` — build a combined `${title ?? ''} ${description ?? ''}` + lowercased haystack, return matched = patterns whose lowercased form is a substring, + flagged = matched.length > 0. Use `.toLowerCase()` + `.includes()` only — NO regex, NO eval + (matches robotic-classifier.evaluateContains). + - `export function computePhishingContentHash(title: string | null, description: string | null): + string` — `createHash('sha256').update(JSON.stringify({ title: title ?? '', description: description ?? '' })).digest('hex')`. + Do NOT include last_activity_date, status, or any bump-prone field (D-04). + + Run vitest again and confirm GREEN. Do not add DB access in this task — pure functions only. + + + npx vitest run lib/services/phishing-detector.test.ts && npx tsc --noEmit --pretty + + + - `lib/services/phishing-detector.test.ts` contains at least 8 assertions covering each locked pattern by name, plus the negative case, case-insensitivity, and hash stability/change + - `npx vitest run lib/services/phishing-detector.test.ts` passes with 0 failures + - `KNOWN_PHISHING_PATTERNS` array has exactly 8 entries matching the locked strings verbatim + - `matchesPhishingPatterns` uses `.toLowerCase()` + `.includes()` (no `RegExp`, no `eval`) — grep confirms no `RegExp(`/`eval(` in the file + - `computePhishingContentHash` uses `createHash('sha256')` and hashes only title+description + - `npx tsc --noEmit --pretty` passes + + Matcher and content-hash are implemented as pure functions, all 8 patterns are covered by passing tests, and type-check is clean. + + + + Task 2: Evidence capture + detectPhishingTicket orchestration with D-04 idempotency + lib/services/phishing-detector.ts + + - lib/services/phishing-detector.ts (current state after Task 1 — extend, don't rewrite) + - lib/services/ticket-reconciliation-service.ts (lazy AutotaskClient singleton pattern lines 34-45; per-row try/catch + result-shape convention) + - lib/services/analyzer/persistence.ts (findExistingAnalysisByContentHash, lines 83-102 — check-before-write idempotency shape to mirror) + - lib/services/autotask-client.ts (getAttachments signature, lines 424-436; Attachment metadata fields) + - lib/services/postgres-client.ts (query/upsert API + parameterized-query convention) + + + Extend `lib/services/phishing-detector.ts` with evidence capture and the shared + orchestration entry point. Add: + + - `interface DetectableTicket { id: number; ticket_number: string | null; title: string | null; + description: string | null; company_id: number | null; contact_id?: number | null; + created_by_contact_id?: number | null }` — the minimal input both callers pass. + + - `async function gatherTicketEvidence(ticket: DetectableTicket): Promise` — + capture EVID-01 evidence into a JSON object: + * company_name: SELECT company_name FROM companies WHERE id = ticket.company_id (null-safe) + * notes: SELECT id, title, description, note_type, creator_resource_id, created_at FROM ticket_notes + WHERE ticket_id = $1 ORDER BY created_at (parameterized) + * time_entries: SELECT id, resource_id, entry_date, hours_worked, start_date_time, end_date_time + FROM time_entries WHERE ticket_id = $1 ORDER BY entry_date (parameterized) + * attachments: call AutotaskClient.getAttachments('Tickets', ticket.id), map to metadata only + ({ fullPath, title, contentType }) — DROP `data`/base64. Wrap the Autotask call in try/catch + so an attachment-API failure degrades to an empty attachments array and logs, never throws out + of detection. Define EvidencePayload with fields: company_name, notes[], time_entries[], attachments[]. + + - `export async function detectPhishingTicket(ticket: DetectableTicket): + Promise<{ flagged: boolean; reportId?: string; skippedUnchanged?: boolean }>`: + 1. const { flagged, matched } = matchesPhishingPatterns(ticket.title, ticket.description); + if (!flagged) return { flagged: false }. + 2. const contentHash = computePhishingContentHash(ticket.title, ticket.description). + 3. SELECT id, content_hash FROM reports WHERE ticket_id = $1 (parameterized). If a row exists and + its content_hash === contentHash, return { flagged: true, skippedUnchanged: true } WITHOUT + re-gathering evidence or writing (D-04 — status/assignee/last_activity bumps don't change the hash). + 4. Otherwise gather evidence, then upsert: + INSERT INTO reports (ticket_id, ticket_number, company_id, company_name, requester_contact_id, + created_by_contact_id, title, description, matched_patterns, content_hash, evidence) + VALUES ($1..$11) ON CONFLICT (ticket_id) DO UPDATE SET ... , updated_at = NOW() + RETURNING id. matched_patterns and evidence bind as JSON (JSON.stringify or ::jsonb cast). + Return { flagged: true, reportId }. + + Do not duplicate the matcher/hash logic — call the Task 1 functions. Use console.error with a + `[PHISHING-DETECT]` prefix for caught errors, matching the project logging convention. + + + npx tsc --noEmit --pretty && npx vitest run lib/services/phishing-detector.test.ts && grep -q "ON CONFLICT (ticket_id)" lib/services/phishing-detector.ts && echo "orchestration OK" + + + - `detectPhishingTicket` is exported and returns `{ flagged, reportId?, skippedUnchanged? }` + - Idempotency branch present: a SELECT of reports by ticket_id compares stored `content_hash` to the freshly computed hash and returns `skippedUnchanged: true` without writing when equal + - The reports upsert uses `ON CONFLICT (ticket_id) DO UPDATE ... updated_at = NOW()` and RETURNING id + - Evidence gathering reads ticket_notes and time_entries via parameterized `$1` queries (no string interpolation of ticket_id) and stores only attachment metadata (fullPath/title/contentType), never base64 `data` + - The Autotask getAttachments call is wrapped so a failure yields an empty attachments array instead of throwing + - `npx tsc --noEmit --pretty` passes and the Task 1 vitest suite still passes + + detectPhishingTicket matches, hashes, checks the content-hash idempotency guard, gathers EVID-01 evidence, and upserts one reports row per candidate — reprocessing only on hash change. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| ticket title/description → matcher + hash | Untrusted ticket content flows into substring match and sha256 | +| ticket content → Postgres reports row | Untrusted content written to DB | +| Autotask Attachments API → evidence | External API metadata captured into evidence JSON | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-15-04 | Tampering (SQL injection) | reports upsert + evidence SELECTs from ticket content | mitigate | All DB access via postgresClient parameterized `$1/$2` placeholders — ticket content is never string-interpolated into SQL (acceptance criterion enforces this) | +| T-15-05 | Elevation / code exec via pattern input | matchesPhishingPatterns over attacker-controlled title | mitigate | Matching is `.toLowerCase()` + `.includes()` only — no `RegExp` construction (no ReDoS), no `eval` (grep-enforced) | +| T-15-06 | Denial of Service | crafted content re-triggering repeated processing | mitigate | D-04 content-hash guard: unchanged content skips before any evidence gathering or write; hash excludes bump-prone fields so status/assignee churn cannot force reprocessing | +| T-15-07 | Server-Side Request Forgery / unsafe fetch | attachment handling | mitigate | Only attachment metadata (fullPath/title/contentType) is stored; the detector never fetches attachment `data`, expands, or executes any URL. Autotask call wrapped in try/catch so it degrades, never blocks | +| T-15-08 | Information Disclosure | evidence JSON holds ticket notes/time entries | accept | No new external read surface in this phase (ACCESS-01 auth gate lands in Phase 18); data already resident in Postgres | +| T-15-SC | Tampering | package installs | accept | No new npm packages — uses built-in `crypto`, existing `pg` client and AutotaskClient. No install task | + + + +- `npx vitest run lib/services/phishing-detector.test.ts` — all 8 pattern cases + negative + hash tests pass +- `npx tsc --noEmit --pretty` clean +- `grep -E "RegExp\(|eval\(" lib/services/phishing-detector.ts` returns nothing +- `grep "ON CONFLICT (ticket_id)" lib/services/phishing-detector.ts` matches +- All reports/notes/time_entries SQL uses `$1`/`$2` parameters (no interpolated ticket_id) + + + +The detector flags tickets matching the 8 known patterns and persists a reports row +(ROADMAP SC#2), reprocesses only when the ticket's title/description content hash changed +(SC#3, D-04), and each reports row stores EVID-01 evidence — company, requester/reporter, +title, description, notes, time entries, and attachment metadata (SC#4). + + + +Create `.planning/phases/15-data-model-detection-ticket-evidence/15-02-SUMMARY.md` when done. + diff --git a/.planning/phases/15-data-model-detection-ticket-evidence/15-03-PLAN.md b/.planning/phases/15-data-model-detection-ticket-evidence/15-03-PLAN.md new file mode 100644 index 0000000..d573593 --- /dev/null +++ b/.planning/phases/15-data-model-detection-ticket-evidence/15-03-PLAN.md @@ -0,0 +1,268 @@ +--- +phase: 15-data-model-detection-ticket-evidence +plan: 03 +type: execute +wave: 3 +depends_on: ["15-02"] +files_modified: + - lib/services/phishing-sweep-service.ts + - lib/services/webhook-service.ts + - lib/services/sync-scheduler.ts + - migrations/098_phishing_sweep_schedule.sql +autonomous: true +requirements: [DETECT-01, DETECT-02] +must_haves: + truths: + - "A newly created ticket webhook fires phishing detection without blocking the webhook response" + - "A scheduled cron sweep re-scans recently-modified tickets through the same detector core" + - "The sweep is bounded (row limit) and reports a scanned/flagged/skipped/errors summary" + - "A phishing-sweep schedule row exists for both fresh installs and existing installs" + artifacts: + - path: "lib/services/phishing-sweep-service.ts" + provides: "sweepPhishingTickets() — bounded reconciliation loop over recently-modified tickets" + min_lines: 40 + - path: "migrations/098_phishing_sweep_schedule.sql" + provides: "phishing-sweep sync_schedules seed row (ON CONFLICT DO NOTHING)" + contains: "phishing-sweep" + key_links: + - from: "webhook-service.ts ticket.created handler" + to: "detectPhishingTicket (Plan 02)" + via: "fire-and-forget .catch()" + pattern: "detectPhishingTicket|triggerPhishingDetection" + - from: "sync-scheduler.ts dispatch" + to: "sweepPhishingTickets" + via: "else-if branch on sync_type === 'phishing-sweep' with dynamic import" + pattern: "phishing-sweep" +--- + + +Wire the Plan 02 detector into Pulse's two established scan triggers (D-01): +(1) the ticket webhook path (near-real-time, fire-and-forget) and (2) a scheduled +cron reconciliation sweep over recently-modified tickets — plus the schedule seed +row for existing installs. This is what makes DETECT-01 scanning and DETECT-02 +idempotency actually run in production. + +Purpose: Mirror the "webhook primary, cron reconciles" pattern already used for +ticket_notes and the pax8-daily / tickets-reconcile schedule registration. +Output: A sweep service, a webhook hook-in, a scheduler branch, and migration 098. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/15-data-model-detection-ticket-evidence/15-CONTEXT.md +@.planning/phases/15-data-model-detection-ticket-evidence/15-PATTERNS.md +@.planning/phases/15-data-model-detection-ticket-evidence/15-02-SUMMARY.md + + + + +Plan 02 export (lib/services/phishing-detector.ts): +detectPhishingTicket(ticket: DetectableTicket): Promise<{ flagged: boolean; reportId?: string; skippedUnchanged?: boolean }> +DetectableTicket = { id, ticket_number, title, description, company_id, contact_id?, created_by_contact_id? } + +webhook-service.ts (existing precedent to copy verbatim): +- Fire-and-forget block, lines 112-117: on payload.entityType === WebhookEntityType.TICKETS && + payload.eventType === WebhookEventType.CREATE, `this.triggerWorkflowEngine(payload).catch(err => console.error(...))`. +- triggerWorkflowEngine (lines 398-420): builds typed ticket data from payload.entity when present, + else uses payload.entityId — copy this "prefer inline entity, else id" shape. +- Imports at top, lines 13-16. + +sync-scheduler.ts (existing precedent): +- ScheduleConfig.sync_type union, line 25 (currently ends `... | 'tickets-reconcile' | 'pax8-daily'`). +- defaultSchedules array — tickets-reconcile entry at lines 294-301 (id, name, description, + cron_expression, sync_type, is_enabled: false). Seeded via ON CONFLICT (id) DO NOTHING (lines 304-318). +- Dispatch chain — tickets-reconcile branch at lines 458-463: `} else if (config.sync_type === 'tickets-reconcile') { + const { reconcileStaleTickets } = await import('@/lib/services/ticket-reconciliation-service'); + const result = await reconcileStaleTickets(); console.log('[SCHEDULER] tickets-reconcile: ...'); }`. + Every branch uses dynamic `await import()` — required per CLAUDE.md (no eager worker/scheduler imports). + +sweep source (ticket-reconciliation-service.ts:51-151): bounded SELECT (LIMIT), per-row try/catch, +aggregate result object, createSyncLogger. tickets columns: id, ticket_number, title, description, +company_id, contact_id, created_by_contact_id, last_activity_date. + +migration seed precedent: migrations/090_ticket_reconcile_schedule.sql (INSERT INTO sync_schedules +(id, name, description, cron_expression, sync_type, years_back, is_enabled) VALUES (...) ON CONFLICT (id) DO NOTHING). +sync_schedules.id is a text PK (values like 'tickets-reconcile'). + + + + + + + Task 1: phishing-sweep-service.ts — bounded reconciliation sweep + lib/services/phishing-sweep-service.ts + + - lib/services/ticket-reconciliation-service.ts (full file — SELECT-with-LIMIT, per-row try/catch, ReconcileResult aggregate shape, createSyncLogger usage to mirror) + - lib/services/phishing-detector.ts (detectPhishingTicket signature + DetectableTicket shape from Plan 02) + - lib/services/postgres-client.ts (query API + parameterized-query convention) + + + Create `lib/services/phishing-sweep-service.ts`. Export + `interface PhishingSweepResult { scanned: number; flagged: number; skippedUnchanged: number; errors: number }` + and `export async function sweepPhishingTickets(): Promise`. + + Implementation (mirror reconcileStaleTickets structure): + - Use `createSyncLogger({ component: 'PhishingSweep' })`. + - Define module constants `SWEEP_WINDOW_DAYS = 7` and `SCAN_LIMIT = 500` (bounded — DoS guard). + - Query recently-modified, non-deleted tickets: + `SELECT id, ticket_number, title, description, company_id, contact_id, created_by_contact_id + FROM tickets WHERE is_deleted = false AND last_activity_date > NOW() - INTERVAL '7 days' + ORDER BY last_activity_date DESC LIMIT $1` with `[SCAN_LIMIT]` (parameterized; the interval + may be inlined as a literal since it is a constant, or bound as a parameter). + - For each row, call `detectPhishingTicket({ id: Number(row.id), ticket_number, title, description, + company_id, contact_id, created_by_contact_id })` inside try/catch. Increment + result.scanned always; result.flagged when `res.flagged && !res.skippedUnchanged`; + result.skippedUnchanged when `res.skippedUnchanged`; result.errors on caught exception + (logger.warn with the ticket id, never rethrow — one bad ticket must not abort the sweep). + - Log a final summary line and return the result. + + Do NOT re-implement matching/hashing — call the shared detectPhishingTicket only (CONTEXT.md: + "no duplicated pattern-matching logic"). Do NOT add a side-effect self-init or eager import. + + + npx tsc --noEmit --pretty && grep -q "detectPhishingTicket" lib/services/phishing-sweep-service.ts && grep -q "LIMIT" lib/services/phishing-sweep-service.ts && echo "sweep OK" + + + - `lib/services/phishing-sweep-service.ts` exports `sweepPhishingTickets` returning `{ scanned, flagged, skippedUnchanged, errors }` + - The ticket query is bounded by `LIMIT` (<= 500) and filters `is_deleted = false` and a recent `last_activity_date` window + - Each ticket is processed by calling `detectPhishingTicket` (the file imports it from `./phishing-detector`) — no duplicated match/hash logic + - A per-row try/catch increments `errors` and logs without aborting the loop (no rethrow) + - No module-level side-effect / self-init / eager worker import + - `npx tsc --noEmit --pretty` passes + + sweepPhishingTickets scans a bounded set of recently-modified tickets through the shared detector and returns an aggregate summary. + + + + Task 2: Webhook hook-in — fire-and-forget phishing detection on ticket.created + lib/services/webhook-service.ts + + - lib/services/webhook-service.ts (full file — imports at 13-16, ticket.created fire-and-forget block at 112-117, triggerWorkflowEngine payload-shaping at 398-420) + - lib/services/phishing-detector.ts (detectPhishingTicket + DetectableTicket shape) + + + Modify `lib/services/webhook-service.ts` to trigger phishing detection on ticket + creation, exactly mirroring the existing workflow-engine fire-and-forget precedent. + + - Add an import for `detectPhishingTicket` from `./phishing-detector` alongside the + existing service imports at the top of the file (lines 13-16). + - Add a private method `triggerPhishingDetection(payload: AutotaskWebhookPayload): Promise` + that builds a DetectableTicket the same "prefer inline payload.entity, else use payload.entityId" + way triggerWorkflowEngine does (lines 398-420): if payload.entity is present, map + title/description/ticketNumber/companyID/contactID/creatorContactID from it; otherwise pass a + minimal ticket with `id: payload.entityId` and null fields (the detector's evidence/hash still + works, and the cron sweep reconciles anything the webhook payload lacked). Call + `await detectPhishingTicket(ticket)`. + - In the existing ticket.created fire-and-forget block (lines 112-117), add a second + identically-shaped call immediately after the triggerWorkflowEngine call: + `this.triggerPhishingDetection(payload).catch(err => console.error('[WEBHOOK] Phishing detection error:', err));`. + Do NOT `await` it in the request path — detection runs after the webhook response, matching the + workflow-engine precedent. Do not change the workflow-engine call or any other webhook behavior. + + + npx tsc --noEmit --pretty && grep -q "triggerPhishingDetection" lib/services/webhook-service.ts && grep -q "Phishing detection error" lib/services/webhook-service.ts && echo "hook OK" + + + - `webhook-service.ts` imports `detectPhishingTicket` from `./phishing-detector` + - A `triggerPhishingDetection` method exists and builds ticket data using the "prefer payload.entity, else payload.entityId" shape + - The ticket.created handler calls `this.triggerPhishingDetection(payload).catch(...)` as fire-and-forget (not awaited in the request path), immediately alongside the existing `triggerWorkflowEngine` call + - The existing workflow-engine trigger and all other webhook behavior are unchanged (grep still finds `triggerWorkflowEngine`) + - `npx tsc --noEmit --pretty` passes + + New ticket webhooks fire phishing detection without blocking the webhook response, mirroring the workflow-engine trigger precedent. + + + + Task 3: Scheduler branch + defaultSchedules entry + migration 098 seed + lib/services/sync-scheduler.ts, migrations/098_phishing_sweep_schedule.sql + + - lib/services/sync-scheduler.ts (sync_type union line 25; defaultSchedules tickets-reconcile entry lines 294-301 + seeding loop 304-318; dispatch chain tickets-reconcile branch 458-463) + - migrations/090_ticket_reconcile_schedule.sql (seed-row shape + ON CONFLICT (id) DO NOTHING) + - lib/services/phishing-sweep-service.ts (sweepPhishingTickets export from Task 1) + + + Register the phishing reconciliation sweep as a scheduled sync, using the + `tickets-reconcile` entry as the closest analog (no external-integration config gate needed). + + In `lib/services/sync-scheduler.ts`: + - Extend the `ScheduleConfig.sync_type` union (line 25) by adding `| 'phishing-sweep'`. + - Add a new entry to the `defaultSchedules` array (copy the tickets-reconcile shape at + lines 294-301): id `'phishing-sweep'`, name `'Phishing Detection Sweep'`, a description stating + it reconciles recently-modified tickets through the phishing detector to catch reports missed by + the webhook path (bounded to 500 tickets, idempotent on content hash), cron_expression a daily + time distinct from other jobs (e.g. `'0 5 * * *'`), sync_type `'phishing-sweep'`, + is_enabled: false (disabled-by-default, matching tickets-reconcile precedent). + - Add a dispatch branch mirroring the tickets-reconcile branch (lines 458-463): + `} else if (config.sync_type === 'phishing-sweep') { + const { sweepPhishingTickets } = await import('@/lib/services/phishing-sweep-service'); + const result = await sweepPhishingTickets(); + console.log('[SCHEDULER] phishing-sweep: scanned=... flagged=... skippedUnchanged=... errors=...'); }` + using dynamic `await import` (required — no eager import). + + Create `migrations/098_phishing_sweep_schedule.sql` (copy migrations/090's shape): a header + comment noting createDefaultSchedules only seeds a virgin table so this covers existing installs, + then `INSERT INTO sync_schedules (id, name, description, cron_expression, sync_type, years_back, + is_enabled) VALUES ('phishing-sweep', 'Phishing Detection Sweep', '', '0 5 * * *', + 'phishing-sweep', NULL, false) ON CONFLICT (id) DO NOTHING;`. Keep the description text identical to + the defaultSchedules entry. Do not edit any committed migration. + + + npx tsc --noEmit --pretty && grep -q "'phishing-sweep'" lib/services/sync-scheduler.ts && grep -q "sweepPhishingTickets" lib/services/sync-scheduler.ts && test -f migrations/098_phishing_sweep_schedule.sql && grep -q "ON CONFLICT (id) DO NOTHING" migrations/098_phishing_sweep_schedule.sql && echo "scheduler+migration OK" + + + - `sync_type` union in sync-scheduler.ts includes `'phishing-sweep'` + - `defaultSchedules` contains a `phishing-sweep` entry with `is_enabled: false` and a valid cron expression + - A dispatch branch `else if (config.sync_type === 'phishing-sweep')` dynamically imports and calls `sweepPhishingTickets` and logs a scanned/flagged/skipped/errors summary + - `migrations/098_phishing_sweep_schedule.sql` exists, inserts the `phishing-sweep` row, and uses `ON CONFLICT (id) DO NOTHING` + - The migration description text matches the defaultSchedules entry description + - `npx tsc --noEmit --pretty` passes + + The phishing sweep is registered in the scheduler (disabled by default) and seeded for existing installs via migration 098, invoking the shared sweep service on each tick. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| Autotask webhook → detection trigger | External webhook event initiates a background code path | +| cron scheduler → sweep | Time-triggered bulk processing over local tickets | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-15-09 | Denial of Service | repeated/replayed ticket.created webhooks | mitigate | Detection is fire-and-forget (never blocks the webhook response) AND idempotent via the Plan 02 content-hash guard — replays of an unchanged ticket skip before any write. Webhook HMAC verification (existing middleware/webhook-service) still gates the endpoint | +| T-15-10 | Denial of Service | unbounded cron sweep | mitigate | sweepPhishingTickets is bounded by `LIMIT 500` and a 7-day recent-activity window; per-row try/catch isolates failures so one ticket cannot abort the run | +| T-15-11 | Tampering | new schedule row on existing installs | mitigate | migration 098 is `ON CONFLICT (id) DO NOTHING` (idempotent) and additive; disabled-by-default (is_enabled=false) so it does nothing until an admin opts in | +| T-15-12 | Repudiation | sweep/detection produces no trace | accept | Both paths log via createSyncLogger / console.error with `[PHISHING-DETECT]`/`[SCHEDULER]` prefixes; full audit_events wiring is Phase 20 scope | +| T-15-SC | Tampering | package installs | accept | No new npm packages — reuses postgresClient, AutotaskClient, node-cron scheduler, and the Plan 02 detector. No install task | + + + +- `npx tsc --noEmit --pretty` clean across all three modified/created files +- `grep "'phishing-sweep'" lib/services/sync-scheduler.ts` finds union member, default entry, and dispatch branch +- `grep "triggerPhishingDetection" lib/services/webhook-service.ts` finds the fire-and-forget hook +- `grep "ON CONFLICT (id) DO NOTHING" migrations/098_phishing_sweep_schedule.sql` matches +- Sweep query bounded by LIMIT (DoS guard) + + + +The detector runs on both triggers per D-01: near-real-time via the webhook ticket.created +path (fire-and-forget, ROADMAP SC#2) and via a bounded scheduled reconciliation sweep, both +calling the same shared detectPhishingTicket so idempotency (SC#3) holds on either path. The +phishing-sweep schedule is registered for fresh and existing installs. + + + +Create `.planning/phases/15-data-model-detection-ticket-evidence/15-03-SUMMARY.md` when done. +