diff --git a/.planning/reports/PHISHING_TRIAGE_LLM_CONTEXT.md b/.planning/reports/PHISHING_TRIAGE_LLM_CONTEXT.md new file mode 100644 index 0000000..9d7092a --- /dev/null +++ b/.planning/reports/PHISHING_TRIAGE_LLM_CONTEXT.md @@ -0,0 +1,215 @@ +# Phishing Triage Automation — LLM Context Document + +> **How to use this document:** Paste this whole file into a new chat session as context +> when you want to brainstorm, design, or spec out new features for Pulse's phishing +> triage system. It is written to be self-contained and information-dense rather than +> narrative — concrete function/table/route names are included so an LLM can reason +> about extension points without guessing. Repo file paths are given as provenance +> (`« path »`), not as something the reading model needs to fetch. +> +> System: **Pulse** — Wulf Consulting's internal PSA dashboard (Next.js 16 + PostgreSQL, +> syncs Autotask PSA data). This document covers one subsystem: **v3.0 Phishing Triage +> Automation**, shipped 2026-07-17, 9 phases / 30 plans / 69 tasks, all 38 requirements +> complete, refined by 5 post-ship fixes through 2026-07-18. + +--- + +## 1. Core value proposition + +A security operator sees every phishing/spam-report ticket automatically triaged, +deduplicated into campaigns, and classified — with any destructive remediation gated +behind explicit human approval. Nothing in this system currently executes a real +destructive action against a mail or identity provider; every remediation action this +milestone is proposed-and-simulated only. + +**Trigger sources:** Autotask tickets created by (a) KnowBe4 phish-alert-button reports, +(b) Microsoft's built-in "Report Message" flow forwarded into a mailbox that syncs to +Autotask, (c) manual employee spam/phishing complaints. + +**Pipeline (single sentence):** detect ticket → extract `.eml` evidence → look up +Mimecast blast radius → deterministically classify → group into a campaign → propose +remediation behind human approval → post sanitized Autotask note → surface on a +ticket-ID-addressable review page reachable via a real Autotask LiveLink button → +optionally automate per-client via an opt-in gate. + +## 2. Explicit non-goals (this milestone) — deferred to a future "v2" + +These are known-missing, not accidentally missing — a future feature request in this +space should assume these don't exist yet: + +- **No real remediation execution.** No code path blocks a sender in Mimecast, deletes/ + moves a Graph mailbox item, purges via Defender/Exchange, revokes a session, or resets + a password. `REMEDEXEC-01..05` in the original requirements doc name these explicitly + as out-of-scope-for-v1. +- **No URL reputation/sandbox lookup** (`ENRICH-01`) — URLs are extracted and displayed, + never scored or detonated. +- **No LLM-backed classification** (`ENRICH-02`) — the classifier is 100% deterministic + code; an LLM layer was deliberately reserved as a future plug-in behind the same + interface, not built. +- **No automatic ticket closure, no auto-created parent incident, no auto-closed + duplicate tickets.** +- **No backfill** of ~267 pre-existing phishing tickets that predate this system — + detection is forward-only from ship date. +- **No per-tenant Mimecast client by default** — a single global Mimecast client serves + all companies (a documented v1 limitation; one post-ship quick task added *optional* + per-tenant client injection, but the automatic/webhook path still uses the global + client). + +## 3. Data model + +`« migrations/097_phishing_triage_schema.sql, 098, 099_indicators_metadata.sql, 100_phishing_automation_gate.sql »` + +| Table | Purpose | Key columns | +|---|---|---| +| `campaigns` | One row per grouped phishing/spam incident | `id`, `campaign_key`, `group_method`, `first_seen_at`, `last_seen_at`, `report_count`, `status` (default `open`) | +| `reports` | One row per detected ticket | `id`, `ticket_id` (FK, UNIQUE), `ticket_number`, `company_id`, `company_name`, `requester_contact_id`, `title`, `description`, `matched_patterns` JSONB, `content_hash`, `evidence` JSONB, `campaign_id` (FK, nullable) | +| `messages` | Parsed `.eml` output, one per report | `id`, `report_id` (FK), `message_id`, `headers`/`urls`/`attachments` JSONB, `body_preview`, `raw_ref` (B2 object key or null) | +| `indicators` | Per-attachment-hash / per-URL / per-sender correlation signals | `id`, `message_id` (FK), `indicator_type`, `value`, `metadata` JSONB | +| `classifications` | Append-only classify-call history | `id`, `campaign_id` (FK), `verdict`, `confidence` NUMERIC, `summary`, `reasons` JSONB, `recommended_actions` JSONB, `requires_approval` BOOLEAN | +| `remediation_actions` | Proposed/approved/completed remediation state | `id`, `campaign_id` (FK), `action_type`, `status` (default `proposed`), `params` JSONB, `approved_by`, `approved_at` | +| `audit_events` | Single append-only audit trail for every state change | `id`, `campaign_id`, `actor`, `event_type`, `payload` JSONB, `created_at` | +| `phishing_automation_gate` | Per-company opt-in automation flags | `company_id` (PK/FK), `auto_parse`/`auto_classify`/`auto_report` BOOLEAN (default false), `updated_by`, `updated_at` | + +`sync_schedules` gains a seeded row `id='phishing-sweep'`, cron `0 5 * * *`, `is_enabled=false`. + +**Verdict vocabulary:** `SPAM | UNWANTED | THREAT | USER_AWARENESS` (the last added in +Phase 23, for confirmed simulation-vendor reports). + +**Action vocabulary (7 types):** each maps to a destructive/non-destructive flag; only +`acknowledge_user` is non-destructive (customer-visible "thanks for reporting" note) and +is the sole action type an automation gate may auto-post without human approval. + +## 4. API surface + +All `/api/phishing/*` routes are `requirePermission('phishing', )`-gated +(`read | analyze | approve | remediate`). All `/api/admin/phishing-automation/*` routes +are `requireAdmin()`-gated. + +| Route | Method | Purpose | +|---|---|---| +| `/api/phishing/campaigns` | GET | Paginated campaign list | +| `/api/phishing/campaigns/{id}` | GET | Full nested detail: reports/messages/indicators/classifications/remediation_actions/audit_events/timeline + fresh blast-radius | +| `/api/phishing/campaigns/{id}/classify` | POST | On-demand classify + audit write | +| `/api/phishing/campaigns/{id}/approve` | POST | Approve one or more recommended actions with param overrides | +| `/api/phishing/campaigns/{id}/remediate` | POST | Execute (simulated) previously-approved actions, idempotent | +| `/api/phishing/campaigns/{id}/mark-false-positive` | POST | Mark false-positive; 409 if any action already approved/completed | +| `/api/phishing/campaigns/{id}/mark-accidental-report` | POST | Parallel resolution path + customer note (added post-ship) | +| `/api/phishing/campaigns/{id}/triage-note` | POST | On-demand internal triage-note generation/post | +| `/api/phishing/reports/{report_id}` | GET | Standalone (ungrouped) report evidence + blast radius | +| `/api/phishing/tickets/{ticket_id}/campaign` | GET | Ticket→campaign resolver — **always 200**, `{found:false}` instead of 404 (so an Autotask LiveLink never dead-ends) | +| `/api/phishing/tickets/{ticket_id}/analyze` | POST | On-demand detect→parse→group for one ticket | +| `/api/admin/phishing-automation` | GET | List all companies + automation-gate flags | +| `/api/admin/phishing-automation/{companyId}` | PATCH | Upsert a company's 3 gate flags | +| `/api/admin/phishing-automation/{companyId}` | DELETE | Revert to all-off default | + +## 5. UI surface + +- `app/phishing/page.tsx` — campaign list (nav entry). +- `app/phishing/tickets/[ticketId]/page.tsx` — the real review surface + **production- + confirmed live Autotask LiveLink target**. 5-state machine: `loading | not-triaged | + ungrouped | ready | error`. Composed of: + - `ClassificationCard` — verdict badge, confidence %, summary, reasons, recommended + actions, permission-gated reclassify. + - `ActionAreaCard` — the only write-surface: checkbox-per-action approve with editable + param overrides, confirm-dialog remediate/mark-false-positive/mark-accidental-report. + Buttons always mounted, disabled+tooltipped when unauthorized or already resolved. + - `EvidenceCard` — tabbed Headers (SPF/DKIM/DMARC) / URLs (**inert `` text with + copy-to-clipboard, never a clickable link — no `dangerouslySetInnerHTML` anywhere in + this feature**) / Attachments (metadata only) / Body preview / Blast Radius (explicit + `unavailable` branch). + - `TimelineCard` — server-pre-sorted chronological feed. +- `app/admin/phishing-automation/page.tsx` — company table, 3 independent `Switch` + toggles per row (parse/classify/report). +- Nav entry "Phishing" visible to all roles (everyone holds `phishing:read`). + +## 6. Architecture decisions worth knowing before proposing new features + +| # | Decision | Rationale | +|---|---|---| +| 1 | Zero-LLM deterministic classifier | Avoids attacker-controlled email content ever reaching a model prompt (prompt-injection surface). Any future AI-assisted classification should sit *behind* the existing verdict interface, not replace it, unless the injection risk is separately solved. | +| 2 | Content-hash idempotency = SHA-256(title + description) only | Deliberately excludes status/assignee/timestamp churn. Evidence (notes/attachments) still refreshes every rescan even on an unchanged hash. | +| 3 | 3-tier `.eml` selection | `rfc.eml` exact name → single non-wrapper `message/rfc822` candidate → `OriginatingEmail.eml` fallback. Content-type alone can't disambiguate (empirically confirmed against 15 real tickets). | +| 4 | Hand-rolled SPF/DKIM/DMARC tokenizer | The obvious library (`mailauth`) forces live DNS/BIMI-HTTP lookups — violates the "never fetch anything from message content" invariant that runs through this whole feature. | +| 5 | 3-tier campaign grouping key | Message-ID → attachment-hash/URL-domain + subject + sender + 24h window → sender + normalized-subject + client + 24h window. Campaigns are **never merged** after creation — explicitly out of scope. | +| 6 | Mimecast blast-radius never throws | Unconditional parallel fan-out (delivered/held/threat-event lookups), 5-min Redis cache, degrades to an explicit `{status:'unavailable', reason:...}` rather than blocking the pipeline. | +| 7 | THREAT requires delivery AND a malicious signal | An auth failure alone on a fully-held message (reached nobody) stays UNWANTED. Malicious signal = hard SPF/DKIM/DMARC fail OR the same attachment-hash/URL recurring across ≥2 reports in the campaign — no external reputation service involved. | +| 8 | Simulation-vendor allowlist is a TypeScript constant | Not a DB table — every rule stays unit-testable pure code. Exact-domain-or-proper-subdomain match only, never substring, to resist lookalike-domain spoofing. | +| 9 | Confidence = point-deduction from 1.0 | Each deduction names the specific missing evidence (no parsed message / no Mimecast data / no indicators) rather than an opaque score — this is a literal requirement (`CLASSIFY-03`), not incidental. | +| 10 | All remediation this milestone is simulated | No block/purge/reset methods exist yet on any integration client — a real "actually block this sender" feature is new integration work, not a flag flip. | +| 11 | `acknowledge_user` is the one auto-postable, non-destructive action | Everything else requires human approval by design; a new "auto-post X" feature should default to requiring the same explicit per-client opt-in Phase 23 established. | +| 12 | Per-company automation = 3 independent booleans, default off | `auto_parse` / `auto_classify` / `auto_report`. Report meaningfully depends on classify, which depends on parse (informational ordering only, not enforced in code). | +| 13 | URLs are inert, copy-only text in the operator UI | Stricter than the triage-note's sanitize-and-describe approach because this is an interactive page a human could click into. | +| 14 | Client-side permission checks reuse the exact server permission function | No bespoke `role === 'admin'` string checks anywhere in this feature — server remains sole enforcement boundary. | + +## 7. Known gaps / tech debt (don't silently rediscover these) + +- **Phase 22 (Approval UI) has 5 outstanding manual browser click-through checks** never + run: full state-machine walkthrough, approve/remediate/mark-false-positive end-to-end + with re-render, non-privileged role gating, Mimecast-unavailable rendering, URL + inertness/clipboard-copy visual confirmation. Code is verified; UI is not yet + browser-hardened. +- **No retry** if a webhook-triggered ticket row isn't yet in Postgres when detection + runs — only a warning log; the backstop cron sweep **ships disabled by default**. +- Webhook doesn't re-trigger detection on ticket *update*, only *create* (deliberate). +- No floor (`GREATEST(..., 0)`) on the campaign report-count decrement; a pre-existing + TOCTOU race in the "already grouped" check. +- The THREAT-tier "clicked > 0" escalation trigger was a reasoned proposal from research, + never formally re-confirmed as a locked decision. +- The automatic (webhook/cron) path only ever reaches Tier-3 (weakest) campaign grouping + until an operator has manually analyzed at least one report in that campaign — the EML + parser is only invoked from the on-demand analyze endpoint, decision #5/#3 above. +- Global (not per-tenant) Mimecast client remains the default for the automatic path. +- 2 pre-existing, unrelated failing tests (`itglue-search.test.ts`) — confirmed to + predate this feature entirely, not phishing-related. + +**3 real defects already found and fixed** (useful precedent — similar bug classes are +worth checking for in any new feature touching the same code): +1. Webhook trigger branching on an Autotask payload field that real webhooks never + populate (fixed by reading the ticket back from Postgres instead of trusting payload + shape). +2. Re-analyzing a single-report campaign created a duplicate campaign row with an + identical grouping key (a "no sibling match found" edge case fell through to + "create new" instead of "keep existing"). +3. Every additional report accumulating into an already-classified simulation campaign + re-triggered a duplicate customer-visible note (no idempotency guard on the + auto-post path, and — found in review of that very fix — the parallel *manual* + approval path had no equivalent guard either). + +**5 post-ship refinements** (2026-07-16 → 2026-07-18): Mimecast blast-radius date-window ++ silently-swallowed-error fix; a confidence-percentage display bug (1.0 confidence +showed as "1%"); expanded simulation-vendor allowlist + parse-idempotency + auto-parse +timing fix; new "mark as accidental report" action; Mimecast held-message false-positive +fix (unbounded hold-queue lookup inflating blast-radius counts). + +## 8. Entry points (for a human engineer, if this context is being used to spec work +that will land back in this repo) + +- `« lib/services/phishing-detector.ts »` — detection front door. +- `« lib/services/campaign-grouping-service.ts »` — trickiest logic; has the most bug + history (see gap #2 above). +- `« lib/services/campaign-classifier.ts »` — deterministic rule engine. +- `« app/phishing/tickets/[ticketId]/page.tsx »` + `« components/phishing/* »` — + operator-facing review surface. +- `« lib/services/phishing-automation-gate.ts »` + `« app/admin/phishing-automation/page.tsx »` + — per-client automation on/off switch. +- `« docs/mimecast-api-guide.md »` — documents the held-message-queue date-range + requirement behind one of the post-ship fixes above. + +## 9. Good directions to brainstorm from here + +Given the explicit non-goals above, natural "what's next" conversation starters: +- **Real remediation execution** (`REMEDEXEC-01..05`): actually calling Mimecast block, + Graph mailbox purge/move, Defender/Exchange purge, behind the existing approve-gate — + the approval/audit/idempotency scaffolding already exists and is designed to have a + real executor slotted in later. +- **URL reputation enrichment** (`ENRICH-01`): a scoring layer over the already-extracted, + already-deduped indicator URLs. +- **An LLM-assisted classification layer** (`ENRICH-02`), explicitly designed to sit + behind the current deterministic verdict interface as an optional enhancement/second + opinion — not a replacement, given decision #1's prompt-injection rationale. +- **Per-tenant Mimecast client as the default**, not just an optional override, closing + gap "Global (not per-tenant) Mimecast client." +- **Automatic campaign grouping parity** with the on-demand path — closing the Tier-3-only + limitation on the fully automatic pipeline. +- **Historical backfill** of the ~267 pre-existing phishing tickets, if retroactive + visibility becomes valuable.