lorentz/wulf-pulse
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions
wulf-pulse/.planning/reports/PHISHING_TRIAGE_LLM_CONTEXT.md

16 KiB
Raw Permalink Blame History

Phishing Triage Automation — LLM Context Document

How to use this document: Paste this whole file into a new chat session as context when you want to brainstorm, design, or spec out new features for Pulse's phishing triage system. It is written to be self-contained and information-dense rather than narrative — concrete function/table/route names are included so an LLM can reason about extension points without guessing. Repo file paths are given as provenance (« path »), not as something the reading model needs to fetch.

System: Pulse — Wulf Consulting's internal PSA dashboard (Next.js 16 + PostgreSQL, syncs Autotask PSA data). This document covers one subsystem: v3.0 Phishing Triage Automation, shipped 2026-07-17, 9 phases / 30 plans / 69 tasks, all 38 requirements complete, refined by 5 post-ship fixes through 2026-07-18.

1. Core value proposition

A security operator sees every phishing/spam-report ticket automatically triaged, deduplicated into campaigns, and classified — with any destructive remediation gated behind explicit human approval. Nothing in this system currently executes a real destructive action against a mail or identity provider; every remediation action this milestone is proposed-and-simulated only.

Trigger sources: Autotask tickets created by (a) KnowBe4 phish-alert-button reports, (b) Microsoft's built-in "Report Message" flow forwarded into a mailbox that syncs to Autotask, (c) manual employee spam/phishing complaints.

Pipeline (single sentence): detect ticket → extract .eml evidence → look up Mimecast blast radius → deterministically classify → group into a campaign → propose remediation behind human approval → post sanitized Autotask note → surface on a ticket-ID-addressable review page reachable via a real Autotask LiveLink button → optionally automate per-client via an opt-in gate.

2. Explicit non-goals (this milestone) — deferred to a future "v2"

These are known-missing, not accidentally missing — a future feature request in this space should assume these don't exist yet:

  • No real remediation execution. No code path blocks a sender in Mimecast, deletes/ moves a Graph mailbox item, purges via Defender/Exchange, revokes a session, or resets a password. REMEDEXEC-01..05 in the original requirements doc name these explicitly as out-of-scope-for-v1.
  • No URL reputation/sandbox lookup (ENRICH-01) — URLs are extracted and displayed, never scored or detonated.
  • No LLM-backed classification (ENRICH-02) — the classifier is 100% deterministic code; an LLM layer was deliberately reserved as a future plug-in behind the same interface, not built.
  • No automatic ticket closure, no auto-created parent incident, no auto-closed duplicate tickets.
  • No backfill of ~267 pre-existing phishing tickets that predate this system — detection is forward-only from ship date.
  • No per-tenant Mimecast client by default — a single global Mimecast client serves all companies (a documented v1 limitation; one post-ship quick task added optional per-tenant client injection, but the automatic/webhook path still uses the global client).

3. Data model

« migrations/097_phishing_triage_schema.sql, 098, 099_indicators_metadata.sql, 100_phishing_automation_gate.sql »

Table Purpose Key columns
campaigns One row per grouped phishing/spam incident id, campaign_key, group_method, first_seen_at, last_seen_at, report_count, status (default open)
reports One row per detected ticket id, ticket_id (FK, UNIQUE), ticket_number, company_id, company_name, requester_contact_id, title, description, matched_patterns JSONB, content_hash, evidence JSONB, campaign_id (FK, nullable)
messages Parsed .eml output, one per report id, report_id (FK), message_id, headers/urls/attachments JSONB, body_preview, raw_ref (B2 object key or null)
indicators Per-attachment-hash / per-URL / per-sender correlation signals id, message_id (FK), indicator_type, value, metadata JSONB
classifications Append-only classify-call history id, campaign_id (FK), verdict, confidence NUMERIC, summary, reasons JSONB, recommended_actions JSONB, requires_approval BOOLEAN
remediation_actions Proposed/approved/completed remediation state id, campaign_id (FK), action_type, status (default proposed), params JSONB, approved_by, approved_at
audit_events Single append-only audit trail for every state change id, campaign_id, actor, event_type, payload JSONB, created_at
phishing_automation_gate Per-company opt-in automation flags company_id (PK/FK), auto_parse/auto_classify/auto_report BOOLEAN (default false), updated_by, updated_at

sync_schedules gains a seeded row id='phishing-sweep', cron 0 5 * * *, is_enabled=false.

Verdict vocabulary: SPAM | UNWANTED | THREAT | USER_AWARENESS (the last added in Phase 23, for confirmed simulation-vendor reports).

Action vocabulary (7 types): each maps to a destructive/non-destructive flag; only acknowledge_user is non-destructive (customer-visible "thanks for reporting" note) and is the sole action type an automation gate may auto-post without human approval.

4. API surface

All /api/phishing/* routes are requirePermission('phishing', <action>)-gated (read | analyze | approve | remediate). All /api/admin/phishing-automation/* routes are requireAdmin()-gated.

Route Method Purpose
/api/phishing/campaigns GET Paginated campaign list
/api/phishing/campaigns/{id} GET Full nested detail: reports/messages/indicators/classifications/remediation_actions/audit_events/timeline + fresh blast-radius
/api/phishing/campaigns/{id}/classify POST On-demand classify + audit write
/api/phishing/campaigns/{id}/approve POST Approve one or more recommended actions with param overrides
/api/phishing/campaigns/{id}/remediate POST Execute (simulated) previously-approved actions, idempotent
/api/phishing/campaigns/{id}/mark-false-positive POST Mark false-positive; 409 if any action already approved/completed
/api/phishing/campaigns/{id}/mark-accidental-report POST Parallel resolution path + customer note (added post-ship)
/api/phishing/campaigns/{id}/triage-note POST On-demand internal triage-note generation/post
/api/phishing/reports/{report_id} GET Standalone (ungrouped) report evidence + blast radius
/api/phishing/tickets/{ticket_id}/campaign GET Ticket→campaign resolver — always 200, {found:false} instead of 404 (so an Autotask LiveLink never dead-ends)
/api/phishing/tickets/{ticket_id}/analyze POST On-demand detect→parse→group for one ticket
/api/admin/phishing-automation GET List all companies + automation-gate flags
/api/admin/phishing-automation/{companyId} PATCH Upsert a company's 3 gate flags
/api/admin/phishing-automation/{companyId} DELETE Revert to all-off default

5. UI surface

  • app/phishing/page.tsx — campaign list (nav entry).
  • app/phishing/tickets/[ticketId]/page.tsx — the real review surface + production- confirmed live Autotask LiveLink target. 5-state machine: loading | not-triaged | ungrouped | ready | error. Composed of:
    • ClassificationCard — verdict badge, confidence %, summary, reasons, recommended actions, permission-gated reclassify.
    • ActionAreaCard — the only write-surface: checkbox-per-action approve with editable param overrides, confirm-dialog remediate/mark-false-positive/mark-accidental-report. Buttons always mounted, disabled+tooltipped when unauthorized or already resolved.
    • EvidenceCard — tabbed Headers (SPF/DKIM/DMARC) / URLs (inert <code> text with copy-to-clipboard, never a clickable link — no dangerouslySetInnerHTML anywhere in this feature) / Attachments (metadata only) / Body preview / Blast Radius (explicit unavailable branch).
    • TimelineCard — server-pre-sorted chronological feed.
  • app/admin/phishing-automation/page.tsx — company table, 3 independent Switch toggles per row (parse/classify/report).
  • Nav entry "Phishing" visible to all roles (everyone holds phishing:read).

6. Architecture decisions worth knowing before proposing new features

# Decision Rationale
1 Zero-LLM deterministic classifier Avoids attacker-controlled email content ever reaching a model prompt (prompt-injection surface). Any future AI-assisted classification should sit behind the existing verdict interface, not replace it, unless the injection risk is separately solved.
2 Content-hash idempotency = SHA-256(title + description) only Deliberately excludes status/assignee/timestamp churn. Evidence (notes/attachments) still refreshes every rescan even on an unchanged hash.
3 3-tier .eml selection rfc.eml exact name → single non-wrapper message/rfc822 candidate → OriginatingEmail.eml fallback. Content-type alone can't disambiguate (empirically confirmed against 15 real tickets).
4 Hand-rolled SPF/DKIM/DMARC tokenizer The obvious library (mailauth) forces live DNS/BIMI-HTTP lookups — violates the "never fetch anything from message content" invariant that runs through this whole feature.
5 3-tier campaign grouping key Message-ID → attachment-hash/URL-domain + subject + sender + 24h window → sender + normalized-subject + client + 24h window. Campaigns are never merged after creation — explicitly out of scope.
6 Mimecast blast-radius never throws Unconditional parallel fan-out (delivered/held/threat-event lookups), 5-min Redis cache, degrades to an explicit {status:'unavailable', reason:...} rather than blocking the pipeline.
7 THREAT requires delivery AND a malicious signal An auth failure alone on a fully-held message (reached nobody) stays UNWANTED. Malicious signal = hard SPF/DKIM/DMARC fail OR the same attachment-hash/URL recurring across ≥2 reports in the campaign — no external reputation service involved.
8 Simulation-vendor allowlist is a TypeScript constant Not a DB table — every rule stays unit-testable pure code. Exact-domain-or-proper-subdomain match only, never substring, to resist lookalike-domain spoofing.
9 Confidence = point-deduction from 1.0 Each deduction names the specific missing evidence (no parsed message / no Mimecast data / no indicators) rather than an opaque score — this is a literal requirement (CLASSIFY-03), not incidental.
10 All remediation this milestone is simulated No block/purge/reset methods exist yet on any integration client — a real "actually block this sender" feature is new integration work, not a flag flip.
11 acknowledge_user is the one auto-postable, non-destructive action Everything else requires human approval by design; a new "auto-post X" feature should default to requiring the same explicit per-client opt-in Phase 23 established.
12 Per-company automation = 3 independent booleans, default off auto_parse / auto_classify / auto_report. Report meaningfully depends on classify, which depends on parse (informational ordering only, not enforced in code).
13 URLs are inert, copy-only text in the operator UI Stricter than the triage-note's sanitize-and-describe approach because this is an interactive page a human could click into.
14 Client-side permission checks reuse the exact server permission function No bespoke role === 'admin' string checks anywhere in this feature — server remains sole enforcement boundary.

7. Known gaps / tech debt (don't silently rediscover these)

  • Phase 22 (Approval UI) has 5 outstanding manual browser click-through checks never run: full state-machine walkthrough, approve/remediate/mark-false-positive end-to-end with re-render, non-privileged role gating, Mimecast-unavailable rendering, URL inertness/clipboard-copy visual confirmation. Code is verified; UI is not yet browser-hardened.
  • No retry if a webhook-triggered ticket row isn't yet in Postgres when detection runs — only a warning log; the backstop cron sweep ships disabled by default.
  • Webhook doesn't re-trigger detection on ticket update, only create (deliberate).
  • No floor (GREATEST(..., 0)) on the campaign report-count decrement; a pre-existing TOCTOU race in the "already grouped" check.
  • The THREAT-tier "clicked > 0" escalation trigger was a reasoned proposal from research, never formally re-confirmed as a locked decision.
  • The automatic (webhook/cron) path only ever reaches Tier-3 (weakest) campaign grouping until an operator has manually analyzed at least one report in that campaign — the EML parser is only invoked from the on-demand analyze endpoint, decision #5/#3 above.
  • Global (not per-tenant) Mimecast client remains the default for the automatic path.
  • 2 pre-existing, unrelated failing tests (itglue-search.test.ts) — confirmed to predate this feature entirely, not phishing-related.

3 real defects already found and fixed (useful precedent — similar bug classes are worth checking for in any new feature touching the same code):

  1. Webhook trigger branching on an Autotask payload field that real webhooks never populate (fixed by reading the ticket back from Postgres instead of trusting payload shape).
  2. Re-analyzing a single-report campaign created a duplicate campaign row with an identical grouping key (a "no sibling match found" edge case fell through to "create new" instead of "keep existing").
  3. Every additional report accumulating into an already-classified simulation campaign re-triggered a duplicate customer-visible note (no idempotency guard on the auto-post path, and — found in review of that very fix — the parallel manual approval path had no equivalent guard either).

5 post-ship refinements (2026-07-16 → 2026-07-18): Mimecast blast-radius date-window

  • silently-swallowed-error fix; a confidence-percentage display bug (1.0 confidence showed as "1%"); expanded simulation-vendor allowlist + parse-idempotency + auto-parse timing fix; new "mark as accidental report" action; Mimecast held-message false-positive fix (unbounded hold-queue lookup inflating blast-radius counts).

8. Entry points (for a human engineer, if this context is being used to spec work

that will land back in this repo)

  • « lib/services/phishing-detector.ts » — detection front door.
  • « lib/services/campaign-grouping-service.ts » — trickiest logic; has the most bug history (see gap #2 above).
  • « lib/services/campaign-classifier.ts » — deterministic rule engine.
  • « app/phishing/tickets/[ticketId]/page.tsx » + « components/phishing/* » — operator-facing review surface.
  • « lib/services/phishing-automation-gate.ts » + « app/admin/phishing-automation/page.tsx » — per-client automation on/off switch.
  • « docs/mimecast-api-guide.md » — documents the held-message-queue date-range requirement behind one of the post-ship fixes above.

9. Good directions to brainstorm from here

Given the explicit non-goals above, natural "what's next" conversation starters:

  • Real remediation execution (REMEDEXEC-01..05): actually calling Mimecast block, Graph mailbox purge/move, Defender/Exchange purge, behind the existing approve-gate — the approval/audit/idempotency scaffolding already exists and is designed to have a real executor slotted in later.
  • URL reputation enrichment (ENRICH-01): a scoring layer over the already-extracted, already-deduped indicator URLs.
  • An LLM-assisted classification layer (ENRICH-02), explicitly designed to sit behind the current deterministic verdict interface as an optional enhancement/second opinion — not a replacement, given decision #1's prompt-injection rationale.
  • Per-tenant Mimecast client as the default, not just an optional override, closing gap "Global (not per-tenant) Mimecast client."
  • Automatic campaign grouping parity with the on-demand path — closing the Tier-3-only limitation on the fully automatic pipeline.
  • Historical backfill of the ~267 pre-existing phishing tickets, if retroactive visibility becomes valuable.