16 KiB
Milestone v3.0 — Phishing Triage Automation — Project Summary
Generated: 2026-07-18 Purpose: Team onboarding and project review
1. Project Overview
Pulse is Wulf Consulting's internal PSA management dashboard (Next.js 16 + Postgres, syncing Autotask data). The v3.0 milestone's core value proposition:
A manager/security operator can see every phishing/spam report ticket automatically triaged, deduplicated into campaigns, and classified — with any destructive remediation gated behind explicit human approval.
Problem it solves: Wulf's technicians receive a constant stream of Autotask tickets from KnowBe4 phish-alert-button reports, Microsoft's built-in "Report Message" flow, and manual spam/phishing complaints. Historically each ticket was triaged by hand — no deduplication across employees reporting the same campaign, no consistent severity call, no blast-radius visibility into Mimecast, and remediation notes/actions were ad hoc.
Who it's for: Wulf's security operators/technicians (triaging tickets day to day) and admins (configuring per-client automation).
What was built: An end-to-end pipeline — detect phishing/spam-report tickets → extract .eml evidence → look up Mimecast blast radius → deterministically classify (SPAM/UNWANTED/THREAT/USER_AWARENESS) → group duplicates into campaigns → propose (never auto-execute) remediation behind human approval → post a sanitized Autotask triage note → surface it all on a ticket-ID-addressable review page reachable via a real, production-confirmed Autotask LiveLink button → optionally automate the whole chain per-client via an opt-in gate.
Status: Shipped 2026-07-17. 9 phases, 30 plans, 69 tasks. All 38 v1 requirements complete.
Explicit non-goals for this milestone (deferred to a future v2):
- No automatic execution of remediation — every action (block sender, purge mailbox, revoke sessions, etc.) is proposed and simulated only; nothing destructive runs against a real mail/identity system yet (
REMEDEXEC-01..05). - No URL sandbox/detonation or URL reputation lookup (
ENRICH-01). - No LLM-backed classification layer — this milestone is intentionally deterministic (
ENRICH-02reserves the interface for later). - No fully automated ticket closure, no auto-created parent incidents, no auto-closed duplicates.
- No backfill of historical phishing tickets (~267 pre-existing tickets) — detection is forward-only from ship date.
2. Architecture & Technical Decisions
| Decision | Why | Phase |
|---|---|---|
| Zero-LLM deterministic classifier | Structured, size-bounded evidence in → deterministic verdict out. Avoids a prompt-injection surface from attacker-controlled email content ever reaching an LLM. | 19 |
| Content-hash idempotency scoped to title+description only | SHA-256 over {title, description} — excludes status/assignee/last-activity so routine ticket churn never re-triggers detection, while evidence (notes/time-entries/attachments) still refreshes on every rescan regardless of hash match. |
15 |
3-tier .eml selection (rfc.eml exact → single non-wrapper message/rfc822 candidate → OriginatingEmail.eml fallback) |
Empirically derived from sampling 15 real production phishing tickets — content-type alone never disambiguates, since every sampled attachment shares message/rfc822. |
16 |
| Hand-rolled SPF/DKIM/DMARC tokenizer, not a library | The obvious library (mailauth) only exposes live-verification functions that perform DNS lookups and BIMI HTTP fetches — a hard violation of "never fetch anything from a message." |
16 |
| 3-tier campaign grouping key: Message-ID → attachment-hash/URL-domain + subject + sender + 24h window → sender + normalized-subject + client + 24h window | Message-ID is the strongest, cheapest key; the fallbacks catch mass-blast duplicates lacking a shared Message-ID. Campaigns are never merged after the fact — out of scope by design. | 18 |
| Mimecast blast-radius as a normalizing abstraction that never throws | Unconditional fan-out over delivered/held/threat-event lookups, 5-minute Redis cache, degrades to an explicit unavailable state on any failure or missing config — the rest of the pipeline never blocks on Mimecast being down. |
17 |
| THREAT requires BOTH delivery AND a malicious signal | An auth failure alone on a message Mimecast fully held (reached no one) isn't a realized threat. Malicious signal = hard SPF/DKIM/DMARC fail OR the same attachment-hash/URL recurring across ≥2 campaign reports. | 19 |
| Simulation-vendor allowlist is a code constant, not a DB table | Every classification rule, including the allowlist, is unit-tested pure code rather than a runtime-editable table that could silently drift. Matches exact domain or proper subdomain only — never substring — so a lookalike domain can't slip through. | 19 |
| Confidence = additive point-deduction from 1.0 | Each deduction (no parsed message, no/failed Mimecast lookup, no attachment/URL indicators) names the specific missing evidence in the response, rather than an opaque score. | 19 |
| All remediation actions are simulated status-only transitions this milestone | No real provider methods exist yet (no block/purge on the Mimecast client, no forwarding-rule/reset on the Graph client) — real execution deliberately deferred to a future v2. | 20 |
acknowledge_user is the one non-destructive action carve-out |
It's a customer-visible "thanks for reporting" note, not a security action — the only action type the automation gate is allowed to auto-post without human approval. | 19/23 |
| Per-company automation gate is 3 independent opt-in booleans (parse/classify/report), all off by default | Mirrors the "propose, don't execute" safety posture — a client only gets automation once an admin deliberately opts them in. | 23 |
| URLs render as inert copy-only text in the review UI, never a clickable link | Stricter than the triage-note's sanitize-and-describe approach, because this is an interactive page an operator could actually click into. | 22 |
| Client-side permission checks call the exact same permission function the server uses | No bespoke role checks in the UI — the server remains the sole real enforcement boundary; the client check is UX-only. | 22 |
3. Phases Delivered
| Phase | Name | Status | One-Liner |
|---|---|---|---|
| 15 | Data Model, Detection & Ticket Evidence | ✅ Complete | Durable 7-table schema + idempotent Autotask ticket scanner and base evidence capture |
| 16 | EML/MIME Evidence Parser | ✅ Complete | Pure, I/O-free RFC822/MIME parser that never executes or fetches anything |
| 17 | Mimecast Blast Radius Lookup | ✅ Complete | Normalized delivery-data abstraction that degrades gracefully when Mimecast isn't configured |
| 18 | Campaign Grouping & Phishing Analysis API | ✅ Complete | Duplicate reports accumulate into campaigns automatically; first /api/phishing/* routes |
| 19 | Classification Engine | ✅ Complete | Deterministic SPAM/UNWANTED/THREAT verdicts from bounded evidence — no LLM anywhere |
| 20 | Remediation, Approval & Audit Safety | ✅ Complete | Proposed-only actions, permission-gated approve/remediate/mark-false-positive, fully audited |
| 21 | Autotask Triage Note | ✅ Complete | Sanitized, human-readable internal note posted to every linked ticket |
| 22 | Approval UI (LiveLink) | ✅ Complete* | Ticket-ID-addressable page, confirmed live as a real Autotask LiveLink target |
| 23 | Classification Disposition + Per-Client Automation Gate | ✅ Complete | Dedicated non-destructive verdict for simulation-vendor reports + per-company opt-in automation |
*Phase 22 is code-complete and verified retroactively, but 5 manual browser click-through checks were never run — see Section 6.
4. Requirements Coverage
All 38 v1 requirements are marked Complete in v3.0-REQUIREMENTS.md.
| ID | Phase | Detail |
|---|---|---|
| DETECT-01/02/03 | 15/18 | Pattern match on 8 locked substrings; content-hash idempotency; on-demand single-ticket analyze endpoint |
| EVID-01..04 | 15/16 | Ticket evidence capture; 3-tier .eml selection; full RFC822/MIME + auth-result normalization; sanitized body preview, zero-network invariant test-enforced |
| CAMP-01..03 | 18 | 3-tier grouping key; report/recipient accumulation; campaign list + detail API |
| BLAST-01/02 | 17 | Normalized blast-radius fan-out; graceful unavailable degradation |
| CLASSIFY-01..06 | 19 | Verdict + confidence + reasons + actions; destructive-action invariant; evidence-naming confidence deductions; sender allowlist; classify API; bounded structured evidence only |
| REMED-01..06 | 20 | No auto-execution path exists (grep-confirmed); approve API; remediate API (simulated); idempotent double-call handling; mark-false-positive with 409 guard; single audit writer |
| NOTE-01 | 21 | Sanitized triage note via existing safe Autotask write path, per-ticket failure isolation |
| ACCESS-01 | 18 | requirePermission() convention applied consistently from Phase 18 onward |
| REVIEW-01..06 | 22 | Ticket-addressable route; timeline; evidence display; classification display; action area wired to APIs; client/server permission parity — all code-verified, live-browser pass outstanding |
| CLASSDISP-01..03 | 23 | USER_AWARENESS verdict; customer-visible acknowledgment note; distinct UI badge |
| AUTOGATE-01..03 | 23 | Automation-gate table + admin API; admin toggle UI; gated webhook chain (idempotency bug found and fixed pre-close) |
5. Key Decisions Log
See Section 2 for the full architecture decision table. Notable individual calls worth calling out for new contributors:
- D-01 (Phase 19): Deterministic classifier chosen over an LLM specifically to avoid attacker-controlled email content ever reaching a model prompt.
- D-05 (Phase 17): A single global Mimecast client (not per-tenant) was an accepted v1 limitation, documented in the module itself — later partially addressed by a post-ship quick task adding optional per-tenant client injection, though the default automatic path still uses the global client.
- D-07 (Phase 18): The EML parser is only invoked from the on-demand
/analyzeendpoint, not the automatic webhook/cron path — meaning fully automatic detection can only ever reach Tier 3 (weakest) campaign grouping until an operator has explicitly analyzed at least one report in that campaign. - D-09 (Phase 22): URLs are rendered as inert, copy-only text in the operator review UI — a deliberately stricter posture than the triage-note's sanitize-and-describe approach, because this page is one an operator might actually click around in.
6. Known Issues, Gaps, and Tech Debt
Real defects caught and fixed before/around ship:
- Phase 15 (pre-ship): The webhook trigger originally branched on an Autotask payload field that real webhooks never populate — detection would have silently never fired against real data. Fixed by reading the ticket back from Postgres instead of trusting the payload shape.
- Phase 18 (found via live production verification, 2026-07-16): Re-analyzing a single-report campaign created a second campaign row with the same grouping key, and the abandoned original campaign's report count was never decremented. Fixed with a signal-diverged guard plus a shared decrement helper; proven via 8 regression tests and a live production re-verification.
- Phase 23 (post-ship gap-closure, before milestone close): Every additional report accumulating into an already-classified simulation campaign re-triggered a duplicate customer-visible "thanks for reporting" note. Fixed via an idempotent, audit-persisting wrapper — plus two second-order bugs found in review of that very fix (a stale-timestamp display bug, and a gap where the manual approval path had no awareness of an auto-posted note). All three fixed in one commit before close.
Process gap: Phase 22 shipped all 6 plans and was marked complete without ever running formal verification — discovered only during the milestone-close requirements traceability check. A retroactive pass found the code correct (6/6 requirements), but 5 manual browser click-through checks remain outstanding: full state-machine walkthrough, approve/remediate/mark-false-positive end-to-end, non-privileged role gating, the Mimecast-unavailable rendering state, and visual confirmation of URL inertness/clipboard-copy. Treat the review UI as code-correct but not yet UI-hardened.
Accepted, still-open items:
- No retry if a webhook-triggered ticket isn't yet in Postgres when detection runs (backstop is a cron sweep that ships disabled by default).
- Webhook doesn't re-trigger detection on ticket update (deliberate, not a bug).
- No floor on the campaign-grouping report-count decrement; a pre-existing race condition in the "already grouped" check.
- The THREAT-tier "clicked > 0" escalation trigger was flagged as a reasoned proposal rather than an explicitly locked decision — never formally reconfirmed.
- Two pre-existing, unrelated failing tests (
itglue-search.test.ts) persist across the whole milestone — confirmed to predate v3.0, out of scope by convention.
5 post-ship fixes (2026-07-16 through 2026-07-18, all real production issues caught after ship):
- Mimecast blast-radius errors were silently swallowed into a false "clean" result; date-window and multi-tenant handling fixed.
- A confidence-percentage display bug showed maximum confidence (1.0) as "1% confidence" — the opposite of its true meaning.
- A real KnowBe4 campaign using rotating lookalike domains was misclassified because only one simulation-vendor domain was allowlisted; also fixed a parse double-invocation race and a too-early auto-parse timing issue.
- Added a new "mark as accidental report" resolution path (feature addition, not a bug fix).
- Mimecast's held-message lookup had no date bounds, so an entire unrelated hold queue could inflate a campaign's blast-radius numbers — fixed with date filtering and sender-relevance matching (false held-message count went from 15 → 1 → 0 across the two fixes).
7. Getting Started
Where to look first:
lib/services/phishing-detector.ts— the front door; what triggers everything downstream.lib/services/campaign-grouping-service.ts— the trickiest logic in the milestone; read its bug history (Section 6, item 2) before touching it.lib/services/campaign-classifier.ts— the deterministic rule engine.app/phishing/tickets/[ticketId]/page.tsxpluscomponents/phishing/*— the operator-facing review surface.lib/services/phishing-automation-gate.ts+app/admin/phishing-automation/page.tsx— the per-client automation on/off switch.
Run a manual sweep/analyze:
- Single ticket:
POST /api/phishing/tickets/{ticket_id}/analyze - Full reconciliation sweep:
lib/services/phishing-sweep-service.ts— wired to a daily cron schedule that ships disabled by default; enable via/adminsync schedules. - Reclassify:
POST /api/phishing/campaigns/{id}/classify
Tests: All coverage lives under lib/services/** (matching this repo's stated test-coverage convention — no app/**/components/** coverage exists anywhere in Pulse). Run npm test. Whole-repo status at milestone close: 439/441 passing (2 pre-existing, unrelated failures).
Docs: docs/mimecast-api-guide.md documents the held-message-queue date-range requirement and the gotcha behind the July 2026 false-positive fix — read before making further Mimecast changes.
Stats
- Timeline: 2026-07-15 → 2026-07-17 (3 days), plus 5 post-ship follow-up fixes through 2026-07-18
- Phases: 9 / 9 complete
- Requirements: 38 / 38 complete
- Plans / Tasks: 30 plans, 69 tasks
- Commits (repo-wide, since v2.0 close 2026-07-12): 330
- Files changed (phishing-specific source paths): 53 files, 9,382 insertions
- Tests: 439 / 441 passing repo-wide at close
- Contributors: lorentz