From f0c09927f4f18c4bc06eb47d3d56e65883f43dd9 Mon Sep 17 00:00:00 2001 From: lorentz Date: Sat, 18 Jul 2026 11:20:31 -0400 Subject: [PATCH] docs(v3.0): generate milestone summary for onboarding --- .planning/reports/MILESTONE_SUMMARY-v3.0.md | 156 ++++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 .planning/reports/MILESTONE_SUMMARY-v3.0.md diff --git a/.planning/reports/MILESTONE_SUMMARY-v3.0.md b/.planning/reports/MILESTONE_SUMMARY-v3.0.md new file mode 100644 index 0000000..4206e36 --- /dev/null +++ b/.planning/reports/MILESTONE_SUMMARY-v3.0.md @@ -0,0 +1,156 @@ +# Milestone v3.0 — Phishing Triage Automation — Project Summary + +**Generated:** 2026-07-18 +**Purpose:** Team onboarding and project review + +--- + +## 1. Project Overview + +Pulse is Wulf Consulting's internal PSA management dashboard (Next.js 16 + Postgres, syncing Autotask data). The v3.0 milestone's core value proposition: + +> A manager/security operator can see every phishing/spam report ticket automatically triaged, deduplicated into campaigns, and classified — with any destructive remediation gated behind explicit human approval. + +**Problem it solves:** Wulf's technicians receive a constant stream of Autotask tickets from KnowBe4 phish-alert-button reports, Microsoft's built-in "Report Message" flow, and manual spam/phishing complaints. Historically each ticket was triaged by hand — no deduplication across employees reporting the same campaign, no consistent severity call, no blast-radius visibility into Mimecast, and remediation notes/actions were ad hoc. + +**Who it's for:** Wulf's security operators/technicians (triaging tickets day to day) and admins (configuring per-client automation). + +**What was built:** An end-to-end pipeline — detect phishing/spam-report tickets → extract `.eml` evidence → look up Mimecast blast radius → deterministically classify (SPAM/UNWANTED/THREAT/USER_AWARENESS) → group duplicates into campaigns → propose (never auto-execute) remediation behind human approval → post a sanitized Autotask triage note → surface it all on a ticket-ID-addressable review page reachable via a real, production-confirmed Autotask LiveLink button → optionally automate the whole chain per-client via an opt-in gate. + +**Status:** Shipped 2026-07-17. 9 phases, 30 plans, 69 tasks. All 38 v1 requirements complete. + +**Explicit non-goals for this milestone** (deferred to a future v2): +- No automatic execution of remediation — every action (block sender, purge mailbox, revoke sessions, etc.) is *proposed* and *simulated* only; nothing destructive runs against a real mail/identity system yet (`REMEDEXEC-01..05`). +- No URL sandbox/detonation or URL reputation lookup (`ENRICH-01`). +- No LLM-backed classification layer — this milestone is intentionally deterministic (`ENRICH-02` reserves the interface for later). +- No fully automated ticket closure, no auto-created parent incidents, no auto-closed duplicates. +- No backfill of historical phishing tickets (~267 pre-existing tickets) — detection is forward-only from ship date. + +--- + +## 2. Architecture & Technical Decisions + +| Decision | Why | Phase | +|---|---|---| +| **Zero-LLM deterministic classifier** | Structured, size-bounded evidence in → deterministic verdict out. Avoids a prompt-injection surface from attacker-controlled email content ever reaching an LLM. | 19 | +| **Content-hash idempotency scoped to title+description only** | SHA-256 over `{title, description}` — excludes status/assignee/last-activity so routine ticket churn never re-triggers detection, while evidence (notes/time-entries/attachments) still refreshes on every rescan regardless of hash match. | 15 | +| **3-tier `.eml` selection** (`rfc.eml` exact → single non-wrapper `message/rfc822` candidate → `OriginatingEmail.eml` fallback) | Empirically derived from sampling 15 real production phishing tickets — content-type alone never disambiguates, since every sampled attachment shares `message/rfc822`. | 16 | +| **Hand-rolled SPF/DKIM/DMARC tokenizer**, not a library | The obvious library (`mailauth`) only exposes live-verification functions that perform DNS lookups and BIMI HTTP fetches — a hard violation of "never fetch anything from a message." | 16 | +| **3-tier campaign grouping key**: Message-ID → attachment-hash/URL-domain + subject + sender + 24h window → sender + normalized-subject + client + 24h window | Message-ID is the strongest, cheapest key; the fallbacks catch mass-blast duplicates lacking a shared Message-ID. Campaigns are never merged after the fact — out of scope by design. | 18 | +| **Mimecast blast-radius as a normalizing abstraction that never throws** | Unconditional fan-out over delivered/held/threat-event lookups, 5-minute Redis cache, degrades to an explicit `unavailable` state on any failure or missing config — the rest of the pipeline never blocks on Mimecast being down. | 17 | +| **THREAT requires BOTH delivery AND a malicious signal** | An auth failure alone on a message Mimecast fully held (reached no one) isn't a realized threat. Malicious signal = hard SPF/DKIM/DMARC fail OR the same attachment-hash/URL recurring across ≥2 campaign reports. | 19 | +| **Simulation-vendor allowlist is a code constant, not a DB table** | Every classification rule, including the allowlist, is unit-tested pure code rather than a runtime-editable table that could silently drift. Matches exact domain or proper subdomain only — never substring — so a lookalike domain can't slip through. | 19 | +| **Confidence = additive point-deduction from 1.0** | Each deduction (no parsed message, no/failed Mimecast lookup, no attachment/URL indicators) names the specific missing evidence in the response, rather than an opaque score. | 19 | +| **All remediation actions are simulated status-only transitions this milestone** | No real provider methods exist yet (no block/purge on the Mimecast client, no forwarding-rule/reset on the Graph client) — real execution deliberately deferred to a future v2. | 20 | +| **`acknowledge_user` is the one non-destructive action carve-out** | It's a customer-visible "thanks for reporting" note, not a security action — the only action type the automation gate is allowed to auto-post without human approval. | 19/23 | +| **Per-company automation gate is 3 independent opt-in booleans** (parse/classify/report), all off by default | Mirrors the "propose, don't execute" safety posture — a client only gets automation once an admin deliberately opts them in. | 23 | +| **URLs render as inert copy-only text in the review UI, never a clickable link** | Stricter than the triage-note's sanitize-and-describe approach, because this is an interactive page an operator could actually click into. | 22 | +| **Client-side permission checks call the exact same permission function the server uses** | No bespoke role checks in the UI — the server remains the sole real enforcement boundary; the client check is UX-only. | 22 | + +--- + +## 3. Phases Delivered + +| Phase | Name | Status | One-Liner | +|-------|------|--------|-----------| +| 15 | Data Model, Detection & Ticket Evidence | ✅ Complete | Durable 7-table schema + idempotent Autotask ticket scanner and base evidence capture | +| 16 | EML/MIME Evidence Parser | ✅ Complete | Pure, I/O-free RFC822/MIME parser that never executes or fetches anything | +| 17 | Mimecast Blast Radius Lookup | ✅ Complete | Normalized delivery-data abstraction that degrades gracefully when Mimecast isn't configured | +| 18 | Campaign Grouping & Phishing Analysis API | ✅ Complete | Duplicate reports accumulate into campaigns automatically; first `/api/phishing/*` routes | +| 19 | Classification Engine | ✅ Complete | Deterministic SPAM/UNWANTED/THREAT verdicts from bounded evidence — no LLM anywhere | +| 20 | Remediation, Approval & Audit Safety | ✅ Complete | Proposed-only actions, permission-gated approve/remediate/mark-false-positive, fully audited | +| 21 | Autotask Triage Note | ✅ Complete | Sanitized, human-readable internal note posted to every linked ticket | +| 22 | Approval UI (LiveLink) | ✅ Complete* | Ticket-ID-addressable page, confirmed live as a real Autotask LiveLink target | +| 23 | Classification Disposition + Per-Client Automation Gate | ✅ Complete | Dedicated non-destructive verdict for simulation-vendor reports + per-company opt-in automation | + +*Phase 22 is code-complete and verified retroactively, but 5 manual browser click-through checks were never run — see Section 6. + +--- + +## 4. Requirements Coverage + +All 38 v1 requirements are marked **Complete** in `v3.0-REQUIREMENTS.md`. + +| ID | Phase | Detail | +|---|---|---| +| DETECT-01/02/03 | 15/18 | Pattern match on 8 locked substrings; content-hash idempotency; on-demand single-ticket analyze endpoint | +| EVID-01..04 | 15/16 | Ticket evidence capture; 3-tier `.eml` selection; full RFC822/MIME + auth-result normalization; sanitized body preview, zero-network invariant test-enforced | +| CAMP-01..03 | 18 | 3-tier grouping key; report/recipient accumulation; campaign list + detail API | +| BLAST-01/02 | 17 | Normalized blast-radius fan-out; graceful `unavailable` degradation | +| CLASSIFY-01..06 | 19 | Verdict + confidence + reasons + actions; destructive-action invariant; evidence-naming confidence deductions; sender allowlist; classify API; bounded structured evidence only | +| REMED-01..06 | 20 | No auto-execution path exists (grep-confirmed); approve API; remediate API (simulated); idempotent double-call handling; mark-false-positive with 409 guard; single audit writer | +| NOTE-01 | 21 | Sanitized triage note via existing safe Autotask write path, per-ticket failure isolation | +| ACCESS-01 | 18 | `requirePermission()` convention applied consistently from Phase 18 onward | +| REVIEW-01..06 | 22 | Ticket-addressable route; timeline; evidence display; classification display; action area wired to APIs; client/server permission parity — all code-verified, live-browser pass outstanding | +| CLASSDISP-01..03 | 23 | `USER_AWARENESS` verdict; customer-visible acknowledgment note; distinct UI badge | +| AUTOGATE-01..03 | 23 | Automation-gate table + admin API; admin toggle UI; gated webhook chain (idempotency bug found and fixed pre-close) | + +--- + +## 5. Key Decisions Log + +See Section 2 for the full architecture decision table. Notable individual calls worth calling out for new contributors: + +- **D-01 (Phase 19):** Deterministic classifier chosen over an LLM specifically to avoid attacker-controlled email content ever reaching a model prompt. +- **D-05 (Phase 17):** A single global Mimecast client (not per-tenant) was an accepted v1 limitation, documented in the module itself — later partially addressed by a post-ship quick task adding optional per-tenant client injection, though the default automatic path still uses the global client. +- **D-07 (Phase 18):** The EML parser is only invoked from the on-demand `/analyze` endpoint, not the automatic webhook/cron path — meaning fully automatic detection can only ever reach Tier 3 (weakest) campaign grouping until an operator has explicitly analyzed at least one report in that campaign. +- **D-09 (Phase 22):** URLs are rendered as inert, copy-only text in the operator review UI — a deliberately stricter posture than the triage-note's sanitize-and-describe approach, because this page is one an operator might actually click around in. + +--- + +## 6. Known Issues, Gaps, and Tech Debt + +**Real defects caught and fixed before/around ship:** + +1. **Phase 15 (pre-ship):** The webhook trigger originally branched on an Autotask payload field that real webhooks never populate — detection would have silently never fired against real data. Fixed by reading the ticket back from Postgres instead of trusting the payload shape. +2. **Phase 18 (found via live production verification, 2026-07-16):** Re-analyzing a single-report campaign created a *second* campaign row with the same grouping key, and the abandoned original campaign's report count was never decremented. Fixed with a signal-diverged guard plus a shared decrement helper; proven via 8 regression tests and a live production re-verification. +3. **Phase 23 (post-ship gap-closure, before milestone close):** Every additional report accumulating into an already-classified simulation campaign re-triggered a duplicate customer-visible "thanks for reporting" note. Fixed via an idempotent, audit-persisting wrapper — plus two second-order bugs found in review of that very fix (a stale-timestamp display bug, and a gap where the *manual* approval path had no awareness of an auto-posted note). All three fixed in one commit before close. + +**Process gap:** Phase 22 shipped all 6 plans and was marked complete without ever running formal verification — discovered only during the milestone-close requirements traceability check. A retroactive pass found the code correct (6/6 requirements), but **5 manual browser click-through checks remain outstanding**: full state-machine walkthrough, approve/remediate/mark-false-positive end-to-end, non-privileged role gating, the Mimecast-unavailable rendering state, and visual confirmation of URL inertness/clipboard-copy. Treat the review UI as code-correct but not yet UI-hardened. + +**Accepted, still-open items:** +- No retry if a webhook-triggered ticket isn't yet in Postgres when detection runs (backstop is a cron sweep that ships **disabled by default**). +- Webhook doesn't re-trigger detection on ticket update (deliberate, not a bug). +- No floor on the campaign-grouping report-count decrement; a pre-existing race condition in the "already grouped" check. +- The THREAT-tier "clicked > 0" escalation trigger was flagged as a reasoned proposal rather than an explicitly locked decision — never formally reconfirmed. +- Two pre-existing, unrelated failing tests (`itglue-search.test.ts`) persist across the whole milestone — confirmed to predate v3.0, out of scope by convention. + +**5 post-ship fixes** (2026-07-16 through 2026-07-18, all real production issues caught after ship): +1. Mimecast blast-radius errors were silently swallowed into a false "clean" result; date-window and multi-tenant handling fixed. +2. A confidence-percentage display bug showed maximum confidence (1.0) as "1% confidence" — the opposite of its true meaning. +3. A real KnowBe4 campaign using rotating lookalike domains was misclassified because only one simulation-vendor domain was allowlisted; also fixed a parse double-invocation race and a too-early auto-parse timing issue. +4. Added a new "mark as accidental report" resolution path (feature addition, not a bug fix). +5. Mimecast's held-message lookup had no date bounds, so an entire unrelated hold queue could inflate a campaign's blast-radius numbers — fixed with date filtering and sender-relevance matching (false held-message count went from 15 → 1 → 0 across the two fixes). + +--- + +## 7. Getting Started + +**Where to look first:** +- `lib/services/phishing-detector.ts` — the front door; what triggers everything downstream. +- `lib/services/campaign-grouping-service.ts` — the trickiest logic in the milestone; read its bug history (Section 6, item 2) before touching it. +- `lib/services/campaign-classifier.ts` — the deterministic rule engine. +- `app/phishing/tickets/[ticketId]/page.tsx` plus `components/phishing/*` — the operator-facing review surface. +- `lib/services/phishing-automation-gate.ts` + `app/admin/phishing-automation/page.tsx` — the per-client automation on/off switch. + +**Run a manual sweep/analyze:** +- Single ticket: `POST /api/phishing/tickets/{ticket_id}/analyze` +- Full reconciliation sweep: `lib/services/phishing-sweep-service.ts` — wired to a daily cron schedule that **ships disabled by default**; enable via `/admin` sync schedules. +- Reclassify: `POST /api/phishing/campaigns/{id}/classify` + +**Tests:** All coverage lives under `lib/services/**` (matching this repo's stated test-coverage convention — no `app/**`/`components/**` coverage exists anywhere in Pulse). Run `npm test`. Whole-repo status at milestone close: 439/441 passing (2 pre-existing, unrelated failures). + +**Docs:** `docs/mimecast-api-guide.md` documents the held-message-queue date-range requirement and the gotcha behind the July 2026 false-positive fix — read before making further Mimecast changes. + +--- + +## Stats + +- **Timeline:** 2026-07-15 → 2026-07-17 (3 days), plus 5 post-ship follow-up fixes through 2026-07-18 +- **Phases:** 9 / 9 complete +- **Requirements:** 38 / 38 complete +- **Plans / Tasks:** 30 plans, 69 tasks +- **Commits (repo-wide, since v2.0 close 2026-07-12):** 330 +- **Files changed (phishing-specific source paths):** 53 files, 9,382 insertions +- **Tests:** 439 / 441 passing repo-wide at close +- **Contributors:** lorentz