From 32cdf55506f6561bac9838aa503ee286786d4b13 Mon Sep 17 00:00:00 2001 From: lorentz Date: Thu, 16 Jul 2026 07:14:54 -0400 Subject: [PATCH] docs(19): capture phase context --- .../19-classification-engine/19-CONTEXT.md | 286 ++++++++++++++++++ .../19-DISCUSSION-LOG.md | 122 ++++++++ 2 files changed, 408 insertions(+) create mode 100644 .planning/phases/19-classification-engine/19-CONTEXT.md create mode 100644 .planning/phases/19-classification-engine/19-DISCUSSION-LOG.md diff --git a/.planning/phases/19-classification-engine/19-CONTEXT.md b/.planning/phases/19-classification-engine/19-CONTEXT.md new file mode 100644 index 0000000..35c119d --- /dev/null +++ b/.planning/phases/19-classification-engine/19-CONTEXT.md @@ -0,0 +1,286 @@ +# Phase 19: Classification Engine - Context + +**Gathered:** 2026-07-16 +**Status:** Ready for planning + + +## Phase Boundary + +Every campaign gets a deterministic `SPAM`/`UNWANTED`/`THREAT` verdict, built +from bounded structured evidence (never raw unbounded email) — confidence, +short summary, evidence-backed reasons, recommended actions, and a +`requires_approval` flag. Destructive recommended actions always force +`requires_approval: true`. Confidence is lowered (with the specific missing +evidence named) when evidence is incomplete. Known security-awareness- +simulation reports (KnowBe4, and Breach Secure Now if research confirms a +detectable pattern) are not classified as `THREAT` absent contrary evidence. +`POST /api/phishing/campaigns/{id}/classify` (re-)triggers classification, +following the Phase 18 auth convention, and the classifier only ever receives +structured, size-bounded evidence. + +Does NOT cover remediation execution/approval (Phase 20) or the Autotask +triage note (Phase 21) — this phase only produces the classification, it +doesn't act on it. + + + + +## Implementation Decisions + +### Architecture: Deterministic Rules Only +- **D-01:** The classifier is a pure deterministic rule engine — no + Anthropic/OpenRouter/LLM calls in this phase. Same style as + `lib/services/robotic-classifier.ts` (evidence-in → rule-eval → + verdict-out). This matches the ROADMAP goal and success criteria literally + — none of them describe an AI call. CLASSIFY-06's "before reaching any AI + layer" language is satisfied by shaping the evidence payload as bounded/ + structured now (see D-08), so a future phase could bolt on an LLM-assist + stage later without reworking the evidence shape — but this phase does not + build one. Rationale: keeps this phase tightly scoped to what the success + criteria actually verify; avoids a mini analyzer-pipeline (prompt template, + cost guard, provider-scoped idempotency) that nothing in this phase's + requirements asks for. + +### Trigger: On-Demand Only +- **D-02:** Classification runs strictly via `POST /api/phishing/campaigns/ + {id}/classify` — no automatic classification on campaign creation/update. + Matches CLASSIFY-05's wording ("operator can (re-)trigger") and the + ROADMAP's "(re-)trigger API" phrasing. Campaign grouping (Phase 18) already + runs automatically on webhook/cron; layering automatic classification on + top would mean every KnowBe4-button click silently produces a verdict no + one asked for. Each classify call inserts a new `classifications` row + (append-only history, matching Phase 18's `GET /campaigns/{id}` which + already returns "classification history" plural) — "current" verdict is + simply the most recent row by `created_at`. + +### Verdict Rules: THREAT Tier +- **D-03:** THREAT requires BOTH (a) evidence the message actually reached + someone — blast-radius shows `delivered > 0` or `clicked > 0` via the + Mimecast abstraction (`getBlastRadius()`, Phase 17) — AND (b) a malicious + signal: a hard SPF/DKIM/DMARC **fail** (not `none`/unchecked — Phase 16's + structured auth verdicts), or a known-bad indicator (attachment-hash/URL + match from Phase 16's `indicators` table). Either signal alone (e.g. auth + failure but Mimecast shows the message was fully held/blocked, reaching no + one) stays at UNWANTED — contained blast radius means it isn't a realized + threat yet. + +### Verdict Rules: SPAM vs. UNWANTED +- **D-04:** SPAM = generic bulk/commercial signals — no spoofing (auth + passes or is absent), no malicious indicator matches, sender not + impersonating anyone (e.g. a newsletter/marketing blast mistakenly + reported). UNWANTED = has a suspicious signal (an indicator match, a mild + auth issue, or Mimecast shows delivery contained to just the reporter) but + doesn't clear the THREAT bar from D-03 — the "legitimately suspicious but + not spreading or spoofed" middle tier. + +### Confidence Scoring +- **D-05:** Point-deduction from a full-evidence baseline of `1.0`. Subtract + a fixed amount per missing evidence source: no `.eml`/`messages` row + parsed, no Mimecast data (`status: 'unavailable'`), no attachment/URL + indicators found. Each deduction is named explicitly in `reasons` + (CLASSIFY-03 requires naming the specific missing evidence, not just a + lower number). Deterministic, testable, and self-explaining in the API + response. Exact deduction weights are a planner/researcher call — not a + user preference, as long as the additive-from-1.0 shape and per-source + naming are preserved. + +### KnowBe4 / Breach Secure Now Simulation Detection +- **D-06:** Detection is a **sender-domain allowlist**, config-driven (not + hardcoded inline), checked against the parsed original message's + `From`/`Return-Path` domain (Phase 16's `NormalizedMessage`) — NOT against + the report-pattern match alone. Phase 15's detector patterns + (`KNOWN_PHISHING_PATTERNS`, e.g. "Source: KnowBe4 Phish Alert Button") + only prove the REPORT arrived via a KnowBe4-button-flavored flow; they do + not prove the underlying reported message is itself a simulation — a real + attacker email reported through that same button would match the same + patterns. The allowlist checks the actual sender of the *original message* + instead. If it matches, force verdict to SPAM/UNWANTED and skip THREAT + regardless of other signals (D-03/D-04 still decide which of the two). +- **D-07 (added mid-discussion):** The user flagged a second security- + awareness-training vendor, **Breach Secure Now (BSN)**, and is not sure how + BSN test tickets are currently marked/tagged — nothing in the codebase + references BSN today (confirmed: zero matches for "Breach Secure Now" or + "BSN" across `*.ts`/`*.md`/`*.sql`). Decision: **research must search + existing Autotask ticket history** for real BSN-originated report examples + (title/body/sender patterns — the same way Phase 15's 8 KnowBe4 patterns + were presumably derived from real tickets) and fold a BSN sender-domain + entry into this phase's allowlist if enough signal exists. The allowlist + structure (D-06) must support multiple vendors from day one so BSN (or any + future vendor) is a config addition, not a rework, even if research can't + confirm a BSN domain in time for this phase. + +### Recommended Actions Vocabulary +- **D-08:** Seven action types, split into two tiers: + - **Non-destructive** (`requires_approval` not forced by the action + itself): `no_action` (SPAM, nothing to do), `warn_user` (UNWANTED, remind + reporter), `disable_forwarding_rule` (easily reversible — toggle back on, + doesn't force approval on its own). + - **Destructive** (always `requires_approval: true` per CLASSIFY-02): + `block_sender`, `purge_message`, `reset_password`, `isolate_endpoint` + (RMM-driven device isolation — reaches into Datto RMM territory; Phase 20 + determines the actual provider path, this phase only needs the + vocabulary and the approval-forcing invariant). + This vocabulary becomes Phase 20's `remediation_actions.action_type` enum + — locking it now avoids a rename later. `disable_forwarding_rule` being + non-destructive is a deliberate exception the invariant test (success + criterion #2) must still hold for: recommending it alone must NOT force + `requires_approval: true`, while recommending it alongside any destructive + action still does (the flag is OR'd across all recommended actions, not + per-action). + +### Claude's Discretion (explicitly deferred to research + planner) +- Exact point-deduction weights for confidence scoring (D-05) — as long as + the shape (additive from 1.0, per-source naming) holds. +- Exact JSON shape of `reasons` / `recommended_actions` in the API response + and `classifications` row — follow the project's camelCase API response + convention; `reasons`/`recommended_actions` are already JSONB columns in + migration 097. +- Whether the KnowBe4/BSN sender-domain allowlist lives as a TypeScript + constant (mirroring `KNOWN_PHISHING_PATTERNS` in `phishing-detector.ts`) or + a small config/DB table — planner's call, but must support multiple + vendors (D-07) and be trivially extendable. +- Exact evaluation order/precedence when multiple rules could apply + (e.g. simulation-allowlist match vs. THREAT-tier signals) — the allowlist + match (D-06) short-circuits before D-03/D-04 tier evaluation per the + decisions above; finer implementation ordering is planner's call. +- How `isolate_endpoint`'s "configured provider path" question interacts + with Phase 20 — Phase 19 only needs to recommend it and mark it + destructive; Phase 20 owns whether/how it's actually executable. + + + + +## Canonical References + +**Downstream agents MUST read these before planning or implementing.** + +### Evidence inputs (this phase reads, does not modify) +- `lib/services/mimecast-blast-radius.ts` — `getBlastRadius()`, + `BlastRadiusResult` (`status: 'ok' | 'unavailable'`, `matched`/`delivered`/ + `held`/`rejected`/`clicked`/`perRecipient`) — the delivery-signal input for + D-03. Ephemeral per Phase 17 D-03 — this phase persists whatever it needs + into `classifications.reasons`. +- `lib/services/eml-parser.ts` — `NormalizedMessage` (headers incl. From/ + Return-Path, structured SPF/DKIM/DMARC verdicts per Phase 16 D-06) — the + auth-failure input for D-03 and the sender-domain input for D-06. +- `lib/services/phishing-eml-service.ts` — writes `indicators` rows + (`indicator_type`: `'attachment_hash'`, `'url'`, `'sender'`) — the + known-bad-indicator input for D-03. +- `lib/services/phishing-detector.ts` — `KNOWN_PHISHING_PATTERNS`, + `matchesPhishingPatterns()` — the EXISTING report-pattern match this + phase's KnowBe4 detection must NOT rely on alone (see D-06 rationale). +- `lib/services/campaign-grouping-service.ts` — campaign/report shape this + phase reads to gather all linked reports/messages/indicators for a + campaign before classifying. + +### Schema (this phase writes) +- `migrations/097_phishing_triage_schema.sql` — `classifications` table + (`verdict`, `confidence` NUMERIC, `summary`, `reasons` JSONB, + `recommended_actions` JSONB, `requires_approval` BOOLEAN NOT NULL DEFAULT + false) — append-only per campaign (D-02); `remediation_actions.action_type` + TEXT column is the eventual home for D-08's vocabulary (Phase 20 concern, + but the enum values should stay consistent with what this phase + recommends). + +### Auth/permissions (established by Phase 18, reused here) +- `lib/permissions.ts` — `phishing` resource; `analyze` action already + granted to `superAdminRole`/`adminRole` (Phase 18 D-05). This phase's + `POST /classify` route should use `requirePermission('phishing', + 'analyze')` — same action as the existing `/analyze` route, not a new + action — per Phase 18 D-06's established per-route permission-check + convention. +- `lib/auth-utils.ts` — `requirePermission(resource, action)` — see + `app/api/phishing/tickets/[ticket_id]/analyze/route.ts` for the exact + call-and-early-return pattern already used in this milestone. + +### Deterministic-classifier precedent (this phase's architectural analog) +- `lib/services/robotic-classifier.ts` — `evaluateContains()` and the + `ClassificationRule` evaluation loop — the existing deterministic, + `.toLowerCase().includes()`-only (no regex/eval) rule-evaluation style + this phase's classifier should follow per D-01. + +### Redaction path (if IT Glue evidence is ever referenced, per CLASSIFY-06) +- `lib/services/analyzer/itglue-search.ts` — the ONLY approved path for any + IT Glue-sourced data if this phase's evidence gathering ever pulls IT Glue + content in; raw `itglue-client.ts` output must never flow into evidence + passed to the classifier or the API response. + +### Prior phase decisions (for consistency) +- `.planning/phases/18-campaign-grouping-phishing-analysis-api/18-CONTEXT.md` + (D-05, D-06 — permission vocabulary + per-route `requirePermission` + convention this phase must follow, not re-litigate) +- `.planning/phases/17-mimecast-blast-radius-lookup/17-CONTEXT.md` (D-03 — + blast-radius is ephemeral, this phase owns persistence) +- `.planning/phases/16-eml-mime-evidence-parser/16-CONTEXT.md` (D-06 — + structured SPF/DKIM/DMARC verdicts, not raw header text, are available) + + + + +## Existing Code Insights + +### Reusable Assets +- `robotic-classifier.ts`'s `evaluateContains()` pattern — precedent for a + small, pure, testable rule-evaluation function; this phase's verdict rules + should follow the same no-regex, deterministic style. +- `phishing-detector.ts`'s `KNOWN_PHISHING_PATTERNS` constant — precedent + for a locked, named pattern list; the KnowBe4/BSN sender-domain allowlist + (D-06/D-07) should follow the same "small exported constant" shape unless + planning finds a reason to make it DB-backed. +- `getBlastRadius()`'s `BlastRadiusResult` discriminated union (`status: 'ok' + | 'unavailable'`) — precedent for how this phase should shape its own + evidence-completeness checks (D-05) — presence/absence of a `status: 'ok'` + result is itself one of the confidence-deduction triggers. + +### Established Patterns +- Factory/service convention: pure functions in `lib/services/`, no classes + needed for stateless evaluation logic (see `mimecast-blast-radius.ts`, + `campaign-grouping-service.ts`). +- `requirePermission(resource, action)` early-return pattern for every + `/api/phishing/*` route (Phase 18 D-06) — this phase's `/classify` route + follows the same shape. +- Append-only history tables with a `campaign_id` FK + `created_at` — same + shape as how `classifications` should be written (D-02), no upsert. + +### Integration Points +- This phase's classifier reads: `campaigns` row, all linked `reports`, + `messages` (parsed .eml data), `indicators`, and a fresh `getBlastRadius()` + call — assembling one bounded evidence payload before rule evaluation. +- Writes: one new `classifications` row per `/classify` call. +- Phase 20 reads `classifications.recommended_actions` / + `requires_approval` to know what it's gating approval on — this phase's + D-08 vocabulary is a direct contract with Phase 20's `remediation_actions` + work. + + + + +## Specific Ideas + +- The user specifically wants Breach Secure Now (BSN) simulation reports + recognized eventually, alongside KnowBe4 — flagged mid-discussion as a + real gap (nothing in the codebase references BSN today). Research should + actively look for BSN ticket examples in Autotask history before planning + locks in the KnowBe4/BSN allowlist (D-07). + + + + +## Deferred Ideas + +- **BSN sender-domain confirmation** — if research finds no usable BSN + pattern in existing ticket history, ship this phase with a KnowBe4-only + allowlist (structured to support more vendors per D-07) and treat adding + BSN as a fast follow-up once a real example ticket is available. +- **`isolate_endpoint` provider wiring** — Phase 19 only recommends and + flags it destructive; which RMM/provider path actually executes it is + Phase 20's concern. +- **Non-BSN/KnowBe4 discussion scope creep** — none surfaced; discussion + stayed within the classification-engine boundary throughout. + + + +--- + +*Phase: 19-classification-engine* +*Context gathered: 2026-07-16* diff --git a/.planning/phases/19-classification-engine/19-DISCUSSION-LOG.md b/.planning/phases/19-classification-engine/19-DISCUSSION-LOG.md new file mode 100644 index 0000000..5cc511d --- /dev/null +++ b/.planning/phases/19-classification-engine/19-DISCUSSION-LOG.md @@ -0,0 +1,122 @@ +# Phase 19: Classification Engine - Discussion Log + +> **Audit trail only.** Do not use as input to planning, research, or execution agents. +> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered. + +**Date:** 2026-07-16 +**Phase:** 19-classification-engine +**Areas discussed:** Rules-only vs. rules+LLM, Verdict signal rules, KnowBe4 simulation handling, Recommended actions vocabulary + +--- + +## Rules-only vs. rules+LLM + +| Option | Description | Selected | +|--------|-------------|----------| +| Deterministic rules only | No AI calls this phase; matches ROADMAP/success criteria literally | ✓ | +| Deterministic core + LLM-assist | Rules + Anthropic/OpenRouter stage for low-confidence verdicts, mini analyzer-pipeline style | | +| Something else | — | | + +**User's choice:** Deterministic rules only. + +| Option | Description | Selected | +|--------|-------------|----------| +| On-demand only via POST /classify | Matches CLASSIFY-05/ROADMAP "(re-)trigger" wording | ✓ | +| Auto-classify on campaign creation | Mirrors Phase 18's automatic grouping trigger | | +| Something else | — | | + +**User's choice:** On-demand only via POST /classify. +**Notes:** No automatic classification pass on campaign creation — grouping already runs automatically (Phase 18); layering auto-classification on top would produce unrequested verdicts on every KnowBe4-button click. + +--- + +## Verdict signal rules + +| Option | Description | Selected | +|--------|-------------|----------| +| Delivery + auth-failure combo | THREAT needs both delivered/clicked>0 AND a hard auth fail or bad indicator | ✓ | +| Any hard auth failure alone | Spoofing alone is sufficient regardless of delivery | | +| Something else | — | | + +**User's choice:** Delivery + auth-failure combo. + +| Option | Description | Selected | +|--------|-------------|----------| +| SPAM = bulk/commercial, UNWANTED = targeted-but-contained | Content-signal-driven tier boundary | ✓ | +| SPAM = single-report, UNWANTED = multi-report campaign | Accumulation-driven tier boundary | | +| Something else | — | | + +**User's choice:** SPAM = bulk/commercial, UNWANTED = targeted-but-contained. + +| Option | Description | Selected | +|--------|-------------|----------| +| Point-deduction from 1.0 | Subtract fixed amount per missing evidence source, name each in reasons | ✓ | +| Discrete tiers (high/medium/low) | Coarser bucket-based confidence | | +| Something else | — | | + +**User's choice:** Point-deduction from 1.0. + +--- + +## KnowBe4 simulation handling + +| Option | Description | Selected | +|--------|-------------|----------| +| Sender-domain allowlist | Check parsed .eml From/Return-Path domain against known simulation-vendor domains | ✓ | +| Trust the report-pattern match alone | Rely on Phase 15's button/report-flow pattern match only | | +| Something else (free text) | User raised Breach Secure Now (BSN) as a second vendor, unsure how it's marked | (led to follow-up) | + +**User's choice (initial "Other"):** "We also have Breach Secure Now Tests - not sure how they are marked -" +**Notes:** Confirmed via grep that nothing in the codebase references "Breach Secure Now" or "BSN" today. Raised as a genuine gap in CLASSIFY-04 coverage — followed up with a dedicated question before finalizing the KnowBe4 question. + +| Option | Description | Selected | +|--------|-------------|----------| +| Defer BSN as follow-up | Scope CLASSIFY-04 to KnowBe4 only this phase, structure allowlist for easy extension | | +| Have research dig into ticket history now | Search existing Autotask tickets for real BSN patterns before planning locks in rules | ✓ | +| Something else | — | | + +**User's choice:** Have research dig into ticket history now. + +| Option | Description | Selected | +|--------|-------------|----------| +| Sender-domain allowlist (config-driven, multi-vendor) | Match against original message's sender domain, not report-pattern alone | ✓ | +| Trust the report-pattern match alone | — | | +| Something else | — | | + +**User's choice:** Sender-domain allowlist (config-driven, multi-vendor). + +--- + +## Recommended actions vocabulary + +| Option | Description | Selected | +|--------|-------------|----------| +| Core set: no_action, warn_user, block_sender, purge_message, reset_password | 5 actions, 2 non-destructive / 3 destructive | | +| Broader set: + disable_forwarding_rule, isolate_endpoint | 7 actions total | ✓ | +| Something else | — | | + +**User's choice:** Broader set (7 actions). + +| Option | Description | Selected | +|--------|-------------|----------| +| Yes, both destructive | disable_forwarding_rule + isolate_endpoint both require approval | | +| isolate_endpoint destructive, disable_forwarding_rule not | Reversibility-based split | ✓ | +| Something else | — | | + +**User's choice:** isolate_endpoint destructive, disable_forwarding_rule not (easily reversible). +**Notes:** Final destructive set: block_sender, purge_message, reset_password, isolate_endpoint. Non-destructive: no_action, warn_user, disable_forwarding_rule. The requires_approval invariant (success criterion #2) is OR'd across all recommended actions — recommending disable_forwarding_rule alone must not force approval, but combined with any destructive action it still does. + +--- + +## Claude's Discretion + +- Exact point-deduction weights for confidence scoring. +- Exact JSON shape of `reasons`/`recommended_actions` fields (camelCase API convention; JSONB columns already exist). +- Whether the KnowBe4/BSN allowlist is a TS constant or config/DB table (must support multiple vendors). +- Exact rule-evaluation order/precedence beyond "simulation allowlist short-circuits before tier evaluation." +- How `isolate_endpoint`'s actual provider path gets wired — explicitly Phase 20's concern. + +## Deferred Ideas + +- BSN sender-domain confirmation, if research finds no usable pattern — ship KnowBe4-only allowlist structured for easy extension, fast-follow BSN later. +- `isolate_endpoint` provider/execution wiring — Phase 20.