docs(19): capture phase context

This commit is contained in:
lorentz 2026-07-16 07:14:54 -04:00
parent b305dd5108
commit 32cdf55506
2 changed files with 408 additions and 0 deletions

View file

@ -0,0 +1,286 @@
# Phase 19: Classification Engine - Context
**Gathered:** 2026-07-16
**Status:** Ready for planning
<domain>
## Phase Boundary
Every campaign gets a deterministic `SPAM`/`UNWANTED`/`THREAT` verdict, built
from bounded structured evidence (never raw unbounded email) — confidence,
short summary, evidence-backed reasons, recommended actions, and a
`requires_approval` flag. Destructive recommended actions always force
`requires_approval: true`. Confidence is lowered (with the specific missing
evidence named) when evidence is incomplete. Known security-awareness-
simulation reports (KnowBe4, and Breach Secure Now if research confirms a
detectable pattern) are not classified as `THREAT` absent contrary evidence.
`POST /api/phishing/campaigns/{id}/classify` (re-)triggers classification,
following the Phase 18 auth convention, and the classifier only ever receives
structured, size-bounded evidence.
Does NOT cover remediation execution/approval (Phase 20) or the Autotask
triage note (Phase 21) — this phase only produces the classification, it
doesn't act on it.
</domain>
<decisions>
## Implementation Decisions
### Architecture: Deterministic Rules Only
- **D-01:** The classifier is a pure deterministic rule engine — no
Anthropic/OpenRouter/LLM calls in this phase. Same style as
`lib/services/robotic-classifier.ts` (evidence-in → rule-eval →
verdict-out). This matches the ROADMAP goal and success criteria literally
— none of them describe an AI call. CLASSIFY-06's "before reaching any AI
layer" language is satisfied by shaping the evidence payload as bounded/
structured now (see D-08), so a future phase could bolt on an LLM-assist
stage later without reworking the evidence shape — but this phase does not
build one. Rationale: keeps this phase tightly scoped to what the success
criteria actually verify; avoids a mini analyzer-pipeline (prompt template,
cost guard, provider-scoped idempotency) that nothing in this phase's
requirements asks for.
### Trigger: On-Demand Only
- **D-02:** Classification runs strictly via `POST /api/phishing/campaigns/
{id}/classify` — no automatic classification on campaign creation/update.
Matches CLASSIFY-05's wording ("operator can (re-)trigger") and the
ROADMAP's "(re-)trigger API" phrasing. Campaign grouping (Phase 18) already
runs automatically on webhook/cron; layering automatic classification on
top would mean every KnowBe4-button click silently produces a verdict no
one asked for. Each classify call inserts a new `classifications` row
(append-only history, matching Phase 18's `GET /campaigns/{id}` which
already returns "classification history" plural) — "current" verdict is
simply the most recent row by `created_at`.
### Verdict Rules: THREAT Tier
- **D-03:** THREAT requires BOTH (a) evidence the message actually reached
someone — blast-radius shows `delivered > 0` or `clicked > 0` via the
Mimecast abstraction (`getBlastRadius()`, Phase 17) — AND (b) a malicious
signal: a hard SPF/DKIM/DMARC **fail** (not `none`/unchecked — Phase 16's
structured auth verdicts), or a known-bad indicator (attachment-hash/URL
match from Phase 16's `indicators` table). Either signal alone (e.g. auth
failure but Mimecast shows the message was fully held/blocked, reaching no
one) stays at UNWANTED — contained blast radius means it isn't a realized
threat yet.
### Verdict Rules: SPAM vs. UNWANTED
- **D-04:** SPAM = generic bulk/commercial signals — no spoofing (auth
passes or is absent), no malicious indicator matches, sender not
impersonating anyone (e.g. a newsletter/marketing blast mistakenly
reported). UNWANTED = has a suspicious signal (an indicator match, a mild
auth issue, or Mimecast shows delivery contained to just the reporter) but
doesn't clear the THREAT bar from D-03 — the "legitimately suspicious but
not spreading or spoofed" middle tier.
### Confidence Scoring
- **D-05:** Point-deduction from a full-evidence baseline of `1.0`. Subtract
a fixed amount per missing evidence source: no `.eml`/`messages` row
parsed, no Mimecast data (`status: 'unavailable'`), no attachment/URL
indicators found. Each deduction is named explicitly in `reasons`
(CLASSIFY-03 requires naming the specific missing evidence, not just a
lower number). Deterministic, testable, and self-explaining in the API
response. Exact deduction weights are a planner/researcher call — not a
user preference, as long as the additive-from-1.0 shape and per-source
naming are preserved.
### KnowBe4 / Breach Secure Now Simulation Detection
- **D-06:** Detection is a **sender-domain allowlist**, config-driven (not
hardcoded inline), checked against the parsed original message's
`From`/`Return-Path` domain (Phase 16's `NormalizedMessage`) — NOT against
the report-pattern match alone. Phase 15's detector patterns
(`KNOWN_PHISHING_PATTERNS`, e.g. "Source: KnowBe4 Phish Alert Button")
only prove the REPORT arrived via a KnowBe4-button-flavored flow; they do
not prove the underlying reported message is itself a simulation — a real
attacker email reported through that same button would match the same
patterns. The allowlist checks the actual sender of the *original message*
instead. If it matches, force verdict to SPAM/UNWANTED and skip THREAT
regardless of other signals (D-03/D-04 still decide which of the two).
- **D-07 (added mid-discussion):** The user flagged a second security-
awareness-training vendor, **Breach Secure Now (BSN)**, and is not sure how
BSN test tickets are currently marked/tagged — nothing in the codebase
references BSN today (confirmed: zero matches for "Breach Secure Now" or
"BSN" across `*.ts`/`*.md`/`*.sql`). Decision: **research must search
existing Autotask ticket history** for real BSN-originated report examples
(title/body/sender patterns — the same way Phase 15's 8 KnowBe4 patterns
were presumably derived from real tickets) and fold a BSN sender-domain
entry into this phase's allowlist if enough signal exists. The allowlist
structure (D-06) must support multiple vendors from day one so BSN (or any
future vendor) is a config addition, not a rework, even if research can't
confirm a BSN domain in time for this phase.
### Recommended Actions Vocabulary
- **D-08:** Seven action types, split into two tiers:
- **Non-destructive** (`requires_approval` not forced by the action
itself): `no_action` (SPAM, nothing to do), `warn_user` (UNWANTED, remind
reporter), `disable_forwarding_rule` (easily reversible — toggle back on,
doesn't force approval on its own).
- **Destructive** (always `requires_approval: true` per CLASSIFY-02):
`block_sender`, `purge_message`, `reset_password`, `isolate_endpoint`
(RMM-driven device isolation — reaches into Datto RMM territory; Phase 20
determines the actual provider path, this phase only needs the
vocabulary and the approval-forcing invariant).
This vocabulary becomes Phase 20's `remediation_actions.action_type` enum
— locking it now avoids a rename later. `disable_forwarding_rule` being
non-destructive is a deliberate exception the invariant test (success
criterion #2) must still hold for: recommending it alone must NOT force
`requires_approval: true`, while recommending it alongside any destructive
action still does (the flag is OR'd across all recommended actions, not
per-action).
### Claude's Discretion (explicitly deferred to research + planner)
- Exact point-deduction weights for confidence scoring (D-05) — as long as
the shape (additive from 1.0, per-source naming) holds.
- Exact JSON shape of `reasons` / `recommended_actions` in the API response
and `classifications` row — follow the project's camelCase API response
convention; `reasons`/`recommended_actions` are already JSONB columns in
migration 097.
- Whether the KnowBe4/BSN sender-domain allowlist lives as a TypeScript
constant (mirroring `KNOWN_PHISHING_PATTERNS` in `phishing-detector.ts`) or
a small config/DB table — planner's call, but must support multiple
vendors (D-07) and be trivially extendable.
- Exact evaluation order/precedence when multiple rules could apply
(e.g. simulation-allowlist match vs. THREAT-tier signals) — the allowlist
match (D-06) short-circuits before D-03/D-04 tier evaluation per the
decisions above; finer implementation ordering is planner's call.
- How `isolate_endpoint`'s "configured provider path" question interacts
with Phase 20 — Phase 19 only needs to recommend it and mark it
destructive; Phase 20 owns whether/how it's actually executable.
</decisions>
<canonical_refs>
## Canonical References
**Downstream agents MUST read these before planning or implementing.**
### Evidence inputs (this phase reads, does not modify)
- `lib/services/mimecast-blast-radius.ts``getBlastRadius()`,
`BlastRadiusResult` (`status: 'ok' | 'unavailable'`, `matched`/`delivered`/
`held`/`rejected`/`clicked`/`perRecipient`) — the delivery-signal input for
D-03. Ephemeral per Phase 17 D-03 — this phase persists whatever it needs
into `classifications.reasons`.
- `lib/services/eml-parser.ts``NormalizedMessage` (headers incl. From/
Return-Path, structured SPF/DKIM/DMARC verdicts per Phase 16 D-06) — the
auth-failure input for D-03 and the sender-domain input for D-06.
- `lib/services/phishing-eml-service.ts` — writes `indicators` rows
(`indicator_type`: `'attachment_hash'`, `'url'`, `'sender'`) — the
known-bad-indicator input for D-03.
- `lib/services/phishing-detector.ts``KNOWN_PHISHING_PATTERNS`,
`matchesPhishingPatterns()` — the EXISTING report-pattern match this
phase's KnowBe4 detection must NOT rely on alone (see D-06 rationale).
- `lib/services/campaign-grouping-service.ts` — campaign/report shape this
phase reads to gather all linked reports/messages/indicators for a
campaign before classifying.
### Schema (this phase writes)
- `migrations/097_phishing_triage_schema.sql``classifications` table
(`verdict`, `confidence` NUMERIC, `summary`, `reasons` JSONB,
`recommended_actions` JSONB, `requires_approval` BOOLEAN NOT NULL DEFAULT
false) — append-only per campaign (D-02); `remediation_actions.action_type`
TEXT column is the eventual home for D-08's vocabulary (Phase 20 concern,
but the enum values should stay consistent with what this phase
recommends).
### Auth/permissions (established by Phase 18, reused here)
- `lib/permissions.ts``phishing` resource; `analyze` action already
granted to `superAdminRole`/`adminRole` (Phase 18 D-05). This phase's
`POST /classify` route should use `requirePermission('phishing',
'analyze')` — same action as the existing `/analyze` route, not a new
action — per Phase 18 D-06's established per-route permission-check
convention.
- `lib/auth-utils.ts``requirePermission(resource, action)` — see
`app/api/phishing/tickets/[ticket_id]/analyze/route.ts` for the exact
call-and-early-return pattern already used in this milestone.
### Deterministic-classifier precedent (this phase's architectural analog)
- `lib/services/robotic-classifier.ts``evaluateContains()` and the
`ClassificationRule` evaluation loop — the existing deterministic,
`.toLowerCase().includes()`-only (no regex/eval) rule-evaluation style
this phase's classifier should follow per D-01.
### Redaction path (if IT Glue evidence is ever referenced, per CLASSIFY-06)
- `lib/services/analyzer/itglue-search.ts` — the ONLY approved path for any
IT Glue-sourced data if this phase's evidence gathering ever pulls IT Glue
content in; raw `itglue-client.ts` output must never flow into evidence
passed to the classifier or the API response.
### Prior phase decisions (for consistency)
- `.planning/phases/18-campaign-grouping-phishing-analysis-api/18-CONTEXT.md`
(D-05, D-06 — permission vocabulary + per-route `requirePermission`
convention this phase must follow, not re-litigate)
- `.planning/phases/17-mimecast-blast-radius-lookup/17-CONTEXT.md` (D-03 —
blast-radius is ephemeral, this phase owns persistence)
- `.planning/phases/16-eml-mime-evidence-parser/16-CONTEXT.md` (D-06 —
structured SPF/DKIM/DMARC verdicts, not raw header text, are available)
</canonical_refs>
<code_context>
## Existing Code Insights
### Reusable Assets
- `robotic-classifier.ts`'s `evaluateContains()` pattern — precedent for a
small, pure, testable rule-evaluation function; this phase's verdict rules
should follow the same no-regex, deterministic style.
- `phishing-detector.ts`'s `KNOWN_PHISHING_PATTERNS` constant — precedent
for a locked, named pattern list; the KnowBe4/BSN sender-domain allowlist
(D-06/D-07) should follow the same "small exported constant" shape unless
planning finds a reason to make it DB-backed.
- `getBlastRadius()`'s `BlastRadiusResult` discriminated union (`status: 'ok'
| 'unavailable'`) — precedent for how this phase should shape its own
evidence-completeness checks (D-05) — presence/absence of a `status: 'ok'`
result is itself one of the confidence-deduction triggers.
### Established Patterns
- Factory/service convention: pure functions in `lib/services/`, no classes
needed for stateless evaluation logic (see `mimecast-blast-radius.ts`,
`campaign-grouping-service.ts`).
- `requirePermission(resource, action)` early-return pattern for every
`/api/phishing/*` route (Phase 18 D-06) — this phase's `/classify` route
follows the same shape.
- Append-only history tables with a `campaign_id` FK + `created_at` — same
shape as how `classifications` should be written (D-02), no upsert.
### Integration Points
- This phase's classifier reads: `campaigns` row, all linked `reports`,
`messages` (parsed .eml data), `indicators`, and a fresh `getBlastRadius()`
call — assembling one bounded evidence payload before rule evaluation.
- Writes: one new `classifications` row per `/classify` call.
- Phase 20 reads `classifications.recommended_actions` /
`requires_approval` to know what it's gating approval on — this phase's
D-08 vocabulary is a direct contract with Phase 20's `remediation_actions`
work.
</code_context>
<specifics>
## Specific Ideas
- The user specifically wants Breach Secure Now (BSN) simulation reports
recognized eventually, alongside KnowBe4 — flagged mid-discussion as a
real gap (nothing in the codebase references BSN today). Research should
actively look for BSN ticket examples in Autotask history before planning
locks in the KnowBe4/BSN allowlist (D-07).
</specifics>
<deferred>
## Deferred Ideas
- **BSN sender-domain confirmation** — if research finds no usable BSN
pattern in existing ticket history, ship this phase with a KnowBe4-only
allowlist (structured to support more vendors per D-07) and treat adding
BSN as a fast follow-up once a real example ticket is available.
- **`isolate_endpoint` provider wiring** — Phase 19 only recommends and
flags it destructive; which RMM/provider path actually executes it is
Phase 20's concern.
- **Non-BSN/KnowBe4 discussion scope creep** — none surfaced; discussion
stayed within the classification-engine boundary throughout.
</deferred>
---
*Phase: 19-classification-engine*
*Context gathered: 2026-07-16*

View file

@ -0,0 +1,122 @@
# Phase 19: Classification Engine - Discussion Log
> **Audit trail only.** Do not use as input to planning, research, or execution agents.
> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
**Date:** 2026-07-16
**Phase:** 19-classification-engine
**Areas discussed:** Rules-only vs. rules+LLM, Verdict signal rules, KnowBe4 simulation handling, Recommended actions vocabulary
---
## Rules-only vs. rules+LLM
| Option | Description | Selected |
|--------|-------------|----------|
| Deterministic rules only | No AI calls this phase; matches ROADMAP/success criteria literally | ✓ |
| Deterministic core + LLM-assist | Rules + Anthropic/OpenRouter stage for low-confidence verdicts, mini analyzer-pipeline style | |
| Something else | — | |
**User's choice:** Deterministic rules only.
| Option | Description | Selected |
|--------|-------------|----------|
| On-demand only via POST /classify | Matches CLASSIFY-05/ROADMAP "(re-)trigger" wording | ✓ |
| Auto-classify on campaign creation | Mirrors Phase 18's automatic grouping trigger | |
| Something else | — | |
**User's choice:** On-demand only via POST /classify.
**Notes:** No automatic classification pass on campaign creation — grouping already runs automatically (Phase 18); layering auto-classification on top would produce unrequested verdicts on every KnowBe4-button click.
---
## Verdict signal rules
| Option | Description | Selected |
|--------|-------------|----------|
| Delivery + auth-failure combo | THREAT needs both delivered/clicked>0 AND a hard auth fail or bad indicator | ✓ |
| Any hard auth failure alone | Spoofing alone is sufficient regardless of delivery | |
| Something else | — | |
**User's choice:** Delivery + auth-failure combo.
| Option | Description | Selected |
|--------|-------------|----------|
| SPAM = bulk/commercial, UNWANTED = targeted-but-contained | Content-signal-driven tier boundary | ✓ |
| SPAM = single-report, UNWANTED = multi-report campaign | Accumulation-driven tier boundary | |
| Something else | — | |
**User's choice:** SPAM = bulk/commercial, UNWANTED = targeted-but-contained.
| Option | Description | Selected |
|--------|-------------|----------|
| Point-deduction from 1.0 | Subtract fixed amount per missing evidence source, name each in reasons | ✓ |
| Discrete tiers (high/medium/low) | Coarser bucket-based confidence | |
| Something else | — | |
**User's choice:** Point-deduction from 1.0.
---
## KnowBe4 simulation handling
| Option | Description | Selected |
|--------|-------------|----------|
| Sender-domain allowlist | Check parsed .eml From/Return-Path domain against known simulation-vendor domains | ✓ |
| Trust the report-pattern match alone | Rely on Phase 15's button/report-flow pattern match only | |
| Something else (free text) | User raised Breach Secure Now (BSN) as a second vendor, unsure how it's marked | (led to follow-up) |
**User's choice (initial "Other"):** "We also have Breach Secure Now Tests - not sure how they are marked -"
**Notes:** Confirmed via grep that nothing in the codebase references "Breach Secure Now" or "BSN" today. Raised as a genuine gap in CLASSIFY-04 coverage — followed up with a dedicated question before finalizing the KnowBe4 question.
| Option | Description | Selected |
|--------|-------------|----------|
| Defer BSN as follow-up | Scope CLASSIFY-04 to KnowBe4 only this phase, structure allowlist for easy extension | |
| Have research dig into ticket history now | Search existing Autotask tickets for real BSN patterns before planning locks in rules | ✓ |
| Something else | — | |
**User's choice:** Have research dig into ticket history now.
| Option | Description | Selected |
|--------|-------------|----------|
| Sender-domain allowlist (config-driven, multi-vendor) | Match against original message's sender domain, not report-pattern alone | ✓ |
| Trust the report-pattern match alone | — | |
| Something else | — | |
**User's choice:** Sender-domain allowlist (config-driven, multi-vendor).
---
## Recommended actions vocabulary
| Option | Description | Selected |
|--------|-------------|----------|
| Core set: no_action, warn_user, block_sender, purge_message, reset_password | 5 actions, 2 non-destructive / 3 destructive | |
| Broader set: + disable_forwarding_rule, isolate_endpoint | 7 actions total | ✓ |
| Something else | — | |
**User's choice:** Broader set (7 actions).
| Option | Description | Selected |
|--------|-------------|----------|
| Yes, both destructive | disable_forwarding_rule + isolate_endpoint both require approval | |
| isolate_endpoint destructive, disable_forwarding_rule not | Reversibility-based split | ✓ |
| Something else | — | |
**User's choice:** isolate_endpoint destructive, disable_forwarding_rule not (easily reversible).
**Notes:** Final destructive set: block_sender, purge_message, reset_password, isolate_endpoint. Non-destructive: no_action, warn_user, disable_forwarding_rule. The requires_approval invariant (success criterion #2) is OR'd across all recommended actions — recommending disable_forwarding_rule alone must not force approval, but combined with any destructive action it still does.
---
## Claude's Discretion
- Exact point-deduction weights for confidence scoring.
- Exact JSON shape of `reasons`/`recommended_actions` fields (camelCase API convention; JSONB columns already exist).
- Whether the KnowBe4/BSN allowlist is a TS constant or config/DB table (must support multiple vendors).
- Exact rule-evaluation order/precedence beyond "simulation allowlist short-circuits before tier evaluation."
- How `isolate_endpoint`'s actual provider path gets wired — explicitly Phase 20's concern.
## Deferred Ideas
- BSN sender-domain confirmation, if research finds no usable pattern — ship KnowBe4-only allowlist structured for easy extension, fast-follow BSN later.
- `isolate_endpoint` provider/execution wiring — Phase 20.