5.9 KiB
Phase 19: Classification Engine - Discussion Log
Audit trail only. Do not use as input to planning, research, or execution agents. Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
Date: 2026-07-16 Phase: 19-classification-engine Areas discussed: Rules-only vs. rules+LLM, Verdict signal rules, KnowBe4 simulation handling, Recommended actions vocabulary
Rules-only vs. rules+LLM
| Option | Description | Selected |
|---|---|---|
| Deterministic rules only | No AI calls this phase; matches ROADMAP/success criteria literally | ✓ |
| Deterministic core + LLM-assist | Rules + Anthropic/OpenRouter stage for low-confidence verdicts, mini analyzer-pipeline style | |
| Something else | — |
User's choice: Deterministic rules only.
| Option | Description | Selected |
|---|---|---|
| On-demand only via POST /classify | Matches CLASSIFY-05/ROADMAP "(re-)trigger" wording | ✓ |
| Auto-classify on campaign creation | Mirrors Phase 18's automatic grouping trigger | |
| Something else | — |
User's choice: On-demand only via POST /classify. Notes: No automatic classification pass on campaign creation — grouping already runs automatically (Phase 18); layering auto-classification on top would produce unrequested verdicts on every KnowBe4-button click.
Verdict signal rules
| Option | Description | Selected |
|---|---|---|
| Delivery + auth-failure combo | THREAT needs both delivered/clicked>0 AND a hard auth fail or bad indicator | ✓ |
| Any hard auth failure alone | Spoofing alone is sufficient regardless of delivery | |
| Something else | — |
User's choice: Delivery + auth-failure combo.
| Option | Description | Selected |
|---|---|---|
| SPAM = bulk/commercial, UNWANTED = targeted-but-contained | Content-signal-driven tier boundary | ✓ |
| SPAM = single-report, UNWANTED = multi-report campaign | Accumulation-driven tier boundary | |
| Something else | — |
User's choice: SPAM = bulk/commercial, UNWANTED = targeted-but-contained.
| Option | Description | Selected |
|---|---|---|
| Point-deduction from 1.0 | Subtract fixed amount per missing evidence source, name each in reasons | ✓ |
| Discrete tiers (high/medium/low) | Coarser bucket-based confidence | |
| Something else | — |
User's choice: Point-deduction from 1.0.
KnowBe4 simulation handling
| Option | Description | Selected |
|---|---|---|
| Sender-domain allowlist | Check parsed .eml From/Return-Path domain against known simulation-vendor domains | ✓ |
| Trust the report-pattern match alone | Rely on Phase 15's button/report-flow pattern match only | |
| Something else (free text) | User raised Breach Secure Now (BSN) as a second vendor, unsure how it's marked | (led to follow-up) |
User's choice (initial "Other"): "We also have Breach Secure Now Tests - not sure how they are marked -" Notes: Confirmed via grep that nothing in the codebase references "Breach Secure Now" or "BSN" today. Raised as a genuine gap in CLASSIFY-04 coverage — followed up with a dedicated question before finalizing the KnowBe4 question.
| Option | Description | Selected |
|---|---|---|
| Defer BSN as follow-up | Scope CLASSIFY-04 to KnowBe4 only this phase, structure allowlist for easy extension | |
| Have research dig into ticket history now | Search existing Autotask tickets for real BSN patterns before planning locks in rules | ✓ |
| Something else | — |
User's choice: Have research dig into ticket history now.
| Option | Description | Selected |
|---|---|---|
| Sender-domain allowlist (config-driven, multi-vendor) | Match against original message's sender domain, not report-pattern alone | ✓ |
| Trust the report-pattern match alone | — | |
| Something else | — |
User's choice: Sender-domain allowlist (config-driven, multi-vendor).
Recommended actions vocabulary
| Option | Description | Selected |
|---|---|---|
| Core set: no_action, warn_user, block_sender, purge_message, reset_password | 5 actions, 2 non-destructive / 3 destructive | |
| Broader set: + disable_forwarding_rule, isolate_endpoint | 7 actions total | ✓ |
| Something else | — |
User's choice: Broader set (7 actions).
| Option | Description | Selected |
|---|---|---|
| Yes, both destructive | disable_forwarding_rule + isolate_endpoint both require approval | |
| isolate_endpoint destructive, disable_forwarding_rule not | Reversibility-based split | ✓ |
| Something else | — |
User's choice: isolate_endpoint destructive, disable_forwarding_rule not (easily reversible). Notes: Final destructive set: block_sender, purge_message, reset_password, isolate_endpoint. Non-destructive: no_action, warn_user, disable_forwarding_rule. The requires_approval invariant (success criterion #2) is OR'd across all recommended actions — recommending disable_forwarding_rule alone must not force approval, but combined with any destructive action it still does.
Claude's Discretion
- Exact point-deduction weights for confidence scoring.
- Exact JSON shape of
reasons/recommended_actionsfields (camelCase API convention; JSONB columns already exist). - Whether the KnowBe4/BSN allowlist is a TS constant or config/DB table (must support multiple vendors).
- Exact rule-evaluation order/precedence beyond "simulation allowlist short-circuits before tier evaluation."
- How
isolate_endpoint's actual provider path gets wired — explicitly Phase 20's concern.
Deferred Ideas
- BSN sender-domain confirmation, if research finds no usable pattern — ship KnowBe4-only allowlist structured for easy extension, fast-follow BSN later.
isolate_endpointprovider/execution wiring — Phase 20.