From 8d4618cf0f4f50b96676d1c77aa17f4c55cd8221 Mon Sep 17 00:00:00 2001 From: lorentz Date: Thu, 16 Jul 2026 07:40:15 -0400 Subject: [PATCH] docs(phase-19): add validation strategy --- .../19-classification-engine/19-VALIDATION.md | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 .planning/phases/19-classification-engine/19-VALIDATION.md diff --git a/.planning/phases/19-classification-engine/19-VALIDATION.md b/.planning/phases/19-classification-engine/19-VALIDATION.md new file mode 100644 index 0000000..b9ae271 --- /dev/null +++ b/.planning/phases/19-classification-engine/19-VALIDATION.md @@ -0,0 +1,78 @@ +--- +phase: 19 +slug: classification-engine +status: draft +nyquist_compliant: false +wave_0_complete: false +created: 2026-07-16 +--- + +# Phase 19 — Validation Strategy + +> Per-phase validation contract for feedback sampling during execution. + +--- + +## Test Infrastructure + +| Property | Value | +|----------|-------| +| **Framework** | vitest 4.1.5 | +| **Config file** | `vitest.config.ts` — `include: ['lib/**/*.test.ts']` (does NOT include `app/**` — no route-handler test precedent exists in this repo) | +| **Quick run command** | `npx vitest run lib/services/campaign-classifier.test.ts` | +| **Full suite command** | `npm test` | +| **Estimated runtime** | ~2 seconds (mocked `postgresClient` and `getBlastRadius`, no real DB/network) | + +--- + +## Sampling Rate + +- **After every task commit:** `npx vitest run lib/services/campaign-classifier.test.ts` +- **After every plan wave:** `npm test` (full suite) +- **Before `/gsd:verify-work`:** Full suite green + `npx tsc --noEmit --pretty` clean + manual curl verification of the `/classify` route's auth behavior +- **Max feedback latency:** ~2 seconds (automated) / manual curl pass for route auth (no automated route-handler test precedent in this repo) + +--- + +## Per-Task Verification Map + +| Task ID | Plan | Wave | Requirement | Threat Ref | Secure Behavior | Test Type | Automated Command | File Exists | Status | +|---------|------|------|-------------|------------|-----------------|-----------|-------------------|-------------|--------| +| 19-01-0x | 01 | 1 | CLASSIFY-01 | — | `classifyCampaign()` returns exactly one of SPAM/UNWANTED/THREAT + confidence + summary + reasons + recommendedActions + requiresApproval | unit | `npx vitest run lib/services/campaign-classifier.test.ts -t "returns exactly one verdict"` | ❌ W0 | ⬜ pending | +| 19-01-0x | 01 | 1 | CLASSIFY-02 | Tampering | Destructive recommended action always forces `requires_approval: true`; `disable_forwarding_rule` alone does NOT | unit | `npx vitest run lib/services/campaign-classifier.test.ts -t "requires_approval invariant"` | ❌ W0 | ⬜ pending | +| 19-01-0x | 01 | 1 | CLASSIFY-03 | — | Incomplete evidence lowers confidence and names the specific missing evidence source in `reasons` | unit | `npx vitest run lib/services/campaign-classifier.test.ts -t "confidence deduction"` | ❌ W0 | ⬜ pending | +| 19-01-0x | 01 | 1 | CLASSIFY-04 | Spoofing | Synthetic KnowBe4 (`it-support.care`) and BSN (`breachsecurenow.com`) simulation fixtures — each with `authResultsOriginal` pass despite `authResults` fail — are not classified THREAT absent contrary evidence | unit | `npx vitest run lib/services/campaign-classifier.test.ts -t "simulation allowlist"` | ❌ W0 | ⬜ pending | +| 19-0x-0x | 0x | 0x | CLASSIFY-05 | V4/V5 | `POST /api/phishing/campaigns/{id}/classify` enforces `requirePermission('phishing','analyze')`, validates campaign `id` as UUID, inserts a new `classifications` row, and rejects unauthorized/malformed requests | manual | curl against dev server (no route-handler test precedent) | n/a | ⬜ pending | +| 19-01-0x | 01 | 1 | CLASSIFY-06 | Denial of Service | Evidence payload is bounded/structured — no raw unbounded body text reaches the classifier or persisted `reasons`; IT Glue-sourced evidence (if any) goes through `itglue-search.ts` | unit | `npx vitest run lib/services/campaign-classifier.test.ts -t "evidence bounding"` | ❌ W0 | ⬜ pending | + +*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky* +*Exact task IDs finalized by the planner — this table's requirement/behavior mapping is fixed by research; row numbering will be updated once PLAN.md exists.* + +--- + +## Wave 0 Requirements + +- [ ] `lib/services/campaign-classifier.test.ts` — new file, covers CLASSIFY-01/02/03/04/06; mock `postgresClient.query` (`vi.mock`, matching `campaign-grouping-service.test.ts`'s existing convention) and mock `getBlastRadius` (`vi.mock('./mimecast-blast-radius')`) +- [ ] Fixture data for a synthetic KnowBe4 (`it-support.care`) message and a synthetic BSN (`breachsecurenow.com`) message, each WITH an `authResultsOriginal` block showing pass despite `authResults` showing fail (auth-verdict-precedence pitfall regression coverage) — no real customer data, synthetic only, matching Phase 16's own fixture convention (`eml-parser.fixtures.ts`) +- [ ] Framework install: none — vitest already configured project-wide + +--- + +## Manual-Only Verifications + +| Behavior | Requirement | Why Manual | Test Instructions | +|----------|-------------|------------|--------------------| +| `POST /api/phishing/campaigns/{id}/classify` orchestration + response shape + auth rejection | CLASSIFY-05 | No route-handler test precedent in this repo; `vitest.config.ts` doesn't even discover `app/**` test files | `curl -X POST http://localhost:3100/api/phishing/campaigns/{id}/classify` with a valid session cookie against a dev server with a known campaign; confirm response includes verdict/confidence/summary/reasons/recommendedActions/requiresApproval. Repeat with no session cookie (expect 401) and a `user`-role session (expect 403, since `analyze` is only granted to admin/super-admin per Phase 18 D-05) | + +--- + +## Validation Sign-Off + +- [ ] All tasks have `` verify or Wave 0 dependencies (service-layer) or documented manual verification (route-layer) +- [ ] Sampling continuity: no 3 consecutive tasks without automated or manual verify +- [ ] Wave 0 covers all MISSING references +- [ ] No watch-mode flags +- [ ] Feedback latency < 5s (automated) — manual route check is a pre-merge/pre-verify gate, not per-commit +- [ ] `nyquist_compliant: true` set in frontmatter + +**Approval:** pending