From a37dcebd2de09d35cd558358ecbfc29a2d71ef1d Mon Sep 17 00:00:00 2001 From: lorentz Date: Wed, 15 Jul 2026 15:39:03 -0400 Subject: [PATCH] docs(18): capture phase context --- .../18-CONTEXT.md | 229 ++++++++++++++++++ .../18-DISCUSSION-LOG.md | 85 +++++++ 2 files changed, 314 insertions(+) create mode 100644 .planning/phases/18-campaign-grouping-phishing-analysis-api/18-CONTEXT.md create mode 100644 .planning/phases/18-campaign-grouping-phishing-analysis-api/18-DISCUSSION-LOG.md diff --git a/.planning/phases/18-campaign-grouping-phishing-analysis-api/18-CONTEXT.md b/.planning/phases/18-campaign-grouping-phishing-analysis-api/18-CONTEXT.md new file mode 100644 index 0000000..6cf91e8 --- /dev/null +++ b/.planning/phases/18-campaign-grouping-phishing-analysis-api/18-CONTEXT.md @@ -0,0 +1,229 @@ +# Phase 18: Campaign Grouping & Phishing Analysis API - Context + +**Gathered:** 2026-07-15 +**Status:** Ready for planning + + +## Phase Boundary + +Duplicate reports of the same phishing/spam campaign are automatically +grouped into a shared `campaigns` row (tiered key: Message-ID, then +attachment-hash/URL-domain + subject + sender + time-window, then sender + +normalized subject + client + time-window), accumulating linked reports and +recipients over time. An operator can trigger on-demand analysis of one +specific ticket (`POST /api/phishing/tickets/{ticket_id}/analyze`) or browse +campaigns (`GET /api/phishing/campaigns`, `GET /api/phishing/campaigns/{id}`) +through a properly access-controlled `/api/phishing/*` surface — the first +API routes in this milestone, establishing the auth convention every later +phishing endpoint (Phases 19-21) must also follow. + + + + +## Implementation Decisions + +### Grouping Trigger Point +- **D-01:** Campaign grouping runs automatically as part of the same + detection path Phase 15's `phishing-detector.ts` already uses (webhook + fire-and-forget `ticket.created` handler + the scheduled cron sweep) — the + moment a `reports` row is created or updated, grouping runs against it so a + campaign forms/accumulates without waiting for anyone to call the API. The + new `POST /api/phishing/tickets/{ticket_id}/analyze` endpoint (DETECT-03) + calls the SAME shared grouping function on-demand for one ticket — not a + separate/duplicate implementation. Rationale: success criterion #2 + ("accumulates... as new duplicate reports arrive over time") implies + grouping must happen automatically, not only when explicitly requested. + +### Grouping Parameters +- **D-02:** Time window for the fallback tiers (attachment-hash/URL-domain + + subject + sender, and sender + normalized-subject + client) is **24 + hours**. Rationale: catches same-day mass-phishing blasts (the realistic + case — one attacker campaign, multiple people reporting the same day) + without over-grouping unrelated reports that happen to share sender+subject + weeks apart. +- **D-03:** Subject normalization for the fallback tiers: strip leading + `Re:`/`Fwd:`/`Fw:` prefixes (case-insensitive, repeated occurrences), + lowercase, trim whitespace. Standard email-threading normalization, nothing + more elaborate. + +### Campaign Merge Behavior +- **D-04:** Campaigns are never merged into each other. Each new report + attaches to at most one existing campaign (the single best/first match by + the tiered key) or creates a new one. If a report's tiered key + theoretically matches two distinct existing open campaigns, this is + treated as a rare edge case — do NOT implement multi-campaign-merge logic + (transactional reassignment of reports/messages/indicators/classifications + across campaigns) in this phase. Attach to the first/best match found and + move on; revisit only if this proves to be a real, recurring problem in + practice. + +### Permission Model +- **D-05:** Add a new `phishing` resource to `lib/permissions.ts`'s + `statement` with the FULL action set from the milestone spec now — + `phishing: ["read", "analyze", "approve", "remediate"]` — but this phase + only GRANTS `read` and `analyze` in role definitions: + - `superAdminRole` / `adminRole`: `phishing: ["read", "analyze"]` + - `userRole`: `phishing: ["read"]` (can browse campaigns, cannot trigger + on-demand analysis) + `approve`/`remediate` are declared in the statement (so the vocabulary + exists) but ungranted to any role until Phase 20 wires them up. Rationale: + establishes the full permission vocabulary once so Phase 20 only adds role + grants, not new statement keys — avoids touching the same statement block + twice across phases. + +### Route Auth Granularity +- **D-06:** Every `/api/phishing/*` route uses the fine-grained + `requirePermission()` check from day one, not plain `requireAuth()`: + - `GET /api/phishing/campaigns` and `GET /api/phishing/campaigns/{id}` → + `requirePermission('phishing', 'read')` + - `POST /api/phishing/tickets/{ticket_id}/analyze` → + `requirePermission('phishing', 'analyze')` + This establishes the exact auth convention (per-route, per-action + permission check) that Phase 19-21's endpoints must copy. Since `userRole` + gets `read` granted per D-05, this doesn't restrict any current user from + browsing — it wires the fine-grained check through now rather than + retrofitting it later. + +### Claude's Discretion (explicitly deferred to research + planner) +- **Exact SQL/query shape for `GET /api/phishing/campaigns/{id}`'s nested + response** (linked reports, messages, indicators, classification history) + — single JOIN-heavy query vs. multiple queries assembled in application + code. Follow whatever the existing `entity-sync.ts` / DetailModal-backing + API routes in this codebase already do for similarly-shaped nested detail + responses. +- **Exact shape of the tiered-key matching queries** (how "attachment-hash/ + URL-domain" is queried against the `indicators` table's `indicator_type` + values `'attachment_hash'`/`'url'`/`'sender'` from Phase 16, how URL-domain + is extracted from a full URL string) — implementation detail, resolve via + research against the actual `indicators` schema and data shapes Phase 16 + produces. +- **Response body shape for `POST /analyze`** — should return the resulting + campaign linkage (campaign ID, grouping method used, whether a new + campaign was created vs. an existing one was matched) — exact field names + are planner's call, following the project's camelCase API response + convention. +- **Where the shared grouping function lives** (new file e.g. + `lib/services/campaign-grouping-service.ts` vs. extending + `phishing-detector.ts`) — planner's call, following whatever composition + pattern is cleanest given `phishing-detector.ts`'s current size and + responsibilities. +- **Whether `campaigns.status` transitions in this phase** (e.g., does + grouping ever set/change `status` beyond the migration's default `'open'`) + — not mentioned in success criteria; likely out of scope for this phase + (status transitions are Phase 19/20's classification/remediation concern), + but confirm during planning. + + + + +## Canonical References + +**Downstream agents MUST read these before planning or implementing.** + +### Phase 15 detection path (grouping hooks into this) +- `lib/services/phishing-detector.ts` — `detectPhishingTicket()`, + `gatherTicketEvidence()`, the `content_hash` idempotency pattern, and the + `reports` upsert (`ON CONFLICT (ticket_id) DO UPDATE`) this phase's + grouping call attaches to. +- `lib/services/webhook-service.ts` — the `ticket.created` fire-and-forget + hook where Phase 15 wired in detection; this phase's grouping call should + follow the same fire-and-forget wiring point (per D-01). +- `lib/services/sync-scheduler.ts` — the cron sweep pattern (`pax8-daily`- + style schedule row) Phase 15 added for reconciliation; grouping runs on + the same sweep per D-01. + +### Phase 16 evidence (grouping keys off this) +- `lib/services/phishing-eml-service.ts` — writes `indicators` rows with + `indicator_type` values `'attachment_hash'`, `'url'`, `'sender'` (confirmed + by reading the file directly) — the CAMP-01 fallback-tier keys come from + these rows. +- `lib/services/eml-parser.ts` — `NormalizedMessage`'s `Message-ID` field + (via `parseEml`) — the CAMP-01 primary-tier key. + +### Auth/permissions (this phase establishes the phishing resource here) +- `lib/permissions.ts` — `statement`, `superAdminRole`, `adminRole`, + `userRole` — add the new `phishing` resource here (D-05). Existing + `rmm: ["read", "execute"]` resource is the closest analog for a + read/action-verb permission shape. +- `lib/auth-utils.ts` — `requirePermission(resource, action)` signature this + phase's routes call (D-06). See `app/api/admin/rmm/settings/route.ts` for + the exact call-and-early-return pattern: + `const { error } = await requirePermission('rmm', 'read'); if (error) return error;` + +### Schema (read/write for this phase) +- `migrations/097_phishing_triage_schema.sql` — `campaigns` table + (`campaign_key`, `group_method`, `first_seen_at`, `last_seen_at`, + `report_count`, `status`), `reports.campaign_id` (nullable FK, populated + starting this phase per the migration's own comment). + +### Prior phase decisions (for consistency) +- `.planning/phases/15-data-model-detection-ticket-evidence/15-CONTEXT.md` + (D-01: webhook + cron "primary path, cron reconciles" pattern; D-04: + content-hash idempotency) — this phase's grouping trigger follows the same + dual-path wiring instinct. +- `.planning/phases/17-mimecast-blast-radius-lookup/17-CONTEXT.md` (D-05: + single-tenant-for-v1, documented-limitation-over-silent-gap pattern) — + shows the project's general preference for "simple now, documented + limitation, defer complexity" when a rare edge case (like campaign + merging) would add substantial implementation cost. + + + + +## Existing Code Insights + +### Reusable Assets +- `phishing-detector.ts`'s `computePhishingContentHash()` — precedent for a + small pure hashing/normalization helper; the new subject-normalization + function (D-03) should follow the same style. +- `requirePermission()` from `lib/auth-utils.ts` — used as-is, no changes + needed to the helper itself, only new resource/action strings. + +### Established Patterns +- Factory/service convention: pure functions in `lib/services/`, called from + both a webhook handler and a cron-scheduled sync route — same shape + `phishing-detector.ts` already established, this phase extends it rather + than inventing a new wiring pattern. +- `ON CONFLICT ... DO UPDATE` upsert pattern (`reports` table) — the + campaign accumulation logic (D-01, CAMP-02) likely needs a similar + upsert-or-create-then-link pattern against `campaigns`. + +### Integration Points +- No `/api/phishing/*` routes exist yet in this repo — this phase creates + the first ones. No existing route to pattern-match against inside this + specific path; use `app/api/admin/rmm/settings/route.ts` (or similar + `requirePermission`-gated routes) as the general Next.js route + auth + pattern instead. + + + + +## Specific Ideas + +No specific UI/output-format requirements — this is a backend abstraction + +API surface with no UI in this phase (`UI hint: no` per ROADMAP.md). + + + + +## Deferred Ideas + +- **Multi-campaign merge logic** — deferred per D-04. If a report's tiered + key ever legitimately matches two distinct existing campaigns in practice + (not just in theory), building proper merge/reassignment logic is future + work, not blocking this phase. +- **`campaigns.status` transition logic** (open → closed/resolved etc.) — + out of scope for this phase; belongs to Phase 19 (classification) or + Phase 20 (remediation). +- **approve/remediate permission grants** — the `phishing` resource's + `approve`/`remediate` actions are declared in the statement (D-05) but not + granted to any role until Phase 20. + +None — discussion otherwise stayed within phase scope. + + + +--- + +*Phase: 18-campaign-grouping-phishing-analysis-api* +*Context gathered: 2026-07-15* diff --git a/.planning/phases/18-campaign-grouping-phishing-analysis-api/18-DISCUSSION-LOG.md b/.planning/phases/18-campaign-grouping-phishing-analysis-api/18-DISCUSSION-LOG.md new file mode 100644 index 0000000..b3cbc59 --- /dev/null +++ b/.planning/phases/18-campaign-grouping-phishing-analysis-api/18-DISCUSSION-LOG.md @@ -0,0 +1,85 @@ +# Phase 18: Campaign Grouping & Phishing Analysis API - Discussion Log + +> **Audit trail only.** Do not use as input to planning, research, or execution agents. +> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered. + +**Date:** 2026-07-15 +**Phase:** 18-campaign-grouping-phishing-analysis-api +**Areas discussed:** Grouping trigger, Grouping params, Campaign merge, Permission model, Route auth + +--- + +## Grouping Trigger + +| Option | Description | Selected | +|--------|-------------|----------| +| Auto-group on every detection | Hook grouping into the same webhook + cron path Phase 15 uses; new `/analyze` endpoint reuses the same shared function on-demand. | ✓ | +| On-demand only via the new API | Grouping only happens when `POST /analyze` is explicitly called. | | + +**User's choice:** Auto-group on every detection (recommended option). +**Notes:** None. + +--- + +## Grouping Params + +| Option | Description | Selected | +|--------|-------------|----------| +| 24h window, strip Re:/Fwd: + lowercase + trim | Same-day mass-phishing blast window; standard subject normalization. | ✓ | +| 72h window, same subject normalization | Wider window, catches slower-trickling reports at cost of looser grouping. | | +| You decide (Claude's discretion) | No strong preference. | | + +**User's choice:** 24h window, strip Re:/Fwd: + lowercase + trim (recommended option). +**Notes:** None. + +--- + +## Campaign Merge + +| Option | Description | Selected | +|--------|-------------|----------| +| Never merge existing campaigns | Attach to single best-matching existing campaign or create new; no merge logic. | ✓ | +| Support merging two campaigns when both match | Transactional reassignment of reports/messages/indicators/classifications across campaigns. | | + +**User's choice:** Never merge existing campaigns (recommended option). +**Notes:** None. + +--- + +## Permission Model + +| Option | Description | Selected | +|--------|-------------|----------| +| New 'phishing' resource: read + analyze actions now, full statement vocabulary declared | `phishing: ["read","analyze","approve","remediate"]` in statement; only read+analyze granted this phase. | ✓ | +| New 'phishing' resource: read + analyze actions only, extend later | Only declare read+analyze now; Phase 20 adds approve/remediate to the statement itself. | | + +**User's choice:** New 'phishing' resource: read + analyze actions now (recommended option). +**Notes:** None. + +--- + +## Route Auth (follow-up) + +| Option | Description | Selected | +|--------|-------------|----------| +| requirePermission('phishing','read'/'analyze') on every route | Fine-grained check on all routes from day one. | ✓ | +| requireAuth() only for GETs, requirePermission for POST /analyze | Looser check on reads. | | + +**User's choice:** requirePermission on every route (recommended option). +**Notes:** Asked as a follow-up after the Permission Model question to nail down route-level specifics. + +--- + +## Claude's Discretion + +- Exact SQL/query shape for `GET /api/phishing/campaigns/{id}`'s nested response +- Exact shape of the tiered-key matching queries against the `indicators` table +- Response body shape for `POST /analyze` +- Where the shared grouping function lives (new file vs. extending `phishing-detector.ts`) +- Whether `campaigns.status` transitions in this phase (likely out of scope) + +## Deferred Ideas + +- Multi-campaign merge logic — deferred per D-04, future work if it proves to be a real recurring problem +- `campaigns.status` transition logic — belongs to Phase 19/20 +- approve/remediate permission grants — declared now (D-05), granted in Phase 20