From d3ff9d088bd70f722e670324a3a3bb7dd8cecda3 Mon Sep 17 00:00:00 2001 From: lorentz Date: Wed, 15 Jul 2026 06:56:37 -0400 Subject: [PATCH] docs(15): capture phase context --- .../15-CONTEXT.md | 174 ++++++++++++++++++ .../15-DISCUSSION-LOG.md | 71 +++++++ 2 files changed, 245 insertions(+) create mode 100644 .planning/phases/15-data-model-detection-ticket-evidence/15-CONTEXT.md create mode 100644 .planning/phases/15-data-model-detection-ticket-evidence/15-DISCUSSION-LOG.md diff --git a/.planning/phases/15-data-model-detection-ticket-evidence/15-CONTEXT.md b/.planning/phases/15-data-model-detection-ticket-evidence/15-CONTEXT.md new file mode 100644 index 0000000..c7d6611 --- /dev/null +++ b/.planning/phases/15-data-model-detection-ticket-evidence/15-CONTEXT.md @@ -0,0 +1,174 @@ +# Phase 15: Data Model, Detection & Ticket Evidence - Context + +**Gathered:** 2026-07-15 +**Status:** Ready for planning + + +## Phase Boundary + +The durable phishing-triage schema exists in Postgres, and Pulse can scan Autotask/Pulse +tickets for known phishing/spam-report patterns idempotently, capturing base +ticket-level evidence for each candidate. Covers DETECT-01, DETECT-02, EVID-01. +Does NOT cover `.eml`/MIME parsing (Phase 16), Mimecast (Phase 17), campaign +grouping or the `/api/phishing/*` surface (Phase 18), classification (Phase 19), +remediation (Phase 20), or the Autotask note (Phase 21). + + + + +## Implementation Decisions + +### Scan Trigger +- **D-01:** Detection is wired into the existing ticket webhook path (near-real-time) + PLUS a scheduled cron sweep for reconciliation — the same "webhook primary, cron + reconciles" pattern already used for `ticket_notes` (see `entity-sync.ts`'s + `syncTicketNotes` docstring: "Webhooks are the primary path; this exists so + missed events (webhook outages, replays) get reconciled by the scheduled sync"). + Concretely: hook detection into `lib/services/webhook-service.ts`'s + `ticket.created` handling (fire-and-forget, matching the workflow-engine + trigger pattern already there), AND add a new `sync-scheduler.ts` cron row + (same shape as `pax8-daily`) that sweeps recently-modified tickets through the + same detector function for reconciliation. + +### Backfill Scope +- **D-02:** Forward-only for this phase. Only tickets created/modified after + this phase ships get scanned by the webhook/cron paths. The existing backlog + of already-reported phishing tickets (production evidence: ~267 in the last + 30 days) is explicitly NOT backfilled in Phase 15 — a manual backfill script + can be run later if needed, but it is not a success criterion here. + +### Match Surface +- **D-03:** The pattern matcher searches ticket `title` + `description` only + (both already columns on the `tickets` table). It does NOT search + `ticket_notes` in this phase, even though that table is already synced + locally and would be a cheap addition — keep the v1 matcher scoped to the + ticket's own fields. (Note for future phases/backlog: broadening to notes + would need explicit follow-up if false-negatives show up in practice.) + +### Idempotency / Reprocessing Key +- **D-04:** Use a content-hash approach, mirroring the analyzer pipeline's + `content_hash` idempotency convention (see `ARCHITECTURE.md` — Stage 0 + computes `content_hash` for idempotency). Hash the matching-relevant fields + (title + description) and store the hash on the `reports` row. Reprocess a + ticket only when its hash changes — NOT on every `last_activity_date` bump + (status changes, assignee changes, etc. must not trigger reprocessing). + +### Claude's Discretion +- Exact migration file number (next available after 096 — confirm at plan + time in case other work landed migrations in between). +- Exact cron schedule cadence/name for the reconciliation sweep (follow the + `pax8-daily` naming/registration pattern in `sync-scheduler.ts`). +- Whether the detector is a single exported function called from both the + webhook path and the cron path, or two thin wrappers over one shared core — + planner/executor's call, as long as both call the same underlying logic + (no duplicated pattern-matching logic). +- Exact `reports` row shape for storing EVID-01 evidence (time entries, + attachment metadata) beyond what's spelled out in ROADMAP.md's success + criteria — planner has discretion on column layout vs. JSON columns, + following the project's snake_case / JSON-column conventions. + + + + +## Canonical References + +**Downstream agents MUST read these before planning or implementing.** + +### Project conventions +- `CLAUDE.md` — migration numbering (`IF NOT EXISTS`, sequential), API route + conventions, auth helper usage, no-ORM/snake_case-DB/camelCase-API rule +- `ARCHITECTURE.md` — background worker vs. fire-and-forget sync-endpoint + patterns, content_hash idempotency precedent (analyzer Stage 0), error + handling conventions (401/403/503/500) +- `INTEGRATIONS.md` — confirms no `attachments` table exists yet (Phase 16 + will need to fetch `.eml` content live via Autotask's Attachments API); + confirms `ticket_notes` already syncs to Postgres; confirms an existing + Autotask note-write pattern already exists (relevant to Phase 21, not this + phase) + +### Reference implementations for this phase +- `lib/services/entity-sync.ts` (`syncTicketNotes`, ~line 1373) — the + "webhook primary, cron reconciles" pattern to mirror for detection +- `lib/services/webhook-service.ts` — where `ticket.created` is currently + handled; detection should hook in here, fire-and-forget, same shape as the + existing workflow-engine trigger +- `lib/services/sync-scheduler.ts` + `migrations/096_pax8_daily_schedule.sql` + — the cron-row registration pattern to copy for the reconciliation sweep + (idempotent seed row, admin-visible, disabled-by-default precedent if + applicable) +- `lib/services/analyzer/pipeline.ts` Stage 0 — the `content_hash` idempotency + pattern to mirror for D-04 + + + + +## Existing Code Insights + +### Reusable Assets +- `tickets` table (migration 001) already has `title`, `description`, + `company_id`, `last_activity_date` — no new columns needed on `tickets` + itself for this phase's matching +- `ticket_notes` table (migration 025) — already synced, available for a + future broadening of match surface even though not used in Phase 15 +- `AutotaskClient.getAttachments(entityName, entityId)` — already exists, + returns `Attachment[]` with metadata (`fullPath`, `title`, `contentType`); + `data` (base64) is typed optional — likely only populated on a per-attachment + fetch, not the list call (confirm in Phase 16, not blocking for Phase 15's + EVID-01 "attachment metadata" success criterion) +- `client.createEntity('TicketNotes', {...})` — existing safe Autotask + note-write pattern (`workflow-engine.ts`, `veeam-rpo-service.ts`) — feeds + Phase 21's NOTE-01, noted here for continuity + +### Established Patterns +- Fire-and-forget `/api//sync` POST + `sync-scheduler.ts` cron row is the + dominant integration pattern in this codebase (PAX8, Veeam, Datto, Zoom, + QBO, etc.) — only the analyzer and RMM Overshell run as always-polling + background workers, and those are the exception, not something to imitate + here +- `EntityType` sync dependency graph (`lib/types/sync.ts`) shows + `TICKET_NOTES` depends on `TICKETS` — same dependency shape will likely + apply to the new phishing tables depending on `tickets`/`companies` + +### Integration Points +- New migration adds `campaigns`, `reports`, `messages`, `indicators`, + `classifications`, `remediation_actions`, `audit_events` tables (this + phase's schema; only `reports` + a stub of the others is populated by + Phase 15's detector — full population of `messages`/`indicators` comes in + Phase 16, `classifications` in Phase 19, etc.) +- Detector hooks into `lib/services/webhook-service.ts` (ticket.created path) + and a new `sync-scheduler.ts` cron entry + + + + +## Specific Ideas + +No specific UI/visual references (this phase has no UI surface). The +concrete behavioral references are the exact known match patterns from the +milestone spec (title/body signatures): "Phishing Report", "Spam Alert", +"Phishing Alert - Email Security Report", "KnowBe4 Phish Alert Report", +"Source: KnowBe4 Phish Alert Button", "userSubmissionsReportMessage", +"reported message destinations", "Microsoft directly" — these are locked via +REQUIREMENTS.md DETECT-01 and must all be covered by the matcher. + + + + +## Deferred Ideas + +- **Backfill scan of existing/historical phishing tickets** — explicitly + deferred per D-02. Could become its own follow-up task/script later + (`scripts/` one-off, per CLAUDE.md's `scripts/` convention) rather than a + roadmap phase, since it's a one-time operational task, not a recurring + capability. +- **Broadening match surface to `ticket_notes`** — deferred per D-03. Revisit + if false-negatives are observed in practice after Phase 15 ships. + +None — discussion stayed within phase scope beyond the two items above. + + + +--- + +*Phase: 15-data-model-detection-ticket-evidence* +*Context gathered: 2026-07-15* diff --git a/.planning/phases/15-data-model-detection-ticket-evidence/15-DISCUSSION-LOG.md b/.planning/phases/15-data-model-detection-ticket-evidence/15-DISCUSSION-LOG.md new file mode 100644 index 0000000..1d0f929 --- /dev/null +++ b/.planning/phases/15-data-model-detection-ticket-evidence/15-DISCUSSION-LOG.md @@ -0,0 +1,71 @@ +# Phase 15: Data Model, Detection & Ticket Evidence - Discussion Log + +> **Audit trail only.** Do not use as input to planning, research, or execution agents. +> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered. + +**Date:** 2026-07-15 +**Phase:** 15-data-model-detection-ticket-evidence +**Areas discussed:** Scan trigger, Backfill scope, Match surface, Idempotency key + +--- + +## Scan Trigger + +| Option | Description | Selected | +|--------|-------------|----------| +| Webhook + cron reconciliation | Hook into ticket.created webhook path for near-real-time detection, plus a scheduled sweep to catch anything missed — mirrors the ticket_notes "webhook primary, cron reconciles" pattern | ✓ | +| Cron-only sweep | Simpler: one new scheduled job scans recent tickets on an interval, no webhook hook-in | | +| On-demand only for this phase | Build just the scanner function; wiring to run automatically deferred to Phase 18 | | + +**User's choice:** Webhook + cron reconciliation (Recommended) +**Notes:** None beyond the recommendation. + +--- + +## Backfill Scope + +| Option | Description | Selected | +|--------|-------------|----------| +| Backfill + forward | One-time scan of existing tickets (~267 in last 30 days) plus catching new ones going forward | | +| Forward-only | Only tickets created/updated after this phase ships get scanned | ✓ | + +**User's choice:** Forward-only +**Notes:** User deviated from the recommended option. Backlog backfill deferred — see Deferred Ideas. + +--- + +## Match Surface + +| Option | Description | Selected | +|--------|-------------|----------| +| Title + description + notes | Also search ticket_notes (already synced locally) since report text sometimes lands in a follow-up note | | +| Title + description only | Simpler first pass, matches only the ticket's own fields | ✓ | + +**User's choice:** Title + description only +**Notes:** User deviated from the recommended option. Broadening to notes deferred — see Deferred Ideas. + +--- + +## Idempotency / Reprocessing Key + +| Option | Description | Selected | +|--------|-------------|----------| +| Content hash | Hash title+description, mirroring the analyzer pipeline's content_hash idempotency pattern | ✓ | +| Timestamp comparison | Compare ticket.last_activity_date against report's processed_at | | + +**User's choice:** Content hash (Recommended) +**Notes:** None beyond the recommendation. + +--- + +## Claude's Discretion + +- Exact migration file number (next available after 096, confirm at plan time) +- Exact cron schedule cadence/name for the reconciliation sweep +- Whether the detector is one shared function called from both paths, or two thin wrappers over shared core logic +- Exact `reports` row shape for EVID-01 evidence beyond ROADMAP.md's stated success criteria + +## Deferred Ideas + +- Backfill scan of existing/historical phishing tickets — noted as a possible future one-off script, not a roadmap phase +- Broadening match surface to `ticket_notes` — revisit if false-negatives are observed in practice