docs(15): capture phase context
This commit is contained in:
parent
0228639f24
commit
d3ff9d088b
2 changed files with 245 additions and 0 deletions
|
|
@ -0,0 +1,174 @@
|
|||
# Phase 15: Data Model, Detection & Ticket Evidence - Context
|
||||
|
||||
**Gathered:** 2026-07-15
|
||||
**Status:** Ready for planning
|
||||
|
||||
<domain>
|
||||
## Phase Boundary
|
||||
|
||||
The durable phishing-triage schema exists in Postgres, and Pulse can scan Autotask/Pulse
|
||||
tickets for known phishing/spam-report patterns idempotently, capturing base
|
||||
ticket-level evidence for each candidate. Covers DETECT-01, DETECT-02, EVID-01.
|
||||
Does NOT cover `.eml`/MIME parsing (Phase 16), Mimecast (Phase 17), campaign
|
||||
grouping or the `/api/phishing/*` surface (Phase 18), classification (Phase 19),
|
||||
remediation (Phase 20), or the Autotask note (Phase 21).
|
||||
|
||||
</domain>
|
||||
|
||||
<decisions>
|
||||
## Implementation Decisions
|
||||
|
||||
### Scan Trigger
|
||||
- **D-01:** Detection is wired into the existing ticket webhook path (near-real-time)
|
||||
PLUS a scheduled cron sweep for reconciliation — the same "webhook primary, cron
|
||||
reconciles" pattern already used for `ticket_notes` (see `entity-sync.ts`'s
|
||||
`syncTicketNotes` docstring: "Webhooks are the primary path; this exists so
|
||||
missed events (webhook outages, replays) get reconciled by the scheduled sync").
|
||||
Concretely: hook detection into `lib/services/webhook-service.ts`'s
|
||||
`ticket.created` handling (fire-and-forget, matching the workflow-engine
|
||||
trigger pattern already there), AND add a new `sync-scheduler.ts` cron row
|
||||
(same shape as `pax8-daily`) that sweeps recently-modified tickets through the
|
||||
same detector function for reconciliation.
|
||||
|
||||
### Backfill Scope
|
||||
- **D-02:** Forward-only for this phase. Only tickets created/modified after
|
||||
this phase ships get scanned by the webhook/cron paths. The existing backlog
|
||||
of already-reported phishing tickets (production evidence: ~267 in the last
|
||||
30 days) is explicitly NOT backfilled in Phase 15 — a manual backfill script
|
||||
can be run later if needed, but it is not a success criterion here.
|
||||
|
||||
### Match Surface
|
||||
- **D-03:** The pattern matcher searches ticket `title` + `description` only
|
||||
(both already columns on the `tickets` table). It does NOT search
|
||||
`ticket_notes` in this phase, even though that table is already synced
|
||||
locally and would be a cheap addition — keep the v1 matcher scoped to the
|
||||
ticket's own fields. (Note for future phases/backlog: broadening to notes
|
||||
would need explicit follow-up if false-negatives show up in practice.)
|
||||
|
||||
### Idempotency / Reprocessing Key
|
||||
- **D-04:** Use a content-hash approach, mirroring the analyzer pipeline's
|
||||
`content_hash` idempotency convention (see `ARCHITECTURE.md` — Stage 0
|
||||
computes `content_hash` for idempotency). Hash the matching-relevant fields
|
||||
(title + description) and store the hash on the `reports` row. Reprocess a
|
||||
ticket only when its hash changes — NOT on every `last_activity_date` bump
|
||||
(status changes, assignee changes, etc. must not trigger reprocessing).
|
||||
|
||||
### Claude's Discretion
|
||||
- Exact migration file number (next available after 096 — confirm at plan
|
||||
time in case other work landed migrations in between).
|
||||
- Exact cron schedule cadence/name for the reconciliation sweep (follow the
|
||||
`pax8-daily` naming/registration pattern in `sync-scheduler.ts`).
|
||||
- Whether the detector is a single exported function called from both the
|
||||
webhook path and the cron path, or two thin wrappers over one shared core —
|
||||
planner/executor's call, as long as both call the same underlying logic
|
||||
(no duplicated pattern-matching logic).
|
||||
- Exact `reports` row shape for storing EVID-01 evidence (time entries,
|
||||
attachment metadata) beyond what's spelled out in ROADMAP.md's success
|
||||
criteria — planner has discretion on column layout vs. JSON columns,
|
||||
following the project's snake_case / JSON-column conventions.
|
||||
|
||||
</decisions>
|
||||
|
||||
<canonical_refs>
|
||||
## Canonical References
|
||||
|
||||
**Downstream agents MUST read these before planning or implementing.**
|
||||
|
||||
### Project conventions
|
||||
- `CLAUDE.md` — migration numbering (`IF NOT EXISTS`, sequential), API route
|
||||
conventions, auth helper usage, no-ORM/snake_case-DB/camelCase-API rule
|
||||
- `ARCHITECTURE.md` — background worker vs. fire-and-forget sync-endpoint
|
||||
patterns, content_hash idempotency precedent (analyzer Stage 0), error
|
||||
handling conventions (401/403/503/500)
|
||||
- `INTEGRATIONS.md` — confirms no `attachments` table exists yet (Phase 16
|
||||
will need to fetch `.eml` content live via Autotask's Attachments API);
|
||||
confirms `ticket_notes` already syncs to Postgres; confirms an existing
|
||||
Autotask note-write pattern already exists (relevant to Phase 21, not this
|
||||
phase)
|
||||
|
||||
### Reference implementations for this phase
|
||||
- `lib/services/entity-sync.ts` (`syncTicketNotes`, ~line 1373) — the
|
||||
"webhook primary, cron reconciles" pattern to mirror for detection
|
||||
- `lib/services/webhook-service.ts` — where `ticket.created` is currently
|
||||
handled; detection should hook in here, fire-and-forget, same shape as the
|
||||
existing workflow-engine trigger
|
||||
- `lib/services/sync-scheduler.ts` + `migrations/096_pax8_daily_schedule.sql`
|
||||
— the cron-row registration pattern to copy for the reconciliation sweep
|
||||
(idempotent seed row, admin-visible, disabled-by-default precedent if
|
||||
applicable)
|
||||
- `lib/services/analyzer/pipeline.ts` Stage 0 — the `content_hash` idempotency
|
||||
pattern to mirror for D-04
|
||||
|
||||
</canonical_refs>
|
||||
|
||||
<code_context>
|
||||
## Existing Code Insights
|
||||
|
||||
### Reusable Assets
|
||||
- `tickets` table (migration 001) already has `title`, `description`,
|
||||
`company_id`, `last_activity_date` — no new columns needed on `tickets`
|
||||
itself for this phase's matching
|
||||
- `ticket_notes` table (migration 025) — already synced, available for a
|
||||
future broadening of match surface even though not used in Phase 15
|
||||
- `AutotaskClient.getAttachments(entityName, entityId)` — already exists,
|
||||
returns `Attachment[]` with metadata (`fullPath`, `title`, `contentType`);
|
||||
`data` (base64) is typed optional — likely only populated on a per-attachment
|
||||
fetch, not the list call (confirm in Phase 16, not blocking for Phase 15's
|
||||
EVID-01 "attachment metadata" success criterion)
|
||||
- `client.createEntity('TicketNotes', {...})` — existing safe Autotask
|
||||
note-write pattern (`workflow-engine.ts`, `veeam-rpo-service.ts`) — feeds
|
||||
Phase 21's NOTE-01, noted here for continuity
|
||||
|
||||
### Established Patterns
|
||||
- Fire-and-forget `/api/<x>/sync` POST + `sync-scheduler.ts` cron row is the
|
||||
dominant integration pattern in this codebase (PAX8, Veeam, Datto, Zoom,
|
||||
QBO, etc.) — only the analyzer and RMM Overshell run as always-polling
|
||||
background workers, and those are the exception, not something to imitate
|
||||
here
|
||||
- `EntityType` sync dependency graph (`lib/types/sync.ts`) shows
|
||||
`TICKET_NOTES` depends on `TICKETS` — same dependency shape will likely
|
||||
apply to the new phishing tables depending on `tickets`/`companies`
|
||||
|
||||
### Integration Points
|
||||
- New migration adds `campaigns`, `reports`, `messages`, `indicators`,
|
||||
`classifications`, `remediation_actions`, `audit_events` tables (this
|
||||
phase's schema; only `reports` + a stub of the others is populated by
|
||||
Phase 15's detector — full population of `messages`/`indicators` comes in
|
||||
Phase 16, `classifications` in Phase 19, etc.)
|
||||
- Detector hooks into `lib/services/webhook-service.ts` (ticket.created path)
|
||||
and a new `sync-scheduler.ts` cron entry
|
||||
|
||||
</code_context>
|
||||
|
||||
<specifics>
|
||||
## Specific Ideas
|
||||
|
||||
No specific UI/visual references (this phase has no UI surface). The
|
||||
concrete behavioral references are the exact known match patterns from the
|
||||
milestone spec (title/body signatures): "Phishing Report", "Spam Alert",
|
||||
"Phishing Alert - Email Security Report", "KnowBe4 Phish Alert Report",
|
||||
"Source: KnowBe4 Phish Alert Button", "userSubmissionsReportMessage",
|
||||
"reported message destinations", "Microsoft directly" — these are locked via
|
||||
REQUIREMENTS.md DETECT-01 and must all be covered by the matcher.
|
||||
|
||||
</specifics>
|
||||
|
||||
<deferred>
|
||||
## Deferred Ideas
|
||||
|
||||
- **Backfill scan of existing/historical phishing tickets** — explicitly
|
||||
deferred per D-02. Could become its own follow-up task/script later
|
||||
(`scripts/` one-off, per CLAUDE.md's `scripts/` convention) rather than a
|
||||
roadmap phase, since it's a one-time operational task, not a recurring
|
||||
capability.
|
||||
- **Broadening match surface to `ticket_notes`** — deferred per D-03. Revisit
|
||||
if false-negatives are observed in practice after Phase 15 ships.
|
||||
|
||||
None — discussion stayed within phase scope beyond the two items above.
|
||||
|
||||
</deferred>
|
||||
|
||||
---
|
||||
|
||||
*Phase: 15-data-model-detection-ticket-evidence*
|
||||
*Context gathered: 2026-07-15*
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
# Phase 15: Data Model, Detection & Ticket Evidence - Discussion Log
|
||||
|
||||
> **Audit trail only.** Do not use as input to planning, research, or execution agents.
|
||||
> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
|
||||
|
||||
**Date:** 2026-07-15
|
||||
**Phase:** 15-data-model-detection-ticket-evidence
|
||||
**Areas discussed:** Scan trigger, Backfill scope, Match surface, Idempotency key
|
||||
|
||||
---
|
||||
|
||||
## Scan Trigger
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| Webhook + cron reconciliation | Hook into ticket.created webhook path for near-real-time detection, plus a scheduled sweep to catch anything missed — mirrors the ticket_notes "webhook primary, cron reconciles" pattern | ✓ |
|
||||
| Cron-only sweep | Simpler: one new scheduled job scans recent tickets on an interval, no webhook hook-in | |
|
||||
| On-demand only for this phase | Build just the scanner function; wiring to run automatically deferred to Phase 18 | |
|
||||
|
||||
**User's choice:** Webhook + cron reconciliation (Recommended)
|
||||
**Notes:** None beyond the recommendation.
|
||||
|
||||
---
|
||||
|
||||
## Backfill Scope
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| Backfill + forward | One-time scan of existing tickets (~267 in last 30 days) plus catching new ones going forward | |
|
||||
| Forward-only | Only tickets created/updated after this phase ships get scanned | ✓ |
|
||||
|
||||
**User's choice:** Forward-only
|
||||
**Notes:** User deviated from the recommended option. Backlog backfill deferred — see Deferred Ideas.
|
||||
|
||||
---
|
||||
|
||||
## Match Surface
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| Title + description + notes | Also search ticket_notes (already synced locally) since report text sometimes lands in a follow-up note | |
|
||||
| Title + description only | Simpler first pass, matches only the ticket's own fields | ✓ |
|
||||
|
||||
**User's choice:** Title + description only
|
||||
**Notes:** User deviated from the recommended option. Broadening to notes deferred — see Deferred Ideas.
|
||||
|
||||
---
|
||||
|
||||
## Idempotency / Reprocessing Key
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| Content hash | Hash title+description, mirroring the analyzer pipeline's content_hash idempotency pattern | ✓ |
|
||||
| Timestamp comparison | Compare ticket.last_activity_date against report's processed_at | |
|
||||
|
||||
**User's choice:** Content hash (Recommended)
|
||||
**Notes:** None beyond the recommendation.
|
||||
|
||||
---
|
||||
|
||||
## Claude's Discretion
|
||||
|
||||
- Exact migration file number (next available after 096, confirm at plan time)
|
||||
- Exact cron schedule cadence/name for the reconciliation sweep
|
||||
- Whether the detector is one shared function called from both paths, or two thin wrappers over shared core logic
|
||||
- Exact `reports` row shape for EVID-01 evidence beyond ROADMAP.md's stated success criteria
|
||||
|
||||
## Deferred Ideas
|
||||
|
||||
- Backfill scan of existing/historical phishing tickets — noted as a possible future one-off script, not a roadmap phase
|
||||
- Broadening match surface to `ticket_notes` — revisit if false-negatives are observed in practice
|
||||
Loading…
Add table
Add a link
Reference in a new issue