# Phase 15: Data Model, Detection & Ticket Evidence - Context

**Gathered:** 2026-07-15
**Status:** Ready for planning

<domain>
## Phase Boundary

The durable phishing-triage schema exists in Postgres, and Pulse can scan Autotask/Pulse
tickets for known phishing/spam-report patterns idempotently, capturing base
ticket-level evidence for each candidate. Covers DETECT-01, DETECT-02, EVID-01.
Does NOT cover `.eml`/MIME parsing (Phase 16), Mimecast (Phase 17), campaign
grouping or the `/api/phishing/*` surface (Phase 18), classification (Phase 19),
remediation (Phase 20), or the Autotask note (Phase 21).

</domain>

<decisions>
## Implementation Decisions

### Scan Trigger
- **D-01:** Detection is wired into the existing ticket webhook path (near-real-time)
  PLUS a scheduled cron sweep for reconciliation — the same "webhook primary, cron
  reconciles" pattern already used for `ticket_notes` (see `entity-sync.ts`'s
  `syncTicketNotes` docstring: "Webhooks are the primary path; this exists so
  missed events (webhook outages, replays) get reconciled by the scheduled sync").
  Concretely: hook detection into `lib/services/webhook-service.ts`'s
  `ticket.created` handling (fire-and-forget, matching the workflow-engine
  trigger pattern already there), AND add a new `sync-scheduler.ts` cron row
  (same shape as `pax8-daily`) that sweeps recently-modified tickets through the
  same detector function for reconciliation.

### Backfill Scope
- **D-02:** Forward-only for this phase. Only tickets created/modified after
  this phase ships get scanned by the webhook/cron paths. The existing backlog
  of already-reported phishing tickets (production evidence: ~267 in the last
  30 days) is explicitly NOT backfilled in Phase 15 — a manual backfill script
  can be run later if needed, but it is not a success criterion here.

### Match Surface
- **D-03:** The pattern matcher searches ticket `title` + `description` only
  (both already columns on the `tickets` table). It does NOT search
  `ticket_notes` in this phase, even though that table is already synced
  locally and would be a cheap addition — keep the v1 matcher scoped to the
  ticket's own fields. (Note for future phases/backlog: broadening to notes
  would need explicit follow-up if false-negatives show up in practice.)

### Idempotency / Reprocessing Key
- **D-04:** Use a content-hash approach, mirroring the analyzer pipeline's
  `content_hash` idempotency convention (see `ARCHITECTURE.md` — Stage 0
  computes `content_hash` for idempotency). Hash the matching-relevant fields
  (title + description) and store the hash on the `reports` row. Reprocess a
  ticket only when its hash changes — NOT on every `last_activity_date` bump
  (status changes, assignee changes, etc. must not trigger reprocessing).

### Claude's Discretion
- Exact migration file number (next available after 096 — confirm at plan
  time in case other work landed migrations in between).
- Exact cron schedule cadence/name for the reconciliation sweep (follow the
  `pax8-daily` naming/registration pattern in `sync-scheduler.ts`).
- Whether the detector is a single exported function called from both the
  webhook path and the cron path, or two thin wrappers over one shared core —
  planner/executor's call, as long as both call the same underlying logic
  (no duplicated pattern-matching logic).
- Exact `reports` row shape for storing EVID-01 evidence (time entries,
  attachment metadata) beyond what's spelled out in ROADMAP.md's success
  criteria — planner has discretion on column layout vs. JSON columns,
  following the project's snake_case / JSON-column conventions.

</decisions>

<canonical_refs>
## Canonical References

**Downstream agents MUST read these before planning or implementing.**

### Project conventions
- `CLAUDE.md` — migration numbering (`IF NOT EXISTS`, sequential), API route
  conventions, auth helper usage, no-ORM/snake_case-DB/camelCase-API rule
- `ARCHITECTURE.md` — background worker vs. fire-and-forget sync-endpoint
  patterns, content_hash idempotency precedent (analyzer Stage 0), error
  handling conventions (401/403/503/500)
- `INTEGRATIONS.md` — confirms no `attachments` table exists yet (Phase 16
  will need to fetch `.eml` content live via Autotask's Attachments API);
  confirms `ticket_notes` already syncs to Postgres; confirms an existing
  Autotask note-write pattern already exists (relevant to Phase 21, not this
  phase)

### Reference implementations for this phase
- `lib/services/entity-sync.ts` (`syncTicketNotes`, ~line 1373) — the
  "webhook primary, cron reconciles" pattern to mirror for detection
- `lib/services/webhook-service.ts` — where `ticket.created` is currently
  handled; detection should hook in here, fire-and-forget, same shape as the
  existing workflow-engine trigger
- `lib/services/sync-scheduler.ts` + `migrations/096_pax8_daily_schedule.sql`
  — the cron-row registration pattern to copy for the reconciliation sweep
  (idempotent seed row, admin-visible, disabled-by-default precedent if
  applicable)
- `lib/services/analyzer/pipeline.ts` Stage 0 — the `content_hash` idempotency
  pattern to mirror for D-04

</canonical_refs>

<code_context>
## Existing Code Insights

### Reusable Assets
- `tickets` table (migration 001) already has `title`, `description`,
  `company_id`, `last_activity_date` — no new columns needed on `tickets`
  itself for this phase's matching
- `ticket_notes` table (migration 025) — already synced, available for a
  future broadening of match surface even though not used in Phase 15
- `AutotaskClient.getAttachments(entityName, entityId)` — already exists,
  returns `Attachment[]` with metadata (`fullPath`, `title`, `contentType`);
  `data` (base64) is typed optional — likely only populated on a per-attachment
  fetch, not the list call (confirm in Phase 16, not blocking for Phase 15's
  EVID-01 "attachment metadata" success criterion)
- `client.createEntity('TicketNotes', {...})` — existing safe Autotask
  note-write pattern (`workflow-engine.ts`, `veeam-rpo-service.ts`) — feeds
  Phase 21's NOTE-01, noted here for continuity

### Established Patterns
- Fire-and-forget `/api/<x>/sync` POST + `sync-scheduler.ts` cron row is the
  dominant integration pattern in this codebase (PAX8, Veeam, Datto, Zoom,
  QBO, etc.) — only the analyzer and RMM Overshell run as always-polling
  background workers, and those are the exception, not something to imitate
  here
- `EntityType` sync dependency graph (`lib/types/sync.ts`) shows
  `TICKET_NOTES` depends on `TICKETS` — same dependency shape will likely
  apply to the new phishing tables depending on `tickets`/`companies`

### Integration Points
- New migration adds `campaigns`, `reports`, `messages`, `indicators`,
  `classifications`, `remediation_actions`, `audit_events` tables (this
  phase's schema; only `reports` + a stub of the others is populated by
  Phase 15's detector — full population of `messages`/`indicators` comes in
  Phase 16, `classifications` in Phase 19, etc.)
- Detector hooks into `lib/services/webhook-service.ts` (ticket.created path)
  and a new `sync-scheduler.ts` cron entry

</code_context>

<specifics>
## Specific Ideas

No specific UI/visual references (this phase has no UI surface). The
concrete behavioral references are the exact known match patterns from the
milestone spec (title/body signatures): "Phishing Report", "Spam Alert",
"Phishing Alert - Email Security Report", "KnowBe4 Phish Alert Report",
"Source: KnowBe4 Phish Alert Button", "userSubmissionsReportMessage",
"reported message destinations", "Microsoft directly" — these are locked via
REQUIREMENTS.md DETECT-01 and must all be covered by the matcher.

</specifics>

<deferred>
## Deferred Ideas

- **Backfill scan of existing/historical phishing tickets** — explicitly
  deferred per D-02. Could become its own follow-up task/script later
  (`scripts/` one-off, per CLAUDE.md's `scripts/` convention) rather than a
  roadmap phase, since it's a one-time operational task, not a recurring
  capability.
- **Broadening match surface to `ticket_notes`** — deferred per D-03. Revisit
  if false-negatives are observed in practice after Phase 15 ships.

None — discussion stayed within phase scope beyond the two items above.

</deferred>

---

*Phase: 15-data-model-detection-ticket-evidence*
*Context gathered: 2026-07-15*
