diff --git a/.planning/phases/16-eml-mime-evidence-parser/16-03-SUMMARY.md b/.planning/phases/16-eml-mime-evidence-parser/16-03-SUMMARY.md new file mode 100644 index 0000000..68d3108 --- /dev/null +++ b/.planning/phases/16-eml-mime-evidence-parser/16-03-SUMMARY.md @@ -0,0 +1,103 @@ +--- +phase: 16-eml-mime-evidence-parser +plan: 03 +subsystem: api +tags: [autotask, backblaze-b2, postgres, mailparser, phishing, vitest] + +# Dependency graph +requires: + - phase: 16-eml-mime-evidence-parser (Plan 01) + provides: "lib/services/eml-parser.ts — selectOriginalMessage, parseEml, MAX_EML_BYTES, NormalizedMessage" + - phase: 16-eml-mime-evidence-parser (Plan 02) + provides: "AutotaskClient.getAttachmentContent, B2 EML_OBJECT_KEY_REGEX + parameterized presignUpload, migrations/099 indicators.metadata" +provides: + - "lib/services/phishing-eml-service.ts — parseAndStoreMessage(reportId, ticketId) orchestration: list -> select -> fetch -> (B2 gated) -> parse -> persist messages+indicators" +affects: [18-campaign-grouping-api, 19-classification] + +# Tech tracking +tech-stack: + added: [] + patterns: + - "Orchestration service (no class, single exported async function) mirroring phishing-detector.ts's fetch->transform->persist shape" + - "Graceful-degrade try/catch around the B2 self-PUT step, gated behind isB2Configured(), so an unconfigured/failed B2 upload never aborts parsing/persistence" + - "Size guard on the decoded buffer BEFORE calling parseEml, returning a no-op result rather than letting parseEml's internal MAX_EML_BYTES guard throw" + +key-files: + created: + - lib/services/phishing-eml-service.ts + - lib/services/phishing-eml-service.test.ts + modified: [] + +key-decisions: + - "Task ordering followed the plan literally: Task 1 built the implementation file, Task 2 built the full test suite against it (not a strict RED-then-GREEN cycle within a single task) — both tasks were tagged tdd=\"true\" in the plan but structured as separate files/commits rather than interleaved test-then-feat commits within one task, matching how the plan's own read_first/action/verify blocks were written per task" + - "indicators inserts are plain INSERT (not upsert) — migration 097's indicators table has no natural-key unique constraint, and each parseAndStoreMessage call creates a fresh messages row, so there is nothing to conflict against" + - "reason codes ('no-eml-attachment', 'no-attachment-content', 'oversized-attachment') added to the { stored: false } result beyond the plan's single named example, to make the three distinct no-op paths distinguishable to a future caller (Phase 18) without inspecting logs" + +patterns-established: + - "Self-PUT to B2 pattern (first instance of Pulse's own server code PUTting bytes to B2 itself, vs. handing a presigned URL to an external collector) — presignUpload(objectKey, 1800, undefined, EML_OBJECT_KEY_REGEX) then fetch(url, { method: 'PUT', body })" + +requirements-completed: [EVID-03, EVID-04] + +# Metrics +duration: ~22min +completed: 2026-07-15 +--- + +# Phase 16 Plan 03: EML/MIME Evidence Orchestration Service Summary + +**`parseAndStoreMessage(reportId, ticketId)` orchestrates the full evidence pipeline — list Autotask attachments, select the original reported message, fetch its content, size-guard it, self-PUT the raw bytes to B2 when configured, parse it, and persist one `messages` row plus per-indicator `indicators` rows (attachment_hash/url/sender) with D-07 metadata — never fetching anything found in the message.** + +## Performance + +- **Duration:** ~22 min +- **Started:** 2026-07-15T14:18:00Z +- **Completed:** 2026-07-15T14:40:23Z +- **Tasks:** 2 completed +- **Files modified:** 2 (both new) + +## Accomplishments +- `lib/services/phishing-eml-service.ts` exports `parseAndStoreMessage({ reportId, ticketId })`, wiring `getAutotaskClient().getAttachments` → Plan 01's `selectOriginalMessage` → `getAttachmentContent` → a `MAX_EML_BYTES` size guard → an `isB2Configured()`-gated B2 self-PUT (`phishing/{reportId}/{attachmentId}.eml`) → Plan 01's `parseEml` → one `messages` INSERT (headers incl. D-06 structured SPF/DKIM/DMARC verdicts, urls, attachments, body_preview, raw_ref) → per-attachment-hash/per-URL/sender `indicators` INSERTs carrying D-07 `metadata` JSONB. +- Returns `{ stored: false, reason }` without throwing and without writing any row when there is no `.eml` attachment, no attachment content, or an oversized decoded buffer; a failed or absent B2 PUT degrades to `raw_ref: null` rather than aborting persistence. +- 6/6 new vitest tests pass, covering the happy path (one messages row + D-06 verdicts reaching `headers`, indicators written), the no-`.eml` no-op, B2-unconfigured (`raw_ref` null, no PUT), B2-configured (`raw_ref` set, exactly one PUT), the no-network-to-message-body-URL invariant, and the attachment-hash indicator's `metadata` payload. +- `npx tsc --noEmit --pretty` is fully clean (zero errors anywhere in the repo, not just this file). + +## Task Commits + +1. **Task 1: phishing-eml-service.ts orchestration (list→select→fetch→B2→parse→persist)** - `5088d8d` (feat) +2. **Task 2: phishing-eml-service.test.ts — orchestration coverage (mocked I/O)** - `ce16e67` (test) + +_Note: both tasks were tagged `tdd="true"` in the plan, but the plan itself structured them as implementation-first (Task 1) then full test coverage (Task 2) rather than an interleaved RED/GREEN pair within a single task — followed literally as written._ + +## Files Created/Modified +- `lib/services/phishing-eml-service.ts` (224 lines) - `parseAndStoreMessage`, `ParseAndStoreInput`/`ParseAndStoreResult` types +- `lib/services/phishing-eml-service.test.ts` (195 lines) - 6 tests: happy path, no-eml no-op, B2-unconfigured, B2-configured, no-network invariant, indicator metadata + +## Decisions Made +- **Plain `INSERT` for `indicators`, not upsert.** Migration 097's `indicators` table has no natural-key unique constraint (unlike `reports.uq_reports_ticket_id`), and each `parseAndStoreMessage` call always creates a fresh `messages` row first — there's nothing to conflict against, so `ON CONFLICT` would be dead code. +- **Three distinct no-op `reason` codes** (`no-eml-attachment`, `no-attachment-content`, `oversized-attachment`) instead of one generic no-op shape — makes the three degrade paths distinguishable to Phase 18's future caller without grepping logs, while still satisfying the plan's single named example (`no-eml-attachment`). +- **Task 1/Task 2 split followed literally**, not force-fit into a same-task RED→GREEN commit pair, since the plan's own per-task `read_first`/`action`/`verify` blocks were already split that way (Task 1's verify is `tsc` only; Task 2's verify is `vitest` + `tsc`). + +## Deviations from Plan + +None - plan executed exactly as written. Both tasks' acceptance criteria (grep checks for `selectOriginalMessage`/`parseEml`/`getAttachmentContent`/`isB2Configured`/`EML_OBJECT_KEY_REGEX`, `INSERT INTO messages`/`INSERT INTO indicators` each with `RETURNING id::text`, `authResults` reaching the headers payload, `metadata` in the indicators INSERT column list, B2 PUT gated behind `isB2Configured()`, no stray `fetch(` targeting message-derived content) were verified via grep and vitest before each commit and all passed on the first attempt. + +## Issues Encountered +None. + +## User Setup Required +None - no external service configuration required. B2 and Autotask credentials are only exercised via mocks in this plan's tests; live credentials are Phase 18's concern when the on-demand trigger route is built. + +## Next Phase Readiness +- `parseAndStoreMessage` is a stable, fully-tested export ready for Phase 18's `POST /api/phishing/tickets/{id}/analyze` route to call directly. +- No blockers identified. `messages`/`indicators` schema (migrations 097 + 099) is fully exercised by this service's insert shape. + +## Self-Check: PASSED + +- `lib/services/phishing-eml-service.ts` — FOUND +- `lib/services/phishing-eml-service.test.ts` — FOUND +- Commit `5088d8d` — FOUND in `git log` +- Commit `ce16e67` — FOUND in `git log` + +--- +*Phase: 16-eml-mime-evidence-parser* +*Completed: 2026-07-15*