docs(16-03): complete EML/MIME evidence orchestration service plan

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.planning/phases/16-eml-mime-evidence-parser/16-03-SUMMARY.md

@@ -0,0 +1,103 @@
+---
+phase: 16-eml-mime-evidence-parser
+plan: 03
+subsystem: api
+tags: [autotask, backblaze-b2, postgres, mailparser, phishing, vitest]
+
+# Dependency graph
+requires:
+  - phase: 16-eml-mime-evidence-parser (Plan 01)
+    provides: "lib/services/eml-parser.ts — selectOriginalMessage, parseEml, MAX_EML_BYTES, NormalizedMessage"
+  - phase: 16-eml-mime-evidence-parser (Plan 02)
+    provides: "AutotaskClient.getAttachmentContent, B2 EML_OBJECT_KEY_REGEX + parameterized presignUpload, migrations/099 indicators.metadata"
+provides:
+  - "lib/services/phishing-eml-service.ts — parseAndStoreMessage(reportId, ticketId) orchestration: list -> select -> fetch -> (B2 gated) -> parse -> persist messages+indicators"
+affects: [18-campaign-grouping-api, 19-classification]
+
+# Tech tracking
+tech-stack:
+  added: []
+  patterns:
+    - "Orchestration service (no class, single exported async function) mirroring phishing-detector.ts's fetch->transform->persist shape"
+    - "Graceful-degrade try/catch around the B2 self-PUT step, gated behind isB2Configured(), so an unconfigured/failed B2 upload never aborts parsing/persistence"
+    - "Size guard on the decoded buffer BEFORE calling parseEml, returning a no-op result rather than letting parseEml's internal MAX_EML_BYTES guard throw"
+
+key-files:
+  created:
+    - lib/services/phishing-eml-service.ts
+    - lib/services/phishing-eml-service.test.ts
+  modified: []
+
+key-decisions:
+  - "Task ordering followed the plan literally: Task 1 built the implementation file, Task 2 built the full test suite against it (not a strict RED-then-GREEN cycle within a single task) — both tasks were tagged tdd=\"true\" in the plan but structured as separate files/commits rather than interleaved test-then-feat commits within one task, matching how the plan's own read_first/action/verify blocks were written per task"
+  - "indicators inserts are plain INSERT (not upsert) — migration 097's indicators table has no natural-key unique constraint, and each parseAndStoreMessage call creates a fresh messages row, so there is nothing to conflict against"
+  - "reason codes ('no-eml-attachment', 'no-attachment-content', 'oversized-attachment') added to the { stored: false } result beyond the plan's single named example, to make the three distinct no-op paths distinguishable to a future caller (Phase 18) without inspecting logs"
+
+patterns-established:
+  - "Self-PUT to B2 pattern (first instance of Pulse's own server code PUTting bytes to B2 itself, vs. handing a presigned URL to an external collector) — presignUpload(objectKey, 1800, undefined, EML_OBJECT_KEY_REGEX) then fetch(url, { method: 'PUT', body })"
+
+requirements-completed: [EVID-03, EVID-04]
+
+# Metrics
+duration: ~22min
+completed: 2026-07-15
+---
+
+# Phase 16 Plan 03: EML/MIME Evidence Orchestration Service Summary
+
+**`parseAndStoreMessage(reportId, ticketId)` orchestrates the full evidence pipeline — list Autotask attachments, select the original reported message, fetch its content, size-guard it, self-PUT the raw bytes to B2 when configured, parse it, and persist one `messages` row plus per-indicator `indicators` rows (attachment_hash/url/sender) with D-07 metadata — never fetching anything found in the message.**
+
+## Performance
+
+- **Duration:** ~22 min
+- **Started:** 2026-07-15T14:18:00Z
+- **Completed:** 2026-07-15T14:40:23Z
+- **Tasks:** 2 completed
+- **Files modified:** 2 (both new)
+
+## Accomplishments
+- `lib/services/phishing-eml-service.ts` exports `parseAndStoreMessage({ reportId, ticketId })`, wiring `getAutotaskClient().getAttachments` → Plan 01's `selectOriginalMessage` → `getAttachmentContent` → a `MAX_EML_BYTES` size guard → an `isB2Configured()`-gated B2 self-PUT (`phishing/{reportId}/{attachmentId}.eml`) → Plan 01's `parseEml` → one `messages` INSERT (headers incl. D-06 structured SPF/DKIM/DMARC verdicts, urls, attachments, body_preview, raw_ref) → per-attachment-hash/per-URL/sender `indicators` INSERTs carrying D-07 `metadata` JSONB.
+- Returns `{ stored: false, reason }` without throwing and without writing any row when there is no `.eml` attachment, no attachment content, or an oversized decoded buffer; a failed or absent B2 PUT degrades to `raw_ref: null` rather than aborting persistence.
+- 6/6 new vitest tests pass, covering the happy path (one messages row + D-06 verdicts reaching `headers`, indicators written), the no-`.eml` no-op, B2-unconfigured (`raw_ref` null, no PUT), B2-configured (`raw_ref` set, exactly one PUT), the no-network-to-message-body-URL invariant, and the attachment-hash indicator's `metadata` payload.
+- `npx tsc --noEmit --pretty` is fully clean (zero errors anywhere in the repo, not just this file).
+
+## Task Commits
+
+1. **Task 1: phishing-eml-service.ts orchestration (list→select→fetch→B2→parse→persist)** - `5088d8d` (feat)
+2. **Task 2: phishing-eml-service.test.ts — orchestration coverage (mocked I/O)** - `ce16e67` (test)
+
+_Note: both tasks were tagged `tdd="true"` in the plan, but the plan itself structured them as implementation-first (Task 1) then full test coverage (Task 2) rather than an interleaved RED/GREEN pair within a single task — followed literally as written._
+
+## Files Created/Modified
+- `lib/services/phishing-eml-service.ts` (224 lines) - `parseAndStoreMessage`, `ParseAndStoreInput`/`ParseAndStoreResult` types
+- `lib/services/phishing-eml-service.test.ts` (195 lines) - 6 tests: happy path, no-eml no-op, B2-unconfigured, B2-configured, no-network invariant, indicator metadata
+
+## Decisions Made
+- **Plain `INSERT` for `indicators`, not upsert.** Migration 097's `indicators` table has no natural-key unique constraint (unlike `reports.uq_reports_ticket_id`), and each `parseAndStoreMessage` call always creates a fresh `messages` row first — there's nothing to conflict against, so `ON CONFLICT` would be dead code.
+- **Three distinct no-op `reason` codes** (`no-eml-attachment`, `no-attachment-content`, `oversized-attachment`) instead of one generic no-op shape — makes the three degrade paths distinguishable to Phase 18's future caller without grepping logs, while still satisfying the plan's single named example (`no-eml-attachment`).
+- **Task 1/Task 2 split followed literally**, not force-fit into a same-task RED→GREEN commit pair, since the plan's own per-task `read_first`/`action`/`verify` blocks were already split that way (Task 1's verify is `tsc` only; Task 2's verify is `vitest` + `tsc`).
+
+## Deviations from Plan
+
+None - plan executed exactly as written. Both tasks' acceptance criteria (grep checks for `selectOriginalMessage`/`parseEml`/`getAttachmentContent`/`isB2Configured`/`EML_OBJECT_KEY_REGEX`, `INSERT INTO messages`/`INSERT INTO indicators` each with `RETURNING id::text`, `authResults` reaching the headers payload, `metadata` in the indicators INSERT column list, B2 PUT gated behind `isB2Configured()`, no stray `fetch(` targeting message-derived content) were verified via grep and vitest before each commit and all passed on the first attempt.
+
+## Issues Encountered
+None.
+
+## User Setup Required
+None - no external service configuration required. B2 and Autotask credentials are only exercised via mocks in this plan's tests; live credentials are Phase 18's concern when the on-demand trigger route is built.
+
+## Next Phase Readiness
+- `parseAndStoreMessage` is a stable, fully-tested export ready for Phase 18's `POST /api/phishing/tickets/{id}/analyze` route to call directly.
+- No blockers identified. `messages`/`indicators` schema (migrations 097 + 099) is fully exercised by this service's insert shape.
+
+## Self-Check: PASSED
+
+- `lib/services/phishing-eml-service.ts` — FOUND
+- `lib/services/phishing-eml-service.test.ts` — FOUND
+- Commit `5088d8d` — FOUND in `git log`
+- Commit `ce16e67` — FOUND in `git log`
+
+---
+*Phase: 16-eml-mime-evidence-parser*
+*Completed: 2026-07-15*