From d7c868a2d7bc4e8e0cbb73f55afc23dc9f86bea2 Mon Sep 17 00:00:00 2001 From: lorentz Date: Wed, 15 Jul 2026 10:31:45 -0400 Subject: [PATCH] docs(16-01): complete EML/MIME evidence parser core plan Summary of Plan 16-01: mailparser + linkify-it dependency install, selectOriginalMessage (EVID-02 three-tier selection), and parseEml + parseAuthResults + extractUrls + buildBodyPreview (EVID-03/EVID-04), with a DoS size guard ahead of simpleParser. 26/26 tests pass, tsc clean. Logs 2 pre-existing unrelated itglue-search.test.ts failures to deferred-items.md (out of scope, not fixed). --- .../16-01-SUMMARY.md | 121 ++++++++++++++++++ .../deferred-items.md | 17 +++ 2 files changed, 138 insertions(+) create mode 100644 .planning/phases/16-eml-mime-evidence-parser/16-01-SUMMARY.md create mode 100644 .planning/phases/16-eml-mime-evidence-parser/deferred-items.md diff --git a/.planning/phases/16-eml-mime-evidence-parser/16-01-SUMMARY.md b/.planning/phases/16-eml-mime-evidence-parser/16-01-SUMMARY.md new file mode 100644 index 0000000..dbb8c3c --- /dev/null +++ b/.planning/phases/16-eml-mime-evidence-parser/16-01-SUMMARY.md @@ -0,0 +1,121 @@ +--- +phase: 16-eml-mime-evidence-parser +plan: 01 +subsystem: api +tags: [mailparser, linkify-it, mime, rfc822, spf, dkim, dmarc, phishing, vitest] + +# Dependency graph +requires: [] +provides: + - "lib/services/eml-parser.ts — selectOriginalMessage, parseEml, parseAuthResults, extractUrls, buildBodyPreview, NormalizedMessage type" + - "mailparser + linkify-it npm dependencies" +affects: [16-02, 16-03, 18-campaign-grouping] + +# Tech tracking +tech-stack: + added: ["mailparser@3.9.14", "linkify-it@6.0.0", "@types/mailparser@3.4.6 (devDependency)"] + patterns: + - "Pure I/O-free transform module (no class, named exports, top-of-file doc-comment stating a hard no-network invariant) — mirrors lib/services/b2/client.ts's module shape" + - "Hand-rolled RFC 8601 Authentication-Results tokenizer instead of a DNS/HTTP-verifying library (mailauth rejected)" + - "describe-per-export vitest structure with synthetic-fixture-only test data" + +key-files: + created: + - lib/services/eml-parser.ts + - lib/services/eml-parser.fixtures.ts + - lib/services/eml-parser.test.ts + - .planning/phases/16-eml-mime-evidence-parser/deferred-items.md + modified: + - package.json + - package-lock.json + +key-decisions: + - "Used a small hand-rolled HTML-to-text stripper for buildBodyPreview's fallback path instead of importing mailparser's undeclared transitive dependency html-to-text directly (avoids depending on an unversioned, un-pinned package.json entry that could silently disappear on a future mailparser bump)" + - "linkify-it constructed with { fuzzyLink: true } to catch scheme-less www.-prefixed phishing URLs (off by default in the library)" + - "Authentication-Results / Authentication-Results-Original collected via mail.headerLines (first occurrence each), not the headers Map, since the Map only exposes the last/one occurrence of a repeated header (Pitfall 4)" + +patterns-established: + - "Pattern: Buffer-in, structured-object-out parsing module with a byte-size guard enforced BEFORE handing untrusted content to a third-party parser (DoS mitigation, T-16-01)" + +requirements-completed: [EVID-02, EVID-03, EVID-04] + +# Metrics +duration: 12min +completed: 2026-07-15 +--- + +# Phase 16 Plan 01: EML/MIME Evidence Parser Core Summary + +**Pure, I/O-free `lib/services/eml-parser.ts` built on `mailparser` + `linkify-it`, implementing three-tier `.eml` attachment selection, RFC822/MIME normalization with hand-rolled structured SPF/DKIM/DMARC verdicts, deduped URL extraction, and a sanitized truncated body preview — all test-enforced to never trigger a network call.** + +## Performance + +- **Duration:** ~12 min +- **Started:** 2026-07-15T14:19:00Z (worktree setup + npm install) +- **Completed:** 2026-07-15T14:30:18Z +- **Tasks:** 3 completed (1 auto, 2 TDD) +- **Files modified:** 5 (2 dependency files + 3 new source files) plus 1 new deferred-items tracking doc + +## Accomplishments +- Installed `mailparser` 3.9.14 + `linkify-it` 6.0.0 (+ `@types/mailparser` devDependency) with a reviewed `git diff package.json` gate confirming no unrelated dependency churn +- Implemented `selectOriginalMessage` — the three-tier `.eml` attachment selection algorithm (exact `rfc.eml` → single non-`OriginatingEmail.eml` `message/rfc822` candidate → `OriginatingEmail.eml` fallback → null), validated against all real-world shapes found in Phase 16 research (Microsoft Report Message, KnowBe4 PhishER, single-attachment legacy tickets, and both ambiguous edge cases) +- Implemented `parseEml` — normalizes From/Reply-To/Return-Path/To/Cc/Subject/Date/Message-ID/Received-chain, structured `authResults`/`authResultsOriginal` verdicts, deduped URLs, and per-attachment metadata (filename/content-type/size/sha256 checksum/`related` flag for inline-CID parts), guarded by `MAX_EML_BYTES` (10 MB, below B2's 25 MB cap) enforced before `simpleParser` is ever called +- Implemented `parseAuthResults` (hand-rolled RFC 8601 tokenizer), `extractUrls` (linkify-it with fuzzy `www.` matching), and `buildBodyPreview` (truncated to 500 chars, HTML-stripped fallback) +- 26/26 vitest tests pass, `tsc --noEmit` clean for all eml-parser files, and a `global.fetch` spy confirms zero network calls across every synthetic fixture (EVID-04 hard invariant) + +## Task Commits + +1. **Task 1: Install mailparser + linkify-it (deliberate, reviewed)** - `0f4dc1f` (chore) +2. **Task 2: Attachment selection (EVID-02, three tiers) — RED** - `2fde115` (test) +2. **Task 2: Attachment selection (EVID-02, three tiers) — GREEN** - `e4718ae` (feat) +3. **Task 3: parseEml + auth-results + URLs + body preview + size guard — RED** - `4df4816` (test) +3. **Task 3: parseEml + auth-results + URLs + body preview + size guard — GREEN** - `654e624` (feat) + +_Note: this is a `type: tdd` plan — Tasks 2 and 3 each have a RED (test) commit followed by a GREEN (feat) commit, no refactor commit was needed._ + +## Files Created/Modified +- `lib/services/eml-parser.ts` (282 lines) - `selectOriginalMessage`, `parseEml`, `parseAuthResults`, `extractUrls`, `buildBodyPreview`, `MAX_EML_BYTES`, `NormalizedMessage`/`AuthResults`/`AttachmentMeta`/`AuthVerdict` types +- `lib/services/eml-parser.fixtures.ts` (227 lines) - synthetic attachment-list fixtures (3 selection tiers + ambiguous/no-eml/empty edge cases) and synthetic raw `.eml` buffers (rich multipart, auth-results-original, inline/CID attachment, long-body, fuzzy-URL, oversized-buffer generator) +- `lib/services/eml-parser.test.ts` (219 lines) - 26 tests across `describe('selectOriginalMessage')`, `describe('parseAuthResults')`, `describe('extractUrls')`, `describe('buildBodyPreview')`, `describe('parseEml')` +- `package.json` / `package-lock.json` - added `mailparser`, `linkify-it`, `@types/mailparser` +- `.planning/phases/16-eml-mime-evidence-parser/deferred-items.md` - logs 2 pre-existing, unrelated `itglue-search.test.ts` failures found via `npm test` (out of scope, not fixed) + +## Decisions Made +- **HTML-to-text fallback implemented by hand, not via `html-to-text`.** RESEARCH.md/the plan suggested using `html-to-text` (already a `mailparser` transitive dependency) for `buildBodyPreview`'s HTML fallback. It ships no bundled TypeScript types and isn't a declared direct dependency — importing it would depend on an un-pinned, unversioned transitive package that could silently disappear on a future `mailparser` version bump. Wrote a small (10-line) regex-based tag/entity stripper instead, fully covered by tests. This was within Claude's explicit discretion per CONTEXT.md ("URL extraction scope... implementation detail, not user-relevant preference"). +- **`linkify-it` constructed with `{ fuzzyLink: true }`.** Verified empirically that linkify-it's default options do NOT enable scheme-less `www.`-prefixed URL matching (`fuzzyLink` defaults to `false`) — without this option, a phishing URL like `www.evil-example.com/login` (no `http://` prefix) would silently not be extracted, undermining EVID-03's URL-extraction requirement. +- **Authentication-Results / Authentication-Results-Original parsed from `mail.headerLines`, not `mail.headers.get(...)`.** Confirmed via research and this session's own live testing that `mail.headers` (a Map) collapses repeated header occurrences; `headerLines` (an ordered array) is the only way to reliably locate both headers when present (Pitfall 4). + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Test discoverability] Renamed a test title to match the plan's required `-t` filter** +- **Found during:** Task 3 verification +- **Issue:** The plan's acceptance criteria require `npx vitest run lib/services/eml-parser.test.ts -t "no network"` to pass. The initial test title ("never makes a network call while parsing any fixture") did not contain the literal substring "no network", so the `-t` filter matched zero tests. +- **Fix:** Renamed the test to "makes no network call (no network fetch) while parsing any fixture". +- **Files modified:** lib/services/eml-parser.test.ts +- **Verification:** `npx vitest run lib/services/eml-parser.test.ts -t "no network"` now passes (1 test). +- **Committed in:** `654e624` (part of Task 3 GREEN commit) + +--- + +**Total deviations:** 1 auto-fixed (test-naming correctness, no behavior change) +**Impact on plan:** Cosmetic only — no scope creep, no behavior change. All three EVID-02/03/04 requirements implemented exactly as specified. + +## Issues Encountered +- This worktree had no `node_modules` at all on start (fresh git worktree checkout) — ran a full `npm install` from the existing `package-lock.json` before the plan's own Task 1 dependency install, so `mailparser`/`linkify-it` could resolve correctly. Not a plan deviation; this is normal worktree bootstrap, not tracked as a commit. +- `npm test` (full suite) surfaced 2 pre-existing failures in `lib/services/analyzer/itglue-search.test.ts`, unrelated to any file this plan touched (last commit on that file predates this plan). Logged to `deferred-items.md`, not fixed, per the executor's scope-boundary rule. + +## User Setup Required + +None - no external service configuration required. `mailparser`/`linkify-it` need no credentials; B2 storage (D-05) and the Autotask attachment-content fetch are Plan 16-03's concern, not this plan's. + +## Next Phase Readiness + +`lib/services/eml-parser.ts`'s exported contract (`selectOriginalMessage`, `parseEml`, `NormalizedMessage`, `AuthResults`, `AttachmentMeta`, `MAX_EML_BYTES`) is stable and ready for Plan 16-03's orchestration layer (fetch attachment content from Autotask → upload to B2 → `parseEml` → persist `messages`/`indicators` rows). No blockers identified for 16-02 or 16-03. + +--- +*Phase: 16-eml-mime-evidence-parser* +*Completed: 2026-07-15* + +## Self-Check: PASSED diff --git a/.planning/phases/16-eml-mime-evidence-parser/deferred-items.md b/.planning/phases/16-eml-mime-evidence-parser/deferred-items.md new file mode 100644 index 0000000..8c927f2 --- /dev/null +++ b/.planning/phases/16-eml-mime-evidence-parser/deferred-items.md @@ -0,0 +1,17 @@ +# Deferred Items — Phase 16 + +Items discovered during execution that are out of scope for the current plan +and were not fixed (per executor scope-boundary rules). + +## From Plan 16-01 + +- **`lib/services/analyzer/itglue-search.test.ts`** — 2 pre-existing test + failures (`returns capped, redacted doc snippets when the org is found`, + `tolerates per-call failures (configurations errors, flex still returns)`), + observed via `npm test` while verifying Plan 16-01. Unrelated to + `eml-parser.ts`/`eml-parser.fixtures.ts`/`eml-parser.test.ts` (no files in + `lib/services/analyzer/` were touched by this plan). Last commit touching + that file (`a0a6e7f fix(itglue): list flexible assets per type to satisfy + API 422 requirement`) predates this plan's work. Not fixed — out of scope + per the executor's scope-boundary rule (only auto-fix issues directly + caused by the current task's changes).