docs(16-01): complete EML/MIME evidence parser core plan
Summary of Plan 16-01: mailparser + linkify-it dependency install, selectOriginalMessage (EVID-02 three-tier selection), and parseEml + parseAuthResults + extractUrls + buildBodyPreview (EVID-03/EVID-04), with a DoS size guard ahead of simpleParser. 26/26 tests pass, tsc clean. Logs 2 pre-existing unrelated itglue-search.test.ts failures to deferred-items.md (out of scope, not fixed).
This commit is contained in:
parent
654e624505
commit
d7c868a2d7
2 changed files with 138 additions and 0 deletions
121
.planning/phases/16-eml-mime-evidence-parser/16-01-SUMMARY.md
Normal file
121
.planning/phases/16-eml-mime-evidence-parser/16-01-SUMMARY.md
Normal file
|
|
@ -0,0 +1,121 @@
|
||||||
|
---
|
||||||
|
phase: 16-eml-mime-evidence-parser
|
||||||
|
plan: 01
|
||||||
|
subsystem: api
|
||||||
|
tags: [mailparser, linkify-it, mime, rfc822, spf, dkim, dmarc, phishing, vitest]
|
||||||
|
|
||||||
|
# Dependency graph
|
||||||
|
requires: []
|
||||||
|
provides:
|
||||||
|
- "lib/services/eml-parser.ts — selectOriginalMessage, parseEml, parseAuthResults, extractUrls, buildBodyPreview, NormalizedMessage type"
|
||||||
|
- "mailparser + linkify-it npm dependencies"
|
||||||
|
affects: [16-02, 16-03, 18-campaign-grouping]
|
||||||
|
|
||||||
|
# Tech tracking
|
||||||
|
tech-stack:
|
||||||
|
added: ["mailparser@3.9.14", "linkify-it@6.0.0", "@types/mailparser@3.4.6 (devDependency)"]
|
||||||
|
patterns:
|
||||||
|
- "Pure I/O-free transform module (no class, named exports, top-of-file doc-comment stating a hard no-network invariant) — mirrors lib/services/b2/client.ts's module shape"
|
||||||
|
- "Hand-rolled RFC 8601 Authentication-Results tokenizer instead of a DNS/HTTP-verifying library (mailauth rejected)"
|
||||||
|
- "describe-per-export vitest structure with synthetic-fixture-only test data"
|
||||||
|
|
||||||
|
key-files:
|
||||||
|
created:
|
||||||
|
- lib/services/eml-parser.ts
|
||||||
|
- lib/services/eml-parser.fixtures.ts
|
||||||
|
- lib/services/eml-parser.test.ts
|
||||||
|
- .planning/phases/16-eml-mime-evidence-parser/deferred-items.md
|
||||||
|
modified:
|
||||||
|
- package.json
|
||||||
|
- package-lock.json
|
||||||
|
|
||||||
|
key-decisions:
|
||||||
|
- "Used a small hand-rolled HTML-to-text stripper for buildBodyPreview's fallback path instead of importing mailparser's undeclared transitive dependency html-to-text directly (avoids depending on an unversioned, un-pinned package.json entry that could silently disappear on a future mailparser bump)"
|
||||||
|
- "linkify-it constructed with { fuzzyLink: true } to catch scheme-less www.-prefixed phishing URLs (off by default in the library)"
|
||||||
|
- "Authentication-Results / Authentication-Results-Original collected via mail.headerLines (first occurrence each), not the headers Map, since the Map only exposes the last/one occurrence of a repeated header (Pitfall 4)"
|
||||||
|
|
||||||
|
patterns-established:
|
||||||
|
- "Pattern: Buffer-in, structured-object-out parsing module with a byte-size guard enforced BEFORE handing untrusted content to a third-party parser (DoS mitigation, T-16-01)"
|
||||||
|
|
||||||
|
requirements-completed: [EVID-02, EVID-03, EVID-04]
|
||||||
|
|
||||||
|
# Metrics
|
||||||
|
duration: 12min
|
||||||
|
completed: 2026-07-15
|
||||||
|
---
|
||||||
|
|
||||||
|
# Phase 16 Plan 01: EML/MIME Evidence Parser Core Summary
|
||||||
|
|
||||||
|
**Pure, I/O-free `lib/services/eml-parser.ts` built on `mailparser` + `linkify-it`, implementing three-tier `.eml` attachment selection, RFC822/MIME normalization with hand-rolled structured SPF/DKIM/DMARC verdicts, deduped URL extraction, and a sanitized truncated body preview — all test-enforced to never trigger a network call.**
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
- **Duration:** ~12 min
|
||||||
|
- **Started:** 2026-07-15T14:19:00Z (worktree setup + npm install)
|
||||||
|
- **Completed:** 2026-07-15T14:30:18Z
|
||||||
|
- **Tasks:** 3 completed (1 auto, 2 TDD)
|
||||||
|
- **Files modified:** 5 (2 dependency files + 3 new source files) plus 1 new deferred-items tracking doc
|
||||||
|
|
||||||
|
## Accomplishments
|
||||||
|
- Installed `mailparser` 3.9.14 + `linkify-it` 6.0.0 (+ `@types/mailparser` devDependency) with a reviewed `git diff package.json` gate confirming no unrelated dependency churn
|
||||||
|
- Implemented `selectOriginalMessage` — the three-tier `.eml` attachment selection algorithm (exact `rfc.eml` → single non-`OriginatingEmail.eml` `message/rfc822` candidate → `OriginatingEmail.eml` fallback → null), validated against all real-world shapes found in Phase 16 research (Microsoft Report Message, KnowBe4 PhishER, single-attachment legacy tickets, and both ambiguous edge cases)
|
||||||
|
- Implemented `parseEml` — normalizes From/Reply-To/Return-Path/To/Cc/Subject/Date/Message-ID/Received-chain, structured `authResults`/`authResultsOriginal` verdicts, deduped URLs, and per-attachment metadata (filename/content-type/size/sha256 checksum/`related` flag for inline-CID parts), guarded by `MAX_EML_BYTES` (10 MB, below B2's 25 MB cap) enforced before `simpleParser` is ever called
|
||||||
|
- Implemented `parseAuthResults` (hand-rolled RFC 8601 tokenizer), `extractUrls` (linkify-it with fuzzy `www.` matching), and `buildBodyPreview` (truncated to 500 chars, HTML-stripped fallback)
|
||||||
|
- 26/26 vitest tests pass, `tsc --noEmit` clean for all eml-parser files, and a `global.fetch` spy confirms zero network calls across every synthetic fixture (EVID-04 hard invariant)
|
||||||
|
|
||||||
|
## Task Commits
|
||||||
|
|
||||||
|
1. **Task 1: Install mailparser + linkify-it (deliberate, reviewed)** - `0f4dc1f` (chore)
|
||||||
|
2. **Task 2: Attachment selection (EVID-02, three tiers) — RED** - `2fde115` (test)
|
||||||
|
2. **Task 2: Attachment selection (EVID-02, three tiers) — GREEN** - `e4718ae` (feat)
|
||||||
|
3. **Task 3: parseEml + auth-results + URLs + body preview + size guard — RED** - `4df4816` (test)
|
||||||
|
3. **Task 3: parseEml + auth-results + URLs + body preview + size guard — GREEN** - `654e624` (feat)
|
||||||
|
|
||||||
|
_Note: this is a `type: tdd` plan — Tasks 2 and 3 each have a RED (test) commit followed by a GREEN (feat) commit, no refactor commit was needed._
|
||||||
|
|
||||||
|
## Files Created/Modified
|
||||||
|
- `lib/services/eml-parser.ts` (282 lines) - `selectOriginalMessage`, `parseEml`, `parseAuthResults`, `extractUrls`, `buildBodyPreview`, `MAX_EML_BYTES`, `NormalizedMessage`/`AuthResults`/`AttachmentMeta`/`AuthVerdict` types
|
||||||
|
- `lib/services/eml-parser.fixtures.ts` (227 lines) - synthetic attachment-list fixtures (3 selection tiers + ambiguous/no-eml/empty edge cases) and synthetic raw `.eml` buffers (rich multipart, auth-results-original, inline/CID attachment, long-body, fuzzy-URL, oversized-buffer generator)
|
||||||
|
- `lib/services/eml-parser.test.ts` (219 lines) - 26 tests across `describe('selectOriginalMessage')`, `describe('parseAuthResults')`, `describe('extractUrls')`, `describe('buildBodyPreview')`, `describe('parseEml')`
|
||||||
|
- `package.json` / `package-lock.json` - added `mailparser`, `linkify-it`, `@types/mailparser`
|
||||||
|
- `.planning/phases/16-eml-mime-evidence-parser/deferred-items.md` - logs 2 pre-existing, unrelated `itglue-search.test.ts` failures found via `npm test` (out of scope, not fixed)
|
||||||
|
|
||||||
|
## Decisions Made
|
||||||
|
- **HTML-to-text fallback implemented by hand, not via `html-to-text`.** RESEARCH.md/the plan suggested using `html-to-text` (already a `mailparser` transitive dependency) for `buildBodyPreview`'s HTML fallback. It ships no bundled TypeScript types and isn't a declared direct dependency — importing it would depend on an un-pinned, unversioned transitive package that could silently disappear on a future `mailparser` version bump. Wrote a small (10-line) regex-based tag/entity stripper instead, fully covered by tests. This was within Claude's explicit discretion per CONTEXT.md ("URL extraction scope... implementation detail, not user-relevant preference").
|
||||||
|
- **`linkify-it` constructed with `{ fuzzyLink: true }`.** Verified empirically that linkify-it's default options do NOT enable scheme-less `www.`-prefixed URL matching (`fuzzyLink` defaults to `false`) — without this option, a phishing URL like `www.evil-example.com/login` (no `http://` prefix) would silently not be extracted, undermining EVID-03's URL-extraction requirement.
|
||||||
|
- **Authentication-Results / Authentication-Results-Original parsed from `mail.headerLines`, not `mail.headers.get(...)`.** Confirmed via research and this session's own live testing that `mail.headers` (a Map) collapses repeated header occurrences; `headerLines` (an ordered array) is the only way to reliably locate both headers when present (Pitfall 4).
|
||||||
|
|
||||||
|
## Deviations from Plan
|
||||||
|
|
||||||
|
### Auto-fixed Issues
|
||||||
|
|
||||||
|
**1. [Rule 1 - Test discoverability] Renamed a test title to match the plan's required `-t` filter**
|
||||||
|
- **Found during:** Task 3 verification
|
||||||
|
- **Issue:** The plan's acceptance criteria require `npx vitest run lib/services/eml-parser.test.ts -t "no network"` to pass. The initial test title ("never makes a network call while parsing any fixture") did not contain the literal substring "no network", so the `-t` filter matched zero tests.
|
||||||
|
- **Fix:** Renamed the test to "makes no network call (no network fetch) while parsing any fixture".
|
||||||
|
- **Files modified:** lib/services/eml-parser.test.ts
|
||||||
|
- **Verification:** `npx vitest run lib/services/eml-parser.test.ts -t "no network"` now passes (1 test).
|
||||||
|
- **Committed in:** `654e624` (part of Task 3 GREEN commit)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Total deviations:** 1 auto-fixed (test-naming correctness, no behavior change)
|
||||||
|
**Impact on plan:** Cosmetic only — no scope creep, no behavior change. All three EVID-02/03/04 requirements implemented exactly as specified.
|
||||||
|
|
||||||
|
## Issues Encountered
|
||||||
|
- This worktree had no `node_modules` at all on start (fresh git worktree checkout) — ran a full `npm install` from the existing `package-lock.json` before the plan's own Task 1 dependency install, so `mailparser`/`linkify-it` could resolve correctly. Not a plan deviation; this is normal worktree bootstrap, not tracked as a commit.
|
||||||
|
- `npm test` (full suite) surfaced 2 pre-existing failures in `lib/services/analyzer/itglue-search.test.ts`, unrelated to any file this plan touched (last commit on that file predates this plan). Logged to `deferred-items.md`, not fixed, per the executor's scope-boundary rule.
|
||||||
|
|
||||||
|
## User Setup Required
|
||||||
|
|
||||||
|
None - no external service configuration required. `mailparser`/`linkify-it` need no credentials; B2 storage (D-05) and the Autotask attachment-content fetch are Plan 16-03's concern, not this plan's.
|
||||||
|
|
||||||
|
## Next Phase Readiness
|
||||||
|
|
||||||
|
`lib/services/eml-parser.ts`'s exported contract (`selectOriginalMessage`, `parseEml`, `NormalizedMessage`, `AuthResults`, `AttachmentMeta`, `MAX_EML_BYTES`) is stable and ready for Plan 16-03's orchestration layer (fetch attachment content from Autotask → upload to B2 → `parseEml` → persist `messages`/`indicators` rows). No blockers identified for 16-02 or 16-03.
|
||||||
|
|
||||||
|
---
|
||||||
|
*Phase: 16-eml-mime-evidence-parser*
|
||||||
|
*Completed: 2026-07-15*
|
||||||
|
|
||||||
|
## Self-Check: PASSED
|
||||||
|
|
@ -0,0 +1,17 @@
|
||||||
|
# Deferred Items — Phase 16
|
||||||
|
|
||||||
|
Items discovered during execution that are out of scope for the current plan
|
||||||
|
and were not fixed (per executor scope-boundary rules).
|
||||||
|
|
||||||
|
## From Plan 16-01
|
||||||
|
|
||||||
|
- **`lib/services/analyzer/itglue-search.test.ts`** — 2 pre-existing test
|
||||||
|
failures (`returns capped, redacted doc snippets when the org is found`,
|
||||||
|
`tolerates per-call failures (configurations errors, flex still returns)`),
|
||||||
|
observed via `npm test` while verifying Plan 16-01. Unrelated to
|
||||||
|
`eml-parser.ts`/`eml-parser.fixtures.ts`/`eml-parser.test.ts` (no files in
|
||||||
|
`lib/services/analyzer/` were touched by this plan). Last commit touching
|
||||||
|
that file (`a0a6e7f fix(itglue): list flexible assets per type to satisfy
|
||||||
|
API 422 requirement`) predates this plan's work. Not fixed — out of scope
|
||||||
|
per the executor's scope-boundary rule (only auto-fix issues directly
|
||||||
|
caused by the current task's changes).
|
||||||
Loading…
Add table
Add a link
Reference in a new issue