wulf-pulse/.planning/phases/16-eml-mime-evidence-parser/16-VERIFICATION.md

---
phase: 16-eml-mime-evidence-parser
verified: 2026-07-15T14:44:12Z
status: passed
score: 8/8 must-haves verified
overrides_applied: 0
---

# Phase 16: EML/MIME Evidence Parser Verification Report

**Phase Goal:** Given a ticket's attachments, Pulse selects the correct original reported message and parses its RFC822/MIME structure into normalized, actionable evidence — without ever executing or fetching anything from the message.
**Verified:** 2026-07-15T14:44:12Z
**Status:** passed
**Re-verification:** No — initial verification

## Goal Achievement

### Observable Truths (ROADMAP Success Criteria + PLAN must_haves, merged)

| # | Truth | Status | Evidence |
|---|-------|--------|----------|
| 1 | Given synthetic fixtures with both `rfc.eml` and `OriginatingEmail.eml` present, selection picks `rfc.eml` matching by `message/rfc822` content-type, not filename alone | ✓ VERIFIED | `lib/services/eml-parser.ts:50-65` `selectOriginalMessage` filters `isMessageRfc822` first, then matches exact `rfc.eml`; test `eml-parser.test.ts:28-37` asserts tier 1 + case-insensitive tier 1; also covers KnowBe4 tier 2 (`:39-43`) and OriginatingEmail-only tier 3 (`:45-49`) and ambiguous/null cases (`:51-69`). All pass live (`npx vitest run` — 49/49 across the 4 phase test files). |
| 2 | Parsing a synthetic `.eml` fixture produces normalized headers (From/displayName/senderEmail/senderDomain/Reply-To/Return-Path/To/Cc/Subject/Date/Message-ID/Received chain, SPF/DKIM/DMARC), URLs, and attachment metadata (name/content-type/size/hash) | ✓ VERIFIED | `parseEml` (`eml-parser.ts:227-282`) builds exactly this shape from `mailparser`'s `ParsedMail`; test `eml-parser.test.ts:137-164` asserts every field against `RICH_MULTIPART_EML`, including `authResults` structured verdicts (not raw text), sha256 checksum format, and `related` flag. Runs against real `simpleParser` (no mocking of the parser itself) — not a tautology. |
| 3 | The parser never executes or fetches any URL found in a message — verified by tests asserting no outbound network calls happen during parsing | ✓ VERIFIED | `eml-parser.test.ts:188-197` spies on `global.fetch` across 5 fixture parses and asserts zero calls; `extractUrls`/`buildBodyPreview` are pure string operations (`eml-parser.ts:140-184`) with no fetch/render path. Orchestration-level (service) invariant separately verified below (Truth 6). |
| 4 | Parsed output includes a sanitized/truncated body preview stored alongside raw evidence, distinct from the full raw body | ✓ VERIFIED | `buildBodyPreview` (`eml-parser.ts:176-184`) truncates to `MAX_BODY_PREVIEW_LENGTH=500` and strips HTML via `stripHtmlToText`; test `eml-parser.test.ts:128-133,178-181` confirms truncation and distinctness on a 2000-char fixture. Raw bytes are never persisted to Postgres — only to B2 (`raw_ref` is an object key, `phishing-eml-service.ts:97-106,142-157`), consistent with D-05. |
| 5 | `npx vitest run` for the new parser test file passes using synthetic fixtures only (no real customer email) | ✓ VERIFIED | Ran live: `eml-parser.test.ts` 26/26 pass. A dedicated test (`eml-parser.test.ts:205-218`) asserts no fixture contains `wulfconsulting.com` and all use RFC 2606 reserved `.test`/`.com`-fake (`evil-example.test`) domains. `deferred-items.md` documents the 2 pre-existing unrelated `itglue-search.test.ts` failures (confirmed via `git log` — last touched at commit `a0a6e7f`/`8f8b5ab`, predating all Phase 16 commits). |
| 6 | (16-02/16-03 supporting truth) AutotaskClient.getAttachmentContent reads `response.items?.[0]`, not `.item`; b2 EML_OBJECT_KEY_REGEX enforces path-traversal-safe `.eml` keys in parallel with the untouched LogLift regex; migration 099 adds `indicators.metadata` JSONB | ✓ VERIFIED | `autotask-client.ts:443-456` reads `response.items?.[0] ?? null`; test asserts an `{item:...}`-shaped response yields null (guards the exact regression risk called out in the plan). `b2/client.ts:41-42` adds `EML_OBJECT_KEY_REGEX` beside the unmodified `OBJECT_KEY_REGEX` (`:31-32`); `presignDownload`/`presignUpload`/`downloadToBuffer` (`:155-218`) take an optional `keyRegex` param defaulting to `OBJECT_KEY_REGEX`; `rmm/executor.ts:337`'s existing 2-arg call site still compiles (confirmed via `tsc --noEmit`, exit 0). Migration 099 applied live: `information_schema.columns` on the dev DB (`pulse-postgres`) reports `indicators.metadata` as `jsonb`. |
| 7 | Orchestration service (16-03) wires list→select→fetch→(B2 gated)→parse→persist end to end, writing one `messages` row + `indicators` rows with D-07 metadata, and no-op (no throw) when no `.eml` attachment or B2 unconfigured | ✓ VERIFIED | `phishing-eml-service.ts:49-224` imports and calls `getAutotaskClient().getAttachments`, `selectOriginalMessage`, `getAttachmentContent`, `isB2Configured`/`presignUpload`/`EML_OBJECT_KEY_REGEX`, `parseEml` — exact function signatures match Plan 01/02's exports (verified by reading both source files side by side). `messages` INSERT (`:142-157`) and 3 `indicators` INSERT loops (`:164-211`) match migration 097/099's actual column list (`report_id, message_id, headers, urls, attachments, body_preview, raw_ref` / `message_id, indicator_type, value, metadata`). Test suite (`phishing-eml-service.test.ts`, 6/6 passing) exercises the happy path (real `parseEml` invoked on `RICH_MULTIPART_EML`, only DB/Autotask/B2 boundaries mocked — not tautological), no-eml no-op, B2-configured/unconfigured branches, no-network invariant, and indicator metadata shape. |
| 8 | No outbound network call is made to any URL found in the message, end to end (service level) | ✓ VERIFIED | `phishing-eml-service.test.ts:155-170` asserts `global.fetch` is called exactly once (the B2 presigned PUT) and never with the fixture's embedded body URL (`http://evil-example.test/verify`). Source review confirms the only `fetch(` call in `phishing-eml-service.ts` targets `uploadUrl` (a B2-presigned URL Pulse itself constructed), never a value derived from `normalized.urls`. |

**Score:** 8/8 truths verified

### Required Artifacts

| Artifact | Expected | Status | Details |
|----------|----------|--------|---------|
| `lib/services/eml-parser.ts` | parseEml, selectOriginalMessage, parseAuthResults, extractUrls, buildBodyPreview, NormalizedMessage (min 120 lines) | ✓ VERIFIED | 282 lines; all 5 functions + types exported exactly per plan interface contract |
| `lib/services/eml-parser.fixtures.ts` | synthetic fixtures for all 3 selection tiers + parse fixtures | ✓ VERIFIED | Exists, imported and exercised by both `eml-parser.test.ts` and `phishing-eml-service.test.ts`; RFC 2606-safe synthetic content only |
| `lib/services/eml-parser.test.ts` | EVID-02/03/04 vitest coverage incl. no-network spy + 3-tier selection (min 100 lines) | ✓ VERIFIED | 219 lines, 26 tests, all passing live |
| `lib/services/autotask-client.ts` | getAttachmentContent(entityName, entityId, attachmentId) | ✓ VERIFIED | Present at line 443, reads `items?.[0] ?? null` |
| `lib/services/autotask-client.test.ts` | first AutotaskClient unit coverage — items[0] behavior (min 30 lines) | ✓ VERIFIED | 73 lines, 3 tests, all passing |
| `lib/services/b2/client.ts` | EML_OBJECT_KEY_REGEX + parameterized key validation | ✓ VERIFIED | Present, OBJECT_KEY_REGEX untouched (git-diff-style visual check against current content confirms LogLift regex line unchanged), all 3 functions parameterized |
| `migrations/099_indicators_metadata.sql` | ALTER TABLE indicators ADD COLUMN metadata JSONB | ✓ VERIFIED | Present, applied live to dev DB (confirmed via information_schema query) |
| `lib/services/phishing-eml-service.ts` | parseAndStoreMessage orchestration (min 90 lines) | ✓ VERIFIED | 224 lines, full list→select→fetch→size-guard→B2→parse→persist flow |
| `lib/services/phishing-eml-service.test.ts` | orchestration coverage with mocked autotask/b2/postgres incl. no-network + B2-gated + no-eml no-op (min 70 lines) | ✓ VERIFIED | 195 lines, 6 tests, all passing |

### Key Link Verification

| From | To | Via | Status | Details |
|------|-----|-----|--------|---------|
| `eml-parser.ts` | `mailparser` | `simpleParser` import w/ `checksumAlgo: 'sha256'` | ✓ WIRED | `eml-parser.ts:21,236` |
| `eml-parser.test.ts` | `global.fetch` | `vi.spyOn` zero-call assertion | ✓ WIRED | `eml-parser.test.ts:188-197` |
| `autotask-client.ts getAttachmentContent` | Autotask `Tickets/{id}/Attachments/{attachmentId}` | `makeApiCall` GET, `items?.[0]` | ✓ WIRED | `autotask-client.ts:443-456`; test confirms both shapes |
| `b2/client.ts presignUpload` | `EML_OBJECT_KEY_REGEX` | optional `keyRegex` param | ✓ WIRED | `b2/client.ts:165-173` |
| `phishing-eml-service.ts` | `eml-parser.ts` | `parseEml`+`selectOriginalMessage` import | ✓ WIRED | `phishing-eml-service.ts:27`; both invoked in the real flow (not stubbed in production code) |
| `phishing-eml-service.ts` | `AutotaskClient.getAttachmentContent` | `getAutotaskClient().getAttachmentContent(...)` | ✓ WIRED | `phishing-eml-service.ts:65-69` |
| `phishing-eml-service.ts` | `messages`/`indicators` tables | `postgresClient.query` INSERT ... RETURNING id::text | ✓ WIRED | `phishing-eml-service.ts:142-157,166-211`; column lists match migration 097+099 schema exactly |
| `phishing-eml-service.ts` | B2 (`presignUpload`+`EML_OBJECT_KEY_REGEX`) | `isB2Configured` gate then self-PUT | ✓ WIRED | `phishing-eml-service.ts:97-122`; gated, graceful-degrades on failure |

### Data-Flow Trace (Level 4)

Not applicable in the traditional UI-rendering sense — this phase is a pure backend service/library. Instead, traced data flow through the orchestration pipeline directly:

| Stage | Input | Output | Verified Real (not hardcoded) |
|-------|-------|--------|-------------------------------|
| `getAttachments` → `selectOriginalMessage` | live Autotask attachment list | selected `Attachment \| null` | ✓ Algorithm is a real filter/find chain over the input array, not a static return |
| `getAttachmentContent` → `Buffer.from(data,'base64')` | live Autotask base64 payload | decoded raw bytes | ✓ Real base64 decode, test proves items[0]-vs-item distinction |
| `parseEml` → `messages` INSERT | real `simpleParser` output | JSONB headers/urls/attachments payload | ✓ `headersPayload` built field-by-field from `normalized.*`, not a static object; test asserts `authResults` reaches the persisted param |
| `normalized.attachments/urls/from` → `indicators` INSERT loops | parsed message | per-indicator rows w/ metadata | ✓ Loops iterate real arrays from the parse result; test asserts attachment_hash metadata shape from an actual parsed fixture |

No hollow/static-return patterns found in the traced path.

### Behavioral Spot-Checks

| Behavior | Command | Result | Status |
|----------|---------|--------|--------|
| Full phase-16 vitest suite | `npx vitest run lib/services/eml-parser.test.ts lib/services/autotask-client.test.ts lib/services/b2/client.test.ts lib/services/phishing-eml-service.test.ts` | 4 files, 49/49 tests passed | ✓ PASS |
| Repo-wide type check | `npx tsc --noEmit --pretty` | exit 0, no output | ✓ PASS |
| Full test suite (regression check) | `npm test` | 25/26 files pass, 282/284 tests pass; only pre-existing unrelated `itglue-search.test.ts` failures (confirmed via `git log` predating Phase 16 commits) | ✓ PASS (no phase-16 regressions) |
| Migration 099 applied to dev DB | `docker exec pulse-postgres psql ... information_schema.columns` | returns `jsonb` | ✓ PASS |
| `rmm/executor.ts` presignUpload call site unaffected | `grep presignUpload lib/services/rmm/executor.ts` + tsc clean | 2-arg call, `presignUpload(objectKey, 1800)`, compiles | ✓ PASS |

### Probe Execution

No `scripts/*/tests/probe-*.sh` probes declared or discovered for this phase; not a migration/tooling phase in that sense. SKIPPED (no probe files apply — verification instead relied on the project's own vitest suite, run directly by the verifier, not narrated by SUMMARY.md).

### Requirements Coverage

| Requirement | Source Plan | Description | Status | Evidence |
|-------------|-------------|-------------|--------|----------|
| EVID-02 | 16-01 | Prefer `rfc.eml` over `OriginatingEmail.eml`, matching by content-type not filename alone | ✓ SATISFIED | `selectOriginalMessage` three-tier algorithm + full test coverage |
| EVID-03 | 16-01, 16-02, 16-03 | Parse RFC822/MIME into normalized headers/auth-verdicts/URLs/attachment metadata, persisted | ✓ SATISFIED | `parseEml` + `messages`/`indicators` persistence, both tested |
| EVID-04 | 16-01, 16-02, 16-03 | Sanitized/truncated body preview stored alongside raw evidence; never fetch/execute message content | ✓ SATISFIED | `buildBodyPreview` + B2 raw storage (D-05) + no-network spy at both parser and service level |

Note: `.planning/REQUIREMENTS.md` still shows EVID-02/03/04 as unchecked/"Pending" — this appears to be a tracking-doc staleness issue (the doc is not updated by the execute-phase workflow), not evidence of non-completion. All three requirements are satisfied by the code and tests as verified above.

### Anti-Patterns Found

None. Scanned all 10 phase-modified/created files for `TBD|FIXME|XXX|TODO|HACK|PLACEHOLDER`, "not yet implemented", "coming soon" — zero matches.

### Deviations Reviewed (from SUMMARY.md, checked against CLAUDE.md conventions)

1. **16-01: hand-rolled HTML stripper instead of `html-to-text`.** Acceptable — avoids depending on an unpinned transitive dependency; consistent with "don't introduce dependencies not required" project posture; fully test-covered.
2. **16-01: test title renamed for `-t` filter discoverability.** Cosmetic, no behavior change.
3. **16-02: migration 099 applied via direct `docker exec` instead of `scripts/apply-migrations.sh`.** Consistent with CLAUDE.md's own documented caveat ("check first; behavior varies") — the script's hardcoded `MIGRATIONS_DIR` didn't see the worktree's file; the same real credentials (`pulse_user`/`pulse_autotask`) were used, not fallback defaults. Verified live on the dev DB.
4. **16-03: self-PUT to B2 (first instance of Pulse's own server code PUTting to B2, vs. handing a presigned URL to an external collector).** Explicitly anticipated and researched in 16-RESEARCH.md as a deliberate new pattern, not an ad hoc deviation; gated behind `isB2Configured()` with graceful degrade on failure.
5. **16-03: Task 1/Task 2 executed as separate implementation-then-test-suite commits rather than interleaved RED/GREEN within one task.** Matches how the plan's own task structure was written (Task 1's verify was tsc-only; Task 2's verify included vitest). No coverage gap — the full suite passes and covers real behavior, not a rubber-stamp.

None of these deviations reduce scope or introduce risk beyond what's already accepted in the phase's own threat model.

### Human Verification Required

None. This phase is a pure backend library + orchestration service with no UI, no new API route, and no external-service live-credential dependency (Autotask/B2/Postgres are all mocked in tests; the live on-demand trigger route is explicitly deferred to Phase 18). All success criteria are mechanically verifiable via source review + live test execution, both performed above.

### Gaps Summary

No gaps found. All 3 requirements (EVID-02, EVID-03, EVID-04) are implemented, wired end-to-end from the pure parser (16-01) through supporting infrastructure (16-02) to the orchestration service (16-03), and covered by tests that exercise real parsing logic (not mocked-to-pass tautologies) with mocks confined to true I/O boundaries (Postgres, Autotask HTTP, B2 HTTP). Live verification (not just SUMMARY.md narrative) confirms: 49/49 phase-specific tests pass, `tsc --noEmit` is clean, the dev-DB migration is actually applied, and the pre-existing unrelated `itglue-search.test.ts` failures are correctly out of scope (confirmed via git history predating this phase).

---
_Verified: 2026-07-15T14:44:12Z_
_Verifier: Claude (gsd-verifier)_