wulf-pulse/.planning/phases/16-eml-mime-evidence-parser/16-VERIFICATION.md

phase	verified	status	score	overrides_applied
16-eml-mime-evidence-parser	2026-07-15T14:44:12Z	passed	8/8 must-haves verified	0

Phase 16: EML/MIME Evidence Parser Verification Report

Phase Goal: Given a ticket's attachments, Pulse selects the correct original reported message and parses its RFC822/MIME structure into normalized, actionable evidence — without ever executing or fetching anything from the message. Verified: 2026-07-15T14:44:12Z Status: passed Re-verification: No — initial verification

Goal Achievement

Observable Truths (ROADMAP Success Criteria + PLAN must_haves, merged)

#	Truth	Status	Evidence
1	Given synthetic fixtures with both rfc.eml and OriginatingEmail.eml present, selection picks rfc.eml matching by message/rfc822 content-type, not filename alone	✓ VERIFIED	lib/services/eml-parser.ts:50-65 selectOriginalMessage filters isMessageRfc822 first, then matches exact rfc.eml; test eml-parser.test.ts:28-37 asserts tier 1 + case-insensitive tier 1; also covers KnowBe4 tier 2 (:39-43) and OriginatingEmail-only tier 3 (:45-49) and ambiguous/null cases (:51-69). All pass live (npx vitest run — 49/49 across the 4 phase test files).
2	Parsing a synthetic .eml fixture produces normalized headers (From/displayName/senderEmail/senderDomain/Reply-To/Return-Path/To/Cc/Subject/Date/Message-ID/Received chain, SPF/DKIM/DMARC), URLs, and attachment metadata (name/content-type/size/hash)	✓ VERIFIED	parseEml (eml-parser.ts:227-282) builds exactly this shape from mailparser's ParsedMail; test eml-parser.test.ts:137-164 asserts every field against RICH_MULTIPART_EML, including authResults structured verdicts (not raw text), sha256 checksum format, and related flag. Runs against real simpleParser (no mocking of the parser itself) — not a tautology.
3	The parser never executes or fetches any URL found in a message — verified by tests asserting no outbound network calls happen during parsing	✓ VERIFIED	eml-parser.test.ts:188-197 spies on global.fetch across 5 fixture parses and asserts zero calls; extractUrls/buildBodyPreview are pure string operations (eml-parser.ts:140-184) with no fetch/render path. Orchestration-level (service) invariant separately verified below (Truth 6).
4	Parsed output includes a sanitized/truncated body preview stored alongside raw evidence, distinct from the full raw body	✓ VERIFIED	buildBodyPreview (eml-parser.ts:176-184) truncates to MAX_BODY_PREVIEW_LENGTH=500 and strips HTML via stripHtmlToText; test eml-parser.test.ts:128-133,178-181 confirms truncation and distinctness on a 2000-char fixture. Raw bytes are never persisted to Postgres — only to B2 (raw_ref is an object key, phishing-eml-service.ts:97-106,142-157), consistent with D-05.
5	npx vitest run for the new parser test file passes using synthetic fixtures only (no real customer email)	✓ VERIFIED	Ran live: eml-parser.test.ts 26/26 pass. A dedicated test (eml-parser.test.ts:205-218) asserts no fixture contains wulfconsulting.com and all use RFC 2606 reserved .test/.com-fake (evil-example.test) domains. deferred-items.md documents the 2 pre-existing unrelated itglue-search.test.ts failures (confirmed via git log — last touched at commit a0a6e7f/8f8b5ab, predating all Phase 16 commits).
6	(16-02/16-03 supporting truth) AutotaskClient.getAttachmentContent reads response.items?.[0], not .item; b2 EML_OBJECT_KEY_REGEX enforces path-traversal-safe .eml keys in parallel with the untouched LogLift regex; migration 099 adds indicators.metadata JSONB	✓ VERIFIED	autotask-client.ts:443-456 reads response.items?.[0] ?? null; test asserts an {item:...}-shaped response yields null (guards the exact regression risk called out in the plan). b2/client.ts:41-42 adds EML_OBJECT_KEY_REGEX beside the unmodified OBJECT_KEY_REGEX (:31-32); presignDownload/presignUpload/downloadToBuffer (:155-218) take an optional keyRegex param defaulting to OBJECT_KEY_REGEX; rmm/executor.ts:337's existing 2-arg call site still compiles (confirmed via tsc --noEmit, exit 0). Migration 099 applied live: information_schema.columns on the dev DB (pulse-postgres) reports indicators.metadata as jsonb.
7	Orchestration service (16-03) wires list→select→fetch→(B2 gated)→parse→persist end to end, writing one messages row + indicators rows with D-07 metadata, and no-op (no throw) when no .eml attachment or B2 unconfigured	✓ VERIFIED	phishing-eml-service.ts:49-224 imports and calls getAutotaskClient().getAttachments, selectOriginalMessage, getAttachmentContent, isB2Configured/presignUpload/EML_OBJECT_KEY_REGEX, parseEml — exact function signatures match Plan 01/02's exports (verified by reading both source files side by side). messages INSERT (:142-157) and 3 indicators INSERT loops (:164-211) match migration 097/099's actual column list (report_id, message_id, headers, urls, attachments, body_preview, raw_ref / message_id, indicator_type, value, metadata). Test suite (phishing-eml-service.test.ts, 6/6 passing) exercises the happy path (real parseEml invoked on RICH_MULTIPART_EML, only DB/Autotask/B2 boundaries mocked — not tautological), no-eml no-op, B2-configured/unconfigured branches, no-network invariant, and indicator metadata shape.
8	No outbound network call is made to any URL found in the message, end to end (service level)	✓ VERIFIED	phishing-eml-service.test.ts:155-170 asserts global.fetch is called exactly once (the B2 presigned PUT) and never with the fixture's embedded body URL (http://evil-example.test/verify). Source review confirms the only fetch( call in phishing-eml-service.ts targets uploadUrl (a B2-presigned URL Pulse itself constructed), never a value derived from normalized.urls.

Score: 8/8 truths verified

Required Artifacts

Artifact	Expected	Status	Details
lib/services/eml-parser.ts	parseEml, selectOriginalMessage, parseAuthResults, extractUrls, buildBodyPreview, NormalizedMessage (min 120 lines)	✓ VERIFIED	282 lines; all 5 functions + types exported exactly per plan interface contract
lib/services/eml-parser.fixtures.ts	synthetic fixtures for all 3 selection tiers + parse fixtures	✓ VERIFIED	Exists, imported and exercised by both eml-parser.test.ts and phishing-eml-service.test.ts; RFC 2606-safe synthetic content only
lib/services/eml-parser.test.ts	EVID-02/03/04 vitest coverage incl. no-network spy + 3-tier selection (min 100 lines)	✓ VERIFIED	219 lines, 26 tests, all passing live
lib/services/autotask-client.ts	getAttachmentContent(entityName, entityId, attachmentId)	✓ VERIFIED	Present at line 443, reads items?.[0] ?? null
lib/services/autotask-client.test.ts	first AutotaskClient unit coverage — items[0] behavior (min 30 lines)	✓ VERIFIED	73 lines, 3 tests, all passing
lib/services/b2/client.ts	EML_OBJECT_KEY_REGEX + parameterized key validation	✓ VERIFIED	Present, OBJECT_KEY_REGEX untouched (git-diff-style visual check against current content confirms LogLift regex line unchanged), all 3 functions parameterized
migrations/099_indicators_metadata.sql	ALTER TABLE indicators ADD COLUMN metadata JSONB	✓ VERIFIED	Present, applied live to dev DB (confirmed via information_schema query)
lib/services/phishing-eml-service.ts	parseAndStoreMessage orchestration (min 90 lines)	✓ VERIFIED	224 lines, full list→select→fetch→size-guard→B2→parse→persist flow
lib/services/phishing-eml-service.test.ts	orchestration coverage with mocked autotask/b2/postgres incl. no-network + B2-gated + no-eml no-op (min 70 lines)	✓ VERIFIED	195 lines, 6 tests, all passing

Key Link Verification

From	To	Via	Status	Details
eml-parser.ts	mailparser	simpleParser import w/ checksumAlgo: 'sha256'	✓ WIRED	eml-parser.ts:21,236
eml-parser.test.ts	global.fetch	vi.spyOn zero-call assertion	✓ WIRED	eml-parser.test.ts:188-197
autotask-client.ts getAttachmentContent	Autotask Tickets/{id}/Attachments/{attachmentId}	makeApiCall GET, items?.[0]	✓ WIRED	autotask-client.ts:443-456; test confirms both shapes
b2/client.ts presignUpload	EML_OBJECT_KEY_REGEX	optional keyRegex param	✓ WIRED	b2/client.ts:165-173
phishing-eml-service.ts	eml-parser.ts	parseEml+selectOriginalMessage import	✓ WIRED	phishing-eml-service.ts:27; both invoked in the real flow (not stubbed in production code)
phishing-eml-service.ts	AutotaskClient.getAttachmentContent	getAutotaskClient().getAttachmentContent(...)	✓ WIRED	phishing-eml-service.ts:65-69
phishing-eml-service.ts	messages/indicators tables	postgresClient.query INSERT ... RETURNING id::text	✓ WIRED	phishing-eml-service.ts:142-157,166-211; column lists match migration 097+099 schema exactly
phishing-eml-service.ts	B2 (presignUpload+EML_OBJECT_KEY_REGEX)	isB2Configured gate then self-PUT	✓ WIRED	phishing-eml-service.ts:97-122; gated, graceful-degrades on failure

Data-Flow Trace (Level 4)

Not applicable in the traditional UI-rendering sense — this phase is a pure backend service/library. Instead, traced data flow through the orchestration pipeline directly:

Stage	Input	Output	Verified Real (not hardcoded)
getAttachments → selectOriginalMessage	live Autotask attachment list	selected Attachment | null	✓ Algorithm is a real filter/find chain over the input array, not a static return
getAttachmentContent → Buffer.from(data,'base64')	live Autotask base64 payload	decoded raw bytes	✓ Real base64 decode, test proves items[0]-vs-item distinction
parseEml → messages INSERT	real simpleParser output	JSONB headers/urls/attachments payload	✓ headersPayload built field-by-field from normalized.*, not a static object; test asserts authResults reaches the persisted param
normalized.attachments/urls/from → indicators INSERT loops	parsed message	per-indicator rows w/ metadata	✓ Loops iterate real arrays from the parse result; test asserts attachment_hash metadata shape from an actual parsed fixture

No hollow/static-return patterns found in the traced path.

Behavioral Spot-Checks

Behavior	Command	Result	Status
Full phase-16 vitest suite	npx vitest run lib/services/eml-parser.test.ts lib/services/autotask-client.test.ts lib/services/b2/client.test.ts lib/services/phishing-eml-service.test.ts	4 files, 49/49 tests passed	✓ PASS
Repo-wide type check	npx tsc --noEmit --pretty	exit 0, no output	✓ PASS
Full test suite (regression check)	npm test	25/26 files pass, 282/284 tests pass; only pre-existing unrelated itglue-search.test.ts failures (confirmed via git log predating Phase 16 commits)	✓ PASS (no phase-16 regressions)
Migration 099 applied to dev DB	docker exec pulse-postgres psql ... information_schema.columns	returns jsonb	✓ PASS
rmm/executor.ts presignUpload call site unaffected	grep presignUpload lib/services/rmm/executor.ts + tsc clean	2-arg call, presignUpload(objectKey, 1800), compiles	✓ PASS

Probe Execution

No scripts/*/tests/probe-*.sh probes declared or discovered for this phase; not a migration/tooling phase in that sense. SKIPPED (no probe files apply — verification instead relied on the project's own vitest suite, run directly by the verifier, not narrated by SUMMARY.md).

Requirements Coverage

Requirement	Source Plan	Description	Status	Evidence
EVID-02	16-01	Prefer rfc.eml over OriginatingEmail.eml, matching by content-type not filename alone	✓ SATISFIED	selectOriginalMessage three-tier algorithm + full test coverage
EVID-03	16-01, 16-02, 16-03	Parse RFC822/MIME into normalized headers/auth-verdicts/URLs/attachment metadata, persisted	✓ SATISFIED	parseEml + messages/indicators persistence, both tested
EVID-04	16-01, 16-02, 16-03	Sanitized/truncated body preview stored alongside raw evidence; never fetch/execute message content	✓ SATISFIED	buildBodyPreview + B2 raw storage (D-05) + no-network spy at both parser and service level

Note: .planning/REQUIREMENTS.md still shows EVID-02/03/04 as unchecked/"Pending" — this appears to be a tracking-doc staleness issue (the doc is not updated by the execute-phase workflow), not evidence of non-completion. All three requirements are satisfied by the code and tests as verified above.

Anti-Patterns Found

None. Scanned all 10 phase-modified/created files for TBD|FIXME|XXX|TODO|HACK|PLACEHOLDER, "not yet implemented", "coming soon" — zero matches.

Deviations Reviewed (from SUMMARY.md, checked against CLAUDE.md conventions)

1. 16-01: hand-rolled HTML stripper instead of html-to-text. Acceptable — avoids depending on an unpinned transitive dependency; consistent with "don't introduce dependencies not required" project posture; fully test-covered.
2. 16-01: test title renamed for -t filter discoverability. Cosmetic, no behavior change.
3. 16-02: migration 099 applied via direct docker exec instead of scripts/apply-migrations.sh. Consistent with CLAUDE.md's own documented caveat ("check first; behavior varies") — the script's hardcoded MIGRATIONS_DIR didn't see the worktree's file; the same real credentials (pulse_user/pulse_autotask) were used, not fallback defaults. Verified live on the dev DB.
4. 16-03: self-PUT to B2 (first instance of Pulse's own server code PUTting to B2, vs. handing a presigned URL to an external collector). Explicitly anticipated and researched in 16-RESEARCH.md as a deliberate new pattern, not an ad hoc deviation; gated behind isB2Configured() with graceful degrade on failure.
5. 16-03: Task 1/Task 2 executed as separate implementation-then-test-suite commits rather than interleaved RED/GREEN within one task. Matches how the plan's own task structure was written (Task 1's verify was tsc-only; Task 2's verify included vitest). No coverage gap — the full suite passes and covers real behavior, not a rubber-stamp.

None of these deviations reduce scope or introduce risk beyond what's already accepted in the phase's own threat model.

Human Verification Required

None. This phase is a pure backend library + orchestration service with no UI, no new API route, and no external-service live-credential dependency (Autotask/B2/Postgres are all mocked in tests; the live on-demand trigger route is explicitly deferred to Phase 18). All success criteria are mechanically verifiable via source review + live test execution, both performed above.

Gaps Summary

No gaps found. All 3 requirements (EVID-02, EVID-03, EVID-04) are implemented, wired end-to-end from the pure parser (16-01) through supporting infrastructure (16-02) to the orchestration service (16-03), and covered by tests that exercise real parsing logic (not mocked-to-pass tautologies) with mocks confined to true I/O boundaries (Postgres, Autotask HTTP, B2 HTTP). Live verification (not just SUMMARY.md narrative) confirms: 49/49 phase-specific tests pass, tsc --noEmit is clean, the dev-DB migration is actually applied, and the pre-existing unrelated itglue-search.test.ts failures are correctly out of scope (confirmed via git history predating this phase).

---

Verified: 2026-07-15T14:44:12Z Verifier: Claude (gsd-verifier)