docs(16): research phase domain
This commit is contained in:
parent
d3e24d968b
commit
dc273c002d
1 changed files with 528 additions and 0 deletions
528
.planning/phases/16-eml-mime-evidence-parser/16-RESEARCH.md
Normal file
528
.planning/phases/16-eml-mime-evidence-parser/16-RESEARCH.md
Normal file
|
|
@ -0,0 +1,528 @@
|
|||
# Phase 16: EML/MIME Evidence Parser - Research
|
||||
|
||||
**Researched:** 2026-07-15
|
||||
**Domain:** RFC822/MIME email parsing (Node.js/TypeScript), Autotask REST attachment retrieval, B2 evidence storage
|
||||
**Confidence:** HIGH (both flagged unknowns resolved with direct empirical verification against the real Autotask API and a real `mailparser` install/parse/checksum run — not just docs)
|
||||
|
||||
## Summary
|
||||
|
||||
Both genuine unknowns flagged in CONTEXT.md are now resolved with primary evidence, not just library docs.
|
||||
|
||||
**MIME parsing:** `mailparser` (nodemailer org, MIT, v3.9.14, actively maintained — latest release 10 days before this research) is the correct default. It parses a raw `.eml` buffer directly into headers (From/To/Cc/Reply-To/Return-Path/Date/Message-ID as structured `AddressObject`s), a `headerLines`/`headers` Map exposing the raw `Authentication-Results` and `Received` header text verbatim, and an `attachments[]` array with `filename`/`contentType`/`size`/`checksum` — the `checksumAlgo: 'sha256'` option computes the exact hash EVID-03 needs with zero extra code. It performs **no network I/O of any kind** during parsing (verified empirically: a `global.fetch` spy recorded zero calls parsing a synthetic fixture with `Content-Type: multipart/mixed`, embedded HTML links, and a base64 attachment) — none of its transitive dependencies (`iconv-lite`, `libmime`, `html-to-text`, `linkify-it`, `@zone-eu/mailsplit`, `nodemailer`, `punycode.js`, `tlds`, `encoding-japanese`, `he`) perform I/O either. **mailparser does not parse `Authentication-Results` into structured SPF/DKIM/DMARC verdicts — that must be hand-rolled** (see below); it only hands back the raw header text.
|
||||
|
||||
**Authentication-Results structured parsing (D-06):** Do **not** use `mailauth` for this. `mailauth`'s public export surface (`authenticate`, `dkimVerify`, `spf`, `dmarc`, `arc`, `bimi`, `sealMessage`, …) is built entirely around **performing live verification** — DNS lookups for SPF/DKIM/DMARC records, and for BIMI, actual HTTP fetches of logo/VMC certificate URLs. That directly violates SC#3 ("never fetch anything from the message"). Its header-only parsing helpers (`parseReceivedHeaders`, `parseDkimArcHeaders`) exist only as internal files (`lib/parse-received.js`, `lib/parse-dkim-headers.js`) and are **not** exported from the package's `main` entrypoint — not usable without reaching into internals. `authentication-results-parser` does not exist on the npm registry (404). The pragmatic, safe choice — and what D-06 actually asks for (parse the verdicts the receiving mail server *already stamped*, don't re-verify) — is a small hand-rolled regex/tokenizer against the RFC 8601 `Authentication-Results` grammar (`authserv-id; method1=result1 [reason]; method2=result2 …`), reading the raw header string mailparser already extracted. `Received-SPF` (older, SPF-only header, seen as a fallback source in real samples) needs the same treatment. This is deliberately hand-rolled per RFC 8601's small, stable grammar — not "hand-rolling a solved problem" (no maintained, safe, non-DNS-performing npm package solves just this parsing step).
|
||||
|
||||
**Autotask attachment content fetch:** Empirically confirmed against the real Autotask API (3 live phishing tickets, read-only GET calls only, no PII persisted). `GET {apiUrl}/Tickets/{id}/Attachments` (the existing `getAttachments()`, unchanged) **always returns `data: null`** — list calls never populate content, regardless of file size. A second, per-attachment-ID call — `GET {apiUrl}/Tickets/{id}/Attachments/{attachmentId}` — **does** return the full base64 content in `data`, decoding to a byte length that exactly matches the list call's `fileSize`. This second call is a genuinely new client method that doesn't exist yet, and its response shape is a documented gotcha: it returns `{ items: [ {...} ], pageDetails }` — an **array**, not the `{ item: {...} }` shape `getEntityById()` uses for every other single-entity-by-ID GET in this codebase. A naive implementer following the existing `getEntityById` pattern will read `response.item` and silently get `undefined`.
|
||||
|
||||
**`.eml` selection (EVID-02):** Sampled 15 real production phishing tickets (both current, from today, and historical, from 2024) and found **three** distinct attachment shapes, not the two implied by the requirement text:
|
||||
1. `rfc.eml` + `OriginatingEmail.eml` (Microsoft "Report Message" add-in flow)
|
||||
2. `phish_alert_sp2_2.0.0.0.eml` + `OriginatingEmail.eml` (KnowBe4 PhishER flow — and DETECT-01 explicitly includes KnowBe4 patterns, so this is squarely in scope)
|
||||
3. `OriginatingEmail.eml` alone (older tickets, no distinct "reported message" attachment at all)
|
||||
|
||||
In every single multi-attachment sample, `OriginatingEmail.eml` was present and was always the larger file (it's the wrapper containing more forwarding context); the smaller, non-`OriginatingEmail.eml`-named `message/rfc822` attachment was the actual originally-reported message. **All attachments in every sample shared `contentType: message/rfc822`** — meaning content-type alone never disambiguates; filename is the only usable signal, and it isn't always literally `"rfc.eml"`. See Common Pitfalls for the recommended selection algorithm.
|
||||
|
||||
**B2 storage reuse (D-05):** `lib/services/b2/client.ts`'s `OBJECT_KEY_REGEX` is hard-coded to the LogLift shape (`{id}/{host}/eventlogs_{timestamp}.json.gz`) and both `presignDownload`/`presignUpload` reject anything else with `B2InvalidObjectKeyError`. This is not a drop-in reuse — it requires a small code change to `client.ts`. The project's own skill doc (`.claude/skills/pulse-overshell-b2-evidence/SKILL.md`) explicitly documents the intended pattern: *"If you need a new payload type, add a new key regex + a new transport rather than loosening the existing one."* Also net-new: every existing `presignUpload` call site (`lib/services/rmm/executor.ts`) hands the URL to an *external* collector to PUT — Phase 16 is the first case where Pulse's own server code would PUT bytes to B2 itself.
|
||||
|
||||
**Primary recommendation:** `mailparser` for RFC822/MIME parsing + attachment hashing, a small hand-rolled RFC 8601 `Authentication-Results` parser for D-06, a new `getAttachmentContent(entityName, entityId, attachmentId)` method on `AutotaskClient` reading `response.items?.[0]`, a three-tier `.eml` selection algorithm (exact `rfc.eml` → non-`OriginatingEmail.eml` `message/rfc822` attachment → `OriginatingEmail.eml` fallback), and a new `EML_OBJECT_KEY_REGEX` + parameterized validation in `b2/client.ts` rather than loosening `OBJECT_KEY_REGEX`.
|
||||
|
||||
## Architectural Responsibility Map
|
||||
|
||||
| Capability | Primary Tier | Secondary Tier | Rationale |
|
||||
|------------|-------------|----------------|-----------|
|
||||
| Fetch attachment metadata + full base64 content from Autotask | API/Backend (service layer) | — | `AutotaskClient` is the sole authorized Autotask access point; no browser/SSR involvement |
|
||||
| Select correct `.eml` from attachment list | API/Backend (service layer) | — | Pure decision logic over metadata already in hand; no I/O |
|
||||
| Parse RFC822/MIME bytes into normalized evidence | API/Backend (service layer) | — | CPU-bound parsing of untrusted bytes; must never reach the browser or an LLM unfiltered (CLASSIFY-06 is downstream in Phase 19) |
|
||||
| Parse Authentication-Results into structured verdicts | API/Backend (service layer) | — | Pure string parsing of a header mailparser already extracted; no DNS/network |
|
||||
| Store raw `.eml` bytes | Database/Storage (B2, not Postgres) | — | D-05 — matches LogLift precedent; raw untrusted bytes never touch Postgres |
|
||||
| Persist normalized headers/URLs/attachments/preview | Database/Storage (Postgres `messages`/`indicators`) | — | Structured, queryable evidence for Phases 18-19 |
|
||||
|
||||
## User Constraints (from CONTEXT.md)
|
||||
|
||||
<user_constraints>
|
||||
|
||||
### Locked Decisions
|
||||
|
||||
- **D-05:** Raw `.eml` bytes are stored in Backblaze B2, reusing the existing LogLift evidence-storage pattern (`lib/services/b2/client.ts`). `messages.raw_ref` stores the B2 object key, not raw bytes or a bare hash.
|
||||
- **D-06:** SPF/DKIM/DMARC are parsed into structured verdicts (pass/fail/none/etc, not raw header text) from the `Authentication-Results` header (and `Received-SPF` as fallback where present). This is deliberately more than "capture the raw header" — Phase 19's classifier needs structured evidence, not raw text it would have to re-parse.
|
||||
- **D-07:** Add a `metadata` JSONB column to the `indicators` table stub (migration 097) via a small Phase 16 migration (`ALTER TABLE indicators ADD COLUMN metadata JSONB`). Lets an attachment-hash indicator carry filename/content-type/size, a URL indicator carry which message part it came from, etc.
|
||||
|
||||
### Claude's Discretion
|
||||
|
||||
- **MIME parsing library or approach.** Resolved by this research: `mailparser`.
|
||||
- **How to actually fetch full `.eml` attachment content from Autotask.** Resolved by this research: per-attachment-ID GET, new client method, `response.items?.[0]` (not `.item`).
|
||||
- **URL extraction scope** (dedup strategy, normalization, which MIME parts to scan) — implementation detail, not user-relevant preference. This research recommends `linkify-it` (already an mailparser transitive dependency's peer in the same org's ecosystem, MIT, ~23M downloads/week) scanning both `mail.text` and `mail.html`, but a plain regex is also viable — genuinely Claude's discretion at plan time.
|
||||
- Exact migration file number for the `indicators` ALTER — confirmed by this research: **099** (098 is the current highest; `migrations/097_phishing_triage_schema.sql` and `migrations/098_phishing_sweep_schedule.sql` already exist).
|
||||
|
||||
### Deferred Ideas (OUT OF SCOPE)
|
||||
|
||||
None — discussion stayed within phase scope. MIME library choice and Autotask attachment-download mechanics were explicitly routed to research rather than deferred to a future phase.
|
||||
|
||||
</user_constraints>
|
||||
|
||||
## Phase Requirements
|
||||
|
||||
<phase_requirements>
|
||||
|
||||
| ID | Description | Research Support |
|
||||
|----|-------------|------------------|
|
||||
| EVID-02 | Prefer `rfc.eml` over `OriginatingEmail.eml`, case-insensitive name + `message/rfc822` content-type match | Empirically sampled 15 real tickets; found 3 real attachment-naming shapes (not the 2 implied by the requirement text) — see Common Pitfalls for the selection algorithm that actually covers all three |
|
||||
| EVID-03 | Parse RFC822/MIME into normalized headers, auth results, URLs, attachment metadata | `mailparser` for headers/attachments/hashing (verified via Context7 docs + live install/parse test); hand-rolled RFC 8601 parser for Authentication-Results (mailauth rejected — see Summary); new `AutotaskClient.getAttachmentContent()` method to actually get the bytes (verified via live Autotask API calls) |
|
||||
| EVID-04 | Sanitized/truncated body preview; never fetch/execute anything from the message | `mailparser` performs zero I/O during parsing (verified via `fetch` spy on a synthetic fixture); body preview should derive from `mail.text` (already-safe plain text) truncated, falling back to `html-to-text`-converted `mail.html` (already a mailparser transitive dependency) rather than raw HTML |
|
||||
|
||||
</phase_requirements>
|
||||
|
||||
## Project Constraints (from CLAUDE.md)
|
||||
|
||||
- No ORM, no server actions, no new state library — none apply here; this phase is a pure service-layer addition plus one new API-adjacent client method.
|
||||
- New npm dependencies are explicitly fine per CLAUDE.md ("no ORM/server-actions/state-lib restriction applies to a MIME parser" — CONTEXT.md canonical_refs) — `mailparser` and (optionally) `linkify-it` are appropriate additions.
|
||||
- All DB columns snake_case; API responses camelCase, transformed manually — applies to any `messages`/`indicators` row-shaping code this phase writes.
|
||||
- New migration: next number after 098 is `099`, `IF NOT EXISTS` guard, no destructive ops.
|
||||
- `console.error()` for caught exceptions with context; no stray `console.log()` (existing `getAttachments()`/`gatherTicketEvidence()` in `phishing-detector.ts` already follow this — match it).
|
||||
- Postgres init only applies `migrations/*.sql` on first volume boot; the running dev/prod Postgres containers need the new migration applied manually (`docker exec pulse-postgres psql ...`) once written, per CLAUDE.md's migration caveat.
|
||||
- `npx tsc --noEmit --pretty` is the safety net for any code path not covered by `npm test` — this phase's parser code should have vitest coverage (`lib/**/*.test.ts`) given SC#5 requires it explicitly, but the Autotask client changes will only be type-checked unless test doubles are added.
|
||||
|
||||
## Standard Stack
|
||||
|
||||
### Core
|
||||
|
||||
| Library | Version | Purpose | Why Standard |
|
||||
|---------|---------|---------|--------------|
|
||||
| `mailparser` | 3.9.14 [VERIFIED: npm registry — 15-yr-old package (created 2011), 2.83M downloads/week, latest release 2026-07-05, MIT, maintained by the nodemailer org] | Parse raw RFC822/MIME `.eml` bytes into headers, address objects, text/html bodies, and attachments with checksums | The de facto standard Node MIME parser; part of the same maintained ecosystem as `nodemailer` itself; verified via live install + parse of a synthetic fixture (see Code Examples) |
|
||||
|
||||
### Supporting
|
||||
|
||||
| Library | Version | Purpose | When to Use |
|
||||
|---------|---------|---------|-------------|
|
||||
| `linkify-it` | 6.0.0 [VERIFIED: npm registry — MIT, ~23M downloads/week (transitively pulled in by `markdown-it`), no postinstall scripts] | Extract URLs (and `mailto:` addresses) from plain text and HTML body content | Recommended over a hand-rolled regex for URL extraction from `mail.text`/`mail.html` — handles schemed URLs, `www.`-prefixed hosts (with `.tlds()` fuzzy matching enabled), and email addresses; verified via a live `.match()` call against a synthetic phishing-style body |
|
||||
|
||||
### Not Recommended (rejected during research)
|
||||
|
||||
| Library | Reason for Rejection |
|
||||
|---------|----------------------|
|
||||
| `mailauth` | Public API (`authenticate`, `spf`, `dkim`, `dmarc`, `arc`, `bimi`) performs live DNS lookups and, for BIMI, HTTP fetches of remote logo/VMC URLs — this is exactly the outbound network activity SC#3 forbids. Its header-only parsers (`parseReceivedHeaders`, `parseDkimArcHeaders`) are internal-only files, not exported from `package.json`'s `main` (`lib/mailauth.js` only exports the verification functions listed above) |
|
||||
| `authentication-results-parser` | Does not exist on the npm registry (`npm view` → 404) — appears to be a hallucinated/non-existent package name; do not use |
|
||||
|
||||
### Alternatives Considered
|
||||
|
||||
| Instead of | Could Use | Tradeoff |
|
||||
|------------|-----------|----------|
|
||||
| `mailparser` | Hand-rolled RFC822 parser | MIME multipart parsing (nested boundaries, encodings, charset conversion) is exactly the kind of "deceptively complex" problem this codebase's "Don't Hand-Roll" philosophy warns about — `mailparser` handles it correctly and is already the ecosystem standard |
|
||||
| `linkify-it` for URLs | Plain regex (`/https?:\/\/\S+/g`) | Simpler, zero new dependency, but misses `www.`-only links, mangles trailing punctuation, and has no IDN/punycode awareness — acceptable for MVP but `linkify-it` is a better default given phishing URLs frequently use non-http-schemed or oddly-punctuated forms |
|
||||
| Hand-rolled Authentication-Results parser | `mailauth`'s internal parser (reached via deep import) | Reaching into a package's non-exported internal files (`mailauth/lib/parse-dkim-headers.js`) is fragile across version bumps and still pulls in the full `mailauth` dependency tree (`undici`, `joi`, `@postalsys/vmc`) for no benefit; RFC 8601's grammar is small and stable enough to hand-roll safely |
|
||||
|
||||
**Installation:**
|
||||
```bash
|
||||
npm install mailparser linkify-it
|
||||
```
|
||||
|
||||
**Version verification:** Confirmed live via `npm view mailparser version` → `3.9.14` (published 2026-07-05) and `npm view linkify-it version` → `6.0.0`, both against the real npm registry at research time — not training-data recollection.
|
||||
|
||||
## Package Legitimacy Audit
|
||||
|
||||
`slopcheck` was installed and run successfully (in an isolated scratch directory, not the project — see note below).
|
||||
|
||||
| Package | Registry | Age | Downloads | Source Repo | slopcheck | Disposition |
|
||||
|---------|----------|-----|-----------|-------------|-----------|-------------|
|
||||
| `mailparser` | npm | ~15 yrs (created 2011-09-14) | 2.83M/wk | github.com/nodemailer/mailparser | [OK] | Approved |
|
||||
| `linkify-it` | npm | mature (markdown-it ecosystem) | 22.8M/wk | github.com/markdown-it/linkify-it | [OK] | Approved |
|
||||
|
||||
**Packages removed due to slopcheck [SLOP] verdict:** none
|
||||
**Packages flagged as suspicious [SUS]:** none
|
||||
|
||||
No postinstall scripts found on `mailparser`, `linkify-it`, or any of `mailparser`'s transitive dependencies (`encoding-japanese`, `he`, `html-to-text`, `iconv-lite`, `libmime`, `@zone-eu/mailsplit`, `nodemailer`, `punycode.js`, `tlds`) — checked via `npm view <pkg> scripts.postinstall` for each. All licenses are permissive (MIT, MIT-0, or dual MIT/EUPL-1.1+ for `@zone-eu/mailsplit`) — no GPL/copyleft exposure.
|
||||
|
||||
**Process note for the planner:** running `slopcheck install <pkg>` (or any bare `npm install <pkg>`) from a shell whose cwd has silently reset to the project root will install the package into the *real* `package.json`/`package-lock.json`. This happened once during this research session and was immediately reverted (`git checkout -- package.json package-lock.json` + `npm install` to resync `node_modules`) before any commit. When executing Phase 16's actual dependency-install task, run it as a deliberate, reviewed step — not inside a throwaway verification script — and confirm `git diff package.json` shows only the intended `mailparser`/`linkify-it` additions.
|
||||
|
||||
## Architecture Patterns
|
||||
|
||||
### System Architecture Diagram
|
||||
|
||||
```
|
||||
Autotask Ticket Attachments (list, metadata only)
|
||||
|
|
||||
v
|
||||
[1] listAttachments() ---------------------------------> AutotaskClient.getAttachments() (existing, unchanged)
|
||||
| filenames + contentType, data:null always
|
||||
v
|
||||
[2] selectOriginalMessage(attachments) (new, pure logic)
|
||||
| picks: exact "rfc.eml" -> else non-"OriginatingEmail.eml" message/rfc822 -> else "OriginatingEmail.eml"
|
||||
v
|
||||
[3] fetchAttachmentContent(entityName, entityId, attachmentId) (new) ---> AutotaskClient.getAttachmentContent()
|
||||
| per-ID GET, base64 `data` populated -- response shape is {items:[...]}, not {item:...}
|
||||
v
|
||||
Buffer of raw RFC822 bytes
|
||||
|
|
||||
+----------------------------------------+
|
||||
| |
|
||||
v v
|
||||
[4] uploadRawEml(buffer) (new) [5] parseEml(buffer) (new)
|
||||
presignUpload(EML_OBJECT_KEY_REGEX- mailparser.simpleParser(buffer, {checksumAlgo:'sha256'})
|
||||
validated key) -> PUT to B2 |
|
||||
| +--> headers (From/To/Cc/ReplyTo/ReturnPath/Date/MessageID)
|
||||
v +--> raw Authentication-Results / Received-SPF header text
|
||||
B2 object key +--> raw Received header lines (ordered)
|
||||
| +--> text / html bodies
|
||||
| +--> attachments[] (filename, contentType, size, checksum)
|
||||
| |
|
||||
| v
|
||||
| [6] parseAuthResults(headerText) (new, hand-rolled RFC 8601)
|
||||
| --> { spf: {result}, dkim: {result}, dmarc: {result} }
|
||||
| |
|
||||
| v
|
||||
| [7] extractUrls(text, html) (new, linkify-it)
|
||||
| --> string[] (deduped, normalized)
|
||||
| |
|
||||
| v
|
||||
| [8] buildBodyPreview(text, html) (new)
|
||||
| --> truncated plain-text preview, distinct from raw body
|
||||
| |
|
||||
v v
|
||||
messages.raw_ref = B2 object key messages.headers / urls / attachments / body_preview (JSONB / TEXT)
|
||||
|
|
||||
v
|
||||
indicators rows (attachment-hash / url / sender, + metadata JSONB per D-07)
|
||||
```
|
||||
|
||||
Every step from [2] onward is local CPU-bound logic or a call to a service the codebase already trusts (Autotask, B2). Nothing in [5]-[8] performs network I/O — this is the property the SC#3 test suite must assert.
|
||||
|
||||
### Recommended Project Structure
|
||||
|
||||
```
|
||||
lib/services/
|
||||
├── eml-parser.ts # NEW — parseEml(buffer) -> NormalizedMessage; wraps mailparser + hand-rolled auth-results parsing + URL extraction + body preview
|
||||
├── eml-parser.test.ts # NEW — synthetic-fixture tests (SC#1-5), including the "no network call" assertion
|
||||
├── autotask-client.ts # MODIFIED — add getAttachmentContent(entityName, entityId, attachmentId): Promise<Attachment | null>
|
||||
├── b2/
|
||||
│ └── client.ts # MODIFIED — add EML_OBJECT_KEY_REGEX; parameterize presignDownload/presignUpload/downloadToBuffer's key validation (default stays OBJECT_KEY_REGEX for LogLift call sites)
|
||||
└── phishing-eml-service.ts # NEW (suggested name, Claude's discretion) — orchestrates: list attachments -> select -> fetch content -> upload to B2 -> parse -> persist messages/indicators rows
|
||||
migrations/
|
||||
└── 099_indicators_metadata.sql # NEW — ALTER TABLE indicators ADD COLUMN metadata JSONB (D-07)
|
||||
```
|
||||
|
||||
### Pattern 1: Selecting the Attachment Content Field via Sub-Resource GET
|
||||
|
||||
**What:** Autotask's `Attachments/{id}` sub-resource returns a list-shaped envelope (`{items:[...], pageDetails}`) even when fetching a single ID — unlike top-level entity-by-ID GETs (`{item: {...}}`).
|
||||
**When to use:** Any new `AutotaskClient` method that fetches a single attachment's full content.
|
||||
**Example:**
|
||||
```typescript
|
||||
// Verified empirically against the real Autotask API (read-only GET,
|
||||
// 2026-07-15) — Tickets/{id}/Attachments/{attachmentId} returns:
|
||||
// { "items": [ { ...attachment fields..., "data": "<base64>" } ], "pageDetails": {...} }
|
||||
// NOT { "item": {...} } like every other single-entity-by-ID GET in this file.
|
||||
|
||||
async getAttachmentContent(
|
||||
entityName: string,
|
||||
entityId: number,
|
||||
attachmentId: number
|
||||
): Promise<Attachment | null> {
|
||||
const url = `${this.config.apiUrl}/${entityName}/${entityId}/Attachments/${attachmentId}`;
|
||||
const response = await this.makeApiCall<ApiResponse<Attachment>>(url, {
|
||||
method: 'GET',
|
||||
headers: this.getAuthHeaders(),
|
||||
});
|
||||
// NOTE: response.items (array), NOT response.item — confirmed live.
|
||||
return response.items?.[0] ?? null;
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 2: Parsing with mailparser (verified live)
|
||||
|
||||
**What:** `simpleParser` accepts a raw RFC822 Buffer directly (no IMAP/transport wrapping needed) and returns structured headers + attachments with checksums in one call.
|
||||
**When to use:** Core of the new `eml-parser.ts` module.
|
||||
**Example:**
|
||||
```typescript
|
||||
// Source: Context7 /nodemailer/mailparser docs + live verification in this
|
||||
// research session (synthetic fixture, checksumAlgo: 'sha256').
|
||||
import { simpleParser } from 'mailparser';
|
||||
|
||||
const mail = await simpleParser(rawEmlBuffer, { checksumAlgo: 'sha256' });
|
||||
|
||||
mail.from?.value[0]?.address; // sender email
|
||||
mail.from?.value[0]?.name; // sender display name
|
||||
mail.headers.get('return-path'); // AddressObject | string, raw header
|
||||
mail.headers.get('authentication-results'); // raw string — hand-parse this (see Pattern 3)
|
||||
mail.messageId; // "<...>"
|
||||
mail.date; // Date | undefined
|
||||
mail.text; // plain-text body (safe for preview)
|
||||
mail.html; // HTML body (do NOT render raw; see body-preview pattern)
|
||||
|
||||
for (const line of mail.headerLines) {
|
||||
if (line.key === 'received') { /* push line.line into an ordered array for the Received chain */ }
|
||||
}
|
||||
|
||||
for (const att of mail.attachments) {
|
||||
att.filename; att.contentType; att.size; att.checksum; // sha256 hex string
|
||||
}
|
||||
```
|
||||
**Empirically confirmed (this session):** parsing a synthetic multipart fixture with an embedded `http://` link in both text and HTML parts, plus a base64 PDF attachment, triggered **zero** calls to a `global.fetch` spy. `checksumAlgo: 'sha256'` produced a correct 64-hex-char digest matching `sha256sum` of the decoded attachment bytes.
|
||||
|
||||
### Pattern 3: Hand-rolled Authentication-Results parsing (D-06)
|
||||
|
||||
**What:** RFC 8601 defines `Authentication-Results:` as `authserv-id; method1/version1=result1 (comment) propspec; method2=result2 ...`. A small tokenizer split on `;` then `method=result` extraction covers the SPF/DKIM/DMARC cases needed.
|
||||
**When to use:** Feed it the raw string from `mail.headers.get('authentication-results')` (and `received-spf` as fallback).
|
||||
**Example (illustrative — write real tests against synthetic fixtures at plan/build time):**
|
||||
```typescript
|
||||
// Real-world sample observed during this research (structure only —
|
||||
// no PII) had headers: Authentication-Results, Authentication-Results-Original,
|
||||
// Received-SPF, DKIM-Signature, ARC-* — Microsoft/Mimecast stack.
|
||||
// A message CAN have more than one Authentication-Results header (added at
|
||||
// each hop) — mail.headers.get() only returns ONE value for a repeated
|
||||
// header in mailparser's Map. Use mail.headerLines (array, preserves all
|
||||
// occurrences) when multiple Authentication-Results headers may be present.
|
||||
|
||||
function parseAuthResults(headerValue: string): {
|
||||
spf?: 'pass' | 'fail' | 'softfail' | 'neutral' | 'none' | 'temperror' | 'permerror';
|
||||
dkim?: 'pass' | 'fail' | 'none' | 'temperror' | 'permerror';
|
||||
dmarc?: 'pass' | 'fail' | 'none' | 'temperror' | 'permerror';
|
||||
} {
|
||||
const result: Record<string, string> = {};
|
||||
// Split on ";" then look for "spf=", "dkim=", "dmarc=" tokens — RFC 8601
|
||||
// resultinfo is always "method=result", optionally followed by a
|
||||
// parenthetical comment and propspec (ptype.property=value) we don't need.
|
||||
for (const clause of headerValue.split(';')) {
|
||||
const m = clause.trim().match(/^(spf|dkim|dmarc)=(\w+)/i);
|
||||
if (m) result[m[1].toLowerCase()] = m[2].toLowerCase();
|
||||
}
|
||||
return result as ReturnType<typeof parseAuthResults>;
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 4: Self-initiated B2 upload (no precedent in this codebase yet)
|
||||
|
||||
**What:** every existing `presignUpload` call (`lib/services/rmm/executor.ts:337`) hands the signed URL to an *external* collector script to PUT. Phase 16 needs Pulse's own server code to PUT the bytes itself.
|
||||
**When to use:** After fetching raw `.eml` bytes from Autotask, before/while parsing.
|
||||
**Example:**
|
||||
```typescript
|
||||
// New pattern — Pulse PUTs to its own presigned URL, no external collector.
|
||||
import { presignUpload } from '@/lib/services/b2/client';
|
||||
|
||||
const objectKey = `phishing/${reportId}/${attachmentId}.eml`; // must match a NEW key regex — see Pitfall below
|
||||
const url = presignUpload(objectKey, 1800);
|
||||
const res = await fetch(url, { method: 'PUT', body: rawEmlBuffer });
|
||||
if (!res.ok) throw new Error(`B2 PUT ${objectKey} failed: ${res.status}`);
|
||||
```
|
||||
|
||||
### Anti-Patterns to Avoid
|
||||
|
||||
- **Rendering `mail.html` anywhere without sanitization.** mailparser explicitly does not sanitize HTML output (per its own docs). Never feed `mail.html` into anything that renders it (browser, markdown viewer) without a dedicated sanitizer — and never fetch anything referenced by it.
|
||||
- **Using `mailauth.authenticate()` "just to get structured verdicts."** It performs live SPF DNS lookups and BIMI HTTP fetches as a side effect of computing those verdicts — this is exactly the network activity SC#3's test suite is written to catch. Parse the *existing* stamped header instead.
|
||||
- **Loosening `OBJECT_KEY_REGEX` in `b2/client.ts` to fit an `.eml` key shape.** The project's own skill doc explicitly forbids this ("add a new key regex + a new transport rather than loosening the existing one") — LogLift's path-traversal guard must not be weakened to accommodate an unrelated payload type.
|
||||
- **Assuming `response.item` on the new `getAttachmentContent()` method.** Confirmed live: it's `response.items[0]`.
|
||||
|
||||
## Don't Hand-Roll
|
||||
|
||||
| Problem | Don't Build | Use Instead | Why |
|
||||
|---------|-------------|-------------|-----|
|
||||
| MIME multipart boundary parsing, charset/encoding conversion, address-header tokenizing | A regex-based `.eml` splitter | `mailparser` | RFC822/MIME has enough edge cases (nested multiparts, base64/quoted-printable, non-UTF-8 charsets, folded headers) that a hand-rolled parser will silently mis-parse real-world messages — exactly the "deceptively complex" category this section exists for |
|
||||
| Attachment content hashing | Manual `crypto.createHash` loop over decoded attachment buffers | `mailparser`'s `checksumAlgo: 'sha256'` option | Already computed correctly as part of parsing; verified to produce a correct 64-char sha256 hex digest in this research session |
|
||||
| URL detection in free text/HTML | Ad-hoc regex | `linkify-it` (or, if avoiding the dependency, a well-tested regex reviewed for `www.`/IDN edge cases) | Phishing URLs deliberately use odd formats (no scheme, punycode domains, unusual TLDs) that a naive `https?://` regex misses |
|
||||
|
||||
**Key insight:** Everything upstream of this phase's *own* decision logic (attachment selection, Authentication-Results interpretation, B2 key shaping) is a solved, actively-maintained ecosystem problem. Everything downstream of it (selection heuristic, structured-verdict extraction, key-shape validation) is Pulse-specific business logic that has no library to reach for — and is exactly what this phase should hand-write, tested against synthetic fixtures.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Pitfall 1: EVID-02's literal selection rule misses a real, in-scope ticket shape
|
||||
|
||||
**What goes wrong:** Implementing EVID-02 literally ("prefer `rfc.eml` over `OriginatingEmail.eml`, else disambiguate by `message/rfc822` content-type") produces no defined behavior for KnowBe4-sourced tickets, where the "real" message is named `phish_alert_sp2_2.0.0.0.eml` — not `rfc.eml` — and *both* attachments share `contentType: message/rfc822`, so content-type never disambiguates.
|
||||
**Why it happens:** The requirement text was written around the Microsoft "Report Message" flow; KnowBe4's PhishER integration (explicitly listed in DETECT-01's pattern set — "KnowBe4 Phish Alert Report", "Source: KnowBe4 Phish Alert Button") uses a different, versioned filename.
|
||||
**How to avoid:** Use a three-tier selection algorithm, empirically validated against 15 real tickets (12 recent + 3 from 2024) spanning both flows plus the attachment-count-1 edge case:
|
||||
1. If an attachment named exactly `rfc.eml` (case-insensitive) exists among `message/rfc822`-content-type attachments → select it.
|
||||
2. Else, among `message/rfc822` attachments, exclude any named exactly `OriginatingEmail.eml` (case-insensitive) → if exactly one candidate remains, select it (covers `phish_alert_sp2_2.0.0.0.eml` and any future versioned KnowBe4 filename).
|
||||
3. Else (0 or >1 ambiguous candidates after step 2) → fall back to `OriginatingEmail.eml` if present; else no email attachment found.
|
||||
This also correctly handles the observed case where **only** `OriginatingEmail.eml` exists (8 of 15 older sampled tickets had a single attachment, no distinct "reported message" file) — step 3's fallback is not theoretical, it's the majority case in that older sample.
|
||||
**Warning signs:** A synthetic-fixture-only test suite (SC#5) that only covers the `rfc.eml` + `OriginatingEmail.eml` pair will pass while silently mis-selecting (or crash-selecting) on real KnowBe4 tickets in production — write a fixture for the KnowBe4 shape and the single-attachment shape too, not just the literal-`rfc.eml` shape the requirement names.
|
||||
|
||||
### Pitfall 2: List-call attachment metadata never has content — a second call is mandatory
|
||||
|
||||
**What goes wrong:** Code that calls the existing `getAttachments()` and reads `.data` off the result will always get `null`/`undefined`, even though `Attachment.data` is typed `string | undefined` (implying it's sometimes populated by that call). It never is.
|
||||
**Why it happens:** Confirmed empirically — 3 separate real tickets' list calls all returned `data: null` for both attachments, at file sizes from 15KB to 100KB (i.e. not a size-based truncation; it's categorically never populated by the list endpoint).
|
||||
**How to avoid:** Always follow up a list call with a per-attachment-ID GET (`Tickets/{id}/Attachments/{attachmentId}`) for the specific attachment selected — and read `response.items?.[0]`, not `response.item` (see Pattern 1).
|
||||
**Warning signs:** `data` is always falsy in code paths that only called the list endpoint; a "successfully parsed 0-byte email" failure mode downstream.
|
||||
|
||||
### Pitfall 3: `B2_KEY_ID`/`B2_APP_KEY` are not set in this dev environment
|
||||
|
||||
**What goes wrong:** Local dev/testing of the D-05 B2-upload path will hit `B2NotConfiguredError` unless credentials are provided.
|
||||
**Why it happens:** Checked directly — `.env` in this repo has no `B2_*` variables at all (LogLift's B2 config is presumably prod-only or configured via a different `.env.local`/deployment secret).
|
||||
**How to avoid:** Gate the B2-upload step behind `isB2Configured()` (already exported from `b2/client.ts`) the same way other optional integrations are gated elsewhere in this codebase, and design tests to mock `presignUpload`/`fetch` rather than requiring real B2 credentials — SC#5 requires synthetic-fixture-only tests anyway, so this should already be the plan.
|
||||
|
||||
### Pitfall 4: A message can have more than one Authentication-Results header
|
||||
|
||||
**What goes wrong:** `mail.headers.get('authentication-results')` (a `Map`) only returns the *last* occurrence if mailparser folds repeated headers, or the first — behavior isn't guaranteed to expose all of them, and a real sample observed in this research had both `Authentication-Results` and `Authentication-Results-Original` (Microsoft's convention when a message transits multiple ARC hops / is remediated).
|
||||
**Why it happens:** SMTP allows repeated header fields; each hop (or Mimecast/M365 remediation step) can add its own `Authentication-Results`. `Authentication-Results-Original` in particular preserves the verdict *before* a security product like Mimecast rewrote the message.
|
||||
**How to avoid:** Iterate `mail.headerLines` (an ordered array preserving every occurrence, unlike the `headers` Map) when collecting Authentication-Results-family headers, and decide explicitly which one(s) to parse (likely: parse the outermost/latest `Authentication-Results` primarily, but consider whether `Authentication-Results-Original` deserves its own structured field — Phase 19's classifier will care about pre-remediation verdicts for Mimecast-protected tenants).
|
||||
**Warning signs:** A test fixture with only one Authentication-Results header passes while a real multi-hop message silently drops verdict data.
|
||||
|
||||
### Pitfall 5: mailparser attachments include inline/CID-referenced content, not just "real" attachments
|
||||
|
||||
**What goes wrong:** Treating every entry in `mail.attachments[]` as a user-facing "attachment" (per EVID-03's "attachment metadata") will also capture inline images referenced by `cid:` in the HTML body (e.g. an email signature logo), inflating attachment counts/hash-based indicators with noise.
|
||||
**Why it happens:** mailparser's default behavior includes both `Content-Disposition: attachment` parts and `Content-Disposition: inline`/CID-referenced parts in the same `attachments` array, distinguished only by the `related` flag (per Context7 docs).
|
||||
**How to avoid:** Decide explicitly (Claude's discretion, per CONTEXT.md) whether to filter `att.related`/inline parts out of the persisted `messages.attachments` and `indicators` rows, or keep them with a distinguishing flag — but decide deliberately rather than defaulting to "whatever mailparser returns."
|
||||
|
||||
## Code Examples
|
||||
|
||||
### Verified: mailparser produces zero network calls while parsing
|
||||
|
||||
```typescript
|
||||
// Verified in this research session — global.fetch spy recorded zero
|
||||
// invocations parsing a synthetic multipart fixture containing an
|
||||
// http:// URL in both text/plain and text/html parts, plus a base64
|
||||
// attachment. This is the shape the SC#3 test should take.
|
||||
import { simpleParser } from 'mailparser';
|
||||
import { vi, expect, it } from 'vitest';
|
||||
|
||||
it('never makes a network call while parsing', async () => {
|
||||
const fetchSpy = vi.spyOn(global, 'fetch');
|
||||
await simpleParser(syntheticEmlBuffer, { checksumAlgo: 'sha256' });
|
||||
expect(fetchSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
```
|
||||
|
||||
### Verified: attachment sha256 checksum matches decoded byte hash
|
||||
|
||||
```typescript
|
||||
// Verified: mailparser's checksum for a synthetic base64-encoded PDF
|
||||
// attachment (63 decoded bytes) was a correct 64-hex-char sha256 digest.
|
||||
const mail = await simpleParser(raw, { checksumAlgo: 'sha256' });
|
||||
mail.attachments[0].checksum; // e.g. "5b6543922ff74615afd92401489840af7b95a8ec0f63cf2b18db6d6f7175e973"
|
||||
```
|
||||
|
||||
## State of the Art
|
||||
|
||||
| Old Approach | Current Approach | When Changed | Impact |
|
||||
|--------------|------------------|---------------|--------|
|
||||
| N/A — no prior art in this codebase | `mailparser` 3.9.14 is the current stable line | Actively released through 2026-07-05 | No deprecation risk in the near term; the package has had a stable public API for years |
|
||||
|
||||
**Deprecated/outdated:** None identified — this is a greenfield capability in this codebase.
|
||||
|
||||
## Assumptions Log
|
||||
|
||||
| # | Claim | Section | Risk if Wrong |
|
||||
|---|-------|---------|---------------|
|
||||
| A1 | Suggested filenames (`eml-parser.ts`, `phishing-eml-service.ts`) and B2 object-key shape (`phishing/{reportId}/{attachmentId}.eml`) | Recommended Project Structure, Pattern 4 | Low — explicitly marked as Claude's discretion in CONTEXT.md; naming has no correctness impact, only consistency |
|
||||
| A2 | `linkify-it` is the best URL-extraction choice over a plain regex | Standard Stack, Don't Hand-Roll | Low-medium — both are viable; a regex is simpler but may miss unusual phishing URL formats. Explicitly flagged as Claude's discretion in CONTEXT.md, not a locked decision |
|
||||
| A3 | Recommendation to filter or flag `related`/inline (CID) attachments separately from "real" attachments | Pitfall 5 | Medium — if the planner doesn't decide this deliberately, indicator noise (e.g. hashing an email signature logo as a phishing indicator) could reduce Phase 19 classification signal quality |
|
||||
|
||||
All other claims in this research are `[VERIFIED]` (live Autotask API calls, live npm registry queries, live `mailparser`/`linkify-it` install-and-run tests, direct reads of this codebase's source) or `[CITED]` (Context7 official mailparser/mailauth documentation, the project's own `SKILL.md`). No claim about the Authentication-Results grammar itself is deeper than RFC 8601's well-established structure, used only as design input for hand-rolled code the planner will fully specify and test.
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Should `Authentication-Results-Original` be parsed into its own structured field, or is the outermost `Authentication-Results` sufficient for D-06?**
|
||||
- What we know: real Mimecast/M365-protected tickets can carry both; they can disagree (e.g. Mimecast remediation can downgrade a verdict after receipt).
|
||||
- What's unclear: whether Phase 19's classifier needs the pre-remediation verdict specifically, or just "the current, authoritative one."
|
||||
- Recommendation: parse both if present under distinct keys (e.g. `authResults` and `authResultsOriginal`) — cheap to add now, expensive to retrofit once Phase 19 depends on a single-verdict shape.
|
||||
|
||||
2. **Should inline/CID-referenced attachments be excluded from `messages.attachments`/`indicators`, or included with a flag?**
|
||||
- What we know: mailparser's `attachment.related` boolean distinguishes them.
|
||||
- What's unclear: whether an inline tracking pixel (a classic phishing technique) should be treated as attachment metadata or promoted to its own indicator type.
|
||||
- Recommendation: keep both, but persist the `related` flag so Phase 19 can weight/filter as needed rather than losing the distinction at parse time.
|
||||
|
||||
3. **What is the practical max size of a real `.eml` this pipeline will see?**
|
||||
- What we know: sampled real `OriginatingEmail.eml` files ranged 26KB-365KB; well under B2's existing 25MB cap.
|
||||
- What's unclear: whether any historical ticket has a pathological outlier (e.g. an email with many large attachments) that would stress mailparser's default buffering behavior (mailparser's own docs recommend streaming for large messages).
|
||||
- Recommendation: buffer (not stream) for the MVP given observed sizes, but add an explicit size guard (e.g. reject/flag anything over a few MB) before calling `simpleParser` — cheap insurance, not premature optimization.
|
||||
|
||||
## Environment Availability
|
||||
|
||||
| Dependency | Required By | Available | Version | Fallback |
|
||||
|------------|------------|-----------|---------|----------|
|
||||
| Autotask REST API (real credentials) | Attachment content fetch | Yes — verified live in this session | v1.0 (`webservices1.autotask.net/atservicesrest/v1.0`) | — |
|
||||
| Backblaze B2 (`B2_KEY_ID`/`B2_APP_KEY`) | D-05 raw `.eml` storage | No — not set in this repo's `.env` | — | Gate behind `isB2Configured()`; tests must mock, not require real credentials (SC#5 mandates synthetic-fixture-only tests anyway) |
|
||||
| `mailparser` (npm) | Core parsing | Not yet installed (needs `npm install`) | 3.9.14 confirmed current | — |
|
||||
| `linkify-it` (npm) | URL extraction | Not yet installed | 6.0.0 confirmed current | Plain regex if the planner opts not to add this dependency |
|
||||
| Postgres (`pulse-postgres` container) | `messages`/`indicators` writes | Yes — running, migrations 001-098 applied | 16 | — |
|
||||
|
||||
**Missing dependencies with no fallback:** none — B2 has an explicit fallback (skip/flag when unconfigured, same convention as other optional integrations in this codebase).
|
||||
|
||||
**Missing dependencies with fallback:**
|
||||
- Backblaze B2 credentials — not configured in this dev `.env`; code must handle `isB2Configured() === false` gracefully (log + skip raw storage, or fail the specific report's evidence capture without blocking the rest of the pipeline — exact behavior is plan-time discretion).
|
||||
|
||||
## Validation Architecture
|
||||
|
||||
### Test Framework
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Framework | vitest 4.1.5 |
|
||||
| Config file | `vitest.config.ts` — `environment: 'node'`, `include: ['lib/**/*.test.ts']` |
|
||||
| Quick run command | `npx vitest run lib/services/eml-parser.test.ts` |
|
||||
| Full suite command | `npm test` |
|
||||
|
||||
### Phase Requirements → Test Map
|
||||
|
||||
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|
||||
|--------|----------|-----------|-------------------|-------------|
|
||||
| EVID-02 | Selects `rfc.eml` over `OriginatingEmail.eml` when both present | unit | `npx vitest run lib/services/eml-parser.test.ts -t "selects rfc.eml"` | ❌ Wave 0 |
|
||||
| EVID-02 | Selects the non-`OriginatingEmail.eml` `message/rfc822` attachment when named differently (KnowBe4 shape) | unit | `npx vitest run lib/services/eml-parser.test.ts -t "KnowBe4"` | ❌ Wave 0 |
|
||||
| EVID-02 | Falls back to `OriginatingEmail.eml` when it's the only `.eml` attachment | unit | `npx vitest run lib/services/eml-parser.test.ts -t "fallback"` | ❌ Wave 0 |
|
||||
| EVID-03 | Parses headers/auth-results/Received chain/URLs/attachment metadata from a synthetic fixture | unit | `npx vitest run lib/services/eml-parser.test.ts -t "normalizes"` | ❌ Wave 0 |
|
||||
| EVID-04 | Never triggers a network call during parsing | unit | `npx vitest run lib/services/eml-parser.test.ts -t "no network"` | ❌ Wave 0 |
|
||||
| EVID-04 | Body preview is truncated/sanitized and distinct from raw body | unit | `npx vitest run lib/services/eml-parser.test.ts -t "body preview"` | ❌ Wave 0 |
|
||||
| EVID-03 | `AutotaskClient.getAttachmentContent()` reads `response.items[0]`, not `response.item` | unit | `npx vitest run lib/services/autotask-client.test.ts -t "getAttachmentContent"` | ❌ Wave 0 (no existing `autotask-client.test.ts` in this repo at all) |
|
||||
|
||||
### Sampling Rate
|
||||
- **Per task commit:** `npx vitest run lib/services/eml-parser.test.ts`
|
||||
- **Per wave merge:** `npm test`
|
||||
- **Phase gate:** Full suite green before `/gsd:verify-work`
|
||||
|
||||
### Wave 0 Gaps
|
||||
- [ ] `lib/services/eml-parser.test.ts` — new file, covers EVID-02/03/04 per the map above
|
||||
- [ ] Synthetic fixtures — at minimum: (a) `rfc.eml` + `OriginatingEmail.eml` pair, (b) KnowBe4-shaped pair (`phish_alert_sp2_2.0.0.0.eml`-style name + `OriginatingEmail.eml`), (c) `OriginatingEmail.eml`-only case, (d) a raw `.eml` buffer fixture with multipart/mixed + text+html bodies + one attachment + an `Authentication-Results` header with spf/dkim/dmarc — all synthetic, no real customer content per the milestone's explicit Out-of-Scope constraint
|
||||
- [ ] `lib/services/autotask-client.test.ts` — does not exist yet in this repo for ANY method, not just the new one; Phase 16 introduces the first coverage for `AutotaskClient` if the planner wants unit coverage on `getAttachmentContent()` specifically (mocking `fetch`)
|
||||
- [ ] Framework install: none — vitest is already configured and used extensively (`lib/services/analyzer/**`, `lib/services/rmm/**`, `lib/services/b2/**`)
|
||||
|
||||
## Security Domain
|
||||
|
||||
### Applicable ASVS Categories
|
||||
|
||||
| ASVS Category | Applies | Standard Control |
|
||||
|---------------|---------|-----------------|
|
||||
| V2 Authentication | No | Phase 16 introduces no new auth surface (pure service-layer parsing; API routes are Phase 18) |
|
||||
| V3 Session Management | No | Same as above |
|
||||
| V4 Access Control | No | Same as above — `ACCESS-01` is explicitly mapped to Phase 18 |
|
||||
| V5 Input Validation | Yes | This phase's entire purpose is safely parsing untrusted, attacker-controlled RFC822/MIME bytes. Use `mailparser` (handles malformed MIME defensively) rather than hand-rolled parsing; enforce a size guard before calling `simpleParser`; never pass `mail.html` to anything that renders or fetches from it |
|
||||
| V6 Cryptography | Yes (limited) | sha256 attachment hashing via `checksumAlgo: 'sha256'` — integrity/identification only, not a security boundary; no key management or signing occurs in this phase |
|
||||
|
||||
### Known Threat Patterns for this stack
|
||||
|
||||
| Pattern | STRIDE | Standard Mitigation |
|
||||
|---------|--------|---------------------|
|
||||
| Malicious/oversized MIME structure (deeply nested multiparts, huge headers) causing resource exhaustion while parsing attacker-controlled content | Denial of Service | Size guard (reject/flag `.eml` content over a threshold, e.g. a few MB) before calling `simpleParser`; rely on `mailparser`'s own defensive parsing rather than hand-rolled boundary splitting |
|
||||
| Spoofed `From`/`Reply-To`/`Return-Path` headers used to impersonate a trusted sender | Spoofing | This is exactly what D-06's structured SPF/DKIM/DMARC verdict extraction exists to surface to Phase 19's classifier — don't trust `From` display name alone; always carry the parsed auth verdicts alongside it |
|
||||
| Malicious URLs or HTML in the message body being rendered, followed, or fetched by Pulse itself | Tampering / Information Disclosure (SSRF-adjacent) | SC#3's hard requirement — never fetch/execute anything found in the message; `mailparser` itself makes no such calls (verified), but any body-preview or URL-extraction code added on top must be equally careful never to `fetch()` an extracted URL "to check if it's alive" or similar |
|
||||
| Raw email bytes (potentially containing malware-laced attachments) persisted somewhere reachable by a browser or naive file server | Information Disclosure / Elevation of Privilege | D-05 already addresses this — B2, not Postgres, not local filesystem; reuse the existing 25MB `MAX_DOWNLOAD_BYTES` cap and object-key validation pattern (with a new, equally strict regex per Pitfall/Anti-Pattern above) |
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
|
||||
- Live Autotask REST API v1.0 (`webservices1.autotask.net/atservicesrest/v1.0`) — 3 read-only `GET .../Tickets/{id}/Attachments` and `GET .../Tickets/{id}/Attachments/{attachmentId}` calls against real production tickets, 2026-07-15 (this session)
|
||||
- `docker exec pulse-postgres psql` — direct query of live `tickets` table for 15 real phishing-pattern ticket titles/IDs used to sample attachment shapes
|
||||
- Context7 `/nodemailer/mailparser` — ParsedMail type definition, attachment/checksum quick-start, security considerations
|
||||
- Context7 `/postalsys/mailauth` — public `authenticate()`/`spf()`/`dmarc()`/`bimi()` API surface (confirms live-verification-only public API)
|
||||
- `npm view mailparser`, `npm view linkify-it`, `npm view mailauth`, `npm view authentication-results-parser` — version, license, dependencies, postinstall scripts, registry existence
|
||||
- `npm pack mailauth` + inspection of `lib/mailauth.js`'s `module.exports` — confirms `parseReceivedHeaders`/`parseDkimArcHeaders` are not part of the public export surface
|
||||
- Live install + `simpleParser()` run against a synthetic fixture with a `global.fetch` spy — confirms zero network calls, correct sha256 checksum, correct header/address parsing
|
||||
- Live install + `linkify-it` `.match()` run against synthetic phishing-style text — confirms URL/mailto extraction behavior
|
||||
- `slopcheck install mailparser linkify-it` (v0.6.1, isolated scratch directory) — both `[OK]`
|
||||
- Direct reads of `/opt/stacks/pulse/lib/services/b2/client.ts`, `lib/services/autotask-client.ts`, `lib/services/phishing-detector.ts`, `lib/types/autotask.ts`, `migrations/097_phishing_triage_schema.sql`, `migrations/098_phishing_sweep_schedule.sql`, `.claude/skills/pulse-overshell-b2-evidence/SKILL.md`, `vitest.config.ts`, `.planning/config.json`, `.planning/ROADMAP.md`
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
|
||||
- npm download-count API (`api.npmjs.org/downloads/point/last-week/...`) for `mailparser` (2.83M/wk) and `linkify-it` (22.8M/wk)
|
||||
|
||||
### Tertiary (LOW confidence)
|
||||
|
||||
- None — every material claim in this document was independently verified during this research session
|
||||
|
||||
## Metadata
|
||||
|
||||
**Confidence breakdown:**
|
||||
- Standard stack (mailparser/linkify-it): HIGH — verified via Context7, live npm registry, and a live install-and-parse test with a network-call spy
|
||||
- Authentication-Results approach (hand-rolled, mailauth rejected): HIGH — verified via inspection of mailauth's actual public export surface (`npm pack` + read `lib/mailauth.js`)
|
||||
- Autotask attachment content-fetch mechanics: HIGH — verified via 5 live read-only API calls against real production data (list + per-ID for 3 tickets, plus a broader 12-ticket + 9-ticket sweep for the selection-heuristic finding)
|
||||
- `.eml` selection algorithm: HIGH — verified via a 15-ticket empirical sample spanning 2024-2026 and both known upstream reporting flows (Microsoft Report Message, KnowBe4 PhishER)
|
||||
- B2 reuse mechanics (D-05): HIGH — verified via direct source read of `b2/client.ts` plus the project's own explicit skill-doc guidance
|
||||
- Security domain / pitfalls: HIGH — grounded in the same empirical findings above, not speculative
|
||||
|
||||
**Research date:** 2026-07-15
|
||||
**Valid until:** 30 days (stable ecosystem; mailparser/linkify-it release cadence is slow-moving; Autotask API attachment behavior is unlikely to change without notice, but re-verify if Autotask API version changes from v1.0)
|
||||
Loading…
Add table
Add a link
Reference in a new issue