docs(16): create phase plan (3 plans, 2 waves)
This commit is contained in:
parent
94b53c11d0
commit
00b8afc546
4 changed files with 685 additions and 2 deletions
|
|
@ -390,7 +390,10 @@ summarizes classification, blast radius, and recommended/approved remediation st
|
|||
3. The parser never executes or fetches any URL found in a message — verified by tests asserting no outbound network calls happen during parsing
|
||||
4. Parsed output includes a sanitized/truncated body preview stored alongside the raw evidence, distinct from the full raw body
|
||||
5. `npx vitest run` for the new parser test file passes using synthetic fixtures only (no real customer email)
|
||||
**Plans**: TBD
|
||||
**Plans**: 3 plans
|
||||
- [ ] 16-01-PLAN.md — Deps (mailparser + linkify-it) + pure EML parser: 3-tier selection, RFC822/MIME normalization, structured SPF/DKIM/DMARC verdicts, sanitized preview, no-network + size-guard tests (EVID-02, EVID-03, EVID-04)
|
||||
- [ ] 16-02-PLAN.md — Supporting infra: AutotaskClient.getAttachmentContent (items[0]), b2 EML_OBJECT_KEY_REGEX + parameterized key validation, migration 099 indicators.metadata JSONB (EVID-03, EVID-04; D-05, D-07)
|
||||
- [ ] 16-03-PLAN.md — phishing-eml-service orchestration: list→select→fetch→B2 (gated)→parse→persist messages/indicators, end-to-end no-network + graceful-degrade tests (EVID-03, EVID-04; D-05, D-06, D-07)
|
||||
**UI hint**: no
|
||||
|
||||
### Phase 17: Mimecast Blast Radius Lookup
|
||||
|
|
@ -478,7 +481,7 @@ Phases execute in numeric order. v1.0 (Phases 1-9.1) shipped 2026-07-10. v2.0 (P
|
|||
| 13. Scheduler & Admin Toggle | v2.0 | 3/3 | Complete | 2026-07-11 |
|
||||
| 14. /pax8 UI Surface | v2.0 | 6/6 | Complete | 2026-07-12 |
|
||||
| 15. Data Model, Detection & Ticket Evidence | v3.0 | 3/3 | Complete | 2026-07-15 |
|
||||
| 16. EML/MIME Evidence Parser | v3.0 | 0/TBD | Not started | - |
|
||||
| 16. EML/MIME Evidence Parser | v3.0 | 0/3 | Planned | - |
|
||||
| 17. Mimecast Blast Radius Lookup | v3.0 | 0/TBD | Not started | - |
|
||||
| 18. Campaign Grouping & Phishing Analysis API | v3.0 | 0/TBD | Not started | - |
|
||||
| 19. Classification Engine | v3.0 | 0/TBD | Not started | - |
|
||||
|
|
|
|||
245
.planning/phases/16-eml-mime-evidence-parser/16-01-PLAN.md
Normal file
245
.planning/phases/16-eml-mime-evidence-parser/16-01-PLAN.md
Normal file
|
|
@ -0,0 +1,245 @@
|
|||
---
|
||||
phase: 16-eml-mime-evidence-parser
|
||||
plan: 01
|
||||
type: tdd
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- package.json
|
||||
- package-lock.json
|
||||
- lib/services/eml-parser.ts
|
||||
- lib/services/eml-parser.fixtures.ts
|
||||
- lib/services/eml-parser.test.ts
|
||||
autonomous: true
|
||||
requirements: [EVID-02, EVID-03, EVID-04]
|
||||
user_setup: []
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "Given a multi-attachment list containing rfc.eml and OriginatingEmail.eml, selection returns rfc.eml"
|
||||
- "Given a KnowBe4-shaped list (phish_alert_sp2_2.0.0.0.eml + OriginatingEmail.eml, both message/rfc822), selection returns the non-OriginatingEmail attachment"
|
||||
- "Given only OriginatingEmail.eml, selection returns OriginatingEmail.eml"
|
||||
- "Parsing a synthetic .eml buffer yields normalized From/displayName/senderEmail/senderDomain/Reply-To/Return-Path/To/Cc/Subject/Date/Message-ID/Received-chain, structured SPF/DKIM/DMARC verdicts, extracted URLs, and per-attachment name/content-type/size/sha256"
|
||||
- "No network call (fetch) occurs during parseEml on any synthetic fixture"
|
||||
- "Body preview is plain-text, truncated, and distinct from the raw HTML/text body"
|
||||
- "A .eml buffer larger than the size guard is rejected before simpleParser is called"
|
||||
artifacts:
|
||||
- path: "lib/services/eml-parser.ts"
|
||||
provides: "parseEml, selectOriginalMessage, parseAuthResults, extractUrls, buildBodyPreview + NormalizedMessage type"
|
||||
min_lines: 120
|
||||
- path: "lib/services/eml-parser.test.ts"
|
||||
provides: "EVID-02/03/04 vitest coverage incl. no-network spy and 3-tier selection"
|
||||
min_lines: 100
|
||||
- path: "lib/services/eml-parser.fixtures.ts"
|
||||
provides: "synthetic .eml buffers + attachment-metadata lists for all three selection tiers"
|
||||
min_lines: 40
|
||||
key_links:
|
||||
- from: "lib/services/eml-parser.ts"
|
||||
to: "mailparser"
|
||||
via: "simpleParser import with checksumAlgo sha256"
|
||||
pattern: "from 'mailparser'"
|
||||
- from: "lib/services/eml-parser.test.ts"
|
||||
to: "global.fetch"
|
||||
via: "vi.spyOn assertion of zero calls"
|
||||
pattern: "spyOn\\(global, 'fetch'\\)"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Add the two npm dependencies (`mailparser`, `linkify-it`) and build the pure,
|
||||
I/O-free EML parser module that is the heart of Phase 16 — attachment selection
|
||||
(EVID-02, three tiers), RFC822/MIME normalization with structured auth verdicts
|
||||
(EVID-03, D-06), and a sanitized truncated body preview that never triggers a
|
||||
network call (EVID-04).
|
||||
|
||||
Purpose: Everything downstream (Plan 03 orchestration → messages/indicators rows,
|
||||
Phase 18 campaign grouping) keys on this module's normalized output. It must be
|
||||
correct, fully test-covered against synthetic fixtures only, and provably free of
|
||||
outbound I/O.
|
||||
Output: `lib/services/eml-parser.ts`, `lib/services/eml-parser.fixtures.ts`,
|
||||
`lib/services/eml-parser.test.ts`, plus the two dependencies in package.json.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/16-eml-mime-evidence-parser/16-RESEARCH.md
|
||||
@.planning/phases/16-eml-mime-evidence-parser/16-PATTERNS.md
|
||||
@lib/services/b2/client.ts
|
||||
@lib/services/b2/client.test.ts
|
||||
@lib/services/phishing-detector.test.ts
|
||||
@lib/types/autotask.ts
|
||||
|
||||
<interfaces>
|
||||
<!-- Contracts this plan CREATES. Downstream plans (03) and Phase 18 consume these. -->
|
||||
|
||||
Attachment (existing, lib/types/autotask.ts): { id:number; attachmentType; fullPath:string; title:string; publish; data?:string; contentType?:string; createDate?; creatorResourceID? }
|
||||
|
||||
New exports from lib/services/eml-parser.ts (define these — they are the contract):
|
||||
|
||||
- type AuthVerdict = 'pass' | 'fail' | 'softfail' | 'neutral' | 'none' | 'temperror' | 'permerror'
|
||||
- interface AuthResults { spf?: AuthVerdict; dkim?: AuthVerdict; dmarc?: AuthVerdict }
|
||||
- interface AttachmentMeta { filename: string | null; contentType: string | null; size: number; checksum: string | null; related: boolean }
|
||||
- interface NormalizedMessage {
|
||||
from: { displayName: string | null; email: string | null; domain: string | null };
|
||||
replyTo: string | null;
|
||||
returnPath: string | null;
|
||||
to: string[];
|
||||
cc: string[];
|
||||
subject: string | null;
|
||||
date: string | null; // ISO 8601 or null
|
||||
messageId: string | null;
|
||||
receivedChain: string[]; // ordered, outermost-first as encountered in headerLines
|
||||
authResults: AuthResults; // parsed from the primary Authentication-Results header
|
||||
authResultsOriginal: AuthResults | null; // parsed from Authentication-Results-Original if present (Open Question 1)
|
||||
urls: string[]; // deduped, from text + html parts
|
||||
attachments: AttachmentMeta[]; // includes inline/related, distinguished by `related`
|
||||
bodyPreview: string; // truncated plain text, distinct from raw body
|
||||
}
|
||||
- function selectOriginalMessage(attachments: Attachment[]): Attachment | null
|
||||
- async function parseEml(rawEmlBuffer: Buffer): Promise<NormalizedMessage>
|
||||
- function parseAuthResults(headerValue: string): AuthResults
|
||||
- function extractUrls(text: string | null | undefined, html: string | null | undefined): string[]
|
||||
- function buildBodyPreview(text: string | null | undefined, html: string | null | undefined): string
|
||||
- const MAX_EML_BYTES: number // size guard, below B2's 25 MB cap
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Install mailparser + linkify-it (deliberate, reviewed)</name>
|
||||
<files>package.json, package-lock.json</files>
|
||||
<read_first>
|
||||
- package.json (confirm neither dependency is already present; confirm current dependency block shape)
|
||||
- .planning/phases/16-eml-mime-evidence-parser/16-RESEARCH.md (Package Legitimacy Audit section + the "Process note for the planner" about cwd drift)
|
||||
</read_first>
|
||||
<action>
|
||||
From the repo root `/opt/stacks/pulse`, run `npm install mailparser linkify-it` as a single deliberate command (NOT inside a throwaway verification script). Target versions confirmed clean by the research slopcheck audit: mailparser 3.9.14, linkify-it 6.0.0 (both MIT, no postinstall scripts, [OK] verdict). Also install the matching `@types/mailparser` and `@types/linkify-it` devDependencies if the packages do not ship their own bundled types (check `npm view <pkg> types` first). After installing, run `git diff --stat package.json` and confirm ONLY `mailparser` and `linkify-it` (plus their `@types/*` if needed) were added to `dependencies`/`devDependencies` — no unrelated churn from a cwd that drifted. If `git diff package.json` shows any package other than these being added, revert with `git checkout -- package.json package-lock.json` and re-run from the confirmed repo root.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /opt/stacks/pulse && node -e "require('mailparser'); require('linkify-it'); console.log('deps-ok')" && git diff package.json | grep -E '^\+' | grep -vE 'mailparser|linkify-it|@types/(mailparser|linkify-it)|^\+\+\+' | grep -E '"[a-z]' && echo "UNEXPECTED_DEP" || echo "clean"</automated>
|
||||
</verify>
|
||||
<done>Both packages import at runtime; `git diff package.json` added lines contain only mailparser, linkify-it, and optionally their @types — the guard command prints "clean" (no UNEXPECTED_DEP).</done>
|
||||
<acceptance_criteria>
|
||||
- `require('mailparser')` and `require('linkify-it')` both succeed (node exits 0, prints "deps-ok")
|
||||
- `package.json` diff introduces exactly mailparser + linkify-it (+ optional @types) and no other new dependency keys
|
||||
- `package-lock.json` is updated consistently (npm install completed without error)
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
|
||||
<task type="tdd">
|
||||
<name>Task 2: Attachment selection (EVID-02, three tiers) — RED then GREEN</name>
|
||||
<files>lib/services/eml-parser.ts, lib/services/eml-parser.fixtures.ts, lib/services/eml-parser.test.ts</files>
|
||||
<read_first>
|
||||
- .planning/phases/16-eml-mime-evidence-parser/16-RESEARCH.md (Common Pitfalls → Pitfall 1: the exact three-tier selection algorithm empirically validated against 15 tickets)
|
||||
- .planning/phases/16-eml-mime-evidence-parser/16-PATTERNS.md (eml-parser.ts module-shape analog + describe-per-export test style)
|
||||
- lib/services/b2/client.ts (module conventions: top-of-file doc comment stating the no-I/O invariant, named exports, no class, `_XXX_INTERNALS` test export)
|
||||
- lib/services/phishing-detector.test.ts (describe-per-export + plain-language `it` titles)
|
||||
- lib/types/autotask.ts lines 287-297 (Attachment interface — title/fullPath/contentType/id fields to select on)
|
||||
</read_first>
|
||||
<behavior>
|
||||
- Tier 1: given attachments where one is named exactly `rfc.eml` (case-insensitive) among `message/rfc822` content-type attachments → selectOriginalMessage returns that attachment
|
||||
- Tier 2 (KnowBe4): given `phish_alert_sp2_2.0.0.0.eml` + `OriginatingEmail.eml`, both `message/rfc822`, no `rfc.eml` → returns the non-`OriginatingEmail.eml` candidate (the phish_alert one) since exactly one candidate remains after excluding OriginatingEmail.eml
|
||||
- Tier 3 (fallback): given only `OriginatingEmail.eml` → returns OriginatingEmail.eml
|
||||
- Ambiguous: given two non-OriginatingEmail `message/rfc822` candidates and no `rfc.eml` → falls back to OriginatingEmail.eml if present, else returns null
|
||||
- Empty / no `.eml`: given an attachment list with no `message/rfc822` and no `.eml` names → returns null
|
||||
- Case-insensitivity: `RFC.EML`, `OriginatingEmail.EML` match their tiers
|
||||
</behavior>
|
||||
<action>
|
||||
Create `lib/services/eml-parser.ts` with the module doc-comment (provenance: mailparser + hand-rolled RFC 8601 parsing; state the "never fetches or executes anything found in a message" invariant up front, mirroring b2/client.ts's doc comment). Export `selectOriginalMessage(attachments: Attachment[]): Attachment | null` implementing the three-tier algorithm from RESEARCH.md Pitfall 1 exactly: (1) exact `rfc.eml` case-insensitive among `message/rfc822`; (2) else the single non-`OriginatingEmail.eml` `message/rfc822` candidate; (3) else `OriginatingEmail.eml` if present; else null. Match filename on both `title` and `fullPath` (use whichever carries the name), lowercased. Treat content-type case-insensitively. Create `lib/services/eml-parser.fixtures.ts` exporting attachment-metadata list fixtures for all three tiers plus the ambiguous and no-eml cases (each a minimal `Attachment[]` with id/title/fullPath/contentType). Create `lib/services/eml-parser.test.ts` with a `describe('selectOriginalMessage', ...)` block covering every case in the behavior list — write the tests FIRST (RED: they fail against a stub), then implement (GREEN). Commit RED then GREEN separately per TDD convention.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /opt/stacks/pulse && npx vitest run lib/services/eml-parser.test.ts -t "selectOriginalMessage" && npx tsc --noEmit --pretty 2>&1 | (grep -i eml-parser && exit 1 || echo tsc-ok)</automated>
|
||||
</verify>
|
||||
<done>All selectOriginalMessage tests pass (rfc.eml, KnowBe4, fallback, ambiguous→OriginatingEmail, no-eml→null, case-insensitive); tsc reports no errors for eml-parser files.</done>
|
||||
<acceptance_criteria>
|
||||
- `npx vitest run lib/services/eml-parser.test.ts -t "selects rfc.eml"` passes
|
||||
- `npx vitest run lib/services/eml-parser.test.ts -t "KnowBe4"` passes
|
||||
- `npx vitest run lib/services/eml-parser.test.ts -t "fallback"` passes
|
||||
- selectOriginalMessage returns null (not throws) for a no-`.eml` list — asserted by a test
|
||||
- eml-parser.ts contains a top-of-file doc comment that literally states the never-fetch/never-execute invariant (grep-checkable string, e.g. "never fetch")
|
||||
- No fenced-code selection heuristic beyond the three tiers; tier order matches RESEARCH.md Pitfall 1
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
|
||||
<task type="tdd">
|
||||
<name>Task 3: parseEml + auth-results + URLs + body preview + size guard (EVID-03, EVID-04, D-06)</name>
|
||||
<files>lib/services/eml-parser.ts, lib/services/eml-parser.fixtures.ts, lib/services/eml-parser.test.ts</files>
|
||||
<read_first>
|
||||
- lib/services/eml-parser.ts (current state after Task 2 — extend the same module)
|
||||
- .planning/phases/16-eml-mime-evidence-parser/16-RESEARCH.md (Pattern 2 mailparser usage, Pattern 3 hand-rolled Authentication-Results, Pitfall 4 multiple Authentication-Results headers, Pitfall 5 inline/CID attachments, Code Examples: no-network spy + sha256 checksum, Open Questions 1-3, Security Domain size-guard threat)
|
||||
- .planning/phases/16-eml-mime-evidence-parser/16-PATTERNS.md (network-call-spy pattern + describe-per-export test style)
|
||||
</read_first>
|
||||
<behavior>
|
||||
- parseEml(buffer) on a synthetic multipart/mixed fixture (text + html + one base64 attachment + Authentication-Results header carrying spf/dkim/dmarc) returns a NormalizedMessage with: from.displayName, from.email, from.domain (domain = substring after '@' of the sender email), replyTo, returnPath, to[], cc[], subject, date (ISO or null), messageId
|
||||
- authResults contains structured verdicts {spf,dkim,dmarc} matching the fixture header (e.g. spf: 'pass', dkim: 'fail', dmarc: 'none') — parsed via parseAuthResults, NOT raw header text
|
||||
- authResultsOriginal is populated when the fixture also has an Authentication-Results-Original header, null otherwise (iterate mail.headerLines, not just the headers Map, to catch repeated/family headers — Pitfall 4)
|
||||
- receivedChain is an ordered string[] of every Received header line
|
||||
- urls is a deduped list containing every http/https/www URL present in both the text and html parts (extractUrls via linkify-it), and never triggers a fetch
|
||||
- attachments[] carries filename/contentType/size/checksum (sha256 hex, 64 chars) for the base64 attachment; the `related` flag distinguishes inline/CID parts (Pitfall 5) — keep both, do not drop inline
|
||||
- bodyPreview is derived from mail.text (falling back to html-to-text of mail.html), truncated to a fixed max length, and is strictly shorter than / distinct from the raw body for a long-body fixture
|
||||
- parseEml never calls global.fetch (asserted with a spy) on any fixture (EVID-04 / SC#3)
|
||||
- parseEml (or a pre-check) throws or rejects a buffer larger than MAX_EML_BYTES BEFORE calling simpleParser (DoS size guard)
|
||||
</behavior>
|
||||
<action>
|
||||
Extend `lib/services/eml-parser.ts` to implement `parseEml`, `parseAuthResults`, `extractUrls`, `buildBodyPreview`, and export `MAX_EML_BYTES` (set a few MB, e.g. 10 * 1024 * 1024 — below b2/client.ts's MAX_DOWNLOAD_BYTES of 25 MB). parseEml: guard `buffer.byteLength > MAX_EML_BYTES` first (throw a clear Error before any parsing — this is the DoS mitigation for untrusted MIME); then call `simpleParser(buffer, { checksumAlgo: 'sha256' })`; map mailparser output into NormalizedMessage per the interfaces block. parseAuthResults: hand-rolled RFC 8601 tokenizer per RESEARCH.md Pattern 3 — split the header value on ';', match `(spf|dkim|dmarc)=<result>` case-insensitively, return the structured verdicts; expose the internal clause-splitter via `_EML_PARSER_INTERNALS` if it needs isolated test access (matching b2/client.ts's `_B2_INTERNALS` convention). Collect Authentication-Results-family headers by iterating `mail.headerLines` (Pitfall 4); parse the primary Authentication-Results into `authResults` and Authentication-Results-Original (if present) into `authResultsOriginal` (Open Question 1 recommendation). extractUrls: use linkify-it with `.tlds()` fuzzy matching enabled, scan mail.text and mail.html, dedupe, and NEVER fetch any extracted URL. buildBodyPreview: prefer mail.text; fall back to html-to-text (already a mailparser transitive dep) of mail.html; truncate to a fixed cap; this is stored distinct from the raw body (raw body is Plan 03's B2 concern, never persisted to Postgres). Keep inline/`related` attachments with the `related` flag rather than dropping them (Pitfall 5). Extend eml-parser.fixtures.ts with a rich multipart fixture (text+html+base64 attachment+Authentication-Results w/ spf pass, dkim fail, dmarc none; a second fixture adding Authentication-Results-Original; a long-body fixture for the preview-truncation test; an oversized buffer for the size-guard test). Add the corresponding `describe('parseEml', ...)`, `describe('parseAuthResults', ...)`, `describe('extractUrls', ...)`, `describe('buildBodyPreview', ...)` blocks including the no-network `vi.spyOn(global, 'fetch')` assertion. Write tests RED first, then implement GREEN. Use `[EML-PARSER]` as the console.error tag for any caught error path (e.g. malformed MIME).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /opt/stacks/pulse && npx vitest run lib/services/eml-parser.test.ts && npx tsc --noEmit --pretty 2>&1 | (grep -i eml-parser && exit 1 || echo tsc-ok)</automated>
|
||||
</verify>
|
||||
<done>Full eml-parser.test.ts suite passes: normalized headers, structured spf/dkim/dmarc verdicts, Received chain, deduped URLs, attachment sha256 metadata (incl. related flag), truncated body preview distinct from raw body, no-network spy asserts zero fetch calls, and oversized buffer is rejected before simpleParser. tsc clean for eml-parser files.</done>
|
||||
<acceptance_criteria>
|
||||
- `npx vitest run lib/services/eml-parser.test.ts -t "normalizes"` passes (headers + auth + received + urls + attachment meta)
|
||||
- `npx vitest run lib/services/eml-parser.test.ts -t "no network"` passes (fetch spy called 0 times)
|
||||
- `npx vitest run lib/services/eml-parser.test.ts -t "body preview"` passes (truncated + distinct from raw)
|
||||
- A test asserts parseEml rejects/throws on a buffer > MAX_EML_BYTES before simpleParser runs
|
||||
- A test asserts authResults.spf/dkim/dmarc are structured verdict strings (not the raw header)
|
||||
- A test asserts attachments[].checksum is a 64-char lowercase hex sha256 string
|
||||
- A test asserts an inline/`related`-flagged attachment is preserved with related === true
|
||||
- `npx vitest run lib/services/eml-parser.test.ts` exits 0 with no real customer email content in any fixture (all synthetic)
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| Autotask attachment bytes → eml-parser | Attacker-controlled RFC822/MIME bytes (a reported phishing email) cross into Pulse's parser |
|
||||
| npm registry → build | Third-party parsing libraries enter the trusted build |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|-------------|-----------------|
|
||||
| T-16-01 | Denial of Service | parseEml over untrusted MIME | mitigate | Size guard: reject buffer > MAX_EML_BYTES (a few MB, below B2's 25 MB cap) BEFORE calling simpleParser; rely on mailparser's defensive multipart parsing rather than hand-rolled boundary splitting |
|
||||
| T-16-02 | Spoofing | forged From/Reply-To/Return-Path | mitigate | Parse Authentication-Results into structured SPF/DKIM/DMARC verdicts (D-06) and carry them in NormalizedMessage alongside From, so Phase 19 never trusts display name alone |
|
||||
| T-16-03 | Tampering / Info Disclosure (SSRF-adjacent) | extractUrls / buildBodyPreview / parseEml | mitigate | Hard invariant, test-enforced: no code path fetches or executes any URL found in the message. `vi.spyOn(global, 'fetch')` asserts zero calls during parseEml; mailparser verified I/O-free; extractUrls only string-matches, never dereferences; mail.html is never rendered or fetched |
|
||||
| T-16-SC | Tampering | npm installs (mailparser, linkify-it) | mitigate | Research slopcheck audit: both [OK] (MIT, 15yr / markdown-it ecosystem, no postinstall scripts). Install is a deliberate reviewed task with a `git diff package.json` gate asserting only the two intended deps were added. Neither is [ASSUMED]/[SUS] → no blocking human checkpoint required |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
- `npx vitest run lib/services/eml-parser.test.ts` — full new suite green
|
||||
- `npx tsc --noEmit --pretty` — no type errors introduced
|
||||
- `git diff package.json` — only mailparser + linkify-it (+ optional @types) added
|
||||
- No fixture contains real customer email (synthetic-only per REQUIREMENTS.md Out of Scope)
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- EVID-02: three-tier selection (rfc.eml → non-OriginatingEmail message/rfc822 → OriginatingEmail.eml) all covered and passing
|
||||
- EVID-03: normalized headers + structured SPF/DKIM/DMARC verdicts + Received chain + URLs + attachment metadata (name/content-type/size/sha256) produced from a synthetic fixture
|
||||
- EVID-04: sanitized truncated body preview distinct from raw body; zero network calls during parsing (spy-asserted)
|
||||
- DoS size guard rejects oversized buffers before parsing
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/16-eml-mime-evidence-parser/16-01-SUMMARY.md` when done
|
||||
</output>
|
||||
220
.planning/phases/16-eml-mime-evidence-parser/16-02-PLAN.md
Normal file
220
.planning/phases/16-eml-mime-evidence-parser/16-02-PLAN.md
Normal file
|
|
@ -0,0 +1,220 @@
|
|||
---
|
||||
phase: 16-eml-mime-evidence-parser
|
||||
plan: 02
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- lib/services/autotask-client.ts
|
||||
- lib/services/autotask-client.test.ts
|
||||
- lib/services/b2/client.ts
|
||||
- lib/services/b2/client.test.ts
|
||||
- migrations/099_indicators_metadata.sql
|
||||
autonomous: true
|
||||
requirements: [EVID-03, EVID-04]
|
||||
user_setup: []
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "AutotaskClient.getAttachmentContent() returns the attachment with populated base64 data by reading response.items[0], not response.item"
|
||||
- "b2/client.ts exports EML_OBJECT_KEY_REGEX matching phishing/{id}/{id}.eml and rejecting path-traversal / non-.eml shapes"
|
||||
- "presignDownload/presignUpload/downloadToBuffer accept an optional key-regex arg defaulting to OBJECT_KEY_REGEX (existing LogLift call sites unchanged)"
|
||||
- "indicators table has a nullable metadata JSONB column after migration 099 applies"
|
||||
artifacts:
|
||||
- path: "lib/services/autotask-client.ts"
|
||||
provides: "getAttachmentContent(entityName, entityId, attachmentId) method"
|
||||
contains: "getAttachmentContent"
|
||||
- path: "lib/services/autotask-client.test.ts"
|
||||
provides: "first AutotaskClient unit coverage — items[0] behavior"
|
||||
min_lines: 30
|
||||
- path: "lib/services/b2/client.ts"
|
||||
provides: "EML_OBJECT_KEY_REGEX + parameterized key validation"
|
||||
contains: "EML_OBJECT_KEY_REGEX"
|
||||
- path: "migrations/099_indicators_metadata.sql"
|
||||
provides: "ALTER TABLE indicators ADD COLUMN metadata JSONB (D-07)"
|
||||
contains: "metadata"
|
||||
key_links:
|
||||
- from: "lib/services/autotask-client.ts getAttachmentContent"
|
||||
to: "Autotask Tickets/{id}/Attachments/{attachmentId}"
|
||||
via: "makeApiCall GET reading response.items?.[0]"
|
||||
pattern: "items\\?\\.\\[0\\]"
|
||||
- from: "lib/services/b2/client.ts presignUpload"
|
||||
to: "EML_OBJECT_KEY_REGEX"
|
||||
via: "optional keyRegex parameter"
|
||||
pattern: "EML_OBJECT_KEY_REGEX"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Build the three independent supporting pieces the Plan 03 orchestrator needs:
|
||||
(1) a new AutotaskClient method that actually fetches full `.eml` base64 content,
|
||||
(2) a separate B2 object-key regex + parameterized validation so raw `.eml` bytes
|
||||
can be stored per D-05 without loosening LogLift's path-traversal guard, and
|
||||
(3) the migration 099 `indicators.metadata` JSONB column (D-07).
|
||||
|
||||
Purpose: These are the load-bearing gotchas the research verified empirically —
|
||||
the per-attachment-ID GET returns `{items:[...]}` (not `{item:...}`), and the B2
|
||||
skill doc forbids loosening the existing regex. Getting them wrong silently
|
||||
returns undefined content or weakens a security guard.
|
||||
Output: modified autotask-client.ts (+ new test), modified b2/client.ts
|
||||
(+ extended test), new migration 099 applied to the dev DB.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/16-eml-mime-evidence-parser/16-RESEARCH.md
|
||||
@.planning/phases/16-eml-mime-evidence-parser/16-PATTERNS.md
|
||||
@lib/services/autotask-client.ts
|
||||
@lib/services/b2/client.ts
|
||||
@lib/services/b2/client.test.ts
|
||||
@migrations/097_phishing_triage_schema.sql
|
||||
@migrations/083_add_user_timezone.sql
|
||||
@.claude/skills/pulse-overshell-b2-evidence/SKILL.md
|
||||
|
||||
<interfaces>
|
||||
Existing (lib/types/autotask.ts):
|
||||
- interface Attachment { id:number; attachmentType; fullPath:string; title:string; publish; data?:string; contentType?:string; ... }
|
||||
- interface ApiResponse<T> { item?: T; items?: T[]; pageDetails?: {...} }
|
||||
|
||||
Existing sibling (lib/services/autotask-client.ts ~line 424) — getAttachments reads `response.items || []` (list convention). uploadAttachment (~line 417) reads `response.item` (entity convention). The NEW method must follow getAttachments' `items` convention, NOT uploadAttachment's `item`.
|
||||
|
||||
Existing (lib/services/b2/client.ts):
|
||||
- export const OBJECT_KEY_REGEX = /^[A-Za-z0-9_-]+\/[A-Za-z0-9_.-]+\/eventlogs_[0-9_]+\.json\.gz$/ (LogLift — DO NOT modify)
|
||||
- export function presignDownload(objectKey, expiresInSeconds=600, cfg=getB2Config()): string
|
||||
- export function presignUpload(objectKey, expiresInSeconds=1800, cfg=getB2Config()): string
|
||||
- export async function downloadToBuffer(objectKey, cfg=getB2Config()): Promise<Buffer>
|
||||
- export class B2InvalidObjectKeyError
|
||||
- Existing call site: lib/services/rmm/executor.ts:337 → presignUpload(objectKey, 1800)
|
||||
|
||||
Existing (migrations/097_phishing_triage_schema.sql lines 100-108) — indicators table:
|
||||
id UUID PK, message_id UUID REFERENCES messages(id), indicator_type TEXT NOT NULL, value TEXT NOT NULL, created_at TIMESTAMPTZ. No metadata column yet.
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 1: Add AutotaskClient.getAttachmentContent() + first client unit test</name>
|
||||
<files>lib/services/autotask-client.ts, lib/services/autotask-client.test.ts</files>
|
||||
<read_first>
|
||||
- lib/services/autotask-client.ts lines 380-436 (uploadAttachment reads response.item; getAttachments reads response.items — the new method must mirror getAttachments)
|
||||
- .planning/phases/16-eml-mime-evidence-parser/16-RESEARCH.md (Pattern 1 + Pitfall 2: per-attachment-ID GET returns {items:[...]}, list call always returns data:null)
|
||||
- .planning/phases/16-eml-mime-evidence-parser/16-PATTERNS.md (autotask-client section: exact method body + the "critical gotcha the new method must NOT copy")
|
||||
- lib/services/b2/client.test.ts (vitest structure to mirror for the new test file)
|
||||
</read_first>
|
||||
<behavior>
|
||||
- getAttachmentContent('Tickets', ticketId, attachmentId) issues GET {apiUrl}/Tickets/{ticketId}/Attachments/{attachmentId} and returns response.items?.[0] ?? null
|
||||
- When the mocked API returns { items: [ { id, data: '<base64>' } ] }, the method returns that attachment object with data populated
|
||||
- When the mocked API returns { items: [] } or { item: {...} } (wrong shape), the method returns null (proving it reads .items, not .item)
|
||||
</behavior>
|
||||
<action>
|
||||
Add `getAttachmentContent(entityName: string, entityId: number, attachmentId: number): Promise<Attachment | null>` to AutotaskClient immediately after `getAttachments` (~line 436). Build the URL as `${this.config.apiUrl}/${entityName}/${entityId}/Attachments/${attachmentId}`, call `this.makeApiCall<ApiResponse<Attachment>>(url, { method: 'GET', headers: this.getAuthHeaders() })`, and return `response.items?.[0] ?? null`. Add a code comment noting the response is `{items:[...]}` (list-shaped) confirmed live — NOT `{item:...}` like getEntityById/uploadAttachment. Do not wrap in try/catch (makeApiCall already catches-and-rethrows, matching sibling methods). Create `lib/services/autotask-client.test.ts` (first coverage for this class): mock `global.fetch` (or the makeApiCall transport) to return the `{items:[...]}` envelope, instantiate a client with a fixture config, and assert getAttachmentContent returns items[0]; add a second test asserting that a response shaped `{item:{...}}` yields null (guards against a future refactor copying the uploadAttachment convention).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /opt/stacks/pulse && npx vitest run lib/services/autotask-client.test.ts && npx tsc --noEmit --pretty 2>&1 | (grep -i autotask-client && exit 1 || echo tsc-ok)</automated>
|
||||
</verify>
|
||||
<done>getAttachmentContent exists, reads response.items?.[0] ?? null; new test file passes both the items[0]-populated and item-shaped-returns-null cases; tsc clean.</done>
|
||||
<acceptance_criteria>
|
||||
- `grep -n "items?.\[0\]" lib/services/autotask-client.ts` matches inside getAttachmentContent
|
||||
- `npx vitest run lib/services/autotask-client.test.ts -t "getAttachmentContent"` passes
|
||||
- A test asserts a `{item:{...}}`-shaped response yields null (not the item)
|
||||
- No try/catch added around the makeApiCall invocation (matches sibling methods)
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: Add EML_OBJECT_KEY_REGEX + parameterize B2 key validation (D-05)</name>
|
||||
<files>lib/services/b2/client.ts, lib/services/b2/client.test.ts</files>
|
||||
<read_first>
|
||||
- lib/services/b2/client.ts lines 24-52 and 145-205 (OBJECT_KEY_REGEX, B2InvalidObjectKeyError, presignDownload/presignUpload/downloadToBuffer signatures)
|
||||
- lib/services/b2/client.test.ts (existing regex/presign test conventions to extend)
|
||||
- .claude/skills/pulse-overshell-b2-evidence/SKILL.md (the rule: add a new key regex + transport, never loosen the existing one)
|
||||
- .planning/phases/16-eml-mime-evidence-parser/16-PATTERNS.md (b2/client.ts section: exact parallel regex + parameterization approach)
|
||||
- lib/services/rmm/executor.ts line 337 (existing presignUpload(objectKey, 1800) call — must keep compiling unchanged)
|
||||
</read_first>
|
||||
<behavior>
|
||||
- EML_OBJECT_KEY_REGEX matches `phishing/<reportId>/<attachmentId>.eml` where each segment is [A-Za-z0-9_-]+
|
||||
- EML_OBJECT_KEY_REGEX rejects path traversal (`phishing/../x.eml`), wrong extension (`phishing/a/b.json`), and the LogLift shape
|
||||
- presignUpload(key, ttl, cfg, EML_OBJECT_KEY_REGEX) validates against the EML regex and throws B2InvalidObjectKeyError for a non-matching key
|
||||
- presignUpload(key, ttl) with no keyRegex still validates against OBJECT_KEY_REGEX (LogLift behavior unchanged) — existing rmm executor call keeps working
|
||||
</behavior>
|
||||
<action>
|
||||
In `lib/services/b2/client.ts`, add `export const EML_OBJECT_KEY_REGEX = /^phishing\/[A-Za-z0-9_-]+\/[A-Za-z0-9_-]+\.eml$/;` immediately after OBJECT_KEY_REGEX, with a doc comment referencing Phase 16 / D-05 and the skill-doc rule (never loosen OBJECT_KEY_REGEX). Add an optional trailing `keyRegex: RegExp = OBJECT_KEY_REGEX` parameter to `presignDownload`, `presignUpload`, and `downloadToBuffer`, and replace their hard-coded `OBJECT_KEY_REGEX.test(...)` checks with `keyRegex.test(...)`. Keep every existing parameter position and default intact so `presignUpload(objectKey, 1800)` (rmm/executor.ts:337) and all LogLift call sites compile and behave identically. Do NOT modify OBJECT_KEY_REGEX itself. Extend `lib/services/b2/client.test.ts` with a `describe('EML_OBJECT_KEY_REGEX', ...)` block asserting it matches a valid `phishing/<id>/<id>.eml` key and rejects traversal / wrong-extension / LogLift-shaped keys, plus a test that `presignUpload(validEmlKey, 1800, FIXTURE_CFG, EML_OBJECT_KEY_REGEX)` succeeds while `presignUpload(logliftKey, 1800, FIXTURE_CFG, EML_OBJECT_KEY_REGEX)` throws B2InvalidObjectKeyError.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /opt/stacks/pulse && npx vitest run lib/services/b2/client.test.ts && npx tsc --noEmit --pretty 2>&1 | (grep -iE 'b2/client|rmm/executor' && exit 1 || echo tsc-ok)</automated>
|
||||
</verify>
|
||||
<done>EML_OBJECT_KEY_REGEX exported and tested; presign*/downloadToBuffer take an optional keyRegex defaulting to OBJECT_KEY_REGEX; OBJECT_KEY_REGEX unchanged; b2 test suite passes; rmm/executor.ts still type-checks.</done>
|
||||
<acceptance_criteria>
|
||||
- `grep -n "EML_OBJECT_KEY_REGEX" lib/services/b2/client.ts` matches the new export
|
||||
- `git diff lib/services/b2/client.ts` shows OBJECT_KEY_REGEX line unchanged
|
||||
- A test asserts EML_OBJECT_KEY_REGEX rejects `phishing/../evil.eml`
|
||||
- A test asserts presignUpload throws B2InvalidObjectKeyError for a LogLift key when passed EML_OBJECT_KEY_REGEX
|
||||
- `npx tsc --noEmit` reports no errors for b2/client.ts or rmm/executor.ts (existing call site intact)
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3: Migration 099 — indicators.metadata JSONB (D-07) + apply to dev DB</name>
|
||||
<files>migrations/099_indicators_metadata.sql</files>
|
||||
<read_first>
|
||||
- migrations/083_add_user_timezone.sql (exact single-column-add migration shape: banner, phase/decision ref, purpose sentence, what-it-does-NOT-change sentence, ALTER ... ADD COLUMN IF NOT EXISTS, COMMENT ON COLUMN)
|
||||
- migrations/097_phishing_triage_schema.sql lines 100-108 (current indicators table this extends)
|
||||
- CLAUDE.md (migration caveat: Postgres init applies migrations on first volume boot only; existing dev DB needs manual apply via docker exec)
|
||||
</read_first>
|
||||
<action>
|
||||
Create `migrations/099_indicators_metadata.sql` following the migrations/083 doc-comment shape: a banner referencing Phase 16 / D-07, a one-sentence purpose ("lets an attachment-hash indicator carry filename/content-type/size, a URL indicator carry which message part it came from, without duplicating into the parent messages row"), and a one-sentence "does not change existing rows" note. The DDL is `ALTER TABLE indicators ADD COLUMN IF NOT EXISTS metadata JSONB;` — nullable, no NOT NULL, no default (per D-07's literal wording; no backfill needed since no rows exist yet). Add a `COMMENT ON COLUMN indicators.metadata IS '...'` line. Then apply it to the running dev database manually (the migration file is the source of truth for fresh installs, but the long-lived dev volume will not auto-run it) via `docker exec -i pulse-postgres psql -U <user> -d <db>` piping the file — confirm the exact container name and credentials from docker-compose.yml / .env before running.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /opt/stacks/pulse && grep -q "ADD COLUMN IF NOT EXISTS metadata JSONB" migrations/099_indicators_metadata.sql && docker exec pulse-postgres psql -U "${POSTGRES_USER:-pulse}" -d "${POSTGRES_DB:-pulse}" -tAc "SELECT data_type FROM information_schema.columns WHERE table_name='indicators' AND column_name='metadata'" | grep -q jsonb && echo "migration-applied"</automated>
|
||||
</verify>
|
||||
<done>migrations/099_indicators_metadata.sql exists with the ALTER + COMMENT following the 083 shape; the metadata JSONB column exists on the live dev indicators table.</done>
|
||||
<acceptance_criteria>
|
||||
- `migrations/099_indicators_metadata.sql` contains `ADD COLUMN IF NOT EXISTS metadata JSONB`
|
||||
- The file includes a banner referencing Phase 16 / D-07 and a COMMENT ON COLUMN statement
|
||||
- `information_schema.columns` reports `indicators.metadata` with data_type `jsonb` on the dev DB
|
||||
- Migration number is 099 (next after 098) and no existing migration file was edited
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| Autotask REST API → AutotaskClient | Fetched base64 attachment content originates from an attacker-controlled reported email |
|
||||
| Pulse server → B2 object store | Object keys constructed for raw `.eml` uploads must not enable path traversal |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|-------------|-----------------|
|
||||
| T-16-04 | Tampering / EoP | B2 object-key construction for raw `.eml` | mitigate | New EML_OBJECT_KEY_REGEX enforces `phishing/<id>/<id>.eml` shape and rejects `../` traversal; LogLift's OBJECT_KEY_REGEX is left untouched (skill-doc rule) — the guard is added in parallel, not by loosening the existing one |
|
||||
| T-16-05 | Information Disclosure | getAttachmentContent returning wrong/empty data | mitigate | Method reads response.items?.[0] (verified live shape); a unit test asserts a `{item:...}`-shaped response yields null so a silent-undefined regression can't ship |
|
||||
| T-16-06 | Tampering | migration 099 DDL on shared dev DB | accept | Additive `ADD COLUMN IF NOT EXISTS metadata JSONB` only — nullable, no data rewrite, idempotent; no destructive risk |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
- `npx vitest run lib/services/autotask-client.test.ts lib/services/b2/client.test.ts` — green
|
||||
- `npx tsc --noEmit --pretty` — no new errors (rmm/executor.ts call site still compiles)
|
||||
- `information_schema` confirms indicators.metadata jsonb on the dev DB
|
||||
- OBJECT_KEY_REGEX unchanged in the diff
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- EVID-03 support: getAttachmentContent fetches full base64 `.eml` content via the items[0] convention
|
||||
- EVID-04 / D-05 support: raw `.eml` bytes can be stored under a dedicated, path-traversal-safe B2 key shape distinct from the persisted preview
|
||||
- D-07: indicators.metadata JSONB column exists for per-indicator context
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/16-eml-mime-evidence-parser/16-02-SUMMARY.md` when done
|
||||
</output>
|
||||
215
.planning/phases/16-eml-mime-evidence-parser/16-03-PLAN.md
Normal file
215
.planning/phases/16-eml-mime-evidence-parser/16-03-PLAN.md
Normal file
|
|
@ -0,0 +1,215 @@
|
|||
---
|
||||
phase: 16-eml-mime-evidence-parser
|
||||
plan: 03
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on: [16-01, 16-02]
|
||||
files_modified:
|
||||
- lib/services/phishing-eml-service.ts
|
||||
- lib/services/phishing-eml-service.test.ts
|
||||
autonomous: true
|
||||
requirements: [EVID-03, EVID-04]
|
||||
user_setup: []
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "Given a report + ticket id, the service lists attachments, selects the original message, fetches its full content, parses it, and writes one messages row linked by report_id"
|
||||
- "The parsed structured auth verdicts (D-06), normalized headers, URLs, and attachment metadata are persisted into messages.headers/urls/attachments/body_preview"
|
||||
- "When B2 is configured, raw .eml bytes are PUT to B2 and messages.raw_ref stores the B2 object key (D-05); when B2 is unconfigured the report's evidence still parses and persists without throwing"
|
||||
- "One indicators row is written per attachment-hash / URL / sender, each carrying its context in the metadata JSONB column (D-07)"
|
||||
- "No outbound network call is made to any URL found in the message — end to end"
|
||||
- "When no .eml attachment is found, the service returns a no-op result without throwing"
|
||||
artifacts:
|
||||
- path: "lib/services/phishing-eml-service.ts"
|
||||
provides: "parseAndStoreMessage orchestration: list→select→fetch→size-guard→B2 upload→parse→persist messages/indicators"
|
||||
min_lines: 90
|
||||
- path: "lib/services/phishing-eml-service.test.ts"
|
||||
provides: "orchestration coverage with mocked autotask/b2/postgres incl. no-network + B2-gated + no-eml no-op"
|
||||
min_lines: 70
|
||||
key_links:
|
||||
- from: "lib/services/phishing-eml-service.ts"
|
||||
to: "lib/services/eml-parser.ts"
|
||||
via: "parseEml + selectOriginalMessage imports"
|
||||
pattern: "from './eml-parser'"
|
||||
- from: "lib/services/phishing-eml-service.ts"
|
||||
to: "AutotaskClient.getAttachmentContent"
|
||||
via: "getAutotaskClient().getAttachmentContent(...)"
|
||||
pattern: "getAttachmentContent"
|
||||
- from: "lib/services/phishing-eml-service.ts"
|
||||
to: "messages / indicators tables"
|
||||
via: "postgresClient INSERT ... RETURNING id::text"
|
||||
pattern: "INSERT INTO messages"
|
||||
- from: "lib/services/phishing-eml-service.ts"
|
||||
to: "B2 (presignUpload + EML_OBJECT_KEY_REGEX)"
|
||||
via: "isB2Configured gate then self-PUT"
|
||||
pattern: "isB2Configured"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Build the orchestration service that turns a detected phishing report into
|
||||
persisted, normalized message evidence: list a ticket's attachments, select the
|
||||
original reported message (Plan 01), fetch its full base64 content (Plan 02),
|
||||
size-guard it, store the raw bytes in B2 under the dedicated `.eml` key (D-05,
|
||||
Plan 02 regex), parse it (Plan 01), and persist a `messages` row plus
|
||||
`indicators` rows (with the D-07 metadata JSONB) — never fetching or executing
|
||||
anything found in the message.
|
||||
|
||||
Purpose: This is the consumer that makes D-05 (B2 raw storage), D-06 (structured
|
||||
verdicts persisted), and D-07 (indicators.metadata) real. It is callable and
|
||||
fully tested here; the live on-demand trigger (`POST /api/phishing/tickets/{id}/analyze`)
|
||||
arrives in Phase 18 (DETECT-03).
|
||||
Output: `lib/services/phishing-eml-service.ts` + `lib/services/phishing-eml-service.test.ts`.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/16-eml-mime-evidence-parser/16-RESEARCH.md
|
||||
@.planning/phases/16-eml-mime-evidence-parser/16-PATTERNS.md
|
||||
@lib/services/phishing-detector.ts
|
||||
@lib/services/eml-parser.ts
|
||||
@lib/services/b2/client.ts
|
||||
@migrations/097_phishing_triage_schema.sql
|
||||
|
||||
<interfaces>
|
||||
Consumes from Plan 01 (lib/services/eml-parser.ts):
|
||||
- selectOriginalMessage(attachments: Attachment[]): Attachment | null
|
||||
- async parseEml(rawEmlBuffer: Buffer): Promise<NormalizedMessage>
|
||||
- const MAX_EML_BYTES: number
|
||||
- NormalizedMessage { from{displayName,email,domain}, replyTo, returnPath, to[], cc[], subject, date, messageId, receivedChain[], authResults{spf,dkim,dmarc}, authResultsOriginal, urls[], attachments[{filename,contentType,size,checksum,related}], bodyPreview }
|
||||
|
||||
Consumes from Plan 02:
|
||||
- AutotaskClient.getAttachmentContent(entityName, entityId, attachmentId): Promise<Attachment | null> (returns attachment with base64 `data`)
|
||||
- b2/client.ts: presignUpload(objectKey, ttl, cfg, keyRegex), isB2Configured(), EML_OBJECT_KEY_REGEX
|
||||
- migrations/099: indicators.metadata JSONB column
|
||||
|
||||
Existing (lib/services/autotask-client.ts): getAttachments('Tickets', ticketId): Promise<Attachment[]> (full list WITH id; note: reports.evidence only stores fullPath/title/contentType, NOT id — so this service must list live to get attachment ids)
|
||||
|
||||
Existing (lib/services/autotask-factory.ts): getAutotaskClient()
|
||||
Existing (lib/services/postgres-client.ts): postgresClient.query<T>(sql, params)
|
||||
|
||||
Target schema (migrations/097 + 099):
|
||||
- messages( id UUID PK, report_id UUID, message_id TEXT, headers JSONB, urls JSONB, attachments JSONB, body_preview TEXT, raw_ref TEXT, created_at )
|
||||
- indicators( id UUID PK, message_id UUID, indicator_type TEXT, value TEXT, metadata JSONB, created_at )
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 1: phishing-eml-service.ts orchestration (list→select→fetch→B2→parse→persist)</name>
|
||||
<files>lib/services/phishing-eml-service.ts</files>
|
||||
<read_first>
|
||||
- lib/services/phishing-detector.ts lines 112-248 (gatherTicketEvidence fetch-then-transform pattern, detectPhishingTicket top-level try/catch + [TAG] console.error + rethrow, ON CONFLICT ... RETURNING id::text upsert shape)
|
||||
- lib/services/eml-parser.ts (the exact exported signatures + NormalizedMessage shape produced in Plan 01)
|
||||
- lib/services/b2/client.ts (isB2Configured, presignUpload signature incl. new keyRegex param, EML_OBJECT_KEY_REGEX, MAX_DOWNLOAD_BYTES)
|
||||
- .planning/phases/16-eml-mime-evidence-parser/16-RESEARCH.md (Pattern 4 self-PUT to B2, Pitfall 3 B2 unconfigured in dev, Open Questions 2-3 inline-attachment + size handling)
|
||||
- .planning/phases/16-eml-mime-evidence-parser/16-PATTERNS.md (phishing-eml-service.ts section: imports, graceful-degrade try/catch, INSERT ... RETURNING id::text convention, camelCase→snake_case boundary)
|
||||
- migrations/097_phishing_triage_schema.sql lines 77-108 (exact messages + indicators column lists)
|
||||
</read_first>
|
||||
<behavior>
|
||||
- parseAndStoreMessage({ reportId, ticketId }) lists attachments via getAutotaskClient().getAttachments('Tickets', ticketId), calls selectOriginalMessage on the full list (needs live list for attachment ids — reports.evidence lacks them)
|
||||
- If selectOriginalMessage returns null → returns a no-op result (e.g. { stored: false, reason: 'no-eml-attachment' }) without throwing and without writing any row
|
||||
- Otherwise fetches full content via getAttachmentContent('Tickets', ticketId, selected.id), base64-decodes `data` into a Buffer
|
||||
- Rejects/flags the report (no throw that crashes the caller) if the decoded buffer exceeds MAX_EML_BYTES — size guard before parseEml
|
||||
- When isB2Configured(): builds objectKey `phishing/{reportId}/{attachmentId}.eml`, presignUpload(objectKey, 1800, undefined, EML_OBJECT_KEY_REGEX), PUTs the raw buffer, sets rawRef = objectKey; when NOT configured: logs + skips B2, rawRef stays null, parsing/persisting still proceeds
|
||||
- Parses via parseEml, writes ONE messages row (report_id, message_id, headers JSONB = full normalized header block incl. authResults/authResultsOriginal per D-06, urls JSONB, attachments JSONB, body_preview, raw_ref) returning its id
|
||||
- Writes indicators rows: one per attachment checksum (indicator_type 'attachment_hash', value=checksum, metadata={filename,contentType,size,related}), one per URL (indicator_type 'url', value=url, metadata={part}), one for the sender (indicator_type 'sender', value=from.email, metadata={displayName,domain}) — each metadata written into the D-07 JSONB column
|
||||
- Never fetches any extracted URL (no fetch of message content beyond the Autotask attachment + B2 PUT)
|
||||
</behavior>
|
||||
<action>
|
||||
Create `lib/services/phishing-eml-service.ts` as a named-function module (no class) with a top doc-comment stating provenance and the never-fetch invariant. Imports: postgresClient from './postgres-client', getAutotaskClient from './autotask-factory', { presignUpload, isB2Configured, EML_OBJECT_KEY_REGEX } from './b2/client', { parseEml, selectOriginalMessage, MAX_EML_BYTES } from './eml-parser'. Export `async function parseAndStoreMessage(input: { reportId: string; ticketId: number }): Promise<{ stored: boolean; messageId?: string; reason?: string }>`. Implement the flow in the behavior block. Follow phishing-detector.ts conventions exactly: graceful-degrade try/catch around the Autotask list + getAttachmentContent + B2 PUT (log with `[PHISHING-EML]` tag, context values after the message string), the `INSERT INTO ... RETURNING id::text AS id` shape for both the messages insert and each indicators insert, and manual camelCase(TS)→snake_case(SQL) mapping at the query-building step (no ORM). Persist the full NormalizedMessage header block (including authResults and authResultsOriginal, D-06) into messages.headers JSONB via `$n::jsonb`. Gate the B2 PUT behind isB2Configured() (Pitfall 3 — dev has no B2 creds); on B2 PUT failure, log and continue with rawRef=null rather than aborting the whole parse. Guard buffer size against MAX_EML_BYTES before parseEml. Wrap the whole body in a top-level try/catch that logs `[PHISHING-EML]` + rethrows so a future caller (Phase 18) decides fail-vs-degrade. Decide inline/`related` attachment handling deliberately (Open Question 2): persist them but keep the `related` flag in indicator metadata so Phase 19 can weight them.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /opt/stacks/pulse && npx tsc --noEmit --pretty 2>&1 | (grep -i phishing-eml && exit 1 || echo tsc-ok)</automated>
|
||||
</verify>
|
||||
<done>phishing-eml-service.ts exports parseAndStoreMessage wiring list→select→fetch→(B2 gated)→parseEml→messages+indicators persistence, with graceful degrade, size guard, [PHISHING-EML] logging, and RETURNING id::text; tsc clean.</done>
|
||||
<acceptance_criteria>
|
||||
- `grep -n "selectOriginalMessage\|parseEml\|getAttachmentContent\|isB2Configured\|EML_OBJECT_KEY_REGEX" lib/services/phishing-eml-service.ts` matches all five
|
||||
- `grep -n "INSERT INTO messages" lib/services/phishing-eml-service.ts` and `INSERT INTO indicators` both present, each with `RETURNING id::text`
|
||||
- messages.headers is written from the NormalizedMessage header block including authResults (D-06) — grep for `authResults` reaching the persisted payload
|
||||
- indicators inserts write the metadata JSONB column (D-07) — grep for `metadata` in the indicators INSERT column list
|
||||
- B2 PUT is inside an `isB2Configured()` guard; a failed/absent B2 path does not abort persistence
|
||||
- No `fetch(` call targets a URL derived from parsed message content (only the Autotask attachment fetch + the B2 presigned PUT)
|
||||
- `npx tsc --noEmit` reports no errors for phishing-eml-service.ts
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: phishing-eml-service.test.ts — orchestration coverage (mocked I/O)</name>
|
||||
<files>lib/services/phishing-eml-service.test.ts</files>
|
||||
<read_first>
|
||||
- lib/services/phishing-eml-service.ts (the module under test, from Task 1)
|
||||
- lib/services/phishing-detector.test.ts (mocking style for postgresClient + autotask factory in this repo)
|
||||
- lib/services/eml-parser.fixtures.ts (reuse the synthetic .eml buffers + attachment lists created in Plan 01)
|
||||
- .planning/phases/16-eml-mime-evidence-parser/16-RESEARCH.md (Pitfall 3 — tests must mock B2, never require real creds; SC#5 synthetic-only)
|
||||
</read_first>
|
||||
<behavior>
|
||||
- Happy path: mock getAutotaskClient to return a 3-tier fixture attachment list + getAttachmentContent returning a synthetic base64 `.eml`; mock postgresClient.query to capture inserts; assert exactly one messages insert and the expected indicators inserts, and that messages.headers payload contains the structured spf/dkim/dmarc verdicts
|
||||
- No-eml no-op: attachment list with no `.eml` → parseAndStoreMessage returns { stored: false } and postgresClient.query is never called with an INSERT INTO messages
|
||||
- B2 unconfigured: with isB2Configured()===false (env unset / mocked), the flow still parses and persists, raw_ref persisted as null, no presignUpload/PUT attempted
|
||||
- B2 configured: with isB2Configured()===true (mocked) and presignUpload/fetch mocked, raw_ref is set to `phishing/{reportId}/{attachmentId}.eml` and the PUT is issued once
|
||||
- No-network invariant: a global.fetch spy records no call to any URL contained in the fixture message body (only the mocked B2 PUT URL, if configured)
|
||||
- metadata JSONB: assert an attachment_hash indicator insert carries {filename,contentType,size,related} in its metadata param
|
||||
</behavior>
|
||||
<action>
|
||||
Create `lib/services/phishing-eml-service.test.ts` mirroring phishing-detector.test.ts's mocking approach (vi.mock the postgres-client and autotask-factory modules; vi.spyOn/stub isB2Configured + global.fetch as needed). Reuse the synthetic fixtures from eml-parser.fixtures.ts (import them) — do NOT introduce real email content. Cover every case in the behavior list. For the no-network assertion, spy on global.fetch and assert it is never called with any URL string that appears in the fixture body's extracted URLs (assert against the specific fixture URL, not just call count, since a configured-B2 test legitimately PUTs to the presigned B2 host). Use `describe('parseAndStoreMessage', ...)` with plain-language `it` titles matching the requirement wording.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /opt/stacks/pulse && npx vitest run lib/services/phishing-eml-service.test.ts && npx tsc --noEmit --pretty 2>&1 | (grep -i phishing-eml && exit 1 || echo tsc-ok)</automated>
|
||||
</verify>
|
||||
<done>phishing-eml-service.test.ts passes all cases: happy-path single messages row + indicators with metadata, no-eml no-op, B2-configured raw_ref set + single PUT, B2-unconfigured raw_ref null + no PUT, and the no-network-to-message-URLs invariant.</done>
|
||||
<acceptance_criteria>
|
||||
- `npx vitest run lib/services/phishing-eml-service.test.ts` exits 0
|
||||
- A test asserts no INSERT INTO messages occurs on the no-`.eml` path (stored:false)
|
||||
- A test asserts raw_ref === null when isB2Configured() is false, and no presignUpload/PUT happens
|
||||
- A test asserts raw_ref === `phishing/{reportId}/{attachmentId}.eml` and exactly one PUT when B2 is configured (mocked)
|
||||
- A test asserts global.fetch is never called with a message-body URL from the fixture
|
||||
- A test asserts the attachment_hash indicator's metadata JSONB carries filename/contentType/size/related
|
||||
- All fixtures are synthetic (imported from eml-parser.fixtures.ts) — no real customer email
|
||||
</acceptance_criteria>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| Autotask attachment content → Pulse server | Attacker-controlled `.eml` bytes fetched and processed server-side |
|
||||
| Message body content → Pulse outbound network | URLs/HTML in the reported email must never be dereferenced by Pulse |
|
||||
| Raw `.eml` bytes → storage | Potentially malware-laced raw email must not land somewhere browser-reachable |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|-------------|-----------------|
|
||||
| T-16-03 | Tampering / Info Disclosure (SSRF-adjacent) | orchestration around parseEml/extractUrls | mitigate | Hard invariant, test-enforced: the service never fetches any URL found in the message; the only outbound calls are the Autotask attachment GET and the B2 presigned PUT. A fetch spy asserts no call to any message-body URL |
|
||||
| T-16-01 | Denial of Service | oversized fetched attachment | mitigate | Size guard against MAX_EML_BYTES on the decoded buffer BEFORE parseEml; degrade (log + skip) rather than crash |
|
||||
| T-16-07 | Information Disclosure / EoP | raw `.eml` byte storage | mitigate | D-05 — raw bytes go to B2 (never Postgres, never local fs) under the path-traversal-safe EML_OBJECT_KEY_REGEX key; only the sanitized truncated preview lands in messages.body_preview |
|
||||
| T-16-08 | Repudiation | parse/persist failures swallowed silently | accept | Top-level try/catch logs `[PHISHING-EML]` with ticket/report context and rethrows so the future Phase 18 caller records the failure; graceful B2 degrade is logged, not silent |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
- `npx vitest run lib/services/phishing-eml-service.test.ts` — green
|
||||
- `npx tsc --noEmit --pretty` — no new errors
|
||||
- Grep confirms messages + indicators inserts, D-06 verdicts in headers payload, D-07 metadata in indicators, isB2Configured gate, and no fetch of message-body URLs
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- EVID-03: full pipeline persists normalized headers (incl. structured auth verdicts, D-06), URLs, and attachment metadata into a messages row + indicators rows
|
||||
- EVID-04 / D-05: raw bytes stored in B2 (raw_ref = object key) when configured; sanitized truncated preview stored distinct from raw; nothing in the message is ever fetched/executed
|
||||
- D-07: indicators.metadata JSONB carries per-indicator context
|
||||
- Graceful degradation when B2 is unconfigured or no `.eml` attachment exists
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/16-eml-mime-evidence-parser/16-03-SUMMARY.md` when done
|
||||
</output>
|
||||
Loading…
Add table
Add a link
Reference in a new issue