diff --git a/.planning/phases/16-eml-mime-evidence-parser/16-CONTEXT.md b/.planning/phases/16-eml-mime-evidence-parser/16-CONTEXT.md new file mode 100644 index 0000000..ca4dfda --- /dev/null +++ b/.planning/phases/16-eml-mime-evidence-parser/16-CONTEXT.md @@ -0,0 +1,150 @@ +# Phase 16: EML/MIME Evidence Parser - Context + +**Gathered:** 2026-07-15 +**Status:** Ready for planning + + +## Phase Boundary + +Given a ticket's attachments, Pulse selects the correct original reported message +(`rfc.eml` preferred over `OriginatingEmail.eml`) and parses its RFC822/MIME +structure into normalized, actionable evidence (headers, auth results, URLs, +attachment metadata, sanitized body preview) — without ever executing or +fetching anything from the message. Covers EVID-02, EVID-03, EVID-04. +Does NOT cover Mimecast (Phase 17), campaign grouping or the `/api/phishing/*` +API surface (Phase 18), classification (Phase 19), remediation (Phase 20), or +the Autotask note (Phase 21). + + + + +## Implementation Decisions + +### Raw .eml Storage +- **D-05:** Raw `.eml` bytes are stored in Backblaze B2, reusing the existing + LogLift evidence-storage pattern (`lib/services/b2/client.ts`). The + `messages.raw_ref` column (already in migration 097's stub schema) stores + the B2 object key, not raw bytes or a bare hash. Rationale: keeps + potentially-malicious raw email bytes out of Postgres, matches the + established precedent for evidence blobs in this codebase, and gives a + natural place to enforce the same size/path-traversal guards B2 client + already has (`lib/services/b2/client.ts`'s object-key validation). + +### Authentication-Results Depth +- **D-06:** SPF/DKIM/DMARC are parsed into structured verdicts (pass/fail/ + none/etc — not just raw header text) from the `Authentication-Results` + header (and `Received-SPF` as fallback where present). This is deliberately + more than "capture the raw header" because Phase 19's classifier + (CLASSIFY-01..06) needs these as structured evidence to reason about + spoofing/impersonation, not raw text it would have to re-parse itself. + +### Indicators Schema +- **D-07:** Add a `metadata` JSONB column to the `indicators` table stub + (migration 097) via a small Phase 16 migration (`ALTER TABLE indicators ADD + COLUMN metadata JSONB`). This lets an attachment-hash indicator carry its + filename/content-type/size alongside the hash, a URL indicator carry which + message part (text/html) it came from, etc., without duplicating that + detail into the parent `messages` row. + +### Claude's Discretion (explicitly deferred to research + planner) +- **MIME parsing library or approach.** No precedent exists in this codebase + (no `mailparser`/similar dependency, no hand-rolled parser). This is a + genuine technical unknown — resolve via research before planning, not a + user preference call. +- **How to actually fetch full `.eml` attachment content from Autotask.** + `AutotaskClient.getAttachments()` (existing, from Phase 15) returns + attachment *metadata* only — `AUTOTASK_API_GUIDE.md` documents attachment + *upload* but not attachment *download*, and the `Attachment.data` field is + typed optional with no confirmed behavior for whether the list call or a + per-ID call actually populates base64 content. This needs to be confirmed + empirically (real Autotask credentials exist in `.env`) or via Autotask API + docs before planning locks in an approach. +- **URL extraction scope** (dedup strategy, normalization, which MIME parts to + scan) — implementation detail, not user-relevant preference. +- Exact migration file number for the `indicators` ALTER (next available + after 098 — confirm at plan time). + + + + +## Canonical References + +**Downstream agents MUST read these before planning or implementing.** + +### Project conventions +- `CLAUDE.md` — migration numbering, no-new-dependency caution (npm deps are + fine when a real gap exists — no ORM/server-actions/state-lib restriction + applies to a MIME parser), auth conventions +- `AUTOTASK_API_GUIDE.md` — documents attachment *upload* only; attachment + *download*/full-content-fetch mechanics are NOT documented here and must be + confirmed via research or empirical testing against the real API +- `migrations/097_phishing_triage_schema.sql` — the `messages` and + `indicators` stub tables this phase populates for real (`messages.headers`, + `messages.urls`, `messages.attachments`, `messages.body_preview`, + `messages.raw_ref`; `indicators.indicator_type` + `indicators.value`, plus + the new `metadata` column per D-07) +- `INTEGRATIONS.md` — confirms `lib/services/b2/client.ts` is the existing B2 + evidence-storage client (LogLift precedent), with object-key format + validation and a 25 MB max download already established + +### Reference implementations for this phase +- `lib/services/b2/client.ts` — B2 upload/download pattern to reuse for raw + `.eml` storage per D-05 (presigned URLs, object-key validation) +- `lib/services/phishing-detector.ts` (Phase 15) — `DetectableTicket`, + `EvidencePayload` shapes and the `gatherTicketEvidence` function this + phase's parser output plugs into (via the `reports` → `messages` link) +- `lib/services/autotask-client.ts` `getAttachments()` (~line 424) — existing + attachment metadata fetch; this phase needs to extend or add to this for + full content retrieval + + + + +## Existing Code Insights + +### Reusable Assets +- `lib/services/b2/client.ts` — presigned URL upload/download, object-key + path-traversal guard, 25 MB max download — directly reusable for raw `.eml` + storage per D-05 +- `migrations/097_phishing_triage_schema.sql`'s `messages`/`indicators` stub + tables already shaped for this phase's output (see canonical_refs) + +### Established Patterns +- No existing MIME/RFC822 parsing code or dependency anywhere in this + codebase — this phase introduces a genuinely new capability, unlike Phase + 15 which reused several existing patterns +- B2 is the established "large/sensitive blob evidence" storage location in + this codebase (LogLift), not Postgres bytea and not local filesystem + +### Integration Points +- This phase reads `reports.evidence` (Phase 15's attachment metadata) to + find which ticket attachments exist, selects the right one (`rfc.eml` over + `OriginatingEmail.eml`), fetches its full content, parses it, and writes a + `messages` row (+ `indicators` rows) linked via `messages.report_id` + + + + +## Specific Ideas + +The exact `.eml` selection logic is locked via REQUIREMENTS.md EVID-02: +prefer `rfc.eml` (case-insensitive name match) over `OriginatingEmail.eml`, +also matching by `message/rfc822` content-type when filename alone is +ambiguous. + + + + +## Deferred Ideas + +None — discussion stayed within phase scope. MIME library choice and +Autotask attachment-download mechanics were explicitly routed to research +rather than deferred to a future phase (they're needed now, just not +user-decidable). + + + +--- + +*Phase: 16-eml-mime-evidence-parser* +*Context gathered: 2026-07-15* diff --git a/.planning/phases/16-eml-mime-evidence-parser/16-DISCUSSION-LOG.md b/.planning/phases/16-eml-mime-evidence-parser/16-DISCUSSION-LOG.md new file mode 100644 index 0000000..246d725 --- /dev/null +++ b/.planning/phases/16-eml-mime-evidence-parser/16-DISCUSSION-LOG.md @@ -0,0 +1,58 @@ +# Phase 16: EML/MIME Evidence Parser - Discussion Log + +> **Audit trail only.** Do not use as input to planning, research, or execution agents. +> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered. + +**Date:** 2026-07-15 +**Phase:** 16-eml-mime-evidence-parser +**Areas discussed:** Raw .eml storage, Authentication-results depth, Indicators schema + +--- + +## Raw .eml Storage + +| Option | Description | Selected | +|--------|-------------|----------| +| Backblaze B2 | Reuse the LogLift evidence-storage pattern; raw_ref stores the B2 object key | ✓ | +| Hash only, no raw storage | raw_ref stores a content hash for dedup/audit; raw bytes never persisted | | +| Postgres bytea column | Store raw bytes directly in a new column | | + +**User's choice:** Backblaze B2 (Recommended) +**Notes:** None beyond the recommendation. + +--- + +## Authentication-Results Depth + +| Option | Description | Selected | +|--------|-------------|----------| +| Structured pass/fail/none extraction | Parse Authentication-Results into structured spf/dkim/dmarc verdicts | ✓ | +| Raw header capture only | Store the raw header text as-is, no structured parsing | | + +**User's choice:** Structured pass/fail/none extraction (Recommended) +**Notes:** None beyond the recommendation. + +--- + +## Indicators Schema + +| Option | Description | Selected | +|--------|-------------|----------| +| Add metadata JSONB column | ALTER TABLE indicators ADD COLUMN metadata JSONB | ✓ | +| Keep type+value only | Put extra detail into the parent messages JSONB columns instead | | + +**User's choice:** Add metadata JSONB column (Recommended) +**Notes:** None beyond the recommendation. + +--- + +## Claude's Discretion + +- MIME parsing library/approach — explicitly routed to research (no codebase precedent) +- How to fetch full .eml attachment content from Autotask — explicitly routed to research/empirical testing (undocumented in AUTOTASK_API_GUIDE.md) +- URL extraction scope (dedup, normalization, which MIME parts to scan) +- Exact migration number for the indicators ALTER + +## Deferred Ideas + +None — discussion stayed within phase scope.