docs(16): capture phase context
This commit is contained in:
parent
d7b8c6b72d
commit
4857a35629
2 changed files with 208 additions and 0 deletions
150
.planning/phases/16-eml-mime-evidence-parser/16-CONTEXT.md
Normal file
150
.planning/phases/16-eml-mime-evidence-parser/16-CONTEXT.md
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
# Phase 16: EML/MIME Evidence Parser - Context
|
||||
|
||||
**Gathered:** 2026-07-15
|
||||
**Status:** Ready for planning
|
||||
|
||||
<domain>
|
||||
## Phase Boundary
|
||||
|
||||
Given a ticket's attachments, Pulse selects the correct original reported message
|
||||
(`rfc.eml` preferred over `OriginatingEmail.eml`) and parses its RFC822/MIME
|
||||
structure into normalized, actionable evidence (headers, auth results, URLs,
|
||||
attachment metadata, sanitized body preview) — without ever executing or
|
||||
fetching anything from the message. Covers EVID-02, EVID-03, EVID-04.
|
||||
Does NOT cover Mimecast (Phase 17), campaign grouping or the `/api/phishing/*`
|
||||
API surface (Phase 18), classification (Phase 19), remediation (Phase 20), or
|
||||
the Autotask note (Phase 21).
|
||||
|
||||
</domain>
|
||||
|
||||
<decisions>
|
||||
## Implementation Decisions
|
||||
|
||||
### Raw .eml Storage
|
||||
- **D-05:** Raw `.eml` bytes are stored in Backblaze B2, reusing the existing
|
||||
LogLift evidence-storage pattern (`lib/services/b2/client.ts`). The
|
||||
`messages.raw_ref` column (already in migration 097's stub schema) stores
|
||||
the B2 object key, not raw bytes or a bare hash. Rationale: keeps
|
||||
potentially-malicious raw email bytes out of Postgres, matches the
|
||||
established precedent for evidence blobs in this codebase, and gives a
|
||||
natural place to enforce the same size/path-traversal guards B2 client
|
||||
already has (`lib/services/b2/client.ts`'s object-key validation).
|
||||
|
||||
### Authentication-Results Depth
|
||||
- **D-06:** SPF/DKIM/DMARC are parsed into structured verdicts (pass/fail/
|
||||
none/etc — not just raw header text) from the `Authentication-Results`
|
||||
header (and `Received-SPF` as fallback where present). This is deliberately
|
||||
more than "capture the raw header" because Phase 19's classifier
|
||||
(CLASSIFY-01..06) needs these as structured evidence to reason about
|
||||
spoofing/impersonation, not raw text it would have to re-parse itself.
|
||||
|
||||
### Indicators Schema
|
||||
- **D-07:** Add a `metadata` JSONB column to the `indicators` table stub
|
||||
(migration 097) via a small Phase 16 migration (`ALTER TABLE indicators ADD
|
||||
COLUMN metadata JSONB`). This lets an attachment-hash indicator carry its
|
||||
filename/content-type/size alongside the hash, a URL indicator carry which
|
||||
message part (text/html) it came from, etc., without duplicating that
|
||||
detail into the parent `messages` row.
|
||||
|
||||
### Claude's Discretion (explicitly deferred to research + planner)
|
||||
- **MIME parsing library or approach.** No precedent exists in this codebase
|
||||
(no `mailparser`/similar dependency, no hand-rolled parser). This is a
|
||||
genuine technical unknown — resolve via research before planning, not a
|
||||
user preference call.
|
||||
- **How to actually fetch full `.eml` attachment content from Autotask.**
|
||||
`AutotaskClient.getAttachments()` (existing, from Phase 15) returns
|
||||
attachment *metadata* only — `AUTOTASK_API_GUIDE.md` documents attachment
|
||||
*upload* but not attachment *download*, and the `Attachment.data` field is
|
||||
typed optional with no confirmed behavior for whether the list call or a
|
||||
per-ID call actually populates base64 content. This needs to be confirmed
|
||||
empirically (real Autotask credentials exist in `.env`) or via Autotask API
|
||||
docs before planning locks in an approach.
|
||||
- **URL extraction scope** (dedup strategy, normalization, which MIME parts to
|
||||
scan) — implementation detail, not user-relevant preference.
|
||||
- Exact migration file number for the `indicators` ALTER (next available
|
||||
after 098 — confirm at plan time).
|
||||
|
||||
</decisions>
|
||||
|
||||
<canonical_refs>
|
||||
## Canonical References
|
||||
|
||||
**Downstream agents MUST read these before planning or implementing.**
|
||||
|
||||
### Project conventions
|
||||
- `CLAUDE.md` — migration numbering, no-new-dependency caution (npm deps are
|
||||
fine when a real gap exists — no ORM/server-actions/state-lib restriction
|
||||
applies to a MIME parser), auth conventions
|
||||
- `AUTOTASK_API_GUIDE.md` — documents attachment *upload* only; attachment
|
||||
*download*/full-content-fetch mechanics are NOT documented here and must be
|
||||
confirmed via research or empirical testing against the real API
|
||||
- `migrations/097_phishing_triage_schema.sql` — the `messages` and
|
||||
`indicators` stub tables this phase populates for real (`messages.headers`,
|
||||
`messages.urls`, `messages.attachments`, `messages.body_preview`,
|
||||
`messages.raw_ref`; `indicators.indicator_type` + `indicators.value`, plus
|
||||
the new `metadata` column per D-07)
|
||||
- `INTEGRATIONS.md` — confirms `lib/services/b2/client.ts` is the existing B2
|
||||
evidence-storage client (LogLift precedent), with object-key format
|
||||
validation and a 25 MB max download already established
|
||||
|
||||
### Reference implementations for this phase
|
||||
- `lib/services/b2/client.ts` — B2 upload/download pattern to reuse for raw
|
||||
`.eml` storage per D-05 (presigned URLs, object-key validation)
|
||||
- `lib/services/phishing-detector.ts` (Phase 15) — `DetectableTicket`,
|
||||
`EvidencePayload` shapes and the `gatherTicketEvidence` function this
|
||||
phase's parser output plugs into (via the `reports` → `messages` link)
|
||||
- `lib/services/autotask-client.ts` `getAttachments()` (~line 424) — existing
|
||||
attachment metadata fetch; this phase needs to extend or add to this for
|
||||
full content retrieval
|
||||
|
||||
</canonical_refs>
|
||||
|
||||
<code_context>
|
||||
## Existing Code Insights
|
||||
|
||||
### Reusable Assets
|
||||
- `lib/services/b2/client.ts` — presigned URL upload/download, object-key
|
||||
path-traversal guard, 25 MB max download — directly reusable for raw `.eml`
|
||||
storage per D-05
|
||||
- `migrations/097_phishing_triage_schema.sql`'s `messages`/`indicators` stub
|
||||
tables already shaped for this phase's output (see canonical_refs)
|
||||
|
||||
### Established Patterns
|
||||
- No existing MIME/RFC822 parsing code or dependency anywhere in this
|
||||
codebase — this phase introduces a genuinely new capability, unlike Phase
|
||||
15 which reused several existing patterns
|
||||
- B2 is the established "large/sensitive blob evidence" storage location in
|
||||
this codebase (LogLift), not Postgres bytea and not local filesystem
|
||||
|
||||
### Integration Points
|
||||
- This phase reads `reports.evidence` (Phase 15's attachment metadata) to
|
||||
find which ticket attachments exist, selects the right one (`rfc.eml` over
|
||||
`OriginatingEmail.eml`), fetches its full content, parses it, and writes a
|
||||
`messages` row (+ `indicators` rows) linked via `messages.report_id`
|
||||
|
||||
</code_context>
|
||||
|
||||
<specifics>
|
||||
## Specific Ideas
|
||||
|
||||
The exact `.eml` selection logic is locked via REQUIREMENTS.md EVID-02:
|
||||
prefer `rfc.eml` (case-insensitive name match) over `OriginatingEmail.eml`,
|
||||
also matching by `message/rfc822` content-type when filename alone is
|
||||
ambiguous.
|
||||
|
||||
</specifics>
|
||||
|
||||
<deferred>
|
||||
## Deferred Ideas
|
||||
|
||||
None — discussion stayed within phase scope. MIME library choice and
|
||||
Autotask attachment-download mechanics were explicitly routed to research
|
||||
rather than deferred to a future phase (they're needed now, just not
|
||||
user-decidable).
|
||||
|
||||
</deferred>
|
||||
|
||||
---
|
||||
|
||||
*Phase: 16-eml-mime-evidence-parser*
|
||||
*Context gathered: 2026-07-15*
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
# Phase 16: EML/MIME Evidence Parser - Discussion Log
|
||||
|
||||
> **Audit trail only.** Do not use as input to planning, research, or execution agents.
|
||||
> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
|
||||
|
||||
**Date:** 2026-07-15
|
||||
**Phase:** 16-eml-mime-evidence-parser
|
||||
**Areas discussed:** Raw .eml storage, Authentication-results depth, Indicators schema
|
||||
|
||||
---
|
||||
|
||||
## Raw .eml Storage
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| Backblaze B2 | Reuse the LogLift evidence-storage pattern; raw_ref stores the B2 object key | ✓ |
|
||||
| Hash only, no raw storage | raw_ref stores a content hash for dedup/audit; raw bytes never persisted | |
|
||||
| Postgres bytea column | Store raw bytes directly in a new column | |
|
||||
|
||||
**User's choice:** Backblaze B2 (Recommended)
|
||||
**Notes:** None beyond the recommendation.
|
||||
|
||||
---
|
||||
|
||||
## Authentication-Results Depth
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| Structured pass/fail/none extraction | Parse Authentication-Results into structured spf/dkim/dmarc verdicts | ✓ |
|
||||
| Raw header capture only | Store the raw header text as-is, no structured parsing | |
|
||||
|
||||
**User's choice:** Structured pass/fail/none extraction (Recommended)
|
||||
**Notes:** None beyond the recommendation.
|
||||
|
||||
---
|
||||
|
||||
## Indicators Schema
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| Add metadata JSONB column | ALTER TABLE indicators ADD COLUMN metadata JSONB | ✓ |
|
||||
| Keep type+value only | Put extra detail into the parent messages JSONB columns instead | |
|
||||
|
||||
**User's choice:** Add metadata JSONB column (Recommended)
|
||||
**Notes:** None beyond the recommendation.
|
||||
|
||||
---
|
||||
|
||||
## Claude's Discretion
|
||||
|
||||
- MIME parsing library/approach — explicitly routed to research (no codebase precedent)
|
||||
- How to fetch full .eml attachment content from Autotask — explicitly routed to research/empirical testing (undocumented in AUTOTASK_API_GUIDE.md)
|
||||
- URL extraction scope (dedup, normalization, which MIME parts to scan)
|
||||
- Exact migration number for the indicators ALTER
|
||||
|
||||
## Deferred Ideas
|
||||
|
||||
None — discussion stayed within phase scope.
|
||||
Loading…
Add table
Add a link
Reference in a new issue