chore: merge executor worktree (worktree-agent-adbfd43ed3e884d1d)
This commit is contained in:
commit
cad1a63e3d
3 changed files with 522 additions and 0 deletions
103
.planning/phases/16-eml-mime-evidence-parser/16-03-SUMMARY.md
Normal file
103
.planning/phases/16-eml-mime-evidence-parser/16-03-SUMMARY.md
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
---
|
||||
phase: 16-eml-mime-evidence-parser
|
||||
plan: 03
|
||||
subsystem: api
|
||||
tags: [autotask, backblaze-b2, postgres, mailparser, phishing, vitest]
|
||||
|
||||
# Dependency graph
|
||||
requires:
|
||||
- phase: 16-eml-mime-evidence-parser (Plan 01)
|
||||
provides: "lib/services/eml-parser.ts — selectOriginalMessage, parseEml, MAX_EML_BYTES, NormalizedMessage"
|
||||
- phase: 16-eml-mime-evidence-parser (Plan 02)
|
||||
provides: "AutotaskClient.getAttachmentContent, B2 EML_OBJECT_KEY_REGEX + parameterized presignUpload, migrations/099 indicators.metadata"
|
||||
provides:
|
||||
- "lib/services/phishing-eml-service.ts — parseAndStoreMessage(reportId, ticketId) orchestration: list -> select -> fetch -> (B2 gated) -> parse -> persist messages+indicators"
|
||||
affects: [18-campaign-grouping-api, 19-classification]
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "Orchestration service (no class, single exported async function) mirroring phishing-detector.ts's fetch->transform->persist shape"
|
||||
- "Graceful-degrade try/catch around the B2 self-PUT step, gated behind isB2Configured(), so an unconfigured/failed B2 upload never aborts parsing/persistence"
|
||||
- "Size guard on the decoded buffer BEFORE calling parseEml, returning a no-op result rather than letting parseEml's internal MAX_EML_BYTES guard throw"
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- lib/services/phishing-eml-service.ts
|
||||
- lib/services/phishing-eml-service.test.ts
|
||||
modified: []
|
||||
|
||||
key-decisions:
|
||||
- "Task ordering followed the plan literally: Task 1 built the implementation file, Task 2 built the full test suite against it (not a strict RED-then-GREEN cycle within a single task) — both tasks were tagged tdd=\"true\" in the plan but structured as separate files/commits rather than interleaved test-then-feat commits within one task, matching how the plan's own read_first/action/verify blocks were written per task"
|
||||
- "indicators inserts are plain INSERT (not upsert) — migration 097's indicators table has no natural-key unique constraint, and each parseAndStoreMessage call creates a fresh messages row, so there is nothing to conflict against"
|
||||
- "reason codes ('no-eml-attachment', 'no-attachment-content', 'oversized-attachment') added to the { stored: false } result beyond the plan's single named example, to make the three distinct no-op paths distinguishable to a future caller (Phase 18) without inspecting logs"
|
||||
|
||||
patterns-established:
|
||||
- "Self-PUT to B2 pattern (first instance of Pulse's own server code PUTting bytes to B2 itself, vs. handing a presigned URL to an external collector) — presignUpload(objectKey, 1800, undefined, EML_OBJECT_KEY_REGEX) then fetch(url, { method: 'PUT', body })"
|
||||
|
||||
requirements-completed: [EVID-03, EVID-04]
|
||||
|
||||
# Metrics
|
||||
duration: ~22min
|
||||
completed: 2026-07-15
|
||||
---
|
||||
|
||||
# Phase 16 Plan 03: EML/MIME Evidence Orchestration Service Summary
|
||||
|
||||
**`parseAndStoreMessage(reportId, ticketId)` orchestrates the full evidence pipeline — list Autotask attachments, select the original reported message, fetch its content, size-guard it, self-PUT the raw bytes to B2 when configured, parse it, and persist one `messages` row plus per-indicator `indicators` rows (attachment_hash/url/sender) with D-07 metadata — never fetching anything found in the message.**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** ~22 min
|
||||
- **Started:** 2026-07-15T14:18:00Z
|
||||
- **Completed:** 2026-07-15T14:40:23Z
|
||||
- **Tasks:** 2 completed
|
||||
- **Files modified:** 2 (both new)
|
||||
|
||||
## Accomplishments
|
||||
- `lib/services/phishing-eml-service.ts` exports `parseAndStoreMessage({ reportId, ticketId })`, wiring `getAutotaskClient().getAttachments` → Plan 01's `selectOriginalMessage` → `getAttachmentContent` → a `MAX_EML_BYTES` size guard → an `isB2Configured()`-gated B2 self-PUT (`phishing/{reportId}/{attachmentId}.eml`) → Plan 01's `parseEml` → one `messages` INSERT (headers incl. D-06 structured SPF/DKIM/DMARC verdicts, urls, attachments, body_preview, raw_ref) → per-attachment-hash/per-URL/sender `indicators` INSERTs carrying D-07 `metadata` JSONB.
|
||||
- Returns `{ stored: false, reason }` without throwing and without writing any row when there is no `.eml` attachment, no attachment content, or an oversized decoded buffer; a failed or absent B2 PUT degrades to `raw_ref: null` rather than aborting persistence.
|
||||
- 6/6 new vitest tests pass, covering the happy path (one messages row + D-06 verdicts reaching `headers`, indicators written), the no-`.eml` no-op, B2-unconfigured (`raw_ref` null, no PUT), B2-configured (`raw_ref` set, exactly one PUT), the no-network-to-message-body-URL invariant, and the attachment-hash indicator's `metadata` payload.
|
||||
- `npx tsc --noEmit --pretty` is fully clean (zero errors anywhere in the repo, not just this file).
|
||||
|
||||
## Task Commits
|
||||
|
||||
1. **Task 1: phishing-eml-service.ts orchestration (list→select→fetch→B2→parse→persist)** - `5088d8d` (feat)
|
||||
2. **Task 2: phishing-eml-service.test.ts — orchestration coverage (mocked I/O)** - `ce16e67` (test)
|
||||
|
||||
_Note: both tasks were tagged `tdd="true"` in the plan, but the plan itself structured them as implementation-first (Task 1) then full test coverage (Task 2) rather than an interleaved RED/GREEN pair within a single task — followed literally as written._
|
||||
|
||||
## Files Created/Modified
|
||||
- `lib/services/phishing-eml-service.ts` (224 lines) - `parseAndStoreMessage`, `ParseAndStoreInput`/`ParseAndStoreResult` types
|
||||
- `lib/services/phishing-eml-service.test.ts` (195 lines) - 6 tests: happy path, no-eml no-op, B2-unconfigured, B2-configured, no-network invariant, indicator metadata
|
||||
|
||||
## Decisions Made
|
||||
- **Plain `INSERT` for `indicators`, not upsert.** Migration 097's `indicators` table has no natural-key unique constraint (unlike `reports.uq_reports_ticket_id`), and each `parseAndStoreMessage` call always creates a fresh `messages` row first — there's nothing to conflict against, so `ON CONFLICT` would be dead code.
|
||||
- **Three distinct no-op `reason` codes** (`no-eml-attachment`, `no-attachment-content`, `oversized-attachment`) instead of one generic no-op shape — makes the three degrade paths distinguishable to Phase 18's future caller without grepping logs, while still satisfying the plan's single named example (`no-eml-attachment`).
|
||||
- **Task 1/Task 2 split followed literally**, not force-fit into a same-task RED→GREEN commit pair, since the plan's own per-task `read_first`/`action`/`verify` blocks were already split that way (Task 1's verify is `tsc` only; Task 2's verify is `vitest` + `tsc`).
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None - plan executed exactly as written. Both tasks' acceptance criteria (grep checks for `selectOriginalMessage`/`parseEml`/`getAttachmentContent`/`isB2Configured`/`EML_OBJECT_KEY_REGEX`, `INSERT INTO messages`/`INSERT INTO indicators` each with `RETURNING id::text`, `authResults` reaching the headers payload, `metadata` in the indicators INSERT column list, B2 PUT gated behind `isB2Configured()`, no stray `fetch(` targeting message-derived content) were verified via grep and vitest before each commit and all passed on the first attempt.
|
||||
|
||||
## Issues Encountered
|
||||
None.
|
||||
|
||||
## User Setup Required
|
||||
None - no external service configuration required. B2 and Autotask credentials are only exercised via mocks in this plan's tests; live credentials are Phase 18's concern when the on-demand trigger route is built.
|
||||
|
||||
## Next Phase Readiness
|
||||
- `parseAndStoreMessage` is a stable, fully-tested export ready for Phase 18's `POST /api/phishing/tickets/{id}/analyze` route to call directly.
|
||||
- No blockers identified. `messages`/`indicators` schema (migrations 097 + 099) is fully exercised by this service's insert shape.
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
- `lib/services/phishing-eml-service.ts` — FOUND
|
||||
- `lib/services/phishing-eml-service.test.ts` — FOUND
|
||||
- Commit `5088d8d` — FOUND in `git log`
|
||||
- Commit `ce16e67` — FOUND in `git log`
|
||||
|
||||
---
|
||||
*Phase: 16-eml-mime-evidence-parser*
|
||||
*Completed: 2026-07-15*
|
||||
195
lib/services/phishing-eml-service.test.ts
Normal file
195
lib/services/phishing-eml-service.test.ts
Normal file
|
|
@ -0,0 +1,195 @@
|
|||
/**
|
||||
* phishing-eml-service.ts — parseAndStoreMessage orchestration tests.
|
||||
*
|
||||
* postgresClient.query, getAutotaskClient (Autotask factory), and the B2
|
||||
* client's isB2Configured/presignUpload are all mocked (following
|
||||
* pax8-sync-service.test.ts / notify.test.ts's mocking discipline) — no real
|
||||
* Postgres, Autotask, or B2 credentials/network are used. All `.eml` content
|
||||
* is imported from eml-parser.fixtures.ts (synthetic only, per this
|
||||
* milestone's Out of Scope constraint — no real customer email).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
|
||||
// Mock postgresClient BEFORE importing the module under test.
|
||||
const queryMock = vi.fn();
|
||||
vi.mock('./postgres-client', () => ({
|
||||
postgresClient: {
|
||||
query: (...args: unknown[]) => queryMock(...args),
|
||||
},
|
||||
}));
|
||||
|
||||
// Mock the Autotask factory so no real credentials/network are required.
|
||||
const getAttachmentsMock = vi.fn();
|
||||
const getAttachmentContentMock = vi.fn();
|
||||
vi.mock('./autotask-factory', () => ({
|
||||
getAutotaskClient: () => ({
|
||||
getAttachments: (...args: unknown[]) => getAttachmentsMock(...args),
|
||||
getAttachmentContent: (...args: unknown[]) => getAttachmentContentMock(...args),
|
||||
}),
|
||||
}));
|
||||
|
||||
// Mock the B2 client — isB2Configured/presignUpload are stubbed per-test.
|
||||
const isB2ConfiguredMock = vi.fn();
|
||||
const presignUploadMock = vi.fn();
|
||||
vi.mock('./b2/client', () => ({
|
||||
isB2Configured: () => isB2ConfiguredMock(),
|
||||
presignUpload: (...args: unknown[]) => presignUploadMock(...args),
|
||||
EML_OBJECT_KEY_REGEX: /^phishing\/[A-Za-z0-9_-]+\/[A-Za-z0-9_-]+\.eml$/,
|
||||
}));
|
||||
|
||||
// Import AFTER the mocks are declared so vi.mock hoisting takes effect.
|
||||
import { parseAndStoreMessage } from './phishing-eml-service';
|
||||
import {
|
||||
RFC_EML_TIER_ATTACHMENTS,
|
||||
NO_EML_ATTACHMENTS,
|
||||
RICH_MULTIPART_EML,
|
||||
} from './eml-parser.fixtures';
|
||||
|
||||
const REPORT_ID = 'report-abc-123';
|
||||
const TICKET_ID = 999;
|
||||
const SELECTED_ATTACHMENT_ID = RFC_EML_TIER_ATTACHMENTS[0].id; // rfc.eml, tier 1
|
||||
// The exact URL embedded in RICH_MULTIPART_EML's text+html body (16-01 fixture).
|
||||
const MESSAGE_BODY_URL = 'http://evil-example.test/verify';
|
||||
|
||||
function selectedAttachmentWithContent() {
|
||||
return {
|
||||
...RFC_EML_TIER_ATTACHMENTS[0],
|
||||
data: RICH_MULTIPART_EML.toString('base64'),
|
||||
};
|
||||
}
|
||||
|
||||
/** Extracts [sql, params] tuples for calls whose SQL contains `needle`. */
|
||||
function callsContaining(needle: string): Array<[string, unknown[]]> {
|
||||
return queryMock.mock.calls
|
||||
.filter(([sql]) => String(sql).includes(needle))
|
||||
.map(([sql, params]) => [String(sql), (params as unknown[]) ?? []]);
|
||||
}
|
||||
|
||||
describe('parseAndStoreMessage', () => {
|
||||
const originalFetch = global.fetch;
|
||||
|
||||
beforeEach(() => {
|
||||
queryMock.mockReset();
|
||||
getAttachmentsMock.mockReset();
|
||||
getAttachmentContentMock.mockReset();
|
||||
isB2ConfiguredMock.mockReset();
|
||||
presignUploadMock.mockReset();
|
||||
|
||||
// Default: any INSERT ... RETURNING resolves with a single row.
|
||||
queryMock.mockResolvedValue({ rows: [{ id: 'message-uuid-1' }], rowCount: 1 });
|
||||
|
||||
global.fetch = vi.fn().mockResolvedValue(new Response('ok', { status: 200 }));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
global.fetch = originalFetch;
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('lists attachments, selects the original message, parses it, and writes one messages row plus indicators (happy path)', async () => {
|
||||
getAttachmentsMock.mockResolvedValue(RFC_EML_TIER_ATTACHMENTS);
|
||||
getAttachmentContentMock.mockResolvedValue(selectedAttachmentWithContent());
|
||||
isB2ConfiguredMock.mockReturnValue(false);
|
||||
|
||||
const result = await parseAndStoreMessage({ reportId: REPORT_ID, ticketId: TICKET_ID });
|
||||
|
||||
expect(result).toEqual({ stored: true, messageId: 'message-uuid-1' });
|
||||
expect(getAttachmentsMock).toHaveBeenCalledWith('Tickets', TICKET_ID);
|
||||
expect(getAttachmentContentMock).toHaveBeenCalledWith('Tickets', TICKET_ID, SELECTED_ATTACHMENT_ID);
|
||||
|
||||
const messageInserts = callsContaining('INSERT INTO messages');
|
||||
expect(messageInserts).toHaveLength(1);
|
||||
|
||||
const [, messageParams] = messageInserts[0];
|
||||
const headersPayload = JSON.parse(messageParams[2] as string);
|
||||
expect(headersPayload.authResults).toEqual({ spf: 'pass', dkim: 'fail', dmarc: 'none' });
|
||||
|
||||
const indicatorInserts = callsContaining('INSERT INTO indicators');
|
||||
expect(indicatorInserts.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('returns { stored: false } and never inserts a messages row when no .eml attachment is found', async () => {
|
||||
getAttachmentsMock.mockResolvedValue(NO_EML_ATTACHMENTS);
|
||||
|
||||
const result = await parseAndStoreMessage({ reportId: REPORT_ID, ticketId: TICKET_ID });
|
||||
|
||||
expect(result).toEqual({ stored: false, reason: 'no-eml-attachment' });
|
||||
expect(getAttachmentContentMock).not.toHaveBeenCalled();
|
||||
expect(callsContaining('INSERT INTO messages')).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('persists with raw_ref null and never calls presignUpload/PUT when B2 is unconfigured', async () => {
|
||||
getAttachmentsMock.mockResolvedValue(RFC_EML_TIER_ATTACHMENTS);
|
||||
getAttachmentContentMock.mockResolvedValue(selectedAttachmentWithContent());
|
||||
isB2ConfiguredMock.mockReturnValue(false);
|
||||
|
||||
await parseAndStoreMessage({ reportId: REPORT_ID, ticketId: TICKET_ID });
|
||||
|
||||
expect(presignUploadMock).not.toHaveBeenCalled();
|
||||
expect(global.fetch).not.toHaveBeenCalled();
|
||||
|
||||
const [, messageParams] = callsContaining('INSERT INTO messages')[0];
|
||||
expect(messageParams[6]).toBeNull(); // raw_ref column
|
||||
});
|
||||
|
||||
it('sets raw_ref to phishing/{reportId}/{attachmentId}.eml and PUTs exactly once when B2 is configured', async () => {
|
||||
getAttachmentsMock.mockResolvedValue(RFC_EML_TIER_ATTACHMENTS);
|
||||
getAttachmentContentMock.mockResolvedValue(selectedAttachmentWithContent());
|
||||
isB2ConfiguredMock.mockReturnValue(true);
|
||||
presignUploadMock.mockReturnValue('https://s3.example.test/presigned-put-url');
|
||||
|
||||
await parseAndStoreMessage({ reportId: REPORT_ID, ticketId: TICKET_ID });
|
||||
|
||||
expect(presignUploadMock).toHaveBeenCalledTimes(1);
|
||||
expect(global.fetch).toHaveBeenCalledTimes(1);
|
||||
expect(global.fetch).toHaveBeenCalledWith(
|
||||
'https://s3.example.test/presigned-put-url',
|
||||
expect.objectContaining({ method: 'PUT' })
|
||||
);
|
||||
|
||||
const [, messageParams] = callsContaining('INSERT INTO messages')[0];
|
||||
expect(messageParams[6]).toBe(`phishing/${REPORT_ID}/${SELECTED_ATTACHMENT_ID}.eml`);
|
||||
});
|
||||
|
||||
it('never calls fetch with any URL extracted from the message body (no-network invariant)', async () => {
|
||||
getAttachmentsMock.mockResolvedValue(RFC_EML_TIER_ATTACHMENTS);
|
||||
getAttachmentContentMock.mockResolvedValue(selectedAttachmentWithContent());
|
||||
isB2ConfiguredMock.mockReturnValue(true);
|
||||
presignUploadMock.mockReturnValue('https://s3.example.test/presigned-put-url');
|
||||
|
||||
await parseAndStoreMessage({ reportId: REPORT_ID, ticketId: TICKET_ID });
|
||||
|
||||
const fetchMock = global.fetch as unknown as ReturnType<typeof vi.fn>;
|
||||
for (const call of fetchMock.mock.calls) {
|
||||
expect(call[0]).not.toBe(MESSAGE_BODY_URL);
|
||||
}
|
||||
// The only fetch call made is the mocked B2 presigned PUT target.
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||
expect(fetchMock).toHaveBeenCalledWith('https://s3.example.test/presigned-put-url', expect.anything());
|
||||
});
|
||||
|
||||
it('writes an attachment_hash indicator whose metadata carries filename/contentType/size/related', async () => {
|
||||
getAttachmentsMock.mockResolvedValue(RFC_EML_TIER_ATTACHMENTS);
|
||||
getAttachmentContentMock.mockResolvedValue(selectedAttachmentWithContent());
|
||||
isB2ConfiguredMock.mockReturnValue(false);
|
||||
|
||||
await parseAndStoreMessage({ reportId: REPORT_ID, ticketId: TICKET_ID });
|
||||
|
||||
const attachmentHashInserts = callsContaining('INSERT INTO indicators').filter(
|
||||
([, params]) => params[1] === 'attachment_hash'
|
||||
);
|
||||
expect(attachmentHashInserts).toHaveLength(1);
|
||||
|
||||
const [, params] = attachmentHashInserts[0];
|
||||
const metadata = JSON.parse(params[3] as string);
|
||||
expect(metadata).toEqual(
|
||||
expect.objectContaining({
|
||||
filename: 'invoice.pdf',
|
||||
contentType: 'application/pdf',
|
||||
size: expect.any(Number),
|
||||
related: false,
|
||||
})
|
||||
);
|
||||
});
|
||||
});
|
||||
224
lib/services/phishing-eml-service.ts
Normal file
224
lib/services/phishing-eml-service.ts
Normal file
|
|
@ -0,0 +1,224 @@
|
|||
/**
|
||||
* Phishing EML/MIME evidence orchestration service (Phase 16, Plan 03).
|
||||
*
|
||||
* Turns a detected phishing report into persisted, normalized message
|
||||
* evidence: lists a ticket's attachments, selects the original reported
|
||||
* message (Plan 01's `selectOriginalMessage`), fetches its full base64
|
||||
* content (Plan 02's `AutotaskClient.getAttachmentContent`), size-guards the
|
||||
* decoded buffer, stores the raw bytes in B2 under a dedicated `.eml` key
|
||||
* when configured (D-05), parses it (Plan 01's `parseEml`), and persists one
|
||||
* `messages` row plus `indicators` rows carrying per-indicator context in
|
||||
* the D-07 `metadata` JSONB column.
|
||||
*
|
||||
* Hard invariant (SC#3 / T-16-03): this service never fetches or executes
|
||||
* anything found in the parsed message. The only outbound network calls are
|
||||
* the Autotask attachment GET (via `AutotaskClient`) and the B2 presigned
|
||||
* PUT of the raw bytes we already hold — never a URL/host derived from the
|
||||
* message content itself.
|
||||
*
|
||||
* The live on-demand trigger (`POST /api/phishing/tickets/{id}/analyze`)
|
||||
* arrives in Phase 18 (DETECT-03); this module is callable and fully tested
|
||||
* here.
|
||||
*/
|
||||
|
||||
import { postgresClient } from './postgres-client';
|
||||
import { getAutotaskClient } from './autotask-factory';
|
||||
import { presignUpload, isB2Configured, EML_OBJECT_KEY_REGEX } from './b2/client';
|
||||
import { parseEml, selectOriginalMessage, MAX_EML_BYTES, type NormalizedMessage } from './eml-parser';
|
||||
|
||||
export interface ParseAndStoreInput {
|
||||
reportId: string;
|
||||
ticketId: number;
|
||||
}
|
||||
|
||||
export interface ParseAndStoreResult {
|
||||
stored: boolean;
|
||||
messageId?: string;
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Orchestrates list -> select -> fetch -> (B2 gated) -> parse -> persist for
|
||||
* a single report/ticket.
|
||||
*
|
||||
* Returns `{ stored: false, reason: ... }` (never throws) when there is
|
||||
* nothing to persist yet (no `.eml` attachment found, no content returned,
|
||||
* or the decoded buffer is oversized). Rethrows on any other failure so a
|
||||
* future caller (Phase 18) can decide fail-vs-degrade (T-16-08).
|
||||
*/
|
||||
export async function parseAndStoreMessage(
|
||||
input: ParseAndStoreInput
|
||||
): Promise<ParseAndStoreResult> {
|
||||
const { reportId, ticketId } = input;
|
||||
|
||||
try {
|
||||
// reports.evidence only stores fullPath/title/contentType (no attachment
|
||||
// id) — list live so we have the attachment id needed for the
|
||||
// per-attachment content fetch below.
|
||||
const attachments = await getAutotaskClient().getAttachments('Tickets', ticketId);
|
||||
const selected = selectOriginalMessage(attachments);
|
||||
|
||||
if (!selected) {
|
||||
return { stored: false, reason: 'no-eml-attachment' };
|
||||
}
|
||||
|
||||
const fullAttachment = await getAutotaskClient().getAttachmentContent(
|
||||
'Tickets',
|
||||
ticketId,
|
||||
selected.id
|
||||
);
|
||||
|
||||
if (!fullAttachment?.data) {
|
||||
console.error(
|
||||
'[PHISHING-EML] Selected attachment had no content for report',
|
||||
reportId,
|
||||
'ticket',
|
||||
ticketId,
|
||||
'attachment',
|
||||
selected.id
|
||||
);
|
||||
return { stored: false, reason: 'no-attachment-content' };
|
||||
}
|
||||
|
||||
const rawEmlBuffer = Buffer.from(fullAttachment.data, 'base64');
|
||||
|
||||
// Size guard BEFORE parseEml (T-16-01 — DoS mitigation): degrade rather
|
||||
// than let parseEml's own internal guard throw and crash the caller.
|
||||
if (rawEmlBuffer.byteLength > MAX_EML_BYTES) {
|
||||
console.error(
|
||||
'[PHISHING-EML] Decoded attachment exceeds MAX_EML_BYTES, skipping report',
|
||||
reportId,
|
||||
'size',
|
||||
rawEmlBuffer.byteLength
|
||||
);
|
||||
return { stored: false, reason: 'oversized-attachment' };
|
||||
}
|
||||
|
||||
let rawRef: string | null = null;
|
||||
if (isB2Configured()) {
|
||||
const objectKey = `phishing/${reportId}/${selected.id}.eml`;
|
||||
try {
|
||||
const uploadUrl = presignUpload(objectKey, 1800, undefined, EML_OBJECT_KEY_REGEX);
|
||||
const putResponse = await fetch(uploadUrl, { method: 'PUT', body: rawEmlBuffer });
|
||||
if (!putResponse.ok) {
|
||||
throw new Error(`B2 PUT ${objectKey} failed: ${putResponse.status}`);
|
||||
}
|
||||
rawRef = objectKey;
|
||||
} catch (error) {
|
||||
// Graceful degrade (RESEARCH.md Pitfall 3) — a failed/absent B2 PUT
|
||||
// must not abort parsing/persistence.
|
||||
console.error(
|
||||
'[PHISHING-EML] Failed to upload raw .eml to B2 for report',
|
||||
reportId,
|
||||
error
|
||||
);
|
||||
rawRef = null;
|
||||
}
|
||||
} else {
|
||||
console.log(
|
||||
'[PHISHING-EML] B2 not configured; skipping raw .eml upload for report',
|
||||
reportId
|
||||
);
|
||||
}
|
||||
|
||||
const normalized: NormalizedMessage = await parseEml(rawEmlBuffer);
|
||||
|
||||
// Full normalized header block, including structured SPF/DKIM/DMARC
|
||||
// verdicts (D-06), persisted into messages.headers JSONB.
|
||||
const headersPayload = {
|
||||
from: normalized.from,
|
||||
replyTo: normalized.replyTo,
|
||||
returnPath: normalized.returnPath,
|
||||
to: normalized.to,
|
||||
cc: normalized.cc,
|
||||
subject: normalized.subject,
|
||||
date: normalized.date,
|
||||
messageId: normalized.messageId,
|
||||
receivedChain: normalized.receivedChain,
|
||||
authResults: normalized.authResults,
|
||||
authResultsOriginal: normalized.authResultsOriginal,
|
||||
};
|
||||
|
||||
const messageInsert = await postgresClient.query<{ id: string }>(
|
||||
`INSERT INTO messages (
|
||||
report_id, message_id, headers, urls, attachments, body_preview, raw_ref
|
||||
)
|
||||
VALUES ($1, $2, $3::jsonb, $4::jsonb, $5::jsonb, $6, $7)
|
||||
RETURNING id::text AS id`,
|
||||
[
|
||||
reportId,
|
||||
normalized.messageId,
|
||||
JSON.stringify(headersPayload),
|
||||
JSON.stringify(normalized.urls),
|
||||
JSON.stringify(normalized.attachments),
|
||||
normalized.bodyPreview,
|
||||
rawRef,
|
||||
]
|
||||
);
|
||||
|
||||
const messageId = messageInsert.rows[0].id;
|
||||
|
||||
// One indicator per attachment checksum (D-07 metadata carries
|
||||
// filename/contentType/size/related so Phase 19 can weight inline parts
|
||||
// differently without a second lookup).
|
||||
for (const attachment of normalized.attachments) {
|
||||
if (!attachment.checksum) continue;
|
||||
await postgresClient.query(
|
||||
`INSERT INTO indicators (message_id, indicator_type, value, metadata)
|
||||
VALUES ($1, $2, $3, $4::jsonb)
|
||||
RETURNING id::text AS id`,
|
||||
[
|
||||
messageId,
|
||||
'attachment_hash',
|
||||
attachment.checksum,
|
||||
JSON.stringify({
|
||||
filename: attachment.filename,
|
||||
contentType: attachment.contentType,
|
||||
size: attachment.size,
|
||||
related: attachment.related,
|
||||
}),
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
// One indicator per extracted URL — never dereferenced, only persisted
|
||||
// as a string value (SC#3 / T-16-03).
|
||||
for (const url of normalized.urls) {
|
||||
await postgresClient.query(
|
||||
`INSERT INTO indicators (message_id, indicator_type, value, metadata)
|
||||
VALUES ($1, $2, $3, $4::jsonb)
|
||||
RETURNING id::text AS id`,
|
||||
[messageId, 'url', url, JSON.stringify({ part: 'body' })]
|
||||
);
|
||||
}
|
||||
|
||||
// One indicator for the sender, if present.
|
||||
if (normalized.from.email) {
|
||||
await postgresClient.query(
|
||||
`INSERT INTO indicators (message_id, indicator_type, value, metadata)
|
||||
VALUES ($1, $2, $3, $4::jsonb)
|
||||
RETURNING id::text AS id`,
|
||||
[
|
||||
messageId,
|
||||
'sender',
|
||||
normalized.from.email,
|
||||
JSON.stringify({
|
||||
displayName: normalized.from.displayName,
|
||||
domain: normalized.from.domain,
|
||||
}),
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
return { stored: true, messageId };
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'[PHISHING-EML] Failed to parse/store message for report',
|
||||
reportId,
|
||||
'ticket',
|
||||
ticketId,
|
||||
error
|
||||
);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue