docs(16-02): complete Autotask attachment content / B2 EML regex / indicators metadata plan

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.planning/phases/16-eml-mime-evidence-parser/16-02-SUMMARY.md

@@ -0,0 +1,118 @@
+---
+phase: 16-eml-mime-evidence-parser
+plan: 02
+subsystem: api
+tags: [autotask, backblaze-b2, postgres, vitest, tdd]
+
+# Dependency graph
+requires:
+  - phase: 15-data-model-detection-ticket-evidence
+    provides: indicators table (migration 097), phishing-detector core, EVID-01 ticket evidence capture
+provides:
+  - AutotaskClient.getAttachmentContent() — fetches full base64 `.eml` attachment content via the items[0] convention
+  - EML_OBJECT_KEY_REGEX + parameterized presignDownload/presignUpload/downloadToBuffer for storing raw `.eml` bytes in B2 under a path-traversal-safe key shape
+  - migrations/099_indicators_metadata.sql — indicators.metadata JSONB column, applied to dev DB
+affects: [16-03-eml-mime-parser-orchestrator, 17-mimecast-blast-radius]
+
+# Tech tracking
+tech-stack:
+  added: []
+  patterns:
+    - "AutotaskClient per-attachment-ID GET reads response.items?.[0] (list-shaped), NOT response.item"
+    - "B2 client: parallel object-key regex per use-case, never loosen an existing regex; keyRegex is an optional trailing param defaulting to the original"
+
+key-files:
+  created:
+    - lib/services/autotask-client.test.ts
+    - migrations/099_indicators_metadata.sql
+  modified:
+    - lib/services/autotask-client.ts
+    - lib/services/b2/client.ts
+    - lib/services/b2/client.test.ts
+
+key-decisions:
+  - "Migration 099 applied to the dev DB via direct docker exec (not scripts/apply-migrations.sh), because that script hardcodes MIGRATIONS_DIR=/opt/stacks/pulse/migrations — the main repo's absolute path, not the worktree's — so it could not see the new file from inside this worktree checkout. Used the same real credentials the script itself defaults to (pulse_user/pulse_autotask), not hand-rolled wrong defaults."
+
+patterns-established:
+  - "First AutotaskClient unit test file (autotask-client.test.ts) — mocks global.fetch, asserts items vs item envelope shapes"
+
+requirements-completed: [EVID-03, EVID-04]
+
+# Metrics
+duration: 25min
+completed: 2026-07-15
+---
+
+# Phase 16 Plan 02: Autotask Attachment Content, B2 EML Key Regex, Indicators Metadata Column Summary
+
+**AutotaskClient.getAttachmentContent() (items[0] convention), a parallel EML_OBJECT_KEY_REGEX + parameterized B2 key validation, and migration 099 (indicators.metadata JSONB) — the three independent supporting pieces the Plan 03 EML/MIME orchestrator depends on.**
+
+## Performance
+
+- **Duration:** ~25 min
+- **Started:** 2026-07-15T14:17:00Z
+- **Completed:** 2026-07-15T14:24:00Z
+- **Tasks:** 3 completed
+- **Files modified:** 5 (2 new, 3 modified)
+
+## Accomplishments
+- `AutotaskClient.getAttachmentContent(entityName, entityId, attachmentId)` fetches a single attachment's full base64 content, correctly reading the live `{items:[...]}` response shape (not `{item:...}`) — verified by a new unit test that also proves an `{item:...}`-shaped response yields `null`, guarding against a future refactor copying the wrong sibling convention.
+- `EML_OBJECT_KEY_REGEX` added to `lib/services/b2/client.ts` alongside the existing (untouched) `OBJECT_KEY_REGEX`, enforcing `phishing/<reportId>/<attachmentId>.eml` and rejecting path traversal. `presignDownload`, `presignUpload`, and `downloadToBuffer` now accept an optional trailing `keyRegex` parameter (defaulting to `OBJECT_KEY_REGEX`), so the existing LogLift call site (`rmm/executor.ts:337`, `presignUpload(objectKey, 1800)`) compiles and behaves identically.
+- `migrations/099_indicators_metadata.sql` adds a nullable `indicators.metadata JSONB` column (D-07), applied live to the dev DB — confirmed via `information_schema.columns`.
+
+## Task Commits
+
+Each task was committed atomically (TDD tasks have separate test → feat commits):
+
+1. **Task 1: AutotaskClient.getAttachmentContent()**
+   - `8be10db` (test) — failing test for items[0] convention + item-shaped-yields-null guard
+   - `9b65de7` (feat) — implementation
+2. **Task 2: EML_OBJECT_KEY_REGEX + parameterized B2 key validation**
+   - `6de92a5` (test) — failing tests for new regex + keyRegex param
+   - `8630fd5` (feat) — implementation
+3. **Task 3: Migration 099 — indicators.metadata JSONB**
+   - `0bf37ce` (feat) — migration file + applied to dev DB
+
+_RED confirmed for both TDD tasks by temporarily removing the implementation and re-running the test suite before restoring it and committing test-then-feat._
+
+## Files Created/Modified
+- `lib/services/autotask-client.ts` - added `getAttachmentContent()` after `getAttachments()`
+- `lib/services/autotask-client.test.ts` - new; first AutotaskClient unit coverage (3 tests)
+- `lib/services/b2/client.ts` - added `EML_OBJECT_KEY_REGEX`; parameterized `keyRegex` on presign*/downloadToBuffer
+- `lib/services/b2/client.test.ts` - extended with EML_OBJECT_KEY_REGEX + custom-keyRegex presignUpload tests (5 new tests)
+- `migrations/099_indicators_metadata.sql` - new; nullable `indicators.metadata JSONB` + COMMENT
+
+## Decisions Made
+- Applied migration 099 to the dev DB via direct `docker exec pulse-postgres psql -U pulse_user -d pulse_autotask` rather than `scripts/apply-migrations.sh`, because that script's `MIGRATIONS_DIR` is hardcoded to the main repo's absolute path and cannot see files that only exist in this worktree checkout. Used the same real project credentials (`pulse_user`/`pulse_autotask`) the script itself would use — not the script's local-non-Docker fallback defaults.
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 3 - Blocking] scripts/apply-migrations.sh could not see the new migration file from inside the worktree**
+- **Found during:** Task 3 (migration 099 apply step)
+- **Issue:** The script hardcodes `MIGRATIONS_DIR="/opt/stacks/pulse/migrations"` — the main repo's path, not this worktree's `.claude/worktrees/agent-a2b7d8cafac0f4270/migrations/`. Running it reported "Migration file 099_indicators_metadata.sql not found" even though the file existed in the worktree.
+- **Fix:** Applied the migration directly with `docker exec -i pulse-postgres psql -U pulse_user -d pulse_autotask < migrations/099_indicators_metadata.sql`, using the identical real credentials the script defaults to for Docker mode (not the script's local/non-Docker psql fallback defaults, which are wrong for this project).
+- **Files modified:** none (operational step only)
+- **Verification:** `information_schema.columns` confirms `indicators.metadata` is `jsonb` on the dev DB.
+- **Committed in:** 0bf37ce (Task 3 commit, migration file only — no code changed by this deviation)
+
+---
+
+**Total deviations:** 1 auto-fixed (1 blocking)
+**Impact on plan:** No scope creep — same migration content and same live-DB outcome the plan specified; only the apply mechanism differed due to a pre-existing worktree/script path mismatch.
+
+## Issues Encountered
+None beyond the deviation above.
+
+## User Setup Required
+None - no external service configuration required.
+
+## Next Phase Readiness
+- `getAttachmentContent()`, `EML_OBJECT_KEY_REGEX`, and `indicators.metadata` are all available for Plan 03's EML/MIME parser orchestrator to consume.
+- `rmm/executor.ts`'s existing `presignUpload(objectKey, 1800)` call site still type-checks unchanged (verified via `npx tsc --noEmit`).
+- No blockers.
+
+---
+*Phase: 16-eml-mime-evidence-parser*
+*Completed: 2026-07-15*