Compare commits
9 commits
672f17b7f9
...
6eafaae40a
| Author | SHA1 | Date | |
|---|---|---|---|
| 6eafaae40a | |||
| db7db98374 | |||
| f447ac33c0 | |||
| 9311f1044b | |||
| d41d6e7c0b | |||
| a0d01cfe8b | |||
| f0c09927f4 | |||
| a65b29055c | |||
| ea8a36b394 |
10 changed files with 1502 additions and 6 deletions
|
|
@ -4,8 +4,8 @@ milestone: v3.0
|
|||
milestone_name: Phishing Triage Automation
|
||||
status: Awaiting next milestone
|
||||
stopped_at: Phase 23 context gathered
|
||||
last_updated: "2026-07-17T10:55:33.806Z"
|
||||
last_activity: 2026-07-17 — Milestone v3.0 completed and archived
|
||||
last_updated: "2026-07-18T15:20:36.300Z"
|
||||
last_activity: "2026-07-18 — Completed quick task 260718-9qg: QBO integration handoff document"
|
||||
progress:
|
||||
total_phases: 9
|
||||
completed_phases: 9
|
||||
|
|
@ -28,7 +28,7 @@ See: .planning/PROJECT.md (updated 2026-07-14)
|
|||
Phase: Milestone v3.0 complete
|
||||
Plan: —
|
||||
Status: Awaiting next milestone
|
||||
Last activity: 2026-07-18 — Completed quick task 260718-7v8: Mimecast blast-radius held-message false-positive fix
|
||||
Last activity: 2026-07-21 — Completed quick task 260721-fy8: fix mimecast and qbo sync scheduler dispatch and reschedule mimecast cron
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
|
|
@ -129,6 +129,8 @@ None yet.
|
|||
| 260717-a19 | Fix phishing simulation-vendor allowlist gaps (3 missing KnowBe4 domains), auto-parse timing race (retry on ticket.update), and parseAndStoreMessage idempotency; reclassified 6 stale Seubert campaigns (all flipped UNWANTED → USER_AWARENESS) | 2026-07-17 | cf04f07 | [260717-a19-fix-phishing-simulation-vendor-allowlist](./quick/260717-a19-fix-phishing-simulation-vendor-allowlist/) |
|
||||
| 260717-v6c | Add "Mark as accidental report" action to the phishing Action Area — closes out a campaign and posts a fixed customer-facing note to the reporter (distinct from the silent "Mark as false positive" action) | 2026-07-18 | 565a0c1 | [260717-v6c-add-a-mark-as-accidental-report-action-t](./quick/260717-v6c-add-a-mark-as-accidental-report-action-t/) |
|
||||
| 260718-7v8 | Fix Mimecast blast-radius false positives — date-scope `getHeldMessages()` and add a sender-domain relevance guard so unrelated held mail in a recipient's queue no longer inflates held/matched counts or overwrites a genuinely delivered recipient's status | 2026-07-18 | b7d6be4 | [260718-7v8-fix-mimecast-blast-radius-held-message-f](./quick/260718-7v8-fix-mimecast-blast-radius-held-message-f/) |
|
||||
| 260718-9qg | Add self-contained `QBO_INTEGRATION_HANDOFF.md` documenting Pulse's QuickBooks Online OAuth2 flow, token storage/refresh, sandbox/production API base URLs, and gotchas (deletion-diffing, CSRF state gap, NEXTAUTH_URL legacy var) for a new app's team | 2026-07-18 | ea8a36b | [260718-9qg-create-a-quickbooks-online-integration-h](./quick/260718-9qg-create-a-quickbooks-online-integration-h/) |
|
||||
| 260721-fy8 | Fix missing `mimecast-sync`/`qbo` scheduler dispatch branches (both silently fell through to a generic Autotask full sync) and reschedule `mimecast-sync` off the 2am 3-way cron collision with `qbo-sync-2am` and `veeam-full` | 2026-07-21 | db7db98 | [260721-fy8-fix-mimecast-and-qbo-sync-scheduler-disp](./quick/260721-fy8-fix-mimecast-and-qbo-sync-scheduler-disp/) |
|
||||
|
||||
## Deferred Items
|
||||
|
||||
|
|
@ -159,9 +161,9 @@ Items acknowledged and deferred at v3.0 milestone close on 2026-07-17 (pre-fligh
|
|||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-07-16T22:37:09.979Z
|
||||
Last session: 2026-07-18T15:20:36.295Z
|
||||
Stopped at: Phase 23 context gathered
|
||||
Resume file: .planning/phases/23-classification-disposition-per-client-automation-gate/23-CONTEXT.md
|
||||
Resume file: None
|
||||
</content>
|
||||
|
||||
## Operator Next Steps
|
||||
|
|
|
|||
|
|
@ -0,0 +1,109 @@
|
|||
---
|
||||
phase: quick-260718-9qg
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified: [QBO_INTEGRATION_HANDOFF.md]
|
||||
autonomous: true
|
||||
requirements: [DOC-QBO-HANDOFF]
|
||||
must_haves:
|
||||
truths:
|
||||
- "A reader with NO access to the Pulse codebase can understand Pulse's QBO integration end to end from this one file"
|
||||
- "The full OAuth2 authorization-code flow is documented step by step (authorize → callback → token exchange → storage → refresh)"
|
||||
- "Env vars, API base URLs (sandbox vs production), scopes, and minor version are stated explicitly"
|
||||
- "Known gotchas — including anything learned from the recent QBO AR diagnostics work — are captured"
|
||||
artifacts:
|
||||
- path: "QBO_INTEGRATION_HANDOFF.md"
|
||||
provides: "Self-contained QBO integration handoff document for a new app"
|
||||
min_lines: 120
|
||||
key_links: []
|
||||
---
|
||||
|
||||
<objective>
|
||||
Produce a single standalone markdown document, `QBO_INTEGRATION_HANDOFF.md` at the repo root, that captures everything Pulse knows about connecting to QuickBooks Online (QBO), so a separate new application's team — who will NOT have access to the Pulse codebase — can implement their own QBO integration with a head start.
|
||||
|
||||
Purpose: Transfer hard-won integration knowledge (OAuth2 flow, token storage/refresh, API base URLs, scopes, gotchas) to a team building a brand-new app.
|
||||
Output: `QBO_INTEGRATION_HANDOFF.md` (new file, repo root).
|
||||
|
||||
**Docs-only task.** No application source code may be modified. The only file written is the handoff document.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/STATE.md
|
||||
|
||||
Known QBO surface in this repo (starting points for the executor's investigation — the executor MUST read these, not assume their contents):
|
||||
- `lib/services/qbo-client.ts` — QBO API client (expect factory pattern per repo convention: `getQboClient()` / `isQboConfigured()`)
|
||||
- `lib/services/qbo-sync-service.ts` — sync service (invoices, AR, etc.)
|
||||
- `app/api/qbo/auth/route.ts` — OAuth2 connect/authorize + callback handling
|
||||
- `app/api/qbo/sync/route.ts` — sync trigger endpoint
|
||||
- `app/api/qbo/disconnect/route.ts` — token teardown
|
||||
- `app/api/qbo/diagnose-ar/route.ts` — AR (accounts receivable) diagnostics (likely source of recently-learned gotchas)
|
||||
- `migrations/051_create_qbo_tables.sql` — QBO tables (token storage, invoices, etc.)
|
||||
- `migrations/088_qbo_invoices_soft_delete.sql` — soft-delete addition
|
||||
- No existing QBO guide under `docs/` — this handoff is net-new.
|
||||
- Note: `.env` did NOT surface `QBO_*` / `QUICKBOOKS_*` / `INTUIT_*` vars by name — the executor MUST grep the source for `process.env.` inside the QBO files to discover the ACTUAL env var names in use.
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Investigate Pulse's QBO integration and write the self-contained handoff doc</name>
|
||||
<files>QBO_INTEGRATION_HANDOFF.md</files>
|
||||
<action>
|
||||
First, INVESTIGATE the existing QBO integration in this repo. Do not write anything until the investigation is complete. Read (not skim) each of the following and extract the concrete details:
|
||||
|
||||
1. Env vars — grep the QBO files for `process.env.` to find the REAL variable names, then record each: client id, client secret, environment (sandbox vs production toggle), redirect/callback URI, realm/company id, and any discovery or token endpoint overrides. Command hint: `grep -rn "process\.env\." lib/services/qbo-client.ts lib/services/qbo-sync-service.ts app/api/qbo/`.
|
||||
|
||||
2. OAuth2 authorization-code flow — from `app/api/qbo/auth/route.ts` and `lib/services/qbo-client.ts`, capture: the authorize URL (Intuit `appcenter.intuit.com/connect/oauth2`), the requested scopes (e.g. `com.intuit.quickbooks.accounting`), the `state` handling / CSRF protection, the redirect URI, the token-exchange URL (`oauth.platform.intuit.com/oauth2/v1/tokens/bearer`), the token-refresh flow, and how the `realmId` (company id) is received on the callback.
|
||||
|
||||
3. Token storage — from `migrations/051_create_qbo_tables.sql` (and `088`), document the exact table(s) and columns used to persist access token, refresh token, realm/company id, and expiry timestamps; and from the client code, how expiry is detected and refresh is triggered (proactive vs on-401).
|
||||
|
||||
4. API base URLs & versioning — record the sandbox base (`sandbox-quickbooks.api.intuit.com`) vs production base (`quickbooks.api.intuit.com`), the `minorversion` query param if used, and the request path shape (`/v3/company/{realmId}/...`).
|
||||
|
||||
5. What QBO is used for in Pulse — read `lib/services/qbo-sync-service.ts` and the api routes to summarize the entities synced (invoices, AR, payments/deposits) and the direction of data flow.
|
||||
|
||||
6. Gotchas / lessons learned — read `app/api/qbo/diagnose-ar/route.ts` and run `git log --oneline -20 -- app/api/qbo lib/services/qbo-client.ts lib/services/qbo-sync-service.ts migrations/051_create_qbo_tables.sql migrations/088_qbo_invoices_soft_delete.sql` (and inspect the "QBO AR diagnostics" work referenced in the recent `chore: check in pending work` commit) to fold in any real gotchas: rate limits, token-refresh failures, sandbox-vs-production quirks, AR/invoice reconciliation issues, soft-delete handling, realm mismatch, etc. Also check the completed quick-task dirs under `.planning/quick/` matching `*qbo*` (e.g. `260519-0oz-add-qbo-createpayment-createdeposit`, `260521-fci-stopgap-nightly-reconciliation`) for SUMMARY notes worth distilling.
|
||||
|
||||
Then WRITE `QBO_INTEGRATION_HANDOFF.md` at the repo root as a self-contained document. It MUST be readable by someone with zero access to this codebase: inline every relevant detail; do NOT leave bare "see lib/services/qbo-client.ts" pointers — if a detail lives in a Pulse file, summarize the detail itself. Pulse file paths may appear only as parenthetical "(in Pulse this lives in …)" provenance notes, never as the sole carrier of information.
|
||||
|
||||
Structure the document roughly as:
|
||||
- **Overview** — what QBO integration does and why (the business purpose).
|
||||
- **Prerequisites** — creating an Intuit Developer app, sandbox vs production, redirect URI registration, required scopes.
|
||||
- **Environment variables** — a table of every env var (real names discovered from source) with description and example/placeholder value. Do NOT copy any real secret values from `.env`; use placeholders.
|
||||
- **OAuth2 connection flow** — a numbered, step-by-step walkthrough: authorize redirect → user consent → callback with `code` + `realmId` → token exchange → persistence → subsequent refresh. Include the exact Intuit URLs.
|
||||
- **Token storage & refresh** — the table schema (columns + purpose) and the refresh strategy, generalized so the new app can adapt it (they may not use Postgres).
|
||||
- **API usage** — base URLs (sandbox/production), path shape, minor version, an example authenticated request.
|
||||
- **Gotchas & lessons learned** — the concrete pitfalls gathered in the investigation.
|
||||
- **Minimal code-flow example** — a distilled, framework-agnostic pseudocode/TypeScript sketch of authorize → exchange → refresh → call (NOT a copy-paste of Pulse code; enough that a QBO-unfamiliar team can implement their own).
|
||||
|
||||
Write directive prose and generalized examples. Do NOT modify any application source, migrations, env files, or existing docs.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>test -f QBO_INTEGRATION_HANDOFF.md && test $(grep -v '^#' QBO_INTEGRATION_HANDOFF.md | wc -l) -ge 100 && grep -qi 'oauth' QBO_INTEGRATION_HANDOFF.md && grep -qi 'refresh' QBO_INTEGRATION_HANDOFF.md && grep -qiE 'sandbox|production' QBO_INTEGRATION_HANDOFF.md && echo OK</automated>
|
||||
</verify>
|
||||
<done>
|
||||
`QBO_INTEGRATION_HANDOFF.md` exists at repo root, is self-contained (no information-bearing bare file-path pointers), and covers overview, prerequisites, env vars, the full OAuth2 flow, token storage/refresh, API base URLs (sandbox + production), gotchas, and a minimal code-flow example. No application source, migration, env, or existing doc files were modified.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
- `git status` shows exactly one new file (`QBO_INTEGRATION_HANDOFF.md`) and no modifications to existing tracked source/docs/migrations.
|
||||
- The document contains no real secret values (placeholders only).
|
||||
- Every major section (overview, prerequisites, env vars, OAuth2 flow, token storage/refresh, API URLs, gotchas, code example) is present and populated with concrete detail drawn from the repo investigation.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
A new-app engineer can read `QBO_INTEGRATION_HANDOFF.md` alone and understand how to register an Intuit app, run the OAuth2 authorization-code flow, store and refresh tokens, hit the correct sandbox/production API endpoints, and avoid the pitfalls Pulse already hit — without ever opening the Pulse codebase.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/quick/260718-9qg-create-a-quickbooks-online-integration-h/260718-9qg-SUMMARY.md` when done.
|
||||
</output>
|
||||
|
|
@ -0,0 +1,92 @@
|
|||
---
|
||||
phase: quick-260718-9qg
|
||||
plan: 01
|
||||
subsystem: docs
|
||||
tags: [quickbooks, qbo, oauth2, integration, handoff, finance]
|
||||
|
||||
# Dependency graph
|
||||
requires: []
|
||||
provides:
|
||||
- "QBO_INTEGRATION_HANDOFF.md — self-contained OAuth2/token/API/gotchas reference for QuickBooks Online, written for a team with no access to Pulse's codebase"
|
||||
affects: [any future work building or maintaining a QBO integration in a different app]
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns: []
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- QBO_INTEGRATION_HANDOFF.md
|
||||
modified: []
|
||||
|
||||
key-decisions:
|
||||
- "Inlined every detail (env vars, exact Intuit URLs, table schemas, request/response shapes) rather than pointing at Pulse source files, per the plan's self-containment requirement"
|
||||
- "Called out two real defects found during investigation as explicit gotchas rather than silently working around them: unverified OAuth `state` (CSRF gap) and the legacy `NEXTAUTH_URL` env var used for the redirect URI instead of Pulse's actual BETTER_AUTH_URL"
|
||||
- "Used placeholder values for all env var examples — no real secrets existed in .env for QBO_* keys to begin with, so nothing needed redaction"
|
||||
|
||||
requirements-completed: [DOC-QBO-HANDOFF]
|
||||
|
||||
# Metrics
|
||||
duration: 25min
|
||||
completed: 2026-07-18
|
||||
---
|
||||
|
||||
# Phase quick-260718-9qg Plan 01: QBO Integration Handoff Doc Summary
|
||||
|
||||
**Self-contained `QBO_INTEGRATION_HANDOFF.md` covering QBO's OAuth2 authorization-code flow, token storage/refresh schema, sandbox vs production API base URLs, minor-version pinning, and 11 concrete gotchas (deletion-diffing, CSRF state, report NoReportData markers, check-number normalization, idempotency, etc.) drawn from Pulse's actual QBO source and its recent AR-reconciliation fix.**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** ~25 min
|
||||
- **Started:** 2026-07-18T11:00:00Z (approx.)
|
||||
- **Completed:** 2026-07-18T11:25:00Z (approx.)
|
||||
- **Tasks:** 1
|
||||
- **Files modified:** 1 (new file)
|
||||
|
||||
## Accomplishments
|
||||
- Investigated Pulse's real QBO integration end to end: `lib/services/qbo-client.ts`, `lib/services/qbo-sync-service.ts`, all four `app/api/qbo/*` routes, `migrations/051_create_qbo_tables.sql`, `migrations/088_qbo_invoices_soft_delete.sql`, `middleware.ts` public-route list, `integration-health.ts` config check, git history (`b98c674`, `ef9b31e`, `672f17b`), and the `260519-0oz` quick-task SUMMARY for the createPayment/createDeposit work.
|
||||
- Wrote `QBO_INTEGRATION_HANDOFF.md` (596 lines) at the repo root with all required sections: Overview, Prerequisites, Environment variables (table), OAuth2 connection flow (11-step numbered walkthrough with exact Intuit URLs), Token storage & refresh (generalized schema + pseudocode), API usage (base URLs, path shape, minorversion, query/write/report examples), Gotchas & lessons learned (11 concrete items), and a framework-agnostic TypeScript pseudocode code-flow example.
|
||||
- Surfaced two real defects discovered during investigation as explicit gotchas for the new team to avoid repeating: (1) Pulse's OAuth `state` value is generated but never persisted/verified on callback — a real CSRF gap; (2) the redirect-URI base URL is read from `NEXTAUTH_URL`, a variable not present anywhere in Pulse's `.env`/`.env.local` (a legacy holdover from a prior auth library), distinct from the app's actual `BETTER_AUTH_URL`.
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1: Investigate Pulse's QBO integration and write the self-contained handoff doc** - `ea8a36b` (docs)
|
||||
|
||||
**Plan metadata:** (this commit, made after SUMMARY.md)
|
||||
|
||||
## Files Created/Modified
|
||||
- `QBO_INTEGRATION_HANDOFF.md` - Self-contained QBO integration handoff document (overview, prerequisites, env vars, OAuth2 flow, token storage/refresh, API usage, gotchas, code-flow example)
|
||||
|
||||
## Decisions Made
|
||||
- Inlined every Pulse-derived detail directly into the doc (no bare `see lib/services/qbo-client.ts` pointers); Pulse file paths appear only as parenthetical provenance notes.
|
||||
- Documented the `NEXTAUTH_URL` / `BETTER_AUTH_URL` mismatch and the unverified OAuth `state` as explicit "don't repeat this" gotchas rather than omitting them, since the plan's objective is transferring hard-won (including negative) knowledge to a new team.
|
||||
- Recommended `crypto.randomUUID()` (or a signed token) plus server-side verification in place of Pulse's current `Math.random()`-based `state`, and recommended adding an explicit Intuit token-revocation call for a real "disconnect," since a new implementation should start from a stronger baseline than Pulse's current code.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None - plan executed exactly as written. This was a docs-only investigation-and-write task; no application source, migrations, env files, or existing docs were touched (confirmed via `git status --short`, which shows only `QBO_INTEGRATION_HANDOFF.md` as an addition).
|
||||
|
||||
## Issues Encountered
|
||||
None.
|
||||
|
||||
## User Setup Required
|
||||
|
||||
None - no external service configuration required. This task only produced a reference document; it does not change any running behavior.
|
||||
|
||||
## Next Phase Readiness
|
||||
- `QBO_INTEGRATION_HANDOFF.md` is ready to be handed directly to the new application's engineering team.
|
||||
- No blockers. If the new team later reports gaps (e.g. QBO Payments/charge-card scope, multi-realm support), a follow-up quick task can extend this document — it was intentionally scoped to what Pulse's own integration actually covers (accounting scope only, single realm).
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
- `QBO_INTEGRATION_HANDOFF.md` exists at repo root — FOUND
|
||||
- Automated verify command (`test -f ... && line count >= 100 && grep oauth/refresh/sandbox|production`) — PASSED (`OK`)
|
||||
- Commit `ea8a36b` exists in git log — FOUND
|
||||
- `git status --short` shows only `QBO_INTEGRATION_HANDOFF.md` as a tracked addition (plus the pre-existing untracked `.planning/quick/260718-9qg-.../` plan directory, unrelated to source/docs/migrations) — CONFIRMED
|
||||
|
||||
---
|
||||
*Phase: quick-260718-9qg*
|
||||
*Completed: 2026-07-18*
|
||||
|
|
@ -0,0 +1,239 @@
|
|||
---
|
||||
phase: quick-260721-fy8
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- lib/services/sync-scheduler.ts
|
||||
- migrations/101_reschedule_mimecast_sync.sql
|
||||
autonomous: true
|
||||
requirements: [FIX-SCHED-DISPATCH, FIX-CRON-COLLISION]
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "The mimecast-sync schedule invokes real Mimecast sync logic, not a generic Autotask full sync"
|
||||
- "The qbo schedules invoke the QBO sync service, not a generic Autotask full sync"
|
||||
- "mimecast-sync no longer shares the 0 2 * * * cron slot with qbo-sync-2am and veeam-full"
|
||||
- "npx tsc --noEmit --pretty passes clean"
|
||||
artifacts:
|
||||
- path: "lib/services/sync-scheduler.ts"
|
||||
provides: "mimecast-sync and qbo dispatch branches in executeScheduledSync()"
|
||||
contains: "config.sync_type === 'mimecast-sync'"
|
||||
- path: "migrations/101_reschedule_mimecast_sync.sql"
|
||||
provides: "Guarded UPDATE moving mimecast-sync off 0 2 * * *"
|
||||
contains: "UPDATE sync_schedules"
|
||||
key_links:
|
||||
- from: "lib/services/sync-scheduler.ts"
|
||||
to: "lib/services/mimecast-sync-service.ts"
|
||||
via: "runMimecastIncrementalSync"
|
||||
pattern: "runMimecastIncrementalSync"
|
||||
- from: "lib/services/sync-scheduler.ts"
|
||||
to: "lib/services/qbo-sync-service.ts"
|
||||
via: "getQboSyncService().incrementalSync"
|
||||
pattern: "getQboSyncService"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Fix the sync scheduler so the `mimecast-sync` and `qbo` scheduled jobs call their real
|
||||
sync logic instead of silently falling through to the generic Autotask `fullSync()`
|
||||
catch-all, and reschedule `mimecast-sync`'s cron off the `0 2 * * *` collision it shares
|
||||
with `qbo-sync-2am` and `veeam-full`.
|
||||
|
||||
Purpose: Two nightly integrations (Mimecast, QBO) never actually run on schedule — they
|
||||
trigger a full Autotask entity sync instead — and their misfire contends for the
|
||||
SyncService singleton mutex, causing the "A sync operation is already in progress" lock
|
||||
errors seen at 2 AM.
|
||||
Output: Two new dispatch branches in `executeScheduledSync()`, `'mimecast-sync'` added to
|
||||
the `sync_type` union, and a new guarded migration (plus live DB apply) moving
|
||||
`mimecast-sync` to a collision-free time.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@CLAUDE.md
|
||||
@lib/services/sync-scheduler.ts
|
||||
|
||||
<interfaces>
|
||||
<!-- Verified from the codebase — use these directly, no exploration needed. -->
|
||||
|
||||
From lib/services/sync-scheduler.ts (line 25) — the sync_type union currently
|
||||
INCLUDES 'qbo' but NOT 'mimecast-sync'. Comparing config.sync_type === 'mimecast-sync'
|
||||
without adding it to the union is a TS "no overlap" error, so the union MUST be
|
||||
extended.
|
||||
|
||||
From lib/services/mimecast-sync-service.ts:
|
||||
export async function runMimecastIncrementalSync(): Promise<MimecastSyncResult>
|
||||
export interface MimecastSyncResult {
|
||||
messagesUpserted: number;
|
||||
threatsUpserted: number;
|
||||
bodiesFetched: number;
|
||||
purgedMessages: number;
|
||||
errors: string[];
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
From lib/services/mimecast-client.ts:
|
||||
export function isMimecastConfigured(): boolean
|
||||
|
||||
From lib/services/qbo-sync-service.ts:
|
||||
async incrementalSync(triggeredBy = 'system'): Promise<QboSyncResult> // class method
|
||||
async fullSync(triggeredBy = 'system'): Promise<QboSyncResult>
|
||||
export function getQboSyncService(): QboSyncService
|
||||
// Manual route app/api/qbo/sync/route.ts uses incrementalSync for the non-full path.
|
||||
|
||||
Existing pax8-daily branch (lines 478-493) — the disable-check pattern to mirror for QBO:
|
||||
const disabledRes = await postgresClient.query...(
|
||||
"SELECT disabled FROM integration_settings WHERE key = 'pax8'"
|
||||
);
|
||||
const isDisabled = disabledRes.rows[0]?.disabled === true;
|
||||
|
||||
Confirmed live: integration_settings has both a 'qbo' row and a 'mimecast' row
|
||||
(disabled = false for both).
|
||||
|
||||
Recent branches (device-link-reconcile, integration-health, appgate, tickets-reconcile,
|
||||
phishing-sweep, pax8-daily) use dynamic `await import('@/lib/services/...')` inside the
|
||||
branch. Match that style for the two new branches — it keeps the change localized and
|
||||
avoids adding top-of-file imports.
|
||||
</interfaces>
|
||||
|
||||
Live sync_schedules cron map (queried 2026-07-21) — used to pick a collision-free slot:
|
||||
0 2 * * * -> mimecast-sync (STALE, moving), qbo-sync-2am, veeam-full
|
||||
0 3 * * 0 -> weekly-full
|
||||
0 4 * * * -> contract-services, pax8-daily
|
||||
30 4 * * * -> tickets-reconcile
|
||||
0 5 * * * -> phishing-sweep
|
||||
15 * * * * -> device-link-reconcile (fires :15 EVERY hour — avoid minute 15)
|
||||
slash-30 -> veeam-* (fires :00 and :30 — avoid minutes 0 and 30)
|
||||
New target: 45 4 * * * (4:45 AM — minute 45 is unused anywhere; no collision)
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Add mimecast-sync and qbo dispatch branches to executeScheduledSync()</name>
|
||||
<files>lib/services/sync-scheduler.ts</files>
|
||||
<action>
|
||||
Two edits, both minimal and matching the existing if/else-if style. Do NOT refactor
|
||||
the chain into a lookup table and do NOT touch any other branch.
|
||||
|
||||
(a) Extend the sync_type union on line 25 (the ScheduleConfig.sync_type type) by
|
||||
adding 'mimecast-sync'. 'qbo' is already present — leave it. Without this, the string
|
||||
comparison in the new mimecast branch is a TS "no overlap" error.
|
||||
|
||||
(b) Insert two new else-if branches into the chain in executeScheduledSync(). Place
|
||||
them BEFORE the final `else if (config.sync_type === 'incremental')` and the final
|
||||
catch-all `else` block, so the catch-all remains reachable only for 'full'/legacy
|
||||
full-sync types. Do not alter the 'incremental' branch or the final else (generic
|
||||
fullSync) behavior.
|
||||
|
||||
Mimecast branch (mirror the engagement-daily/zoom-daily configured-gate pattern, but
|
||||
use dynamic import to match the recent branch style):
|
||||
- else if (config.sync_type === 'mimecast-sync')
|
||||
- Dynamically import isMimecastConfigured from @/lib/services/mimecast-client and
|
||||
runMimecastIncrementalSync from @/lib/services/mimecast-sync-service.
|
||||
- If not configured, console.log a skip line matching the engagement/zoom wording,
|
||||
e.g. "[SCHEDULER] Skipping mimecast-sync — Mimecast not configured".
|
||||
- Otherwise call runMimecastIncrementalSync(), capture the result, and log a one-line
|
||||
summary using the ACTUAL MimecastSyncResult fields (do not invent fields):
|
||||
messagesUpserted, threatsUpserted, bodiesFetched, purgedMessages,
|
||||
errors.length, durationMs — following the one-line summary format used by the
|
||||
phishing-sweep and device-link-reconcile branches.
|
||||
|
||||
QBO branch (mirror the pax8-daily integration_settings disable-check pattern):
|
||||
- else if (config.sync_type === 'qbo')
|
||||
- Query "SELECT disabled FROM integration_settings WHERE key = 'qbo'" exactly like the
|
||||
pax8 branch, compute isDisabled = rows[0]?.disabled === true.
|
||||
- If disabled, console.log "[SCHEDULER] Skipping qbo sync — QBO disabled via /admin/integrations"
|
||||
and do nothing else.
|
||||
- Otherwise dynamically import getQboSyncService from @/lib/services/qbo-sync-service
|
||||
and call getQboSyncService().incrementalSync('scheduled'). Use incrementalSync, not
|
||||
fullSync — that matches the manual /api/qbo/sync non-full path and is correct for a
|
||||
twice-daily recurring job. QBO has no env-var configured check like pax8 (it uses
|
||||
stored OAuth tokens), so only the disabled check is needed.
|
||||
|
||||
Because both branches replace a fall-through to the SyncService singleton, this also
|
||||
removes mimecast-sync and qbo from contending for that mutex.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /opt/stacks/pulse && npx tsc --noEmit --pretty 2>&1 | tail -5 && grep -c "config.sync_type === 'mimecast-sync'" lib/services/sync-scheduler.ts && grep -c "getQboSyncService" lib/services/sync-scheduler.ts && grep -c "runMimecastIncrementalSync" lib/services/sync-scheduler.ts</automated>
|
||||
</verify>
|
||||
<done>
|
||||
tsc passes clean; both new branches present; catch-all fullSync and incremental
|
||||
branches unchanged; 'mimecast-sync' added to the sync_type union. No test file exists
|
||||
for sync-scheduler (confirmed — only mimecast-client.test.ts), so no test changes are
|
||||
made, per the constraint against adding a new scheduler test file.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Create migration 101 to reschedule mimecast-sync + apply to live DB</name>
|
||||
<files>migrations/101_reschedule_mimecast_sync.sql</files>
|
||||
<action>
|
||||
Create migrations/101_reschedule_mimecast_sync.sql following the repo's schedule-table
|
||||
migration precedent (migrations/098_phishing_sweep_schedule.sql,
|
||||
migrations/090_ticket_reconcile_schedule.sql) — but an UPDATE, not an INSERT, since the
|
||||
mimecast-sync row already exists.
|
||||
|
||||
The UPDATE must be safe to re-run and must NOT clobber an admin's manual change. Guard
|
||||
it on the stale value so it is a no-op if already moved:
|
||||
UPDATE sync_schedules
|
||||
SET cron_expression = '45 4 * * *', updated_at = NOW()
|
||||
WHERE id = 'mimecast-sync' AND cron_expression = '0 2 * * *';
|
||||
|
||||
Add a leading SQL comment block explaining WHY (2 AM 3-way collision with qbo-sync-2am
|
||||
+ veeam-full, plus SyncService mutex contention) — matching the commenting style of
|
||||
migration 098.
|
||||
|
||||
Chosen new time: 45 4 * * * (4:45 AM). Verified collision-free against the live table:
|
||||
minute 45 is used by nothing; it avoids 0 4 (contract-services, pax8-daily), 30 4
|
||||
(tickets-reconcile), the 15 * * * * hourly device-link job, and the */30 :00/:30 veeam
|
||||
jobs.
|
||||
|
||||
Then apply the same UPDATE directly to the live container so it takes effect without
|
||||
waiting for a fresh-volume Postgres init (per CLAUDE.md, migrations only auto-apply on
|
||||
first volume boot). Run:
|
||||
docker exec pulse-postgres psql -U pulse_user -d pulse_autotask -c "UPDATE sync_schedules SET cron_expression = '45 4 * * *', updated_at = NOW() WHERE id = 'mimecast-sync' AND cron_expression = '0 2 * * *';"
|
||||
|
||||
Note in the SUMMARY: the running in-memory cron task keeps the old time until the app
|
||||
container restarts (the scheduler re-loads schedules from the DB on init) or
|
||||
reloadAllSchedules() is invoked. Deploying the Task 1 code change restarts the
|
||||
container, which reloads the new cron from the DB — so no separate restart step is
|
||||
needed as long as the code fix is deployed.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /opt/stacks/pulse && test -f migrations/101_reschedule_mimecast_sync.sql && grep -q "UPDATE sync_schedules" migrations/101_reschedule_mimecast_sync.sql && echo MIGRATION_FILE_OK && docker exec pulse-postgres psql -U pulse_user -d pulse_autotask -t -c "SELECT cron_expression FROM sync_schedules WHERE id = 'mimecast-sync';"</automated>
|
||||
</verify>
|
||||
<done>
|
||||
migrations/101_reschedule_mimecast_sync.sql exists with a guarded UPDATE keyed on the
|
||||
stale '0 2 * * *' value; live DB query returns '45 4 * * *' for mimecast-sync; no other
|
||||
schedule row modified.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
- npx tsc --noEmit --pretty passes clean.
|
||||
- executeScheduledSync() contains a 'mimecast-sync' branch calling runMimecastIncrementalSync()
|
||||
behind isMimecastConfigured(), and a 'qbo' branch calling getQboSyncService().incrementalSync('scheduled')
|
||||
behind an integration_settings disabled check.
|
||||
- The generic catch-all and the 'incremental' branch behave exactly as before.
|
||||
- No existing branch (phishing-sweep, pax8-daily, veeam-*, etc.) is modified.
|
||||
- Live sync_schedules shows mimecast-sync at '45 4 * * *' with no cron collision.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Mimecast and QBO scheduled jobs run their real sync logic on schedule.
|
||||
- The 0 2 * * * three-way collision is eliminated (mimecast-sync moved to 4:45 AM).
|
||||
- Migration 101 is safe to re-run and safe against an admin's manual cron change.
|
||||
- Type check passes; change is minimal and matches existing dispatch style.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/quick/260721-fy8-fix-mimecast-and-qbo-sync-scheduler-disp/260721-fy8-SUMMARY.md` when done.
|
||||
</output>
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
---
|
||||
phase: quick-260721-fy8
|
||||
plan: 01
|
||||
subsystem: sync-scheduler
|
||||
tags: [scheduler, mimecast, qbo, migration, bugfix]
|
||||
key-decisions:
|
||||
- mimecast-sync and qbo dispatch branches use incrementalSync (not fullSync) — matches the manual /api/qbo/sync non-full path and is correct for a twice-daily recurring job
|
||||
- Chose 45 4 * * * for mimecast-sync's new cron slot — minute 45 is unused anywhere in the live sync_schedules table, avoiding 0 4 (contract-services, pax8-daily), 30 4 (tickets-reconcile), the hourly :15 device-link job, and the veeam */30 :00/:30 jobs
|
||||
- Migration 101 guards the UPDATE on the stale '0 2 * * *' value so it's a no-op if already moved and won't clobber an admin's manual schedule change made via /admin after this ships
|
||||
status: complete
|
||||
---
|
||||
|
||||
# Quick Task 260721-fy8: Fix mimecast and qbo sync scheduler dispatch and reschedule mimecast cron
|
||||
|
||||
**One-liner:** Added missing `mimecast-sync` and `qbo` dispatch branches to `sync-scheduler.ts` (both previously fell through to a generic Autotask full sync, silently never running their real sync logic) and rescheduled `mimecast-sync` off the 2am three-way cron collision with `qbo-sync-2am` and `veeam-full`.
|
||||
|
||||
## What was done
|
||||
|
||||
### Task 1 — Dispatch branches (`lib/services/sync-scheduler.ts`, commit `9311f10`)
|
||||
- Extended the `ScheduleConfig.sync_type` union to add `'mimecast-sync'` (`'qbo'` was already present).
|
||||
- Added an `else if (config.sync_type === 'mimecast-sync')` branch: dynamically imports `isMimecastConfigured` from `mimecast-client.ts`; if not configured, logs a skip line matching the engagement/zoom wording. Otherwise dynamically imports and calls `runMimecastIncrementalSync()` from `mimecast-sync-service.ts`, logging a one-line summary using the real `MimecastSyncResult` fields (`messagesUpserted`, `threatsUpserted`, `bodiesFetched`, `purgedMessages`, `errors.length`, `durationMs`).
|
||||
- Added an `else if (config.sync_type === 'qbo')` branch: queries `integration_settings` for `key = 'qbo'` (mirroring the `pax8-daily` disable-check pattern exactly); if disabled, logs a skip line; otherwise dynamically imports `getQboSyncService` and calls `.incrementalSync('scheduled')`.
|
||||
- Both new branches sit before the final `'incremental'` branch and catch-all `else`, which are unchanged — the catch-all remains reachable only for `'full'`/legacy full-sync types.
|
||||
- `npx tsc --noEmit --pretty` passes clean.
|
||||
|
||||
### Task 2 — Reschedule migration (`migrations/101_reschedule_mimecast_sync.sql`, commit `f447ac3`)
|
||||
- New guarded migration: `UPDATE sync_schedules SET cron_expression = '45 4 * * *', updated_at = NOW() WHERE id = 'mimecast-sync' AND cron_expression = '0 2 * * *'` — a no-op if already moved, safe against clobbering a manual admin change.
|
||||
- Applied the same UPDATE directly against the live `pulse-postgres` container (migrations only auto-apply on first volume boot per CLAUDE.md) — confirmed live: `mimecast-sync` now reads `45 4 * * *`; `qbo-sync-2am` and `veeam-full` unchanged at `0 2 * * *` (collision eliminated for mimecast-sync specifically, as intended — qbo-sync-2am and veeam-full no longer contend with mimecast-sync, though qbo-sync-2am and veeam-full still share 2am with each other, which was out of scope for this task).
|
||||
|
||||
## Verification
|
||||
|
||||
- `npx tsc --noEmit --pretty` — clean.
|
||||
- Live DB: `SELECT cron_expression FROM sync_schedules WHERE id = 'mimecast-sync'` → `45 4 * * *`.
|
||||
- No existing branch (`phishing-sweep`, `pax8-daily`, `veeam-*`, `incremental`, catch-all) modified.
|
||||
- No test file exists for `sync-scheduler.ts` (only `mimecast-client.test.ts`), so no test changes were made, per plan constraint.
|
||||
|
||||
## Deviations from plan
|
||||
|
||||
None. Both tasks executed exactly as planned.
|
||||
|
||||
## Note on this SUMMARY.md
|
||||
|
||||
This file was originally written by the executor inside its isolated git worktree but was lost when the orchestrator removed the worktree (`git worktree remove --force`) without first running the standard rescue-before-remove step. Reconstructed immediately after from the executor's final report text and the actual `git show` output of both commits — content is accurate but not verbatim the original (same recovery situation documented previously in this repo for quick task 260717-a19).
|
||||
156
.planning/reports/MILESTONE_SUMMARY-v3.0.md
Normal file
156
.planning/reports/MILESTONE_SUMMARY-v3.0.md
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
# Milestone v3.0 — Phishing Triage Automation — Project Summary
|
||||
|
||||
**Generated:** 2026-07-18
|
||||
**Purpose:** Team onboarding and project review
|
||||
|
||||
---
|
||||
|
||||
## 1. Project Overview
|
||||
|
||||
Pulse is Wulf Consulting's internal PSA management dashboard (Next.js 16 + Postgres, syncing Autotask data). The v3.0 milestone's core value proposition:
|
||||
|
||||
> A manager/security operator can see every phishing/spam report ticket automatically triaged, deduplicated into campaigns, and classified — with any destructive remediation gated behind explicit human approval.
|
||||
|
||||
**Problem it solves:** Wulf's technicians receive a constant stream of Autotask tickets from KnowBe4 phish-alert-button reports, Microsoft's built-in "Report Message" flow, and manual spam/phishing complaints. Historically each ticket was triaged by hand — no deduplication across employees reporting the same campaign, no consistent severity call, no blast-radius visibility into Mimecast, and remediation notes/actions were ad hoc.
|
||||
|
||||
**Who it's for:** Wulf's security operators/technicians (triaging tickets day to day) and admins (configuring per-client automation).
|
||||
|
||||
**What was built:** An end-to-end pipeline — detect phishing/spam-report tickets → extract `.eml` evidence → look up Mimecast blast radius → deterministically classify (SPAM/UNWANTED/THREAT/USER_AWARENESS) → group duplicates into campaigns → propose (never auto-execute) remediation behind human approval → post a sanitized Autotask triage note → surface it all on a ticket-ID-addressable review page reachable via a real, production-confirmed Autotask LiveLink button → optionally automate the whole chain per-client via an opt-in gate.
|
||||
|
||||
**Status:** Shipped 2026-07-17. 9 phases, 30 plans, 69 tasks. All 38 v1 requirements complete.
|
||||
|
||||
**Explicit non-goals for this milestone** (deferred to a future v2):
|
||||
- No automatic execution of remediation — every action (block sender, purge mailbox, revoke sessions, etc.) is *proposed* and *simulated* only; nothing destructive runs against a real mail/identity system yet (`REMEDEXEC-01..05`).
|
||||
- No URL sandbox/detonation or URL reputation lookup (`ENRICH-01`).
|
||||
- No LLM-backed classification layer — this milestone is intentionally deterministic (`ENRICH-02` reserves the interface for later).
|
||||
- No fully automated ticket closure, no auto-created parent incidents, no auto-closed duplicates.
|
||||
- No backfill of historical phishing tickets (~267 pre-existing tickets) — detection is forward-only from ship date.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture & Technical Decisions
|
||||
|
||||
| Decision | Why | Phase |
|
||||
|---|---|---|
|
||||
| **Zero-LLM deterministic classifier** | Structured, size-bounded evidence in → deterministic verdict out. Avoids a prompt-injection surface from attacker-controlled email content ever reaching an LLM. | 19 |
|
||||
| **Content-hash idempotency scoped to title+description only** | SHA-256 over `{title, description}` — excludes status/assignee/last-activity so routine ticket churn never re-triggers detection, while evidence (notes/time-entries/attachments) still refreshes on every rescan regardless of hash match. | 15 |
|
||||
| **3-tier `.eml` selection** (`rfc.eml` exact → single non-wrapper `message/rfc822` candidate → `OriginatingEmail.eml` fallback) | Empirically derived from sampling 15 real production phishing tickets — content-type alone never disambiguates, since every sampled attachment shares `message/rfc822`. | 16 |
|
||||
| **Hand-rolled SPF/DKIM/DMARC tokenizer**, not a library | The obvious library (`mailauth`) only exposes live-verification functions that perform DNS lookups and BIMI HTTP fetches — a hard violation of "never fetch anything from a message." | 16 |
|
||||
| **3-tier campaign grouping key**: Message-ID → attachment-hash/URL-domain + subject + sender + 24h window → sender + normalized-subject + client + 24h window | Message-ID is the strongest, cheapest key; the fallbacks catch mass-blast duplicates lacking a shared Message-ID. Campaigns are never merged after the fact — out of scope by design. | 18 |
|
||||
| **Mimecast blast-radius as a normalizing abstraction that never throws** | Unconditional fan-out over delivered/held/threat-event lookups, 5-minute Redis cache, degrades to an explicit `unavailable` state on any failure or missing config — the rest of the pipeline never blocks on Mimecast being down. | 17 |
|
||||
| **THREAT requires BOTH delivery AND a malicious signal** | An auth failure alone on a message Mimecast fully held (reached no one) isn't a realized threat. Malicious signal = hard SPF/DKIM/DMARC fail OR the same attachment-hash/URL recurring across ≥2 campaign reports. | 19 |
|
||||
| **Simulation-vendor allowlist is a code constant, not a DB table** | Every classification rule, including the allowlist, is unit-tested pure code rather than a runtime-editable table that could silently drift. Matches exact domain or proper subdomain only — never substring — so a lookalike domain can't slip through. | 19 |
|
||||
| **Confidence = additive point-deduction from 1.0** | Each deduction (no parsed message, no/failed Mimecast lookup, no attachment/URL indicators) names the specific missing evidence in the response, rather than an opaque score. | 19 |
|
||||
| **All remediation actions are simulated status-only transitions this milestone** | No real provider methods exist yet (no block/purge on the Mimecast client, no forwarding-rule/reset on the Graph client) — real execution deliberately deferred to a future v2. | 20 |
|
||||
| **`acknowledge_user` is the one non-destructive action carve-out** | It's a customer-visible "thanks for reporting" note, not a security action — the only action type the automation gate is allowed to auto-post without human approval. | 19/23 |
|
||||
| **Per-company automation gate is 3 independent opt-in booleans** (parse/classify/report), all off by default | Mirrors the "propose, don't execute" safety posture — a client only gets automation once an admin deliberately opts them in. | 23 |
|
||||
| **URLs render as inert copy-only text in the review UI, never a clickable link** | Stricter than the triage-note's sanitize-and-describe approach, because this is an interactive page an operator could actually click into. | 22 |
|
||||
| **Client-side permission checks call the exact same permission function the server uses** | No bespoke role checks in the UI — the server remains the sole real enforcement boundary; the client check is UX-only. | 22 |
|
||||
|
||||
---
|
||||
|
||||
## 3. Phases Delivered
|
||||
|
||||
| Phase | Name | Status | One-Liner |
|
||||
|-------|------|--------|-----------|
|
||||
| 15 | Data Model, Detection & Ticket Evidence | ✅ Complete | Durable 7-table schema + idempotent Autotask ticket scanner and base evidence capture |
|
||||
| 16 | EML/MIME Evidence Parser | ✅ Complete | Pure, I/O-free RFC822/MIME parser that never executes or fetches anything |
|
||||
| 17 | Mimecast Blast Radius Lookup | ✅ Complete | Normalized delivery-data abstraction that degrades gracefully when Mimecast isn't configured |
|
||||
| 18 | Campaign Grouping & Phishing Analysis API | ✅ Complete | Duplicate reports accumulate into campaigns automatically; first `/api/phishing/*` routes |
|
||||
| 19 | Classification Engine | ✅ Complete | Deterministic SPAM/UNWANTED/THREAT verdicts from bounded evidence — no LLM anywhere |
|
||||
| 20 | Remediation, Approval & Audit Safety | ✅ Complete | Proposed-only actions, permission-gated approve/remediate/mark-false-positive, fully audited |
|
||||
| 21 | Autotask Triage Note | ✅ Complete | Sanitized, human-readable internal note posted to every linked ticket |
|
||||
| 22 | Approval UI (LiveLink) | ✅ Complete* | Ticket-ID-addressable page, confirmed live as a real Autotask LiveLink target |
|
||||
| 23 | Classification Disposition + Per-Client Automation Gate | ✅ Complete | Dedicated non-destructive verdict for simulation-vendor reports + per-company opt-in automation |
|
||||
|
||||
*Phase 22 is code-complete and verified retroactively, but 5 manual browser click-through checks were never run — see Section 6.
|
||||
|
||||
---
|
||||
|
||||
## 4. Requirements Coverage
|
||||
|
||||
All 38 v1 requirements are marked **Complete** in `v3.0-REQUIREMENTS.md`.
|
||||
|
||||
| ID | Phase | Detail |
|
||||
|---|---|---|
|
||||
| DETECT-01/02/03 | 15/18 | Pattern match on 8 locked substrings; content-hash idempotency; on-demand single-ticket analyze endpoint |
|
||||
| EVID-01..04 | 15/16 | Ticket evidence capture; 3-tier `.eml` selection; full RFC822/MIME + auth-result normalization; sanitized body preview, zero-network invariant test-enforced |
|
||||
| CAMP-01..03 | 18 | 3-tier grouping key; report/recipient accumulation; campaign list + detail API |
|
||||
| BLAST-01/02 | 17 | Normalized blast-radius fan-out; graceful `unavailable` degradation |
|
||||
| CLASSIFY-01..06 | 19 | Verdict + confidence + reasons + actions; destructive-action invariant; evidence-naming confidence deductions; sender allowlist; classify API; bounded structured evidence only |
|
||||
| REMED-01..06 | 20 | No auto-execution path exists (grep-confirmed); approve API; remediate API (simulated); idempotent double-call handling; mark-false-positive with 409 guard; single audit writer |
|
||||
| NOTE-01 | 21 | Sanitized triage note via existing safe Autotask write path, per-ticket failure isolation |
|
||||
| ACCESS-01 | 18 | `requirePermission()` convention applied consistently from Phase 18 onward |
|
||||
| REVIEW-01..06 | 22 | Ticket-addressable route; timeline; evidence display; classification display; action area wired to APIs; client/server permission parity — all code-verified, live-browser pass outstanding |
|
||||
| CLASSDISP-01..03 | 23 | `USER_AWARENESS` verdict; customer-visible acknowledgment note; distinct UI badge |
|
||||
| AUTOGATE-01..03 | 23 | Automation-gate table + admin API; admin toggle UI; gated webhook chain (idempotency bug found and fixed pre-close) |
|
||||
|
||||
---
|
||||
|
||||
## 5. Key Decisions Log
|
||||
|
||||
See Section 2 for the full architecture decision table. Notable individual calls worth calling out for new contributors:
|
||||
|
||||
- **D-01 (Phase 19):** Deterministic classifier chosen over an LLM specifically to avoid attacker-controlled email content ever reaching a model prompt.
|
||||
- **D-05 (Phase 17):** A single global Mimecast client (not per-tenant) was an accepted v1 limitation, documented in the module itself — later partially addressed by a post-ship quick task adding optional per-tenant client injection, though the default automatic path still uses the global client.
|
||||
- **D-07 (Phase 18):** The EML parser is only invoked from the on-demand `/analyze` endpoint, not the automatic webhook/cron path — meaning fully automatic detection can only ever reach Tier 3 (weakest) campaign grouping until an operator has explicitly analyzed at least one report in that campaign.
|
||||
- **D-09 (Phase 22):** URLs are rendered as inert, copy-only text in the operator review UI — a deliberately stricter posture than the triage-note's sanitize-and-describe approach, because this page is one an operator might actually click around in.
|
||||
|
||||
---
|
||||
|
||||
## 6. Known Issues, Gaps, and Tech Debt
|
||||
|
||||
**Real defects caught and fixed before/around ship:**
|
||||
|
||||
1. **Phase 15 (pre-ship):** The webhook trigger originally branched on an Autotask payload field that real webhooks never populate — detection would have silently never fired against real data. Fixed by reading the ticket back from Postgres instead of trusting the payload shape.
|
||||
2. **Phase 18 (found via live production verification, 2026-07-16):** Re-analyzing a single-report campaign created a *second* campaign row with the same grouping key, and the abandoned original campaign's report count was never decremented. Fixed with a signal-diverged guard plus a shared decrement helper; proven via 8 regression tests and a live production re-verification.
|
||||
3. **Phase 23 (post-ship gap-closure, before milestone close):** Every additional report accumulating into an already-classified simulation campaign re-triggered a duplicate customer-visible "thanks for reporting" note. Fixed via an idempotent, audit-persisting wrapper — plus two second-order bugs found in review of that very fix (a stale-timestamp display bug, and a gap where the *manual* approval path had no awareness of an auto-posted note). All three fixed in one commit before close.
|
||||
|
||||
**Process gap:** Phase 22 shipped all 6 plans and was marked complete without ever running formal verification — discovered only during the milestone-close requirements traceability check. A retroactive pass found the code correct (6/6 requirements), but **5 manual browser click-through checks remain outstanding**: full state-machine walkthrough, approve/remediate/mark-false-positive end-to-end, non-privileged role gating, the Mimecast-unavailable rendering state, and visual confirmation of URL inertness/clipboard-copy. Treat the review UI as code-correct but not yet UI-hardened.
|
||||
|
||||
**Accepted, still-open items:**
|
||||
- No retry if a webhook-triggered ticket isn't yet in Postgres when detection runs (backstop is a cron sweep that ships **disabled by default**).
|
||||
- Webhook doesn't re-trigger detection on ticket update (deliberate, not a bug).
|
||||
- No floor on the campaign-grouping report-count decrement; a pre-existing race condition in the "already grouped" check.
|
||||
- The THREAT-tier "clicked > 0" escalation trigger was flagged as a reasoned proposal rather than an explicitly locked decision — never formally reconfirmed.
|
||||
- Two pre-existing, unrelated failing tests (`itglue-search.test.ts`) persist across the whole milestone — confirmed to predate v3.0, out of scope by convention.
|
||||
|
||||
**5 post-ship fixes** (2026-07-16 through 2026-07-18, all real production issues caught after ship):
|
||||
1. Mimecast blast-radius errors were silently swallowed into a false "clean" result; date-window and multi-tenant handling fixed.
|
||||
2. A confidence-percentage display bug showed maximum confidence (1.0) as "1% confidence" — the opposite of its true meaning.
|
||||
3. A real KnowBe4 campaign using rotating lookalike domains was misclassified because only one simulation-vendor domain was allowlisted; also fixed a parse double-invocation race and a too-early auto-parse timing issue.
|
||||
4. Added a new "mark as accidental report" resolution path (feature addition, not a bug fix).
|
||||
5. Mimecast's held-message lookup had no date bounds, so an entire unrelated hold queue could inflate a campaign's blast-radius numbers — fixed with date filtering and sender-relevance matching (false held-message count went from 15 → 1 → 0 across the two fixes).
|
||||
|
||||
---
|
||||
|
||||
## 7. Getting Started
|
||||
|
||||
**Where to look first:**
|
||||
- `lib/services/phishing-detector.ts` — the front door; what triggers everything downstream.
|
||||
- `lib/services/campaign-grouping-service.ts` — the trickiest logic in the milestone; read its bug history (Section 6, item 2) before touching it.
|
||||
- `lib/services/campaign-classifier.ts` — the deterministic rule engine.
|
||||
- `app/phishing/tickets/[ticketId]/page.tsx` plus `components/phishing/*` — the operator-facing review surface.
|
||||
- `lib/services/phishing-automation-gate.ts` + `app/admin/phishing-automation/page.tsx` — the per-client automation on/off switch.
|
||||
|
||||
**Run a manual sweep/analyze:**
|
||||
- Single ticket: `POST /api/phishing/tickets/{ticket_id}/analyze`
|
||||
- Full reconciliation sweep: `lib/services/phishing-sweep-service.ts` — wired to a daily cron schedule that **ships disabled by default**; enable via `/admin` sync schedules.
|
||||
- Reclassify: `POST /api/phishing/campaigns/{id}/classify`
|
||||
|
||||
**Tests:** All coverage lives under `lib/services/**` (matching this repo's stated test-coverage convention — no `app/**`/`components/**` coverage exists anywhere in Pulse). Run `npm test`. Whole-repo status at milestone close: 439/441 passing (2 pre-existing, unrelated failures).
|
||||
|
||||
**Docs:** `docs/mimecast-api-guide.md` documents the held-message-queue date-range requirement and the gotcha behind the July 2026 false-positive fix — read before making further Mimecast changes.
|
||||
|
||||
---
|
||||
|
||||
## Stats
|
||||
|
||||
- **Timeline:** 2026-07-15 → 2026-07-17 (3 days), plus 5 post-ship follow-up fixes through 2026-07-18
|
||||
- **Phases:** 9 / 9 complete
|
||||
- **Requirements:** 38 / 38 complete
|
||||
- **Plans / Tasks:** 30 plans, 69 tasks
|
||||
- **Commits (repo-wide, since v2.0 close 2026-07-12):** 330
|
||||
- **Files changed (phishing-specific source paths):** 53 files, 9,382 insertions
|
||||
- **Tests:** 439 / 441 passing repo-wide at close
|
||||
- **Contributors:** lorentz
|
||||
215
.planning/reports/PHISHING_TRIAGE_LLM_CONTEXT.md
Normal file
215
.planning/reports/PHISHING_TRIAGE_LLM_CONTEXT.md
Normal file
|
|
@ -0,0 +1,215 @@
|
|||
# Phishing Triage Automation — LLM Context Document
|
||||
|
||||
> **How to use this document:** Paste this whole file into a new chat session as context
|
||||
> when you want to brainstorm, design, or spec out new features for Pulse's phishing
|
||||
> triage system. It is written to be self-contained and information-dense rather than
|
||||
> narrative — concrete function/table/route names are included so an LLM can reason
|
||||
> about extension points without guessing. Repo file paths are given as provenance
|
||||
> (`« path »`), not as something the reading model needs to fetch.
|
||||
>
|
||||
> System: **Pulse** — Wulf Consulting's internal PSA dashboard (Next.js 16 + PostgreSQL,
|
||||
> syncs Autotask PSA data). This document covers one subsystem: **v3.0 Phishing Triage
|
||||
> Automation**, shipped 2026-07-17, 9 phases / 30 plans / 69 tasks, all 38 requirements
|
||||
> complete, refined by 5 post-ship fixes through 2026-07-18.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core value proposition
|
||||
|
||||
A security operator sees every phishing/spam-report ticket automatically triaged,
|
||||
deduplicated into campaigns, and classified — with any destructive remediation gated
|
||||
behind explicit human approval. Nothing in this system currently executes a real
|
||||
destructive action against a mail or identity provider; every remediation action this
|
||||
milestone is proposed-and-simulated only.
|
||||
|
||||
**Trigger sources:** Autotask tickets created by (a) KnowBe4 phish-alert-button reports,
|
||||
(b) Microsoft's built-in "Report Message" flow forwarded into a mailbox that syncs to
|
||||
Autotask, (c) manual employee spam/phishing complaints.
|
||||
|
||||
**Pipeline (single sentence):** detect ticket → extract `.eml` evidence → look up
|
||||
Mimecast blast radius → deterministically classify → group into a campaign → propose
|
||||
remediation behind human approval → post sanitized Autotask note → surface on a
|
||||
ticket-ID-addressable review page reachable via a real Autotask LiveLink button →
|
||||
optionally automate per-client via an opt-in gate.
|
||||
|
||||
## 2. Explicit non-goals (this milestone) — deferred to a future "v2"
|
||||
|
||||
These are known-missing, not accidentally missing — a future feature request in this
|
||||
space should assume these don't exist yet:
|
||||
|
||||
- **No real remediation execution.** No code path blocks a sender in Mimecast, deletes/
|
||||
moves a Graph mailbox item, purges via Defender/Exchange, revokes a session, or resets
|
||||
a password. `REMEDEXEC-01..05` in the original requirements doc name these explicitly
|
||||
as out-of-scope-for-v1.
|
||||
- **No URL reputation/sandbox lookup** (`ENRICH-01`) — URLs are extracted and displayed,
|
||||
never scored or detonated.
|
||||
- **No LLM-backed classification** (`ENRICH-02`) — the classifier is 100% deterministic
|
||||
code; an LLM layer was deliberately reserved as a future plug-in behind the same
|
||||
interface, not built.
|
||||
- **No automatic ticket closure, no auto-created parent incident, no auto-closed
|
||||
duplicate tickets.**
|
||||
- **No backfill** of ~267 pre-existing phishing tickets that predate this system —
|
||||
detection is forward-only from ship date.
|
||||
- **No per-tenant Mimecast client by default** — a single global Mimecast client serves
|
||||
all companies (a documented v1 limitation; one post-ship quick task added *optional*
|
||||
per-tenant client injection, but the automatic/webhook path still uses the global
|
||||
client).
|
||||
|
||||
## 3. Data model
|
||||
|
||||
`« migrations/097_phishing_triage_schema.sql, 098, 099_indicators_metadata.sql, 100_phishing_automation_gate.sql »`
|
||||
|
||||
| Table | Purpose | Key columns |
|
||||
|---|---|---|
|
||||
| `campaigns` | One row per grouped phishing/spam incident | `id`, `campaign_key`, `group_method`, `first_seen_at`, `last_seen_at`, `report_count`, `status` (default `open`) |
|
||||
| `reports` | One row per detected ticket | `id`, `ticket_id` (FK, UNIQUE), `ticket_number`, `company_id`, `company_name`, `requester_contact_id`, `title`, `description`, `matched_patterns` JSONB, `content_hash`, `evidence` JSONB, `campaign_id` (FK, nullable) |
|
||||
| `messages` | Parsed `.eml` output, one per report | `id`, `report_id` (FK), `message_id`, `headers`/`urls`/`attachments` JSONB, `body_preview`, `raw_ref` (B2 object key or null) |
|
||||
| `indicators` | Per-attachment-hash / per-URL / per-sender correlation signals | `id`, `message_id` (FK), `indicator_type`, `value`, `metadata` JSONB |
|
||||
| `classifications` | Append-only classify-call history | `id`, `campaign_id` (FK), `verdict`, `confidence` NUMERIC, `summary`, `reasons` JSONB, `recommended_actions` JSONB, `requires_approval` BOOLEAN |
|
||||
| `remediation_actions` | Proposed/approved/completed remediation state | `id`, `campaign_id` (FK), `action_type`, `status` (default `proposed`), `params` JSONB, `approved_by`, `approved_at` |
|
||||
| `audit_events` | Single append-only audit trail for every state change | `id`, `campaign_id`, `actor`, `event_type`, `payload` JSONB, `created_at` |
|
||||
| `phishing_automation_gate` | Per-company opt-in automation flags | `company_id` (PK/FK), `auto_parse`/`auto_classify`/`auto_report` BOOLEAN (default false), `updated_by`, `updated_at` |
|
||||
|
||||
`sync_schedules` gains a seeded row `id='phishing-sweep'`, cron `0 5 * * *`, `is_enabled=false`.
|
||||
|
||||
**Verdict vocabulary:** `SPAM | UNWANTED | THREAT | USER_AWARENESS` (the last added in
|
||||
Phase 23, for confirmed simulation-vendor reports).
|
||||
|
||||
**Action vocabulary (7 types):** each maps to a destructive/non-destructive flag; only
|
||||
`acknowledge_user` is non-destructive (customer-visible "thanks for reporting" note) and
|
||||
is the sole action type an automation gate may auto-post without human approval.
|
||||
|
||||
## 4. API surface
|
||||
|
||||
All `/api/phishing/*` routes are `requirePermission('phishing', <action>)`-gated
|
||||
(`read | analyze | approve | remediate`). All `/api/admin/phishing-automation/*` routes
|
||||
are `requireAdmin()`-gated.
|
||||
|
||||
| Route | Method | Purpose |
|
||||
|---|---|---|
|
||||
| `/api/phishing/campaigns` | GET | Paginated campaign list |
|
||||
| `/api/phishing/campaigns/{id}` | GET | Full nested detail: reports/messages/indicators/classifications/remediation_actions/audit_events/timeline + fresh blast-radius |
|
||||
| `/api/phishing/campaigns/{id}/classify` | POST | On-demand classify + audit write |
|
||||
| `/api/phishing/campaigns/{id}/approve` | POST | Approve one or more recommended actions with param overrides |
|
||||
| `/api/phishing/campaigns/{id}/remediate` | POST | Execute (simulated) previously-approved actions, idempotent |
|
||||
| `/api/phishing/campaigns/{id}/mark-false-positive` | POST | Mark false-positive; 409 if any action already approved/completed |
|
||||
| `/api/phishing/campaigns/{id}/mark-accidental-report` | POST | Parallel resolution path + customer note (added post-ship) |
|
||||
| `/api/phishing/campaigns/{id}/triage-note` | POST | On-demand internal triage-note generation/post |
|
||||
| `/api/phishing/reports/{report_id}` | GET | Standalone (ungrouped) report evidence + blast radius |
|
||||
| `/api/phishing/tickets/{ticket_id}/campaign` | GET | Ticket→campaign resolver — **always 200**, `{found:false}` instead of 404 (so an Autotask LiveLink never dead-ends) |
|
||||
| `/api/phishing/tickets/{ticket_id}/analyze` | POST | On-demand detect→parse→group for one ticket |
|
||||
| `/api/admin/phishing-automation` | GET | List all companies + automation-gate flags |
|
||||
| `/api/admin/phishing-automation/{companyId}` | PATCH | Upsert a company's 3 gate flags |
|
||||
| `/api/admin/phishing-automation/{companyId}` | DELETE | Revert to all-off default |
|
||||
|
||||
## 5. UI surface
|
||||
|
||||
- `app/phishing/page.tsx` — campaign list (nav entry).
|
||||
- `app/phishing/tickets/[ticketId]/page.tsx` — the real review surface + **production-
|
||||
confirmed live Autotask LiveLink target**. 5-state machine: `loading | not-triaged |
|
||||
ungrouped | ready | error`. Composed of:
|
||||
- `ClassificationCard` — verdict badge, confidence %, summary, reasons, recommended
|
||||
actions, permission-gated reclassify.
|
||||
- `ActionAreaCard` — the only write-surface: checkbox-per-action approve with editable
|
||||
param overrides, confirm-dialog remediate/mark-false-positive/mark-accidental-report.
|
||||
Buttons always mounted, disabled+tooltipped when unauthorized or already resolved.
|
||||
- `EvidenceCard` — tabbed Headers (SPF/DKIM/DMARC) / URLs (**inert `<code>` text with
|
||||
copy-to-clipboard, never a clickable link — no `dangerouslySetInnerHTML` anywhere in
|
||||
this feature**) / Attachments (metadata only) / Body preview / Blast Radius (explicit
|
||||
`unavailable` branch).
|
||||
- `TimelineCard` — server-pre-sorted chronological feed.
|
||||
- `app/admin/phishing-automation/page.tsx` — company table, 3 independent `Switch`
|
||||
toggles per row (parse/classify/report).
|
||||
- Nav entry "Phishing" visible to all roles (everyone holds `phishing:read`).
|
||||
|
||||
## 6. Architecture decisions worth knowing before proposing new features
|
||||
|
||||
| # | Decision | Rationale |
|
||||
|---|---|---|
|
||||
| 1 | Zero-LLM deterministic classifier | Avoids attacker-controlled email content ever reaching a model prompt (prompt-injection surface). Any future AI-assisted classification should sit *behind* the existing verdict interface, not replace it, unless the injection risk is separately solved. |
|
||||
| 2 | Content-hash idempotency = SHA-256(title + description) only | Deliberately excludes status/assignee/timestamp churn. Evidence (notes/attachments) still refreshes every rescan even on an unchanged hash. |
|
||||
| 3 | 3-tier `.eml` selection | `rfc.eml` exact name → single non-wrapper `message/rfc822` candidate → `OriginatingEmail.eml` fallback. Content-type alone can't disambiguate (empirically confirmed against 15 real tickets). |
|
||||
| 4 | Hand-rolled SPF/DKIM/DMARC tokenizer | The obvious library (`mailauth`) forces live DNS/BIMI-HTTP lookups — violates the "never fetch anything from message content" invariant that runs through this whole feature. |
|
||||
| 5 | 3-tier campaign grouping key | Message-ID → attachment-hash/URL-domain + subject + sender + 24h window → sender + normalized-subject + client + 24h window. Campaigns are **never merged** after creation — explicitly out of scope. |
|
||||
| 6 | Mimecast blast-radius never throws | Unconditional parallel fan-out (delivered/held/threat-event lookups), 5-min Redis cache, degrades to an explicit `{status:'unavailable', reason:...}` rather than blocking the pipeline. |
|
||||
| 7 | THREAT requires delivery AND a malicious signal | An auth failure alone on a fully-held message (reached nobody) stays UNWANTED. Malicious signal = hard SPF/DKIM/DMARC fail OR the same attachment-hash/URL recurring across ≥2 reports in the campaign — no external reputation service involved. |
|
||||
| 8 | Simulation-vendor allowlist is a TypeScript constant | Not a DB table — every rule stays unit-testable pure code. Exact-domain-or-proper-subdomain match only, never substring, to resist lookalike-domain spoofing. |
|
||||
| 9 | Confidence = point-deduction from 1.0 | Each deduction names the specific missing evidence (no parsed message / no Mimecast data / no indicators) rather than an opaque score — this is a literal requirement (`CLASSIFY-03`), not incidental. |
|
||||
| 10 | All remediation this milestone is simulated | No block/purge/reset methods exist yet on any integration client — a real "actually block this sender" feature is new integration work, not a flag flip. |
|
||||
| 11 | `acknowledge_user` is the one auto-postable, non-destructive action | Everything else requires human approval by design; a new "auto-post X" feature should default to requiring the same explicit per-client opt-in Phase 23 established. |
|
||||
| 12 | Per-company automation = 3 independent booleans, default off | `auto_parse` / `auto_classify` / `auto_report`. Report meaningfully depends on classify, which depends on parse (informational ordering only, not enforced in code). |
|
||||
| 13 | URLs are inert, copy-only text in the operator UI | Stricter than the triage-note's sanitize-and-describe approach because this is an interactive page a human could click into. |
|
||||
| 14 | Client-side permission checks reuse the exact server permission function | No bespoke `role === 'admin'` string checks anywhere in this feature — server remains sole enforcement boundary. |
|
||||
|
||||
## 7. Known gaps / tech debt (don't silently rediscover these)
|
||||
|
||||
- **Phase 22 (Approval UI) has 5 outstanding manual browser click-through checks** never
|
||||
run: full state-machine walkthrough, approve/remediate/mark-false-positive end-to-end
|
||||
with re-render, non-privileged role gating, Mimecast-unavailable rendering, URL
|
||||
inertness/clipboard-copy visual confirmation. Code is verified; UI is not yet
|
||||
browser-hardened.
|
||||
- **No retry** if a webhook-triggered ticket row isn't yet in Postgres when detection
|
||||
runs — only a warning log; the backstop cron sweep **ships disabled by default**.
|
||||
- Webhook doesn't re-trigger detection on ticket *update*, only *create* (deliberate).
|
||||
- No floor (`GREATEST(..., 0)`) on the campaign report-count decrement; a pre-existing
|
||||
TOCTOU race in the "already grouped" check.
|
||||
- The THREAT-tier "clicked > 0" escalation trigger was a reasoned proposal from research,
|
||||
never formally re-confirmed as a locked decision.
|
||||
- The automatic (webhook/cron) path only ever reaches Tier-3 (weakest) campaign grouping
|
||||
until an operator has manually analyzed at least one report in that campaign — the EML
|
||||
parser is only invoked from the on-demand analyze endpoint, decision #5/#3 above.
|
||||
- Global (not per-tenant) Mimecast client remains the default for the automatic path.
|
||||
- 2 pre-existing, unrelated failing tests (`itglue-search.test.ts`) — confirmed to
|
||||
predate this feature entirely, not phishing-related.
|
||||
|
||||
**3 real defects already found and fixed** (useful precedent — similar bug classes are
|
||||
worth checking for in any new feature touching the same code):
|
||||
1. Webhook trigger branching on an Autotask payload field that real webhooks never
|
||||
populate (fixed by reading the ticket back from Postgres instead of trusting payload
|
||||
shape).
|
||||
2. Re-analyzing a single-report campaign created a duplicate campaign row with an
|
||||
identical grouping key (a "no sibling match found" edge case fell through to
|
||||
"create new" instead of "keep existing").
|
||||
3. Every additional report accumulating into an already-classified simulation campaign
|
||||
re-triggered a duplicate customer-visible note (no idempotency guard on the
|
||||
auto-post path, and — found in review of that very fix — the parallel *manual*
|
||||
approval path had no equivalent guard either).
|
||||
|
||||
**5 post-ship refinements** (2026-07-16 → 2026-07-18): Mimecast blast-radius date-window
|
||||
+ silently-swallowed-error fix; a confidence-percentage display bug (1.0 confidence
|
||||
showed as "1%"); expanded simulation-vendor allowlist + parse-idempotency + auto-parse
|
||||
timing fix; new "mark as accidental report" action; Mimecast held-message false-positive
|
||||
fix (unbounded hold-queue lookup inflating blast-radius counts).
|
||||
|
||||
## 8. Entry points (for a human engineer, if this context is being used to spec work
|
||||
that will land back in this repo)
|
||||
|
||||
- `« lib/services/phishing-detector.ts »` — detection front door.
|
||||
- `« lib/services/campaign-grouping-service.ts »` — trickiest logic; has the most bug
|
||||
history (see gap #2 above).
|
||||
- `« lib/services/campaign-classifier.ts »` — deterministic rule engine.
|
||||
- `« app/phishing/tickets/[ticketId]/page.tsx »` + `« components/phishing/* »` —
|
||||
operator-facing review surface.
|
||||
- `« lib/services/phishing-automation-gate.ts »` + `« app/admin/phishing-automation/page.tsx »`
|
||||
— per-client automation on/off switch.
|
||||
- `« docs/mimecast-api-guide.md »` — documents the held-message-queue date-range
|
||||
requirement behind one of the post-ship fixes above.
|
||||
|
||||
## 9. Good directions to brainstorm from here
|
||||
|
||||
Given the explicit non-goals above, natural "what's next" conversation starters:
|
||||
- **Real remediation execution** (`REMEDEXEC-01..05`): actually calling Mimecast block,
|
||||
Graph mailbox purge/move, Defender/Exchange purge, behind the existing approve-gate —
|
||||
the approval/audit/idempotency scaffolding already exists and is designed to have a
|
||||
real executor slotted in later.
|
||||
- **URL reputation enrichment** (`ENRICH-01`): a scoring layer over the already-extracted,
|
||||
already-deduped indicator URLs.
|
||||
- **An LLM-assisted classification layer** (`ENRICH-02`), explicitly designed to sit
|
||||
behind the current deterministic verdict interface as an optional enhancement/second
|
||||
opinion — not a replacement, given decision #1's prompt-injection rationale.
|
||||
- **Per-tenant Mimecast client as the default**, not just an optional override, closing
|
||||
gap "Global (not per-tenant) Mimecast client."
|
||||
- **Automatic campaign grouping parity** with the on-demand path — closing the Tier-3-only
|
||||
limitation on the fully automatic pipeline.
|
||||
- **Historical backfill** of the ~267 pre-existing phishing tickets, if retroactive
|
||||
visibility becomes valuable.
|
||||
596
QBO_INTEGRATION_HANDOFF.md
Normal file
596
QBO_INTEGRATION_HANDOFF.md
Normal file
|
|
@ -0,0 +1,596 @@
|
|||
# QuickBooks Online Integration — Handoff Document
|
||||
|
||||
This document captures everything the Pulse team learned building a QuickBooks
|
||||
Online (QBO) integration, so a separate team building a new application can
|
||||
implement their own QBO connection with a head start. It assumes zero access
|
||||
to the Pulse codebase — every detail that matters is spelled out inline.
|
||||
Where useful, a parenthetical note says "(in Pulse this lives in …)" purely as
|
||||
provenance; treat those as historical footnotes, not as things you need to go
|
||||
look up.
|
||||
|
||||
Pulse's implementation was originally built March 2026, extended with
|
||||
Payment/Deposit write support in May 2026, and had a soft-delete + AR
|
||||
reconciliation fix land in July 2026 after a real production discrepancy (see
|
||||
Gotchas below). This document reflects the accumulated, working state as of
|
||||
July 2026.
|
||||
|
||||
## 1. Overview
|
||||
|
||||
QuickBooks Online is Intuit's hosted accounting SaaS. Pulse connects to a
|
||||
single QBO company ("realm") belonging to the business the dashboard serves,
|
||||
and uses the connection for two purposes:
|
||||
|
||||
1. **Read / sync (one-way, QBO → Pulse):** invoices, customer payments, bank
|
||||
deposits, purchases/expenses, journal entries, and pre-built financial
|
||||
reports (Profit & Loss, Balance Sheet, Cash Flow, and Aged Receivable
|
||||
reports) are pulled on a schedule and mirrored into Pulse's own database.
|
||||
This lets Pulse show finance dashboards, accounts-receivable aging, and
|
||||
engagement/finance widgets without hitting QBO's API on every page load.
|
||||
2. **Write (one-way, Pulse → QBO):** Pulse can also *create* records in QBO —
|
||||
specifically `Payment` (a "Receive Payment" applied against one or more open
|
||||
invoices) and `Deposit` (grouping several payments into a single bank
|
||||
deposit line so QBO's bank-feed reconciliation matches the actual bank
|
||||
deposit slip). This was added to support a semi-automated bank remittance
|
||||
reconciliation workflow: a CLI script parses a bank's remittance file,
|
||||
maps each check to an invoice, and posts Payments + a single Deposit to
|
||||
QBO via the API instead of a human re-keying each line into the QBO UI.
|
||||
|
||||
The integration is intentionally single-tenant: it connects to exactly one
|
||||
QBO company (one `realmId`) per deployment, configured via environment
|
||||
variables. There is no concept of "connect a QBO account per customer" — this
|
||||
was built for one business's own books, not as a multi-tenant SaaS feature.
|
||||
|
||||
## 2. Prerequisites
|
||||
|
||||
Before writing any code, set up an app in Intuit's developer ecosystem:
|
||||
|
||||
1. Create an account at the Intuit Developer portal (https://developer.intuit.com)
|
||||
and create a new app under "My Apps".
|
||||
2. Choose the **QuickBooks Online and Payments** (or just **QuickBooks
|
||||
Online Accounting API**) scope for the app — this is what exposes the
|
||||
`com.intuit.quickbooks.accounting` OAuth scope.
|
||||
3. Every Intuit app ships with **two separate credential sets**: one for the
|
||||
**Sandbox** environment (a fake QBO company you can safely test writes
|
||||
against) and one for **Production** (real customer/company data). Treat
|
||||
these as two entirely different client id/secret pairs — do not assume
|
||||
sandbox credentials work against the production API base URL or vice
|
||||
versa.
|
||||
4. Register a **redirect URI** (Intuit calls this the "Redirect URI" under the
|
||||
app's Keys & OAuth section) that exactly matches the callback URL your app
|
||||
will use — e.g. `https://your-app.example.com/api/qbo/auth`. Intuit
|
||||
validates this on every authorization request; a mismatch (including
|
||||
trailing slash or http vs https) fails the authorization step outright.
|
||||
5. Intuit requires a **published EULA and Privacy Policy URL** on the app
|
||||
listing before you can move an app from Sandbox-only to Production/Go Live
|
||||
— Pulse added simple static `/legal/eula` and `/legal/privacy` pages
|
||||
specifically to satisfy Intuit's app review/assessment step. Budget time
|
||||
for this if you haven't already published one.
|
||||
6. Decide up front which QBO scope(s) you need. Pulse only ever requests
|
||||
`com.intuit.quickbooks.accounting` (the general accounting scope that
|
||||
covers invoices, payments, deposits, purchases, journal entries, and
|
||||
reports). If you also need QBO Payments (charging cards) that's a
|
||||
different, additional scope not covered here.
|
||||
|
||||
## 3. Environment variables
|
||||
|
||||
Pulse's QBO code reads the following variables directly via `process.env` —
|
||||
these are the *real* names in use, discovered by grepping the source (not
|
||||
assumed from a `.env.example`, since none of these currently appear in
|
||||
Pulse's committed `.env` — meaning this integration is present in code but
|
||||
not yet live-configured in this particular deployment).
|
||||
|
||||
| Variable | Purpose | Example / placeholder |
|
||||
|---|---|---|
|
||||
| `QBO_CLIENT_ID` | Intuit app's OAuth2 client id (Sandbox or Production, depending on `QBO_SANDBOX`) | `ABxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` |
|
||||
| `QBO_CLIENT_SECRET` | Intuit app's OAuth2 client secret (matching the same environment as the client id) | `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` |
|
||||
| `QBO_REALM_ID` | The QBO company id ("realm") this deployment talks to. Received once from Intuit on the OAuth callback, then hardcoded into config for a single-tenant deployment | `123146xxxxxxxxx` |
|
||||
| `QBO_SANDBOX` | String toggle. When exactly `"true"`, the client points at the Sandbox API base URL; any other value (including unset) means Production | `true` or `false` |
|
||||
|
||||
Two things worth calling out explicitly:
|
||||
|
||||
- Pulse's health-check / integration-status layer only checks for
|
||||
`QBO_CLIENT_ID` and `QBO_CLIENT_SECRET` being present to decide whether to
|
||||
show QBO as "configured" in its admin integrations UI — it does **not**
|
||||
separately validate `QBO_REALM_ID`. A new app should validate all three
|
||||
(`CLIENT_ID`, `CLIENT_SECRET`, `REALM_ID`) before attempting any API call,
|
||||
because the actual `QboClient` constructor throws if any of the three is
|
||||
missing.
|
||||
- The OAuth redirect-URI construction (see next section) reads a **fourth**,
|
||||
unrelated env var — `NEXTAUTH_URL` — for the base URL used to build both
|
||||
the callback redirect URI sent to Intuit and the post-auth redirect back
|
||||
into the admin UI. This is a legacy holdover from an earlier auth library
|
||||
Pulse used before switching to its current auth stack; it is **not** the
|
||||
same variable as the app's primary "base URL" env var used everywhere else
|
||||
in the codebase. In Pulse's actual `.env` this variable is not set at all,
|
||||
which means in production the redirect URI silently becomes
|
||||
`undefined/api/qbo/auth` unless something else sets `NEXTAUTH_URL`. **Do
|
||||
not repeat this mistake** — in your new app, use a single canonical
|
||||
"public base URL" env var for constructing OAuth redirect URIs, and make
|
||||
sure it's actually present wherever the OAuth code runs.
|
||||
|
||||
No `client id`/`secret` values from Pulse's real `.env` are reproduced here —
|
||||
`.env` did not contain `QBO_*`/`QUICKBOOKS_*`/`INTUIT_*` keys at all at the
|
||||
time this document was written, so there was nothing to redact; the examples
|
||||
above are placeholders only.
|
||||
|
||||
## 4. OAuth2 connection flow
|
||||
|
||||
QBO uses the standard OAuth2 **authorization-code** grant. Two Intuit
|
||||
endpoints matter:
|
||||
|
||||
- **Authorize URL:** `https://appcenter.intuit.com/connect/oauth2`
|
||||
- **Token URL (exchange + refresh):** `https://oauth.platform.intuit.com/oauth2/v1/tokens/bearer`
|
||||
|
||||
Step by step, this is exactly what Pulse's implementation does:
|
||||
|
||||
1. **User clicks "Connect to QBO"** in the admin UI, which hits a single
|
||||
backend endpoint (in Pulse: `GET /api/qbo/auth`, with no query params).
|
||||
2. **Server builds the authorize redirect.** It generates a `state` value
|
||||
(Pulse currently uses `Math.random().toString(36).slice(2)` — cryptographically
|
||||
weak; a new app should use a proper random token, e.g. `crypto.randomUUID()`
|
||||
or a signed value, and *should* persist/verify it server-side before
|
||||
trusting the callback — Pulse's implementation generates `state` but does
|
||||
**not** persist or verify it on callback, which is a real CSRF gap worth
|
||||
closing in a new implementation). It then redirects the browser to:
|
||||
```
|
||||
https://appcenter.intuit.com/connect/oauth2
|
||||
?client_id=<QBO_CLIENT_ID>
|
||||
&scope=com.intuit.quickbooks.accounting
|
||||
&redirect_uri=<url-encoded redirect URI, e.g. https://your-app.example.com/api/qbo/auth>
|
||||
&response_type=code
|
||||
&state=<state>
|
||||
```
|
||||
3. **User authenticates with Intuit and approves the connection**, choosing
|
||||
which QBO company ("realm") to connect if they have access to more than
|
||||
one.
|
||||
4. **Intuit redirects back** to the exact `redirect_uri` registered on the
|
||||
Intuit app, appending query params: `code` (the authorization code),
|
||||
`realmId` (the QBO company id the user just authorized — this is how you
|
||||
discover the realm; it is not something you choose ahead of time unless
|
||||
you already know the company), and `state` (echoed back — verify it here).
|
||||
If the user declines or something goes wrong, Intuit instead appends an
|
||||
`error` param and no `code`/`realmId`.
|
||||
5. **Server exchanges the code for tokens.** POST to the token URL:
|
||||
- Header: `Authorization: Basic <base64(client_id:client_secret)>`
|
||||
- Header: `Content-Type: application/x-www-form-urlencoded`
|
||||
- Header: `Accept: application/json`
|
||||
- Body (form-encoded): `grant_type=authorization_code&code=<code>&redirect_uri=<same redirect_uri used in step 2, byte-for-byte>`
|
||||
|
||||
The redirect_uri sent here **must match** the one used in the initial
|
||||
authorize redirect exactly, or Intuit rejects the exchange.
|
||||
6. **Intuit responds with a token payload:**
|
||||
```json
|
||||
{
|
||||
"access_token": "...",
|
||||
"refresh_token": "...",
|
||||
"expires_in": 3600,
|
||||
"x_refresh_token_expires_in": 8726400,
|
||||
"token_type": "bearer"
|
||||
}
|
||||
```
|
||||
`expires_in` is in seconds and is **short** — access tokens are valid for
|
||||
1 hour. `x_refresh_token_expires_in` is also in seconds and is **long**
|
||||
— the refresh token is valid for 100 days (8,726,400 seconds) from
|
||||
issuance, but note: **every time you use a refresh token to get a new
|
||||
access token, Intuit issues a brand-new refresh token as well**, and the
|
||||
100-day clock resets. You must persist the new refresh token every single
|
||||
time or the old one on file will eventually go stale and you'll be forced
|
||||
back through the full user-consent flow (step 1).
|
||||
7. **Persist the tokens** (see section 5) keyed by `realmId`, computing
|
||||
absolute expiry timestamps from `expires_in` / `x_refresh_token_expires_in`
|
||||
at the moment of exchange (`now + expires_in seconds`, etc.) rather than
|
||||
storing the raw relative seconds.
|
||||
8. **Redirect the user back into your own app's UI** (not back to Intuit) —
|
||||
e.g. an admin settings page — with a success indicator.
|
||||
9. **On every subsequent API call**, before hitting QBO, check whether the
|
||||
stored access token's expiry has passed. If it has, and the refresh token
|
||||
is still valid, silently call the **refresh** grant (same token URL, same
|
||||
Basic-auth header) with `grant_type=refresh_token&refresh_token=<stored
|
||||
refresh token>`, persist the new access + refresh token pair, and proceed.
|
||||
If the *refresh* token itself has also expired, there is no way to
|
||||
recover programmatically — you must re-run the full authorize flow (step
|
||||
1) with a human present.
|
||||
|
||||
Pulse's implementation checks and refreshes **proactively** (before each API
|
||||
call, based on stored expiry timestamps) rather than reactively (catching a
|
||||
401 from the API and retrying) — this is simpler and avoids ever sending a
|
||||
guaranteed-to-fail request, at the cost of trusting your stored clock/expiry
|
||||
bookkeeping to be accurate.
|
||||
|
||||
## 5. Token storage & refresh
|
||||
|
||||
Pulse stores exactly one row per connected realm in a dedicated tokens table.
|
||||
Generalized (not Postgres-specific — use whatever your new app's data layer
|
||||
supports, even a simple encrypted-at-rest key/value store), the record shape
|
||||
that must be persisted is:
|
||||
|
||||
| Field | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `realm_id` | string, unique | The QBO company id this token pair belongs to. Unique constraint because Pulse only ever supports one active connection per realm |
|
||||
| `access_token` | string (secret) | Short-lived bearer token sent on every API call |
|
||||
| `refresh_token` | string (secret) | Long-lived token used to mint new access tokens |
|
||||
| `access_token_expires_at` | timestamp | `now + expires_in seconds` at time of exchange/refresh |
|
||||
| `refresh_token_expires_at` | timestamp | `now + x_refresh_token_expires_in seconds` at time of exchange/refresh |
|
||||
| `created_at` / `updated_at` | timestamp | Standard audit columns |
|
||||
|
||||
Refresh strategy (generalized pseudocode):
|
||||
|
||||
```
|
||||
function getValidAccessToken(realmId):
|
||||
token = loadToken(realmId)
|
||||
if token is null:
|
||||
raise "not connected — run OAuth flow"
|
||||
if now() < token.access_token_expires_at:
|
||||
return token.access_token
|
||||
if now() >= token.refresh_token_expires_at:
|
||||
raise "refresh token expired — re-authorization required"
|
||||
return refreshAccessToken(token.refresh_token) # calls token URL, persists new pair, returns new access_token
|
||||
```
|
||||
|
||||
Treat both tokens as secrets: encrypt at rest if your storage layer supports
|
||||
it, never log them, and never return them in any API response (Pulse's
|
||||
"connection status" endpoint returns only a derived `valid | expired |
|
||||
missing` enum — never the token values themselves — and the codebase's own
|
||||
security posture doc calls out that this table's contents, plus the whole
|
||||
`.env` file, must never be echoed or logged verbatim).
|
||||
|
||||
**Disconnect is soft, not a full teardown.** Pulse's disconnect action only
|
||||
deletes the local token row; it does **not** call Intuit's token-revocation
|
||||
endpoint. This means Intuit's side may still consider the connection
|
||||
"authorized" until the refresh token naturally expires. A more correct
|
||||
implementation should also POST to Intuit's revoke endpoint
|
||||
(`https://developer.api.intuit.com/v2/oauth2/tokens/revoke`, same
|
||||
Basic-auth pattern, body `token=<access_token or refresh_token>`) as part of
|
||||
disconnect, so the authorization is actually torn down on Intuit's side too.
|
||||
|
||||
## 6. API usage
|
||||
|
||||
Base URLs (these differ **only** by the `sandbox-` prefix):
|
||||
|
||||
- **Sandbox:** `https://sandbox-quickbooks.api.intuit.com`
|
||||
- **Production:** `https://quickbooks.api.intuit.com`
|
||||
|
||||
Path shape for every accounting-entity call:
|
||||
|
||||
```
|
||||
{baseUrl}/v3/company/{realmId}/{resource-or-query}
|
||||
```
|
||||
|
||||
Pulse pins **`minorversion=65`** as a query parameter on every request. QBO's
|
||||
"minor version" controls which incremental API revision you get (new fields,
|
||||
behavior changes); pin an explicit value rather than omitting it, so QBO
|
||||
adding a new minor version later doesn't silently change your response
|
||||
shape. Check Intuit's current minor-version changelog before picking a value
|
||||
for a new integration — don't just copy `65` — but the pattern of pinning one
|
||||
explicit version is the right one to keep.
|
||||
|
||||
Every authenticated request needs:
|
||||
```
|
||||
Authorization: Bearer <valid access token>
|
||||
Accept: application/json
|
||||
Content-Type: application/json
|
||||
```
|
||||
|
||||
**Reading data** — QBO exposes a SQL-like query language over each entity.
|
||||
Example: list invoices, paginated (QBO caps each response at a max page
|
||||
size; keep paging with `STARTPOSITION`/`MAXRESULTS` until you get a short
|
||||
page back):
|
||||
|
||||
```
|
||||
GET {baseUrl}/v3/company/{realmId}/query
|
||||
?query=SELECT * FROM Invoice STARTPOSITION 1 MAXRESULTS 1000
|
||||
&minorversion=65
|
||||
```
|
||||
|
||||
For incremental sync, add a `WHERE` clause filtering on QBO's own
|
||||
last-modified metadata, e.g.:
|
||||
```
|
||||
SELECT * FROM Invoice WHERE MetaData.LastUpdatedTime > '2026-07-01T00:00:00.000Z'
|
||||
STARTPOSITION 1 MAXRESULTS 1000
|
||||
```
|
||||
|
||||
**Writing data** — POST the entity's JSON body to its dedicated endpoint,
|
||||
e.g. creating a Payment:
|
||||
```
|
||||
POST {baseUrl}/v3/company/{realmId}/payment?minorversion=65
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"CustomerRef": { "value": "<QBO customer id>" },
|
||||
"TotalAmt": 539.40,
|
||||
"TxnDate": "2026-05-18",
|
||||
"DepositToAccountRef": { "value": "<QBO bank account id>" },
|
||||
"PaymentRefNum": "0000997294",
|
||||
"Line": [
|
||||
{ "Amount": 539.40, "LinkedTxn": [ { "TxnId": "<invoice id>", "TxnType": "Invoice" } ] }
|
||||
]
|
||||
}
|
||||
```
|
||||
QBO echoes back the full created object (including its new `Id` and
|
||||
`SyncToken`) on success — always check for that `Id` in the response before
|
||||
considering the write successful; a 200-with-unexpected-body is possible and
|
||||
should be treated as a failure.
|
||||
|
||||
**Reports** — pre-aggregated reports (Profit & Loss, Balance Sheet, Cash
|
||||
Flow, Aged Receivable Detail/Summary) are separate GET endpoints under
|
||||
`/v3/company/{realmId}/reports/{ReportName}` with their own query params
|
||||
(`start_date`, `end_date`, `accounting_method`, `report_date`,
|
||||
`aging_method`, plus `showrows=all&showcols=all` to get fully expanded
|
||||
rows/columns rather than a collapsed summary). Reports come back as a deeply
|
||||
nested `Header` / `Columns` / `Rows` tree that must be walked recursively —
|
||||
there is no flat tabular shape. Fetch the company's accounting preference
|
||||
(`GET /v3/company/{realmId}/preferences?minorversion=65`, look at
|
||||
`Preferences.ReportPrefs.ReportBasis`, e.g. `"Accrual"` or `"Cash"`) once and
|
||||
pass it explicitly as `accounting_method` on P&L/Balance Sheet requests —
|
||||
don't assume Accrual.
|
||||
|
||||
**Rate limits / transient errors** — Intuit enforces per-app-per-realm rate
|
||||
limits (roughly 500 requests/minute per realm at time of writing — verify
|
||||
current limits in Intuit's docs, they change). Every QBO response includes
|
||||
an `intuit_tid` header (sometimes lowercased as `intuit-tid` depending on
|
||||
proxy/CDN normalization — check both) — capture it in any error you log or
|
||||
surface, since Intuit support requests require this transaction id to
|
||||
investigate a specific failed call.
|
||||
|
||||
## 7. Gotchas & lessons learned
|
||||
|
||||
These are real issues Pulse hit, in rough order of how much time they cost:
|
||||
|
||||
1. **QBO never reports deletions — you must diff to find them.** The
|
||||
Invoice (and other entity) query endpoints only return records that
|
||||
*currently exist* in QBO. When an invoice is voided or deleted in QBO, it
|
||||
simply stops appearing in query results going forward — there is no
|
||||
"deleted" flag, no tombstone record, nothing to subscribe to. If your
|
||||
sync only *upserts* what it gets back, deleted/voided invoices silently
|
||||
linger forever in your own database and keep inflating any aggregate
|
||||
(e.g. "Total Accounts Receivable") that sums over them. Pulse hit this in
|
||||
production (an invoice for one customer had been voided in QBO but kept
|
||||
counting toward Pulse's headline A/R for weeks). **Fix:** on every *full*
|
||||
sync, collect the full set of entity IDs QBO actually returned, then mark
|
||||
(soft-delete) any locally-stored row for that realm whose id was *not* in
|
||||
that set. Do **not** apply this same logic on an *incremental* sync
|
||||
(one that only asks for rows changed since a timestamp) — an incremental
|
||||
response is deliberately a small subset of all records, so diffing
|
||||
against "everything QBO returned this call" would wrongly tombstone your
|
||||
entire historical ledger. Only do the tombstone diff on syncs that fetch
|
||||
the complete current entity set.
|
||||
|
||||
2. **Reconcile against QBO's own reports periodically, not just raw entity
|
||||
sync.** Even with the tombstone fix above, subtle drift can still appear
|
||||
between "sum of invoice balances I have stored" and "what QBO's own Aged
|
||||
Receivable report says." Building a small reconciliation
|
||||
check — pull QBO's live Aged-Receivable-Detail report, pull your own
|
||||
stored open-invoice balances, and diff by invoice/document number in
|
||||
three buckets (present in QBO but missing locally = sync miss; present
|
||||
locally but missing from QBO = likely stale/tombstone candidate; present
|
||||
in both but balance differs = a data-mapping or timing bug) — is cheap
|
||||
insurance and caught issues that a pure "did the sync job succeed"
|
||||
check would not have surfaced.
|
||||
|
||||
3. **Report endpoints reject the wrong parameter combinations.** Early
|
||||
attempts to fetch P&L/Balance Sheet passed a `summarize_column_by`
|
||||
parameter that QBO's API rejected outright; the fix was to drop it and
|
||||
instead explicitly pass `accounting_method` (sourced from the company's
|
||||
real preference, see section 6) plus `showrows=all&showcols=all`. If a
|
||||
report call returns an error, check the exact param set against Intuit's
|
||||
current report-endpoint docs rather than assuming your first guess is
|
||||
right — report endpoints are pickier about params than the generic
|
||||
`query` endpoint.
|
||||
|
||||
4. **Empty report periods return a distinct "no data" marker, not an empty
|
||||
array.** When you request a report for a period where the company had no
|
||||
activity, QBO returns a response with `Header.NoReportData` set to the
|
||||
*string* `"true"` rather than a normal empty report body. If you don't
|
||||
check for this explicitly, you'll either crash trying to parse a
|
||||
report with no rows, or you'll store a garbage "empty" report row and
|
||||
have to distinguish "the business had zero P&L that month" from "we
|
||||
never successfully fetched this month" later. Check for
|
||||
`Header.NoReportData === 'true'` and skip persisting anything for that
|
||||
period.
|
||||
|
||||
5. **Sandbox and Production are separate universes with separate
|
||||
credentials.** It is easy to assume "sandbox" is just a URL flag with the
|
||||
same client id/secret — it isn't. Each Intuit app has independent
|
||||
Sandbox and Production OAuth client id/secret pairs, independent
|
||||
authorized companies, and independent data. A `QBO_SANDBOX=true` toggle
|
||||
only changes which API *base URL* you hit — you still need to swap the
|
||||
client id/secret pair to match, or every request will fail
|
||||
authentication. Keep the credential pair and the sandbox/production flag
|
||||
changed together, never independently.
|
||||
|
||||
6. **Bank check numbers don't round-trip cleanly.** When reconciling bank
|
||||
remittance files against invoices/payments, physical paper checks often
|
||||
have leading zeros in their check number (e.g. `0000996226`) that the
|
||||
bank's own electronic remittance file strips down to a bare number
|
||||
(`996226`). If you're matching check numbers between an external file and
|
||||
QBO/your own records, normalize both sides (e.g. strip leading zeros
|
||||
before comparing) or you'll get false "no match found" failures on
|
||||
otherwise-correct data.
|
||||
|
||||
7. **Idempotency for write operations needs its own mechanism — QBO does
|
||||
not give you one for free.** POSTing the same Payment or Deposit twice
|
||||
creates two separate records in QBO; there's no natural dedupe key QBO
|
||||
enforces on your behalf for a client-supplied write like an idempotency
|
||||
key header would provide. Any script or job that writes to QBO
|
||||
(Payments, Deposits, or otherwise) needs its own tracking of "have I
|
||||
already successfully posted this specific record" (Pulse's reconciliation
|
||||
script does this with a sibling JSON manifest that records the QBO-assigned
|
||||
id after each successful POST, and skips any item already recorded on
|
||||
re-run) so that retrying a partially-failed batch job doesn't duplicate
|
||||
the successful half.
|
||||
|
||||
8. **Lazy-construct the API client if any code path shouldn't require
|
||||
credentials.** A tool that has a "dry run" or "preview" mode (e.g.
|
||||
validate + print what *would* be sent, without actually calling QBO)
|
||||
should not eagerly construct the QBO client at the top of its execution
|
||||
path, because a client constructor that validates `CLIENT_ID` /
|
||||
`CLIENT_SECRET` / `REALM_ID` up front will throw immediately in
|
||||
environments where those aren't configured — even though the dry run
|
||||
never needed a real connection. Defer client construction to the exact
|
||||
point where you're about to make a real, live API call.
|
||||
|
||||
9. **The disconnect action is local-only unless you explicitly call revoke.**
|
||||
See section 5 — a "disconnect" button that only deletes your locally
|
||||
stored tokens does not actually revoke the authorization on Intuit's
|
||||
side. Decide deliberately whether "disconnect" in your new app means
|
||||
"forget locally" or "forget locally AND revoke with Intuit," and
|
||||
implement the revoke call if you mean the latter.
|
||||
|
||||
10. **CSRF `state` handling needs to be real, not decorative.** Generating a
|
||||
`state` value and sending it to Intuit is only half of CSRF protection —
|
||||
you must also store what you sent (session, signed cookie, or a
|
||||
short-lived server-side record) and verify the value that comes back on
|
||||
the callback matches, before trusting `code`/`realmId`. Skipping the
|
||||
verification half (which Pulse currently does) leaves the callback
|
||||
endpoint accepting any `code`/`realmId` pair presented to it with no way
|
||||
to confirm it originated from an authorization flow you actually
|
||||
initiated.
|
||||
|
||||
11. **The OAuth callback and any scheduler-triggered sync endpoints must be
|
||||
reachable without a logged-in session, but everything else should stay
|
||||
behind auth.** Intuit's redirect back to your app after user consent is
|
||||
an unauthenticated browser request — if your app's normal auth
|
||||
middleware requires a session cookie on every route, the OAuth callback
|
||||
route (and, if you expose a webhook/scheduler-triggered "run a sync now"
|
||||
endpoint, that route too) need to be carved out as public routes, while
|
||||
read-oriented admin/diagnostic endpoints stay behind normal
|
||||
authorization (Pulse gates its AR-diagnostic endpoint behind an
|
||||
admin-only check, for example, while the OAuth callback and the sync
|
||||
trigger are unauthenticated by design).
|
||||
|
||||
## 8. Minimal code-flow example
|
||||
|
||||
Framework-agnostic sketch (TypeScript-flavored pseudocode) of the whole
|
||||
lifecycle — this is illustrative, not a copy-paste of Pulse's actual files:
|
||||
|
||||
```ts
|
||||
const AUTHORIZE_URL = 'https://appcenter.intuit.com/connect/oauth2';
|
||||
const TOKEN_URL = 'https://oauth.platform.intuit.com/oauth2/v1/tokens/bearer';
|
||||
const REVOKE_URL = 'https://developer.api.intuit.com/v2/oauth2/tokens/revoke';
|
||||
|
||||
function basicAuthHeader(clientId: string, clientSecret: string): string {
|
||||
return 'Basic ' + Buffer.from(`${clientId}:${clientSecret}`).toString('base64');
|
||||
}
|
||||
|
||||
// 1. Kick off the flow
|
||||
function buildAuthorizeUrl(clientId: string, redirectUri: string, state: string): string {
|
||||
const params = new URLSearchParams({
|
||||
client_id: clientId,
|
||||
scope: 'com.intuit.quickbooks.accounting',
|
||||
redirect_uri: redirectUri,
|
||||
response_type: 'code',
|
||||
state,
|
||||
});
|
||||
return `${AUTHORIZE_URL}?${params}`;
|
||||
}
|
||||
// -> persist `state` server-side (session/db) before redirecting the browser here
|
||||
|
||||
// 2. Handle the callback
|
||||
async function handleCallback(code: string, realmId: string, state: string, expectedState: string, redirectUri: string) {
|
||||
if (state !== expectedState) throw new Error('CSRF: state mismatch');
|
||||
const tokens = await exchangeCodeForTokens(code, redirectUri);
|
||||
await saveTokens(realmId, tokens);
|
||||
}
|
||||
|
||||
async function exchangeCodeForTokens(code: string, redirectUri: string) {
|
||||
const res = await fetch(TOKEN_URL, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
Authorization: basicAuthHeader(CLIENT_ID, CLIENT_SECRET),
|
||||
'Content-Type': 'application/x-www-form-urlencoded',
|
||||
Accept: 'application/json',
|
||||
},
|
||||
body: new URLSearchParams({ grant_type: 'authorization_code', code, redirect_uri: redirectUri }),
|
||||
});
|
||||
if (!res.ok) throw new Error(`token exchange failed: ${res.status} ${await res.text()}`);
|
||||
return res.json(); // { access_token, refresh_token, expires_in, x_refresh_token_expires_in, ... }
|
||||
}
|
||||
|
||||
async function saveTokens(realmId: string, tokens: any) {
|
||||
const now = Date.now();
|
||||
await tokenStore.upsert(realmId, {
|
||||
accessToken: tokens.access_token,
|
||||
refreshToken: tokens.refresh_token,
|
||||
accessTokenExpiresAt: new Date(now + tokens.expires_in * 1000),
|
||||
refreshTokenExpiresAt: new Date(now + tokens.x_refresh_token_expires_in * 1000),
|
||||
});
|
||||
}
|
||||
|
||||
// 3. Get a valid access token for any API call, refreshing if needed
|
||||
async function getValidAccessToken(realmId: string): Promise<string> {
|
||||
const token = await tokenStore.load(realmId);
|
||||
if (!token) throw new Error('not connected');
|
||||
if (Date.now() < token.accessTokenExpiresAt.getTime()) return token.accessToken;
|
||||
if (Date.now() >= token.refreshTokenExpiresAt.getTime()) throw new Error('refresh token expired; reconnect required');
|
||||
|
||||
const res = await fetch(TOKEN_URL, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
Authorization: basicAuthHeader(CLIENT_ID, CLIENT_SECRET),
|
||||
'Content-Type': 'application/x-www-form-urlencoded',
|
||||
Accept: 'application/json',
|
||||
},
|
||||
body: new URLSearchParams({ grant_type: 'refresh_token', refresh_token: token.refreshToken }),
|
||||
});
|
||||
if (!res.ok) throw new Error(`token refresh failed: ${res.status} ${await res.text()}`);
|
||||
const fresh = await res.json();
|
||||
await saveTokens(realmId, fresh);
|
||||
return fresh.access_token;
|
||||
}
|
||||
|
||||
// 4. Call the actual API
|
||||
async function qboRequest<T>(realmId: string, path: string, options: RequestInit = {}): Promise<T> {
|
||||
const accessToken = await getValidAccessToken(realmId);
|
||||
const base = USE_SANDBOX ? 'https://sandbox-quickbooks.api.intuit.com' : 'https://quickbooks.api.intuit.com';
|
||||
const res = await fetch(`${base}/v3/company/${realmId}${path}`, {
|
||||
...options,
|
||||
headers: {
|
||||
Authorization: `Bearer ${accessToken}`,
|
||||
Accept: 'application/json',
|
||||
'Content-Type': 'application/json',
|
||||
...options.headers,
|
||||
},
|
||||
});
|
||||
const tid = res.headers.get('intuit_tid') ?? res.headers.get('intuit-tid') ?? 'unknown';
|
||||
if (!res.ok) throw new Error(`QBO ${res.status} on ${path} [intuit_tid=${tid}]: ${await res.text()}`);
|
||||
return res.json() as Promise<T>;
|
||||
}
|
||||
|
||||
// Example read
|
||||
async function getInvoicesUpdatedSince(realmId: string, since: Date) {
|
||||
const q = encodeURIComponent(
|
||||
`SELECT * FROM Invoice WHERE MetaData.LastUpdatedTime > '${since.toISOString()}' STARTPOSITION 1 MAXRESULTS 1000`
|
||||
);
|
||||
const data = await qboRequest<any>(realmId, `/query?query=${q}&minorversion=65`);
|
||||
return data.QueryResponse.Invoice ?? [];
|
||||
}
|
||||
|
||||
// Example write
|
||||
async function createReceivePayment(realmId: string, payload: any) {
|
||||
const data = await qboRequest<any>(realmId, `/payment?minorversion=65`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify(payload),
|
||||
});
|
||||
if (!data?.Payment?.Id) throw new Error('createPayment returned no Payment.Id');
|
||||
return data.Payment;
|
||||
}
|
||||
|
||||
// Optional: full teardown (not just forgetting locally stored tokens)
|
||||
async function revokeToken(token: string) {
|
||||
await fetch(REVOKE_URL, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
Authorization: basicAuthHeader(CLIENT_ID, CLIENT_SECRET),
|
||||
'Content-Type': 'application/x-www-form-urlencoded',
|
||||
Accept: 'application/json',
|
||||
},
|
||||
body: new URLSearchParams({ token }),
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
That's the whole lifecycle: authorize redirect → callback with verified
|
||||
`state` → code-for-token exchange → persisted token record → transparent
|
||||
refresh on every call → reads via the `query` endpoint or report endpoints →
|
||||
writes via entity-specific POST endpoints, each guarded by your own
|
||||
idempotency bookkeeping since QBO provides none for you.
|
||||
|
|
@ -22,7 +22,7 @@ export interface ScheduleConfig {
|
|||
name: string;
|
||||
description: string;
|
||||
cron_expression: string;
|
||||
sync_type: 'incremental' | 'full' | 'veeam-incremental' | 'veeam-full' | 'veeam-rpo-check' | 'contract-services' | 'engagement-daily' | 'zoom-daily' | 'morning-summary' | 'ticket-digest-daily' | 'ticket-digest-weekly' | 'ticket-digest-monthly' | 'device-link-reconcile' | 'integration-health' | 'qbo' | 'appgate-sessions' | 'appgate-daily' | 'tickets-reconcile' | 'pax8-daily' | 'phishing-sweep';
|
||||
sync_type: 'incremental' | 'full' | 'veeam-incremental' | 'veeam-full' | 'veeam-rpo-check' | 'contract-services' | 'engagement-daily' | 'zoom-daily' | 'morning-summary' | 'ticket-digest-daily' | 'ticket-digest-weekly' | 'ticket-digest-monthly' | 'device-link-reconcile' | 'integration-health' | 'qbo' | 'mimecast-sync' | 'appgate-sessions' | 'appgate-daily' | 'tickets-reconcile' | 'pax8-daily' | 'phishing-sweep';
|
||||
years_back?: number;
|
||||
is_enabled: boolean;
|
||||
last_run?: Date;
|
||||
|
|
@ -491,6 +491,28 @@ class SyncScheduler {
|
|||
await getPax8SyncService().fullSync('scheduled');
|
||||
}
|
||||
}
|
||||
} else if (config.sync_type === 'mimecast-sync') {
|
||||
const { isMimecastConfigured } = await import('@/lib/services/mimecast-client');
|
||||
if (!isMimecastConfigured()) {
|
||||
console.log('[SCHEDULER] Skipping mimecast-sync — Mimecast not configured');
|
||||
} else {
|
||||
const { runMimecastIncrementalSync } = await import('@/lib/services/mimecast-sync-service');
|
||||
const result = await runMimecastIncrementalSync();
|
||||
console.log(
|
||||
`[SCHEDULER] mimecast-sync: messagesUpserted=${result.messagesUpserted} threatsUpserted=${result.threatsUpserted} bodiesFetched=${result.bodiesFetched} purgedMessages=${result.purgedMessages} errors=${result.errors.length} durationMs=${result.durationMs}`
|
||||
);
|
||||
}
|
||||
} else if (config.sync_type === 'qbo') {
|
||||
const disabledRes = await postgresClient.query<{ disabled: boolean }>(
|
||||
`SELECT disabled FROM integration_settings WHERE key = 'qbo'`
|
||||
);
|
||||
const isDisabled = disabledRes.rows[0]?.disabled === true;
|
||||
if (isDisabled) {
|
||||
console.log('[SCHEDULER] Skipping qbo sync — QBO disabled via /admin/integrations');
|
||||
} else {
|
||||
const { getQboSyncService } = await import('@/lib/services/qbo-sync-service');
|
||||
await getQboSyncService().incrementalSync('scheduled');
|
||||
}
|
||||
} else if (config.sync_type === 'incremental') {
|
||||
await this.syncService.incrementalSync('scheduled');
|
||||
} else {
|
||||
|
|
|
|||
22
migrations/101_reschedule_mimecast_sync.sql
Normal file
22
migrations/101_reschedule_mimecast_sync.sql
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
-- Migration 101: Reschedule mimecast-sync off the 0 2 * * * cron collision.
|
||||
--
|
||||
-- mimecast-sync was seeded at '0 2 * * *', the same slot as qbo-sync-2am and
|
||||
-- veeam-full. Because the sync-scheduler dispatch table had no branches for
|
||||
-- 'mimecast-sync' or 'qbo' (fixed alongside this migration), both jobs were
|
||||
-- silently falling through to the generic Autotask fullSync() catch-all,
|
||||
-- which meant they never ran their real sync logic AND contended with
|
||||
-- veeam-full for the SyncService singleton mutex — producing
|
||||
-- "A sync operation is already in progress" lock errors at 2 AM.
|
||||
--
|
||||
-- Moves mimecast-sync to 4:45 AM (minute 45 is unused anywhere in the live
|
||||
-- sync_schedules table — avoids 0 4 (contract-services, pax8-daily), 30 4
|
||||
-- (tickets-reconcile), the 15 * * * * hourly device-link job, and the
|
||||
-- */30 :00/:30 veeam jobs).
|
||||
--
|
||||
-- Guarded on the stale cron value so this is a no-op if already moved (safe
|
||||
-- to re-run) and won't clobber an admin's manual schedule change made via
|
||||
-- /admin after this ships.
|
||||
|
||||
UPDATE sync_schedules
|
||||
SET cron_expression = '45 4 * * *', updated_at = NOW()
|
||||
WHERE id = 'mimecast-sync' AND cron_expression = '0 2 * * *';
|
||||
Loading…
Add table
Add a link
Reference in a new issue