diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index f2cc17b..15e3882 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -284,7 +284,12 @@ render, including the manual-resolution workflow for flagged companies. 2. At sync time, each PAX8 company is automatically matched to an Autotask company by fuzzy name similarity when a sufficiently confident match exists, and the match is persisted 3. A PAX8 company with no match, or with multiple similarly-scored Autotask candidates, is persisted with a flagged/needs-review status instead of being auto-assigned 4. Re-running the sync does not overwrite a match that has already been manually confirmed/resolved (idempotent with respect to human decisions) -**Plans**: TBD +**Plans**: 5 plans +- [ ] 12-01-PLAN.md — Migration 092 (pg_trgm + pax8_order_items/pax8_companies columns) + Pax8Invoice/Pax8InvoiceItem types (PAX8-06, PAX8-10, PAX8-11) +- [ ] 12-02-PLAN.md — pax8-client listAllInvoices/listAllInvoiceItems + tests + live field-mapping spot-check (PAX8-06) +- [ ] 12-03-PLAN.md — pax8-company-matcher.ts (pg_trgm similarity, 0.90 threshold, tie/empty/idempotency policy) + tests (PAX8-10, PAX8-11) +- [ ] 12-04-PLAN.md — syncOrders + syncCompanyMatches wired into Pax8SyncService.fullSync + sync-service tests (PAX8-06, PAX8-10, PAX8-11) +- [ ] 12-05-PLAN.md — Live full-sync verification of all 4 success criteria + human-verify checkpoint (PAX8-06, PAX8-10, PAX8-11) **UI hint**: no ### Phase 13: Scheduler & Admin Toggle diff --git a/.planning/phases/12-orders-invoices-company-matching/12-01-PLAN.md b/.planning/phases/12-orders-invoices-company-matching/12-01-PLAN.md new file mode 100644 index 0000000..48c3dce --- /dev/null +++ b/.planning/phases/12-orders-invoices-company-matching/12-01-PLAN.md @@ -0,0 +1,232 @@ +--- +phase: 12-orders-invoices-company-matching +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - migrations/092_pax8_orders_company_matching.sql + - lib/types/pax8.ts +autonomous: true +requirements: [PAX8-06, PAX8-10, PAX8-11] + +must_haves: + truths: + - "pg_trgm extension is enabled in the dev Postgres database" + - "pax8_order_items has per-company id, billing period, and dual-cost columns" + - "pax8_companies has auto-match columns (autotask_company_id, match_confidence, match_method, matched_at)" + - "TypeScript exposes Pax8Invoice and Pax8InvoiceItem types matching the live PAX8 /invoices field shape" + artifacts: + - path: "migrations/092_pax8_orders_company_matching.sql" + provides: "pg_trgm + additive columns on pax8_order_items and pax8_companies" + contains: "CREATE EXTENSION IF NOT EXISTS pg_trgm" + - path: "lib/types/pax8.ts" + provides: "Pax8Invoice / Pax8InvoiceItem API types" + contains: "Pax8InvoiceItem" + key_links: + - from: "migrations/092_pax8_orders_company_matching.sql" + to: "pax8_order_items" + via: "ALTER TABLE ADD COLUMN IF NOT EXISTS" + pattern: "ALTER TABLE pax8_order_items" + - from: "migrations/092_pax8_orders_company_matching.sql" + to: "pax8_companies" + via: "ALTER TABLE ADD COLUMN IF NOT EXISTS" + pattern: "ALTER TABLE pax8_companies" +--- + + +Lay the schema + type foundation for both halves of Phase 12: historical +invoice/line-item cost storage (PAX8-06) and fuzzy company matching +(PAX8-10, PAX8-11). + +Live PAX8 verification (12-RESEARCH.md) proved two schema gaps in the +already-committed migration 091: `pax8_order_items` has no per-company id and +no billing-period columns (invoice headers carry no per-customer data — +`companyId` is always NULL on the header), and `pax8_companies` has nowhere to +store a confident auto-match. This plan closes both gaps with a new additive +migration and enables the `pg_trgm` extension the matcher needs. + +Purpose: Everything downstream (client methods, matcher, sync wiring) depends +on these columns and types existing first. +Output: migrations/092_pax8_orders_company_matching.sql applied to the dev DB; +Pax8Invoice / Pax8InvoiceItem types in lib/types/pax8.ts. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/12-orders-invoices-company-matching/12-CONTEXT.md +@.planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md +@.planning/phases/12-orders-invoices-company-matching/12-PATTERNS.md + + +Existing schema this migration extends (migrations/091_pax8_tables.sql): +- pax8_order_items already has: id UUID PK, order_id UUID NOT NULL REFERENCES + pax8_orders(id) ON DELETE CASCADE, product_id UUID, quantity INTEGER, + unit_price NUMERIC(12,2), line_total NUMERIC(12,2), currency CHAR(3), + raw_payload JSONB, synced_at, is_deleted, deleted_at. +- pax8_companies already has: id UUID PK, name TEXT NOT NULL, external_id, + website, status, city, state_or_province, postal_code, country, + raw_payload, synced_at, is_deleted, deleted_at. +- companies (Autotask, migrations/001): id BIGINT PK, company_name + VARCHAR(255), is_active BOOLEAN. + +Extension-enable precedent (migrations/069, 070): `CREATE EXTENSION IF NOT +EXISTS pgcrypto;` + +Existing stale stubs to replace in lib/types/pax8.ts (currently unused — no +file imports Pax8Order/Pax8OrderItem): Pax8Order (lines 65-75), +Pax8OrderItem (lines 79-88). Keep Pax8EntitySyncResult / Pax8SyncResult +unchanged. + + + + + + + Task 1: Write migration 092 (pg_trgm + additive columns) and apply to dev DB + migrations/092_pax8_orders_company_matching.sql + + - migrations/091_pax8_tables.sql (current pax8_order_items and pax8_companies definitions this migration ALTERs — confirm which columns already exist so ADD COLUMN IF NOT EXISTS is correct) + - migrations/069_create_analyzer_tables.sql (CREATE EXTENSION IF NOT EXISTS pgcrypto precedent — copy this exact statement shape for pg_trgm) + - .planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md (Code Examples > migration skeleton; Pitfall 1 and Pitfall 2 explain why each column exists) + - scripts/apply-migrations.sh (how migrations are applied to the running container; DB_USER default pulse_user, DB_NAME default pulse_autotask) + + + Create migrations/092_pax8_orders_company_matching.sql, additive-only, using + CREATE EXTENSION IF NOT EXISTS and ADD COLUMN IF NOT EXISTS / CREATE INDEX IF + NOT EXISTS throughout (never edit committed migration 091). Include a header + comment stating this is Phase 12, that it enables pg_trgm and closes the + per-company/period gap on pax8_order_items (invoice headers carry no + per-customer data — see 12-RESEARCH.md Pitfall 1), and that + pax8_orders.pax8_company_id will intentionally stay NULL for every real row. + + Statement 1: `CREATE EXTENSION IF NOT EXISTS pg_trgm;` + + Statement 2: ALTER TABLE pax8_order_items ADD COLUMN IF NOT EXISTS these nine + columns: pax8_company_id UUID (soft ref to pax8_companies(id), no hard FK — + matches the soft-ref convention on pax8_subscriptions.pax8_company_id), + subscription_id UUID (soft ref to pax8_subscriptions(id)), item_type TEXT + (free-form — 'subscription'/'prorate'/'one-time' and possibly others; NO + CHECK constraint, matching pax8_subscriptions.status), sku TEXT, description + TEXT, start_period TIMESTAMPTZ, end_period TIMESTAMPTZ, partner_cost + NUMERIC(12,2), partner_cost_total NUMERIC(12,2). + + Statement 3: CREATE INDEX IF NOT EXISTS idx_pax8_order_items_company ON + pax8_order_items(pax8_company_id); and idx_pax8_order_items_period ON + pax8_order_items(start_period, end_period). + + Statement 4: ALTER TABLE pax8_companies ADD COLUMN IF NOT EXISTS + autotask_company_id BIGINT (soft ref to companies(id), no hard FK), + match_confidence NUMERIC(4,3) (raw pg_trgm score 0.000-1.000), match_method + TEXT ('pg_trgm' | 'manual'), matched_at TIMESTAMPTZ. + + Statement 5: CREATE INDEX IF NOT EXISTS idx_pax8_companies_autotask ON + pax8_companies(autotask_company_id). + + Do NOT touch pax8_orders, pax8_order_items existing columns, or + pax8_company_match_review — they are already correctly shaped by migration 091. + + Then apply to the running dev database (existing volume — Postgres init does + not re-run migrations per CLAUDE.md): run + `docker exec -i pulse-postgres psql -U pulse_user -d pulse_autotask < migrations/092_pax8_orders_company_matching.sql`. + If the container name or credentials differ, read scripts/apply-migrations.sh + and adapt. + + + - File migrations/092_pax8_orders_company_matching.sql exists and contains the literal string `CREATE EXTENSION IF NOT EXISTS pg_trgm` + - `grep -c "ADD COLUMN IF NOT EXISTS" migrations/092_pax8_orders_company_matching.sql` returns 13 (9 on pax8_order_items + 4 on pax8_companies) + - After apply, `docker exec pulse-postgres psql -U pulse_user -d pulse_autotask -tAc "SELECT 1 FROM pg_extension WHERE extname='pg_trgm'"` prints `1` + - After apply, `docker exec pulse-postgres psql -U pulse_user -d pulse_autotask -tAc "SELECT count(*) FROM information_schema.columns WHERE table_name='pax8_order_items' AND column_name IN ('pax8_company_id','subscription_id','item_type','sku','description','start_period','end_period','partner_cost','partner_cost_total')"` prints `9` + - After apply, the same query for table_name='pax8_companies' and column_name IN ('autotask_company_id','match_confidence','match_method','matched_at') prints `4` + - No ALTER or DROP against pax8_orders / pax8_company_match_review appears in the file + + + docker exec pulse-postgres psql -U pulse_user -d pulse_autotask -tAc "SELECT (SELECT count(*) FROM pg_extension WHERE extname='pg_trgm') || '/' || (SELECT count(*) FROM information_schema.columns WHERE table_name='pax8_order_items' AND column_name IN ('pax8_company_id','subscription_id','item_type','sku','description','start_period','end_period','partner_cost','partner_cost_total')) || '/' || (SELECT count(*) FROM information_schema.columns WHERE table_name='pax8_companies' AND column_name IN ('autotask_company_id','match_confidence','match_method','matched_at'))" + Expected output: 1/9/4 + + pg_trgm enabled; all 13 additive columns and 3 indexes present in the dev DB; migration is additive-only and does not edit migration 091. + + + + Task 2: Add Pax8Invoice / Pax8InvoiceItem types to lib/types/pax8.ts + lib/types/pax8.ts + + - lib/types/pax8.ts (current Pax8Order/Pax8OrderItem stubs at lines 65-88 to replace; Pax8Company/Pax8Subscription convention with the `[key: string]: unknown` escape hatch to mirror) + - .planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md (Pitfall 2 — the live-verified invoice item JSON shape is the source of truth for field names/types) + + + Replace the two stale speculative stubs Pax8Order and Pax8OrderItem (currently + unused — grep confirms no importers) with live-verified types named + Pax8Invoice and Pax8InvoiceItem (favor these names per 12-RESEARCH.md + Recommended Project Structure, matching the client methods listAllInvoices / + listAllInvoiceItems added in Plan 02). Keep the `[key: string]: unknown` + escape hatch on both. + + Pax8Invoice (header — the partner's consolidated monthly bill): id: string; + companyId: string | null (add a comment: always null on the header for this + single-tenant reseller account — per-company data lives on items); invoiceDate: + string | null; total: number | null; status: string | null; currencyCode: + string | null; plus the escape hatch. + + Pax8InvoiceItem (line item — per-company, per-period cost): id: string; type: + string | null; externalId: string | null; companyId: string | null; + forCompanyId: string | null; companyName: string | null; startPeriod: string | + null; endPeriod: string | null; quantity: number | null; unitOfMeasure: string + | null; term: string | null; sku: string | null; description: string | null; + rateType: string | null; chargeType: string | null; price: number | null; + subTotal: number | null; cost: number | null; costTotal: number | null; total: + number | null; amountDue: number | null; productId: string | null; + productName: string | null; vendorName: string | null; currencyCode: string | + null; subscriptionId: string | null; plus the escape hatch. + + Do not change Pax8Company, Pax8Subscription, Pax8Product, Pax8PageEnvelope, + Pax8EntitySyncResult, or Pax8SyncResult. + + + - lib/types/pax8.ts exports interfaces named exactly `Pax8Invoice` and `Pax8InvoiceItem` + - `grep -c "Pax8Order" lib/types/pax8.ts` returns 0 (stale stubs removed) + - Pax8InvoiceItem declares fields `amountDue`, `cost`, `costTotal`, `subscriptionId`, `startPeriod`, `endPeriod`, `companyId`, `type` + - `npx tsc --noEmit --pretty` passes with no new errors + + + npx tsc --noEmit --pretty 2>&1 | tail -5; grep -c "Pax8Order" lib/types/pax8.ts + + Pax8Invoice and Pax8InvoiceItem exist with the live-verified field set; no Pax8Order/Pax8OrderItem references remain; type-check passes. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| migration file → dev Postgres | DDL applied to a live database with existing PAX8 data | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-12-01 | Tampering | migration 092 DDL | mitigate | Additive-only (ADD COLUMN IF NOT EXISTS, CREATE INDEX IF NOT EXISTS, CREATE EXTENSION IF NOT EXISTS); no DROP/ALTER of existing columns; re-runnable without data loss | +| T-12-SC | Tampering | package installs | accept | No new npm/pip/cargo packages this phase; pg_trgm is a Postgres 16 contrib extension verified live against the project's own container (12-RESEARCH.md) — not a registry package | + + + +- Migration is idempotent: re-applying it against the dev DB produces no error and no duplicate columns. +- `npx tsc --noEmit --pretty` passes. + + + +pg_trgm is enabled; pax8_order_items and pax8_companies carry the new columns; Pax8Invoice/Pax8InvoiceItem types match the live PAX8 field shape; type-check green. + + + +Create `.planning/phases/12-orders-invoices-company-matching/12-01-SUMMARY.md` when done. + diff --git a/.planning/phases/12-orders-invoices-company-matching/12-02-PLAN.md b/.planning/phases/12-orders-invoices-company-matching/12-02-PLAN.md new file mode 100644 index 0000000..1255e34 --- /dev/null +++ b/.planning/phases/12-orders-invoices-company-matching/12-02-PLAN.md @@ -0,0 +1,233 @@ +--- +phase: 12-orders-invoices-company-matching +plan: 02 +type: execute +wave: 2 +depends_on: ["12-01"] +files_modified: + - lib/services/pax8-client.ts + - lib/services/pax8-client.test.ts + - scripts/verify-pax8-invoice-items.ts +autonomous: true +requirements: [PAX8-06] + +must_haves: + truths: + - "Pax8Client can page through every invoice header via listAllInvoices()" + - "Pax8Client can page through a single invoice's line items via listAllInvoiceItems(invoiceId)" + - "Both new methods are GET-only (no mutating verb), preserving PAX8-08" + - "The live invoice-item cost field mapping (unit_price/line_total/partner_cost/partner_cost_total) is confirmed against real prorate + one-time + subscription items" + artifacts: + - path: "lib/services/pax8-client.ts" + provides: "listAllInvoices + listAllInvoiceItems read methods" + contains: "listAllInvoiceItems" + - path: "lib/services/pax8-client.test.ts" + provides: "pagination + GET-only tests for the new methods" + contains: "listAllInvoices" + - path: "scripts/verify-pax8-invoice-items.ts" + provides: "live field-mapping spot-check resolving RESEARCH Open Question 1" + key_links: + - from: "lib/services/pax8-client.ts" + to: "paginateAll" + via: "reuse existing private helper for both new methods" + pattern: "paginateAll +Add the two read-only PAX8 client methods this phase's invoice sync needs +(PAX8-06), and resolve the one open field-mapping question from research before +the sync service (Plan 04) writes any cost columns. + +`/invoices` is a flat list (94 headers, full history since 2019). Invoice items +are a per-invoice child resource (`/invoices/{id}/items`) — there is no flat +items endpoint (12-RESEARCH.md Pitfall 3; `/orders` is unreliable, returns 504 +— do NOT add it). Both methods reuse the existing generic `paginateAll` +helper unchanged. + +The spot-check script closes RESEARCH Open Question 1: confirm that the billed +line amount is `amountDue` (not `total`), and that partner cost maps from +`cost`/`costTotal`, across all three observed item types. + +Purpose: Correct client surface + confirmed cost mapping before Plan 04 codes +the upsert SQL. +Output: two new client methods, their tests, and a verification script. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md +@.planning/phases/12-orders-invoices-company-matching/12-PATTERNS.md + + +Existing pax8-client.ts (Plan 01 added Pax8Invoice/Pax8InvoiceItem to lib/types/pax8.ts): +- private paginateAll((page, size) => Promise>): Promise + — size=200, loops until page.number >= page.totalPages - 1; inherits fetchJson's + 429/Retry-After backoff. GET-only. +- Existing analog to copy: listAllProducts() (lines 117-122) returns + paginateAll((page, size) => this.fetchJson>(`/products?page=${page}&size=${size}`)). +- Import Pax8Invoice, Pax8InvoiceItem from '@/lib/types/pax8'. + +Existing test helpers (pax8-client.test.ts): +- makeMultiPageFetchMock({ resource, pages }) — matches url.includes(`/${resource}`), + keys page body by `page=N`. Works for `/invoices` and for `/invoices/{id}/items` + (both contain `/invoices`). +- SECRET constant + the GET-only / secret-never-leaked assertion patterns. + +verify script precedent: scripts/verify-pax8-auth.ts — dotenv from ../.env.local, +getPax8Client(), prints ONLY a summary, never the token/secret. Run with npx tsx. + + + + + + + Task 1: Add listAllInvoices() and listAllInvoiceItems(invoiceId) to Pax8Client + lib/services/pax8-client.ts + + - lib/services/pax8-client.ts (paginateAll helper lines 86-101; listAllProducts lines 117-122 as the exact shape to copy; the import line 7 to extend) + - lib/types/pax8.ts (Pax8Invoice / Pax8InvoiceItem added in Plan 01) + - .planning/phases/12-orders-invoices-company-matching/12-PATTERNS.md (pax8-client.ts section — nested per-invoice method shape) + + + Extend the type import on line 7 to include Pax8Invoice and Pax8InvoiceItem. + + Add listAllInvoices(): Promise<Pax8Invoice[]> — identical shape to + listAllProducts but hitting `/invoices?page=${page}&size=${size}`, returning + paginateAll<Pax8Invoice>. Add a doc comment marking it read-only / GET-only + (PAX8-08), consistent with the other listAll* comments. + + Add listAllInvoiceItems(invoiceId: string): Promise<Pax8InvoiceItem[]> — + paginateAll<Pax8InvoiceItem> hitting the nested child path + `/invoices/${invoiceId}/items?page=${page}&size=${size}`. Same GET-only doc + comment. Do NOT add any `/orders` method (12-RESEARCH.md Pitfall 3 — the + endpoint is unreliable and not part of the design). + + paginateAll and fetchJson need no changes — both new methods are purely + additive call sites. + + + - pax8-client.ts declares `async listAllInvoices(): Promise` and `async listAllInvoiceItems(invoiceId: string): Promise` + - listAllInvoiceItems interpolates the invoiceId into the path segment `/invoices/${invoiceId}/items` + - `grep -c "/orders" lib/services/pax8-client.ts` returns 0 + - No `method:` other than the default GET appears in either new method + - `npx tsc --noEmit --pretty` passes + + + npx tsc --noEmit --pretty 2>&1 | tail -5; grep -c "/orders" lib/services/pax8-client.ts + + Both read-only methods exist, reuse paginateAll, and type-check passes; no /orders call added. + + + + Task 2: Add pagination + GET-only tests for the new client methods + lib/services/pax8-client.test.ts + + - listAllInvoices() concatenates content across all pages in order (multi-page mock, resource 'invoices'), and every request is size=200 + - listAllInvoiceItems(invoiceId) requests the nested path containing that invoiceId (e.g. `/invoices/inv-123/items`) and concatenates its pages + - Every data request issued by both methods is a GET with an Authorization: Bearer header, none use a mutating method (extends the existing PAX8-08 assertion to the new methods) + + + - lib/services/pax8-client.test.ts (makeMultiPageFetchMock lines 49-98; the listAllSubscriptions/listAllProducts test cases lines 165-223 to copy; the GET-only test lines 225-240) + + + Add test cases mirroring the existing listAllSubscriptions/listAllProducts + tests. For listAllInvoices use makeMultiPageFetchMock with resource: 'invoices' + and assert the concatenated order plus size=200 on each call. + + For listAllInvoiceItems, the shared helper's url.includes('/invoices') already + routes the nested path, but it does not assert the invoiceId segment — add an + explicit assertion that at least one issued data-call URL contains + `/invoices/inv-123/items` (use invoiceId 'inv-123'). If the shared helper's + page-keying does not cleanly disambiguate the nested path, write a small + bespoke fetch mock for this one case following makeMultiPageFetchMock's shape. + + Extend the existing GET-only / no-mutating-method assertion to also cover a + listAllInvoiceItems call so PAX8-08 stays proven for the new surface. + + + - Test file contains cases titled to cover listAllInvoices and listAllInvoiceItems + - The listAllInvoiceItems test asserts a data-call URL includes the literal substring `/invoices/inv-123/items` + - `npx vitest run lib/services/pax8-client.test.ts` passes with the new cases included + + + npx vitest run lib/services/pax8-client.test.ts + + New tests pass, proving in-order pagination, the nested invoice-item path, and GET-only behavior for both methods. + + + + Task 3: Live field-mapping spot-check script (resolves RESEARCH Open Question 1) + scripts/verify-pax8-invoice-items.ts + + - scripts/verify-pax8-auth.ts (the dotenv-from-.env.local + getPax8Client() + secret-safe logging pattern to copy) + - .planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md (Open Question 1 and Pitfall 2 — the exact fields and the hypothesized mapping to confirm) + - lib/services/pax8-client.ts (the listAllInvoices / listAllInvoiceItems methods from Task 1) + + + Create scripts/verify-pax8-invoice-items.ts (run via `npx tsx`), copying the + dotenv + getPax8Client + secret-safe logging discipline from + verify-pax8-auth.ts (never print the token or client secret — summary output + only). + + The script fetches the first invoice via listAllInvoices(), then its items via + listAllInvoiceItems(invoice.id). It locates one item of each observed type + ('subscription', 'prorate', 'one-time') where present, and for each prints: + type, quantity, price, subTotal, cost, costTotal, total, amountDue, + companyId (non-null?), subscriptionId, startPeriod, endPeriod. It then prints + a one-line CONFIRM/DIVERGENCE verdict for the mapping the sync service (Plan 04) + will use: unit_price ← price, line_total ← amountDue, partner_cost ← cost, + partner_cost_total ← costTotal — flagging any type where amountDue is not the + plausible billed amount so Plan 04 can adjust before writing the upsert SQL. + + The script must run read-only (it only calls listAll* GET methods). Print a + final summary line with the count of item types inspected. + + + - Running `npx tsx scripts/verify-pax8-invoice-items.ts` prints, for at least the 'subscription' type, the fields price/subTotal/cost/costTotal/total/amountDue and a CONFIRM or DIVERGENCE verdict line for the unit_price/line_total/partner_cost/partner_cost_total mapping + - The script output never contains the client secret or access token + - The script issues only GET calls (uses only listAllInvoices / listAllInvoiceItems) + - The SUMMARY (12-02-SUMMARY.md) records the confirmed mapping (or any divergence) so Plan 04 can rely on it + + + npx tsx scripts/verify-pax8-invoice-items.ts 2>&1 | tail -20 + Manual read of output: confirm the mapping verdict line and that no secret is printed. + + Live field mapping confirmed (or divergence documented) for the invoice-item cost columns; result recorded in the SUMMARY for Plan 04 to consume. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| PAX8 API → Pulse client | External JSON responses (untrusted) parsed into typed objects | +| PAX8 credentials → logs/output | Client secret + bearer token must never surface in logs or script output | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-12-03 | Information Disclosure | verify-pax8-invoice-items.ts, new client error paths | mitigate | Copy verify-pax8-auth.ts's summary-only logging; error messages include HTTP status but never the secret (existing pax8-client.test.ts asserts `rejects.not.toThrow(new RegExp(SECRET))`) | +| T-12-05 | Elevation of Privilege | new client methods | mitigate | Both methods are GET-only via paginateAll; no mutating verb ever set (PAX8-08); covered by the extended GET-only test | +| T-12-SC | Tampering | package installs | accept | No new packages; methods reuse existing client internals | + + + +- `npx vitest run lib/services/pax8-client.test.ts` green. +- `npx tsc --noEmit --pretty` passes. +- Spot-check script runs read-only and prints the mapping verdict with no secret leakage. + + + +listAllInvoices + listAllInvoiceItems exist, are GET-only, tested; the invoice-item cost field mapping is confirmed live and recorded for Plan 04. + + + +Create `.planning/phases/12-orders-invoices-company-matching/12-02-SUMMARY.md` when done. Record the confirmed invoice-item field mapping (or any divergence) explicitly. + diff --git a/.planning/phases/12-orders-invoices-company-matching/12-03-PLAN.md b/.planning/phases/12-orders-invoices-company-matching/12-03-PLAN.md new file mode 100644 index 0000000..d7264cd --- /dev/null +++ b/.planning/phases/12-orders-invoices-company-matching/12-03-PLAN.md @@ -0,0 +1,287 @@ +--- +phase: 12-orders-invoices-company-matching +plan: 03 +type: execute +wave: 2 +depends_on: ["12-01"] +files_modified: + - lib/services/pax8-company-matcher.ts + - lib/services/pax8-company-matcher.test.ts +autonomous: true +requirements: [PAX8-10, PAX8-11] + +must_haves: + truths: + - "A PAX8 company with a single candidate scoring >= 0.90 and no near-tie is auto-linked (columns written on pax8_companies)" + - "A PAX8 company with the best score < 0.90, or a second candidate within 0.05 of the top, is flagged with the top-3 candidates (never auto-linked)" + - "A PAX8 company with zero candidates above the 0.3 floor is flagged with an empty candidate_company_ids array (D-03)" + - "A human-resolved match is never overwritten by re-running the matcher (D-05 / SC#4)" + - "PAX8 company names are passed as bound parameters into similarity(), never string-interpolated" + artifacts: + - path: "lib/services/pax8-company-matcher.ts" + provides: "matchPax8Companies() + candidate/decision/apply/conflict helpers" + exports: ["matchPax8Companies"] + min_lines: 120 + - path: "lib/services/pax8-company-matcher.test.ts" + provides: "auto-link / review / empty-candidate / idempotency unit tests" + contains: "matchPax8Companies" + key_links: + - from: "lib/services/pax8-company-matcher.ts" + to: "companies" + via: "similarity($1, company_name) with is_active filter" + pattern: "similarity\\(\\$1, company_name\\)" + - from: "lib/services/pax8-company-matcher.ts" + to: "pax8_companies" + via: "UPDATE auto-match columns guarded against resolved rows" + pattern: "UPDATE pax8_companies" + - from: "lib/services/pax8-company-matcher.ts" + to: "pax8_company_match_review" + via: "upsert on the open-review partial unique index" + pattern: "INSERT INTO pax8_company_match_review" +--- + + +Build the fuzzy company matcher (PAX8-10, PAX8-11) as a close structural port of +lib/services/device-link-reconciler.ts, scored with Postgres `pg_trgm` +similarity() at a conservative, tunable 0.90 auto-link floor with a 0.05 +tie-margin — the thresholds research validated live against this project's real +118 pax8_companies vs 242 active companies (every genuine match scored 1.00; the +highest non-match was 0.70). + +Match policy (locked decisions): +- D-01: auto-link only at similarity >= 0.90 (AUTO_LINK_THRESHOLD, initial & + tunable) with a single candidate and no near-tie. +- D-02: never silently tie-break — a second candidate within 0.05 (TIE_MARGIN) + of the top score forces review even at score 1.0. +- D-03: zero candidates above a 0.3 floor still creates a review row with an + empty candidate_company_ids array — never silently dropped. +- D-04: ambiguous review rows carry the top 3 candidates. +- D-05 / SC#4: only unresolved rows are (re)scored every sync; a human-resolved + match (resolved review row, or match_method='manual') is never overwritten. + +Purpose: The hardest problem of the phase, isolated in its own service + test +plan (matching logic + SQL is context-heavy). Runs from Pax8SyncService in Plan 04. +Output: lib/services/pax8-company-matcher.ts and its unit tests. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/phases/12-orders-invoices-company-matching/12-CONTEXT.md +@.planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md +@.planning/phases/12-orders-invoices-company-matching/12-PATTERNS.md + + +Precedent to port (lib/services/device-link-reconciler.ts): +- findBySerial/findByMac/findByHostnameInCompany — parameterized candidate + queries filtered on is_deleted=false (adapt to similarity() + is_active=true). +- pickBestCandidate — ambiguity guard (adapt to numeric TIE_MARGIN). +- applyLink — UPDATE ... WHERE ... AND configuration_item_id IS NULL idempotency + guard (adapt to a resolved-row guard). +- recordConflict — INSERT ... ON CONFLICT (subject) WHERE resolved_at IS NULL + DO UPDATE (device_link_review). pax8_company_match_review has the identical + partial-unique-index shape. +- reconcileUnlinkedDevices(opts?: { limit?; dryRun? }) — top-level scan loop + + ReconcileResult rollup. + +Schema (after Plan 01's migration 092): +- pax8_companies: id UUID, name TEXT, is_deleted BOOLEAN, autotask_company_id + BIGINT, match_confidence NUMERIC(4,3), match_method TEXT, matched_at TIMESTAMPTZ. +- companies (Autotask): id BIGINT, company_name VARCHAR(255), is_active BOOLEAN. +- pax8_company_match_review (migration 091): id UUID, pax8_company_id UUID NOT + NULL, candidate_company_ids BIGINT[] NOT NULL, match_confidences TEXT[] NOT + NULL, detected_at, resolved_at TIMESTAMPTZ, resolved_by_user_id TEXT, + resolved_to_company_id BIGINT, resolution_note TEXT. Partial unique index + uq_pax8_company_match_review_open on (pax8_company_id) WHERE resolved_at IS NULL. + +postgresClient: default export from '@/lib/services/postgres-client'; +postgresClient.query(sql, params) — parameterized only, never interpolate. + + + + + + + Task 1: Create lib/services/pax8-company-matcher.ts + lib/services/pax8-company-matcher.ts + + - lib/services/device-link-reconciler.ts (full file — the structural precedent: findBy*, applyLink, recordConflict, pickBestCandidate, reconcileUnlinkedDevices) + - migrations/091_pax8_tables.sql (pax8_company_match_review shape + partial unique index) + - .planning/phases/12-orders-invoices-company-matching/12-PATTERNS.md (pax8-company-matcher.ts section — the exact adapted findCandidates/decide/applyLink/recordConflict snippets AND the "IMPORTANT" note on the resolved_at re-scoring gate) + - .planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md (Pattern 2; Pitfall 4 case-insensitivity; Pitfall 5 is_active filter; Security Domain SQL-injection mitigation) + - lib/types/pax8.ts (Pax8EntitySyncResult shape the matcher result maps into) + + + Create lib/services/pax8-company-matcher.ts. File header comment: states this + is the PAX8 company↔Autotask fuzzy matcher, ported from + device-link-reconciler.ts, and that unlike the reconciler it runs from + Pax8SyncService.fullSync() (Plan 04) — NOT a standalone cron (no scheduler + wiring this phase). Import postgresClient default from + '@/lib/services/postgres-client'. + + Module constants (export them so they are documented/tunable and testable): + AUTO_LINK_THRESHOLD = 0.90 (D-01, conservative initial/tunable value), + TIE_MARGIN = 0.05 (D-02), CANDIDATE_FLOOR = 0.3 (keeps the candidate list + small). Add a comment citing the live evidence: genuine matches clustered at + 1.00, highest observed non-match 0.70, so 0.90 sits in the empty gap. + + Types: CompanyCandidate { autotask_company_id: number; score: number }. Export + interface Pax8CompanyMatchResult { scanned: number; autoLinked: number; + flaggedAmbiguous: number; flaggedNoCandidate: number; durationMs: number }. + + findCandidates(pax8Name: string): parameterized query — SELECT id::text, + similarity($1, company_name)::text AS score FROM companies WHERE is_active = + true AND similarity($1, company_name) > CANDIDATE_FLOOR ORDER BY score DESC + LIMIT 5. Bind pax8Name as $1 (NEVER interpolate — Security Domain). Return + rows mapped to CompanyCandidate (Number(id), Number(score)). Rely on + pg_trgm's built-in case-insensitivity (Pitfall 4) — do not add LOWER(); TRIM + the input name for cleaner review display only. + + decide(candidates): returns { kind: 'auto'; match } | { kind: 'review'; top3 }. + Zero candidates → review with empty top3 (D-03). Else let [best, second] = + candidates; tie = second exists AND best.score - second.score < TIE_MARGIN. + If best.score >= AUTO_LINK_THRESHOLD AND NOT tie → auto. Otherwise review with + candidates.slice(0, 3) (D-02/D-04). + + applyLink(pax8CompanyId, autotaskCompanyId, score): UPDATE pax8_companies SET + autotask_company_id=$2, match_confidence=$3, match_method='pg_trgm', + matched_at=NOW() WHERE id=$1 AND match_method IS DISTINCT FROM 'manual' AND + NOT EXISTS (SELECT 1 FROM pax8_company_match_review r WHERE + r.pax8_company_id = pax8_companies.id AND r.resolved_at IS NOT NULL). This is + the SC#4 / D-05 idempotency guard: never overwrite a manually-resolved match. + Then close any open (never-human-touched) review row for that company: DELETE + FROM pax8_company_match_review WHERE pax8_company_id=$1 AND resolved_at IS NULL + (a now-confident match supersedes an open flag; human-resolved rows have + resolved_at set and are untouched). Pass score as a NUMERIC bound param. + + recordConflict(pax8CompanyId, top3): first, if this company previously held a + pg_trgm auto-match that is now ambiguous/below-threshold, clear the stale + columns conservatively — UPDATE pax8_companies SET autotask_company_id=NULL, + match_confidence=NULL, match_method=NULL, matched_at=NULL WHERE id=$1 AND + match_method='pg_trgm' (never clears 'manual'). Then upsert the review row: + INSERT INTO pax8_company_match_review (pax8_company_id, candidate_company_ids, + match_confidences) VALUES ($1, $2::bigint[], $3::text[]) ON CONFLICT + (pax8_company_id) WHERE resolved_at IS NULL DO UPDATE SET candidate_company_ids + = EXCLUDED.candidate_company_ids, match_confidences = EXCLUDED.match_confidences, + detected_at = NOW(). candidate_company_ids = top3 ids (empty [] when no + candidates, D-03); match_confidences = top3 scores formatted with toFixed(3) + as TEXT[] (consistent precision with match_confidence NUMERIC(4,3)). + + matchPax8Companies(opts?: { limit?: number; dryRun?: boolean }): + Promise. Select the re-scoring-eligible set (D-05): + SELECT id::text, name FROM pax8_companies WHERE is_deleted = false AND + match_method IS DISTINCT FROM 'manual' AND NOT EXISTS (SELECT 1 FROM + pax8_company_match_review r WHERE r.pax8_company_id = pax8_companies.id AND + r.resolved_at IS NOT NULL) ORDER BY name LIMIT $1 (default limit 1000). Loop: + scanned++; candidates = findCandidates(name.trim()); d = decide(candidates); + if kind auto → (unless dryRun) applyLink + autoLinked++; if kind review with + empty top3 → (unless dryRun) recordConflict(id, []) + flaggedNoCandidate++; + else review → (unless dryRun) recordConflict(id, top3) + flaggedAmbiguous++. + Wrap in try/catch per the CLAUDE.md convention; return the result with + durationMs. Console.log a one-line summary at the end. + + Use error handling shape: catch (err) { const msg = err instanceof Error ? + err.message : String(err); console.error('[Pax8Match] failed:', msg); ... }. + + + - Exports `matchPax8Companies`, `AUTO_LINK_THRESHOLD` (= 0.90), `TIE_MARGIN` (= 0.05), and interface `Pax8CompanyMatchResult` + - The candidate query passes the PAX8 name as a bound `$1` param and filters `is_active = true`; `grep -c "company_name +" lib/services/pax8-company-matcher.ts` returns 0 (no string concatenation into SQL) + - The applyLink UPDATE contains both `match_method IS DISTINCT FROM 'manual'` and a `resolved_at IS NOT NULL` NOT EXISTS guard + - The recordConflict upsert targets `pax8_company_match_review` with `ON CONFLICT (pax8_company_id) WHERE resolved_at IS NULL` + - The eligibility SELECT excludes rows where a review row has `resolved_at IS NOT NULL` and where `match_method` = 'manual' + - `npx tsc --noEmit --pretty` passes + + + npx tsc --noEmit --pretty 2>&1 | tail -5; grep -c "similarity(\$1, company_name)" lib/services/pax8-company-matcher.ts + + Matcher implements the D-01..D-05 policy with parameterized queries, the resolved-row idempotency guard, and the top-3/empty-array review semantics; type-check passes. + + + + Task 2: Create lib/services/pax8-company-matcher.test.ts + lib/services/pax8-company-matcher.test.ts + + - auto-link: eligible company, one candidate scoring 0.95 → applyLink UPDATE against pax8_companies is issued; no INSERT into pax8_company_match_review + - review (below threshold): best candidate 0.80 → an INSERT/UPSERT into pax8_company_match_review with that candidate; no auto-link UPDATE writing autotask_company_id + - review (near-tie): candidates 0.95 and 0.92 (margin < 0.05) → review row with top candidates, no auto-link even though top >= 0.90 (D-02) + - empty candidates: zero rows above the floor → review row with an empty candidate_company_ids array (D-03) + - idempotent (D-05/SC#4): the applyLink UPDATE SQL includes the `resolved_at IS NOT NULL` guard AND `match_method IS DISTINCT FROM 'manual'`; the eligibility SELECT excludes manually-resolved rows + - dryRun: no write queries (no UPDATE/INSERT/DELETE) are issued when dryRun is true + + + - lib/services/pax8-client.test.ts (vi.fn / vi.stubGlobal mocking discipline to mirror, applied to postgresClient.query instead of fetch) + - lib/services/pax8-company-matcher.ts (Task 1 — the query shapes to assert against) + - .planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md (Validation Architecture > Phase Requirements → Test Map; Wave 0 Gaps recommends mocking postgresClient.query) + + + Create the matcher test with vitest. Mock the postgres client module: + vi.mock('@/lib/services/postgres-client', () => ({ default: { query: vi.fn() } })). + Import the mocked query and, per test, program its return sequence so the + first call (eligibility SELECT) yields the pax8_companies rows under test and + subsequent findCandidates calls yield the candidate score rows. Assert + behavior by inspecting the SQL string + params of the mock's calls: for + auto-link assert a call whose SQL matches /UPDATE pax8_companies/ ran; for + review assert a call whose SQL matches /INSERT INTO pax8_company_match_review/ + ran (and, for the empty case, that its bound candidate id array is empty). For + the idempotency test, assert the applyLink SQL string contains both + 'resolved_at IS NOT NULL' and "IS DISTINCT FROM 'manual'". For dryRun, assert + no call SQL matches /UPDATE|INSERT|DELETE/ after the read-only eligibility and + candidate SELECTs. + + Follow the RESEARCH test-name map so the -t filters resolve: name the cases so + 'auto-link', 'review', 'empty candidates', and 'idempotent' each appear. + similarity() runs in Postgres, so these tests never touch a real DB — the mock + returns canned score rows (keeps the suite fast and consistent with the rest of + lib/services/*.test.ts, all pure-mock, environment: 'node'). + + + - Test titles include the substrings 'auto-link', 'review', 'empty candidates', and 'idempotent' + - The auto-link test asserts a mock query call whose SQL matches `/UPDATE pax8_companies/` + - The empty-candidates test asserts the review upsert's bound `candidate_company_ids` param is an empty array + - The idempotent test asserts the applyLink SQL contains `resolved_at IS NOT NULL` and `IS DISTINCT FROM 'manual'` + - `npx vitest run lib/services/pax8-company-matcher.test.ts` passes + + + npx vitest run lib/services/pax8-company-matcher.test.ts + + All five decision branches (auto, below-threshold review, near-tie review, empty-candidate review, idempotency guard) plus dryRun are proven with mocked postgresClient.query. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| PAX8 company name → SQL | External string fed into a similarity() comparison — must be a bound parameter | +| matcher writes → billing-sensitive match state | A wrong auto-link misattributes cost data between companies | +| re-sync → human-resolved rows | Automated re-scoring must never overwrite a manual decision | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-12-01 | Tampering | findCandidates similarity() query | mitigate | PAX8 name bound as `$1`, never string-interpolated (parameterized query per postgresClient convention); asserted by the no-concatenation acceptance check | +| T-12-02 | Tampering / Repudiation | decide() / applyLink() | mitigate | Conservative 0.90 threshold + 0.05 tie-margin + top-3 review; unit tests cover below-threshold, near-tie, and empty-candidate branches so wrong-company cost attribution is caught in CI | +| T-12-04 | Tampering | applyLink() re-scoring | mitigate | UPDATE guarded with `match_method IS DISTINCT FROM 'manual'` + `resolved_at IS NOT NULL` NOT EXISTS; eligibility SELECT excludes human-resolved rows; idempotency unit test asserts the guard | +| T-12-06 | Denial of Service | matchPax8Companies loop | accept | Bounded by LIMIT (default 1000) and CANDIDATE_FLOOR keeping candidate lists to ≤5; dataset is ~120 pax8 companies — no scaling concern | +| T-12-SC | Tampering | package installs | accept | No new packages; pg_trgm is a Postgres contrib extension enabled in Plan 01 | + + + +- `npx vitest run lib/services/pax8-company-matcher.test.ts` green. +- `npx tsc --noEmit --pretty` passes. +- No string concatenation of company names into SQL (grep gate). + + + +The matcher auto-links only on conservative, unambiguous, high-confidence matches; flags everything else (including zero-candidate and near-tie cases); never overwrites a human decision; all parameterized. Unit tests prove each branch. + + + +Create `.planning/phases/12-orders-invoices-company-matching/12-03-SUMMARY.md` when done. + diff --git a/.planning/phases/12-orders-invoices-company-matching/12-04-PLAN.md b/.planning/phases/12-orders-invoices-company-matching/12-04-PLAN.md new file mode 100644 index 0000000..d9d9e4a --- /dev/null +++ b/.planning/phases/12-orders-invoices-company-matching/12-04-PLAN.md @@ -0,0 +1,284 @@ +--- +phase: 12-orders-invoices-company-matching +plan: 04 +type: execute +wave: 3 +depends_on: ["12-02", "12-03"] +files_modified: + - lib/services/pax8-sync-service.ts + - lib/services/pax8-sync-service.test.ts +autonomous: true +requirements: [PAX8-06, PAX8-10, PAX8-11] + +must_haves: + truths: + - "fullSync() runs the invoice/line-item sync and the company matcher as two additional entity steps" + - "syncOrders() upserts invoice headers to pax8_orders and line items to pax8_order_items with per-company id, billing period, and dual-cost columns populated" + - "syncOrders() tombstones headers and items no longer returned by PAX8 using the id <> ALL($1) soft-delete pattern" + - "syncCompanyMatches() delegates to matchPax8Companies() and reports a Pax8EntitySyncResult" + artifacts: + - path: "lib/services/pax8-sync-service.ts" + provides: "syncOrders + syncCompanyMatches steps wired into fullSync" + contains: "syncOrders" + - path: "lib/services/pax8-sync-service.test.ts" + provides: "upsert/tombstone + matcher-delegation unit tests" + contains: "syncOrders" + key_links: + - from: "lib/services/pax8-sync-service.ts" + to: "pax8-client listAllInvoices/listAllInvoiceItems" + via: "nested per-invoice fetch loop" + pattern: "listAllInvoiceItems" + - from: "lib/services/pax8-sync-service.ts" + to: "pax8-company-matcher matchPax8Companies" + via: "syncCompanyMatches wrapper" + pattern: "matchPax8Companies" + - from: "lib/services/pax8-sync-service.ts" + to: "fullSync entities array" + via: "entities.push(ordersResult) and entities.push(matchResult)" + pattern: "entities.push" +--- + + +Wire both halves of Phase 12 into the existing Pax8SyncService.fullSync() +orchestration: a new syncOrders() step (historical invoice headers + per-company +line items, PAX8-06) and a new syncCompanyMatches() step delegating to the Plan +03 matcher (PAX8-10, PAX8-11). Both conform to the existing Pax8EntitySyncResult +shape and slot into fullSync's entities.push accumulation unchanged. + +Invoice → items is a nested per-parent fetch (12-RESEARCH.md Pattern 1): page all +94 headers once, then for each header page its ~500-700 items. Per-company cost +lives ONLY on the item (`companyId` is always NULL on the header — Pitfall 1), so +pax8_order_items.pax8_company_id is populated from the item; pax8_orders stays +header-only. Cost column mapping was confirmed live in Plan 02 +(unit_price←price, line_total←amountDue, partner_cost←cost, +partner_cost_total←costTotal) — use the mapping recorded in 12-02-SUMMARY.md. + +Purpose: The integration point that makes the phase's data actually populate on a +sync run. +Output: syncOrders + syncCompanyMatches in pax8-sync-service.ts, wired into +fullSync, plus the service's first unit tests. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md +@.planning/phases/12-orders-invoices-company-matching/12-PATTERNS.md +@.planning/phases/12-orders-invoices-company-matching/12-02-SUMMARY.md + + +Existing pax8-sync-service.ts: +- fullSync(triggeredBy): builds entities: Pax8EntitySyncResult[]; currently pushes + syncCompanies(), syncSubscriptions() (returns { result, referencedProductIds }), + syncProducts(referencedProductIds). Add ordersResult then matchResult after + productsResult, before the rollup that computes success/status/totals. +- syncSubscriptions() (lines 175-253) is the closest upsert+tombstone analog: + `seen: string[]`, INSERT ... ON CONFLICT (id) DO UPDATE SET ..., synced_at=NOW(), + is_deleted=false, deleted_at=NULL; then tombstone UPDATE ... WHERE is_deleted=false + AND id <> ALL($1::uuid[]). Per-method try/catch returning a failed + Pax8EntitySyncResult on error. + +New client methods (Plan 02): this.client.listAllInvoices(): Promise; +this.client.listAllInvoiceItems(invoiceId: string): Promise. + +Matcher (Plan 03): import { matchPax8Companies, Pax8CompanyMatchResult } from +'./pax8-company-matcher'. matchPax8Companies(opts?: { limit?; dryRun? }). + +pax8_orders columns (migration 091): id UUID, pax8_company_id UUID (stays NULL), +order_date TIMESTAMPTZ, total NUMERIC(12,2), status TEXT, currency CHAR(3), +raw_payload JSONB, synced_at, is_deleted, deleted_at. +pax8_order_items columns (091 + 092): id UUID, order_id UUID NOT NULL (FK CASCADE), +product_id UUID, quantity INTEGER, unit_price NUMERIC(12,2), line_total +NUMERIC(12,2), currency CHAR(3), raw_payload, synced_at, is_deleted, deleted_at, ++ pax8_company_id UUID, subscription_id UUID, item_type TEXT, sku TEXT, +description TEXT, start_period TIMESTAMPTZ, end_period TIMESTAMPTZ, partner_cost +NUMERIC(12,2), partner_cost_total NUMERIC(12,2). + + + + + + + Task 1: Add syncOrders() (nested invoice → item upsert + tombstone) + lib/services/pax8-sync-service.ts + + - lib/services/pax8-sync-service.ts (syncSubscriptions lines 175-253 for the upsert+tombstone shape; syncCompanies lines 113-173 for the single-table baseline; the import block lines 12-21; the per-method try/catch error shape lines 168-172) + - lib/services/itglue-sync-service.ts (the syncModels() per-parent nested-fetch iteration referenced by 12-PATTERNS.md Pattern 1 — parent loop calling a child endpoint per id) + - .planning/phases/12-orders-invoices-company-matching/12-02-SUMMARY.md (the CONFIRMED invoice-item cost field mapping — authoritative for the item upsert) + - .planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md (Pitfall 1 header companyId always NULL; Pitfall 2 field mapping; Pattern 1 nested pagination) + + + Extend the type import block to add Pax8Invoice, Pax8InvoiceItem from + '@/lib/types/pax8'. Add a private async syncOrders(): Promise<Pax8EntitySyncResult> + modeled on syncSubscriptions. + + Fetch headers: const invoices = await this.client.listAllInvoices(). Track + seenOrderIds: string[] and seenItemIds: string[]; count ordersUpserted and + itemsUpserted. For each invoice with an id: push id to seenOrderIds; upsert + pax8_orders — INSERT (id, pax8_company_id, order_date, total, status, currency, + raw_payload, synced_at, is_deleted, deleted_at) VALUES ($1, NULL, $2..., NOW(), + false, NULL) ON CONFLICT (id) DO UPDATE SET the same fields + synced_at=NOW(), + is_deleted=false, deleted_at=NULL. Map order_date←invoice.invoiceDate, + total←invoice.total, status←invoice.status, currency←invoice.currencyCode ?? + 'USD'. Keep pax8_company_id NULL (Pitfall 1 — comment why). + + Then fetch that invoice's items: const items = await + this.client.listAllInvoiceItems(invoice.id). For each item with an id: push id + to seenItemIds; upsert pax8_order_items — INSERT (id, order_id, product_id, + quantity, unit_price, line_total, currency, raw_payload, synced_at, is_deleted, + deleted_at, pax8_company_id, subscription_id, item_type, sku, description, + start_period, end_period, partner_cost, partner_cost_total) with order_id = + invoice.id, and the CONFIRMED mapping from 12-02-SUMMARY.md: unit_price←price, + line_total←amountDue, partner_cost←cost, partner_cost_total←costTotal, + pax8_company_id←companyId, subscription_id←subscriptionId, item_type←type, + sku←sku, description←description, start_period←startPeriod, end_period←endPeriod, + product_id←productId, quantity←quantity, currency←currencyCode ?? 'USD', + raw_payload←JSON.stringify(item). ON CONFLICT (id) DO UPDATE SET all the above + + synced_at=NOW(), is_deleted=false, deleted_at=NULL. + + After all invoices processed, tombstone in child-then-parent order to respect + the FK: first UPDATE pax8_order_items SET is_deleted=true, deleted_at=NOW() + WHERE is_deleted=false AND id <> ALL($1::uuid[]) with seenItemIds; then the + same on pax8_orders with seenOrderIds. Guard each with the seen.length === 0 ? + 0 : (...).rowCount ?? 0 pattern from syncSubscriptions. Sum tombstoned across + both tables. + + Return { entity: 'orders', success: true, upserted: ordersUpserted + + itemsUpserted, tombstoned, durationMs: Date.now() - start }. Wrap in the + standard try/catch returning a failed result with the error message + ('[Pax8Sync] Order sync failed:'). + + + - syncOrders() upserts into both `pax8_orders` and `pax8_order_items` + - The item upsert binds `pax8_company_id` from the item's `companyId` and sets `line_total` from `amountDue`, `partner_cost` from `cost`, `partner_cost_total` from `costTotal`, `unit_price` from `price` (per 12-02-SUMMARY.md) + - The header insert leaves pax8_orders.pax8_company_id NULL (Pitfall 1) + - Tombstone runs on pax8_order_items before pax8_orders, both using `id <> ALL($1::uuid[])` + - Returns a Pax8EntitySyncResult with `entity: 'orders'` + - `npx tsc --noEmit --pretty` passes + + + npx tsc --noEmit --pretty 2>&1 | tail -5; grep -c "listAllInvoiceItems" lib/services/pax8-sync-service.ts + + syncOrders performs the nested invoice→item sync with the confirmed cost mapping, per-company id on items, and child-then-parent tombstoning; type-check passes. + + + + Task 2: Add syncCompanyMatches() and wire both steps into fullSync() + lib/services/pax8-sync-service.ts + + - lib/services/pax8-sync-service.ts (fullSync lines 42-109 — the entities.push accumulation and rollup) + - lib/services/pax8-company-matcher.ts (matchPax8Companies signature + Pax8CompanyMatchResult from Plan 03) + - lib/types/pax8.ts (Pax8EntitySyncResult) + + + Import { matchPax8Companies } from './pax8-company-matcher' (and the + Pax8CompanyMatchResult type). Add a private async syncCompanyMatches(): + Promise<Pax8EntitySyncResult> that calls matchPax8Companies() (no dryRun), + then maps its result into a Pax8EntitySyncResult: entity: 'company_matches', + success: true, upserted: result.autoLinked, tombstoned: result.flaggedAmbiguous + + result.flaggedNoCandidate (repurposed here as the flagged-for-review count — + document this in a comment), durationMs: result.durationMs. Wrap in try/catch + returning a failed result ('[Pax8Sync] Company match failed:'). + + In fullSync(), after `entities.push(productsResult);` add: const ordersResult = + await this.syncOrders(); entities.push(ordersResult); then const matchResult = + await this.syncCompanyMatches(); entities.push(matchResult); Matching runs + AFTER orders and companies so pax8_companies is fully populated first. The + existing rollup (success = entities.every, totals = reduce) picks both up + unchanged. + + + - fullSync() contains `entities.push(ordersResult)` and `entities.push(matchResult)` after the products step + - syncCompanyMatches() calls `matchPax8Companies()` and returns a Pax8EntitySyncResult with `entity: 'company_matches'` + - The matcher step runs after syncCompanies (companies populated before matching) + - `npx tsc --noEmit --pretty` passes + + + npx tsc --noEmit --pretty 2>&1 | tail -5; grep -c "matchPax8Companies\|entities.push" lib/services/pax8-sync-service.ts + + Both new steps are wired into fullSync and roll up into the sync result and sync_history counts. + + + + Task 3: Create lib/services/pax8-sync-service.test.ts + lib/services/pax8-sync-service.test.ts + + - syncOrders (via fullSync or a directly-invoked instance with a mock client): a two-invoice fixture with items produces upserts into both pax8_orders and pax8_order_items; the item upsert binds companyId into the pax8_company_id parameter position + - unseen headers/items are tombstoned via id <> ALL (the tombstone UPDATE runs) + - syncCompanyMatches delegates to the mocked matchPax8Companies and returns entity 'company_matches' + - fullSync's entities array includes an 'orders' and a 'company_matches' result + + + - lib/services/pax8-client.test.ts (vi.fn / vi.mock discipline to mirror) + - lib/services/pax8-sync-service.ts (the constructor accepts an optional client: `constructor(client?: Pax8Client)` — inject a mock client; the query shapes to assert) + - .planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md (Validation Architecture — Wave 0 note that no prior Pax8SyncService test exists; mock postgresClient.query) + + + Create the service's first unit test. Mock '@/lib/services/postgres-client' + (default: { query: vi.fn() }) and mock './pax8-company-matcher' so + matchPax8Companies returns a canned Pax8CompanyMatchResult ({ scanned: 2, + autoLinked: 1, flaggedAmbiguous: 1, flaggedNoCandidate: 0, durationMs: 1 }). + Construct the service with an injected mock Pax8Client exposing + listAllCompanies/listAllSubscriptions/listAllProducts (return []) and + listAllInvoices (two headers) + listAllInvoiceItems (items per header, one with + a companyId + subscriptionId + amountDue/cost/costTotal). + + Assert: after fullSync (or a direct syncOrders call if you expose it via the + instance), the mocked postgresClient.query received an INSERT INTO pax8_orders + and an INSERT INTO pax8_order_items call; for the item insert, the params array + includes the fixture's companyId value (pax8_company_id) and its amountDue value + (line_total). Assert a tombstone UPDATE against pax8_order_items and one against + pax8_orders were issued. Assert the returned Pax8SyncResult.entities contains an + object with entity 'orders' and one with entity 'company_matches'. Keep DB-free + (all mocked), environment node. + + + - Test asserts INSERT calls against both `pax8_orders` and `pax8_order_items` + - Test asserts the item insert params include the fixture companyId (pax8_company_id) and amountDue (line_total) + - Test asserts tombstone UPDATEs against both tables + - Test asserts fullSync entities include `entity: 'orders'` and `entity: 'company_matches'` + - `npx vitest run lib/services/pax8-sync-service.test.ts` passes + + + npx vitest run lib/services/pax8-sync-service.test.ts + + The sync service's first unit tests prove the nested order upsert (with per-company mapping), tombstoning, and matcher delegation. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| PAX8 API → sync service | External invoice/item JSON persisted to Postgres | +| sync writes → Postgres | Upserts/tombstones on billing-history tables | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-12-01 | Tampering | syncOrders upsert SQL | mitigate | All PAX8 field values bound as `$n` params via postgresClient.query (no interpolation), matching the existing syncSubscriptions convention | +| T-12-07 | Denial of Service | nested invoice→item fetch | accept | ~94 headers × ~500-700 items (≈56k rows), well within the client's 1000/min handling and normal Postgres sizes (12-RESEARCH.md Assumption A3); full sync only, no unbounded growth path | +| T-12-08 | Repudiation | sync_history rollup | mitigate | orders + company_matches counts fold into the existing sync_history record via fullSync's rollup, preserving the audit trail of what each run changed | +| T-12-05 | Elevation of Privilege | client calls | mitigate | Only read (GET) client methods are called (listAll*); PAX8-08 read-only invariant preserved | +| T-12-SC | Tampering | package installs | accept | No new packages | + + + +- `npx vitest run lib/services/pax8-sync-service.test.ts` and `npx vitest run lib/services/pax8-client.test.ts lib/services/pax8-company-matcher.test.ts` green. +- `npx tsc --noEmit --pretty` passes. +- `npm test` (full suite) green before the wave merges. + + + +fullSync runs orders sync + company matching as first-class entity steps; invoice items land with per-company id, billing period, and dual-cost columns; unseen rows tombstone; matcher delegation reports through the sync result. Unit tests prove it. + + + +Create `.planning/phases/12-orders-invoices-company-matching/12-04-SUMMARY.md` when done. + diff --git a/.planning/phases/12-orders-invoices-company-matching/12-05-PLAN.md b/.planning/phases/12-orders-invoices-company-matching/12-05-PLAN.md new file mode 100644 index 0000000..b2b043e --- /dev/null +++ b/.planning/phases/12-orders-invoices-company-matching/12-05-PLAN.md @@ -0,0 +1,170 @@ +--- +phase: 12-orders-invoices-company-matching +plan: 05 +type: execute +wave: 4 +depends_on: ["12-04"] +files_modified: + - scripts/verify-pax8-orders-matching.ts +autonomous: false +requirements: [PAX8-06, PAX8-10, PAX8-11] + +must_haves: + truths: + - "A real full sync populates pax8_order_items with per-company id and billing period rows (SC#1)" + - "At least one pax8_companies row carries a confident auto-match (autotask_company_id + match_confidence >= 0.90) (SC#2)" + - "No-match and ambiguous PAX8 companies have pax8_company_match_review rows (empty array for no-match) (SC#3)" + - "Running the sync a second time does not change already-resolved matches (SC#4 idempotency)" + artifacts: + - path: "scripts/verify-pax8-orders-matching.ts" + provides: "live full-sync + DB assertion harness for the phase's 4 success criteria" + key_links: + - from: "scripts/verify-pax8-orders-matching.ts" + to: "pax8-sync-service fullSync" + via: "invokes a real sync then queries the resulting rows" + pattern: "fullSync|/api/pax8/sync" +--- + + +Prove the phase's four ROADMAP success criteria against real PAX8 data and the +real dev Postgres — the truest verification, mirroring Phase 11's live-run +verification plan (11-03). Unit tests (Plans 02-04) prove the logic with mocks; +this plan proves the integration end-to-end and lets the developer eyeball the +match distribution, since conservative billing-reconciliation matching (D-01) is +a judgment the user explicitly cares about. + +Purpose: End-to-end confidence that orders populate, auto-matches persist, +flags are created (never silently guessed), and a human decision survives a +re-sync. +Output: a reusable verification script + a human confirmation of the results. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md +@.planning/phases/12-orders-invoices-company-matching/12-CONTEXT.md + + +- POST /api/pax8/sync (existing route) fires a fire-and-forget fullSync; + GET /api/pax8/sync returns inProgress + counts + history. App runs on port 3100. +- Alternatively invoke getPax8SyncService().fullSync('verify') directly from a + tsx script (dotenv from ../.env.local), like scripts/verify-pax8-auth.ts. +- DB access for assertions: docker exec pulse-postgres psql -U pulse_user -d pulse_autotask. +- Migration 092 (Plan 01) must already be applied to the dev DB. + + + + + + + Task 1: Write scripts/verify-pax8-orders-matching.ts (live full-sync + assertions) + scripts/verify-pax8-orders-matching.ts + + - scripts/verify-pax8-auth.ts (dotenv + getPax8Client + secret-safe logging pattern; run via npx tsx) + - lib/services/pax8-sync-service.ts (getPax8SyncService / fullSync from Plan 04) + - .planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md (Live-verified distribution — expected ~80 exact matches at 1.00, so auto-links should be substantial; ~56k order items total) + + + Create scripts/verify-pax8-orders-matching.ts (npx tsx), dotenv from + ../.env.local, secret-safe logging only. It runs getPax8SyncService().fullSync + ('verify') once, awaiting completion, then queries the dev DB (via a direct + postgresClient.query, since the script runs in-process) and prints a report: + + - SC#1: count of pax8_order_items WHERE is_deleted=false AND pax8_company_id + IS NOT NULL, and count with start_period IS NOT NULL. Both must be > 0. + - SC#2: count of pax8_companies WHERE autotask_company_id IS NOT NULL AND + match_confidence >= 0.90 AND match_method = 'pg_trgm'. Print a sample of 5 + (pax8 name, matched autotask company_name, score). Count must be > 0. + - SC#3: count of pax8_company_match_review rows WHERE resolved_at IS NULL; of + those, count with candidate_company_ids = '{}' (no-match, D-03) and count + with array_length >= 1 (ambiguous/below-threshold). Report both. + - SC#4: capture the (pax8_company_id, autotask_company_id, match_confidence) + for auto-matched rows into a snapshot; then run fullSync('verify') a SECOND + time; re-read the same rows and assert none of the previously-auto-matched + rows changed AND that no row with a resolved review (simulate none exist yet + — Phase 14 owns manual resolution) was mutated. Print PASS/FAIL for + idempotency (auto-match set stable across two runs). + + Print a final verdict block: SC#1..SC#4 each PASS/FAIL with the underlying + numbers. Exit non-zero if any SC fails so the verify command surfaces it. + Never log the client secret or token. + + + - `npx tsx scripts/verify-pax8-orders-matching.ts` runs a real sync and prints a verdict block with SC#1, SC#2, SC#3, SC#4 each marked PASS or FAIL and their counts + - SC#1 count of pax8_order_items with non-null pax8_company_id is > 0 + - SC#2 count of pax8_companies with autotask_company_id set and match_confidence >= 0.90 is > 0 + - SC#3 reports both the empty-candidate (no-match) review count and the ambiguous review count + - SC#4 shows the auto-match set is identical across two consecutive full syncs (idempotency PASS) + - Script exits 0 only when all four criteria pass; no secret/token appears in output + + + npx tsx scripts/verify-pax8-orders-matching.ts; echo "exit=$?" + + A live full sync populates orders/items and match state; all four success criteria print PASS with real numbers; the auto-match set is stable across two runs. + + + + Task 2: Human review of the auto-match sample + success-criteria verdict + PAUSE for the developer. This is a human-verify checkpoint: the developer reviews the SC#1..SC#4 verdict block and the sample of auto-matches produced by Task 1's script, confirming the conservative matches are correct (D-01) and the flagged queue is genuinely ambiguous/no-match. Do not auto-approve — resume only on the developer signal below. + + Historical PAX8 invoice line items are synced into pax8_order_items (with + per-company id + billing period + dual cost), PAX8 companies are auto-linked to + Autotask companies at similarity >= 0.90 (conservative, no near-ties), and + no-match/ambiguous companies are flagged in pax8_company_match_review rather + than silently guessed. The verification script ran two full syncs and confirmed + idempotency. + + + 1. Review the verdict block from `npx tsx scripts/verify-pax8-orders-matching.ts` + (SC#1..SC#4 all PASS). + 2. Eyeball the SC#2 sample of 5 auto-matches: each PAX8 company name should + clearly be the same real company as its matched Autotask company_name. + Conservative intent (D-01): if any sampled auto-match looks wrong, that is a + problem — report it (the threshold may need raising). + 3. Optionally spot-check the review queue: + `docker exec pulse-postgres psql -U pulse_user -d pulse_autotask -c "SELECT p.name, r.candidate_company_ids, r.match_confidences FROM pax8_company_match_review r JOIN pax8_companies p ON p.id=r.pax8_company_id WHERE r.resolved_at IS NULL LIMIT 10"` + — confirm flagged companies are genuinely ambiguous or genuinely have no + clear Autotask match. + 4. Confirm a per-company cost query returns rows: + `docker exec pulse-postgres psql -U pulse_user -d pulse_autotask -c "SELECT pax8_company_id, count(*), sum(line_total) FROM pax8_order_items WHERE is_deleted=false GROUP BY 1 ORDER BY 2 DESC LIMIT 5"`. + + Type "approved" if the auto-matches look correct and all SC pass, or describe any wrong match / bad flag so the threshold or mapping can be adjusted. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| live PAX8 API → dev Postgres | Real historical billing data written during verification | +| verification output → developer | Match results reviewed by a human before the phase is accepted | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-12-02 | Tampering / Repudiation | auto-match correctness | mitigate | Human-verify checkpoint reviews a sample of auto-matches; conservative D-01 threshold means wrong auto-links should be near-zero and any spotted one blocks approval | +| T-12-04 | Tampering | idempotency | mitigate | SC#4 double-run asserts the auto-match set is stable and (by design) resolved rows are never overwritten | +| T-12-03 | Information Disclosure | verification script logging | mitigate | Secret-safe logging copied from verify-pax8-auth.ts; summary/counts only | +| T-12-SC | Tampering | package installs | accept | No new packages | + + + +- Verification script exits 0 with SC#1..SC#4 all PASS. +- Human confirms the auto-match sample is correct. +- Full test suite (`npm test`) still green. + + + +All four ROADMAP Phase 12 success criteria proven against live data; auto-match sample human-approved; idempotency confirmed across two syncs. + + + +Create `.planning/phases/12-orders-invoices-company-matching/12-05-SUMMARY.md` when done. Record the SC counts and the human verdict. +