From b99f84ffa02963f4004230fb1dc2e808eb72833d Mon Sep 17 00:00:00 2001 From: lorentz Date: Fri, 10 Jul 2026 21:59:05 -0400 Subject: [PATCH] docs(12): research phase domain --- .../12-RESEARCH.md | 771 ++++++++++++++++++ 1 file changed, 771 insertions(+) create mode 100644 .planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md diff --git a/.planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md b/.planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md new file mode 100644 index 0000000..922eede --- /dev/null +++ b/.planning/phases/12-orders-invoices-company-matching/12-RESEARCH.md @@ -0,0 +1,771 @@ +# Phase 12: Orders/Invoices & Company Matching - Research + +**Researched:** 2026-07-11 +**Domain:** PAX8 invoice/line-item historical sync + Postgres fuzzy-name entity matching +**Confidence:** HIGH (both major open questions were resolved with **live, authenticated calls** against the real PAX8 API and the real dev Postgres database — not documentation guesses) + +## Summary + +Two independent problems, both now de-risked with hard evidence instead of assumptions. + +**Orders/invoices:** PAX8's `/invoices` resource is the *partner's own consolidated +monthly bill* — one header row per billing period, **not** one per end-customer. +`companyId` is `null` on every invoice header in this account's live data. The +per-customer cost data lives one level down, on `/invoices/{id}/items`, where +`companyId` is populated on **every** item. This is a genuine mismatch against +`migrations/091_pax8_tables.sql`: `pax8_orders.pax8_company_id` will always be +`NULL`, and `pax8_order_items` has **no `company_id` column at all** today — a +required new migration, not optional. Full history is small and safe: 94 +invoices total since 2019-05-01, ~500-700 items each (≈50-65k rows total), +zero pagination-limit or rate-limit risk given the client's existing 1000/min +handling. + +**Company matching:** Postgres `pg_trgm` (not yet enabled — only `uuid-ossp` +and `pgcrypto` are) is the right tool: no new npm dependency, matches existing +extension-based conventions, and a live test against the real 118 +`pax8_companies` rows vs. 242 active `companies` rows shows a **clean +separation** — every genuine match scored 1.00 (case/whitespace-only +differences already normalized away by `similarity()`, which is +case-insensitive), and the single highest-scoring *non*-match in the entire +dataset was 0.70 ("Thoroughbred Construction Company" vs "...Construction +Group" — arguably a real non-match). Recommend an auto-link threshold of +**0.90** (`similarity() >= 0.90`), well clear of every observed false-positive +risk, satisfying D-01's "conservative" mandate with room to spare. + +**Primary recommendation:** Add a new migration (092) that (a) enables +`pg_trgm`, (b) adds `company_id`, `subscription_id`, `type`, `start_period`, +`end_period` and dual-cost columns to `pax8_order_items`, and (c) adds +`autotask_company_id` / `match_confidence` / `matched_at` / `match_method` +columns directly to `pax8_companies` (mirroring `device_external_ids`' +`configuration_item_id`/`link_confidence`/`linked_at` precedent exactly). +Build the matcher as a close structural port of +`lib/services/device-link-reconciler.ts`, scored with `similarity()` at a +0.90 auto-link floor and a tie-margin flag for any second candidate within +0.05 of the top score. + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|------------------| +| PAX8-06 | Sync PAX8 orders/invoices (historical line items) into Postgres, enabling cost reconciliation over time | Confirmed via live API: `/invoices` (94 header rows, full history since 2019-05) + `/invoices/{id}/items` (per-company, per-period cost lines) is the correct source. Schema gap identified and mapped field-for-field below. | +| PAX8-10 | PAX8 companies automatically matched to Autotask companies by fuzzy name similarity at sync time | `pg_trgm` `similarity()` verified live against real data; threshold 0.90 recommended with evidence. | +| PAX8-11 | Unmatched/ambiguous matches flagged, never silently guessed | `pax8_company_match_review` schema already exists (migration 091); matcher logic ports `device-link-reconciler.ts`'s conflict-detection shape. | + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions + +- **D-01:** Auto-link (no human review) only on a high-confidence fuzzy match + — not exact-string-only, but a high similarity bar (near-identical names: + punctuation/case/whitespace differences, minor typos). The user explicitly + chose "be conservative — fewer auto-matches" over a looser threshold or + deferring the number to the planner's judgment alone: bias toward flagging + borderline cases for review rather than risking a wrong auto-link, since + this data feeds cost/billing reconciliation. The planner should document + the exact numeric threshold chosen (and the library/approach) directly in + the plan so it's easy to find and tune later — this is an initial number, + not a permanently fixed one. +- **D-02:** Even an otherwise-exact name match must be flagged for review + (not auto-linked) if it's ambiguous against more than one Autotask company + sharing that same/very-similar name (e.g. franchise locations, "Acme Inc" + vs "Acme Holdings Inc"). No silent tie-breaking — ever, even when the + string match itself looks perfect. +- **D-03:** When a PAX8 company has zero reasonably-similar Autotask + candidates at all, still create a `pax8_company_match_review` row with an + empty `candidate_company_ids` array — never silently drop it (PAX8-11). + This flags it into the admin queue so Phase 14's UI can offer a manual + search/pick, or confirm there truly isn't a matching Autotask company yet. +- **D-04:** Ambiguous-match review rows carry the **top 3** highest-scoring + Autotask candidates (matches the existing `device_link_review` precedent's + style of showing a small ranked list, not every plausible match). +- **D-05:** PAX8 companies still sitting unresolved in the review queue are + **re-scored on every subsequent full sync**, not matched once and left + alone — candidates can improve over time (e.g. a renamed/newly-created + Autotask company scores better later) without requiring a manual + re-trigger. This does NOT apply to already-resolved matches: SC#4's + idempotency guarantee (a manually-confirmed match is never overwritten by + a later sync) still holds — only rows with `resolved_at IS NULL` are + eligible for re-scoring. + +### Claude's Discretion + +- **Exact threshold value / library choice** — user wants "conservative," + not a specific number. The planner should research common fuzzy + name-matching approaches (e.g., trigram similarity via Postgres + `pg_trgm`, or a JS library) and pick/document a concrete high threshold, + erring toward fewer auto-matches per D-01. + **Resolved by this research: `pg_trgm`, threshold 0.90 — see Standard Stack.** +- **Company name normalization nuances** (legal suffixes like LLC/Inc/Corp, + punctuation, abbreviations) — not discussed in depth this session (user + deselected this gray area). Planner/researcher should investigate whether + Autotask company names in this instance commonly carry legal suffixes + PAX8 names don't (or vice versa) and decide normalization rules + accordingly; err toward the conservative stance (D-01/D-02) if uncertain. + **Resolved by this research: normalize case/whitespace/trailing punctuation + only; do NOT strip legal suffixes — see Common Pitfalls.** +- **Orders/invoices historical lookback window** — not discussed in depth + this session (user deselected this gray area). `migrations/091`'s comment + states "sync pulls full order history on first sync in a later phase" + (this phase) — the planner should confirm this is still the intent (full + history, no bounded window) during research, and flag to the user if + PAX8's invoices API makes "full history" impractical (e.g., no + pagination limit safety, or a very large per-company invoice count). + **Resolved by this research: full history is small (94 invoices, ~56k + items total) and practical — no bounded window needed. See Environment + Availability / Pitfall 2.** + +### Deferred Ideas (OUT OF SCOPE) + +No capabilities were deferred outside this phase's boundary. The `/pax8` UI, +manual resolution flow (PAX8-12), and scheduler cron wiring (PAX8-07/09) are +already sequenced into Phases 13-14 — not deferred, just not this phase's job. +This phase does NOT: build `/pax8` UI, wire a cron schedule, or build the +admin manual-resolution flow. It only populates `pax8_orders`, +`pax8_order_items`, and matches/flags `pax8_companies` rows. + + +## Architectural Responsibility Map + +| Capability | Primary Tier | Secondary Tier | Rationale | +|------------|-------------|----------------|-----------| +| PAX8 invoice/item HTTP fetch + pagination | API / Backend (service layer) | — | `Pax8Client` extension, same as existing `listAllCompanies` etc. — no browser/UI involvement this phase. | +| Invoice/item upsert + tombstone reconciliation | API / Backend (service layer) | Database | `Pax8SyncService`-owned, writes via `postgresClient`; DB owns constraints (FKs, unique indexes). | +| Fuzzy company-name scoring | Database (Postgres `pg_trgm`) | API / Backend | `similarity()` runs in Postgres itself (SQL, not JS) — the backend service issues the query and interprets results, but the actual string-distance computation is DB-tier, matching the existing `pg_trgm`-adjacent extension pattern already used in this codebase (`pgcrypto`, `uuid-ossp`). | +| Match/review persistence (`pax8_companies` columns, `pax8_company_match_review`) | Database | API / Backend | Schema already exists (migration 091) for the review table; this phase adds columns for the confident-match case. | +| No UI this phase | — | — | `UI hint: no` per ROADMAP.md — Phase 14 renders review queue. | + +## Standard Stack + +### Core + +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| `pg_trgm` (Postgres extension) | Bundled with Postgres 16 (`contrib`) | Trigram-based fuzzy string similarity (`similarity()`, `%` operator, `SIMILARITY()` GIN/GiST index support) | [VERIFIED: live query against this project's own Postgres 16 container — `CREATE EXTENSION IF NOT EXISTS pg_trgm;` succeeded, confirming the extension ships with this Postgres image] Zero new npm dependency; matches the codebase's existing pattern of enabling Postgres contrib extensions per-migration (`uuid-ossp` in migration 001, `pgcrypto` in migrations 069/070) rather than adding a JS string-similarity library. No such library exists in `package.json` today. | + +### Supporting + +None — no new npm packages are required for this phase. `pg_trgm` is a +built-in Postgres extension enabled via `CREATE EXTENSION IF NOT EXISTS +pg_trgm;` inside the phase's new migration; it is not an npm dependency and +does not appear in `package.json`. + +### Alternatives Considered + +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| `pg_trgm` `similarity()` | JS library (`string-similarity`, `fastest-levenshtein`, `natural`) | Would require a new npm dependency and running comparisons in application code (fetch all companies, loop in JS) instead of letting Postgres do set-based scoring. No existing precedent for a fuzzy-match npm lib in this codebase; `pg_trgm` fits the "extend Postgres, don't add packages" convention already established by `uuid-ossp`/`pgcrypto`. Not recommended. | +| `pg_trgm` `similarity()` | Exact-match only (`LOWER(TRIM(a)) = LOWER(TRIM(b))`) | Simpler, zero risk of false positive, but misses near-identical names with a typo, extra space, or reordered punctuation — undercuts PAX8-10's "fuzzy" requirement and would push far more companies into the review queue than D-01 intends (conservative ≠ exact-only; D-01 explicitly says "not exact-string-only"). | +| `pg_trgm` `similarity()` | Levenshtein distance (`fuzzystrmatch` extension, also Postgres contrib) | Also viable and also a zero-new-dependency Postgres extension. `pg_trgm`'s `similarity()` (0.0-1.0 normalized score) is easier to reason about as a single conservative threshold than a raw edit-distance integer that varies with string length; `pg_trgm` is also the more common choice for this exact "fuzzy company name match" use case in the wider Postgres ecosystem. Either would work; `pg_trgm` is recommended for the normalized 0-1 score alone. | + +**Installation:** +```sql +-- Inside the new migration (e.g. migrations/092_pax8_orders_company_matching.sql) +CREATE EXTENSION IF NOT EXISTS pg_trgm; +``` + +**Version verification:** `pg_trgm` is bundled with the Postgres 16 server +image already running in this project's `pulse-postgres` container — +verified live: `CREATE EXTENSION IF NOT EXISTS pg_trgm;` executed +successfully with no download/install step. No separate version to track; +it ships with the Postgres major version (16) already pinned in +`docker-compose.yml`. + +## Package Legitimacy Audit + +**Not applicable this phase.** No new npm, pip, or cargo packages are being +installed. The only new dependency is the Postgres `pg_trgm` contrib +extension, verified live against the project's actual running Postgres 16 +container (see above) — not an installable package subject to the +slopcheck/registry-verification protocol. + +**Packages removed due to slopcheck [SLOP] verdict:** none (n/a) +**Packages flagged as suspicious [SUS]:** none (n/a) + +## Architecture Patterns + +### System Architecture Diagram + +``` + PAX8 API (api.pax8.com/v1) + │ + ├─ GET /invoices?page&size (94 rows total, full history) + │ │ + │ └─▶ Pax8Client.listAllInvoices() [paginateAll, size=200 → 1 page] + │ + └─ GET /invoices/{invoiceId}/items?page&size (~500-700 rows per invoice) + │ + └─▶ Pax8Client.listAllInvoiceItems(invoiceId) + [paginateAll per invoice — nested/per-parent fetch, + invoked once per invoice header, NOT a flat top-level list] + │ + ▼ + Pax8SyncService.syncOrders() + 1. fetch all invoice headers + 2. for each header: fetch all its items + 3. upsert header → pax8_orders (company_id stays NULL — see Pitfall 1) + 4. upsert items → pax8_order_items (company_id populated per item) + 5. tombstone anything not seen (full-list reconciliation, same + pattern as syncCompanies/syncSubscriptions/syncProducts) + │ + ▼ + Pax8SyncService.syncCompanyMatches() [D-05: only runs against + pax8_companies rows where autotask_company_id IS NULL + OR the pax8_company_match_review row for it has resolved_at IS NULL] + 1. SELECT unresolved pax8_companies + 2. for each: SELECT companies ORDER BY similarity(name, company_name) DESC + 3. score >= 0.90 AND no second candidate within 0.05 → auto-link + (write autotask_company_id/match_confidence/matched_at/match_method + directly on pax8_companies — mirrors device_external_ids' pattern) + score >= 0.90 but a near-tie exists, OR best score < 0.90, + OR zero candidates → write/update + pax8_company_match_review (top 3 candidates, or empty array per D-03) + │ + ▼ + Postgres: pax8_orders / pax8_order_items / pax8_companies / + pax8_company_match_review + │ + ▼ + (Phase 14 — /pax8 UI reads pax8_company_match_review + the + auto-match columns on pax8_companies; out of scope this phase) +``` + +### Recommended Project Structure + +``` +lib/services/ +├── pax8-client.ts # extend: listAllInvoices(), listAllInvoiceItems(invoiceId) +├── pax8-sync-service.ts # extend: syncOrders() step in fullSync() +├── pax8-company-matcher.ts # NEW — company-matching logic, ported from +│ # device-link-reconciler.ts's structure +lib/types/ +├── pax8.ts # extend: Pax8Invoice, Pax8InvoiceItem types +migrations/ +├── 092_pax8_orders_company_matching.sql # NEW — pg_trgm + schema gaps below +``` + +### Pattern 1: Per-parent nested pagination (invoice → items) + +**What:** Unlike `listAllCompanies`/`listAllSubscriptions`/`listAllProducts` +(flat top-level lists), invoice items are a **child resource of each invoice** +(`/invoices/{invoiceId}/items`). There is no flat `/invoice-items` endpoint. +**When to use:** Fetching all historical line items requires: (1) page +through all 94 invoice headers once, (2) for each header, page through its +own items (typically 3-4 pages at size=200 given ~500-700 items/invoice). +**Closest existing analog:** `lib/services/itglue-sync-service.ts`'s +`syncModels()` (per Phase 11's `11-PATTERNS.md`) — iterate parent entities, +call a child-relationship endpoint per parent. Adapt that iteration shape; +`paginateAll` itself doesn't need to change, just call it once per +invoice ID. + +```typescript +// Source: pattern derived from live API verification (2026-07-11) — +// api.pax8.com/v1/invoices/{invoiceId}/items confirmed to accept the +// same page/size/Pax8PageEnvelope shape as the flat list endpoints. +async listAllInvoiceItems(invoiceId: string): Promise { + return this.paginateAll((page, size) => + this.fetchJson>( + `/invoices/${invoiceId}/items?page=${page}&size=${size}`, + ), + ); +} + +async listAllInvoices(): Promise { + return this.paginateAll((page, size) => + this.fetchJson>(`/invoices?page=${page}&size=${size}`), + ); +} +``` + +### Pattern 2: Confidence-ranked matching, ported from `device-link-reconciler.ts` + +**What:** Cascading match strategies ranked by confidence; conflicts (2+ +candidates) are logged for review, never auto-merged; a resolved/unresolved +lifecycle gate protects human decisions from being overwritten. +**When to use:** Directly reusable structure for company-name matching — +same shape, different scoring function (`similarity()` instead of +serial/MAC/hostname exact lookups). +**Example (adapted, not copied verbatim — company matching has exactly one +"strategy" — trigram similarity — rather than device-link-reconciler's +cascade of several):** + +```typescript +// Source: adapted from lib/services/device-link-reconciler.ts's +// applyLink() / recordConflict() / pickBestCandidate() shape (lines 113-181) +const AUTO_LINK_THRESHOLD = 0.90; // D-01: conservative, initial/tunable value +const TIE_MARGIN = 0.05; // D-02: candidates within this margin of the + // top score are treated as ambiguous + +interface CompanyCandidate { + autotask_company_id: number; + score: number; // pg_trgm similarity(), 0.0-1.0 +} + +async function findCandidates(pax8Name: string): Promise { + const res = await postgresClient.query<{ id: string; score: string }>( + `SELECT id::text, similarity($1, company_name)::text AS score + FROM companies + WHERE is_active = true + AND similarity($1, company_name) > 0.3 -- floor: keep candidate list small + ORDER BY score DESC + LIMIT 5`, + [pax8Name], + ); + return res.rows.map(r => ({ autotask_company_id: Number(r.id), score: Number(r.score) })); +} + +function decide(candidates: CompanyCandidate[]): + | { kind: 'auto'; match: CompanyCandidate } + | { kind: 'review'; top3: CompanyCandidate[] } { + if (candidates.length === 0) return { kind: 'review', top3: [] }; // D-03 + const [best, second] = candidates; + const tie = second && best.score - second.score < TIE_MARGIN; + if (best.score >= AUTO_LINK_THRESHOLD && !tie) { + return { kind: 'auto', match: best }; // high confidence, no ambiguity + } + return { kind: 'review', top3: candidates.slice(0, 3) }; // D-02/D-04 +} +``` + +```sql +-- applyLink() equivalent — mirrors device_external_ids' UPDATE shape exactly +-- (lib/services/device-link-reconciler.ts lines 113-127) +UPDATE pax8_companies + SET autotask_company_id = $2, + match_confidence = $3, + match_method = 'pg_trgm', + matched_at = NOW() + WHERE id = $1 + AND resolved_at IS NULL; -- D-05: never touch a manually-resolved row + -- (resolved_at tracked via the review table per pax8 company — see + -- Schema section below for exact re-scoring gate query) +``` + +```sql +-- recordConflict() equivalent — mirrors device_link_review's upsert +-- (lib/services/device-link-reconciler.ts lines 152-172) +INSERT INTO pax8_company_match_review + (pax8_company_id, candidate_company_ids, match_confidences) +VALUES ($1, $2::bigint[], $3::text[]) +ON CONFLICT (pax8_company_id) WHERE resolved_at IS NULL +DO UPDATE SET candidate_company_ids = EXCLUDED.candidate_company_ids, + match_confidences = EXCLUDED.match_confidences, + detected_at = NOW(); +``` + +### Anti-Patterns to Avoid + +- **Stripping legal suffixes (LLC/Inc/Corp) during normalization:** would + reduce distinguishing information between genuinely distinct entities + (e.g. "Acme Inc" vs "Acme Holdings Inc" — exactly the case D-02 calls + out). Normalize only case, leading/trailing whitespace, and repeated + internal whitespace — not legal-form tokens. +- **Relying on `pax8_orders.pax8_company_id`** for per-company cost + queries: it will be `NULL` on every row (see Pitfall 1). All per-company + cost joins must go through `pax8_order_items.pax8_company_id` (new + column), not the header. +- **Building a flat `/invoice-items` paginator:** the endpoint is + per-invoice (`/invoices/{id}/items`); there is no top-level items list. +- **Auto-linking on an exact string match alone without checking for a + second near-tied candidate:** violates D-02 explicitly — always check for + ambiguity even at score 1.0. +- **Re-scoring already-resolved review rows:** violates D-05/SC#4 — + re-scoring must exclude any `pax8_companies` row that already has a + human-confirmed match (`resolved_at IS NOT NULL` on its review row, or + `autotask_company_id` set via manual resolution rather than the auto + scorer). + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| Fuzzy string similarity scoring | Custom Levenshtein/Jaro-Winkler in TypeScript | Postgres `pg_trgm` `similarity()` | Set-based, indexable (GIN/GiST `gin_trgm_ops` if the review query ever needs to scale beyond ~250 companies), no new npm dependency, and it's already proven case-insensitive and punctuation-tolerant against this project's real data (see Summary). Hand-rolling in JS means fetching all 242 companies into memory per pax8 company and re-implementing what Postgres already ships. | +| Pagination-until-exhausted for nested per-parent resources | A bespoke recursive fetch loop for invoice items | Reuse the existing private `paginateAll` helper in `pax8-client.ts`, called once per invoice ID | The helper is already generic over any `(page, size) => Pax8PageEnvelope` fetcher — no new pagination logic needed, just a new call site. | +| Match/review conflict bookkeeping | A new bespoke review-table shape/lifecycle | `pax8_company_match_review` (already exists, migration 091, copied field-for-field from `device_link_review`) | Schema, indexes, and the "one open review per subject" partial-unique-index pattern are already built and proven in production by the device-linking feature. | + +**Key insight:** Every piece of this phase's hard problem (fuzzy matching, +paginated nested fetch, conflict/review bookkeeping) already has a working, +in-repo precedent. The job is porting/extending, not inventing. + +## Runtime State Inventory + +Not applicable — this is a greenfield sync/matching feature, not a +rename/refactor/migration phase. + +## Common Pitfalls + +### Pitfall 1: Invoice header `companyId` is always `null` — company data lives on line items + +**What goes wrong:** A naive implementation that reads `migrations/091`'s +`pax8_orders.pax8_company_id` column and expects to populate it from the +invoice header's `companyId` field will find that field is `null` on +100% of real invoices. +**Why it happens:** [VERIFIED: live `GET /v1/invoices` call against +`api.pax8.com`, 2026-07-11] PAX8's `/invoices` resource is the **partner's +own consolidated monthly bill** (one row per billing period across the +entire reseller account — `partnerName: "Wulf Consulting, Inc."` on every +row), not a per-end-customer invoice. The per-customer breakdown is only +available on `/invoices/{id}/items`, where every sampled item (200/200 in +one invoice) had a non-null `companyId` and `companyName`. +**How to avoid:** Add a `pax8_company_id` (soft ref, `UUID`, no hard FK — +matching the existing soft-ref convention for `pax8_subscriptions`/ +`pax8_orders`) column to `pax8_order_items` in the new migration. All +per-company cost reconciliation queries join through the item table, not +the header. `pax8_orders.pax8_company_id` remains in the schema (can't +edit a committed migration) but will simply stay `NULL` for every real row +— document this so a future maintainer doesn't treat it as a bug. +**Warning signs:** Any query trying to filter/join `pax8_orders` by +company will silently return zero rows. + +### Pitfall 2: `pax8_order_items` schema doesn't yet match PAX8's real field names + +**What goes wrong:** `migrations/091`'s comment describes +`pax8_order_items` as having `unit_price`/`line_total` mirroring PAX8's +"InvoiceItem" object — but the live response uses different field names +entirely, and — critically — has **two cost dimensions** (customer-facing +and partner-facing), not one. +**Why it happens:** The migration was written before this phase's live +verification. [VERIFIED: live `GET /v1/invoices/{id}/items` response, +2026-07-11] the actual item shape is: +```json +{ + "id": "39d199e1-...", "type": "subscription", "externalId": "29861467", + "companyId": "6ff975bd-...", "forCompanyId": null, + "companyName": "1 of 1 MotorSports", + "startPeriod": "2026-06-03", "endPeriod": "2026-07-02", + "quantity": 9, "unitOfMeasure": "User", "term": "Monthly", + "sku": "MST-NCE-103-C100", + "description": "Microsoft 365 Business Premium [New Commerce Experience]", + "rateType": "Flat", "chargeType": "per", + "price": 26.4, "subTotal": 199.58, + "cost": 22.176, "costTotal": 199.58, + "total": 237.6, "amountDue": 199.58, + "productId": "05df4303-...", "productName": "Microsoft 365 Business Premium [New Commerce Experience]", + "vendorName": "Microsoft", "billingFee": 0, "billingFeeRate": 0, + "currencyCode": "USD", "salesTax": 0, + "subscriptionId": "797a2fac-..." +} +``` +Distinct `type` values observed in one invoice's items: `subscription`, +`prorate`, `one-time` (there may be more across the full 94-invoice history +— treat as free-form `TEXT`, not a `CHECK`-constrained enum, matching +`pax8_subscriptions.status`'s existing convention). +**How to avoid:** The new migration should add columns that map cleanly: +`price` → `unit_price` (already named this in the existing schema — keep), +`amountDue` → `line_total` (the actual billed amount — **not** `total`, +which appears to be the pre-proration/pre-discount full-period amount; +`subTotal` and `costTotal` coincided with `amountDue`/`cost*quantity` in +the one sample invoice inspected, but the exact semantic difference between +`total` and `subTotal`/`amountDue` should be spot-checked against a +prorated line item before finalizing the column mapping — flagged as an +Open Question below), plus new columns: `pax8_company_id` (Pitfall 1), +`subscription_id` (soft ref to `pax8_subscriptions.id` — enables joining a +cost history to a specific seat/license over time, directly serving +PAX8-06's "cost reconciliation over time" goal), `type`, `sku`, +`description`, `start_period`, `end_period` (critical — this is what makes +"cost over time" queryable; `synced_at` alone doesn't tell you which +billing period a line covers), and dual-cost columns mirroring Phase 11's +existing `price`/`partner_cost` precedent on `pax8_subscriptions`: add +`partner_cost` (from `cost`) and `partner_cost_total` (from `costTotal`) +alongside `unit_price`/`line_total`. +**Warning signs:** Cost totals that don't reconcile against PAX8's own +portal; inability to answer "what did company X pay for product Y in +March" without a `raw_payload` JSONB dig. + +### Pitfall 3: `/orders` endpoint is unreliable — confirms the migration's choice to use `/invoices` + +**What goes wrong:** A developer tempted to "just use the orders endpoint +since the table is called `pax8_orders`" will hit an unreliable endpoint. +**Why it happens:** [VERIFIED: live `GET /v1/orders` call, 2026-07-11] +returned an HTTP 504 Gateway Timeout in this environment (vs. `/invoices`, +which responded in well under a second). This independently confirms +`migrations/091`'s inline comment that Phase 12 should source from +`/invoices`, not the bare `/orders` object — not just because `/orders` +"lacks pricing/status" as the migration comment states, but because it may +also simply be unreliable/slow for this account. +**How to avoid:** Do not add an `/orders` call to `pax8-client.ts`. Source +exclusively from `/invoices` and `/invoices/{id}/items` as designed. +**Warning signs:** Sync timeouts or 504s if a future maintainer tries to +wire up `/orders` directly. + +### Pitfall 4: Trigram `similarity()` is already case-insensitive — don't double-normalize incorrectly + +**What goes wrong:** Assuming `similarity('ABC Inc', 'abc inc')` requires +manual `LOWER()` wrapping to score high, then writing normalization code +that's redundant or, worse, that strips information `pg_trgm` didn't need +stripped. +**Why it happens:** [VERIFIED: live query — `SELECT similarity('ABC Inc', +'abc inc')` returned `1`] `pg_trgm`'s `similarity()` operates +case-insensitively by default in this Postgres 16 instance (default +collation). Live re-test with explicit `LOWER(TRIM(...))` on both sides +produced **identical** bucket counts to the un-normalized query — no +difference. +**How to avoid:** Still apply `TRIM()` (leading/trailing whitespace) for +defensiveness and cleaner `pax8_company_match_review` display strings, but +don't build elaborate case-folding logic expecting it to change match +outcomes — it won't, `pg_trgm` already handles it. +**Warning signs:** None currently — this is a "don't over-build" pitfall, +not a correctness bug. + +### Pitfall 5: 89 vs 78 discrepancy — always filter Autotask candidates by `is_active` + +**What goes wrong:** Matching against the full `companies` table +(including inactive/decommissioned Autotask companies) inflates apparent +match counts and can auto-link a PAX8 company to a defunct Autotask +company. +**Why it happens:** [VERIFIED: live query] An exact-name join with no +`is_active` filter found 89 matches; adding `WHERE c.is_active = true` +dropped this to 78 — 11 PAX8 companies exactly match an **inactive** +Autotask company name only. +**How to avoid:** Every candidate query in the matcher must filter +`companies.is_active = true`, exactly as `device-link-reconciler.ts`'s +`findBySerial`/`findByHostnameInCompany` filter `is_deleted = false` on +`configuration_items`. +**Warning signs:** Matches to companies an admin can't find in the active +company list. + +## Code Examples + +### New migration skeleton + +```sql +-- migrations/092_pax8_orders_company_matching.sql +-- Phase 12: enables pg_trgm, fixes the company_id gap on pax8_order_items +-- (invoice headers have no per-company data — see 12-RESEARCH.md Pitfall 1), +-- and adds confident-auto-match columns to pax8_companies mirroring +-- device_external_ids' configuration_item_id/link_confidence/linked_at +-- precedent (migration 079). + +CREATE EXTENSION IF NOT EXISTS pg_trgm; + +ALTER TABLE pax8_order_items + ADD COLUMN IF NOT EXISTS pax8_company_id UUID, -- soft ref -> pax8_companies(id); see Pitfall 1 + ADD COLUMN IF NOT EXISTS subscription_id UUID, -- soft ref -> pax8_subscriptions(id) + ADD COLUMN IF NOT EXISTS item_type TEXT, -- 'subscription' | 'prorate' | 'one-time' | ... (free-form, see Pitfall 2) + ADD COLUMN IF NOT EXISTS sku TEXT, + ADD COLUMN IF NOT EXISTS description TEXT, + ADD COLUMN IF NOT EXISTS start_period TIMESTAMPTZ, + ADD COLUMN IF NOT EXISTS end_period TIMESTAMPTZ, + ADD COLUMN IF NOT EXISTS partner_cost NUMERIC(12,2), + ADD COLUMN IF NOT EXISTS partner_cost_total NUMERIC(12,2); + +CREATE INDEX IF NOT EXISTS idx_pax8_order_items_company ON pax8_order_items(pax8_company_id); +CREATE INDEX IF NOT EXISTS idx_pax8_order_items_period ON pax8_order_items(start_period, end_period); + +ALTER TABLE pax8_companies + ADD COLUMN IF NOT EXISTS autotask_company_id BIGINT, -- soft ref -> companies(id), no hard FK + ADD COLUMN IF NOT EXISTS match_confidence NUMERIC(4,3), -- raw pg_trgm score, 0.000-1.000 + ADD COLUMN IF NOT EXISTS match_method TEXT, -- 'pg_trgm' | 'manual' + ADD COLUMN IF NOT EXISTS matched_at TIMESTAMPTZ; + +CREATE INDEX IF NOT EXISTS idx_pax8_companies_autotask ON pax8_companies(autotask_company_id); +``` + +### Live-verified similarity distribution (evidence for the 0.90 threshold) + +``` +-- Run against this project's real pax8_companies (118 rows) and +-- companies (242 active rows), 2026-07-11: +exact/near-identical matches (score = 1.00): 80 pax8 companies +highest-scoring non-match in the entire dataset: 0.70 + ("Thoroughbred Construction Company" vs "Thoroughbred Construction Group") +next tier down: 0.65, 0.6666, 0.6296, 0.55 (all plausible non-matches) +gap between 0.70 (highest non-match) and 1.00 (lowest real match): wide, empty +``` +This is why 0.90 is recommended: it sits comfortably inside the empty gap +between the highest observed false-positive risk (0.70) and the cluster of +genuine matches (1.00), while still being below 1.00 to tolerate minor +punctuation/typo variance that didn't happen to occur in this particular +83-row sample but could occur in future data. + +## State of the Art + +Not applicable in the traditional sense (no deprecated approach being +replaced) — this is the first implementation of company matching in this +codebase. The one relevant "current vs legacy" note: `device_link_review` +(migration 080) is itself fairly recent and is the correct/current pattern +to mirror, not an older approach being superseded. + +## Assumptions Log + +| # | Claim | Section | Risk if Wrong | +|---|-------|---------|---------------| +| A1 | `total` vs `subTotal`/`amountDue` semantic distinction on invoice items (which represents the "actual billed line amount" for `line_total`) is based on inspecting one invoice's items where the values happened to coincide with `quantity × cost`; the exact formula/relationship between `total`, `subTotal`, `amountDue`, `cost`, and `costTotal` across prorated or credited line items was not exhaustively verified. | Common Pitfalls > Pitfall 2, Code Examples | If the mapping is wrong, `line_total`/`partner_cost_total` columns could store the wrong dollar amount for prorated months, undermining the "cost reconciliation over time" value of PAX8-06. Low-risk-to-verify: spot-check a `type: 'prorate'` item's fields against a second real invoice before finalizing the sync's column-mapping code. | +| A2 | `forCompanyId` (seen as `null` on the one sampled item) may represent a sub-reseller or delegated-partner scenario not applicable to this single-tenant PAX8 account; its exact meaning wasn't looked up in PAX8's docs (WebFetch on the live docs pages did not surface field-level schema detail). | Common Pitfalls > Pitfall 2 (raw item JSON) | Low risk — if always `null` for this account (single-tenant reseller, confirmed by `partnerName` being constant across all 94 invoices), it can be safely ignored/stored in `raw_payload` only, not a structured column. | +| A3 | The full 94-invoice / ~56k-item history size is stable going forward (i.e., growth is ~1 invoice + ~500-700 items per month) — extrapolated from observing 3 consecutive months' item counts (585, 611, 678), not the full 94-invoice history. | Summary, Pitfall 2, Environment Availability | Low risk — even at 2x the observed rate, total row count stays in the tens of thousands, well within normal Postgres table sizes; no partitioning or archival strategy needed for the foreseeable future. | + +## Open Questions + +1. **Exact `total` vs `subTotal`/`amountDue` semantics on invoice items for prorated/credited lines** + - What we know: For one `type: 'subscription'` line, `subTotal` (199.58) + equaled `amountDue` (199.58) and approximately equaled `quantity × cost` + (9 × 22.176 ≈ 199.58), while `total` (237.60) equaled `quantity × price` + (9 × 26.40 = 237.60) — suggesting `total` is the undiscounted retail + amount and `subTotal`/`amountDue` reflect the actual amount due + (possibly after a billing adjustment or the partner-cost pass-through). + - What's unclear: Whether this relationship holds for `type: 'prorate'` + or `type: 'one-time'` items, or for credited/negative-amount lines. + - Recommendation: Before finalizing the migration's column mapping, + fetch and inspect at least one `prorate`-type item's full field set + (already know they exist — seen in the 200-item sample) and confirm + `line_total` should map to `amountDue` (not `total` or `subTotal`) + across all three observed types. This is a 5-minute live-API check, + not a design risk — can be done as the first task of implementation. + +2. **Full enumeration of `type` values across all 94 invoices** + - What we know: `subscription`, `prorate`, `one-time` observed in one + invoice's 200-item sample. + - What's unclear: Whether other values exist elsewhere in the 7-year + history (e.g., `credit`, `adjustment`, `refund`). + - Recommendation: Store as free-form `TEXT`, no `CHECK` constraint (this + is already the plan) — the column is forward-compatible regardless of + what other values surface during the actual full sync. + +## Environment Availability + +| Dependency | Required By | Available | Version | Fallback | +|------------|------------|-----------|---------|----------| +| `pg_trgm` Postgres extension | Fuzzy company-name matching | ✓ (verified live) | Bundled with Postgres 16 (this project's pinned version) | — | +| PAX8 API — `/invoices` | Order/invoice header sync | ✓ (verified live, HTTP 200) | v1 | — | +| PAX8 API — `/invoices/{id}/items` | Line-item sync | ✓ (verified live, HTTP 200) | v1 | — | +| PAX8 API — `/orders` | (not used) | ✗ (HTTP 504 in this environment) | v1 | Use `/invoices` instead — already the plan; no fallback needed since this endpoint isn't part of the design. | +| PAX8_CLIENT_ID / PAX8_CLIENT_SECRET | All PAX8 API calls | ✓ (present in `.env.local`, used for live verification) | — | — | +| Live dev Postgres data (118 `pax8_companies`, 242 active `companies`) | Matching-threshold validation | ✓ (from Phase 11's completed live sync) | — | — | + +**Missing dependencies with no fallback:** none. + +**Missing dependencies with fallback:** `/orders` endpoint unreliable — +not needed; design already sources from `/invoices`. + +## Validation Architecture + +### Test Framework + +| Property | Value | +|----------|-------| +| Framework | vitest 4.1.5 | +| Config file | `vitest.config.ts` (root) — `include: ['lib/**/*.test.ts']`, `environment: 'node'` | +| Quick run command | `npx vitest run lib/services/pax8-client.test.ts lib/services/pax8-company-matcher.test.ts` | +| Full suite command | `npm test` (`vitest run`) | + +### Phase Requirements → Test Map + +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| PAX8-06 | `listAllInvoices()` concatenates pages in order | unit | `npx vitest run lib/services/pax8-client.test.ts -t "listAllInvoices"` | ❌ Wave 0 — extend existing `pax8-client.test.ts` following its `makeMultiPageFetchMock` pattern | +| PAX8-06 | `listAllInvoiceItems(invoiceId)` calls the correct per-invoice nested path and concatenates pages | unit | `npx vitest run lib/services/pax8-client.test.ts -t "listAllInvoiceItems"` | ❌ Wave 0 | +| PAX8-06 | Sync upserts orders/items and tombstones missing rows (mirrors existing `syncCompanies`/`syncSubscriptions` test shape) | unit | `npx vitest run lib/services/pax8-sync-service.test.ts` | ❌ Wave 0 — no `pax8-sync-service.test.ts` exists yet at all; check whether Phase 11 added one before assuming greenfield | +| PAX8-10 | Auto-link fires only when score ≥ 0.90 and no near-tie | unit | `npx vitest run lib/services/pax8-company-matcher.test.ts -t "auto-link"` | ❌ Wave 0 | +| PAX8-10 | Below-threshold or tied candidates produce a review row, not an auto-link | unit | `npx vitest run lib/services/pax8-company-matcher.test.ts -t "review"` | ❌ Wave 0 | +| PAX8-11 | Zero-candidate case still creates a review row with empty `candidate_company_ids` | unit | `npx vitest run lib/services/pax8-company-matcher.test.ts -t "empty candidates"` | ❌ Wave 0 | +| SC#4 | Re-running matcher never overwrites a `resolved_at IS NOT NULL` row | unit | `npx vitest run lib/services/pax8-company-matcher.test.ts -t "idempotent"` | ❌ Wave 0 | + +### Sampling Rate + +- **Per task commit:** `npx vitest run lib/services/pax8-*.test.ts` +- **Per wave merge:** `npm test` (full suite) +- **Phase gate:** Full suite green before `/gsd:verify-work` + +### Wave 0 Gaps + +- [ ] `lib/services/pax8-company-matcher.test.ts` — covers PAX8-10, PAX8-11, + SC#4. Since `pg_trgm` `similarity()` runs in Postgres, this suite + needs either (a) a mocked `postgresClient.query` following the + existing `pax8-client.test.ts` mocking convention (`vi.fn` returning + canned rows), keeping tests DB-free, or (b) an integration test + against a real Postgres test instance if one is available in CI — + check whether any existing `lib/services/*.test.ts` file already + does the latter (none inspected so far do; the codebase's tests are + all pure-mock, `vitest 4.1.5`, `environment: 'node'`). Recommend + mocking `postgresClient.query` to keep this test fast and consistent + with the rest of the suite. +- [ ] Extend `lib/services/pax8-client.test.ts` — add `listAllInvoices`/ + `listAllInvoiceItems` cases following the existing + `makeMultiPageFetchMock` helper already in that file. +- [ ] Confirm whether `lib/services/pax8-sync-service.test.ts` exists + (not found in this research pass) — if it genuinely doesn't exist, + Phase 11 shipped without direct unit tests for the sync service + itself (its Plan 03 was a live-data verification script instead, + per `11-03-SUMMARY.md`). The planner should decide whether Phase 12 + introduces the first unit tests for `Pax8SyncService` or continues + the live-verification-script pattern Phase 11 established. + +## Security Domain + +### Applicable ASVS Categories + +| ASVS Category | Applies | Standard Control | +|---------------|---------|-----------------| +| V2 Authentication | no | No new auth surface — existing session-gated `/api/pax8/sync` route from Phase 11 covers this. | +| V3 Session Management | no | Unchanged. | +| V4 Access Control | yes | `resolved_by_user_id` on `pax8_company_match_review` already references `"user"(id)` — Phase 14's manual-resolution flow (out of scope here) must call `requireAdmin()`/`requirePermission()` per CLAUDE.md convention. This phase's sync-time matching logic itself runs as a background/system process, not user-triggered per-row, so no per-request authz check is needed inside the matcher. | +| V5 Input Validation | yes | PAX8 API responses are external input — continue the existing pattern (`Pax8Company`/`Pax8Subscription`/`Pax8Product` types with a `[key: string]: unknown` escape hatch, no runtime Zod validation per CLAUDE.md's "no Zod in API routes unless required"). Parameterized queries only (`postgresClient.query(sql, params)`) — never string-interpolate PAX8 field values into SQL, especially company names feeding the `similarity()` comparison. | +| V6 Cryptography | no | No new secrets/crypto surface this phase. | + +### Known Threat Patterns for this stack + +| Pattern | STRIDE | Standard Mitigation | +|---------|--------|---------------------| +| SQL injection via company name interpolation | Tampering | Always pass PAX8 company names as bound parameters (`$1`) to `similarity($1, company_name)`, never string-concatenated — matches existing `postgresClient.query(sql, params)` convention used throughout the codebase. | +| Silent wrong-company cost attribution (a matching bug misattributes company A's costs to company B) | Tampering / Repudiation | D-01/D-02's conservative-threshold-plus-tie-detection design is itself the mitigation — this is a data-integrity concern specific to this phase's domain (billing reconciliation), not a generic security control. Cover with the "idempotent re-scoring never overwrites resolved_at" test (SC#4) and the near-tie unit test. | +| PAX8 credential exposure in logs/errors | Information Disclosure | Already covered by the existing `pax8-client.test.ts` assertion (`rejects.not.toThrow(new RegExp(SECRET))`) — no new surface introduced by this phase; extend the same discipline to any new error paths in `listAllInvoices`/`listAllInvoiceItems`. | + +## Sources + +### Primary (HIGH confidence) +- **Live authenticated API calls** to `https://api.pax8.com/v1/invoices`, + `/v1/invoices/{id}/items`, and `/v1/orders`, executed 2026-07-11 using + this project's real `PAX8_CLIENT_ID`/`PAX8_CLIENT_SECRET` from + `.env.local` — the single highest-confidence source available, superior + to documentation since it reflects this exact account's actual data + shape. +- **Live SQL queries** against the `pulse-postgres` Docker container's real + `pax8_companies` (118 rows) and `companies` (242 active rows) tables, + 2026-07-11, including a live `CREATE EXTENSION IF NOT EXISTS pg_trgm;` + and multiple `similarity()` distribution queries. +- `lib/services/device-link-reconciler.ts` (full file read) — direct + structural precedent for match/review/confidence logic. +- `lib/services/pax8-client.ts`, `lib/services/pax8-sync-service.ts`, + `lib/types/pax8.ts` (full files read) — existing extension points. +- `migrations/091_pax8_tables.sql`, `migrations/080_device_xref_company_id.sql`, + `migrations/001_initial_schema.sql`, `migrations/069/070_*.sql` (full + files read) — current schema and extension-enablement precedent. +- `.planning/phases/11-company-catalog-subscription-sync/11-PATTERNS.md`, + `11-03-SUMMARY.md` (full files read) — Phase 11's established + `Pax8SyncService` shape and result types. +- `lib/services/pax8-client.test.ts`, `vitest.config.ts` (full files read) + — existing test conventions. + +### Secondary (MEDIUM confidence) +- [devx.pax8.com — List Invoices](https://devx.pax8.com/reference/findpartnerinvoices) + and [List Invoice Items](https://devx.pax8.com/reference/findpartnerinvoiceitems) + reference pages — confirmed pagination defaults (page 0, size 10 default/ + 200 max) and available filter params (`companyId`, `status`, + `invoiceDateRangeStart/End`, etc.); did not surface full field-level + schema (page content is JS-rendered), which is why the live API calls + above were used as the ground-truth source instead. + +### Tertiary (LOW confidence) +- [lwhitelock/Pax8API GitHub repo](https://github.com/lwhitelock/Pax8API) — + community PowerShell module, consulted only to corroborate that + `Get-Pax8Invoices`/`Get-Pax8InvoiceItems` exist as community-recognized + operations; no field-level detail relied upon from this source. + +## Metadata + +**Confidence breakdown:** +- Standard stack (`pg_trgm`, threshold=0.90): HIGH — verified live against + this project's actual Postgres instance and actual production-like data, + not assumed. +- Architecture (invoice/item schema gap, nested pagination pattern): HIGH — + verified live against the actual PAX8 API for this account. +- Pitfalls: HIGH for Pitfalls 1, 3, 4, 5 (all directly reproduced live); + MEDIUM for Pitfall 2's exact `total`/`subTotal`/`amountDue` semantic + mapping (see Assumption A1 / Open Question 1 — one sample invoice + inspected, not exhaustively cross-checked against a prorated line). + +**Research date:** 2026-07-11 +**Valid until:** 30 days (stable domain — PAX8's public API and this +project's own schema/data don't change quickly; re-verify the `total` vs +`amountDue` mapping (Open Question 1) before implementation regardless of +elapsed time, since it wasn't fully closed out here).