diff --git a/.planning/phases/11-company-catalog-subscription-sync/11-CONTEXT.md b/.planning/phases/11-company-catalog-subscription-sync/11-CONTEXT.md new file mode 100644 index 0000000..5aaa9e5 --- /dev/null +++ b/.planning/phases/11-company-catalog-subscription-sync/11-CONTEXT.md @@ -0,0 +1,190 @@ +# Phase 11: Company, Catalog & Subscription Sync - Context + +**Gathered:** 2026-07-10 +**Status:** Ready for planning + + +## Phase Boundary + +PAX8 companies, the product/SKU catalog, and current subscriptions are synced +into Postgres, human-readable (joined to catalog, not bare SKU IDs) — the +"current state" half of the integration. No orders/invoices (Phase 12), no +company-matching to Autotask (Phase 12), no scheduler wiring (Phase 13), no +`/pax8` UI (Phase 14). This phase is sync-service logic only, writing into the +tables `10-01`/`10-02` already created (`pax8_companies`, `pax8_products`, +`pax8_subscriptions`). + + + + +## Implementation Decisions + +### Product Catalog Scope +- **D-01:** Lazy/referenced-only catalog sync — only fetch and store + `pax8_products` entries for SKUs that actually appear in at least one + synced subscription. Do not sync PAX8's full catalog independent of what's + referenced. +- **D-02:** When a subscription references a product ID not yet in the local + `pax8_products` table, fetch it inline during the same sync run (call + PAX8's product-detail endpoint for that SKU, upsert, then continue) — not a + separate two-pass batch step. + +### Subscription Cost Fields +- **D-03:** Store both PAX8's list/retail price and the actual partner/ + reseller cost as separate columns on `pax8_subscriptions`, if PAX8's API + exposes both per-subscription. This directly serves the milestone's stated + margin/profitability reconciliation goal (see SEED-002). If research + reveals PAX8 only exposes one of the two at the subscription level, note + the gap rather than fabricating the missing figure. +- **D-04:** Cost columns store the raw per-billing-period amount exactly as + PAX8 returns it (paired with the existing `billing_term` field) — no + monthly-equivalent normalization at sync time. Any "normalize to monthly" + math is a read-time/API-layer concern for a later phase (Phase 14's + `/pax8` page or SEED-003's data assistant), not this sync service. + +### Stale/Removed Entity Handling +- **D-05:** Soft-delete convention — when a company, subscription, or + catalog entry that existed in a prior sync no longer comes back from PAX8, + mark it `is_deleted = true`, `deleted_at = now()` rather than removing the + row. Matches the audit-column convention already used elsewhere in Pulse + (Autotask tables, per `CLAUDE.md`'s "Audit columns convention"). +- **D-06:** Apply the soft-delete pattern consistently across all three + tables touched by this phase — `pax8_companies`, `pax8_subscriptions`, AND + `pax8_products` — not just companies/subscriptions. A cancelled + subscription still needs to display its (possibly now-discontinued) + product name, so catalog rows are never hard-deleted either. + +### Sync Strategy +- **D-07:** Incremental sync where PAX8's API supports a modified-since / + delta mechanism, following the `entity-sync.ts` pattern already used for + Autotask (read last-sync timestamp, request only changed records). If + research during planning finds PAX8 has no such filter for a given entity + type, fall back to full sync for that entity type specifically — this is + a per-entity-type decision, not all-or-nothing across companies/ + subscriptions/catalog. +- **D-08:** Reconcile incremental sync with reliable removal detection via a + periodic full reconciliation pass — incremental syncs handle routine + updates, but the sync service must also periodically (e.g., once per + sync run, or on a longer cadence — planner's call on frequency) fetch the + full list of company/subscription/product IDs currently in PAX8, diff + against what's `is_deleted = false` in Postgres, and soft-delete anything + no longer present. This directly satisfies the D-05/D-06 soft-delete + decisions, which an incremental-only pull cannot detect on its own. + +### Claude's Discretion +- **Reconciliation-pass cadence** — whether the full-ID reconciliation pass + (D-08) runs on every sync invocation or on a separate, less-frequent + schedule is left to the planner, informed by whatever sync-trigger + mechanism this phase builds (Phase 13 owns the actual cron wiring, but + this phase's sync service should expose whatever hooks that eventual + schedule needs). +- **Missing partner-cost field fallback** — if research shows PAX8's + subscription API doesn't expose reseller/partner cost distinctly from list + price, the planner should decide the fallback column shape (e.g., a single + `cost` column vs `list_price` + nullable `partner_cost`) rather than + blocking on this ambiguity. + + + + +## Canonical References + +**Downstream agents MUST read these before planning or implementing.** + +### Project scope & requirements +- `.planning/PROJECT.md` — Current Milestone: v2.0 PAX8 Integration section +- `.planning/REQUIREMENTS.md` — PAX8-03, PAX8-04, PAX8-05, PAX8-08 (this + phase's requirement IDs) +- `.planning/seeds/SEED-002-pax8-integration.md` — original exploration: + "cost reconciliation" and "margin/profitability reporting" as explicit + goals (informs D-03); "build the Postgres schema with [SEED-003] in mind — + clean, well-typed tables, avoid PAX8-API-shaped blobs" + +### Prior phase (10) foundation this phase builds on +- `.planning/phases/10-pax8-client-auth-foundation/10-CONTEXT.md` — records + that product catalog scope (this phase's D-01/D-02) and raw_payload + columns across all PAX8 tables were explicitly deferred here from Phase 10 +- `lib/services/pax8-client.ts` / `lib/services/pax8-factory.ts` — the + OAuth2 client this phase's sync service must call (`getPax8Client()`) +- `migrations/091_pax8_tables.sql` — existing schema for `pax8_companies`, + `pax8_products`, `pax8_subscriptions`, `pax8_orders`, `pax8_order_items`, + `pax8_company_match_review`; all four non-order tables already have + `raw_payload JSONB` per the Phase 10 discretion decision — this phase + populates them, does not alter schema unless a genuine gap is found +- `.planning/phases/10-pax8-client-auth-foundation/10-RESEARCH.md` — Phase + 10's PAX8 API research; check first before re-researching auth/base-URL + basics in Phase 11's own research pass + +### Existing patterns to follow +- `lib/services/entity-sync.ts` — canonical "last sync timestamp + + incremental pull + batch upsert" pattern (D-07 follows this shape) +- `lib/services/itglue-sync-service.ts`, `lib/services/veeam-sync-service.ts` + — sibling sync-service examples for structure/conventions +- `postgresClient.bulkUpsert()` — batch upsert method, per + `lib/services/postgres-client.ts` +- Audit columns convention (`created_at`, `updated_at`, `synced_at`, + `is_deleted`, `deleted_at`) — per `CLAUDE.md`; D-05/D-06 rely on this + already-established convention, columns already exist on the Phase 10 + schema +- `/api//sync` fire-and-forget POST pattern (e.g., `/api/veeam/sync`, + `/api/itglue/sync`) — likely shape for this phase's sync trigger endpoint, + though scheduler cron wiring itself is Phase 13's scope, not this phase's + + + + +## Existing Code Insights + +### Reusable Assets +- `lib/services/entity-sync.ts` — direct template for incremental sync + orchestration (D-07/D-08) +- `lib/services/msgraph-client.ts` pattern (already followed by + `pax8-client.ts`) — no new auth work needed, `getPax8Client()` is ready +- `postgresClient.bulkUpsert()` — batch write mechanism for all three + entity types + +### Established Patterns +- Sync services live at `lib/services/-sync-service.ts`; a + `pax8-sync-service.ts` (or similarly named) is the expected new file per + this convention +- Fire-and-forget sync trigger: `POST /api//sync` returns immediately, + sync runs async — matches Veeam/IT Glue/SentinelOne/Zoom/QBO precedent +- Soft-delete via `is_deleted`/`deleted_at` columns already exists on the + Phase 10 migration for all relevant tables — no new migration needed for + D-05/D-06 unless research finds a gap + +### Integration Points +- New file: `lib/services/pax8-sync-service.ts` (or equivalent name per + planner) +- New route: `/api/pax8/sync` (POST, fire-and-forget) — trigger only, no + scheduler registration yet (Phase 13) +- No route/nav/UI integration in this phase (`UI hint: no` per ROADMAP.md) + + + + +## Specific Ideas + +No specific UI or behavioral references were given — this phase is sync +service logic only. The eight numbered decisions above (D-01 through D-08) +are the concrete specifics: lazy catalog sync with inline SKU fetch, +dual cost columns (list + partner) stored per-billing-period as-is, +consistent soft-delete across all three tables, and incremental sync with a +periodic full-reconciliation pass for removal detection. + + + + +## Deferred Ideas + +None — discussion stayed within phase scope. (Orders/invoices, company +matching, scheduler cron wiring, and the `/pax8` UI are already sequenced +into Phases 12-14 per ROADMAP.md and REQUIREMENTS.md — not deferred from +this discussion, just out of this phase's boundary.) + + + +--- + +*Phase: 11-company-catalog-subscription-sync* +*Context gathered: 2026-07-10* diff --git a/.planning/phases/11-company-catalog-subscription-sync/11-DISCUSSION-LOG.md b/.planning/phases/11-company-catalog-subscription-sync/11-DISCUSSION-LOG.md new file mode 100644 index 0000000..789be5d --- /dev/null +++ b/.planning/phases/11-company-catalog-subscription-sync/11-DISCUSSION-LOG.md @@ -0,0 +1,99 @@ +# Phase 11: Company, Catalog & Subscription Sync - Discussion Log + +> **Audit trail only.** Do not use as input to planning, research, or execution agents. +> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered. + +**Date:** 2026-07-10 +**Phase:** 11-company-catalog-subscription-sync +**Areas discussed:** Product catalog scope, Subscription cost fields, Stale/removed entity handling, Sync strategy (full vs incremental) + +--- + +## Product Catalog Scope + +| Option | Description | Selected | +|--------|-------------|----------| +| Full catalog sync | Fetch and store PAX8's complete product catalog every sync, independent of subscriptions | | +| Lazy/referenced-only | Only fetch/store catalog entries for SKUs referenced by a synced subscription | ✓ | +| You decide | Let planner choose based on PAX8's actual catalog API shape | | + +**User's choice:** Lazy/referenced-only +**Notes:** Directly carries forward the gray area left open in Phase 10's CONTEXT.md discretion section. + +| Option | Description | Selected | +|--------|-------------|----------| +| Fetch it inline during the same sync run | Call PAX8's product-detail endpoint immediately when an unrecognized SKU is hit | ✓ | +| Two-pass sync | First pass collects referenced product IDs, second pass batch-fetches them | | + +**User's choice:** Fetch it inline during the same sync run + +--- + +## Subscription Cost Fields + +| Option | Description | Selected | +|--------|-------------|----------| +| Your partner/reseller cost only | Store only what PAX8 bills the reseller | | +| List price + partner cost, both | Store both as separate columns, if PAX8 exposes both | ✓ | +| Whatever PAX8's subscription endpoint returns, typed generically | Don't presuppose the pricing model | | + +**User's choice:** List price + partner cost, both +**Notes:** Ties directly to SEED-002's stated margin/profitability reconciliation goal. + +| Option | Description | Selected | +|--------|-------------|----------| +| Store as-is, normalize at read time | Keep raw per-billing-period amount; monthly-equivalent math happens later | ✓ | +| Normalize to monthly at sync time | Compute and store a monthly-equivalent column at sync time | | + +**User's choice:** Store as-is, normalize at read time + +--- + +## Stale/Removed Entity Handling + +| Option | Description | Selected | +|--------|-------------|----------| +| Soft-delete (is_deleted flag) | Mark is_deleted=true, deleted_at=now() instead of removing | ✓ | +| Leave rows as-is (no tracking) | Don't track removal this phase | | + +**User's choice:** Soft-delete (is_deleted flag) +**Notes:** Matches existing Pulse audit-column convention. + +| Option | Description | Selected | +|--------|-------------|----------| +| Companies + subscriptions only | Catalog stays as historical reference, never soft-deleted | | +| All three tables consistently | Apply is_deleted/deleted_at to pax8_products too | ✓ | + +**User's choice:** All three tables consistently + +--- + +## Sync Strategy (Full vs Incremental) + +| Option | Description | Selected | +|--------|-------------|----------| +| Full resync every run | Fetch and upsert everything each sync, no last-sync tracking | | +| Incremental where PAX8 supports it | Follow entity-sync.ts pattern; modified-since filter, fall back to full if unsupported | ✓ | +| You decide | Let planner/researcher decide after seeing PAX8's actual API | | + +**User's choice:** Incremental where PAX8 supports it + +**Follow-up tension surfaced:** Incremental sync alone can't detect entities PAX8 stops returning (no signal that something is now missing), which conflicts with the soft-delete decision above. + +| Option | Description | Selected | +|--------|-------------|----------| +| Periodic full reconciliation pass | Incremental for routine updates + periodic full ID-listing diff to catch removals | ✓ | +| Full sync only for existence, incremental for details | Always fetch full ID list, but incremental-fetch only changed record details | | + +**User's choice:** Periodic full reconciliation pass + +--- + +## Claude's Discretion + +- **Reconciliation-pass cadence** — whether the full-ID reconciliation pass runs every sync invocation or on a separate schedule, informed by whatever trigger mechanism this phase builds (Phase 13 owns actual cron wiring). +- **Missing partner-cost field fallback** — if PAX8's API doesn't expose partner cost distinctly from list price, planner decides the fallback column shape. + +## Deferred Ideas + +None — discussion stayed within phase scope.