docs(quick-260521-foj): Fix weekly-full FK error - widen Companies filter + defensive ticket company_id validation
This commit is contained in:
parent
62c529fb91
commit
078e087d5f
3 changed files with 497 additions and 1 deletions
|
|
@ -28,7 +28,7 @@ See: .planning/PROJECT.md (updated 2026-05-03)
|
||||||
Phase: 09.1 (ntfy-backend-fix) — EXECUTING
|
Phase: 09.1 (ntfy-backend-fix) — EXECUTING
|
||||||
Plan: 1 of 1
|
Plan: 1 of 1
|
||||||
Status: Executing Phase 09.1
|
Status: Executing Phase 09.1
|
||||||
Last activity: 2026-05-21 - Completed quick task 260521-fci: Stopgap nightly reconciliation for stale open tickets in postgres mirror
|
Last activity: 2026-05-21 - Completed quick task 260521-foj: Fix weekly-full FK error: widen Companies filter + defensive ticket company_id validation
|
||||||
|
|
||||||
Progress: [░░░░░░░░░░] 0%
|
Progress: [░░░░░░░░░░] 0%
|
||||||
|
|
||||||
|
|
@ -95,6 +95,7 @@ None yet.
|
||||||
|---|-------------|------|--------|-----------|
|
|---|-------------|------|--------|-----------|
|
||||||
| 260519-0oz | Add QBO createPayment + createDeposit + .FH reconciliation script | 2026-05-19 | 5497458 | [260519-0oz-add-qbo-createpayment-createdeposit-fh-r](./quick/260519-0oz-add-qbo-createpayment-createdeposit-fh-r/) |
|
| 260519-0oz | Add QBO createPayment + createDeposit + .FH reconciliation script | 2026-05-19 | 5497458 | [260519-0oz-add-qbo-createpayment-createdeposit-fh-r](./quick/260519-0oz-add-qbo-createpayment-createdeposit-fh-r/) |
|
||||||
| 260521-fci | Stopgap nightly reconciliation for stale open tickets in postgres mirror | 2026-05-21 | badd718 | [260521-fci-stopgap-nightly-reconciliation-for-stale](./quick/260521-fci-stopgap-nightly-reconciliation-for-stale/) |
|
| 260521-fci | Stopgap nightly reconciliation for stale open tickets in postgres mirror | 2026-05-21 | badd718 | [260521-fci-stopgap-nightly-reconciliation-for-stale](./quick/260521-fci-stopgap-nightly-reconciliation-for-stale/) |
|
||||||
|
| 260521-foj | Fix weekly-full FK error: widen Companies filter + defensive ticket company_id validation | 2026-05-21 | 62c529f | [260521-foj-fix-weekly-full-fk-error-widen-companies](./quick/260521-foj-fix-weekly-full-fk-error-widen-companies/) |
|
||||||
|
|
||||||
## Session Continuity
|
## Session Continuity
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,373 @@
|
||||||
|
---
|
||||||
|
phase: quick-260521-foj
|
||||||
|
plan: 01
|
||||||
|
type: execute
|
||||||
|
wave: 1
|
||||||
|
depends_on: []
|
||||||
|
files_modified:
|
||||||
|
- lib/utils/sync-helpers.ts
|
||||||
|
- lib/services/entity-sync.ts
|
||||||
|
autonomous: true
|
||||||
|
requirements:
|
||||||
|
- QUICK-260521-FOJ
|
||||||
|
must_haves:
|
||||||
|
truths:
|
||||||
|
- "Full sync of Companies entity fetches all companies (active + inactive) — no isActive=true filter applied"
|
||||||
|
- "Tickets sync with a company_id pointing at a company missing from the Pulse mirror does not raise tickets_company_id_fkey; the company_id is nullified and the ticket is preserved"
|
||||||
|
- "`hasAppliedFilters` remains true on Companies full sync, so soft-delete of companies is NOT triggered"
|
||||||
|
- "Existing behavior unchanged for all other entities — getActiveField/buildActiveFilter still works as before for resources, contacts, statuses, etc."
|
||||||
|
- "`npx tsc --noEmit --pretty` passes after both edits"
|
||||||
|
artifacts:
|
||||||
|
- path: "lib/utils/sync-helpers.ts"
|
||||||
|
provides: "New exported `buildCompaniesFilter()` returning `[{ field: 'id', op: 'gt', value: 0 }]`"
|
||||||
|
contains: "export function buildCompaniesFilter"
|
||||||
|
- path: "lib/services/entity-sync.ts"
|
||||||
|
provides: "New `getValidCompanyIds()` private method + COMPANIES branch in full-sync filter chain + ticket company_id nullification block"
|
||||||
|
contains: "buildCompaniesFilter"
|
||||||
|
key_links:
|
||||||
|
- from: "lib/services/entity-sync.ts (full-sync filter chain ~line 133-165)"
|
||||||
|
to: "buildCompaniesFilter (lib/utils/sync-helpers.ts)"
|
||||||
|
via: "import + branch above the generic `else` that calls buildActiveFilter"
|
||||||
|
pattern: "entity === EntityType.COMPANIES.*buildCompaniesFilter"
|
||||||
|
- from: "lib/services/entity-sync.ts (ticket validation block)"
|
||||||
|
to: "companies table (is_deleted = false)"
|
||||||
|
via: "getValidCompanyIds() called inside the `if (entity === EntityType.TICKETS)` block"
|
||||||
|
pattern: "getValidCompanyIds"
|
||||||
|
---
|
||||||
|
|
||||||
|
<objective>
|
||||||
|
Fix the weekly-full and full ticket sync FK error (`tickets_company_id_fkey`) caused by Companies sync filtering out inactive companies that nonetheless have tickets in the same sync window.
|
||||||
|
|
||||||
|
Purpose: Every full and weekly-full sync since 2026-05-15 fails because Companies full sync applies `isActive=true` (via `buildActiveFilter`), which misses inactive companies that have tickets. The ticket transaction rolls back on the FK violation. Live evidence in `sync_history`: every recent full sync shows `[DATABASE_CONSTRAINT_ERROR] ... tickets_company_id_fkey`.
|
||||||
|
|
||||||
|
Output:
|
||||||
|
- A new `buildCompaniesFilter()` in `lib/utils/sync-helpers.ts` mirroring the `buildProjectPhasesFilter` template (`id > 0` to fetch all).
|
||||||
|
- A new branch in `entity-sync.ts` full-sync filter chain that routes COMPANIES through `buildCompaniesFilter()` instead of the generic `buildActiveFilter`.
|
||||||
|
- A defensive ticket `company_id` nullification block in `entity-sync.ts` (belt-and-suspenders for the rare case of truly hard-deleted Autotask companies).
|
||||||
|
</objective>
|
||||||
|
|
||||||
|
<execution_context>
|
||||||
|
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||||
|
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||||
|
</execution_context>
|
||||||
|
|
||||||
|
<context>
|
||||||
|
@./CLAUDE.md
|
||||||
|
@lib/utils/sync-helpers.ts
|
||||||
|
@lib/services/entity-sync.ts
|
||||||
|
@lib/services/postgres-client.ts
|
||||||
|
|
||||||
|
<interfaces>
|
||||||
|
<!-- Key contracts the executor will use. Already in the codebase — no exploration needed. -->
|
||||||
|
|
||||||
|
From `lib/utils/sync-helpers.ts` — existing filter-builder templates this task must mirror:
|
||||||
|
```ts
|
||||||
|
/**
|
||||||
|
* Build filter for contract services (requires contractID filter — fetch all via contractIDs)
|
||||||
|
* @returns Query filter array for active contract services
|
||||||
|
*/
|
||||||
|
export function buildContractServicesFilter(): Array<{ field: string; op: string; value: any }> {
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
field: 'contractID',
|
||||||
|
op: 'gt',
|
||||||
|
value: 0,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build filter for project phases (Phases endpoint requires a filter)
|
||||||
|
* @returns Query filter array for all phases
|
||||||
|
*/
|
||||||
|
export function buildProjectPhasesFilter(): Array<{ field: string; op: string; value: any }> {
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
field: 'id',
|
||||||
|
op: 'gt',
|
||||||
|
value: 0,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get active status field name for entity
|
||||||
|
* Returns 'isActive' for COMPANIES — DO NOT change this mapping. It is still
|
||||||
|
* referenced by `buildActiveFilter` for other entities (resources, contacts, etc.).
|
||||||
|
*/
|
||||||
|
export function getActiveField(entity: EntityType): string | null { /* ... */ }
|
||||||
|
```
|
||||||
|
|
||||||
|
From `lib/services/entity-sync.ts` — the full-sync filter chain (~lines 130-170) is where the new COMPANIES branch goes:
|
||||||
|
```ts
|
||||||
|
const filters: Array<{ field: string; op: string; value: any }> = [];
|
||||||
|
|
||||||
|
// Special handling for entities that require filters
|
||||||
|
if (entity === EntityType.CONTRACTS) {
|
||||||
|
filters.push(...buildContractsFilter());
|
||||||
|
entityLogger.info('Full sync with status filter for active contracts');
|
||||||
|
} else if (entity === EntityType.CONTRACT_SERVICES) {
|
||||||
|
filters.push(...buildContractServicesFilter());
|
||||||
|
entityLogger.info('Full sync of all contract services');
|
||||||
|
} else if (entity === EntityType.PROJECTS) {
|
||||||
|
filters.push(...buildProjectsFilter());
|
||||||
|
entityLogger.info('Full sync with status filter for non-completed projects');
|
||||||
|
} else if (entity === EntityType.PROJECT_PHASES) {
|
||||||
|
filters.push(...buildProjectPhasesFilter());
|
||||||
|
entityLogger.info('Full sync of all project phases');
|
||||||
|
} else if (entity === EntityType.TIME_ENTRIES) { /* ... */
|
||||||
|
} else if (entity === EntityType.BILLING_ITEMS) { /* ... */
|
||||||
|
} else {
|
||||||
|
// Add active filter if applicable
|
||||||
|
const activeFilter = buildActiveFilter(entity);
|
||||||
|
if (activeFilter) {
|
||||||
|
filters.push(...activeFilter);
|
||||||
|
entityLogger.info('Full sync with active filter');
|
||||||
|
}
|
||||||
|
// ... date range filter ...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
From `lib/services/entity-sync.ts` — the existing `getValidResourceIds` template the new method must mirror (lines 710-718):
|
||||||
|
```ts
|
||||||
|
/**
|
||||||
|
* Get all valid resource IDs from the database
|
||||||
|
* Used to validate foreign key references before insert
|
||||||
|
* @returns Set of valid resource IDs
|
||||||
|
*/
|
||||||
|
private async getValidResourceIds(): Promise<Set<number>> {
|
||||||
|
const query = 'SELECT id FROM resources WHERE is_deleted = false';
|
||||||
|
const result = await postgresClient.query<{ id: number }>(query);
|
||||||
|
return new Set(result.rows.map(row => Number(row.id)));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
From `lib/services/entity-sync.ts` — the existing tickets validation block where the new `company_id` validation slots in (lines 235-285):
|
||||||
|
- Lines 238-252: filters out tickets without `company_id` (`recordsWithoutCompany`). KEEP AS-IS.
|
||||||
|
- Lines 255-285: nullifies invalid `assigned_resource_id` / `first_response_*` resource FKs against `getValidResourceIds`. KEEP AS-IS.
|
||||||
|
- NEW VALIDATION goes immediately after the `recordsWithoutCompany` filter, before or alongside the resource validation block. Must use the same `mappedRecords = mappedRecords.map(...)` mutation pattern and the same `entityLogger.warn` shape.
|
||||||
|
|
||||||
|
From `lib/services/postgres-client.ts` — singleton DB interface:
|
||||||
|
```ts
|
||||||
|
postgresClient.query<{ id: number }>(sql, params?: any[]): Promise<QueryResult<{ id: number }>>
|
||||||
|
```
|
||||||
|
</interfaces>
|
||||||
|
</context>
|
||||||
|
|
||||||
|
<tasks>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 1: Add buildCompaniesFilter and wire COMPANIES into the full-sync filter chain</name>
|
||||||
|
<files>lib/utils/sync-helpers.ts, lib/services/entity-sync.ts</files>
|
||||||
|
<action>
|
||||||
|
**Step 1 — `lib/utils/sync-helpers.ts`:**
|
||||||
|
|
||||||
|
Add a new exported function `buildCompaniesFilter()` adjacent to `buildContractServicesFilter` / `buildProjectPhasesFilter` (around line 92, right after `buildContractServicesFilter`). Mirror the JSDoc style of `buildContractServicesFilter` exactly:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
/**
|
||||||
|
* Build filter for companies (Companies endpoint requires a filter — use id > 0 to fetch all
|
||||||
|
* companies regardless of isActive. Active-only filtering misses inactive companies that
|
||||||
|
* have tickets, causing tickets_company_id_fkey violations on full sync.)
|
||||||
|
* @returns Query filter array for all companies
|
||||||
|
*/
|
||||||
|
export function buildCompaniesFilter(): Array<{ field: string; op: string; value: any }> {
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
field: 'id',
|
||||||
|
op: 'gt',
|
||||||
|
value: 0,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
DO NOT touch `getActiveField(COMPANIES)` — leave it returning `'isActive'`. It's still referenced by `buildActiveFilter` for other entities and by call sites we are not changing.
|
||||||
|
|
||||||
|
**Step 2 — `lib/services/entity-sync.ts`:**
|
||||||
|
|
||||||
|
1. Update the sync-helpers import at the top of the file (currently lines 13-25). Add `buildCompaniesFilter` to the named imports — alphabetize alongside the other `build*` helpers. The import block becomes:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import {
|
||||||
|
getAutotaskEntityName,
|
||||||
|
buildIncrementalFilter,
|
||||||
|
buildActiveFilter,
|
||||||
|
buildDateRangeFilter,
|
||||||
|
buildCompaniesFilter,
|
||||||
|
buildContractsFilter,
|
||||||
|
buildContractServicesFilter,
|
||||||
|
buildProjectsFilter,
|
||||||
|
buildProjectPhasesFilter,
|
||||||
|
buildTimeEntriesFilter,
|
||||||
|
buildBillingItemsFilter,
|
||||||
|
getTableName
|
||||||
|
} from '../utils/sync-helpers';
|
||||||
|
```
|
||||||
|
|
||||||
|
2. In the full-sync filter chain (around lines 133-165), add a new branch for COMPANIES. Place it as the FIRST branch in the chain (just after the `const filters: Array<...> = [];` declaration and before the existing `if (entity === EntityType.CONTRACTS)` branch). The chain after edit:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
const filters: Array<{ field: string; op: string; value: any }> = [];
|
||||||
|
|
||||||
|
// Special handling for entities that require filters
|
||||||
|
if (entity === EntityType.COMPANIES) {
|
||||||
|
filters.push(...buildCompaniesFilter());
|
||||||
|
entityLogger.info('Full sync of all companies (active + inactive)');
|
||||||
|
} else if (entity === EntityType.CONTRACTS) {
|
||||||
|
filters.push(...buildContractsFilter());
|
||||||
|
entityLogger.info('Full sync with status filter for active contracts');
|
||||||
|
} else if (entity === EntityType.CONTRACT_SERVICES) {
|
||||||
|
// ... existing branches unchanged ...
|
||||||
|
```
|
||||||
|
|
||||||
|
Critical: the new COMPANIES branch MUST execute BEFORE the generic `else` block that calls `buildActiveFilter`. Result: the active filter is no longer applied to Companies on full sync, so inactive companies are fetched too.
|
||||||
|
|
||||||
|
**Why this is safe (state this in your verification reasoning, do not add it as a code comment):**
|
||||||
|
- `hasAppliedFilters` (entity-sync.ts:173) stays true because the new filter array is non-empty (`id > 0`). So `softDeleteMissingRecords` is NOT triggered for companies — no accidental mass soft-delete.
|
||||||
|
- The existing DB already has 145 `is_active=false` companies referenced by tickets; downstream UI/API code tolerates inactive companies.
|
||||||
|
- Companies don't support incremental sync (entity-sync.ts:106 `supportsIncremental` excludes COMPANIES), so this code path runs on every Companies sync — full, weekly-full, and the incremental→full fallback path.
|
||||||
|
|
||||||
|
DO NOT touch:
|
||||||
|
- `getActiveField`.
|
||||||
|
- The `lastActivityDate` filter on tickets (out of scope).
|
||||||
|
- Any other branch in the filter chain.
|
||||||
|
- Adjacent formatting / unrelated code.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd /opt/stacks/pulse && npx tsc --noEmit --pretty 2>&1 | tail -40 && echo "---" && grep -n "buildCompaniesFilter" lib/utils/sync-helpers.ts lib/services/entity-sync.ts && echo "---" && grep -n "entity === EntityType.COMPANIES" lib/services/entity-sync.ts</automated>
|
||||||
|
</verify>
|
||||||
|
<done>
|
||||||
|
- `buildCompaniesFilter` exported from `lib/utils/sync-helpers.ts` with the exact JSDoc and body shown above.
|
||||||
|
- `buildCompaniesFilter` imported in `lib/services/entity-sync.ts` named-imports block.
|
||||||
|
- Exactly one new COMPANIES branch in the full-sync filter chain, placed before all other `if/else if` branches, with the `entityLogger.info('Full sync of all companies (active + inactive)')` line.
|
||||||
|
- `getActiveField` and `buildActiveFilter` definitions UNCHANGED.
|
||||||
|
- `npx tsc --noEmit --pretty` passes with zero errors.
|
||||||
|
- `grep -n "buildCompaniesFilter" lib/services/entity-sync.ts` shows exactly two matches (one import, one usage).
|
||||||
|
</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 2: Defensive company_id nullification in tickets sync (belt-and-suspenders)</name>
|
||||||
|
<files>lib/services/entity-sync.ts</files>
|
||||||
|
<action>
|
||||||
|
This task adds a second layer of defense after Task 1: even if a ticket somehow references a company that doesn't exist in the Pulse mirror (e.g. hard-deleted from Autotask), nullify the `company_id` rather than letting the FK violation roll back the whole ticket transaction.
|
||||||
|
|
||||||
|
**Step 1 — Add new private method `getValidCompanyIds`:**
|
||||||
|
|
||||||
|
Find the existing `getValidResourceIds` method (currently lines ~710-718, just after `calculateMonthlyChunks` is defined further down — search for `private async getValidResourceIds`). Add the new method immediately AFTER it, before `getValidContactIds`. Match the exact pattern:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
/**
|
||||||
|
* Get all valid company IDs from the database
|
||||||
|
* Used to validate foreign key references before insert
|
||||||
|
* @returns Set of valid company IDs
|
||||||
|
*/
|
||||||
|
private async getValidCompanyIds(): Promise<Set<number>> {
|
||||||
|
const query = 'SELECT id FROM companies WHERE is_deleted = false';
|
||||||
|
const result = await postgresClient.query<{ id: number }>(query);
|
||||||
|
return new Set(result.rows.map(row => Number(row.id)));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2 — Add the company_id validation block in the tickets validation pipeline:**
|
||||||
|
|
||||||
|
Find the existing `if (entity === EntityType.TICKETS) {` block that nullifies invalid resource IDs (currently lines ~254-285, the block that uses `getValidResourceIds`). Add a NEW `if (entity === EntityType.TICKETS) { ... }` block IMMEDIATELY ABOVE it (between the `recordsWithoutCompany` filter block at lines ~238-252 and the resource validation block at lines ~254-285). The new block:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
// Validate company_id foreign keys for tickets (defensive — Task 1 widening
|
||||||
|
// the Companies filter should make this a near-zero count, but catches the
|
||||||
|
// truly hard-deleted Autotask company case).
|
||||||
|
if (entity === EntityType.TICKETS) {
|
||||||
|
const validCompanyIds = await this.getValidCompanyIds();
|
||||||
|
let nullifiedCompanyCount = 0;
|
||||||
|
const nullifiedCompanySamples: number[] = [];
|
||||||
|
mappedRecords = mappedRecords.map(ticket => {
|
||||||
|
if (ticket.company_id && !validCompanyIds.has(ticket.company_id)) {
|
||||||
|
if (nullifiedCompanySamples.length < 5) {
|
||||||
|
nullifiedCompanySamples.push(ticket.id);
|
||||||
|
}
|
||||||
|
ticket.company_id = null;
|
||||||
|
nullifiedCompanyCount++;
|
||||||
|
}
|
||||||
|
return ticket;
|
||||||
|
});
|
||||||
|
if (nullifiedCompanyCount > 0) {
|
||||||
|
entityLogger.warn('Nullified ticket company_id for companies missing from mirror', {
|
||||||
|
nullifiedCount: nullifiedCompanyCount,
|
||||||
|
sampleTicketIds: nullifiedCompanySamples,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Ordering rationale (must be respected):
|
||||||
|
1. `recordsWithoutCompany` filter (existing) — drops tickets without any `company_id`. Runs first so we only validate tickets that have a `company_id`.
|
||||||
|
2. NEW company_id nullification block — nullifies `company_id` that doesn't exist in mirror. Runs second.
|
||||||
|
3. Resource validation block (existing) — nullifies invalid resource refs. Runs third.
|
||||||
|
4. Everything else (`bulkUpsert` at ~line 372) runs after.
|
||||||
|
|
||||||
|
The `tickets.company_id` column is nullable (confirmed via the recent stopgap reconciliation work and existing handling that filters rows without `company_id`). Nullifying it is non-destructive — it preserves the ticket row and just severs the company link, matching the resource-FK handling pattern already used in the same function.
|
||||||
|
|
||||||
|
DO NOT touch:
|
||||||
|
- The existing `recordsWithoutCompany` filter block.
|
||||||
|
- The existing resource validation block.
|
||||||
|
- The chunked tickets sync (`syncTicketsChunked`) — out of scope. That path has its own (separate) flow; this fix is for `syncEntity`.
|
||||||
|
- `cachedValidResourceIds` or `cachedValidContactIds` fields.
|
||||||
|
- Adjacent code / formatting.
|
||||||
|
|
||||||
|
DO NOT manually trigger a sync from this task. Note in SUMMARY.md only:
|
||||||
|
```
|
||||||
|
Manual verify (user runs after merge):
|
||||||
|
curl -X POST http://localhost:3100/api/sync/full \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"triggeredBy":"manual-verify"}'
|
||||||
|
Then check `sync_history` for the next ticket sync row — it should complete without `tickets_company_id_fkey` in `error_message`.
|
||||||
|
```
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd /opt/stacks/pulse && npx tsc --noEmit --pretty 2>&1 | tail -40 && echo "---" && grep -cn "getValidCompanyIds" lib/services/entity-sync.ts && echo "---" && grep -n "Nullified ticket company_id for companies missing from mirror" lib/services/entity-sync.ts && echo "---" && grep -n "if (entity === EntityType.TICKETS)" lib/services/entity-sync.ts</automated>
|
||||||
|
</verify>
|
||||||
|
<done>
|
||||||
|
- `getValidCompanyIds` method exists exactly once in `lib/services/entity-sync.ts`, mirroring `getValidResourceIds`.
|
||||||
|
- A new `if (entity === EntityType.TICKETS)` block exists between the `recordsWithoutCompany` filter and the resource validation block.
|
||||||
|
- The new block calls `await this.getValidCompanyIds()` and logs via `entityLogger.warn('Nullified ticket company_id for companies missing from mirror', ...)` when count > 0.
|
||||||
|
- `grep -c "getValidCompanyIds" lib/services/entity-sync.ts` returns 2 (one definition, one call site).
|
||||||
|
- `grep -n "if (entity === EntityType.TICKETS)" lib/services/entity-sync.ts` returns at least 2 matches (the new block + the existing resource validation block); ordering must put the new block earlier in the file (lower line number) than the resource block.
|
||||||
|
- `npx tsc --noEmit --pretty` passes with zero errors.
|
||||||
|
- No edits to `syncTicketsChunked`, `getValidResourceIds`, `getValidContactIds`, or `getValidProjectIds`.
|
||||||
|
</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
</tasks>
|
||||||
|
|
||||||
|
<verification>
|
||||||
|
1. Type check passes: `npx tsc --noEmit --pretty` from repo root — zero errors.
|
||||||
|
2. Targeted greps:
|
||||||
|
- `grep -n "buildCompaniesFilter" lib/utils/sync-helpers.ts lib/services/entity-sync.ts` → 3 matches (one definition + import + usage).
|
||||||
|
- `grep -c "getValidCompanyIds" lib/services/entity-sync.ts` → 2.
|
||||||
|
- `grep -n "entity === EntityType.COMPANIES" lib/services/entity-sync.ts` → 1 new match in the filter chain.
|
||||||
|
3. Negative greps (proving we didn't touch out-of-scope code):
|
||||||
|
- `git diff lib/utils/sync-helpers.ts -- ':!**/buildCompaniesFilter*'` should show only the new function added.
|
||||||
|
- `git diff lib/services/entity-sync.ts` should show: import addition, COMPANIES filter branch, new `getValidCompanyIds` method, and new ticket company_id validation block. NOTHING else.
|
||||||
|
4. Read-back sanity:
|
||||||
|
- `entity-sync.ts` line ~152 (was the start of the generic `else` calling `buildActiveFilter`) is now NOT reached for COMPANIES because the new branch fires first.
|
||||||
|
- `hasAppliedFilters` (line ~173) is still true for COMPANIES full sync because `buildCompaniesFilter()` returns a non-empty filter.
|
||||||
|
</verification>
|
||||||
|
|
||||||
|
<success_criteria>
|
||||||
|
- `lib/utils/sync-helpers.ts` exports `buildCompaniesFilter()` returning `[{ field: 'id', op: 'gt', value: 0 }]`.
|
||||||
|
- `lib/services/entity-sync.ts` routes COMPANIES through `buildCompaniesFilter` on full sync; the generic `buildActiveFilter` branch is no longer hit for COMPANIES.
|
||||||
|
- `lib/services/entity-sync.ts` defensively nullifies ticket `company_id` for any company missing from the Pulse mirror, mirroring the existing resource-FK nullification pattern.
|
||||||
|
- `npx tsc --noEmit --pretty` passes.
|
||||||
|
- No changes to `getActiveField`, `buildActiveFilter`, `syncTicketsChunked`, the reconciliation service from quick-260521-fci, or the sync-progress-lock code.
|
||||||
|
- SUMMARY.md notes the manual `curl -X POST http://localhost:3100/api/sync/full` verification step but does NOT execute it.
|
||||||
|
</success_criteria>
|
||||||
|
|
||||||
|
<output>
|
||||||
|
After completion, create `.planning/quick/260521-foj-fix-weekly-full-fk-error-widen-companies/260521-foj-SUMMARY.md` with:
|
||||||
|
- What changed in each file (1-2 lines per file).
|
||||||
|
- Why this fixes the FK error (1 paragraph: Companies sync was filtering on `isActive=true` via `buildActiveFilter`, missing inactive companies referenced by tickets; widening to `id > 0` closes the gap; the defensive nullification handles the residual hard-deleted-in-Autotask edge case).
|
||||||
|
- Manual verify step for the user: `curl -X POST http://localhost:3100/api/sync/full -H 'content-type: application/json' -d '{"triggeredBy":"manual-verify"}'`, then check `sync_history` for the next full ticket sync.
|
||||||
|
- Explicit note: did NOT modify `getActiveField`, `syncTicketsChunked`, or the reconciliation service.
|
||||||
|
</output>
|
||||||
|
|
@ -0,0 +1,122 @@
|
||||||
|
---
|
||||||
|
phase: quick-260521-foj
|
||||||
|
plan: 01
|
||||||
|
subsystem: sync
|
||||||
|
tags:
|
||||||
|
- autotask-sync
|
||||||
|
- companies
|
||||||
|
- tickets
|
||||||
|
- foreign-key
|
||||||
|
- bugfix
|
||||||
|
requirements:
|
||||||
|
- QUICK-260521-FOJ
|
||||||
|
dependency_graph:
|
||||||
|
requires:
|
||||||
|
- lib/utils/sync-helpers.ts (existing build*Filter pattern)
|
||||||
|
- lib/services/entity-sync.ts (existing full-sync filter chain + tickets validation pipeline)
|
||||||
|
- lib/services/postgres-client.ts (singleton query interface)
|
||||||
|
provides:
|
||||||
|
- buildCompaniesFilter() (lib/utils/sync-helpers.ts)
|
||||||
|
- EntitySyncService.getValidCompanyIds() (private)
|
||||||
|
- Defensive ticket company_id nullification in syncEntity()
|
||||||
|
affects:
|
||||||
|
- Companies full sync (now fetches active + inactive)
|
||||||
|
- Tickets full + weekly-full sync (no longer rolls back on tickets_company_id_fkey)
|
||||||
|
tech_stack:
|
||||||
|
added: []
|
||||||
|
patterns:
|
||||||
|
- Mirror existing buildProjectPhasesFilter / buildContractServicesFilter "id > 0" template for any Autotask entity that requires a filter but where we want everything
|
||||||
|
- Mirror existing getValidResourceIds + null-on-miss pattern for FK validation in syncEntity()
|
||||||
|
key_files:
|
||||||
|
created: []
|
||||||
|
modified:
|
||||||
|
- lib/utils/sync-helpers.ts
|
||||||
|
- lib/services/entity-sync.ts
|
||||||
|
decisions:
|
||||||
|
- Widen Companies full sync to id > 0 (fetch all) rather than carrying isActive=true via buildActiveFilter, so inactive Autotask companies referenced by tickets are mirrored.
|
||||||
|
- Add a belt-and-suspenders ticket company_id nullification block for the truly hard-deleted Autotask company case. Non-destructive — preserves the ticket row.
|
||||||
|
- Did NOT touch getActiveField — still returns 'isActive' for COMPANIES because other entities and call sites use that mapping.
|
||||||
|
- Did NOT touch syncTicketsChunked — out of scope; that path has its own flow and is not on the failing weekly-full path.
|
||||||
|
metrics:
|
||||||
|
duration: 1m 49s
|
||||||
|
completed_date: 2026-05-21
|
||||||
|
---
|
||||||
|
|
||||||
|
# Quick 260521-foj: Fix weekly-full FK error — widen Companies full sync Summary
|
||||||
|
|
||||||
|
One-liner: Widen Companies full sync to fetch active+inactive companies (id > 0) and defensively nullify ticket.company_id for hard-deleted-in-Autotask companies — eliminates the `tickets_company_id_fkey` rollback that has been failing every full and weekly-full sync since 2026-05-15.
|
||||||
|
|
||||||
|
## What changed
|
||||||
|
|
||||||
|
**`lib/utils/sync-helpers.ts`**
|
||||||
|
- Added new exported `buildCompaniesFilter()` returning `[{ field: 'id', op: 'gt', value: 0 }]`, mirroring the `buildProjectPhasesFilter` template. JSDoc explains the rationale (Companies endpoint requires a filter; active-only misses inactive companies referenced by tickets).
|
||||||
|
|
||||||
|
**`lib/services/entity-sync.ts`**
|
||||||
|
- Added `buildCompaniesFilter` to the named-imports block from `../utils/sync-helpers`.
|
||||||
|
- Inserted a new `COMPANIES` branch as the first branch in the full-sync filter chain (placed before `CONTRACTS`). It pushes the `id > 0` filter and logs `Full sync of all companies (active + inactive)`. This routes COMPANIES around the generic `else` block that would otherwise apply `buildActiveFilter` (`isActive=true`).
|
||||||
|
- Added a new private `getValidCompanyIds()` method that selects all `companies.id WHERE is_deleted = false`, mirroring `getValidResourceIds`.
|
||||||
|
- Added a new `if (entity === EntityType.TICKETS)` block between the existing `recordsWithoutCompany` filter and the existing resource-FK nullification block. It iterates `mappedRecords`, nullifies `ticket.company_id` when the referenced company is missing from the Pulse mirror, tracks up to 5 sample ticket IDs, and emits a `warn` log when the count is non-zero. Pattern matches the surrounding resource nullification block exactly.
|
||||||
|
|
||||||
|
## Why this fixes the FK error
|
||||||
|
|
||||||
|
Every full and weekly-full sync since 2026-05-15 was failing with `[DATABASE_CONSTRAINT_ERROR] ... tickets_company_id_fkey`, rolling back the tickets bulkUpsert transaction. Root cause: Companies full sync was applying `buildActiveFilter`, which sends `isActive=true` to the Autotask API. Companies marked inactive in Autotask but still referenced by open tickets were therefore not mirrored into Pulse. When the tickets sync ran next, the FK constraint on `tickets.company_id → companies.id` fired and rolled the whole batch back. Widening the Companies filter to `id > 0` closes the primary gap by fetching active+inactive companies. The defensive nullification covers the residual edge case where a company was hard-deleted from Autotask entirely — instead of rolling back, the ticket survives with `company_id = null` (the column is nullable, and the existing `recordsWithoutCompany` filter only drops tickets that have no `company_id` from the start, so the bulkUpsert path handles `null` cleanly via the resource-nullification precedent).
|
||||||
|
|
||||||
|
`hasAppliedFilters` stays true on Companies full sync because the new filter array is non-empty (`id > 0`), so `softDeleteMissingRecords` is NOT triggered — no accidental mass soft-delete of companies. Companies do not support incremental sync (per `supportsIncremental` at entity-sync.ts line 106), so this code path runs on every Companies sync (full, weekly-full, and the incremental→full fallback).
|
||||||
|
|
||||||
|
## Commits
|
||||||
|
|
||||||
|
| Task | Description | Commit | Files |
|
||||||
|
| ---- | ------------------------------------------------------------------------------------ | ------- | ------------------------------------------------ |
|
||||||
|
| 1 | Add buildCompaniesFilter + wire COMPANIES into the full-sync filter chain | 1ecaefe | lib/utils/sync-helpers.ts, lib/services/entity-sync.ts |
|
||||||
|
| 2 | Add getValidCompanyIds + defensive ticket company_id nullification | 62c529f | lib/services/entity-sync.ts |
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
Type check passed cleanly:
|
||||||
|
|
||||||
|
```
|
||||||
|
cd /opt/stacks/pulse && npx tsc --noEmit --pretty
|
||||||
|
# (zero errors)
|
||||||
|
```
|
||||||
|
|
||||||
|
Targeted greps confirm shape:
|
||||||
|
- `grep -n "buildCompaniesFilter" lib/utils/sync-helpers.ts lib/services/entity-sync.ts` → 3 matches (definition + import + usage)
|
||||||
|
- `grep -c "getValidCompanyIds" lib/services/entity-sync.ts` → 2 (definition + call site)
|
||||||
|
- `grep -n "entity === EntityType.COMPANIES" lib/services/entity-sync.ts` → 1 new match (line 134, in the filter chain)
|
||||||
|
- `grep -n "if (entity === EntityType.TICKETS)" lib/services/entity-sync.ts` → 2 matches: line 261 (new company-validation block) BEFORE line 284 (existing resource-validation block), ordering correct.
|
||||||
|
|
||||||
|
## Manual verify step (user runs after merge)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:3100/api/sync/full \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"triggeredBy":"manual-verify"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Then check `sync_history` for the next ticket sync row — `error_message` should no longer contain `tickets_company_id_fkey`, and `status` should be `completed`. Also sanity-check `SELECT count(*) FROM companies WHERE is_active = false AND is_deleted = false;` — the count should be ≥ the previous ~145 (or whatever the prior baseline was) once Companies full sync picks up inactive companies that weren't being fetched before.
|
||||||
|
|
||||||
|
## Out-of-scope (explicit non-changes)
|
||||||
|
|
||||||
|
- `getActiveField` in `lib/utils/sync-helpers.ts` — unchanged. Still returns `'isActive'` for COMPANIES (used by other call sites and for other entities like RESOURCES, CONTACTS, etc.).
|
||||||
|
- `buildActiveFilter` — unchanged.
|
||||||
|
- `syncTicketsChunked` — unchanged. The chunked path has its own (separate) flow and was not on the failing weekly-full path. Touching it was explicitly out of scope per the plan.
|
||||||
|
- The reconciliation service from `quick-260521-fci` — unchanged.
|
||||||
|
- `cachedValidResourceIds` / `cachedValidContactIds` — unchanged.
|
||||||
|
- The `lastActivityDate` filter on tickets — unchanged.
|
||||||
|
- Other branches in the full-sync filter chain (CONTRACTS, CONTRACT_SERVICES, PROJECTS, PROJECT_PHASES, TIME_ENTRIES, BILLING_ITEMS) — unchanged.
|
||||||
|
|
||||||
|
## Deviations from Plan
|
||||||
|
|
||||||
|
None — plan executed exactly as written.
|
||||||
|
|
||||||
|
## Known Stubs
|
||||||
|
|
||||||
|
None — both functions are wired end-to-end into the live sync path.
|
||||||
|
|
||||||
|
## Self-Check: PASSED
|
||||||
|
|
||||||
|
- `lib/utils/sync-helpers.ts` exists and contains `export function buildCompaniesFilter` at line 100.
|
||||||
|
- `lib/services/entity-sync.ts` exists; import (line 18), usage (line 135), COMPANIES branch (line 134), `getValidCompanyIds` method, and the new TICKETS company-validation block (line 261) all present.
|
||||||
|
- Commit `1ecaefe` exists in `git log` (Task 1).
|
||||||
|
- Commit `62c529f` exists in `git log` (Task 2).
|
||||||
|
- `npx tsc --noEmit --pretty` exits 0.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue