docs(quick-260521-fci): Stopgap nightly reconciliation for stale open tickets in postgres mirror

2026-05-21 11:11:33 -04:00 · 2026-05-21 11:11:33 -04:00 · d02796e863
commit d02796e863
parent badd718194
3 changed files with 649 additions and 1 deletions
--- a/.planning/STATE.md
+++ b/.planning/STATE.md
@ -28,7 +28,7 @@ See: .planning/PROJECT.md (updated 2026-05-03)
 Phase: 09.1 (ntfy-backend-fix) — EXECUTING
 Plan: 1 of 1
 Status: Executing Phase 09.1
-Last activity: 2026-05-19 - Completed quick task 260519-0oz: Add QBO createPayment + createDeposit + .FH reconciliation script
+Last activity: 2026-05-21 - Completed quick task 260521-fci: Stopgap nightly reconciliation for stale open tickets in postgres mirror

 Progress: [░░░░░░░░░░] 0%

@ -94,6 +94,7 @@ None yet.
 | # | Description | Date | Commit | Directory |
 |---|-------------|------|--------|-----------|
 | 260519-0oz | Add QBO createPayment + createDeposit + .FH reconciliation script | 2026-05-19 | 5497458 | [260519-0oz-add-qbo-createpayment-createdeposit-fh-r](./quick/260519-0oz-add-qbo-createpayment-createdeposit-fh-r/) |
+| 260521-fci | Stopgap nightly reconciliation for stale open tickets in postgres mirror | 2026-05-21 | badd718 | [260521-fci-stopgap-nightly-reconciliation-for-stale](./quick/260521-fci-stopgap-nightly-reconciliation-for-stale/) |

 ## Session Continuity

--- a/.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-PLAN.md
+++ b/.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-PLAN.md
@ -0,0 +1,525 @@
+---
+phase: quick-260521-fci
+plan: 01
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+  - lib/services/ticket-reconciliation-service.ts
+  - app/api/sync/reconcile-tickets/route.ts
+  - lib/services/sync-scheduler.ts
+  - migrations/090_ticket_reconcile_schedule.sql
+autonomous: true
+requirements:
+  - QUICK-260521-FCI-01
+must_haves:
+  truths:
+    - "A nightly job scans the postgres tickets mirror for rows where is_deleted=false AND status<>5 AND synced_at < NOW() - INTERVAL '7 days' (capped at LIMIT 500)"
+    - "For each stale ticket id, the job fetches the current Autotask record via AutotaskClient.getEntityById('Tickets', id) and either upserts (using the existing mapAutotaskToDatabase + webhook upsert SQL) or soft-deletes (is_deleted=true, deleted_at=NOW()) when Autotask returns null"
+    - "The job returns { scanned, updated, statusFlippedToComplete, softDeleted, errors } and writes one summary log line per run"
+    - "POST /api/sync/reconcile-tickets triggers the job non-blocking (fire-and-forget), mirroring /api/sync/incremental"
+    - "A 'tickets-reconcile' default schedule (id=tickets-reconcile, cron='30 4 * * *') exists in sync_schedules and is dispatched via the scheduler's switch (disabled by default, like other defaults)"
+    - "Migration 090 idempotently seeds the schedule row with ON CONFLICT (id) DO NOTHING so it lands on a fresh DB without disturbing existing installs"
+    - "`npx tsc --noEmit --pretty` passes — type-check is the safety net (no tests for sync services)"
+  artifacts:
+    - path: "lib/services/ticket-reconciliation-service.ts"
+      provides: "reconcileStaleTickets() function returning ReconcileResult"
+      exports: ["reconcileStaleTickets", "ReconcileResult"]
+    - path: "app/api/sync/reconcile-tickets/route.ts"
+      provides: "POST handler — fire-and-forget trigger for reconcileStaleTickets()"
+      exports: ["POST"]
+    - path: "lib/services/sync-scheduler.ts"
+      provides: "Updated sync_type union + dispatch case + default schedule entry for tickets-reconcile"
+      contains: "tickets-reconcile"
+    - path: "migrations/090_ticket_reconcile_schedule.sql"
+      provides: "Idempotent INSERT for tickets-reconcile schedule row"
+      contains: "ON CONFLICT (id) DO NOTHING"
+  key_links:
+    - from: "lib/services/sync-scheduler.ts"
+      to: "lib/services/ticket-reconciliation-service.ts"
+      via: "dynamic import + direct call in executeScheduledSync switch (matches device-link-reconcile / integration-health pattern)"
+      pattern: "reconcileStaleTickets\\("
+    - from: "app/api/sync/reconcile-tickets/route.ts"
+      to: "lib/services/ticket-reconciliation-service.ts"
+      via: "fire-and-forget call with .catch() — mirrors /api/sync/incremental"
+      pattern: "reconcileStaleTickets\\(\\)\\.catch"
+    - from: "lib/services/ticket-reconciliation-service.ts"
+      to: "lib/utils/entity-mapper.ts"
+      via: "mapAutotaskToDatabase(EntityType.TICKETS, entity) — reuse, do not parallel-implement"
+      pattern: "mapAutotaskToDatabase\\(EntityType\\.TICKETS"
+---
+
+<objective>
+Build a STOPGAP nightly reconciliation job that detects "drift" between the
+postgres `tickets` mirror and Autotask: tickets that have not been re-synced in
+>7 days and are still marked open locally. For each, fetch the canonical
+Autotask record by id and either re-upsert (status may have flipped to Complete
+out-of-band, e.g. via a missed webhook) or soft-delete (Autotask no longer
+returns the row). Wire it into the existing scheduler as a nightly cron.
+
+Purpose: The webhook can drop events (Autotask retries are limited, network
+hiccups happen), and the incremental sync uses a high-water-mark filter that
+will miss a ticket whose `lastActivityDate` never advances after a status
+change. This job is a backstop — not a replacement for fixing the incremental
+filter, but a way to bound staleness at 7 days while that fix is scoped
+separately.
+
+Output:
+- `lib/services/ticket-reconciliation-service.ts` (new)
+- `app/api/sync/reconcile-tickets/route.ts` (new)
+- Edits to `lib/services/sync-scheduler.ts` (union + default + dispatch)
+- `migrations/090_ticket_reconcile_schedule.sql` (new)
+</objective>
+
+<execution_context>
+@$HOME/.claude/get-shit-done/workflows/execute-plan.md
+@$HOME/.claude/get-shit-done/templates/summary.md
+</execution_context>
+
+<context>
+@.planning/STATE.md
+@CLAUDE.md
+
+<!-- Existing patterns the executor must mirror exactly -->
+@lib/services/webhook-service.ts
+@lib/services/sync-scheduler.ts
+@app/api/sync/incremental/route.ts
+@lib/utils/entity-mapper.ts
+@lib/utils/sync-helpers.ts
+@lib/utils/sync-logger.ts
+@lib/services/autotask-client.ts
+@lib/services/rate-limiter.ts
+@lib/services/postgres-client.ts
+@lib/types/sync.ts
+@middleware.ts
+
+<interfaces>
+<!-- Key contracts extracted from the codebase. Executor uses these directly. -->
+
+From `lib/services/autotask-client.ts:164`:
+```typescript
+async getEntityById<T>(entityName: string, id: number): Promise<T | null>
+// Internally calls this.rateLimiter.throttle() (10 req/sec). For Tickets, pass 'Tickets'.
+```
+
+From `lib/utils/entity-mapper.ts`:
+```typescript
+export function mapAutotaskToDatabase(
+  entity: EntityType,
+  data: any
+): Record<string, any>
+// For EntityType.TICKETS, runs the ticket-specific mapper and appends synced_at=new Date()
+```
+
+From `lib/utils/sync-helpers.ts`:
+```typescript
+export function getTableName(entity: EntityType): string
+// getTableName(EntityType.TICKETS) -> 'tickets'
+```
+
+From `lib/utils/sync-logger.ts:250`:
+```typescript
+export function createSyncLogger(context?: LogContext): SyncLogger
+// Use: const logger = createSyncLogger({ component: 'TicketReconciliation' })
+// Methods: logger.info(msg, ctx?), logger.warn(msg, ctx?, err?), logger.error(msg, ctx?, err?)
+```
+
+From `lib/services/rate-limiter.ts:114`:
+```typescript
+export const autotaskRateLimiter: RateLimiter
+// Use: await autotaskRateLimiter.throttle(async () => { ... })
+```
+
+From `lib/types/sync.ts`:
+```typescript
+export enum EntityType { TICKETS = 'tickets', ... }
+```
+
+From `lib/services/postgres-client.ts`:
+```typescript
+postgresClient.query<T>(sql: string, params?: any[]): Promise<QueryResult<T>>
+// .rows: T[]
+```
+
+**Webhook upsert pattern to mirror exactly** (`lib/services/webhook-service.ts:201-234`):
+```typescript
+const mappedData = mapAutotaskToDatabase(internalEntityType, entityData);
+const tableName = getTableName(internalEntityType);
+const keys = Object.keys(mappedData);
+const values = Object.values(mappedData);
+const placeholders = keys.map((_, i) => `$${i + 1}`).join(', ');
+const updateClause = keys
+  .filter(k => k !== 'id')
+  .map(k => `${k} = EXCLUDED.${k}`)
+  .join(', ');
+const query = `
+  INSERT INTO ${tableName} (${keys.join(', ')})
+  VALUES (${placeholders})
+  ON CONFLICT (id)
+  DO UPDATE SET ${updateClause}, updated_at = NOW()
+`;
+await postgresClient.query(query, values);
+```
+
+**Tickets table schema reminders:**
+- `id` is `BIGINT` (Autotask id)
+- Audit cols: `created_at`, `updated_at`, `synced_at`, `is_deleted`, `deleted_at`
+- Status 5 = Complete in Autotask convention (see `ticket-digest-service.ts:193` precedent: `status NOT IN (5)`)
+
+**Existing dispatch precedent** (`sync-scheduler.ts:425-449`):
+The `device-link-reconcile` and `integration-health` cases use dynamic `await import()`
+and call the service directly (not via HTTP). The dispatch logs a one-line summary.
+Mirror that pattern — do NOT POST to the API route from the scheduler. (The API
+route exists for manual/admin triggering, the scheduler calls the service directly.)
+
+**Middleware note:** `/api/sync` is already in `publicRoutes` (middleware.ts:30),
+so the new `/api/sync/reconcile-tickets` route is auto-allowed. No middleware
+edit needed.
+
+**Default-schedule seeding caveat:** `createDefaultSchedules()` in
+`sync-scheduler.ts` only inserts when the table is empty (line 173 check).
+For existing DBs the in-code default won't fire — that's why migration 090
+seeds the row explicitly with `ON CONFLICT (id) DO NOTHING`. On a fresh DB
+both paths are idempotent.
+</interfaces>
+</context>
+
+<tasks>
+
+<task type="auto">
+  <name>Task 1: Build reconciliation service + API route</name>
+  <files>
+    lib/services/ticket-reconciliation-service.ts,
+    app/api/sync/reconcile-tickets/route.ts
+  </files>
+  <action>
+**Create `lib/services/ticket-reconciliation-service.ts`:**
+
+```typescript
+/**
+ * Ticket Reconciliation Service (STOPGAP)
+ *
+ * Nightly backstop for drift between the postgres `tickets` mirror and Autotask.
+ * Webhooks can drop events and the incremental sync's high-water-mark filter
+ * can miss tickets whose lastActivityDate never advances after a status change.
+ * This job bounds local staleness at 7 days by re-fetching each stale-open
+ * ticket from Autotask and either re-upserting or soft-deleting.
+ *
+ * NOTE: This is a stopgap. The underlying incremental-filter issue is tracked
+ * separately and should fix the root cause.
+ */
+
+import { postgresClient } from './postgres-client';
+import { AutotaskClient } from './autotask-client';
+import { autotaskRateLimiter } from './rate-limiter';
+import { mapAutotaskToDatabase } from '../utils/entity-mapper';
+import { getTableName } from '../utils/sync-helpers';
+import { EntityType } from '../types/sync';
+import { createSyncLogger } from '../utils/sync-logger';
+
+export interface ReconcileResult {
+  scanned: number;
+  updated: number;
+  statusFlippedToComplete: number;
+  softDeleted: number;
+  errors: number;
+}
+
+const STALE_DAYS = 7;
+const SCAN_LIMIT = 500;
+const COMPLETE_STATUS = 5; // Autotask status id for Complete
+
+let _client: AutotaskClient | null = null;
+function getClient(): AutotaskClient {
+  if (!_client) {
+    _client = new AutotaskClient({
+      apiUrl: process.env.AUTOTASK_API_URL || '',
+      username: process.env.AUTOTASK_USERNAME || '',
+      password: process.env.AUTOTASK_SECRET || '',
+      apiIntegrationCode: process.env.AUTOTASK_API_INTEGRATION_CODE || '',
+    });
+  }
+  return _client;
+}
+
+/**
+ * Re-fetch up to SCAN_LIMIT stale-open tickets from Autotask and reconcile
+ * the local mirror. Idempotent and safe to run alongside the regular sync.
+ */
+export async function reconcileStaleTickets(): Promise<ReconcileResult> {
+  const logger = createSyncLogger({ component: 'TicketReconciliation' });
+  const startedAt = Date.now();
+  const result: ReconcileResult = {
+    scanned: 0,
+    updated: 0,
+    statusFlippedToComplete: 0,
+    softDeleted: 0,
+    errors: 0,
+  };
+
+  // 1. Find stale-open tickets in the postgres mirror.
+  const staleQuery = `
+    SELECT id, status
+    FROM tickets
+    WHERE is_deleted = false
+      AND status <> $1
+      AND synced_at < NOW() - INTERVAL '${STALE_DAYS} days'
+    ORDER BY synced_at ASC
+    LIMIT $2
+  `;
+  const stale = await postgresClient.query<{ id: string; status: number | null }>(
+    staleQuery,
+    [COMPLETE_STATUS, SCAN_LIMIT]
+  );
+  result.scanned = stale.rows.length;
+
+  logger.info(`Scanning ${result.scanned} stale-open tickets (>${STALE_DAYS}d, limit=${SCAN_LIMIT})`);
+
+  if (result.scanned === 0) {
+    logger.info('No stale tickets found', { duration: Date.now() - startedAt });
+    return result;
+  }
+
+  const client = getClient();
+  const tableName = getTableName(EntityType.TICKETS); // 'tickets'
+
+  // 2. Walk each id. Use the singleton rate limiter as an extra guard on top of
+  //    AutotaskClient's internal limiter (defense in depth for parallel callers).
+  for (const row of stale.rows) {
+    const ticketId = Number(row.id);
+    const priorStatus = row.status;
+    try {
+      const entity = await autotaskRateLimiter.throttle(() =>
+        client.getEntityById<Record<string, any>>('Tickets', ticketId)
+      );
+
+      if (!entity) {
+        // Not in Autotask anymore -> soft-delete locally.
+        await postgresClient.query(
+          `UPDATE ${tableName}
+             SET is_deleted = true, deleted_at = NOW(), updated_at = NOW()
+           WHERE id = $1`,
+          [ticketId]
+        );
+        result.softDeleted += 1;
+        continue;
+      }
+
+      // 3. Upsert using the same SQL shape the webhook handler uses.
+      const mapped = mapAutotaskToDatabase(EntityType.TICKETS, entity);
+      if (!mapped) {
+        throw new Error(`mapAutotaskToDatabase returned null for ticket ${ticketId}`);
+      }
+      const keys = Object.keys(mapped);
+      const values = Object.values(mapped);
+      const placeholders = keys.map((_, i) => `$${i + 1}`).join(', ');
+      const updateClause = keys
+        .filter(k => k !== 'id')
+        .map(k => `${k} = EXCLUDED.${k}`)
+        .join(', ');
+      const upsertSql = `
+        INSERT INTO ${tableName} (${keys.join(', ')})
+        VALUES (${placeholders})
+        ON CONFLICT (id)
+        DO UPDATE SET ${updateClause}, updated_at = NOW()
+      `;
+      await postgresClient.query(upsertSql, values);
+      result.updated += 1;
+
+      const newStatus = (mapped.status ?? null) as number | null;
+      if (priorStatus !== COMPLETE_STATUS && newStatus === COMPLETE_STATUS) {
+        result.statusFlippedToComplete += 1;
+      }
+    } catch (err) {
+      result.errors += 1;
+      logger.warn(
+        `Reconcile failed for ticket ${ticketId}`,
+        { ticketId },
+        err instanceof Error ? err : new Error(String(err))
+      );
+    }
+  }
+
+  logger.info(
+    `Reconciliation complete: scanned=${result.scanned} updated=${result.updated} flippedComplete=${result.statusFlippedToComplete} softDeleted=${result.softDeleted} errors=${result.errors}`,
+    { duration: Date.now() - startedAt }
+  );
+
+  return result;
+}
+```
+
+**Create `app/api/sync/reconcile-tickets/route.ts`:**
+
+```typescript
+/**
+ * Ticket Reconciliation API Endpoint
+ * POST /api/sync/reconcile-tickets - Trigger a stale-ticket reconciliation pass.
+ * Public per middleware.ts (matches /api/sync/incremental). Fire-and-forget.
+ */
+
+import { NextRequest, NextResponse } from 'next/server';
+import { reconcileStaleTickets } from '@/lib/services/ticket-reconciliation-service';
+
+export async function POST(request: NextRequest) {
+  try {
+    const body = await request.json().catch(() => ({}));
+    const triggeredBy = body.triggeredBy || 'api';
+
+    // Non-blocking — same pattern as /api/sync/incremental.
+    reconcileStaleTickets().catch((error) => {
+      console.error('[RECONCILE] Ticket reconciliation failed:', error);
+    });
+
+    return NextResponse.json({
+      message: 'Ticket reconciliation started',
+      triggeredBy,
+    });
+  } catch (error) {
+    console.error('Failed to start ticket reconciliation:', error);
+    return NextResponse.json(
+      { error: 'Failed to start ticket reconciliation' },
+      { status: 500 }
+    );
+  }
+}
+```
+
+**Constraints reminders (CLAUDE.md):**
+- No Zod validation in route.
+- No `requireAuth()` — `/api/sync` is in middleware's `publicRoutes`; matches `incremental/route.ts`.
+- `postgresClient.query()`, manual snake_case ↔ camelCase via `mapAutotaskToDatabase`.
+- Errors: `try/catch` + `NextResponse.json({ error, message }, { status })`. 500 for runtime.
+- kebab-case filenames (both are). Use `@/lib/...` and `@/lib/services/...` imports — no relative `../../`.
+- Do NOT introduce a parallel mapping path. Do NOT touch `sync-helpers.ts`.
+  </action>
+  <verify>
+    <automated>npx tsc --noEmit --pretty 2>&amp;1 | grep -E "(ticket-reconciliation|reconcile-tickets)" ; echo "exit=$?"</automated>
+  </verify>
+  <done>
+- Both files exist at the listed paths.
+- `npx tsc --noEmit --pretty` is clean for the two new files (no errors referencing
+  `ticket-reconciliation-service.ts` or `app/api/sync/reconcile-tickets/route.ts`).
+- `reconcileStaleTickets()` exports a `ReconcileResult` shape matching
+  `{ scanned, updated, statusFlippedToComplete, softDeleted, errors }`.
+- The route is structured exactly like `/api/sync/incremental` (no auth, body parse
+  with `.catch(() => ({}))`, fire-and-forget call with `.catch()`).
+- Upsert SQL is the webhook-service pattern, not a new shape.
+  </done>
+</task>
+
+<task type="auto">
+  <name>Task 2: Wire scheduler + migration</name>
+  <files>
+    lib/services/sync-scheduler.ts,
+    migrations/090_ticket_reconcile_schedule.sql
+  </files>
+  <action>
+**Edit `lib/services/sync-scheduler.ts`:**
+
+1. **Extend the `sync_type` union** on the `ScheduleConfig` interface (line ~25). Append `| 'tickets-reconcile'` to the existing union literal:
+
+```typescript
+  sync_type: 'incremental' | 'full' | 'veeam-incremental' | 'veeam-full' | 'veeam-rpo-check' | 'contract-services' | 'engagement-daily' | 'zoom-daily' | 'morning-summary' | 'ticket-digest-daily' | 'ticket-digest-weekly' | 'ticket-digest-monthly' | 'device-link-reconcile' | 'integration-health' | 'qbo' | 'appgate-sessions' | 'appgate-daily' | 'tickets-reconcile';
+```
+
+2. **Add a default schedule entry** inside `createDefaultSchedules()` `defaultSchedules` array (alongside the `integration-health` entry around line 286-293). Append:
+
+```typescript
+      {
+        id: 'tickets-reconcile',
+        name: 'Tickets Reconciliation',
+        description: 'STOPGAP backstop: re-fetches stale-open tickets (>7d since last sync) from Autotask and reconciles status / soft-deletes missing rows. Runs daily at 4:30 AM. Capped at 500 tickets per run.',
+        cron_expression: '30 4 * * *',
+        sync_type: 'tickets-reconcile',
+        is_enabled: false,
+      },
+```
+
+(Match the other defaults: `is_enabled: false` so this seeds disabled and an admin enables it from `/admin`.)
+
+3. **Add a dispatch case** in `executeScheduledSync()`'s switch (the long `else if` chain starting at line ~397). Insert it next to `device-link-reconcile` / `integration-health` (anywhere in the chain before the trailing `incremental` / fallthrough cases). Use the dynamic-import pattern those two cases use:
+
+```typescript
+      } else if (config.sync_type === 'tickets-reconcile') {
+        const { reconcileStaleTickets } = await import('@/lib/services/ticket-reconciliation-service');
+        const result = await reconcileStaleTickets();
+        console.log(
+          `[SCHEDULER] tickets-reconcile: scanned=${result.scanned} updated=${result.updated} flippedComplete=${result.statusFlippedToComplete} softDeleted=${result.softDeleted} errors=${result.errors}`
+        );
+```
+
+**Do NOT** change anything about `weekly-full`, the morning summary, or any other existing schedule. Do NOT modify `sync-helpers.ts`.
+
+**Create `migrations/090_ticket_reconcile_schedule.sql`:**
+
+```sql
+-- Migration 090: Seed the tickets-reconcile sync schedule (STOPGAP).
+--
+-- The sync_scheduler.createDefaultSchedules() path only seeds defaults on a
+-- virgin sync_schedules table; this migration covers existing installs.
+-- Idempotent via ON CONFLICT (id) DO NOTHING.
+
+INSERT INTO sync_schedules (
+  id,
+  name,
+  description,
+  cron_expression,
+  sync_type,
+  years_back,
+  is_enabled
+) VALUES (
+  'tickets-reconcile',
+  'Tickets Reconciliation',
+  'STOPGAP backstop: re-fetches stale-open tickets (>7d since last sync) from Autotask and reconciles status / soft-deletes missing rows. Runs daily at 4:30 AM. Capped at 500 tickets per run.',
+  '30 4 * * *',
+  'tickets-reconcile',
+  NULL,
+  false
+)
+ON CONFLICT (id) DO NOTHING;
+```
+
+**Notes:**
+- Schedule starts **disabled** (`is_enabled = false`) — admin enables from `/admin/sync-schedules` (or equivalent UI) when ready. Matches the convention of nearly every other default.
+- Cron `30 4 * * *` = 4:30 AM daily, deliberately staggered from the 4:00 AM contract-services job to avoid Autotask rate-limit contention.
+- The migration is idempotent: on a fresh DB the default-schedule seeder may insert first; on an existing DB this migration is the only insert path. Either way `ON CONFLICT DO NOTHING` makes the order irrelevant.
+- No new migration numbers conflict — `ls migrations/` shows highest is `089_appgate_tables.sql`.
+  </action>
+  <verify>
+    <automated>npx tsc --noEmit --pretty 2>&amp;1 | grep -E "sync-scheduler" ; echo "exit=$?"</automated>
+  </verify>
+  <done>
+- `lib/services/sync-scheduler.ts` includes `'tickets-reconcile'` in the `sync_type` union, a new `defaultSchedules` entry with id `tickets-reconcile`, and a dispatch case calling `reconcileStaleTickets()` via dynamic import.
+- `migrations/090_ticket_reconcile_schedule.sql` exists with `ON CONFLICT (id) DO NOTHING` and the same id/cron/description as the default entry.
+- `npx tsc --noEmit --pretty` is clean for `sync-scheduler.ts` (no errors mentioning the file).
+- Manual smoke check (optional, not required for done): grep confirms no leftover edits to `weekly-full` or `sync-helpers.ts`.
+  </done>
+</task>
+
+</tasks>
+
+<verification>
+- `npx tsc --noEmit --pretty` exits clean across the whole repo (the project's only safety net for sync code).
+- `grep -n "tickets-reconcile" lib/services/sync-scheduler.ts` shows three hits: union, default entry, dispatch case.
+- `grep -n "reconcileStaleTickets" lib/services/` shows the export in the new service file and the import in the scheduler.
+- `ls migrations/090_*.sql` returns the new file.
+- Sanity-only manual check (do NOT execute as part of automated verify): a one-shot trigger via `curl -X POST http://localhost:3100/api/sync/reconcile-tickets -H 'Content-Type: application/json' -d '{"triggeredBy":"manual-smoke"}'` should return `{ "message": "Ticket reconciliation started", ... }` and the next container log line should be `[TicketReconciliation] Scanning N stale-open tickets...`. Document this in the SUMMARY, don't gate on it.
+</verification>
+
+<success_criteria>
+- All four artifacts (service, route, scheduler edits, migration) exist and type-check.
+- The job's behavioral contract holds: stale-open tickets are either updated or soft-deleted, never silently left behind.
+- No CLAUDE.md violations: no ORM, no Zod-in-route, no parallel mapper, no auth on `/api/sync/*`, kebab-case filenames, `@/...` imports.
+- Migration 090 is idempotent. The schedule lands disabled by default.
+- Scope discipline: zero changes to `weekly-full`, `sync-helpers.ts`, or any non-listed file.
+</success_criteria>
+
+<output>
+After completion, create `.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-SUMMARY.md` with:
+- Files created/modified (paths + 1-line role each)
+- The `ReconcileResult` shape and where it's logged
+- Cron + default-disabled note (for the eventual ops handoff)
+- Any deviations from this plan and why
+- Pointer to the underlying incremental-filter issue this stopgaps (link to existing tracking, or note "no separate tracker yet")
+</output>
--- a/.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-SUMMARY.md
+++ b/.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-SUMMARY.md
@ -0,0 +1,122 @@
+---
+phase: quick-260521-fci
+plan: 01
+subsystem: sync
+tags: [stopgap, autotask, reconciliation, scheduler]
+requires:
+  - existing AutotaskClient.getEntityById
+  - existing mapAutotaskToDatabase / getTableName / autotaskRateLimiter
+  - existing sync_schedules table + sync-scheduler dispatch chain
+provides:
+  - reconcileStaleTickets() — re-fetches stale-open tickets and reconciles
+  - POST /api/sync/reconcile-tickets — fire-and-forget trigger
+  - tickets-reconcile cron schedule (disabled by default, 30 4 * * *)
+affects:
+  - lib/services/sync-scheduler.ts (sync_type union + default schedule + dispatch case)
+tech_stack:
+  added: []
+  patterns:
+    - dynamic-import-in-scheduler-dispatch
+    - webhook-style ON CONFLICT (id) DO UPDATE upsert
+    - rate-limited per-row Autotask getEntityById walk
+key_files:
+  created:
+    - lib/services/ticket-reconciliation-service.ts
+    - app/api/sync/reconcile-tickets/route.ts
+    - migrations/090_ticket_reconcile_schedule.sql
+  modified:
+    - lib/services/sync-scheduler.ts
+decisions:
+  - STOPGAP only — does NOT fix the underlying incremental-filter root cause
+  - LIMIT 500 per run + 10 req/sec rate limit caps worst-case API cost
+  - 4:30 AM cron deliberately staggered from 4:00 AM contract-services job
+  - Soft-delete (not hard-delete) when Autotask returns null — preserves audit
+  - Schedule lands DISABLED on both fresh + existing DBs; admin enables via UI
+metrics:
+  duration_seconds: 129
+  completed_at: 2026-05-21T15:09:45Z
+  tasks_completed: 2
+  files_created: 3
+  files_modified: 1
+---
+
+# Quick Task 260521-fci: STOPGAP nightly reconciliation for stale tickets — Summary
+
+One-liner: A nightly cron walks postgres tickets that are still open locally but
+haven't synced in >7 days, re-fetches each from Autotask via getEntityById, and
+either upserts the canonical row or soft-deletes it — backstopping missed
+webhooks and the incremental sync's high-water-mark blind spot.
+
+## Files Created / Modified
+
+| File | Role |
+| ---- | ---- |
+| `lib/services/ticket-reconciliation-service.ts` (created) | `reconcileStaleTickets()` — finds stale-open tickets, re-fetches from Autotask, upserts or soft-deletes. Returns `ReconcileResult`. |
+| `app/api/sync/reconcile-tickets/route.ts` (created) | POST endpoint, fire-and-forget trigger mirroring `/api/sync/incremental` (no auth — public per existing `/api/sync` middleware allowlist). |
+| `lib/services/sync-scheduler.ts` (modified) | Extended `sync_type` union with `'tickets-reconcile'`, added a `defaultSchedules` entry (disabled), and a dispatch case using the dynamic-import pattern matching `device-link-reconcile` / `integration-health`. |
+| `migrations/090_ticket_reconcile_schedule.sql` (created) | Idempotent `INSERT … ON CONFLICT (id) DO NOTHING` so existing installs pick up the schedule row without disturbing the fresh-DB default-seed path. |
+
+## Result Shape
+
+`ReconcileResult` is logged once per run in two places:
+
+```ts
+interface ReconcileResult {
+  scanned: number;                  // rows pulled from postgres for processing
+  updated: number;                  // successfully upserted from Autotask
+  statusFlippedToComplete: number;  // priorStatus != 5, newStatus == 5
+  softDeleted: number;              // Autotask returned null -> is_deleted=true
+  errors: number;                   // per-row failures (logged, do not abort run)
+}
+```
+
+Log lines:
+1. **Service:** `[TicketReconciliation] Reconciliation complete: scanned=N updated=N flippedComplete=N softDeleted=N errors=N`
+   (via `createSyncLogger({ component: 'TicketReconciliation' })`)
+2. **Scheduler:** `[SCHEDULER] tickets-reconcile: scanned=N updated=N flippedComplete=N softDeleted=N errors=N`
+
+## Operational Notes (handoff)
+
+- **Cron:** `30 4 * * *` (4:30 AM daily). Staggered 30 min after the 4:00 AM `contract-services` job to avoid rate-limit contention.
+- **Default state:** `is_enabled = false`. Enable via the existing `/admin` sync-schedules UI when ready.
+- **Capacity:** Hard-capped at `LIMIT 500` per run. With the 10 req/sec Autotask limiter that's ~50s minimum walltime if every ticket is stale; usually far less.
+- **Manual trigger:** `curl -X POST http://localhost:3100/api/sync/reconcile-tickets -H 'Content-Type: application/json' -d '{"triggeredBy":"manual-smoke"}'` — returns `{ "message": "Ticket reconciliation started", "triggeredBy": "manual-smoke" }` immediately; result lines appear in container logs.
+- **Idempotency:** Safe to run alongside the regular incremental sync — the upsert path uses the same SQL the webhook uses, and the soft-delete is column-level (no row deletion).
+- **Failure mode:** Per-row exceptions increment `errors` and continue; only postgres infra failures abort the run.
+
+## Pointer to Underlying Issue
+
+This is a **STOPGAP**. The root cause is that Autotask's incremental sync uses
+`lastTrackedModificationDateTime` (a.k.a. `lastActivityDate`-style high-water-mark)
+which can fail to advance after some status transitions, so a status change to
+Complete may never trigger an incremental row in our pipeline. The webhook path
+also drops events on rare network/timeout edges.
+
+**No separate tracker exists yet** for the root cause. When that work is scoped
+(likely as a "fix incremental filter for ticket status transitions" plan), this
+job can either be retired or downgraded to a weekly safety net.
+
+## Deviations from Plan
+
+None — the plan was followed exactly as written. The only nuance was that the
+union already contained `qbo | appgate-sessions | appgate-daily` from earlier
+work in this worktree (pre-existing uncommitted changes), so the edit appended
+`| 'tickets-reconcile'` to that already-extended union rather than the one
+literally shown in the plan. Behavior is identical.
+
+## Verification
+
+- `npx tsc --noEmit --pretty` — clean (no output, full repo passes)
+- `grep -n "tickets-reconcile" lib/services/sync-scheduler.ts` — three hits (union L25, default L295/L299, dispatch L458)
+- `grep -rn "reconcileStaleTickets" lib/services/ app/api/sync/reconcile-tickets/` — one export, two imports (scheduler + route)
+- `ls migrations/090_*.sql` — present
+- Manual smoke (not run as part of automated verify, per plan): the curl POST above is the suggested ops handoff smoke test.
+
+## Self-Check: PASSED
+
+- `lib/services/ticket-reconciliation-service.ts` — FOUND
+- `app/api/sync/reconcile-tickets/route.ts` — FOUND
+- `migrations/090_ticket_reconcile_schedule.sql` — FOUND
+- Commit `51f0b32` (Task 1: service + route) — FOUND
+- Commit `badd718` (Task 2: scheduler + migration) — FOUND
+- Full `npx tsc --noEmit --pretty` — clean