diff --git a/.planning/STATE.md b/.planning/STATE.md index 2d55ac6..1d07cf4 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -28,7 +28,7 @@ See: .planning/PROJECT.md (updated 2026-05-03) Phase: 09.1 (ntfy-backend-fix) — EXECUTING Plan: 1 of 1 Status: Executing Phase 09.1 -Last activity: 2026-05-19 - Completed quick task 260519-0oz: Add QBO createPayment + createDeposit + .FH reconciliation script +Last activity: 2026-05-21 - Completed quick task 260521-fci: Stopgap nightly reconciliation for stale open tickets in postgres mirror Progress: [░░░░░░░░░░] 0% @@ -94,6 +94,7 @@ None yet. | # | Description | Date | Commit | Directory | |---|-------------|------|--------|-----------| | 260519-0oz | Add QBO createPayment + createDeposit + .FH reconciliation script | 2026-05-19 | 5497458 | [260519-0oz-add-qbo-createpayment-createdeposit-fh-r](./quick/260519-0oz-add-qbo-createpayment-createdeposit-fh-r/) | +| 260521-fci | Stopgap nightly reconciliation for stale open tickets in postgres mirror | 2026-05-21 | badd718 | [260521-fci-stopgap-nightly-reconciliation-for-stale](./quick/260521-fci-stopgap-nightly-reconciliation-for-stale/) | ## Session Continuity diff --git a/.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-PLAN.md b/.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-PLAN.md new file mode 100644 index 0000000..4d4753d --- /dev/null +++ b/.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-PLAN.md @@ -0,0 +1,525 @@ +--- +phase: quick-260521-fci +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - lib/services/ticket-reconciliation-service.ts + - app/api/sync/reconcile-tickets/route.ts + - lib/services/sync-scheduler.ts + - migrations/090_ticket_reconcile_schedule.sql +autonomous: true +requirements: + - QUICK-260521-FCI-01 +must_haves: + truths: + - "A nightly job scans the postgres tickets mirror for rows where is_deleted=false AND status<>5 AND synced_at < NOW() - INTERVAL '7 days' (capped at LIMIT 500)" + - "For each stale ticket id, the job fetches the current Autotask record via AutotaskClient.getEntityById('Tickets', id) and either upserts (using the existing mapAutotaskToDatabase + webhook upsert SQL) or soft-deletes (is_deleted=true, deleted_at=NOW()) when Autotask returns null" + - "The job returns { scanned, updated, statusFlippedToComplete, softDeleted, errors } and writes one summary log line per run" + - "POST /api/sync/reconcile-tickets triggers the job non-blocking (fire-and-forget), mirroring /api/sync/incremental" + - "A 'tickets-reconcile' default schedule (id=tickets-reconcile, cron='30 4 * * *') exists in sync_schedules and is dispatched via the scheduler's switch (disabled by default, like other defaults)" + - "Migration 090 idempotently seeds the schedule row with ON CONFLICT (id) DO NOTHING so it lands on a fresh DB without disturbing existing installs" + - "`npx tsc --noEmit --pretty` passes — type-check is the safety net (no tests for sync services)" + artifacts: + - path: "lib/services/ticket-reconciliation-service.ts" + provides: "reconcileStaleTickets() function returning ReconcileResult" + exports: ["reconcileStaleTickets", "ReconcileResult"] + - path: "app/api/sync/reconcile-tickets/route.ts" + provides: "POST handler — fire-and-forget trigger for reconcileStaleTickets()" + exports: ["POST"] + - path: "lib/services/sync-scheduler.ts" + provides: "Updated sync_type union + dispatch case + default schedule entry for tickets-reconcile" + contains: "tickets-reconcile" + - path: "migrations/090_ticket_reconcile_schedule.sql" + provides: "Idempotent INSERT for tickets-reconcile schedule row" + contains: "ON CONFLICT (id) DO NOTHING" + key_links: + - from: "lib/services/sync-scheduler.ts" + to: "lib/services/ticket-reconciliation-service.ts" + via: "dynamic import + direct call in executeScheduledSync switch (matches device-link-reconcile / integration-health pattern)" + pattern: "reconcileStaleTickets\\(" + - from: "app/api/sync/reconcile-tickets/route.ts" + to: "lib/services/ticket-reconciliation-service.ts" + via: "fire-and-forget call with .catch() — mirrors /api/sync/incremental" + pattern: "reconcileStaleTickets\\(\\)\\.catch" + - from: "lib/services/ticket-reconciliation-service.ts" + to: "lib/utils/entity-mapper.ts" + via: "mapAutotaskToDatabase(EntityType.TICKETS, entity) — reuse, do not parallel-implement" + pattern: "mapAutotaskToDatabase\\(EntityType\\.TICKETS" +--- + + +Build a STOPGAP nightly reconciliation job that detects "drift" between the +postgres `tickets` mirror and Autotask: tickets that have not been re-synced in +>7 days and are still marked open locally. For each, fetch the canonical +Autotask record by id and either re-upsert (status may have flipped to Complete +out-of-band, e.g. via a missed webhook) or soft-delete (Autotask no longer +returns the row). Wire it into the existing scheduler as a nightly cron. + +Purpose: The webhook can drop events (Autotask retries are limited, network +hiccups happen), and the incremental sync uses a high-water-mark filter that +will miss a ticket whose `lastActivityDate` never advances after a status +change. This job is a backstop — not a replacement for fixing the incremental +filter, but a way to bound staleness at 7 days while that fix is scoped +separately. + +Output: +- `lib/services/ticket-reconciliation-service.ts` (new) +- `app/api/sync/reconcile-tickets/route.ts` (new) +- Edits to `lib/services/sync-scheduler.ts` (union + default + dispatch) +- `migrations/090_ticket_reconcile_schedule.sql` (new) + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/STATE.md +@CLAUDE.md + + +@lib/services/webhook-service.ts +@lib/services/sync-scheduler.ts +@app/api/sync/incremental/route.ts +@lib/utils/entity-mapper.ts +@lib/utils/sync-helpers.ts +@lib/utils/sync-logger.ts +@lib/services/autotask-client.ts +@lib/services/rate-limiter.ts +@lib/services/postgres-client.ts +@lib/types/sync.ts +@middleware.ts + + + + +From `lib/services/autotask-client.ts:164`: +```typescript +async getEntityById(entityName: string, id: number): Promise +// Internally calls this.rateLimiter.throttle() (10 req/sec). For Tickets, pass 'Tickets'. +``` + +From `lib/utils/entity-mapper.ts`: +```typescript +export function mapAutotaskToDatabase( + entity: EntityType, + data: any +): Record +// For EntityType.TICKETS, runs the ticket-specific mapper and appends synced_at=new Date() +``` + +From `lib/utils/sync-helpers.ts`: +```typescript +export function getTableName(entity: EntityType): string +// getTableName(EntityType.TICKETS) -> 'tickets' +``` + +From `lib/utils/sync-logger.ts:250`: +```typescript +export function createSyncLogger(context?: LogContext): SyncLogger +// Use: const logger = createSyncLogger({ component: 'TicketReconciliation' }) +// Methods: logger.info(msg, ctx?), logger.warn(msg, ctx?, err?), logger.error(msg, ctx?, err?) +``` + +From `lib/services/rate-limiter.ts:114`: +```typescript +export const autotaskRateLimiter: RateLimiter +// Use: await autotaskRateLimiter.throttle(async () => { ... }) +``` + +From `lib/types/sync.ts`: +```typescript +export enum EntityType { TICKETS = 'tickets', ... } +``` + +From `lib/services/postgres-client.ts`: +```typescript +postgresClient.query(sql: string, params?: any[]): Promise> +// .rows: T[] +``` + +**Webhook upsert pattern to mirror exactly** (`lib/services/webhook-service.ts:201-234`): +```typescript +const mappedData = mapAutotaskToDatabase(internalEntityType, entityData); +const tableName = getTableName(internalEntityType); +const keys = Object.keys(mappedData); +const values = Object.values(mappedData); +const placeholders = keys.map((_, i) => `$${i + 1}`).join(', '); +const updateClause = keys + .filter(k => k !== 'id') + .map(k => `${k} = EXCLUDED.${k}`) + .join(', '); +const query = ` + INSERT INTO ${tableName} (${keys.join(', ')}) + VALUES (${placeholders}) + ON CONFLICT (id) + DO UPDATE SET ${updateClause}, updated_at = NOW() +`; +await postgresClient.query(query, values); +``` + +**Tickets table schema reminders:** +- `id` is `BIGINT` (Autotask id) +- Audit cols: `created_at`, `updated_at`, `synced_at`, `is_deleted`, `deleted_at` +- Status 5 = Complete in Autotask convention (see `ticket-digest-service.ts:193` precedent: `status NOT IN (5)`) + +**Existing dispatch precedent** (`sync-scheduler.ts:425-449`): +The `device-link-reconcile` and `integration-health` cases use dynamic `await import()` +and call the service directly (not via HTTP). The dispatch logs a one-line summary. +Mirror that pattern — do NOT POST to the API route from the scheduler. (The API +route exists for manual/admin triggering, the scheduler calls the service directly.) + +**Middleware note:** `/api/sync` is already in `publicRoutes` (middleware.ts:30), +so the new `/api/sync/reconcile-tickets` route is auto-allowed. No middleware +edit needed. + +**Default-schedule seeding caveat:** `createDefaultSchedules()` in +`sync-scheduler.ts` only inserts when the table is empty (line 173 check). +For existing DBs the in-code default won't fire — that's why migration 090 +seeds the row explicitly with `ON CONFLICT (id) DO NOTHING`. On a fresh DB +both paths are idempotent. + + + + + + + Task 1: Build reconciliation service + API route + + lib/services/ticket-reconciliation-service.ts, + app/api/sync/reconcile-tickets/route.ts + + +**Create `lib/services/ticket-reconciliation-service.ts`:** + +```typescript +/** + * Ticket Reconciliation Service (STOPGAP) + * + * Nightly backstop for drift between the postgres `tickets` mirror and Autotask. + * Webhooks can drop events and the incremental sync's high-water-mark filter + * can miss tickets whose lastActivityDate never advances after a status change. + * This job bounds local staleness at 7 days by re-fetching each stale-open + * ticket from Autotask and either re-upserting or soft-deleting. + * + * NOTE: This is a stopgap. The underlying incremental-filter issue is tracked + * separately and should fix the root cause. + */ + +import { postgresClient } from './postgres-client'; +import { AutotaskClient } from './autotask-client'; +import { autotaskRateLimiter } from './rate-limiter'; +import { mapAutotaskToDatabase } from '../utils/entity-mapper'; +import { getTableName } from '../utils/sync-helpers'; +import { EntityType } from '../types/sync'; +import { createSyncLogger } from '../utils/sync-logger'; + +export interface ReconcileResult { + scanned: number; + updated: number; + statusFlippedToComplete: number; + softDeleted: number; + errors: number; +} + +const STALE_DAYS = 7; +const SCAN_LIMIT = 500; +const COMPLETE_STATUS = 5; // Autotask status id for Complete + +let _client: AutotaskClient | null = null; +function getClient(): AutotaskClient { + if (!_client) { + _client = new AutotaskClient({ + apiUrl: process.env.AUTOTASK_API_URL || '', + username: process.env.AUTOTASK_USERNAME || '', + password: process.env.AUTOTASK_SECRET || '', + apiIntegrationCode: process.env.AUTOTASK_API_INTEGRATION_CODE || '', + }); + } + return _client; +} + +/** + * Re-fetch up to SCAN_LIMIT stale-open tickets from Autotask and reconcile + * the local mirror. Idempotent and safe to run alongside the regular sync. + */ +export async function reconcileStaleTickets(): Promise { + const logger = createSyncLogger({ component: 'TicketReconciliation' }); + const startedAt = Date.now(); + const result: ReconcileResult = { + scanned: 0, + updated: 0, + statusFlippedToComplete: 0, + softDeleted: 0, + errors: 0, + }; + + // 1. Find stale-open tickets in the postgres mirror. + const staleQuery = ` + SELECT id, status + FROM tickets + WHERE is_deleted = false + AND status <> $1 + AND synced_at < NOW() - INTERVAL '${STALE_DAYS} days' + ORDER BY synced_at ASC + LIMIT $2 + `; + const stale = await postgresClient.query<{ id: string; status: number | null }>( + staleQuery, + [COMPLETE_STATUS, SCAN_LIMIT] + ); + result.scanned = stale.rows.length; + + logger.info(`Scanning ${result.scanned} stale-open tickets (>${STALE_DAYS}d, limit=${SCAN_LIMIT})`); + + if (result.scanned === 0) { + logger.info('No stale tickets found', { duration: Date.now() - startedAt }); + return result; + } + + const client = getClient(); + const tableName = getTableName(EntityType.TICKETS); // 'tickets' + + // 2. Walk each id. Use the singleton rate limiter as an extra guard on top of + // AutotaskClient's internal limiter (defense in depth for parallel callers). + for (const row of stale.rows) { + const ticketId = Number(row.id); + const priorStatus = row.status; + try { + const entity = await autotaskRateLimiter.throttle(() => + client.getEntityById>('Tickets', ticketId) + ); + + if (!entity) { + // Not in Autotask anymore -> soft-delete locally. + await postgresClient.query( + `UPDATE ${tableName} + SET is_deleted = true, deleted_at = NOW(), updated_at = NOW() + WHERE id = $1`, + [ticketId] + ); + result.softDeleted += 1; + continue; + } + + // 3. Upsert using the same SQL shape the webhook handler uses. + const mapped = mapAutotaskToDatabase(EntityType.TICKETS, entity); + if (!mapped) { + throw new Error(`mapAutotaskToDatabase returned null for ticket ${ticketId}`); + } + const keys = Object.keys(mapped); + const values = Object.values(mapped); + const placeholders = keys.map((_, i) => `$${i + 1}`).join(', '); + const updateClause = keys + .filter(k => k !== 'id') + .map(k => `${k} = EXCLUDED.${k}`) + .join(', '); + const upsertSql = ` + INSERT INTO ${tableName} (${keys.join(', ')}) + VALUES (${placeholders}) + ON CONFLICT (id) + DO UPDATE SET ${updateClause}, updated_at = NOW() + `; + await postgresClient.query(upsertSql, values); + result.updated += 1; + + const newStatus = (mapped.status ?? null) as number | null; + if (priorStatus !== COMPLETE_STATUS && newStatus === COMPLETE_STATUS) { + result.statusFlippedToComplete += 1; + } + } catch (err) { + result.errors += 1; + logger.warn( + `Reconcile failed for ticket ${ticketId}`, + { ticketId }, + err instanceof Error ? err : new Error(String(err)) + ); + } + } + + logger.info( + `Reconciliation complete: scanned=${result.scanned} updated=${result.updated} flippedComplete=${result.statusFlippedToComplete} softDeleted=${result.softDeleted} errors=${result.errors}`, + { duration: Date.now() - startedAt } + ); + + return result; +} +``` + +**Create `app/api/sync/reconcile-tickets/route.ts`:** + +```typescript +/** + * Ticket Reconciliation API Endpoint + * POST /api/sync/reconcile-tickets - Trigger a stale-ticket reconciliation pass. + * Public per middleware.ts (matches /api/sync/incremental). Fire-and-forget. + */ + +import { NextRequest, NextResponse } from 'next/server'; +import { reconcileStaleTickets } from '@/lib/services/ticket-reconciliation-service'; + +export async function POST(request: NextRequest) { + try { + const body = await request.json().catch(() => ({})); + const triggeredBy = body.triggeredBy || 'api'; + + // Non-blocking — same pattern as /api/sync/incremental. + reconcileStaleTickets().catch((error) => { + console.error('[RECONCILE] Ticket reconciliation failed:', error); + }); + + return NextResponse.json({ + message: 'Ticket reconciliation started', + triggeredBy, + }); + } catch (error) { + console.error('Failed to start ticket reconciliation:', error); + return NextResponse.json( + { error: 'Failed to start ticket reconciliation' }, + { status: 500 } + ); + } +} +``` + +**Constraints reminders (CLAUDE.md):** +- No Zod validation in route. +- No `requireAuth()` — `/api/sync` is in middleware's `publicRoutes`; matches `incremental/route.ts`. +- `postgresClient.query()`, manual snake_case ↔ camelCase via `mapAutotaskToDatabase`. +- Errors: `try/catch` + `NextResponse.json({ error, message }, { status })`. 500 for runtime. +- kebab-case filenames (both are). Use `@/lib/...` and `@/lib/services/...` imports — no relative `../../`. +- Do NOT introduce a parallel mapping path. Do NOT touch `sync-helpers.ts`. + + + npx tsc --noEmit --pretty 2>&1 | grep -E "(ticket-reconciliation|reconcile-tickets)" ; echo "exit=$?" + + +- Both files exist at the listed paths. +- `npx tsc --noEmit --pretty` is clean for the two new files (no errors referencing + `ticket-reconciliation-service.ts` or `app/api/sync/reconcile-tickets/route.ts`). +- `reconcileStaleTickets()` exports a `ReconcileResult` shape matching + `{ scanned, updated, statusFlippedToComplete, softDeleted, errors }`. +- The route is structured exactly like `/api/sync/incremental` (no auth, body parse + with `.catch(() => ({}))`, fire-and-forget call with `.catch()`). +- Upsert SQL is the webhook-service pattern, not a new shape. + + + + + Task 2: Wire scheduler + migration + + lib/services/sync-scheduler.ts, + migrations/090_ticket_reconcile_schedule.sql + + +**Edit `lib/services/sync-scheduler.ts`:** + +1. **Extend the `sync_type` union** on the `ScheduleConfig` interface (line ~25). Append `| 'tickets-reconcile'` to the existing union literal: + +```typescript + sync_type: 'incremental' | 'full' | 'veeam-incremental' | 'veeam-full' | 'veeam-rpo-check' | 'contract-services' | 'engagement-daily' | 'zoom-daily' | 'morning-summary' | 'ticket-digest-daily' | 'ticket-digest-weekly' | 'ticket-digest-monthly' | 'device-link-reconcile' | 'integration-health' | 'qbo' | 'appgate-sessions' | 'appgate-daily' | 'tickets-reconcile'; +``` + +2. **Add a default schedule entry** inside `createDefaultSchedules()` `defaultSchedules` array (alongside the `integration-health` entry around line 286-293). Append: + +```typescript + { + id: 'tickets-reconcile', + name: 'Tickets Reconciliation', + description: 'STOPGAP backstop: re-fetches stale-open tickets (>7d since last sync) from Autotask and reconciles status / soft-deletes missing rows. Runs daily at 4:30 AM. Capped at 500 tickets per run.', + cron_expression: '30 4 * * *', + sync_type: 'tickets-reconcile', + is_enabled: false, + }, +``` + +(Match the other defaults: `is_enabled: false` so this seeds disabled and an admin enables it from `/admin`.) + +3. **Add a dispatch case** in `executeScheduledSync()`'s switch (the long `else if` chain starting at line ~397). Insert it next to `device-link-reconcile` / `integration-health` (anywhere in the chain before the trailing `incremental` / fallthrough cases). Use the dynamic-import pattern those two cases use: + +```typescript + } else if (config.sync_type === 'tickets-reconcile') { + const { reconcileStaleTickets } = await import('@/lib/services/ticket-reconciliation-service'); + const result = await reconcileStaleTickets(); + console.log( + `[SCHEDULER] tickets-reconcile: scanned=${result.scanned} updated=${result.updated} flippedComplete=${result.statusFlippedToComplete} softDeleted=${result.softDeleted} errors=${result.errors}` + ); +``` + +**Do NOT** change anything about `weekly-full`, the morning summary, or any other existing schedule. Do NOT modify `sync-helpers.ts`. + +**Create `migrations/090_ticket_reconcile_schedule.sql`:** + +```sql +-- Migration 090: Seed the tickets-reconcile sync schedule (STOPGAP). +-- +-- The sync_scheduler.createDefaultSchedules() path only seeds defaults on a +-- virgin sync_schedules table; this migration covers existing installs. +-- Idempotent via ON CONFLICT (id) DO NOTHING. + +INSERT INTO sync_schedules ( + id, + name, + description, + cron_expression, + sync_type, + years_back, + is_enabled +) VALUES ( + 'tickets-reconcile', + 'Tickets Reconciliation', + 'STOPGAP backstop: re-fetches stale-open tickets (>7d since last sync) from Autotask and reconciles status / soft-deletes missing rows. Runs daily at 4:30 AM. Capped at 500 tickets per run.', + '30 4 * * *', + 'tickets-reconcile', + NULL, + false +) +ON CONFLICT (id) DO NOTHING; +``` + +**Notes:** +- Schedule starts **disabled** (`is_enabled = false`) — admin enables from `/admin/sync-schedules` (or equivalent UI) when ready. Matches the convention of nearly every other default. +- Cron `30 4 * * *` = 4:30 AM daily, deliberately staggered from the 4:00 AM contract-services job to avoid Autotask rate-limit contention. +- The migration is idempotent: on a fresh DB the default-schedule seeder may insert first; on an existing DB this migration is the only insert path. Either way `ON CONFLICT DO NOTHING` makes the order irrelevant. +- No new migration numbers conflict — `ls migrations/` shows highest is `089_appgate_tables.sql`. + + + npx tsc --noEmit --pretty 2>&1 | grep -E "sync-scheduler" ; echo "exit=$?" + + +- `lib/services/sync-scheduler.ts` includes `'tickets-reconcile'` in the `sync_type` union, a new `defaultSchedules` entry with id `tickets-reconcile`, and a dispatch case calling `reconcileStaleTickets()` via dynamic import. +- `migrations/090_ticket_reconcile_schedule.sql` exists with `ON CONFLICT (id) DO NOTHING` and the same id/cron/description as the default entry. +- `npx tsc --noEmit --pretty` is clean for `sync-scheduler.ts` (no errors mentioning the file). +- Manual smoke check (optional, not required for done): grep confirms no leftover edits to `weekly-full` or `sync-helpers.ts`. + + + + + + +- `npx tsc --noEmit --pretty` exits clean across the whole repo (the project's only safety net for sync code). +- `grep -n "tickets-reconcile" lib/services/sync-scheduler.ts` shows three hits: union, default entry, dispatch case. +- `grep -n "reconcileStaleTickets" lib/services/` shows the export in the new service file and the import in the scheduler. +- `ls migrations/090_*.sql` returns the new file. +- Sanity-only manual check (do NOT execute as part of automated verify): a one-shot trigger via `curl -X POST http://localhost:3100/api/sync/reconcile-tickets -H 'Content-Type: application/json' -d '{"triggeredBy":"manual-smoke"}'` should return `{ "message": "Ticket reconciliation started", ... }` and the next container log line should be `[TicketReconciliation] Scanning N stale-open tickets...`. Document this in the SUMMARY, don't gate on it. + + + +- All four artifacts (service, route, scheduler edits, migration) exist and type-check. +- The job's behavioral contract holds: stale-open tickets are either updated or soft-deleted, never silently left behind. +- No CLAUDE.md violations: no ORM, no Zod-in-route, no parallel mapper, no auth on `/api/sync/*`, kebab-case filenames, `@/...` imports. +- Migration 090 is idempotent. The schedule lands disabled by default. +- Scope discipline: zero changes to `weekly-full`, `sync-helpers.ts`, or any non-listed file. + + + +After completion, create `.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-SUMMARY.md` with: +- Files created/modified (paths + 1-line role each) +- The `ReconcileResult` shape and where it's logged +- Cron + default-disabled note (for the eventual ops handoff) +- Any deviations from this plan and why +- Pointer to the underlying incremental-filter issue this stopgaps (link to existing tracking, or note "no separate tracker yet") + diff --git a/.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-SUMMARY.md b/.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-SUMMARY.md new file mode 100644 index 0000000..cb2c54c --- /dev/null +++ b/.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-SUMMARY.md @@ -0,0 +1,122 @@ +--- +phase: quick-260521-fci +plan: 01 +subsystem: sync +tags: [stopgap, autotask, reconciliation, scheduler] +requires: + - existing AutotaskClient.getEntityById + - existing mapAutotaskToDatabase / getTableName / autotaskRateLimiter + - existing sync_schedules table + sync-scheduler dispatch chain +provides: + - reconcileStaleTickets() — re-fetches stale-open tickets and reconciles + - POST /api/sync/reconcile-tickets — fire-and-forget trigger + - tickets-reconcile cron schedule (disabled by default, 30 4 * * *) +affects: + - lib/services/sync-scheduler.ts (sync_type union + default schedule + dispatch case) +tech_stack: + added: [] + patterns: + - dynamic-import-in-scheduler-dispatch + - webhook-style ON CONFLICT (id) DO UPDATE upsert + - rate-limited per-row Autotask getEntityById walk +key_files: + created: + - lib/services/ticket-reconciliation-service.ts + - app/api/sync/reconcile-tickets/route.ts + - migrations/090_ticket_reconcile_schedule.sql + modified: + - lib/services/sync-scheduler.ts +decisions: + - STOPGAP only — does NOT fix the underlying incremental-filter root cause + - LIMIT 500 per run + 10 req/sec rate limit caps worst-case API cost + - 4:30 AM cron deliberately staggered from 4:00 AM contract-services job + - Soft-delete (not hard-delete) when Autotask returns null — preserves audit + - Schedule lands DISABLED on both fresh + existing DBs; admin enables via UI +metrics: + duration_seconds: 129 + completed_at: 2026-05-21T15:09:45Z + tasks_completed: 2 + files_created: 3 + files_modified: 1 +--- + +# Quick Task 260521-fci: STOPGAP nightly reconciliation for stale tickets — Summary + +One-liner: A nightly cron walks postgres tickets that are still open locally but +haven't synced in >7 days, re-fetches each from Autotask via getEntityById, and +either upserts the canonical row or soft-deletes it — backstopping missed +webhooks and the incremental sync's high-water-mark blind spot. + +## Files Created / Modified + +| File | Role | +| ---- | ---- | +| `lib/services/ticket-reconciliation-service.ts` (created) | `reconcileStaleTickets()` — finds stale-open tickets, re-fetches from Autotask, upserts or soft-deletes. Returns `ReconcileResult`. | +| `app/api/sync/reconcile-tickets/route.ts` (created) | POST endpoint, fire-and-forget trigger mirroring `/api/sync/incremental` (no auth — public per existing `/api/sync` middleware allowlist). | +| `lib/services/sync-scheduler.ts` (modified) | Extended `sync_type` union with `'tickets-reconcile'`, added a `defaultSchedules` entry (disabled), and a dispatch case using the dynamic-import pattern matching `device-link-reconcile` / `integration-health`. | +| `migrations/090_ticket_reconcile_schedule.sql` (created) | Idempotent `INSERT … ON CONFLICT (id) DO NOTHING` so existing installs pick up the schedule row without disturbing the fresh-DB default-seed path. | + +## Result Shape + +`ReconcileResult` is logged once per run in two places: + +```ts +interface ReconcileResult { + scanned: number; // rows pulled from postgres for processing + updated: number; // successfully upserted from Autotask + statusFlippedToComplete: number; // priorStatus != 5, newStatus == 5 + softDeleted: number; // Autotask returned null -> is_deleted=true + errors: number; // per-row failures (logged, do not abort run) +} +``` + +Log lines: +1. **Service:** `[TicketReconciliation] Reconciliation complete: scanned=N updated=N flippedComplete=N softDeleted=N errors=N` + (via `createSyncLogger({ component: 'TicketReconciliation' })`) +2. **Scheduler:** `[SCHEDULER] tickets-reconcile: scanned=N updated=N flippedComplete=N softDeleted=N errors=N` + +## Operational Notes (handoff) + +- **Cron:** `30 4 * * *` (4:30 AM daily). Staggered 30 min after the 4:00 AM `contract-services` job to avoid rate-limit contention. +- **Default state:** `is_enabled = false`. Enable via the existing `/admin` sync-schedules UI when ready. +- **Capacity:** Hard-capped at `LIMIT 500` per run. With the 10 req/sec Autotask limiter that's ~50s minimum walltime if every ticket is stale; usually far less. +- **Manual trigger:** `curl -X POST http://localhost:3100/api/sync/reconcile-tickets -H 'Content-Type: application/json' -d '{"triggeredBy":"manual-smoke"}'` — returns `{ "message": "Ticket reconciliation started", "triggeredBy": "manual-smoke" }` immediately; result lines appear in container logs. +- **Idempotency:** Safe to run alongside the regular incremental sync — the upsert path uses the same SQL the webhook uses, and the soft-delete is column-level (no row deletion). +- **Failure mode:** Per-row exceptions increment `errors` and continue; only postgres infra failures abort the run. + +## Pointer to Underlying Issue + +This is a **STOPGAP**. The root cause is that Autotask's incremental sync uses +`lastTrackedModificationDateTime` (a.k.a. `lastActivityDate`-style high-water-mark) +which can fail to advance after some status transitions, so a status change to +Complete may never trigger an incremental row in our pipeline. The webhook path +also drops events on rare network/timeout edges. + +**No separate tracker exists yet** for the root cause. When that work is scoped +(likely as a "fix incremental filter for ticket status transitions" plan), this +job can either be retired or downgraded to a weekly safety net. + +## Deviations from Plan + +None — the plan was followed exactly as written. The only nuance was that the +union already contained `qbo | appgate-sessions | appgate-daily` from earlier +work in this worktree (pre-existing uncommitted changes), so the edit appended +`| 'tickets-reconcile'` to that already-extended union rather than the one +literally shown in the plan. Behavior is identical. + +## Verification + +- `npx tsc --noEmit --pretty` — clean (no output, full repo passes) +- `grep -n "tickets-reconcile" lib/services/sync-scheduler.ts` — three hits (union L25, default L295/L299, dispatch L458) +- `grep -rn "reconcileStaleTickets" lib/services/ app/api/sync/reconcile-tickets/` — one export, two imports (scheduler + route) +- `ls migrations/090_*.sql` — present +- Manual smoke (not run as part of automated verify, per plan): the curl POST above is the suggested ops handoff smoke test. + +## Self-Check: PASSED + +- `lib/services/ticket-reconciliation-service.ts` — FOUND +- `app/api/sync/reconcile-tickets/route.ts` — FOUND +- `migrations/090_ticket_reconcile_schedule.sql` — FOUND +- Commit `51f0b32` (Task 1: service + route) — FOUND +- Commit `badd718` (Task 2: scheduler + migration) — FOUND +- Full `npx tsc --noEmit --pretty` — clean