---
phase: quick-260521-fci
plan: 01
subsystem: sync
tags: [stopgap, autotask, reconciliation, scheduler]
requires:
  - existing AutotaskClient.getEntityById
  - existing mapAutotaskToDatabase / getTableName / autotaskRateLimiter
  - existing sync_schedules table + sync-scheduler dispatch chain
provides:
  - reconcileStaleTickets() — re-fetches stale-open tickets and reconciles
  - POST /api/sync/reconcile-tickets — fire-and-forget trigger
  - tickets-reconcile cron schedule (disabled by default, 30 4 * * *)
affects:
  - lib/services/sync-scheduler.ts (sync_type union + default schedule + dispatch case)
tech_stack:
  added: []
  patterns:
    - dynamic-import-in-scheduler-dispatch
    - webhook-style ON CONFLICT (id) DO UPDATE upsert
    - rate-limited per-row Autotask getEntityById walk
key_files:
  created:
    - lib/services/ticket-reconciliation-service.ts
    - app/api/sync/reconcile-tickets/route.ts
    - migrations/090_ticket_reconcile_schedule.sql
  modified:
    - lib/services/sync-scheduler.ts
decisions:
  - STOPGAP only — does NOT fix the underlying incremental-filter root cause
  - LIMIT 500 per run + 10 req/sec rate limit caps worst-case API cost
  - 4:30 AM cron deliberately staggered from 4:00 AM contract-services job
  - Soft-delete (not hard-delete) when Autotask returns null — preserves audit
  - Schedule lands DISABLED on both fresh + existing DBs; admin enables via UI
metrics:
  duration_seconds: 129
  completed_at: 2026-05-21T15:09:45Z
  tasks_completed: 2
  files_created: 3
  files_modified: 1
---

# Quick Task 260521-fci: STOPGAP nightly reconciliation for stale tickets — Summary

One-liner: A nightly cron walks postgres tickets that are still open locally but
haven't synced in >7 days, re-fetches each from Autotask via getEntityById, and
either upserts the canonical row or soft-deletes it — backstopping missed
webhooks and the incremental sync's high-water-mark blind spot.

## Files Created / Modified

| File | Role |
| ---- | ---- |
| `lib/services/ticket-reconciliation-service.ts` (created) | `reconcileStaleTickets()` — finds stale-open tickets, re-fetches from Autotask, upserts or soft-deletes. Returns `ReconcileResult`. |
| `app/api/sync/reconcile-tickets/route.ts` (created) | POST endpoint, fire-and-forget trigger mirroring `/api/sync/incremental` (no auth — public per existing `/api/sync` middleware allowlist). |
| `lib/services/sync-scheduler.ts` (modified) | Extended `sync_type` union with `'tickets-reconcile'`, added a `defaultSchedules` entry (disabled), and a dispatch case using the dynamic-import pattern matching `device-link-reconcile` / `integration-health`. |
| `migrations/090_ticket_reconcile_schedule.sql` (created) | Idempotent `INSERT … ON CONFLICT (id) DO NOTHING` so existing installs pick up the schedule row without disturbing the fresh-DB default-seed path. |

## Result Shape

`ReconcileResult` is logged once per run in two places:

```ts
interface ReconcileResult {
  scanned: number;                  // rows pulled from postgres for processing
  updated: number;                  // successfully upserted from Autotask
  statusFlippedToComplete: number;  // priorStatus != 5, newStatus == 5
  softDeleted: number;              // Autotask returned null -> is_deleted=true
  errors: number;                   // per-row failures (logged, do not abort run)
}
```

Log lines:
1. **Service:** `[TicketReconciliation] Reconciliation complete: scanned=N updated=N flippedComplete=N softDeleted=N errors=N`
   (via `createSyncLogger({ component: 'TicketReconciliation' })`)
2. **Scheduler:** `[SCHEDULER] tickets-reconcile: scanned=N updated=N flippedComplete=N softDeleted=N errors=N`

## Operational Notes (handoff)

- **Cron:** `30 4 * * *` (4:30 AM daily). Staggered 30 min after the 4:00 AM `contract-services` job to avoid rate-limit contention.
- **Default state:** `is_enabled = false`. Enable via the existing `/admin` sync-schedules UI when ready.
- **Capacity:** Hard-capped at `LIMIT 500` per run. With the 10 req/sec Autotask limiter that's ~50s minimum walltime if every ticket is stale; usually far less.
- **Manual trigger:** `curl -X POST http://localhost:3100/api/sync/reconcile-tickets -H 'Content-Type: application/json' -d '{"triggeredBy":"manual-smoke"}'` — returns `{ "message": "Ticket reconciliation started", "triggeredBy": "manual-smoke" }` immediately; result lines appear in container logs.
- **Idempotency:** Safe to run alongside the regular incremental sync — the upsert path uses the same SQL the webhook uses, and the soft-delete is column-level (no row deletion).
- **Failure mode:** Per-row exceptions increment `errors` and continue; only postgres infra failures abort the run.

## Pointer to Underlying Issue

This is a **STOPGAP**. The root cause is that Autotask's incremental sync uses
`lastTrackedModificationDateTime` (a.k.a. `lastActivityDate`-style high-water-mark)
which can fail to advance after some status transitions, so a status change to
Complete may never trigger an incremental row in our pipeline. The webhook path
also drops events on rare network/timeout edges.

**No separate tracker exists yet** for the root cause. When that work is scoped
(likely as a "fix incremental filter for ticket status transitions" plan), this
job can either be retired or downgraded to a weekly safety net.

## Deviations from Plan

None — the plan was followed exactly as written. The only nuance was that the
union already contained `qbo | appgate-sessions | appgate-daily` from earlier
work in this worktree (pre-existing uncommitted changes), so the edit appended
`| 'tickets-reconcile'` to that already-extended union rather than the one
literally shown in the plan. Behavior is identical.

## Verification

- `npx tsc --noEmit --pretty` — clean (no output, full repo passes)
- `grep -n "tickets-reconcile" lib/services/sync-scheduler.ts` — three hits (union L25, default L295/L299, dispatch L458)
- `grep -rn "reconcileStaleTickets" lib/services/ app/api/sync/reconcile-tickets/` — one export, two imports (scheduler + route)
- `ls migrations/090_*.sql` — present
- Manual smoke (not run as part of automated verify, per plan): the curl POST above is the suggested ops handoff smoke test.

## Self-Check: PASSED

- `lib/services/ticket-reconciliation-service.ts` — FOUND
- `app/api/sync/reconcile-tickets/route.ts` — FOUND
- `migrations/090_ticket_reconcile_schedule.sql` — FOUND
- Commit `51f0b32` (Task 1: service + route) — FOUND
- Commit `badd718` (Task 2: scheduler + migration) — FOUND
- Full `npx tsc --noEmit --pretty` — clean
