---
phase: quick-260521-fci
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - lib/services/ticket-reconciliation-service.ts
  - app/api/sync/reconcile-tickets/route.ts
  - lib/services/sync-scheduler.ts
  - migrations/090_ticket_reconcile_schedule.sql
autonomous: true
requirements:
  - QUICK-260521-FCI-01
must_haves:
  truths:
    - "A nightly job scans the postgres tickets mirror for rows where is_deleted=false AND status<>5 AND synced_at < NOW() - INTERVAL '7 days' (capped at LIMIT 500)"
    - "For each stale ticket id, the job fetches the current Autotask record via AutotaskClient.getEntityById('Tickets', id) and either upserts (using the existing mapAutotaskToDatabase + webhook upsert SQL) or soft-deletes (is_deleted=true, deleted_at=NOW()) when Autotask returns null"
    - "The job returns { scanned, updated, statusFlippedToComplete, softDeleted, errors } and writes one summary log line per run"
    - "POST /api/sync/reconcile-tickets triggers the job non-blocking (fire-and-forget), mirroring /api/sync/incremental"
    - "A 'tickets-reconcile' default schedule (id=tickets-reconcile, cron='30 4 * * *') exists in sync_schedules and is dispatched via the scheduler's switch (disabled by default, like other defaults)"
    - "Migration 090 idempotently seeds the schedule row with ON CONFLICT (id) DO NOTHING so it lands on a fresh DB without disturbing existing installs"
    - "`npx tsc --noEmit --pretty` passes — type-check is the safety net (no tests for sync services)"
  artifacts:
    - path: "lib/services/ticket-reconciliation-service.ts"
      provides: "reconcileStaleTickets() function returning ReconcileResult"
      exports: ["reconcileStaleTickets", "ReconcileResult"]
    - path: "app/api/sync/reconcile-tickets/route.ts"
      provides: "POST handler — fire-and-forget trigger for reconcileStaleTickets()"
      exports: ["POST"]
    - path: "lib/services/sync-scheduler.ts"
      provides: "Updated sync_type union + dispatch case + default schedule entry for tickets-reconcile"
      contains: "tickets-reconcile"
    - path: "migrations/090_ticket_reconcile_schedule.sql"
      provides: "Idempotent INSERT for tickets-reconcile schedule row"
      contains: "ON CONFLICT (id) DO NOTHING"
  key_links:
    - from: "lib/services/sync-scheduler.ts"
      to: "lib/services/ticket-reconciliation-service.ts"
      via: "dynamic import + direct call in executeScheduledSync switch (matches device-link-reconcile / integration-health pattern)"
      pattern: "reconcileStaleTickets\\("
    - from: "app/api/sync/reconcile-tickets/route.ts"
      to: "lib/services/ticket-reconciliation-service.ts"
      via: "fire-and-forget call with .catch() — mirrors /api/sync/incremental"
      pattern: "reconcileStaleTickets\\(\\)\\.catch"
    - from: "lib/services/ticket-reconciliation-service.ts"
      to: "lib/utils/entity-mapper.ts"
      via: "mapAutotaskToDatabase(EntityType.TICKETS, entity) — reuse, do not parallel-implement"
      pattern: "mapAutotaskToDatabase\\(EntityType\\.TICKETS"
---

<objective>
Build a STOPGAP nightly reconciliation job that detects "drift" between the
postgres `tickets` mirror and Autotask: tickets that have not been re-synced in
>7 days and are still marked open locally. For each, fetch the canonical
Autotask record by id and either re-upsert (status may have flipped to Complete
out-of-band, e.g. via a missed webhook) or soft-delete (Autotask no longer
returns the row). Wire it into the existing scheduler as a nightly cron.

Purpose: The webhook can drop events (Autotask retries are limited, network
hiccups happen), and the incremental sync uses a high-water-mark filter that
will miss a ticket whose `lastActivityDate` never advances after a status
change. This job is a backstop — not a replacement for fixing the incremental
filter, but a way to bound staleness at 7 days while that fix is scoped
separately.

Output:
- `lib/services/ticket-reconciliation-service.ts` (new)
- `app/api/sync/reconcile-tickets/route.ts` (new)
- Edits to `lib/services/sync-scheduler.ts` (union + default + dispatch)
- `migrations/090_ticket_reconcile_schedule.sql` (new)
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/STATE.md
@CLAUDE.md

<!-- Existing patterns the executor must mirror exactly -->
@lib/services/webhook-service.ts
@lib/services/sync-scheduler.ts
@app/api/sync/incremental/route.ts
@lib/utils/entity-mapper.ts
@lib/utils/sync-helpers.ts
@lib/utils/sync-logger.ts
@lib/services/autotask-client.ts
@lib/services/rate-limiter.ts
@lib/services/postgres-client.ts
@lib/types/sync.ts
@middleware.ts

<interfaces>
<!-- Key contracts extracted from the codebase. Executor uses these directly. -->

From `lib/services/autotask-client.ts:164`:
```typescript
async getEntityById<T>(entityName: string, id: number): Promise<T | null>
// Internally calls this.rateLimiter.throttle() (10 req/sec). For Tickets, pass 'Tickets'.
```

From `lib/utils/entity-mapper.ts`:
```typescript
export function mapAutotaskToDatabase(
  entity: EntityType,
  data: any
): Record<string, any>
// For EntityType.TICKETS, runs the ticket-specific mapper and appends synced_at=new Date()
```

From `lib/utils/sync-helpers.ts`:
```typescript
export function getTableName(entity: EntityType): string
// getTableName(EntityType.TICKETS) -> 'tickets'
```

From `lib/utils/sync-logger.ts:250`:
```typescript
export function createSyncLogger(context?: LogContext): SyncLogger
// Use: const logger = createSyncLogger({ component: 'TicketReconciliation' })
// Methods: logger.info(msg, ctx?), logger.warn(msg, ctx?, err?), logger.error(msg, ctx?, err?)
```

From `lib/services/rate-limiter.ts:114`:
```typescript
export const autotaskRateLimiter: RateLimiter
// Use: await autotaskRateLimiter.throttle(async () => { ... })
```

From `lib/types/sync.ts`:
```typescript
export enum EntityType { TICKETS = 'tickets', ... }
```

From `lib/services/postgres-client.ts`:
```typescript
postgresClient.query<T>(sql: string, params?: any[]): Promise<QueryResult<T>>
// .rows: T[]
```

**Webhook upsert pattern to mirror exactly** (`lib/services/webhook-service.ts:201-234`):
```typescript
const mappedData = mapAutotaskToDatabase(internalEntityType, entityData);
const tableName = getTableName(internalEntityType);
const keys = Object.keys(mappedData);
const values = Object.values(mappedData);
const placeholders = keys.map((_, i) => `$${i + 1}`).join(', ');
const updateClause = keys
  .filter(k => k !== 'id')
  .map(k => `${k} = EXCLUDED.${k}`)
  .join(', ');
const query = `
  INSERT INTO ${tableName} (${keys.join(', ')})
  VALUES (${placeholders})
  ON CONFLICT (id)
  DO UPDATE SET ${updateClause}, updated_at = NOW()
`;
await postgresClient.query(query, values);
```

**Tickets table schema reminders:**
- `id` is `BIGINT` (Autotask id)
- Audit cols: `created_at`, `updated_at`, `synced_at`, `is_deleted`, `deleted_at`
- Status 5 = Complete in Autotask convention (see `ticket-digest-service.ts:193` precedent: `status NOT IN (5)`)

**Existing dispatch precedent** (`sync-scheduler.ts:425-449`):
The `device-link-reconcile` and `integration-health` cases use dynamic `await import()`
and call the service directly (not via HTTP). The dispatch logs a one-line summary.
Mirror that pattern — do NOT POST to the API route from the scheduler. (The API
route exists for manual/admin triggering, the scheduler calls the service directly.)

**Middleware note:** `/api/sync` is already in `publicRoutes` (middleware.ts:30),
so the new `/api/sync/reconcile-tickets` route is auto-allowed. No middleware
edit needed.

**Default-schedule seeding caveat:** `createDefaultSchedules()` in
`sync-scheduler.ts` only inserts when the table is empty (line 173 check).
For existing DBs the in-code default won't fire — that's why migration 090
seeds the row explicitly with `ON CONFLICT (id) DO NOTHING`. On a fresh DB
both paths are idempotent.
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Build reconciliation service + API route</name>
  <files>
    lib/services/ticket-reconciliation-service.ts,
    app/api/sync/reconcile-tickets/route.ts
  </files>
  <action>
**Create `lib/services/ticket-reconciliation-service.ts`:**

```typescript
/**
 * Ticket Reconciliation Service (STOPGAP)
 *
 * Nightly backstop for drift between the postgres `tickets` mirror and Autotask.
 * Webhooks can drop events and the incremental sync's high-water-mark filter
 * can miss tickets whose lastActivityDate never advances after a status change.
 * This job bounds local staleness at 7 days by re-fetching each stale-open
 * ticket from Autotask and either re-upserting or soft-deleting.
 *
 * NOTE: This is a stopgap. The underlying incremental-filter issue is tracked
 * separately and should fix the root cause.
 */

import { postgresClient } from './postgres-client';
import { AutotaskClient } from './autotask-client';
import { autotaskRateLimiter } from './rate-limiter';
import { mapAutotaskToDatabase } from '../utils/entity-mapper';
import { getTableName } from '../utils/sync-helpers';
import { EntityType } from '../types/sync';
import { createSyncLogger } from '../utils/sync-logger';

export interface ReconcileResult {
  scanned: number;
  updated: number;
  statusFlippedToComplete: number;
  softDeleted: number;
  errors: number;
}

const STALE_DAYS = 7;
const SCAN_LIMIT = 500;
const COMPLETE_STATUS = 5; // Autotask status id for Complete

let _client: AutotaskClient | null = null;
function getClient(): AutotaskClient {
  if (!_client) {
    _client = new AutotaskClient({
      apiUrl: process.env.AUTOTASK_API_URL || '',
      username: process.env.AUTOTASK_USERNAME || '',
      password: process.env.AUTOTASK_SECRET || '',
      apiIntegrationCode: process.env.AUTOTASK_API_INTEGRATION_CODE || '',
    });
  }
  return _client;
}

/**
 * Re-fetch up to SCAN_LIMIT stale-open tickets from Autotask and reconcile
 * the local mirror. Idempotent and safe to run alongside the regular sync.
 */
export async function reconcileStaleTickets(): Promise<ReconcileResult> {
  const logger = createSyncLogger({ component: 'TicketReconciliation' });
  const startedAt = Date.now();
  const result: ReconcileResult = {
    scanned: 0,
    updated: 0,
    statusFlippedToComplete: 0,
    softDeleted: 0,
    errors: 0,
  };

  // 1. Find stale-open tickets in the postgres mirror.
  const staleQuery = `
    SELECT id, status
    FROM tickets
    WHERE is_deleted = false
      AND status <> $1
      AND synced_at < NOW() - INTERVAL '${STALE_DAYS} days'
    ORDER BY synced_at ASC
    LIMIT $2
  `;
  const stale = await postgresClient.query<{ id: string; status: number | null }>(
    staleQuery,
    [COMPLETE_STATUS, SCAN_LIMIT]
  );
  result.scanned = stale.rows.length;

  logger.info(`Scanning ${result.scanned} stale-open tickets (>${STALE_DAYS}d, limit=${SCAN_LIMIT})`);

  if (result.scanned === 0) {
    logger.info('No stale tickets found', { duration: Date.now() - startedAt });
    return result;
  }

  const client = getClient();
  const tableName = getTableName(EntityType.TICKETS); // 'tickets'

  // 2. Walk each id. Use the singleton rate limiter as an extra guard on top of
  //    AutotaskClient's internal limiter (defense in depth for parallel callers).
  for (const row of stale.rows) {
    const ticketId = Number(row.id);
    const priorStatus = row.status;
    try {
      const entity = await autotaskRateLimiter.throttle(() =>
        client.getEntityById<Record<string, any>>('Tickets', ticketId)
      );

      if (!entity) {
        // Not in Autotask anymore -> soft-delete locally.
        await postgresClient.query(
          `UPDATE ${tableName}
             SET is_deleted = true, deleted_at = NOW(), updated_at = NOW()
           WHERE id = $1`,
          [ticketId]
        );
        result.softDeleted += 1;
        continue;
      }

      // 3. Upsert using the same SQL shape the webhook handler uses.
      const mapped = mapAutotaskToDatabase(EntityType.TICKETS, entity);
      if (!mapped) {
        throw new Error(`mapAutotaskToDatabase returned null for ticket ${ticketId}`);
      }
      const keys = Object.keys(mapped);
      const values = Object.values(mapped);
      const placeholders = keys.map((_, i) => `$${i + 1}`).join(', ');
      const updateClause = keys
        .filter(k => k !== 'id')
        .map(k => `${k} = EXCLUDED.${k}`)
        .join(', ');
      const upsertSql = `
        INSERT INTO ${tableName} (${keys.join(', ')})
        VALUES (${placeholders})
        ON CONFLICT (id)
        DO UPDATE SET ${updateClause}, updated_at = NOW()
      `;
      await postgresClient.query(upsertSql, values);
      result.updated += 1;

      const newStatus = (mapped.status ?? null) as number | null;
      if (priorStatus !== COMPLETE_STATUS && newStatus === COMPLETE_STATUS) {
        result.statusFlippedToComplete += 1;
      }
    } catch (err) {
      result.errors += 1;
      logger.warn(
        `Reconcile failed for ticket ${ticketId}`,
        { ticketId },
        err instanceof Error ? err : new Error(String(err))
      );
    }
  }

  logger.info(
    `Reconciliation complete: scanned=${result.scanned} updated=${result.updated} flippedComplete=${result.statusFlippedToComplete} softDeleted=${result.softDeleted} errors=${result.errors}`,
    { duration: Date.now() - startedAt }
  );

  return result;
}
```

**Create `app/api/sync/reconcile-tickets/route.ts`:**

```typescript
/**
 * Ticket Reconciliation API Endpoint
 * POST /api/sync/reconcile-tickets - Trigger a stale-ticket reconciliation pass.
 * Public per middleware.ts (matches /api/sync/incremental). Fire-and-forget.
 */

import { NextRequest, NextResponse } from 'next/server';
import { reconcileStaleTickets } from '@/lib/services/ticket-reconciliation-service';

export async function POST(request: NextRequest) {
  try {
    const body = await request.json().catch(() => ({}));
    const triggeredBy = body.triggeredBy || 'api';

    // Non-blocking — same pattern as /api/sync/incremental.
    reconcileStaleTickets().catch((error) => {
      console.error('[RECONCILE] Ticket reconciliation failed:', error);
    });

    return NextResponse.json({
      message: 'Ticket reconciliation started',
      triggeredBy,
    });
  } catch (error) {
    console.error('Failed to start ticket reconciliation:', error);
    return NextResponse.json(
      { error: 'Failed to start ticket reconciliation' },
      { status: 500 }
    );
  }
}
```

**Constraints reminders (CLAUDE.md):**
- No Zod validation in route.
- No `requireAuth()` — `/api/sync` is in middleware's `publicRoutes`; matches `incremental/route.ts`.
- `postgresClient.query()`, manual snake_case ↔ camelCase via `mapAutotaskToDatabase`.
- Errors: `try/catch` + `NextResponse.json({ error, message }, { status })`. 500 for runtime.
- kebab-case filenames (both are). Use `@/lib/...` and `@/lib/services/...` imports — no relative `../../`.
- Do NOT introduce a parallel mapping path. Do NOT touch `sync-helpers.ts`.
  </action>
  <verify>
    <automated>npx tsc --noEmit --pretty 2>&amp;1 | grep -E "(ticket-reconciliation|reconcile-tickets)" ; echo "exit=$?"</automated>
  </verify>
  <done>
- Both files exist at the listed paths.
- `npx tsc --noEmit --pretty` is clean for the two new files (no errors referencing
  `ticket-reconciliation-service.ts` or `app/api/sync/reconcile-tickets/route.ts`).
- `reconcileStaleTickets()` exports a `ReconcileResult` shape matching
  `{ scanned, updated, statusFlippedToComplete, softDeleted, errors }`.
- The route is structured exactly like `/api/sync/incremental` (no auth, body parse
  with `.catch(() => ({}))`, fire-and-forget call with `.catch()`).
- Upsert SQL is the webhook-service pattern, not a new shape.
  </done>
</task>

<task type="auto">
  <name>Task 2: Wire scheduler + migration</name>
  <files>
    lib/services/sync-scheduler.ts,
    migrations/090_ticket_reconcile_schedule.sql
  </files>
  <action>
**Edit `lib/services/sync-scheduler.ts`:**

1. **Extend the `sync_type` union** on the `ScheduleConfig` interface (line ~25). Append `| 'tickets-reconcile'` to the existing union literal:

```typescript
  sync_type: 'incremental' | 'full' | 'veeam-incremental' | 'veeam-full' | 'veeam-rpo-check' | 'contract-services' | 'engagement-daily' | 'zoom-daily' | 'morning-summary' | 'ticket-digest-daily' | 'ticket-digest-weekly' | 'ticket-digest-monthly' | 'device-link-reconcile' | 'integration-health' | 'qbo' | 'appgate-sessions' | 'appgate-daily' | 'tickets-reconcile';
```

2. **Add a default schedule entry** inside `createDefaultSchedules()` `defaultSchedules` array (alongside the `integration-health` entry around line 286-293). Append:

```typescript
      {
        id: 'tickets-reconcile',
        name: 'Tickets Reconciliation',
        description: 'STOPGAP backstop: re-fetches stale-open tickets (>7d since last sync) from Autotask and reconciles status / soft-deletes missing rows. Runs daily at 4:30 AM. Capped at 500 tickets per run.',
        cron_expression: '30 4 * * *',
        sync_type: 'tickets-reconcile',
        is_enabled: false,
      },
```

(Match the other defaults: `is_enabled: false` so this seeds disabled and an admin enables it from `/admin`.)

3. **Add a dispatch case** in `executeScheduledSync()`'s switch (the long `else if` chain starting at line ~397). Insert it next to `device-link-reconcile` / `integration-health` (anywhere in the chain before the trailing `incremental` / fallthrough cases). Use the dynamic-import pattern those two cases use:

```typescript
      } else if (config.sync_type === 'tickets-reconcile') {
        const { reconcileStaleTickets } = await import('@/lib/services/ticket-reconciliation-service');
        const result = await reconcileStaleTickets();
        console.log(
          `[SCHEDULER] tickets-reconcile: scanned=${result.scanned} updated=${result.updated} flippedComplete=${result.statusFlippedToComplete} softDeleted=${result.softDeleted} errors=${result.errors}`
        );
```

**Do NOT** change anything about `weekly-full`, the morning summary, or any other existing schedule. Do NOT modify `sync-helpers.ts`.

**Create `migrations/090_ticket_reconcile_schedule.sql`:**

```sql
-- Migration 090: Seed the tickets-reconcile sync schedule (STOPGAP).
--
-- The sync_scheduler.createDefaultSchedules() path only seeds defaults on a
-- virgin sync_schedules table; this migration covers existing installs.
-- Idempotent via ON CONFLICT (id) DO NOTHING.

INSERT INTO sync_schedules (
  id,
  name,
  description,
  cron_expression,
  sync_type,
  years_back,
  is_enabled
) VALUES (
  'tickets-reconcile',
  'Tickets Reconciliation',
  'STOPGAP backstop: re-fetches stale-open tickets (>7d since last sync) from Autotask and reconciles status / soft-deletes missing rows. Runs daily at 4:30 AM. Capped at 500 tickets per run.',
  '30 4 * * *',
  'tickets-reconcile',
  NULL,
  false
)
ON CONFLICT (id) DO NOTHING;
```

**Notes:**
- Schedule starts **disabled** (`is_enabled = false`) — admin enables from `/admin/sync-schedules` (or equivalent UI) when ready. Matches the convention of nearly every other default.
- Cron `30 4 * * *` = 4:30 AM daily, deliberately staggered from the 4:00 AM contract-services job to avoid Autotask rate-limit contention.
- The migration is idempotent: on a fresh DB the default-schedule seeder may insert first; on an existing DB this migration is the only insert path. Either way `ON CONFLICT DO NOTHING` makes the order irrelevant.
- No new migration numbers conflict — `ls migrations/` shows highest is `089_appgate_tables.sql`.
  </action>
  <verify>
    <automated>npx tsc --noEmit --pretty 2>&amp;1 | grep -E "sync-scheduler" ; echo "exit=$?"</automated>
  </verify>
  <done>
- `lib/services/sync-scheduler.ts` includes `'tickets-reconcile'` in the `sync_type` union, a new `defaultSchedules` entry with id `tickets-reconcile`, and a dispatch case calling `reconcileStaleTickets()` via dynamic import.
- `migrations/090_ticket_reconcile_schedule.sql` exists with `ON CONFLICT (id) DO NOTHING` and the same id/cron/description as the default entry.
- `npx tsc --noEmit --pretty` is clean for `sync-scheduler.ts` (no errors mentioning the file).
- Manual smoke check (optional, not required for done): grep confirms no leftover edits to `weekly-full` or `sync-helpers.ts`.
  </done>
</task>

</tasks>

<verification>
- `npx tsc --noEmit --pretty` exits clean across the whole repo (the project's only safety net for sync code).
- `grep -n "tickets-reconcile" lib/services/sync-scheduler.ts` shows three hits: union, default entry, dispatch case.
- `grep -n "reconcileStaleTickets" lib/services/` shows the export in the new service file and the import in the scheduler.
- `ls migrations/090_*.sql` returns the new file.
- Sanity-only manual check (do NOT execute as part of automated verify): a one-shot trigger via `curl -X POST http://localhost:3100/api/sync/reconcile-tickets -H 'Content-Type: application/json' -d '{"triggeredBy":"manual-smoke"}'` should return `{ "message": "Ticket reconciliation started", ... }` and the next container log line should be `[TicketReconciliation] Scanning N stale-open tickets...`. Document this in the SUMMARY, don't gate on it.
</verification>

<success_criteria>
- All four artifacts (service, route, scheduler edits, migration) exist and type-check.
- The job's behavioral contract holds: stale-open tickets are either updated or soft-deleted, never silently left behind.
- No CLAUDE.md violations: no ORM, no Zod-in-route, no parallel mapper, no auth on `/api/sync/*`, kebab-case filenames, `@/...` imports.
- Migration 090 is idempotent. The schedule lands disabled by default.
- Scope discipline: zero changes to `weekly-full`, `sync-helpers.ts`, or any non-listed file.
</success_criteria>

<output>
After completion, create `.planning/quick/260521-fci-stopgap-nightly-reconciliation-for-stale/260521-fci-SUMMARY.md` with:
- Files created/modified (paths + 1-line role each)
- The `ReconcileResult` shape and where it's logged
- Cron + default-disabled note (for the eventual ops handoff)
- Any deviations from this plan and why
- Pointer to the underlying incremental-filter issue this stopgaps (link to existing tracking, or note "no separate tracker yet")
</output>
