# Sync Date Filter Bug Fix

## Critical Bug Fixed

**Issue**: Date-filtered syncs were incorrectly soft-deleting ALL records not in the fetched set, including records outside the sync window.

## Problem Description

### What Happened
When syncing entities with date filters (Tickets, Tasks, Time Entries, Projects, Contracts):
1. User syncs "Last 7 Days" of tickets
2. Sync fetches tickets from the last 7 days
3. Sync soft-deletes ALL tickets NOT in that 7-day window
4. **79,849 tickets were incorrectly deleted**

### Example
```sql
-- User syncs tickets for "Last 7 Days" (Nov 4, 2025 backwards)
-- Sync fetches: 1,655 tickets from Nov 4-Oct 28, 2025

-- BUG: Soft-deletes everything else
UPDATE tickets 
SET is_deleted = true, deleted_at = CURRENT_TIMESTAMP
WHERE id NOT IN (1655 recent ticket IDs)
  AND is_deleted = false;
-- Result: 79,849 tickets deleted (including ticket 621428 from Sept 3, 2025)
```

### Impact
- **Time entry enrichment failed** - Tickets referenced by time entries were deleted
- **Data loss appearance** - Historical tickets appeared deleted
- **Incorrect behavior** - Sync should mirror Autotask, not delete historical data

## Root Cause

### Code Analysis

**File**: `/opt/stacks/pulse/lib/services/entity-sync.ts`

**Problematic Logic**:
```typescript
// For full sync, soft delete records not in the fetched set
if (!isIncremental) {
  const activeIds = mappedRecords.map(r => r.id);
  deletedCount = await softDeleteMissingRecords(entity, activeIds);
  // ❌ This deletes ALL records not in activeIds
  // ❌ Doesn't consider date filters
}
```

**Soft Delete Function**:
```typescript
// lib/utils/db-helpers.ts
export async function softDeleteMissingRecords(
  entity: EntityType,
  activeIds: (number | string)[]
): Promise<number> {
  const query = `
    UPDATE ${tableName}
    SET is_deleted = true, deleted_at = CURRENT_TIMESTAMP
    WHERE id NOT IN (${activeIds.join(',')})
      AND is_deleted = false
  `;
  // ❌ No date range consideration
  // ❌ Deletes everything outside activeIds
}
```

### Design Flaw
The sync was designed for **full entity syncs** (e.g., all companies, all resources) where soft-deleting missing records makes sense. But it was incorrectly applied to **date-filtered syncs** where only a subset of data is fetched.

## Solution

### Code Fix

**File**: `/opt/stacks/pulse/lib/services/entity-sync.ts`

```typescript
// For full sync, soft delete records not in the fetched set
// IMPORTANT: Only delete for entities without date filters
let deletedCount = 0;
const hasDateFilter = entity === EntityType.TICKETS || 
                     entity === EntityType.TASKS || 
                     entity === EntityType.TIME_ENTRIES ||
                     entity === EntityType.PROJECTS ||
                     entity === EntityType.CONTRACTS;

if (!isIncremental && !hasDateFilter) {
  // ✅ Only soft-delete for entities that fetch ALL records
  const activeIds = mappedRecords.map(r => r.id);
  deletedCount = await softDeleteMissingRecords(entity, activeIds);
  console.log(`[${entity}] Soft deleted ${deletedCount} missing records`);
} else if (!isIncremental && hasDateFilter) {
  // ✅ Skip soft-delete for date-filtered entities
  console.log(`[${entity}] Skipping soft-delete for date-filtered sync`);
}
```

### Data Restoration

**Migration 009**: Restore incorrectly deleted tickets

```sql
-- Restore all soft-deleted tickets
UPDATE tickets
SET is_deleted = false, deleted_at = NULL
WHERE is_deleted = true;
-- Restored: 79,849 tickets
```

**Run**:
```bash
docker exec pulse-postgres psql -U pulse_user -d pulse_autotask \
  -c "UPDATE tickets SET is_deleted = false, deleted_at = NULL WHERE is_deleted = true;"
```

## Entities Affected

### Date-Filtered Entities (Fixed)
These entities now **skip soft-delete**:
- ✅ **Tickets** - Filtered by `createDate` (yearsBack parameter)
- ✅ **Tasks** - Filtered by date range
- ✅ **Time Entries** - Filtered by `dateWorked` (yearsBack parameter)
- ✅ **Projects** - Filtered by status and date
- ✅ **Contracts** - Filtered by status

### Full-Sync Entities (Unchanged)
These entities still **perform soft-delete** (correct behavior):
- ✅ **Companies** - Fetches all active companies
- ✅ **Resources** - Fetches all active resources
- ✅ **Contacts** - Fetches all contacts
- ✅ **Configuration Items** - Fetches all items
- ✅ **Billing Items** - Fetches all items

## Correct Sync Behavior

### Before Fix ❌
```
Sync "Last 7 Days" of Tickets:
1. Fetch 1,655 tickets from last 7 days
2. Upsert 1,655 tickets
3. Soft-delete 79,849 tickets NOT in the 7-day window ❌ WRONG
```

### After Fix ✅
```
Sync "Last 7 Days" of Tickets:
1. Fetch 1,655 tickets from last 7 days
2. Upsert 1,655 tickets
3. Skip soft-delete (date-filtered sync) ✅ CORRECT
```

### Full Sync (No Date Filter) ✅
```
Sync All Companies:
1. Fetch ALL companies from Autotask
2. Upsert companies
3. Soft-delete companies NOT in Autotask ✅ CORRECT
   (These are truly deleted in Autotask)
```

## Design Principle

**Sync Goal**: Mirror what's available via the Autotask API

### Rules
1. **Upsert fetched records** - Always update/insert what we fetch
2. **Never modify records outside sync window** - Don't touch data we didn't fetch
3. **Only soft-delete for full syncs** - Only when we fetch ALL records of an entity
4. **Respect date filters** - Date-filtered syncs are partial, not complete

### Examples

**✅ Correct**: Sync all companies, delete companies not in Autotask
- Fetches: ALL companies
- Deletes: Companies that don't exist in Autotask anymore

**❌ Incorrect**: Sync last 7 days of tickets, delete tickets older than 7 days
- Fetches: Only recent tickets
- Should NOT delete: Historical tickets outside the window

**✅ Correct**: Sync last 7 days of tickets, update only those tickets
- Fetches: Only recent tickets
- Updates: Only those tickets
- Leaves alone: All other tickets

## Testing

### Verify Fix

1. **Check ticket count before sync**:
```sql
SELECT COUNT(*) FROM tickets WHERE is_deleted = false;
-- Should be: ~81,414 tickets
```

2. **Sync last 7 days**:
```bash
# From UI: Select Tickets, set "Last 7 Days", sync
```

3. **Check ticket count after sync**:
```sql
SELECT COUNT(*) FROM tickets WHERE is_deleted = false;
-- Should be: ~81,414 tickets (unchanged except for updates)
```

4. **Verify no deletions**:
```sql
SELECT COUNT(*) FROM tickets WHERE is_deleted = true;
-- Should be: 0 (or only tickets truly deleted in Autotask)
```

### Test Enrichment

1. Navigate to `/admin/data-browser/time-entries`
2. Click "Enrich" button
3. Verify ticket numbers appear (e.g., T20250903.0081)
4. No missing ticket numbers for recent entries

## Future Improvements

### Option 1: Smart Soft-Delete (Advanced)
For date-filtered syncs, only soft-delete records **within the sync window** that weren't fetched:

```typescript
// Pseudo-code
if (hasDateFilter) {
  // Only delete records in the date range that weren't fetched
  const startDate = calculateStartDate(yearsBack);
  const endDate = new Date();
  
  const query = `
    UPDATE ${tableName}
    SET is_deleted = true
    WHERE create_date BETWEEN $1 AND $2
      AND id NOT IN (${activeIds})
      AND is_deleted = false
  `;
  await postgresClient.query(query, [startDate, endDate, ...activeIds]);
}
```

**Pros**: Detects deletions within sync window  
**Cons**: Complex, requires date column mapping per entity

### Option 2: Deletion Sync (Separate Process)
Create a separate sync that specifically checks for deleted records:

```typescript
// Fetch all IDs from Autotask (lightweight query)
const autotaskIds = await autotaskClient.queryIds(entity);

// Mark records not in Autotask as deleted
await softDeleteMissingRecords(entity, autotaskIds);
```

**Pros**: Accurate deletion detection  
**Cons**: Extra API calls, rate limiting concerns

### Option 3: Current Approach (Recommended)
Keep the current fix - skip soft-delete for date-filtered entities:

**Pros**: Simple, safe, prevents data loss  
**Cons**: Doesn't detect deletions (acceptable trade-off)

## Conclusion

**Fixed**: Date-filtered syncs no longer delete records outside the sync window  
**Restored**: 79,849 incorrectly deleted tickets  
**Principle**: Sync mirrors Autotask API, never modifies data outside sync scope  

The sync now correctly implements the principle: **"Mirror what's available via the API, never modify items outside the range of the sync."**

## Files Modified

- `/opt/stacks/pulse/lib/services/entity-sync.ts` - Added date filter check, skip soft-delete
- `/opt/stacks/pulse/migrations/009_restore_deleted_tickets.sql` - Restore deleted tickets
- `/opt/stacks/pulse/docs/SYNC_DATE_FILTER_FIX.md` - This documentation
