wulf-pulse/docs/SYNC_BEHAVIOR.md

# Sync Behavior Documentation

## Overview

This document provides comprehensive details about how the Autotask sync system works, including the differences between sync types, date range filtering, database operations, and entity-specific behaviors.

---

## Table of Contents

- [Sync Types](#sync-types)
- [Date Range Filtering](#date-range-filtering)
- [Database Operations](#database-operations)
- [Entity-Specific Behaviors](#entity-specific-behaviors)
- [Foreign Key Validation](#foreign-key-validation)
- [Soft Deletes](#soft-deletes)
- [Performance Considerations](#performance-considerations)

---

## Sync Types

### Full Sync

**What it does:**
- Fetches **ALL** records from Autotask that match the configured filters
- Performs an **UPSERT** operation on every fetched record
- Optionally performs **soft deletes** for missing records (entity-dependent)

**When to use:**
- Initial setup or first-time sync
- After major data changes in Autotask
- To ensure complete data consistency
- When you suspect data drift between systems

**Characteristics:**
- Processes all records within filter criteria (date range, status, etc.)
- Does NOT check if records have changed - upserts everything
- Slower than incremental sync
- More thorough and ensures data consistency

**Example:**
```
Full Sync of Tickets (last 2 years):
1. Fetch all tickets created in last 2 years from Autotask
2. Upsert all fetched tickets to PostgreSQL
3. Skip soft deletes (has date filter)
```

### Incremental Sync

**What it does:**
- Fetches only records **modified since the last successful sync**
- Uses `lastModifiedDate` or similar timestamp fields
- Performs UPSERT on changed records only
- Does NOT perform soft deletes

**When to use:**
- Regular scheduled syncs (hourly, daily)
- To keep data up-to-date with minimal overhead
- When you only need recent changes

**Characteristics:**
- Only processes records that changed since last sync
- Much faster than full sync
- Requires a previous successful sync to establish baseline
- Falls back to full sync if no previous sync found

**Important Limitations:**
- **Companies and Resources** do NOT support incremental sync
  - These entities automatically fall back to full sync
  - Autotask API does not support date-based filtering for these entities
  - This is by design and prevents sync failures

**Example:**
```
Incremental Sync of Tickets:
1. Get last sync time: 2026-01-24 08:00:00
2. Fetch tickets where lastActivityDate >= 2026-01-24 08:00:00
3. Upsert only the changed tickets
4. No soft deletes

Incremental Sync of Companies (automatic fallback):
1. Request incremental sync
2. System detects Companies doesn't support incremental
3. Automatically performs full sync instead
4. Logs: "companies does not support incremental sync, performing full sync instead"
```

### Sync Selected (Entity-Specific)

**What it does:**
- Syncs only the selected entities
- Uses full sync behavior for each selected entity
- Respects all entity-specific filters and behaviors

**When to use:**
- To sync specific entities without syncing everything
- After fixing entity-specific issues
- For testing or troubleshooting

---

## Date Range Filtering

### Entities with Required Filters

These entities **REQUIRE** filters - the Autotask API will return an error without them:

| Entity | Filter Field | Default Range | Reason |
|--------|-------------|---------------|---------|
| **Time Entries** | `dateWorked` | Last 2 years | API requirement |
| **Billing Items** | `itemDate` | Last 2 years | API requirement |
| **Contracts** | `status` | Active only (status=1) | API requirement |
| **Projects** | `status` | Non-completed (status≠5) | API requirement |

### Entities with Optional Date Filters

These entities have date filters applied for **performance** and **data management**:

| Entity | Filter Field | Default Range | Configurable |
|--------|-------------|---------------|--------------|
| **Tickets** | `createDate` | Last 2 years | Yes (via yearsBack) |
| **Tasks** | `createDateTime` | Last 2 years | Yes (via yearsBack) |

### Entities Without Date Filters

These entities sync **ALL** records (no date filtering):

- **Companies** - All active companies
- **Resources** - All active resources
- **Contacts** - All active contacts
- **Configuration Items** - All active items
- **Picklists** (Statuses, Issue Types, Sub-Issue Types, Work Types) - All values

### Configuring Date Range

The `yearsBack` parameter controls the date range:

```typescript
// Default: 2 years
syncService.fullSync(2);

// Custom: 5 years
syncService.fullSync(5);

// Via API
POST /api/sync/full
{
  "yearsBack": 3,
  "triggeredBy": "user@example.com"
}
```

**Important Notes:**

1. **Date filters apply to BOTH full and incremental sync**
   - Full sync: Fetches all records within date range
   - Incremental sync: Fetches changed records within date range

2. **Soft deletes are skipped for date-filtered entities**
   - Prevents deleting old records outside the sync window
   - See [Soft Deletes](#soft-deletes) section

---

## Database Operations

### UPSERT Behavior

All syncs use PostgreSQL's `ON CONFLICT DO UPDATE` (UPSERT):

```sql
INSERT INTO table_name (id, field1, field2, ...)
VALUES ($1, $2, $3, ...)
ON CONFLICT (id) 
DO UPDATE SET
  field1 = EXCLUDED.field1,
  field2 = EXCLUDED.field2,
  ...
  synced_at = NOW()
```

**What this means:**

- **If record exists**: All fields are updated with new values from Autotask
- **If record doesn't exist**: Record is inserted as new
- **Every fetched record is processed**, regardless of whether data actually changed
- The `synced_at` timestamp is always updated

### Record Counts

Sync history tracks three metrics:

| Metric | Description | How It's Calculated |
|--------|-------------|---------------------|
| `records_added` | New records inserted | **Estimated** (~10% of upserted) |
| `records_updated` | Existing records updated | **Estimated** (~90% of upserted) |
| `records_deleted` | Records soft-deleted | **Actual count** from soft delete operation |

**Important:** `records_added` and `records_updated` are **estimates** based on the total upserted count. PostgreSQL doesn't easily distinguish between inserts and updates in bulk UPSERT operations.

### Bulk Processing

Records are processed in batches for performance:

- **Fetch batch size**: 500 records per API call
- **Upsert batch size**: 100 records per database transaction
- **Pagination**: Automatic for large datasets

---

## Entity-Specific Behaviors

### Picklist Entities

**Entities:** Statuses, Issue Types, Sub-Issue Types, Work Types

**Special Handling:**
- Fetched from `/entity/entityInformation/fields` endpoint (not `/query`)
- No date filtering
- No soft deletes
- Synced as key-value pairs

**Example:**
```typescript
// Statuses from Tickets.status field
GET /Tickets/entityInformation/fields

// Maps to:
{ value: 1, label: "New", is_active: true }
{ value: 5, label: "Complete", is_active: true }
```

### Tickets

**Filters:**
- Date: `createDate >= (now - 2 years)`
- Active: `isActive = true` (if applicable)

**Foreign Key Validation:**
- `assigned_resource_id` → Resources
- `first_response_assigned_resource_id` → Resources
- `first_response_initiating_resource_id` → Resources
- Invalid references are set to `null`

**Soft Deletes:** ❌ Skipped (has date filter)

### Tasks

**Filters:**
- Date: `createDateTime >= (now - 2 years)`

**Foreign Key Validation:**
- `project_id` → Projects
- `assigned_resource_id` → Resources
- `creator_resource_id` → Resources
- `completed_by_resource_id` → Resources
- `last_activity_resource_id` → Resources
- Invalid references are set to `null`

**Soft Deletes:** ❌ Skipped (has date filter)

### Time Entries

**Filters:**
- **Required**: `dateWorked >= (now - 2 years)`

**Special Notes:**
- `company_id` is nullable (time entries can exist without a company)
- No foreign key validation needed

**Soft Deletes:** ❌ Skipped (has date filter)

### Billing Items

**Filters:**
- **Required**: `itemDate >= (now - 2 years)`

**Special Notes:**
- API requires a filter parameter
- Returns 500 error without filter

**Soft Deletes:** ❌ Skipped (has date filter)

### Companies

**Filters:**
- Active: `isActive = true`

**Special Notes:**
- No date filtering - syncs all active companies
- Foundation entity for many foreign keys

**Soft Deletes:** ✅ Yes (no date filter)

### Resources

**Filters:**
- Active: `isActive = true`

**Special Notes:**
- No date filtering - syncs all active resources
- Referenced by Tickets, Tasks, Time Entries

**Soft Deletes:** ✅ Yes (no date filter)

### Contacts

**Filters:**
- Active: `isActive = true`

**Special Notes:**
- No date filtering - syncs all active contacts
- Referenced by Configuration Items

**Soft Deletes:** ✅ Yes (no date filter)

### Configuration Items

**Filters:**
- Active: `isActive = true`

**Foreign Key Validation:**
- `contact_id` → Contacts
- Invalid references are set to `null`

**Soft Deletes:** ✅ Yes (no date filter)

### Contracts

**Filters:**
- **Required**: `status = 1` (Active)

**Special Notes:**
- API requires status filter
- Only syncs active contracts

**Soft Deletes:** ❌ Skipped (has status filter)

### Projects

**Filters:**
- **Required**: `status != 5` (Not Complete)

**Special Notes:**
- API requires status filter
- Excludes completed projects

**Soft Deletes:** ❌ Skipped (has status filter)

---

## Foreign Key Validation

### Why It's Needed

Autotask data may reference entities that:
- Haven't been synced yet
- Are outside the sync date range
- Have been deleted in Autotask
- Don't exist due to data inconsistencies

Without validation, these references cause PostgreSQL foreign key constraint violations.

### How It Works

Before upserting records, the sync validates foreign key references:

1. **Fetch valid IDs** from the database
   ```typescript
   const validResourceIds = await getValidResourceIds();
   // Returns Set of all resource IDs where is_deleted = false
   ```

2. **Validate each record**
   ```typescript
   if (task.assigned_resource_id && !validResourceIds.has(task.assigned_resource_id)) {
     task.assigned_resource_id = null; // Nullify invalid reference
   }
   ```

3. **Log warnings** for nullified references
   ```
   [WARN] Nullified invalid resource references: 15 records
   ```

### Validated Relationships

| Entity | Foreign Key Field | References | Action on Invalid |
|--------|------------------|------------|-------------------|
| **Tickets** | `assigned_resource_id` | Resources | Set to null |
| **Tickets** | `first_response_assigned_resource_id` | Resources | Set to null |
| **Tickets** | `first_response_initiating_resource_id` | Resources | Set to null |
| **Tasks** | `project_id` | Projects | Set to null |
| **Tasks** | `assigned_resource_id` | Resources | Set to null |
| **Tasks** | `creator_resource_id` | Resources | Set to null |
| **Tasks** | `completed_by_resource_id` | Resources | Set to null |
| **Tasks** | `last_activity_resource_id` | Resources | Set to null |
| **Configuration Items** | `contact_id` | Contacts | Set to null |

### Performance Impact

Foreign key validation queries are:
- Executed once per sync (cached)
- Indexed queries (fast)
- Minimal overhead compared to API fetching

---

## Soft Deletes

### What Are Soft Deletes?

Instead of permanently deleting records, the system marks them as deleted:

```sql
UPDATE table_name 
SET is_deleted = true, deleted_at = NOW()
WHERE id NOT IN (fetched_ids)
  AND is_deleted = false
```

### When Soft Deletes Occur

**Only during Full Sync** for entities **without date filters**:

| Entity | Soft Deletes? | Reason |
|--------|--------------|---------|
| Companies | ✅ Yes | No date filter - safe to delete |
| Resources | ✅ Yes | No date filter - safe to delete |
| Contacts | ✅ Yes | No date filter - safe to delete |
| Configuration Items | ✅ Yes | No date filter - safe to delete |
| Tickets | ❌ No | Has date filter - would delete old records |
| Tasks | ❌ No | Has date filter - would delete old records |
| Time Entries | ❌ No | Has date filter - would delete old records |
| Billing Items | ❌ No | Has date filter - would delete old records |
| Projects | ❌ No | Has status filter - would delete completed |
| Contracts | ❌ No | Has status filter - would delete inactive |
| Picklists | ❌ No | Picklist values don't get deleted |

### Why Skip Soft Deletes for Date-Filtered Entities?

**Problem:**
If we soft-deleted records not in the fetched set for date-filtered entities, we would incorrectly delete old records outside the sync window.

**Example:**
```
Full Sync of Tickets (last 2 years):
1. Fetch tickets created >= 2024-01-24
2. Ticket #12345 created in 2020 exists in DB
3. Ticket #12345 NOT in fetched set (outside date range)
4. ❌ Would incorrectly soft-delete Ticket #12345

Solution: Skip soft deletes for date-filtered entities
```

### Restoring Soft-Deleted Records

If a record is soft-deleted but appears in a later sync:

```sql
-- Record is automatically restored
UPDATE table_name
SET is_deleted = false, deleted_at = NULL, ...
WHERE id = $1
```

---

## Performance Considerations

### Sync Duration Factors

| Factor | Impact | Mitigation |
|--------|--------|------------|
| **Number of records** | High | Use date range filters |
| **API rate limits** | Medium | Built-in rate limiting (10 req/sec) |
| **Network latency** | Medium | Pagination, batch processing |
| **Database operations** | Low | Bulk upserts, indexed queries |
| **Foreign key validation** | Low | Cached, indexed queries |

### Estimated Sync Times

Based on typical data volumes:

| Entity | Record Count | Full Sync Time | Incremental Sync Time |
|--------|-------------|----------------|----------------------|
| Companies | 500 | ~30 seconds | ~5 seconds |
| Resources | 100 | ~10 seconds | ~2 seconds |
| Tickets (2 years) | 10,000 | ~10 minutes | ~1 minute |
| Tasks (2 years) | 5,000 | ~5 minutes | ~30 seconds |
| Time Entries (2 years) | 50,000 | ~45 minutes | ~5 minutes |
| Billing Items (2 years) | 50,000+ | ~45-60 minutes | ~5 minutes |

**Note:** Times vary based on data volume, network speed, and system load.

### Optimization Tips

1. **Use Incremental Sync for regular updates**
   - Schedule incremental syncs hourly or daily
   - Reserve full syncs for weekly/monthly maintenance

2. **Adjust date range based on needs**
   - Default 2 years is usually sufficient
   - Reduce to 1 year for faster syncs if older data isn't needed
   - Increase to 5+ years only if historical data is required

3. **Sync entities in dependency order**
   - Companies → Resources → Tickets → Tasks
   - Reduces foreign key validation overhead

4. **Monitor sync history**
   - Check for failures and patterns
   - Identify slow entities for optimization

5. **Use Sync Selected for troubleshooting**
   - Test individual entities
   - Isolate and fix issues without full sync

---

## Sync Workflow Examples

### Example 1: Initial Setup

```
1. Full Sync - All Entities (2 years)
   ├─ Companies (all active)
   ├─ Resources (all active)
   ├─ Contacts (all active)
   ├─ Configuration Items (all active)
   ├─ Contracts (active only)
   ├─ Projects (non-completed)
   ├─ Tickets (last 2 years)
   ├─ Tasks (last 2 years)
   ├─ Time Entries (last 2 years)
   ├─ Billing Items (last 2 years)
   └─ Picklists (all values)

Result: Complete database populated with all relevant data
```

### Example 2: Daily Maintenance

```
1. Incremental Sync - All Entities
   ├─ Fetch only records modified since last sync
   ├─ Upsert changed records
   └─ No soft deletes

Result: Database updated with latest changes (fast)
```

### Example 3: Fixing Specific Entity

```
1. Identify failed entity: Billing Items
2. Sync Selected - Billing Items only
   ├─ Full sync with date filter (last 2 years)
   ├─ Upsert all billing items
   └─ No soft deletes

Result: Billing Items data refreshed and consistent
```

### Example 4: Historical Data Sync

```
1. Full Sync - Tickets (5 years)
   ├─ Adjust yearsBack parameter to 5
   ├─ Fetch tickets created >= 2021-01-24
   ├─ Upsert all fetched tickets
   └─ Skip soft deletes

Result: 5 years of ticket history in database
```

---

## Troubleshooting

### Common Issues

#### Issue: Foreign Key Constraint Violations

**Symptoms:**
```
[DATABASE_CONSTRAINT_ERROR] violates foreign key constraint "tasks_project_id_fkey"
```

**Cause:** Referenced entity hasn't been synced or is outside date range

**Solution:**
1. Sync dependency entities first (Companies → Resources → Projects)
2. Foreign key validation will nullify invalid references
3. Check logs for nullified reference warnings

#### Issue: API Filter Required Error

**Symptoms:**
```
[API_ERROR] Value cannot be null. Parameter name: filters
```

**Cause:** Entity requires a filter (Time Entries, Billing Items, Contracts, Projects)

**Solution:** Filters are automatically applied - this shouldn't occur. If it does, check entity-sync.ts for proper filter configuration.

#### Issue: Slow Sync Performance

**Symptoms:** Sync takes longer than expected

**Solutions:**
1. Reduce `yearsBack` parameter (e.g., from 2 to 1 year)
2. Use incremental sync instead of full sync
3. Sync entities individually instead of all at once
4. Check network connectivity and API response times

#### Issue: Records Not Appearing

**Symptoms:** Records exist in Autotask but not in database

**Possible Causes:**
1. Outside date range filter
2. Not active (filtered out by `isActive`)
3. Foreign key validation nullified references
4. Sync failed (check sync history)

**Solutions:**
1. Check sync history for errors
2. Verify record meets filter criteria
3. Check logs for validation warnings
4. Run full sync for the entity

---

## Best Practices

### 1. Sync Strategy

- **Initial Setup:** Full sync all entities (2 years)
- **Daily Maintenance:** Incremental sync all entities
- **Weekly/Monthly:** Full sync for data consistency
- **Troubleshooting:** Sync selected entities as needed

### 2. Date Range Configuration

- **Default (2 years):** Good for most use cases
- **1 year:** Faster syncs, less historical data
- **3-5 years:** More historical data, slower syncs
- **Adjust based on:** Business needs, performance requirements

### 3. Monitoring

- Review sync history regularly
- Set up alerts for failed syncs
- Monitor sync duration trends
- Check for foreign key validation warnings

### 4. Dependency Management

Sync entities in this order for best results:
1. Companies
2. Resources
3. Contacts
4. Configuration Items
5. Contracts
6. Projects
7. Tickets
8. Tasks
9. Time Entries
10. Billing Items
11. Picklists

### 5. Error Handling

- Failed syncs are logged in `sync_history` table
- Check error messages for specific issues
- Re-run failed entity syncs individually
- Contact support if errors persist

---

## API Reference

### Full Sync

```typescript
POST /api/sync/full
{
  "yearsBack": 2,
  "triggeredBy": "user@example.com"
}
```

### Incremental Sync

```typescript
POST /api/sync/incremental
{
  "triggeredBy": "user@example.com"
}
```

### Sync Selected Entities

```typescript
POST /api/sync/entity
{
  "entities": ["tickets", "tasks"],
  "yearsBack": 2,
  "triggeredBy": "user@example.com"
}
```

### Get Sync History

```typescript
GET /api/sync/history?limit=50&offset=0
```

### Get Sync Status

```typescript
GET /api/sync/status
```

---

## Related Documentation

- [Sync Interface Guide](./SYNC_INTERFACE_GUIDE.md) - User guide for the sync UI
- [Sync Logging Improvements](./SYNC_LOGGING_IMPROVEMENTS.md) - Structured logging details
- [Database Schema](../migrations/001_initial_schema.sql) - PostgreSQL table definitions

---

## Changelog

### 2026-01-24
- Added billing items filter requirement
- Added task foreign key validation
- Documented sync behavior comprehensively

### 2026-01-23
- Fixed statuses and work types sync (picklist API)
- Added structured logging
- Created sync interface documentation

---

## Support

For issues or questions:
1. Check sync history in the admin interface
2. Review error messages and logs
3. Consult this documentation
4. Contact system administrator