wulf-pulse/docs/SYNC_INTERFACE_GUIDE.md

# Sync Interface User Guide

## Overview

The Sync Interface allows you to synchronize data from Autotask to your PostgreSQL database. This guide explains each sync option, when to use it, and what to expect.

## Sync Interface Location

Access the sync interface at: **Admin Dashboard → Sync Controls**

---

## Sync Options

### 1. **Full Sync** 🔄

**What it does:**
- Syncs **all entities** from Autotask to PostgreSQL
- Fetches all active records for each entity type
- **Soft-deletes** records that exist in your database but are no longer in Autotask
- Respects the date range filter for time-based entities (Tickets, Tasks, Projects)

**When to use:**
- **Initial setup** - First time syncing your Autotask data
- **Data recovery** - After database issues or data corruption
- **Periodic refresh** - Monthly or quarterly to ensure data consistency
- **After major Autotask changes** - When records have been bulk-deleted or modified in Autotask

**Important notes:**
- ⚠️ **Requires confirmation** - Shows a warning dialog before starting
- ⏱️ **Takes longest** - Can take 10-30 minutes depending on data volume
- 🗑️ **Soft-deletes missing records** - Sets `is_deleted = true` for records not found in Autotask
- 📅 **Respects date range** - For Tickets/Tasks/Projects, only syncs within selected date range

**Example use case:**
> "It's the first Monday of the month. I want to ensure all our data is fresh and any deleted records in Autotask are marked as deleted in our database."

**What gets synced (in order):**
1. Companies
2. Resources (Users)
3. Statuses
4. Issue Types
5. Sub-Issue Types
6. Work Types
7. Contacts
8. Projects
9. Tickets (within date range)
10. Tasks (within date range)
11. Configuration Items
12. Contracts
13. Billing Items
14. Time Entries (within date range)

---

### 2. **Incremental Sync** ⚡

**What it does:**
- Syncs **only records modified since the last sync**
- Uses Autotask's `lastModifiedDate` field to filter records
- Much faster than Full Sync
- Does **NOT** soft-delete missing records

**When to use:**
- **Daily updates** - Regular scheduled syncs to keep data current
- **Quick refresh** - After making changes in Autotask
- **Automated syncs** - Set up on a cron schedule (e.g., every 4 hours)
- **Low-impact updates** - When you need fresh data without heavy processing

**Important notes:**
- ✅ **No confirmation needed** - Starts immediately
- ⚡ **Fast** - Typically completes in 1-5 minutes
- 🔄 **Additive only** - Adds/updates records but doesn't delete
- 📊 **Best for routine updates** - Ideal for keeping data fresh

**Example use case:**
> "A technician just closed several tickets in Autotask. I want to pull those updates into our dashboard without running a full sync."

**How it works:**
1. Checks `sync_history` table for last successful sync of each entity
2. Queries Autotask for records where `lastModifiedDate > last_sync_time`
3. Updates existing records or inserts new ones
4. Skips entities that have no modifications

---

### 3. **Chunked Tickets** 📦

**What it does:**
- Syncs **only Tickets** in monthly date chunks
- Breaks large date ranges into manageable pieces
- Prevents API timeouts and memory issues
- Shows real-time progress for each chunk

**When to use:**
- **Large date ranges** - Syncing 3+ years of ticket history
- **Initial ticket sync** - First time pulling historical tickets
- **Timeout prevention** - When regular ticket sync fails with timeouts
- **Off-hours processing** - Running overnight syncs for historical data

**Important notes:**
- 📊 **Progress tracking** - Shows which month is currently syncing
- 🔄 **Resilient** - Continues even if individual chunks fail
- 🎯 **Tickets only** - Does not sync other entities
- ⏱️ **Slower but reliable** - Takes longer but handles large datasets

**Example use case:**
> "I need to sync 5 years of ticket history. A regular sync times out, so I'll use Chunked Tickets to process it month by month."

**How it works:**
1. Calculates monthly chunks based on selected date range
2. Processes each month sequentially (e.g., "Jan 2024", "Feb 2024", etc.)
3. Shows progress: "Processing chunk 15/60: Mar 2023"
4. Continues even if some months fail (logs failures)
5. Displays final summary with total records processed

**Date range examples:**
- Last 7 Days → ~1 chunk
- Last 2 Years → ~24 chunks
- Last 5 Years → ~60 chunks
- All Time → Could be 100+ chunks

---

### 4. **Sync Selected** 🎯

**What it does:**
- Syncs **only the entities you select** from the entity list
- Full sync for selected entities (not incremental)
- Respects date range for time-based entities
- Does **NOT** soft-delete missing records

**When to use:**
- **Targeted updates** - Only need to refresh specific entities
- **Dependency sync** - Sync Companies before syncing Tickets
- **Troubleshooting** - Re-sync a specific entity that had errors
- **Testing** - Verify sync works for a single entity

**Important notes:**
- ✅ **Requires entity selection** - Must select at least one entity
- 🎯 **Selective** - Only syncs what you choose
- ⚠️ **Respect dependencies** - Ensure parent entities are synced first
- 📅 **Uses date range** - For Tickets/Tasks/Projects

**Example use case:**
> "We just imported 50 new companies into Autotask. I only need to sync Companies and Contacts, not everything else."

**Entity dependencies to consider:**
- **Contacts** → Requires Companies
- **Tickets** → Requires Companies, Resources, Contacts
- **Tasks** → Requires Resources, Projects, Tickets
- **Configuration Items** → Requires Companies, Contacts
- **Time Entries** → Requires Companies, Resources, Projects, Tasks, Tickets

**How to use:**
1. Click entity checkboxes to select (e.g., Companies, Contacts)
2. Set date range if syncing time-based entities
3. Click "Sync Selected (2)" button
4. Monitor progress in sync history


---

## Date Range Selector 📅

**Applies to:** Tickets, Tasks, Projects, Time Entries

**Available ranges:**
- **Last 7 Days** (0.019 years) - Quick updates, minimal data
- **Last 30 Days** (0.082 years) - Monthly refresh
- **Last 90 Days** (0.25 years) - Quarterly data
- **Last 1 Year** - Recent history
- **Last 2 Years** ⭐ **Recommended** - Good balance of history and performance
- **Last 3 Years** - Extended history
- **Last 5 Years** - Long-term analysis
- **Last 10 Years** - Comprehensive history
- **All Time** ⚠️ - Complete history (very slow, use off-hours)

**Why limit date ranges?**
1. **Performance** - Fewer records = faster sync
2. **API limits** - Autotask has rate limits and timeout thresholds
3. **Relevance** - Most reporting focuses on recent data
4. **Resource usage** - Large syncs consume more memory and CPU

**Recommendation:**
- **Daily syncs:** Last 7-30 Days
- **Weekly syncs:** Last 90 Days
- **Monthly syncs:** Last 1-2 Years
- **Initial setup:** Last 2 Years, then run "All Time" overnight if needed

---

## Entity Selector

**Available entities:**

| Entity | Description | Dependencies |
|--------|-------------|--------------|
| **Companies** | Customer/vendor organizations | None |
| **Resources** | Autotask users/technicians | None |
| **Statuses** | Ticket/task statuses | None |
| **Issue Types** | Ticket categorization | None |
| **Sub-Issue Types** | Ticket sub-categories | None |
| **Work Types** | Time entry work types | None |
| **Contacts** | People at companies | Companies |
| **Projects** | Customer projects | Companies, Resources |
| **Tickets** | Support tickets | Companies, Resources, Contacts |
| **Tasks** | Project/ticket tasks | Resources, Projects, Tickets |
| **Configuration Items** | Assets/devices | Companies, Contacts |
| **Contracts** | Service agreements | Companies, Contacts |
| **Billing Items** | Billable items | Companies, Tasks, Tickets, Projects |
| **Time Entries** | Time tracking | Companies, Resources, Projects, Tasks, Tickets |

**Selection tips:**
- ✅ **Select dependencies first** - e.g., Companies before Contacts
- 🔄 **Use "Select All"** for full sync
- 🎯 **Deselect unneeded** - Skip entities you don't use
- ⚠️ **Check order** - Entities sync in dependency order automatically

---

## Sync Progress & Monitoring

### Real-time Progress
- **Progress bars** - Visual indication of sync status
- **Record counts** - Shows records added/updated/deleted
- **Chunk tracking** - For chunked syncs, shows current month
- **Error alerts** - Immediate notification of failures

### Sync History Table
- **Recent syncs** - Last 50 sync operations
- **Status indicators** - ✅ Completed, ❌ Failed, ⏳ In Progress
- **Timing data** - Duration and timestamps
- **Error messages** - Detailed failure information
- **Filter options** - By entity, status, or date

### Background Processing
- Syncs run in the **background** (server-side)
- UI remains responsive during sync
- Dashboard auto-refreshes every 30 seconds
- Can close browser - sync continues on server

---

## Best Practices

### Daily Operations
```
Morning: Incremental Sync (Last 7 Days)
→ Fast refresh of overnight changes
→ 2-5 minutes
```

### Weekly Maintenance
```
Monday: Full Sync (Last 90 Days)
→ Ensure data consistency
→ Clean up deleted records
→ 10-15 minutes
```

### Monthly Deep Sync
```
First of Month: Full Sync (Last 2 Years)
→ Comprehensive data refresh
→ Historical accuracy
→ 20-30 minutes
```

### Initial Setup
```
Step 1: Full Sync (Last 2 Years)
→ Get recent operational data
→ 20-30 minutes

Step 2: Chunked Tickets (All Time) - Optional
→ Run overnight for complete history
→ 1-3 hours depending on volume
```

---

## Troubleshooting

### "Sync is taking too long"
**Solution:** Use Chunked Tickets for large date ranges
- Switch from "All Time" to "Last 2 Years"
- Use Chunked Tickets for historical data
- Run large syncs during off-hours

### "Sync failed with timeout error"
**Solution:** Reduce scope
- Decrease date range (e.g., from 5 years to 1 year)
- Use Chunked Tickets instead of Full Sync
- Sync entities individually using Sync Selected

### "Missing records in database"
**Solution:** Run Full Sync
- Full Sync will fetch all active records
- Check date range - records outside range won't sync
- Verify records exist and are active in Autotask

### "Deleted records still showing"
**Solution:** Run Full Sync
- Only Full Sync performs soft-deletes
- Incremental and Entity syncs don't remove records
- Check `is_deleted` flag in database

### "Foreign key constraint errors"
**Solution:** Sync dependencies first
- Sync Companies before Contacts
- Sync Resources before Tickets
- Use Full Sync to handle dependencies automatically

### "Some entities failed, others succeeded"
**Check:** Sync History table
- Review error messages for failed entities
- Re-sync failed entities individually
- Check Autotask API connectivity
- Review logs: `npx tsx scripts/analyze-sync-failures.ts`

---

## Understanding Sync Results

### Record Counts

**Records Added** (+)
- New records inserted into database
- First time seeing this record from Autotask

**Records Updated** (~)
- Existing records modified
- Changes detected from Autotask

**Records Deleted** (-)
- Records soft-deleted (Full Sync only)
- Set `is_deleted = true` when not found in Autotask
- Only happens during Full Sync

### Example Results
```
✅ Companies: +5 ~23 -2
   → 5 new companies
   → 23 companies updated
   → 2 companies soft-deleted (no longer in Autotask)

✅ Tickets: +142 ~67 -0
   → 142 new tickets
   → 67 tickets updated
   → 0 deleted (incremental sync doesn't delete)
```

---

## Sync Timing Recommendations

### Business Hours (9 AM - 5 PM)
- ✅ Incremental Sync (Last 7-30 Days)
- ✅ Sync Selected (specific entities)
- ⚠️ Avoid Full Sync with large date ranges
- ⚠️ Avoid Chunked Tickets for All Time

### Off-Hours (Evening/Night)
- ✅ Full Sync (Last 2+ Years)
- ✅ Chunked Tickets (All Time)
- ✅ Large historical syncs
- ✅ Data recovery operations

### Automated Schedule (Recommended)
```
Every 4 hours: Incremental Sync
→ Keeps data fresh throughout the day

Daily at 2 AM: Full Sync (Last 90 Days)
→ Maintains data consistency

Weekly Sunday 1 AM: Full Sync (Last 2 Years)
→ Deep refresh and cleanup
```

---

## API Rate Limits & Performance

### Autotask API Limits
- **Rate limit:** ~500 requests per hour per integration
- **Timeout:** 60 seconds per request
- **Page size:** 500 records maximum per page

### Sync Performance Tips
1. **Use Incremental Sync** for routine updates
2. **Limit date ranges** to reduce API calls
3. **Avoid concurrent syncs** - wait for one to finish
4. **Use Chunked Tickets** for large datasets
5. **Schedule heavy syncs** during off-hours

### Estimated Sync Times

| Sync Type | Date Range | Estimated Time |
|-----------|------------|----------------|
| Incremental | N/A | 2-5 minutes |
| Full Sync | Last 7 Days | 5-10 minutes |
| Full Sync | Last 90 Days | 10-15 minutes |
| Full Sync | Last 2 Years | 20-30 minutes |
| Chunked Tickets | Last 5 Years | 1-2 hours |
| Chunked Tickets | All Time | 2-4 hours |

*Times vary based on data volume and API performance*

---

## Analyzing Sync Failures

Use the sync failure analysis script to investigate issues:

```bash
# Show failures from last 7 days
npx tsx scripts/analyze-sync-failures.ts

# Show all syncs from yesterday
npx tsx scripts/analyze-sync-failures.ts --days 1 --all

# Check specific entity failures
npx tsx scripts/analyze-sync-failures.ts --days 30 --entity tickets

# Show all completed syncs
npx tsx scripts/analyze-sync-failures.ts --status completed --all
```

The script provides:
- Failure summary by entity
- Error categorization (NETWORK_ERROR, DATABASE_ERROR, etc.)
- Success rate statistics
- Detailed error messages
- Timing information

---

## Quick Reference

| Need | Use This | Time |
|------|----------|------|
| Quick update | Incremental Sync | 2-5 min |
| Daily refresh | Incremental (Last 7 Days) | 2-5 min |
| Weekly cleanup | Full Sync (Last 90 Days) | 10-15 min |
| Monthly deep sync | Full Sync (Last 2 Years) | 20-30 min |
| Initial setup | Full Sync (Last 2 Years) | 20-30 min |
| Historical tickets | Chunked Tickets (All Time) | 1-3 hours |
| Single entity | Sync Selected | 1-10 min |
| After timeout | Chunked Tickets | Varies |
| Data recovery | Full Sync (appropriate range) | Varies |

---

## Support & Additional Resources

- **Sync Logging Documentation:** `docs/SYNC_LOGGING_IMPROVEMENTS.md`
- **Analysis Script:** `scripts/analyze-sync-failures.ts`
- **Sync History:** Admin Dashboard → Sync History Table
- **Error Logs:** Check structured logs for detailed error information

---

## Summary

**For most users:**
- Run **Incremental Sync** daily for routine updates
- Run **Full Sync** weekly/monthly for data consistency
- Use **Chunked Tickets** for large historical syncs
- Use **Sync Selected** for targeted entity updates

**Remember:**
- ✅ Incremental = Fast, additive only
- 🔄 Full = Complete, includes soft-deletes
- 📦 Chunked = Reliable for large datasets
- 🎯 Selected = Targeted, specific entities

**Date ranges matter:**
- Smaller ranges = Faster syncs
- Larger ranges = More complete data
- Balance based on your needs and available time