diff --git a/docs/SYNC_INTERFACE_GUIDE.md b/docs/SYNC_INTERFACE_GUIDE.md new file mode 100644 index 0000000..4d4e61e --- /dev/null +++ b/docs/SYNC_INTERFACE_GUIDE.md @@ -0,0 +1,482 @@ +# Sync Interface User Guide + +## Overview + +The Sync Interface allows you to synchronize data from Autotask to your PostgreSQL database. This guide explains each sync option, when to use it, and what to expect. + +## Sync Interface Location + +Access the sync interface at: **Admin Dashboard → Sync Controls** + +--- + +## Sync Options + +### 1. **Full Sync** 🔄 + +**What it does:** +- Syncs **all entities** from Autotask to PostgreSQL +- Fetches all active records for each entity type +- **Soft-deletes** records that exist in your database but are no longer in Autotask +- Respects the date range filter for time-based entities (Tickets, Tasks, Projects) + +**When to use:** +- **Initial setup** - First time syncing your Autotask data +- **Data recovery** - After database issues or data corruption +- **Periodic refresh** - Monthly or quarterly to ensure data consistency +- **After major Autotask changes** - When records have been bulk-deleted or modified in Autotask + +**Important notes:** +- ⚠️ **Requires confirmation** - Shows a warning dialog before starting +- ⏱️ **Takes longest** - Can take 10-30 minutes depending on data volume +- 🗑️ **Soft-deletes missing records** - Sets `is_deleted = true` for records not found in Autotask +- 📅 **Respects date range** - For Tickets/Tasks/Projects, only syncs within selected date range + +**Example use case:** +> "It's the first Monday of the month. I want to ensure all our data is fresh and any deleted records in Autotask are marked as deleted in our database." + +**What gets synced (in order):** +1. Companies +2. Resources (Users) +3. Statuses +4. Issue Types +5. Sub-Issue Types +6. Work Types +7. Contacts +8. Projects +9. Tickets (within date range) +10. Tasks (within date range) +11. Configuration Items +12. Contracts +13. Billing Items +14. Time Entries (within date range) + +--- + +### 2. **Incremental Sync** ⚡ + +**What it does:** +- Syncs **only records modified since the last sync** +- Uses Autotask's `lastModifiedDate` field to filter records +- Much faster than Full Sync +- Does **NOT** soft-delete missing records + +**When to use:** +- **Daily updates** - Regular scheduled syncs to keep data current +- **Quick refresh** - After making changes in Autotask +- **Automated syncs** - Set up on a cron schedule (e.g., every 4 hours) +- **Low-impact updates** - When you need fresh data without heavy processing + +**Important notes:** +- ✅ **No confirmation needed** - Starts immediately +- ⚡ **Fast** - Typically completes in 1-5 minutes +- 🔄 **Additive only** - Adds/updates records but doesn't delete +- 📊 **Best for routine updates** - Ideal for keeping data fresh + +**Example use case:** +> "A technician just closed several tickets in Autotask. I want to pull those updates into our dashboard without running a full sync." + +**How it works:** +1. Checks `sync_history` table for last successful sync of each entity +2. Queries Autotask for records where `lastModifiedDate > last_sync_time` +3. Updates existing records or inserts new ones +4. Skips entities that have no modifications + +--- + +### 3. **Chunked Tickets** 📦 + +**What it does:** +- Syncs **only Tickets** in monthly date chunks +- Breaks large date ranges into manageable pieces +- Prevents API timeouts and memory issues +- Shows real-time progress for each chunk + +**When to use:** +- **Large date ranges** - Syncing 3+ years of ticket history +- **Initial ticket sync** - First time pulling historical tickets +- **Timeout prevention** - When regular ticket sync fails with timeouts +- **Off-hours processing** - Running overnight syncs for historical data + +**Important notes:** +- 📊 **Progress tracking** - Shows which month is currently syncing +- 🔄 **Resilient** - Continues even if individual chunks fail +- 🎯 **Tickets only** - Does not sync other entities +- ⏱️ **Slower but reliable** - Takes longer but handles large datasets + +**Example use case:** +> "I need to sync 5 years of ticket history. A regular sync times out, so I'll use Chunked Tickets to process it month by month." + +**How it works:** +1. Calculates monthly chunks based on selected date range +2. Processes each month sequentially (e.g., "Jan 2024", "Feb 2024", etc.) +3. Shows progress: "Processing chunk 15/60: Mar 2023" +4. Continues even if some months fail (logs failures) +5. Displays final summary with total records processed + +**Date range examples:** +- Last 7 Days → ~1 chunk +- Last 2 Years → ~24 chunks +- Last 5 Years → ~60 chunks +- All Time → Could be 100+ chunks + +--- + +### 4. **Sync Selected** 🎯 + +**What it does:** +- Syncs **only the entities you select** from the entity list +- Full sync for selected entities (not incremental) +- Respects date range for time-based entities +- Does **NOT** soft-delete missing records + +**When to use:** +- **Targeted updates** - Only need to refresh specific entities +- **Dependency sync** - Sync Companies before syncing Tickets +- **Troubleshooting** - Re-sync a specific entity that had errors +- **Testing** - Verify sync works for a single entity + +**Important notes:** +- ✅ **Requires entity selection** - Must select at least one entity +- 🎯 **Selective** - Only syncs what you choose +- ⚠️ **Respect dependencies** - Ensure parent entities are synced first +- 📅 **Uses date range** - For Tickets/Tasks/Projects + +**Example use case:** +> "We just imported 50 new companies into Autotask. I only need to sync Companies and Contacts, not everything else." + +**Entity dependencies to consider:** +- **Contacts** → Requires Companies +- **Tickets** → Requires Companies, Resources, Contacts +- **Tasks** → Requires Resources, Projects, Tickets +- **Configuration Items** → Requires Companies, Contacts +- **Time Entries** → Requires Companies, Resources, Projects, Tasks, Tickets + +**How to use:** +1. Click entity checkboxes to select (e.g., Companies, Contacts) +2. Set date range if syncing time-based entities +3. Click "Sync Selected (2)" button +4. Monitor progress in sync history + + +--- + +## Date Range Selector 📅 + +**Applies to:** Tickets, Tasks, Projects, Time Entries + +**Available ranges:** +- **Last 7 Days** (0.019 years) - Quick updates, minimal data +- **Last 30 Days** (0.082 years) - Monthly refresh +- **Last 90 Days** (0.25 years) - Quarterly data +- **Last 1 Year** - Recent history +- **Last 2 Years** ⭐ **Recommended** - Good balance of history and performance +- **Last 3 Years** - Extended history +- **Last 5 Years** - Long-term analysis +- **Last 10 Years** - Comprehensive history +- **All Time** ⚠️ - Complete history (very slow, use off-hours) + +**Why limit date ranges?** +1. **Performance** - Fewer records = faster sync +2. **API limits** - Autotask has rate limits and timeout thresholds +3. **Relevance** - Most reporting focuses on recent data +4. **Resource usage** - Large syncs consume more memory and CPU + +**Recommendation:** +- **Daily syncs:** Last 7-30 Days +- **Weekly syncs:** Last 90 Days +- **Monthly syncs:** Last 1-2 Years +- **Initial setup:** Last 2 Years, then run "All Time" overnight if needed + +--- + +## Entity Selector + +**Available entities:** + +| Entity | Description | Dependencies | +|--------|-------------|--------------| +| **Companies** | Customer/vendor organizations | None | +| **Resources** | Autotask users/technicians | None | +| **Statuses** | Ticket/task statuses | None | +| **Issue Types** | Ticket categorization | None | +| **Sub-Issue Types** | Ticket sub-categories | None | +| **Work Types** | Time entry work types | None | +| **Contacts** | People at companies | Companies | +| **Projects** | Customer projects | Companies, Resources | +| **Tickets** | Support tickets | Companies, Resources, Contacts | +| **Tasks** | Project/ticket tasks | Resources, Projects, Tickets | +| **Configuration Items** | Assets/devices | Companies, Contacts | +| **Contracts** | Service agreements | Companies, Contacts | +| **Billing Items** | Billable items | Companies, Tasks, Tickets, Projects | +| **Time Entries** | Time tracking | Companies, Resources, Projects, Tasks, Tickets | + +**Selection tips:** +- ✅ **Select dependencies first** - e.g., Companies before Contacts +- 🔄 **Use "Select All"** for full sync +- 🎯 **Deselect unneeded** - Skip entities you don't use +- ⚠️ **Check order** - Entities sync in dependency order automatically + +--- + +## Sync Progress & Monitoring + +### Real-time Progress +- **Progress bars** - Visual indication of sync status +- **Record counts** - Shows records added/updated/deleted +- **Chunk tracking** - For chunked syncs, shows current month +- **Error alerts** - Immediate notification of failures + +### Sync History Table +- **Recent syncs** - Last 50 sync operations +- **Status indicators** - ✅ Completed, ❌ Failed, ⏳ In Progress +- **Timing data** - Duration and timestamps +- **Error messages** - Detailed failure information +- **Filter options** - By entity, status, or date + +### Background Processing +- Syncs run in the **background** (server-side) +- UI remains responsive during sync +- Dashboard auto-refreshes every 30 seconds +- Can close browser - sync continues on server + +--- + +## Best Practices + +### Daily Operations +``` +Morning: Incremental Sync (Last 7 Days) +→ Fast refresh of overnight changes +→ 2-5 minutes +``` + +### Weekly Maintenance +``` +Monday: Full Sync (Last 90 Days) +→ Ensure data consistency +→ Clean up deleted records +→ 10-15 minutes +``` + +### Monthly Deep Sync +``` +First of Month: Full Sync (Last 2 Years) +→ Comprehensive data refresh +→ Historical accuracy +→ 20-30 minutes +``` + +### Initial Setup +``` +Step 1: Full Sync (Last 2 Years) +→ Get recent operational data +→ 20-30 minutes + +Step 2: Chunked Tickets (All Time) - Optional +→ Run overnight for complete history +→ 1-3 hours depending on volume +``` + +--- + +## Troubleshooting + +### "Sync is taking too long" +**Solution:** Use Chunked Tickets for large date ranges +- Switch from "All Time" to "Last 2 Years" +- Use Chunked Tickets for historical data +- Run large syncs during off-hours + +### "Sync failed with timeout error" +**Solution:** Reduce scope +- Decrease date range (e.g., from 5 years to 1 year) +- Use Chunked Tickets instead of Full Sync +- Sync entities individually using Sync Selected + +### "Missing records in database" +**Solution:** Run Full Sync +- Full Sync will fetch all active records +- Check date range - records outside range won't sync +- Verify records exist and are active in Autotask + +### "Deleted records still showing" +**Solution:** Run Full Sync +- Only Full Sync performs soft-deletes +- Incremental and Entity syncs don't remove records +- Check `is_deleted` flag in database + +### "Foreign key constraint errors" +**Solution:** Sync dependencies first +- Sync Companies before Contacts +- Sync Resources before Tickets +- Use Full Sync to handle dependencies automatically + +### "Some entities failed, others succeeded" +**Check:** Sync History table +- Review error messages for failed entities +- Re-sync failed entities individually +- Check Autotask API connectivity +- Review logs: `npx tsx scripts/analyze-sync-failures.ts` + +--- + +## Understanding Sync Results + +### Record Counts + +**Records Added** (+) +- New records inserted into database +- First time seeing this record from Autotask + +**Records Updated** (~) +- Existing records modified +- Changes detected from Autotask + +**Records Deleted** (-) +- Records soft-deleted (Full Sync only) +- Set `is_deleted = true` when not found in Autotask +- Only happens during Full Sync + +### Example Results +``` +✅ Companies: +5 ~23 -2 + → 5 new companies + → 23 companies updated + → 2 companies soft-deleted (no longer in Autotask) + +✅ Tickets: +142 ~67 -0 + → 142 new tickets + → 67 tickets updated + → 0 deleted (incremental sync doesn't delete) +``` + +--- + +## Sync Timing Recommendations + +### Business Hours (9 AM - 5 PM) +- ✅ Incremental Sync (Last 7-30 Days) +- ✅ Sync Selected (specific entities) +- ⚠️ Avoid Full Sync with large date ranges +- ⚠️ Avoid Chunked Tickets for All Time + +### Off-Hours (Evening/Night) +- ✅ Full Sync (Last 2+ Years) +- ✅ Chunked Tickets (All Time) +- ✅ Large historical syncs +- ✅ Data recovery operations + +### Automated Schedule (Recommended) +``` +Every 4 hours: Incremental Sync +→ Keeps data fresh throughout the day + +Daily at 2 AM: Full Sync (Last 90 Days) +→ Maintains data consistency + +Weekly Sunday 1 AM: Full Sync (Last 2 Years) +→ Deep refresh and cleanup +``` + +--- + +## API Rate Limits & Performance + +### Autotask API Limits +- **Rate limit:** ~500 requests per hour per integration +- **Timeout:** 60 seconds per request +- **Page size:** 500 records maximum per page + +### Sync Performance Tips +1. **Use Incremental Sync** for routine updates +2. **Limit date ranges** to reduce API calls +3. **Avoid concurrent syncs** - wait for one to finish +4. **Use Chunked Tickets** for large datasets +5. **Schedule heavy syncs** during off-hours + +### Estimated Sync Times + +| Sync Type | Date Range | Estimated Time | +|-----------|------------|----------------| +| Incremental | N/A | 2-5 minutes | +| Full Sync | Last 7 Days | 5-10 minutes | +| Full Sync | Last 90 Days | 10-15 minutes | +| Full Sync | Last 2 Years | 20-30 minutes | +| Chunked Tickets | Last 5 Years | 1-2 hours | +| Chunked Tickets | All Time | 2-4 hours | + +*Times vary based on data volume and API performance* + +--- + +## Analyzing Sync Failures + +Use the sync failure analysis script to investigate issues: + +```bash +# Show failures from last 7 days +npx tsx scripts/analyze-sync-failures.ts + +# Show all syncs from yesterday +npx tsx scripts/analyze-sync-failures.ts --days 1 --all + +# Check specific entity failures +npx tsx scripts/analyze-sync-failures.ts --days 30 --entity tickets + +# Show all completed syncs +npx tsx scripts/analyze-sync-failures.ts --status completed --all +``` + +The script provides: +- Failure summary by entity +- Error categorization (NETWORK_ERROR, DATABASE_ERROR, etc.) +- Success rate statistics +- Detailed error messages +- Timing information + +--- + +## Quick Reference + +| Need | Use This | Time | +|------|----------|------| +| Quick update | Incremental Sync | 2-5 min | +| Daily refresh | Incremental (Last 7 Days) | 2-5 min | +| Weekly cleanup | Full Sync (Last 90 Days) | 10-15 min | +| Monthly deep sync | Full Sync (Last 2 Years) | 20-30 min | +| Initial setup | Full Sync (Last 2 Years) | 20-30 min | +| Historical tickets | Chunked Tickets (All Time) | 1-3 hours | +| Single entity | Sync Selected | 1-10 min | +| After timeout | Chunked Tickets | Varies | +| Data recovery | Full Sync (appropriate range) | Varies | + +--- + +## Support & Additional Resources + +- **Sync Logging Documentation:** `docs/SYNC_LOGGING_IMPROVEMENTS.md` +- **Analysis Script:** `scripts/analyze-sync-failures.ts` +- **Sync History:** Admin Dashboard → Sync History Table +- **Error Logs:** Check structured logs for detailed error information + +--- + +## Summary + +**For most users:** +- Run **Incremental Sync** daily for routine updates +- Run **Full Sync** weekly/monthly for data consistency +- Use **Chunked Tickets** for large historical syncs +- Use **Sync Selected** for targeted entity updates + +**Remember:** +- ✅ Incremental = Fast, additive only +- 🔄 Full = Complete, includes soft-deletes +- 📦 Chunked = Reliable for large datasets +- 🎯 Selected = Targeted, specific entities + +**Date ranges matter:** +- Smaller ranges = Faster syncs +- Larger ranges = More complete data +- Balance based on your needs and available time