wulf-pulse/docs/SYNC_INTERFACE_GUIDE.md

Sync Interface User Guide

Overview

The Sync Interface allows you to synchronize data from Autotask to your PostgreSQL database. This guide explains each sync option, when to use it, and what to expect.

Sync Interface Location

Access the sync interface at: Admin Dashboard → Sync Controls

---

Sync Options

1. Full Sync 🔄

What it does:

• Syncs all entities from Autotask to PostgreSQL
• Fetches all active records for each entity type
• Soft-deletes records that exist in your database but are no longer in Autotask
• Respects the date range filter for time-based entities (Tickets, Tasks, Projects)

When to use:

• Initial setup - First time syncing your Autotask data
• Data recovery - After database issues or data corruption
• Periodic refresh - Monthly or quarterly to ensure data consistency
• After major Autotask changes - When records have been bulk-deleted or modified in Autotask

Important notes:

• ⚠️ Requires confirmation - Shows a warning dialog before starting
• ⏱️ Takes longest - Can take 10-30 minutes depending on data volume
• 🗑️ Soft-deletes missing records - Sets is_deleted = true for records not found in Autotask
• 📅 Respects date range - For Tickets/Tasks/Projects, only syncs within selected date range

Example use case:

"It's the first Monday of the month. I want to ensure all our data is fresh and any deleted records in Autotask are marked as deleted in our database."

What gets synced (in order):

1. Companies
2. Resources (Users)
3. Statuses
4. Issue Types
5. Sub-Issue Types
6. Work Types
7. Contacts
8. Projects
9. Tickets (within date range)
10. Tasks (within date range)
11. Configuration Items
12. Contracts
13. Billing Items
14. Time Entries (within date range)

---

2. Incremental Sync ⚡

What it does:

• Syncs only records modified since the last sync
• Uses Autotask's lastModifiedDate field to filter records
• Much faster than Full Sync
• Does NOT soft-delete missing records

When to use:

• Daily updates - Regular scheduled syncs to keep data current
• Quick refresh - After making changes in Autotask
• Automated syncs - Set up on a cron schedule (e.g., every 4 hours)
• Low-impact updates - When you need fresh data without heavy processing

Important notes:

• ✅ No confirmation needed - Starts immediately
• ⚡ Fast - Typically completes in 1-5 minutes
• 🔄 Additive only - Adds/updates records but doesn't delete
• 📊 Best for routine updates - Ideal for keeping data fresh

Example use case:

"A technician just closed several tickets in Autotask. I want to pull those updates into our dashboard without running a full sync."

How it works:

1. Checks sync_history table for last successful sync of each entity
2. Queries Autotask for records where lastModifiedDate > last_sync_time
3. Updates existing records or inserts new ones
4. Skips entities that have no modifications

---

3. Chunked Tickets 📦

What it does:

• Syncs only Tickets in monthly date chunks
• Breaks large date ranges into manageable pieces
• Prevents API timeouts and memory issues
• Shows real-time progress for each chunk

When to use:

• Large date ranges - Syncing 3+ years of ticket history
• Initial ticket sync - First time pulling historical tickets
• Timeout prevention - When regular ticket sync fails with timeouts
• Off-hours processing - Running overnight syncs for historical data

Important notes:

• 📊 Progress tracking - Shows which month is currently syncing
• 🔄 Resilient - Continues even if individual chunks fail
• 🎯 Tickets only - Does not sync other entities
• ⏱️ Slower but reliable - Takes longer but handles large datasets

Example use case:

"I need to sync 5 years of ticket history. A regular sync times out, so I'll use Chunked Tickets to process it month by month."

How it works:

1. Calculates monthly chunks based on selected date range
2. Processes each month sequentially (e.g., "Jan 2024", "Feb 2024", etc.)
3. Shows progress: "Processing chunk 15/60: Mar 2023"
4. Continues even if some months fail (logs failures)
5. Displays final summary with total records processed

Date range examples:

• Last 7 Days → ~1 chunk
• Last 2 Years → ~24 chunks
• Last 5 Years → ~60 chunks
• All Time → Could be 100+ chunks

---

4. Sync Selected 🎯

What it does:

• Syncs only the entities you select from the entity list
• Full sync for selected entities (not incremental)
• Respects date range for time-based entities
• Does NOT soft-delete missing records

When to use:

• Targeted updates - Only need to refresh specific entities
• Dependency sync - Sync Companies before syncing Tickets
• Troubleshooting - Re-sync a specific entity that had errors
• Testing - Verify sync works for a single entity

Important notes:

• ✅ Requires entity selection - Must select at least one entity
• 🎯 Selective - Only syncs what you choose
• ⚠️ Respect dependencies - Ensure parent entities are synced first
• 📅 Uses date range - For Tickets/Tasks/Projects

Example use case:

"We just imported 50 new companies into Autotask. I only need to sync Companies and Contacts, not everything else."

Entity dependencies to consider:

• Contacts → Requires Companies
• Tickets → Requires Companies, Resources, Contacts
• Tasks → Requires Resources, Projects, Tickets
• Configuration Items → Requires Companies, Contacts
• Time Entries → Requires Companies, Resources, Projects, Tasks, Tickets

How to use:

1. Click entity checkboxes to select (e.g., Companies, Contacts)
2. Set date range if syncing time-based entities
3. Click "Sync Selected (2)" button
4. Monitor progress in sync history

---

Date Range Selector 📅

Applies to: Tickets, Tasks, Projects, Time Entries

Available ranges:

• Last 7 Days (0.019 years) - Quick updates, minimal data
• Last 30 Days (0.082 years) - Monthly refresh
• Last 90 Days (0.25 years) - Quarterly data
• Last 1 Year - Recent history
• Last 2 Years ⭐ Recommended - Good balance of history and performance
• Last 3 Years - Extended history
• Last 5 Years - Long-term analysis
• Last 10 Years - Comprehensive history
• All Time ⚠️ - Complete history (very slow, use off-hours)

Why limit date ranges?

1. Performance - Fewer records = faster sync
2. API limits - Autotask has rate limits and timeout thresholds
3. Relevance - Most reporting focuses on recent data
4. Resource usage - Large syncs consume more memory and CPU

Recommendation:

• Daily syncs: Last 7-30 Days
• Weekly syncs: Last 90 Days
• Monthly syncs: Last 1-2 Years
• Initial setup: Last 2 Years, then run "All Time" overnight if needed

---

Entity Selector

Available entities:

Entity	Description	Dependencies
Companies	Customer/vendor organizations	None
Resources	Autotask users/technicians	None
Statuses	Ticket/task statuses	None
Issue Types	Ticket categorization	None
Sub-Issue Types	Ticket sub-categories	None
Work Types	Time entry work types	None
Contacts	People at companies	Companies
Projects	Customer projects	Companies, Resources
Tickets	Support tickets	Companies, Resources, Contacts
Tasks	Project/ticket tasks	Resources, Projects, Tickets
Configuration Items	Assets/devices	Companies, Contacts
Contracts	Service agreements	Companies, Contacts
Billing Items	Billable items	Companies, Tasks, Tickets, Projects
Time Entries	Time tracking	Companies, Resources, Projects, Tasks, Tickets

Selection tips:

• ✅ Select dependencies first - e.g., Companies before Contacts
• 🔄 Use "Select All" for full sync
• 🎯 Deselect unneeded - Skip entities you don't use
• ⚠️ Check order - Entities sync in dependency order automatically

---

Sync Progress & Monitoring

Real-time Progress

• Progress bars - Visual indication of sync status
• Record counts - Shows records added/updated/deleted
• Chunk tracking - For chunked syncs, shows current month
• Error alerts - Immediate notification of failures

Sync History Table

• Recent syncs - Last 50 sync operations
• Status indicators - ✅ Completed, ❌ Failed, ⏳ In Progress
• Timing data - Duration and timestamps
• Error messages - Detailed failure information
• Filter options - By entity, status, or date

Background Processing

• Syncs run in the background (server-side)
• UI remains responsive during sync
• Dashboard auto-refreshes every 30 seconds
• Can close browser - sync continues on server

---

Best Practices

Daily Operations

Morning: Incremental Sync (Last 7 Days)
→ Fast refresh of overnight changes
→ 2-5 minutes

Weekly Maintenance

Monday: Full Sync (Last 90 Days)
→ Ensure data consistency
→ Clean up deleted records
→ 10-15 minutes

Monthly Deep Sync

First of Month: Full Sync (Last 2 Years)
→ Comprehensive data refresh
→ Historical accuracy
→ 20-30 minutes

Initial Setup

Step 1: Full Sync (Last 2 Years)
→ Get recent operational data
→ 20-30 minutes

Step 2: Chunked Tickets (All Time) - Optional
→ Run overnight for complete history
→ 1-3 hours depending on volume

---

Troubleshooting

"Sync is taking too long"

Solution: Use Chunked Tickets for large date ranges

• Switch from "All Time" to "Last 2 Years"
• Use Chunked Tickets for historical data
• Run large syncs during off-hours

"Sync failed with timeout error"

Solution: Reduce scope

• Decrease date range (e.g., from 5 years to 1 year)
• Use Chunked Tickets instead of Full Sync
• Sync entities individually using Sync Selected

"Missing records in database"

Solution: Run Full Sync

• Full Sync will fetch all active records
• Check date range - records outside range won't sync
• Verify records exist and are active in Autotask

"Deleted records still showing"

Solution: Run Full Sync

• Only Full Sync performs soft-deletes
• Incremental and Entity syncs don't remove records
• Check is_deleted flag in database

"Foreign key constraint errors"

Solution: Sync dependencies first

• Sync Companies before Contacts
• Sync Resources before Tickets
• Use Full Sync to handle dependencies automatically

"Some entities failed, others succeeded"

Check: Sync History table

• Review error messages for failed entities
• Re-sync failed entities individually
• Check Autotask API connectivity
• Review logs: npx tsx scripts/analyze-sync-failures.ts

---

Understanding Sync Results

Record Counts

Records Added (+)

• New records inserted into database
• First time seeing this record from Autotask

Records Updated (~)

• Existing records modified
• Changes detected from Autotask

Records Deleted (-)

• Records soft-deleted (Full Sync only)
• Set is_deleted = true when not found in Autotask
• Only happens during Full Sync

Example Results

✅ Companies: +5 ~23 -2
   → 5 new companies
   → 23 companies updated
   → 2 companies soft-deleted (no longer in Autotask)

✅ Tickets: +142 ~67 -0
   → 142 new tickets
   → 67 tickets updated
   → 0 deleted (incremental sync doesn't delete)

---

Sync Timing Recommendations

Business Hours (9 AM - 5 PM)

• ✅ Incremental Sync (Last 7-30 Days)
• ✅ Sync Selected (specific entities)
• ⚠️ Avoid Full Sync with large date ranges
• ⚠️ Avoid Chunked Tickets for All Time

Off-Hours (Evening/Night)

• ✅ Full Sync (Last 2+ Years)
• ✅ Chunked Tickets (All Time)
• ✅ Large historical syncs
• ✅ Data recovery operations

Automated Schedule (Recommended)

Every 4 hours: Incremental Sync
→ Keeps data fresh throughout the day

Daily at 2 AM: Full Sync (Last 90 Days)
→ Maintains data consistency

Weekly Sunday 1 AM: Full Sync (Last 2 Years)
→ Deep refresh and cleanup

---

API Rate Limits & Performance

Autotask API Limits

• Rate limit: ~500 requests per hour per integration
• Timeout: 60 seconds per request
• Page size: 500 records maximum per page

Sync Performance Tips

1. Use Incremental Sync for routine updates
2. Limit date ranges to reduce API calls
3. Avoid concurrent syncs - wait for one to finish
4. Use Chunked Tickets for large datasets
5. Schedule heavy syncs during off-hours

Estimated Sync Times

Sync Type	Date Range	Estimated Time
Incremental	N/A	2-5 minutes
Full Sync	Last 7 Days	5-10 minutes
Full Sync	Last 90 Days	10-15 minutes
Full Sync	Last 2 Years	20-30 minutes
Chunked Tickets	Last 5 Years	1-2 hours
Chunked Tickets	All Time	2-4 hours

Times vary based on data volume and API performance

---

Analyzing Sync Failures

Use the sync failure analysis script to investigate issues:

# Show failures from last 7 days
npx tsx scripts/analyze-sync-failures.ts

# Show all syncs from yesterday
npx tsx scripts/analyze-sync-failures.ts --days 1 --all

# Check specific entity failures
npx tsx scripts/analyze-sync-failures.ts --days 30 --entity tickets

# Show all completed syncs
npx tsx scripts/analyze-sync-failures.ts --status completed --all

The script provides:

• Failure summary by entity
• Error categorization (NETWORK_ERROR, DATABASE_ERROR, etc.)
• Success rate statistics
• Detailed error messages
• Timing information

---

Quick Reference

Need	Use This	Time
Quick update	Incremental Sync	2-5 min
Daily refresh	Incremental (Last 7 Days)	2-5 min
Weekly cleanup	Full Sync (Last 90 Days)	10-15 min
Monthly deep sync	Full Sync (Last 2 Years)	20-30 min
Initial setup	Full Sync (Last 2 Years)	20-30 min
Historical tickets	Chunked Tickets (All Time)	1-3 hours
Single entity	Sync Selected	1-10 min
After timeout	Chunked Tickets	Varies
Data recovery	Full Sync (appropriate range)	Varies

---

Support & Additional Resources

• Sync Logging Documentation: docs/SYNC_LOGGING_IMPROVEMENTS.md
• Analysis Script: scripts/analyze-sync-failures.ts
• Sync History: Admin Dashboard → Sync History Table
• Error Logs: Check structured logs for detailed error information

---

Summary

For most users:

• Run Incremental Sync daily for routine updates
• Run Full Sync weekly/monthly for data consistency
• Use Chunked Tickets for large historical syncs
• Use Sync Selected for targeted entity updates

Remember:

• ✅ Incremental = Fast, additive only
• 🔄 Full = Complete, includes soft-deletes
• 📦 Chunked = Reliable for large datasets
• 🎯 Selected = Targeted, specific entities

Date ranges matter:

• Smaller ranges = Faster syncs
• Larger ranges = More complete data
• Balance based on your needs and available time