wulf-pulse/docs/SYNC_BEHAVIOR.md

Sync Behavior Documentation

Overview

This document provides comprehensive details about how the Autotask sync system works, including the differences between sync types, date range filtering, database operations, and entity-specific behaviors.

---

Table of Contents

• Sync Types
• Date Range Filtering
• Database Operations
• Entity-Specific Behaviors
• Foreign Key Validation
• Soft Deletes
• Performance Considerations

---

Sync Types

Full Sync

What it does:

• Fetches ALL records from Autotask that match the configured filters
• Performs an UPSERT operation on every fetched record
• Optionally performs soft deletes for missing records (entity-dependent)

When to use:

• Initial setup or first-time sync
• After major data changes in Autotask
• To ensure complete data consistency
• When you suspect data drift between systems

Characteristics:

• Processes all records within filter criteria (date range, status, etc.)
• Does NOT check if records have changed - upserts everything
• Slower than incremental sync
• More thorough and ensures data consistency

Example:

Full Sync of Tickets (last 2 years):
1. Fetch all tickets created in last 2 years from Autotask
2. Upsert all fetched tickets to PostgreSQL
3. Skip soft deletes (has date filter)

Incremental Sync

What it does:

• Fetches only records modified since the last successful sync
• Uses lastModifiedDate or similar timestamp fields
• Performs UPSERT on changed records only
• Does NOT perform soft deletes

When to use:

• Regular scheduled syncs (hourly, daily)
• To keep data up-to-date with minimal overhead
• When you only need recent changes

Characteristics:

• Only processes records that changed since last sync
• Much faster than full sync
• Requires a previous successful sync to establish baseline
• Falls back to full sync if no previous sync found

Important Limitations:

• Companies and Resources do NOT support incremental sync
  • These entities automatically fall back to full sync
  • Autotask API does not support date-based filtering for these entities
  • This is by design and prevents sync failures

Example:

Incremental Sync of Tickets:
1. Get last sync time: 2026-01-24 08:00:00
2. Fetch tickets where lastActivityDate >= 2026-01-24 08:00:00
3. Upsert only the changed tickets
4. No soft deletes

Incremental Sync of Companies (automatic fallback):
1. Request incremental sync
2. System detects Companies doesn't support incremental
3. Automatically performs full sync instead
4. Logs: "companies does not support incremental sync, performing full sync instead"

Sync Selected (Entity-Specific)

What it does:

• Syncs only the selected entities
• Uses full sync behavior for each selected entity
• Respects all entity-specific filters and behaviors

When to use:

• To sync specific entities without syncing everything
• After fixing entity-specific issues
• For testing or troubleshooting

---

Date Range Filtering

Entities with Required Filters

These entities REQUIRE filters - the Autotask API will return an error without them:

Entity	Filter Field	Default Range	Reason
Time Entries	dateWorked	Last 2 years	API requirement
Billing Items	itemDate	Last 2 years	API requirement
Contracts	status	Active only (status=1)	API requirement
Projects	status	Non-completed (status≠5)	API requirement

Entities with Optional Date Filters

These entities have date filters applied for performance and data management:

Entity	Filter Field	Default Range	Configurable
Tickets	createDate	Last 2 years	Yes (via yearsBack)
Tasks	createDateTime	Last 2 years	Yes (via yearsBack)

Entities Without Date Filters

These entities sync ALL records (no date filtering):

• Companies - All active companies
• Resources - All active resources
• Contacts - All active contacts
• Configuration Items - All active items
• Picklists (Statuses, Issue Types, Sub-Issue Types, Work Types) - All values

Configuring Date Range

The yearsBack parameter controls the date range:

// Default: 2 years
syncService.fullSync(2);

// Custom: 5 years
syncService.fullSync(5);

// Via API
POST /api/sync/full
{
  "yearsBack": 3,
  "triggeredBy": "user@example.com"
}

Important Notes:

1. Date filters apply to BOTH full and incremental sync

   • Full sync: Fetches all records within date range
   • Incremental sync: Fetches changed records within date range

2. Soft deletes are skipped for date-filtered entities

   • Prevents deleting old records outside the sync window
   • See Soft Deletes section

---

Database Operations

UPSERT Behavior

All syncs use PostgreSQL's ON CONFLICT DO UPDATE (UPSERT):

INSERT INTO table_name (id, field1, field2, ...)
VALUES ($1, $2, $3, ...)
ON CONFLICT (id) 
DO UPDATE SET
  field1 = EXCLUDED.field1,
  field2 = EXCLUDED.field2,
  ...
  synced_at = NOW()

What this means:

• If record exists: All fields are updated with new values from Autotask
• If record doesn't exist: Record is inserted as new
• Every fetched record is processed, regardless of whether data actually changed
• The synced_at timestamp is always updated

Record Counts

Sync history tracks three metrics:

Metric	Description	How It's Calculated
records_added	New records inserted	Estimated (~10% of upserted)
records_updated	Existing records updated	Estimated (~90% of upserted)
records_deleted	Records soft-deleted	Actual count from soft delete operation

Important: records_added and records_updated are estimates based on the total upserted count. PostgreSQL doesn't easily distinguish between inserts and updates in bulk UPSERT operations.

Bulk Processing

Records are processed in batches for performance:

• Fetch batch size: 500 records per API call
• Upsert batch size: 100 records per database transaction
• Pagination: Automatic for large datasets

---

Entity-Specific Behaviors

Picklist Entities

Entities: Statuses, Issue Types, Sub-Issue Types, Work Types

Special Handling:

• Fetched from /entity/entityInformation/fields endpoint (not /query)
• No date filtering
• No soft deletes
• Synced as key-value pairs

Example:

// Statuses from Tickets.status field
GET /Tickets/entityInformation/fields

// Maps to:
{ value: 1, label: "New", is_active: true }
{ value: 5, label: "Complete", is_active: true }

Tickets

Filters:

• Date: createDate >= (now - 2 years)
• Active: isActive = true (if applicable)

Foreign Key Validation:

• assigned_resource_id → Resources
• first_response_assigned_resource_id → Resources
• first_response_initiating_resource_id → Resources
• Invalid references are set to null

Soft Deletes: ❌ Skipped (has date filter)

Tasks

Filters:

• Date: createDateTime >= (now - 2 years)

Foreign Key Validation:

• project_id → Projects
• assigned_resource_id → Resources
• creator_resource_id → Resources
• completed_by_resource_id → Resources
• last_activity_resource_id → Resources
• Invalid references are set to null

Soft Deletes: ❌ Skipped (has date filter)

Time Entries

Filters:

• Required: dateWorked >= (now - 2 years)

Special Notes:

• company_id is nullable (time entries can exist without a company)
• No foreign key validation needed

Soft Deletes: ❌ Skipped (has date filter)

Billing Items

Filters:

• Required: itemDate >= (now - 2 years)

Special Notes:

• API requires a filter parameter
• Returns 500 error without filter

Soft Deletes: ❌ Skipped (has date filter)

Companies

Filters:

• Active: isActive = true

Special Notes:

• No date filtering - syncs all active companies
• Foundation entity for many foreign keys

Soft Deletes: ✅ Yes (no date filter)

Resources

Filters:

• Active: isActive = true

Special Notes:

• No date filtering - syncs all active resources
• Referenced by Tickets, Tasks, Time Entries

Soft Deletes: ✅ Yes (no date filter)

Contacts

Filters:

• Active: isActive = true

Special Notes:

• No date filtering - syncs all active contacts
• Referenced by Configuration Items

Soft Deletes: ✅ Yes (no date filter)

Configuration Items

Filters:

• Active: isActive = true

Foreign Key Validation:

• contact_id → Contacts
• Invalid references are set to null

Soft Deletes: ✅ Yes (no date filter)

Contracts

Filters:

• Required: status = 1 (Active)

Special Notes:

• API requires status filter
• Only syncs active contracts

Soft Deletes: ❌ Skipped (has status filter)

Projects

Filters:

• Required: status != 5 (Not Complete)

Special Notes:

• API requires status filter
• Excludes completed projects

Soft Deletes: ❌ Skipped (has status filter)

---

Foreign Key Validation

Why It's Needed

Autotask data may reference entities that:

• Haven't been synced yet
• Are outside the sync date range
• Have been deleted in Autotask
• Don't exist due to data inconsistencies

Without validation, these references cause PostgreSQL foreign key constraint violations.

How It Works

Before upserting records, the sync validates foreign key references:

1. Fetch valid IDs from the database

   const validResourceIds = await getValidResourceIds();
   // Returns Set of all resource IDs where is_deleted = false
   

2. Validate each record

   if (task.assigned_resource_id && !validResourceIds.has(task.assigned_resource_id)) {
     task.assigned_resource_id = null; // Nullify invalid reference
   }
   

3. Log warnings for nullified references

   [WARN] Nullified invalid resource references: 15 records
   

Validated Relationships

Entity	Foreign Key Field	References	Action on Invalid
Tickets	assigned_resource_id	Resources	Set to null
Tickets	first_response_assigned_resource_id	Resources	Set to null
Tickets	first_response_initiating_resource_id	Resources	Set to null
Tasks	project_id	Projects	Set to null
Tasks	assigned_resource_id	Resources	Set to null
Tasks	creator_resource_id	Resources	Set to null
Tasks	completed_by_resource_id	Resources	Set to null
Tasks	last_activity_resource_id	Resources	Set to null
Configuration Items	contact_id	Contacts	Set to null

Performance Impact

Foreign key validation queries are:

• Executed once per sync (cached)
• Indexed queries (fast)
• Minimal overhead compared to API fetching

---

Soft Deletes

What Are Soft Deletes?

Instead of permanently deleting records, the system marks them as deleted:

UPDATE table_name 
SET is_deleted = true, deleted_at = NOW()
WHERE id NOT IN (fetched_ids)
  AND is_deleted = false

When Soft Deletes Occur

Only during Full Sync for entities without date filters:

Entity	Soft Deletes?	Reason
Companies	✅ Yes	No date filter - safe to delete
Resources	✅ Yes	No date filter - safe to delete
Contacts	✅ Yes	No date filter - safe to delete
Configuration Items	✅ Yes	No date filter - safe to delete
Tickets	❌ No	Has date filter - would delete old records
Tasks	❌ No	Has date filter - would delete old records
Time Entries	❌ No	Has date filter - would delete old records
Billing Items	❌ No	Has date filter - would delete old records
Projects	❌ No	Has status filter - would delete completed
Contracts	❌ No	Has status filter - would delete inactive
Picklists	❌ No	Picklist values don't get deleted

Why Skip Soft Deletes for Date-Filtered Entities?

Problem: If we soft-deleted records not in the fetched set for date-filtered entities, we would incorrectly delete old records outside the sync window.

Example:

Full Sync of Tickets (last 2 years):
1. Fetch tickets created >= 2024-01-24
2. Ticket #12345 created in 2020 exists in DB
3. Ticket #12345 NOT in fetched set (outside date range)
4. ❌ Would incorrectly soft-delete Ticket #12345

Solution: Skip soft deletes for date-filtered entities

Restoring Soft-Deleted Records

If a record is soft-deleted but appears in a later sync:

-- Record is automatically restored
UPDATE table_name
SET is_deleted = false, deleted_at = NULL, ...
WHERE id = $1

---

Performance Considerations

Sync Duration Factors

Factor	Impact	Mitigation
Number of records	High	Use date range filters
API rate limits	Medium	Built-in rate limiting (10 req/sec)
Network latency	Medium	Pagination, batch processing
Database operations	Low	Bulk upserts, indexed queries
Foreign key validation	Low	Cached, indexed queries

Estimated Sync Times

Based on typical data volumes:

Entity	Record Count	Full Sync Time	Incremental Sync Time
Companies	500	~30 seconds	~5 seconds
Resources	100	~10 seconds	~2 seconds
Tickets (2 years)	10,000	~10 minutes	~1 minute
Tasks (2 years)	5,000	~5 minutes	~30 seconds
Time Entries (2 years)	50,000	~45 minutes	~5 minutes
Billing Items (2 years)	50,000+	~45-60 minutes	~5 minutes

Note: Times vary based on data volume, network speed, and system load.

Optimization Tips

1. Use Incremental Sync for regular updates

   • Schedule incremental syncs hourly or daily
   • Reserve full syncs for weekly/monthly maintenance

2. Adjust date range based on needs

   • Default 2 years is usually sufficient
   • Reduce to 1 year for faster syncs if older data isn't needed
   • Increase to 5+ years only if historical data is required

3. Sync entities in dependency order

   • Companies → Resources → Tickets → Tasks
   • Reduces foreign key validation overhead

4. Monitor sync history

   • Check for failures and patterns
   • Identify slow entities for optimization

5. Use Sync Selected for troubleshooting

   • Test individual entities
   • Isolate and fix issues without full sync

---

Sync Workflow Examples

Example 1: Initial Setup

1. Full Sync - All Entities (2 years)
   ├─ Companies (all active)
   ├─ Resources (all active)
   ├─ Contacts (all active)
   ├─ Configuration Items (all active)
   ├─ Contracts (active only)
   ├─ Projects (non-completed)
   ├─ Tickets (last 2 years)
   ├─ Tasks (last 2 years)
   ├─ Time Entries (last 2 years)
   ├─ Billing Items (last 2 years)
   └─ Picklists (all values)

Result: Complete database populated with all relevant data

Example 2: Daily Maintenance

1. Incremental Sync - All Entities
   ├─ Fetch only records modified since last sync
   ├─ Upsert changed records
   └─ No soft deletes

Result: Database updated with latest changes (fast)

Example 3: Fixing Specific Entity

1. Identify failed entity: Billing Items
2. Sync Selected - Billing Items only
   ├─ Full sync with date filter (last 2 years)
   ├─ Upsert all billing items
   └─ No soft deletes

Result: Billing Items data refreshed and consistent

Example 4: Historical Data Sync

1. Full Sync - Tickets (5 years)
   ├─ Adjust yearsBack parameter to 5
   ├─ Fetch tickets created >= 2021-01-24
   ├─ Upsert all fetched tickets
   └─ Skip soft deletes

Result: 5 years of ticket history in database

---

Troubleshooting

Common Issues

Issue: Foreign Key Constraint Violations

Symptoms:

[DATABASE_CONSTRAINT_ERROR] violates foreign key constraint "tasks_project_id_fkey"

Cause: Referenced entity hasn't been synced or is outside date range

Solution:

1. Sync dependency entities first (Companies → Resources → Projects)
2. Foreign key validation will nullify invalid references
3. Check logs for nullified reference warnings

Issue: API Filter Required Error

Symptoms:

[API_ERROR] Value cannot be null. Parameter name: filters

Cause: Entity requires a filter (Time Entries, Billing Items, Contracts, Projects)

Solution: Filters are automatically applied - this shouldn't occur. If it does, check entity-sync.ts for proper filter configuration.

Issue: Slow Sync Performance

Symptoms: Sync takes longer than expected

Solutions:

1. Reduce yearsBack parameter (e.g., from 2 to 1 year)
2. Use incremental sync instead of full sync
3. Sync entities individually instead of all at once
4. Check network connectivity and API response times

Issue: Records Not Appearing

Symptoms: Records exist in Autotask but not in database

Possible Causes:

1. Outside date range filter
2. Not active (filtered out by isActive)
3. Foreign key validation nullified references
4. Sync failed (check sync history)

Solutions:

1. Check sync history for errors
2. Verify record meets filter criteria
3. Check logs for validation warnings
4. Run full sync for the entity

---

Best Practices

1. Sync Strategy

• Initial Setup: Full sync all entities (2 years)
• Daily Maintenance: Incremental sync all entities
• Weekly/Monthly: Full sync for data consistency
• Troubleshooting: Sync selected entities as needed

2. Date Range Configuration

• Default (2 years): Good for most use cases
• 1 year: Faster syncs, less historical data
• 3-5 years: More historical data, slower syncs
• Adjust based on: Business needs, performance requirements

3. Monitoring

• Review sync history regularly
• Set up alerts for failed syncs
• Monitor sync duration trends
• Check for foreign key validation warnings

4. Dependency Management

Sync entities in this order for best results:

1. Companies
2. Resources
3. Contacts
4. Configuration Items
5. Contracts
6. Projects
7. Tickets
8. Tasks
9. Time Entries
10. Billing Items
11. Picklists

5. Error Handling

• Failed syncs are logged in sync_history table
• Check error messages for specific issues
• Re-run failed entity syncs individually
• Contact support if errors persist

---

API Reference

Full Sync

POST /api/sync/full
{
  "yearsBack": 2,
  "triggeredBy": "user@example.com"
}

Incremental Sync

POST /api/sync/incremental
{
  "triggeredBy": "user@example.com"
}

Sync Selected Entities

POST /api/sync/entity
{
  "entities": ["tickets", "tasks"],
  "yearsBack": 2,
  "triggeredBy": "user@example.com"
}

Get Sync History

GET /api/sync/history?limit=50&offset=0

Get Sync Status

GET /api/sync/status

---

Related Documentation

• Sync Interface Guide - User guide for the sync UI
• Sync Logging Improvements - Structured logging details
• Database Schema - PostgreSQL table definitions

---

Changelog

2026-01-24

• Added billing items filter requirement
• Added task foreign key validation
• Documented sync behavior comprehensively

2026-01-23

• Fixed statuses and work types sync (picklist API)
• Added structured logging
• Created sync interface documentation

---

Support

For issues or questions:

1. Check sync history in the admin interface
2. Review error messages and logs
3. Consult this documentation
4. Contact system administrator