From 8b50cae71c57a588266e1d3ccb2909d4b29daf3f Mon Sep 17 00:00:00 2001 From: root Date: Sat, 24 Jan 2026 08:43:35 -0500 Subject: [PATCH] docs: add comprehensive sync behavior documentation Created detailed documentation covering: - Sync types (Full, Incremental, Sync Selected) - Date range filtering and entity-specific behaviors - Database operations (UPSERT, soft deletes) - Foreign key validation mechanisms - Performance considerations and optimization tips - Entity-specific filters and requirements - Troubleshooting guide and best practices - Workflow examples and API reference This complements the existing SYNC_INTERFACE_GUIDE.md with technical implementation details and behavioral specifications. --- docs/SYNC_BEHAVIOR.md | 747 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 747 insertions(+) create mode 100644 docs/SYNC_BEHAVIOR.md diff --git a/docs/SYNC_BEHAVIOR.md b/docs/SYNC_BEHAVIOR.md new file mode 100644 index 0000000..684e1c3 --- /dev/null +++ b/docs/SYNC_BEHAVIOR.md @@ -0,0 +1,747 @@ +# Sync Behavior Documentation + +## Overview + +This document provides comprehensive details about how the Autotask sync system works, including the differences between sync types, date range filtering, database operations, and entity-specific behaviors. + +--- + +## Table of Contents + +- [Sync Types](#sync-types) +- [Date Range Filtering](#date-range-filtering) +- [Database Operations](#database-operations) +- [Entity-Specific Behaviors](#entity-specific-behaviors) +- [Foreign Key Validation](#foreign-key-validation) +- [Soft Deletes](#soft-deletes) +- [Performance Considerations](#performance-considerations) + +--- + +## Sync Types + +### Full Sync + +**What it does:** +- Fetches **ALL** records from Autotask that match the configured filters +- Performs an **UPSERT** operation on every fetched record +- Optionally performs **soft deletes** for missing records (entity-dependent) + +**When to use:** +- Initial setup or first-time sync +- After major data changes in Autotask +- To ensure complete data consistency +- When you suspect data drift between systems + +**Characteristics:** +- Processes all records within filter criteria (date range, status, etc.) +- Does NOT check if records have changed - upserts everything +- Slower than incremental sync +- More thorough and ensures data consistency + +**Example:** +``` +Full Sync of Tickets (last 2 years): +1. Fetch all tickets created in last 2 years from Autotask +2. Upsert all fetched tickets to PostgreSQL +3. Skip soft deletes (has date filter) +``` + +### Incremental Sync + +**What it does:** +- Fetches only records **modified since the last successful sync** +- Uses `lastModifiedDate` or similar timestamp fields +- Performs UPSERT on changed records only +- Does NOT perform soft deletes + +**When to use:** +- Regular scheduled syncs (hourly, daily) +- To keep data up-to-date with minimal overhead +- When you only need recent changes + +**Characteristics:** +- Only processes records that changed since last sync +- Much faster than full sync +- Requires a previous successful sync to establish baseline +- Falls back to full sync if no previous sync found + +**Example:** +``` +Incremental Sync of Companies: +1. Get last sync time: 2026-01-24 08:00:00 +2. Fetch companies where lastModifiedDate >= 2026-01-24 08:00:00 +3. Upsert only the changed companies +4. No soft deletes +``` + +### Sync Selected (Entity-Specific) + +**What it does:** +- Syncs only the selected entities +- Uses full sync behavior for each selected entity +- Respects all entity-specific filters and behaviors + +**When to use:** +- To sync specific entities without syncing everything +- After fixing entity-specific issues +- For testing or troubleshooting + +--- + +## Date Range Filtering + +### Entities with Required Filters + +These entities **REQUIRE** filters - the Autotask API will return an error without them: + +| Entity | Filter Field | Default Range | Reason | +|--------|-------------|---------------|---------| +| **Time Entries** | `dateWorked` | Last 2 years | API requirement | +| **Billing Items** | `itemDate` | Last 2 years | API requirement | +| **Contracts** | `status` | Active only (status=1) | API requirement | +| **Projects** | `status` | Non-completed (status≠5) | API requirement | + +### Entities with Optional Date Filters + +These entities have date filters applied for **performance** and **data management**: + +| Entity | Filter Field | Default Range | Configurable | +|--------|-------------|---------------|--------------| +| **Tickets** | `createDate` | Last 2 years | Yes (via yearsBack) | +| **Tasks** | `createDateTime` | Last 2 years | Yes (via yearsBack) | + +### Entities Without Date Filters + +These entities sync **ALL** records (no date filtering): + +- **Companies** - All active companies +- **Resources** - All active resources +- **Contacts** - All active contacts +- **Configuration Items** - All active items +- **Picklists** (Statuses, Issue Types, Sub-Issue Types, Work Types) - All values + +### Configuring Date Range + +The `yearsBack` parameter controls the date range: + +```typescript +// Default: 2 years +syncService.fullSync(2); + +// Custom: 5 years +syncService.fullSync(5); + +// Via API +POST /api/sync/full +{ + "yearsBack": 3, + "triggeredBy": "user@example.com" +} +``` + +**Important Notes:** + +1. **Date filters apply to BOTH full and incremental sync** + - Full sync: Fetches all records within date range + - Incremental sync: Fetches changed records within date range + +2. **Soft deletes are skipped for date-filtered entities** + - Prevents deleting old records outside the sync window + - See [Soft Deletes](#soft-deletes) section + +--- + +## Database Operations + +### UPSERT Behavior + +All syncs use PostgreSQL's `ON CONFLICT DO UPDATE` (UPSERT): + +```sql +INSERT INTO table_name (id, field1, field2, ...) +VALUES ($1, $2, $3, ...) +ON CONFLICT (id) +DO UPDATE SET + field1 = EXCLUDED.field1, + field2 = EXCLUDED.field2, + ... + synced_at = NOW() +``` + +**What this means:** + +- **If record exists**: All fields are updated with new values from Autotask +- **If record doesn't exist**: Record is inserted as new +- **Every fetched record is processed**, regardless of whether data actually changed +- The `synced_at` timestamp is always updated + +### Record Counts + +Sync history tracks three metrics: + +| Metric | Description | How It's Calculated | +|--------|-------------|---------------------| +| `records_added` | New records inserted | **Estimated** (~10% of upserted) | +| `records_updated` | Existing records updated | **Estimated** (~90% of upserted) | +| `records_deleted` | Records soft-deleted | **Actual count** from soft delete operation | + +**Important:** `records_added` and `records_updated` are **estimates** based on the total upserted count. PostgreSQL doesn't easily distinguish between inserts and updates in bulk UPSERT operations. + +### Bulk Processing + +Records are processed in batches for performance: + +- **Fetch batch size**: 500 records per API call +- **Upsert batch size**: 100 records per database transaction +- **Pagination**: Automatic for large datasets + +--- + +## Entity-Specific Behaviors + +### Picklist Entities + +**Entities:** Statuses, Issue Types, Sub-Issue Types, Work Types + +**Special Handling:** +- Fetched from `/entity/entityInformation/fields` endpoint (not `/query`) +- No date filtering +- No soft deletes +- Synced as key-value pairs + +**Example:** +```typescript +// Statuses from Tickets.status field +GET /Tickets/entityInformation/fields + +// Maps to: +{ value: 1, label: "New", is_active: true } +{ value: 5, label: "Complete", is_active: true } +``` + +### Tickets + +**Filters:** +- Date: `createDate >= (now - 2 years)` +- Active: `isActive = true` (if applicable) + +**Foreign Key Validation:** +- `assigned_resource_id` → Resources +- `first_response_assigned_resource_id` → Resources +- `first_response_initiating_resource_id` → Resources +- Invalid references are set to `null` + +**Soft Deletes:** ❌ Skipped (has date filter) + +### Tasks + +**Filters:** +- Date: `createDateTime >= (now - 2 years)` + +**Foreign Key Validation:** +- `project_id` → Projects +- `assigned_resource_id` → Resources +- `creator_resource_id` → Resources +- `completed_by_resource_id` → Resources +- `last_activity_resource_id` → Resources +- Invalid references are set to `null` + +**Soft Deletes:** ❌ Skipped (has date filter) + +### Time Entries + +**Filters:** +- **Required**: `dateWorked >= (now - 2 years)` + +**Special Notes:** +- `company_id` is nullable (time entries can exist without a company) +- No foreign key validation needed + +**Soft Deletes:** ❌ Skipped (has date filter) + +### Billing Items + +**Filters:** +- **Required**: `itemDate >= (now - 2 years)` + +**Special Notes:** +- API requires a filter parameter +- Returns 500 error without filter + +**Soft Deletes:** ❌ Skipped (has date filter) + +### Companies + +**Filters:** +- Active: `isActive = true` + +**Special Notes:** +- No date filtering - syncs all active companies +- Foundation entity for many foreign keys + +**Soft Deletes:** ✅ Yes (no date filter) + +### Resources + +**Filters:** +- Active: `isActive = true` + +**Special Notes:** +- No date filtering - syncs all active resources +- Referenced by Tickets, Tasks, Time Entries + +**Soft Deletes:** ✅ Yes (no date filter) + +### Contacts + +**Filters:** +- Active: `isActive = true` + +**Special Notes:** +- No date filtering - syncs all active contacts +- Referenced by Configuration Items + +**Soft Deletes:** ✅ Yes (no date filter) + +### Configuration Items + +**Filters:** +- Active: `isActive = true` + +**Foreign Key Validation:** +- `contact_id` → Contacts +- Invalid references are set to `null` + +**Soft Deletes:** ✅ Yes (no date filter) + +### Contracts + +**Filters:** +- **Required**: `status = 1` (Active) + +**Special Notes:** +- API requires status filter +- Only syncs active contracts + +**Soft Deletes:** ❌ Skipped (has status filter) + +### Projects + +**Filters:** +- **Required**: `status != 5` (Not Complete) + +**Special Notes:** +- API requires status filter +- Excludes completed projects + +**Soft Deletes:** ❌ Skipped (has status filter) + +--- + +## Foreign Key Validation + +### Why It's Needed + +Autotask data may reference entities that: +- Haven't been synced yet +- Are outside the sync date range +- Have been deleted in Autotask +- Don't exist due to data inconsistencies + +Without validation, these references cause PostgreSQL foreign key constraint violations. + +### How It Works + +Before upserting records, the sync validates foreign key references: + +1. **Fetch valid IDs** from the database + ```typescript + const validResourceIds = await getValidResourceIds(); + // Returns Set of all resource IDs where is_deleted = false + ``` + +2. **Validate each record** + ```typescript + if (task.assigned_resource_id && !validResourceIds.has(task.assigned_resource_id)) { + task.assigned_resource_id = null; // Nullify invalid reference + } + ``` + +3. **Log warnings** for nullified references + ``` + [WARN] Nullified invalid resource references: 15 records + ``` + +### Validated Relationships + +| Entity | Foreign Key Field | References | Action on Invalid | +|--------|------------------|------------|-------------------| +| **Tickets** | `assigned_resource_id` | Resources | Set to null | +| **Tickets** | `first_response_assigned_resource_id` | Resources | Set to null | +| **Tickets** | `first_response_initiating_resource_id` | Resources | Set to null | +| **Tasks** | `project_id` | Projects | Set to null | +| **Tasks** | `assigned_resource_id` | Resources | Set to null | +| **Tasks** | `creator_resource_id` | Resources | Set to null | +| **Tasks** | `completed_by_resource_id` | Resources | Set to null | +| **Tasks** | `last_activity_resource_id` | Resources | Set to null | +| **Configuration Items** | `contact_id` | Contacts | Set to null | + +### Performance Impact + +Foreign key validation queries are: +- Executed once per sync (cached) +- Indexed queries (fast) +- Minimal overhead compared to API fetching + +--- + +## Soft Deletes + +### What Are Soft Deletes? + +Instead of permanently deleting records, the system marks them as deleted: + +```sql +UPDATE table_name +SET is_deleted = true, deleted_at = NOW() +WHERE id NOT IN (fetched_ids) + AND is_deleted = false +``` + +### When Soft Deletes Occur + +**Only during Full Sync** for entities **without date filters**: + +| Entity | Soft Deletes? | Reason | +|--------|--------------|---------| +| Companies | ✅ Yes | No date filter - safe to delete | +| Resources | ✅ Yes | No date filter - safe to delete | +| Contacts | ✅ Yes | No date filter - safe to delete | +| Configuration Items | ✅ Yes | No date filter - safe to delete | +| Tickets | ❌ No | Has date filter - would delete old records | +| Tasks | ❌ No | Has date filter - would delete old records | +| Time Entries | ❌ No | Has date filter - would delete old records | +| Billing Items | ❌ No | Has date filter - would delete old records | +| Projects | ❌ No | Has status filter - would delete completed | +| Contracts | ❌ No | Has status filter - would delete inactive | +| Picklists | ❌ No | Picklist values don't get deleted | + +### Why Skip Soft Deletes for Date-Filtered Entities? + +**Problem:** +If we soft-deleted records not in the fetched set for date-filtered entities, we would incorrectly delete old records outside the sync window. + +**Example:** +``` +Full Sync of Tickets (last 2 years): +1. Fetch tickets created >= 2024-01-24 +2. Ticket #12345 created in 2020 exists in DB +3. Ticket #12345 NOT in fetched set (outside date range) +4. ❌ Would incorrectly soft-delete Ticket #12345 + +Solution: Skip soft deletes for date-filtered entities +``` + +### Restoring Soft-Deleted Records + +If a record is soft-deleted but appears in a later sync: + +```sql +-- Record is automatically restored +UPDATE table_name +SET is_deleted = false, deleted_at = NULL, ... +WHERE id = $1 +``` + +--- + +## Performance Considerations + +### Sync Duration Factors + +| Factor | Impact | Mitigation | +|--------|--------|------------| +| **Number of records** | High | Use date range filters | +| **API rate limits** | Medium | Built-in rate limiting (10 req/sec) | +| **Network latency** | Medium | Pagination, batch processing | +| **Database operations** | Low | Bulk upserts, indexed queries | +| **Foreign key validation** | Low | Cached, indexed queries | + +### Estimated Sync Times + +Based on typical data volumes: + +| Entity | Record Count | Full Sync Time | Incremental Sync Time | +|--------|-------------|----------------|----------------------| +| Companies | 500 | ~30 seconds | ~5 seconds | +| Resources | 100 | ~10 seconds | ~2 seconds | +| Tickets (2 years) | 10,000 | ~10 minutes | ~1 minute | +| Tasks (2 years) | 5,000 | ~5 minutes | ~30 seconds | +| Time Entries (2 years) | 50,000 | ~45 minutes | ~5 minutes | +| Billing Items (2 years) | 50,000+ | ~45-60 minutes | ~5 minutes | + +**Note:** Times vary based on data volume, network speed, and system load. + +### Optimization Tips + +1. **Use Incremental Sync for regular updates** + - Schedule incremental syncs hourly or daily + - Reserve full syncs for weekly/monthly maintenance + +2. **Adjust date range based on needs** + - Default 2 years is usually sufficient + - Reduce to 1 year for faster syncs if older data isn't needed + - Increase to 5+ years only if historical data is required + +3. **Sync entities in dependency order** + - Companies → Resources → Tickets → Tasks + - Reduces foreign key validation overhead + +4. **Monitor sync history** + - Check for failures and patterns + - Identify slow entities for optimization + +5. **Use Sync Selected for troubleshooting** + - Test individual entities + - Isolate and fix issues without full sync + +--- + +## Sync Workflow Examples + +### Example 1: Initial Setup + +``` +1. Full Sync - All Entities (2 years) + ├─ Companies (all active) + ├─ Resources (all active) + ├─ Contacts (all active) + ├─ Configuration Items (all active) + ├─ Contracts (active only) + ├─ Projects (non-completed) + ├─ Tickets (last 2 years) + ├─ Tasks (last 2 years) + ├─ Time Entries (last 2 years) + ├─ Billing Items (last 2 years) + └─ Picklists (all values) + +Result: Complete database populated with all relevant data +``` + +### Example 2: Daily Maintenance + +``` +1. Incremental Sync - All Entities + ├─ Fetch only records modified since last sync + ├─ Upsert changed records + └─ No soft deletes + +Result: Database updated with latest changes (fast) +``` + +### Example 3: Fixing Specific Entity + +``` +1. Identify failed entity: Billing Items +2. Sync Selected - Billing Items only + ├─ Full sync with date filter (last 2 years) + ├─ Upsert all billing items + └─ No soft deletes + +Result: Billing Items data refreshed and consistent +``` + +### Example 4: Historical Data Sync + +``` +1. Full Sync - Tickets (5 years) + ├─ Adjust yearsBack parameter to 5 + ├─ Fetch tickets created >= 2021-01-24 + ├─ Upsert all fetched tickets + └─ Skip soft deletes + +Result: 5 years of ticket history in database +``` + +--- + +## Troubleshooting + +### Common Issues + +#### Issue: Foreign Key Constraint Violations + +**Symptoms:** +``` +[DATABASE_CONSTRAINT_ERROR] violates foreign key constraint "tasks_project_id_fkey" +``` + +**Cause:** Referenced entity hasn't been synced or is outside date range + +**Solution:** +1. Sync dependency entities first (Companies → Resources → Projects) +2. Foreign key validation will nullify invalid references +3. Check logs for nullified reference warnings + +#### Issue: API Filter Required Error + +**Symptoms:** +``` +[API_ERROR] Value cannot be null. Parameter name: filters +``` + +**Cause:** Entity requires a filter (Time Entries, Billing Items, Contracts, Projects) + +**Solution:** Filters are automatically applied - this shouldn't occur. If it does, check entity-sync.ts for proper filter configuration. + +#### Issue: Slow Sync Performance + +**Symptoms:** Sync takes longer than expected + +**Solutions:** +1. Reduce `yearsBack` parameter (e.g., from 2 to 1 year) +2. Use incremental sync instead of full sync +3. Sync entities individually instead of all at once +4. Check network connectivity and API response times + +#### Issue: Records Not Appearing + +**Symptoms:** Records exist in Autotask but not in database + +**Possible Causes:** +1. Outside date range filter +2. Not active (filtered out by `isActive`) +3. Foreign key validation nullified references +4. Sync failed (check sync history) + +**Solutions:** +1. Check sync history for errors +2. Verify record meets filter criteria +3. Check logs for validation warnings +4. Run full sync for the entity + +--- + +## Best Practices + +### 1. Sync Strategy + +- **Initial Setup:** Full sync all entities (2 years) +- **Daily Maintenance:** Incremental sync all entities +- **Weekly/Monthly:** Full sync for data consistency +- **Troubleshooting:** Sync selected entities as needed + +### 2. Date Range Configuration + +- **Default (2 years):** Good for most use cases +- **1 year:** Faster syncs, less historical data +- **3-5 years:** More historical data, slower syncs +- **Adjust based on:** Business needs, performance requirements + +### 3. Monitoring + +- Review sync history regularly +- Set up alerts for failed syncs +- Monitor sync duration trends +- Check for foreign key validation warnings + +### 4. Dependency Management + +Sync entities in this order for best results: +1. Companies +2. Resources +3. Contacts +4. Configuration Items +5. Contracts +6. Projects +7. Tickets +8. Tasks +9. Time Entries +10. Billing Items +11. Picklists + +### 5. Error Handling + +- Failed syncs are logged in `sync_history` table +- Check error messages for specific issues +- Re-run failed entity syncs individually +- Contact support if errors persist + +--- + +## API Reference + +### Full Sync + +```typescript +POST /api/sync/full +{ + "yearsBack": 2, + "triggeredBy": "user@example.com" +} +``` + +### Incremental Sync + +```typescript +POST /api/sync/incremental +{ + "triggeredBy": "user@example.com" +} +``` + +### Sync Selected Entities + +```typescript +POST /api/sync/entity +{ + "entities": ["tickets", "tasks"], + "yearsBack": 2, + "triggeredBy": "user@example.com" +} +``` + +### Get Sync History + +```typescript +GET /api/sync/history?limit=50&offset=0 +``` + +### Get Sync Status + +```typescript +GET /api/sync/status +``` + +--- + +## Related Documentation + +- [Sync Interface Guide](./SYNC_INTERFACE_GUIDE.md) - User guide for the sync UI +- [Sync Logging Improvements](./SYNC_LOGGING_IMPROVEMENTS.md) - Structured logging details +- [Database Schema](../migrations/001_initial_schema.sql) - PostgreSQL table definitions + +--- + +## Changelog + +### 2026-01-24 +- Added billing items filter requirement +- Added task foreign key validation +- Documented sync behavior comprehensively + +### 2026-01-23 +- Fixed statuses and work types sync (picklist API) +- Added structured logging +- Created sync interface documentation + +--- + +## Support + +For issues or questions: +1. Check sync history in the admin interface +2. Review error messages and logs +3. Consult this documentation +4. Contact system administrator