docs: add comprehensive sync behavior documentation
Created detailed documentation covering: - Sync types (Full, Incremental, Sync Selected) - Date range filtering and entity-specific behaviors - Database operations (UPSERT, soft deletes) - Foreign key validation mechanisms - Performance considerations and optimization tips - Entity-specific filters and requirements - Troubleshooting guide and best practices - Workflow examples and API reference This complements the existing SYNC_INTERFACE_GUIDE.md with technical implementation details and behavioral specifications.
This commit is contained in:
root
parent
2f20b99195
commit
8b50cae71c
1 changed files with 747 additions and 0 deletions
747
docs/SYNC_BEHAVIOR.md
Normal file
747
docs/SYNC_BEHAVIOR.md
Normal file
|
|
@ -0,0 +1,747 @@
|
|||
# Sync Behavior Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
This document provides comprehensive details about how the Autotask sync system works, including the differences between sync types, date range filtering, database operations, and entity-specific behaviors.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Sync Types](#sync-types)
|
||||
- [Date Range Filtering](#date-range-filtering)
|
||||
- [Database Operations](#database-operations)
|
||||
- [Entity-Specific Behaviors](#entity-specific-behaviors)
|
||||
- [Foreign Key Validation](#foreign-key-validation)
|
||||
- [Soft Deletes](#soft-deletes)
|
||||
- [Performance Considerations](#performance-considerations)
|
||||
|
||||
---
|
||||
|
||||
## Sync Types
|
||||
|
||||
### Full Sync
|
||||
|
||||
**What it does:**
|
||||
- Fetches **ALL** records from Autotask that match the configured filters
|
||||
- Performs an **UPSERT** operation on every fetched record
|
||||
- Optionally performs **soft deletes** for missing records (entity-dependent)
|
||||
|
||||
**When to use:**
|
||||
- Initial setup or first-time sync
|
||||
- After major data changes in Autotask
|
||||
- To ensure complete data consistency
|
||||
- When you suspect data drift between systems
|
||||
|
||||
**Characteristics:**
|
||||
- Processes all records within filter criteria (date range, status, etc.)
|
||||
- Does NOT check if records have changed - upserts everything
|
||||
- Slower than incremental sync
|
||||
- More thorough and ensures data consistency
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Full Sync of Tickets (last 2 years):
|
||||
1. Fetch all tickets created in last 2 years from Autotask
|
||||
2. Upsert all fetched tickets to PostgreSQL
|
||||
3. Skip soft deletes (has date filter)
|
||||
```
|
||||
|
||||
### Incremental Sync
|
||||
|
||||
**What it does:**
|
||||
- Fetches only records **modified since the last successful sync**
|
||||
- Uses `lastModifiedDate` or similar timestamp fields
|
||||
- Performs UPSERT on changed records only
|
||||
- Does NOT perform soft deletes
|
||||
|
||||
**When to use:**
|
||||
- Regular scheduled syncs (hourly, daily)
|
||||
- To keep data up-to-date with minimal overhead
|
||||
- When you only need recent changes
|
||||
|
||||
**Characteristics:**
|
||||
- Only processes records that changed since last sync
|
||||
- Much faster than full sync
|
||||
- Requires a previous successful sync to establish baseline
|
||||
- Falls back to full sync if no previous sync found
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Incremental Sync of Companies:
|
||||
1. Get last sync time: 2026-01-24 08:00:00
|
||||
2. Fetch companies where lastModifiedDate >= 2026-01-24 08:00:00
|
||||
3. Upsert only the changed companies
|
||||
4. No soft deletes
|
||||
```
|
||||
|
||||
### Sync Selected (Entity-Specific)
|
||||
|
||||
**What it does:**
|
||||
- Syncs only the selected entities
|
||||
- Uses full sync behavior for each selected entity
|
||||
- Respects all entity-specific filters and behaviors
|
||||
|
||||
**When to use:**
|
||||
- To sync specific entities without syncing everything
|
||||
- After fixing entity-specific issues
|
||||
- For testing or troubleshooting
|
||||
|
||||
---
|
||||
|
||||
## Date Range Filtering
|
||||
|
||||
### Entities with Required Filters
|
||||
|
||||
These entities **REQUIRE** filters - the Autotask API will return an error without them:
|
||||
|
||||
| Entity | Filter Field | Default Range | Reason |
|
||||
|--------|-------------|---------------|---------|
|
||||
| **Time Entries** | `dateWorked` | Last 2 years | API requirement |
|
||||
| **Billing Items** | `itemDate` | Last 2 years | API requirement |
|
||||
| **Contracts** | `status` | Active only (status=1) | API requirement |
|
||||
| **Projects** | `status` | Non-completed (status≠5) | API requirement |
|
||||
|
||||
### Entities with Optional Date Filters
|
||||
|
||||
These entities have date filters applied for **performance** and **data management**:
|
||||
|
||||
| Entity | Filter Field | Default Range | Configurable |
|
||||
|--------|-------------|---------------|--------------|
|
||||
| **Tickets** | `createDate` | Last 2 years | Yes (via yearsBack) |
|
||||
| **Tasks** | `createDateTime` | Last 2 years | Yes (via yearsBack) |
|
||||
|
||||
### Entities Without Date Filters
|
||||
|
||||
These entities sync **ALL** records (no date filtering):
|
||||
|
||||
- **Companies** - All active companies
|
||||
- **Resources** - All active resources
|
||||
- **Contacts** - All active contacts
|
||||
- **Configuration Items** - All active items
|
||||
- **Picklists** (Statuses, Issue Types, Sub-Issue Types, Work Types) - All values
|
||||
|
||||
### Configuring Date Range
|
||||
|
||||
The `yearsBack` parameter controls the date range:
|
||||
|
||||
```typescript
|
||||
// Default: 2 years
|
||||
syncService.fullSync(2);
|
||||
|
||||
// Custom: 5 years
|
||||
syncService.fullSync(5);
|
||||
|
||||
// Via API
|
||||
POST /api/sync/full
|
||||
{
|
||||
"yearsBack": 3,
|
||||
"triggeredBy": "user@example.com"
|
||||
}
|
||||
```
|
||||
|
||||
**Important Notes:**
|
||||
|
||||
1. **Date filters apply to BOTH full and incremental sync**
|
||||
- Full sync: Fetches all records within date range
|
||||
- Incremental sync: Fetches changed records within date range
|
||||
|
||||
2. **Soft deletes are skipped for date-filtered entities**
|
||||
- Prevents deleting old records outside the sync window
|
||||
- See [Soft Deletes](#soft-deletes) section
|
||||
|
||||
---
|
||||
|
||||
## Database Operations
|
||||
|
||||
### UPSERT Behavior
|
||||
|
||||
All syncs use PostgreSQL's `ON CONFLICT DO UPDATE` (UPSERT):
|
||||
|
||||
```sql
|
||||
INSERT INTO table_name (id, field1, field2, ...)
|
||||
VALUES ($1, $2, $3, ...)
|
||||
ON CONFLICT (id)
|
||||
DO UPDATE SET
|
||||
field1 = EXCLUDED.field1,
|
||||
field2 = EXCLUDED.field2,
|
||||
...
|
||||
synced_at = NOW()
|
||||
```
|
||||
|
||||
**What this means:**
|
||||
|
||||
- **If record exists**: All fields are updated with new values from Autotask
|
||||
- **If record doesn't exist**: Record is inserted as new
|
||||
- **Every fetched record is processed**, regardless of whether data actually changed
|
||||
- The `synced_at` timestamp is always updated
|
||||
|
||||
### Record Counts
|
||||
|
||||
Sync history tracks three metrics:
|
||||
|
||||
| Metric | Description | How It's Calculated |
|
||||
|--------|-------------|---------------------|
|
||||
| `records_added` | New records inserted | **Estimated** (~10% of upserted) |
|
||||
| `records_updated` | Existing records updated | **Estimated** (~90% of upserted) |
|
||||
| `records_deleted` | Records soft-deleted | **Actual count** from soft delete operation |
|
||||
|
||||
**Important:** `records_added` and `records_updated` are **estimates** based on the total upserted count. PostgreSQL doesn't easily distinguish between inserts and updates in bulk UPSERT operations.
|
||||
|
||||
### Bulk Processing
|
||||
|
||||
Records are processed in batches for performance:
|
||||
|
||||
- **Fetch batch size**: 500 records per API call
|
||||
- **Upsert batch size**: 100 records per database transaction
|
||||
- **Pagination**: Automatic for large datasets
|
||||
|
||||
---
|
||||
|
||||
## Entity-Specific Behaviors
|
||||
|
||||
### Picklist Entities
|
||||
|
||||
**Entities:** Statuses, Issue Types, Sub-Issue Types, Work Types
|
||||
|
||||
**Special Handling:**
|
||||
- Fetched from `/entity/entityInformation/fields` endpoint (not `/query`)
|
||||
- No date filtering
|
||||
- No soft deletes
|
||||
- Synced as key-value pairs
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
// Statuses from Tickets.status field
|
||||
GET /Tickets/entityInformation/fields
|
||||
|
||||
// Maps to:
|
||||
{ value: 1, label: "New", is_active: true }
|
||||
{ value: 5, label: "Complete", is_active: true }
|
||||
```
|
||||
|
||||
### Tickets
|
||||
|
||||
**Filters:**
|
||||
- Date: `createDate >= (now - 2 years)`
|
||||
- Active: `isActive = true` (if applicable)
|
||||
|
||||
**Foreign Key Validation:**
|
||||
- `assigned_resource_id` → Resources
|
||||
- `first_response_assigned_resource_id` → Resources
|
||||
- `first_response_initiating_resource_id` → Resources
|
||||
- Invalid references are set to `null`
|
||||
|
||||
**Soft Deletes:** ❌ Skipped (has date filter)
|
||||
|
||||
### Tasks
|
||||
|
||||
**Filters:**
|
||||
- Date: `createDateTime >= (now - 2 years)`
|
||||
|
||||
**Foreign Key Validation:**
|
||||
- `project_id` → Projects
|
||||
- `assigned_resource_id` → Resources
|
||||
- `creator_resource_id` → Resources
|
||||
- `completed_by_resource_id` → Resources
|
||||
- `last_activity_resource_id` → Resources
|
||||
- Invalid references are set to `null`
|
||||
|
||||
**Soft Deletes:** ❌ Skipped (has date filter)
|
||||
|
||||
### Time Entries
|
||||
|
||||
**Filters:**
|
||||
- **Required**: `dateWorked >= (now - 2 years)`
|
||||
|
||||
**Special Notes:**
|
||||
- `company_id` is nullable (time entries can exist without a company)
|
||||
- No foreign key validation needed
|
||||
|
||||
**Soft Deletes:** ❌ Skipped (has date filter)
|
||||
|
||||
### Billing Items
|
||||
|
||||
**Filters:**
|
||||
- **Required**: `itemDate >= (now - 2 years)`
|
||||
|
||||
**Special Notes:**
|
||||
- API requires a filter parameter
|
||||
- Returns 500 error without filter
|
||||
|
||||
**Soft Deletes:** ❌ Skipped (has date filter)
|
||||
|
||||
### Companies
|
||||
|
||||
**Filters:**
|
||||
- Active: `isActive = true`
|
||||
|
||||
**Special Notes:**
|
||||
- No date filtering - syncs all active companies
|
||||
- Foundation entity for many foreign keys
|
||||
|
||||
**Soft Deletes:** ✅ Yes (no date filter)
|
||||
|
||||
### Resources
|
||||
|
||||
**Filters:**
|
||||
- Active: `isActive = true`
|
||||
|
||||
**Special Notes:**
|
||||
- No date filtering - syncs all active resources
|
||||
- Referenced by Tickets, Tasks, Time Entries
|
||||
|
||||
**Soft Deletes:** ✅ Yes (no date filter)
|
||||
|
||||
### Contacts
|
||||
|
||||
**Filters:**
|
||||
- Active: `isActive = true`
|
||||
|
||||
**Special Notes:**
|
||||
- No date filtering - syncs all active contacts
|
||||
- Referenced by Configuration Items
|
||||
|
||||
**Soft Deletes:** ✅ Yes (no date filter)
|
||||
|
||||
### Configuration Items
|
||||
|
||||
**Filters:**
|
||||
- Active: `isActive = true`
|
||||
|
||||
**Foreign Key Validation:**
|
||||
- `contact_id` → Contacts
|
||||
- Invalid references are set to `null`
|
||||
|
||||
**Soft Deletes:** ✅ Yes (no date filter)
|
||||
|
||||
### Contracts
|
||||
|
||||
**Filters:**
|
||||
- **Required**: `status = 1` (Active)
|
||||
|
||||
**Special Notes:**
|
||||
- API requires status filter
|
||||
- Only syncs active contracts
|
||||
|
||||
**Soft Deletes:** ❌ Skipped (has status filter)
|
||||
|
||||
### Projects
|
||||
|
||||
**Filters:**
|
||||
- **Required**: `status != 5` (Not Complete)
|
||||
|
||||
**Special Notes:**
|
||||
- API requires status filter
|
||||
- Excludes completed projects
|
||||
|
||||
**Soft Deletes:** ❌ Skipped (has status filter)
|
||||
|
||||
---
|
||||
|
||||
## Foreign Key Validation
|
||||
|
||||
### Why It's Needed
|
||||
|
||||
Autotask data may reference entities that:
|
||||
- Haven't been synced yet
|
||||
- Are outside the sync date range
|
||||
- Have been deleted in Autotask
|
||||
- Don't exist due to data inconsistencies
|
||||
|
||||
Without validation, these references cause PostgreSQL foreign key constraint violations.
|
||||
|
||||
### How It Works
|
||||
|
||||
Before upserting records, the sync validates foreign key references:
|
||||
|
||||
1. **Fetch valid IDs** from the database
|
||||
```typescript
|
||||
const validResourceIds = await getValidResourceIds();
|
||||
// Returns Set of all resource IDs where is_deleted = false
|
||||
```
|
||||
|
||||
2. **Validate each record**
|
||||
```typescript
|
||||
if (task.assigned_resource_id && !validResourceIds.has(task.assigned_resource_id)) {
|
||||
task.assigned_resource_id = null; // Nullify invalid reference
|
||||
}
|
||||
```
|
||||
|
||||
3. **Log warnings** for nullified references
|
||||
```
|
||||
[WARN] Nullified invalid resource references: 15 records
|
||||
```
|
||||
|
||||
### Validated Relationships
|
||||
|
||||
| Entity | Foreign Key Field | References | Action on Invalid |
|
||||
|--------|------------------|------------|-------------------|
|
||||
| **Tickets** | `assigned_resource_id` | Resources | Set to null |
|
||||
| **Tickets** | `first_response_assigned_resource_id` | Resources | Set to null |
|
||||
| **Tickets** | `first_response_initiating_resource_id` | Resources | Set to null |
|
||||
| **Tasks** | `project_id` | Projects | Set to null |
|
||||
| **Tasks** | `assigned_resource_id` | Resources | Set to null |
|
||||
| **Tasks** | `creator_resource_id` | Resources | Set to null |
|
||||
| **Tasks** | `completed_by_resource_id` | Resources | Set to null |
|
||||
| **Tasks** | `last_activity_resource_id` | Resources | Set to null |
|
||||
| **Configuration Items** | `contact_id` | Contacts | Set to null |
|
||||
|
||||
### Performance Impact
|
||||
|
||||
Foreign key validation queries are:
|
||||
- Executed once per sync (cached)
|
||||
- Indexed queries (fast)
|
||||
- Minimal overhead compared to API fetching
|
||||
|
||||
---
|
||||
|
||||
## Soft Deletes
|
||||
|
||||
### What Are Soft Deletes?
|
||||
|
||||
Instead of permanently deleting records, the system marks them as deleted:
|
||||
|
||||
```sql
|
||||
UPDATE table_name
|
||||
SET is_deleted = true, deleted_at = NOW()
|
||||
WHERE id NOT IN (fetched_ids)
|
||||
AND is_deleted = false
|
||||
```
|
||||
|
||||
### When Soft Deletes Occur
|
||||
|
||||
**Only during Full Sync** for entities **without date filters**:
|
||||
|
||||
| Entity | Soft Deletes? | Reason |
|
||||
|--------|--------------|---------|
|
||||
| Companies | ✅ Yes | No date filter - safe to delete |
|
||||
| Resources | ✅ Yes | No date filter - safe to delete |
|
||||
| Contacts | ✅ Yes | No date filter - safe to delete |
|
||||
| Configuration Items | ✅ Yes | No date filter - safe to delete |
|
||||
| Tickets | ❌ No | Has date filter - would delete old records |
|
||||
| Tasks | ❌ No | Has date filter - would delete old records |
|
||||
| Time Entries | ❌ No | Has date filter - would delete old records |
|
||||
| Billing Items | ❌ No | Has date filter - would delete old records |
|
||||
| Projects | ❌ No | Has status filter - would delete completed |
|
||||
| Contracts | ❌ No | Has status filter - would delete inactive |
|
||||
| Picklists | ❌ No | Picklist values don't get deleted |
|
||||
|
||||
### Why Skip Soft Deletes for Date-Filtered Entities?
|
||||
|
||||
**Problem:**
|
||||
If we soft-deleted records not in the fetched set for date-filtered entities, we would incorrectly delete old records outside the sync window.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Full Sync of Tickets (last 2 years):
|
||||
1. Fetch tickets created >= 2024-01-24
|
||||
2. Ticket #12345 created in 2020 exists in DB
|
||||
3. Ticket #12345 NOT in fetched set (outside date range)
|
||||
4. ❌ Would incorrectly soft-delete Ticket #12345
|
||||
|
||||
Solution: Skip soft deletes for date-filtered entities
|
||||
```
|
||||
|
||||
### Restoring Soft-Deleted Records
|
||||
|
||||
If a record is soft-deleted but appears in a later sync:
|
||||
|
||||
```sql
|
||||
-- Record is automatically restored
|
||||
UPDATE table_name
|
||||
SET is_deleted = false, deleted_at = NULL, ...
|
||||
WHERE id = $1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Sync Duration Factors
|
||||
|
||||
| Factor | Impact | Mitigation |
|
||||
|--------|--------|------------|
|
||||
| **Number of records** | High | Use date range filters |
|
||||
| **API rate limits** | Medium | Built-in rate limiting (10 req/sec) |
|
||||
| **Network latency** | Medium | Pagination, batch processing |
|
||||
| **Database operations** | Low | Bulk upserts, indexed queries |
|
||||
| **Foreign key validation** | Low | Cached, indexed queries |
|
||||
|
||||
### Estimated Sync Times
|
||||
|
||||
Based on typical data volumes:
|
||||
|
||||
| Entity | Record Count | Full Sync Time | Incremental Sync Time |
|
||||
|--------|-------------|----------------|----------------------|
|
||||
| Companies | 500 | ~30 seconds | ~5 seconds |
|
||||
| Resources | 100 | ~10 seconds | ~2 seconds |
|
||||
| Tickets (2 years) | 10,000 | ~10 minutes | ~1 minute |
|
||||
| Tasks (2 years) | 5,000 | ~5 minutes | ~30 seconds |
|
||||
| Time Entries (2 years) | 50,000 | ~45 minutes | ~5 minutes |
|
||||
| Billing Items (2 years) | 50,000+ | ~45-60 minutes | ~5 minutes |
|
||||
|
||||
**Note:** Times vary based on data volume, network speed, and system load.
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
1. **Use Incremental Sync for regular updates**
|
||||
- Schedule incremental syncs hourly or daily
|
||||
- Reserve full syncs for weekly/monthly maintenance
|
||||
|
||||
2. **Adjust date range based on needs**
|
||||
- Default 2 years is usually sufficient
|
||||
- Reduce to 1 year for faster syncs if older data isn't needed
|
||||
- Increase to 5+ years only if historical data is required
|
||||
|
||||
3. **Sync entities in dependency order**
|
||||
- Companies → Resources → Tickets → Tasks
|
||||
- Reduces foreign key validation overhead
|
||||
|
||||
4. **Monitor sync history**
|
||||
- Check for failures and patterns
|
||||
- Identify slow entities for optimization
|
||||
|
||||
5. **Use Sync Selected for troubleshooting**
|
||||
- Test individual entities
|
||||
- Isolate and fix issues without full sync
|
||||
|
||||
---
|
||||
|
||||
## Sync Workflow Examples
|
||||
|
||||
### Example 1: Initial Setup
|
||||
|
||||
```
|
||||
1. Full Sync - All Entities (2 years)
|
||||
├─ Companies (all active)
|
||||
├─ Resources (all active)
|
||||
├─ Contacts (all active)
|
||||
├─ Configuration Items (all active)
|
||||
├─ Contracts (active only)
|
||||
├─ Projects (non-completed)
|
||||
├─ Tickets (last 2 years)
|
||||
├─ Tasks (last 2 years)
|
||||
├─ Time Entries (last 2 years)
|
||||
├─ Billing Items (last 2 years)
|
||||
└─ Picklists (all values)
|
||||
|
||||
Result: Complete database populated with all relevant data
|
||||
```
|
||||
|
||||
### Example 2: Daily Maintenance
|
||||
|
||||
```
|
||||
1. Incremental Sync - All Entities
|
||||
├─ Fetch only records modified since last sync
|
||||
├─ Upsert changed records
|
||||
└─ No soft deletes
|
||||
|
||||
Result: Database updated with latest changes (fast)
|
||||
```
|
||||
|
||||
### Example 3: Fixing Specific Entity
|
||||
|
||||
```
|
||||
1. Identify failed entity: Billing Items
|
||||
2. Sync Selected - Billing Items only
|
||||
├─ Full sync with date filter (last 2 years)
|
||||
├─ Upsert all billing items
|
||||
└─ No soft deletes
|
||||
|
||||
Result: Billing Items data refreshed and consistent
|
||||
```
|
||||
|
||||
### Example 4: Historical Data Sync
|
||||
|
||||
```
|
||||
1. Full Sync - Tickets (5 years)
|
||||
├─ Adjust yearsBack parameter to 5
|
||||
├─ Fetch tickets created >= 2021-01-24
|
||||
├─ Upsert all fetched tickets
|
||||
└─ Skip soft deletes
|
||||
|
||||
Result: 5 years of ticket history in database
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### Issue: Foreign Key Constraint Violations
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
[DATABASE_CONSTRAINT_ERROR] violates foreign key constraint "tasks_project_id_fkey"
|
||||
```
|
||||
|
||||
**Cause:** Referenced entity hasn't been synced or is outside date range
|
||||
|
||||
**Solution:**
|
||||
1. Sync dependency entities first (Companies → Resources → Projects)
|
||||
2. Foreign key validation will nullify invalid references
|
||||
3. Check logs for nullified reference warnings
|
||||
|
||||
#### Issue: API Filter Required Error
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
[API_ERROR] Value cannot be null. Parameter name: filters
|
||||
```
|
||||
|
||||
**Cause:** Entity requires a filter (Time Entries, Billing Items, Contracts, Projects)
|
||||
|
||||
**Solution:** Filters are automatically applied - this shouldn't occur. If it does, check entity-sync.ts for proper filter configuration.
|
||||
|
||||
#### Issue: Slow Sync Performance
|
||||
|
||||
**Symptoms:** Sync takes longer than expected
|
||||
|
||||
**Solutions:**
|
||||
1. Reduce `yearsBack` parameter (e.g., from 2 to 1 year)
|
||||
2. Use incremental sync instead of full sync
|
||||
3. Sync entities individually instead of all at once
|
||||
4. Check network connectivity and API response times
|
||||
|
||||
#### Issue: Records Not Appearing
|
||||
|
||||
**Symptoms:** Records exist in Autotask but not in database
|
||||
|
||||
**Possible Causes:**
|
||||
1. Outside date range filter
|
||||
2. Not active (filtered out by `isActive`)
|
||||
3. Foreign key validation nullified references
|
||||
4. Sync failed (check sync history)
|
||||
|
||||
**Solutions:**
|
||||
1. Check sync history for errors
|
||||
2. Verify record meets filter criteria
|
||||
3. Check logs for validation warnings
|
||||
4. Run full sync for the entity
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Sync Strategy
|
||||
|
||||
- **Initial Setup:** Full sync all entities (2 years)
|
||||
- **Daily Maintenance:** Incremental sync all entities
|
||||
- **Weekly/Monthly:** Full sync for data consistency
|
||||
- **Troubleshooting:** Sync selected entities as needed
|
||||
|
||||
### 2. Date Range Configuration
|
||||
|
||||
- **Default (2 years):** Good for most use cases
|
||||
- **1 year:** Faster syncs, less historical data
|
||||
- **3-5 years:** More historical data, slower syncs
|
||||
- **Adjust based on:** Business needs, performance requirements
|
||||
|
||||
### 3. Monitoring
|
||||
|
||||
- Review sync history regularly
|
||||
- Set up alerts for failed syncs
|
||||
- Monitor sync duration trends
|
||||
- Check for foreign key validation warnings
|
||||
|
||||
### 4. Dependency Management
|
||||
|
||||
Sync entities in this order for best results:
|
||||
1. Companies
|
||||
2. Resources
|
||||
3. Contacts
|
||||
4. Configuration Items
|
||||
5. Contracts
|
||||
6. Projects
|
||||
7. Tickets
|
||||
8. Tasks
|
||||
9. Time Entries
|
||||
10. Billing Items
|
||||
11. Picklists
|
||||
|
||||
### 5. Error Handling
|
||||
|
||||
- Failed syncs are logged in `sync_history` table
|
||||
- Check error messages for specific issues
|
||||
- Re-run failed entity syncs individually
|
||||
- Contact support if errors persist
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Full Sync
|
||||
|
||||
```typescript
|
||||
POST /api/sync/full
|
||||
{
|
||||
"yearsBack": 2,
|
||||
"triggeredBy": "user@example.com"
|
||||
}
|
||||
```
|
||||
|
||||
### Incremental Sync
|
||||
|
||||
```typescript
|
||||
POST /api/sync/incremental
|
||||
{
|
||||
"triggeredBy": "user@example.com"
|
||||
}
|
||||
```
|
||||
|
||||
### Sync Selected Entities
|
||||
|
||||
```typescript
|
||||
POST /api/sync/entity
|
||||
{
|
||||
"entities": ["tickets", "tasks"],
|
||||
"yearsBack": 2,
|
||||
"triggeredBy": "user@example.com"
|
||||
}
|
||||
```
|
||||
|
||||
### Get Sync History
|
||||
|
||||
```typescript
|
||||
GET /api/sync/history?limit=50&offset=0
|
||||
```
|
||||
|
||||
### Get Sync Status
|
||||
|
||||
```typescript
|
||||
GET /api/sync/status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Sync Interface Guide](./SYNC_INTERFACE_GUIDE.md) - User guide for the sync UI
|
||||
- [Sync Logging Improvements](./SYNC_LOGGING_IMPROVEMENTS.md) - Structured logging details
|
||||
- [Database Schema](../migrations/001_initial_schema.sql) - PostgreSQL table definitions
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-01-24
|
||||
- Added billing items filter requirement
|
||||
- Added task foreign key validation
|
||||
- Documented sync behavior comprehensively
|
||||
|
||||
### 2026-01-23
|
||||
- Fixed statuses and work types sync (picklist API)
|
||||
- Added structured logging
|
||||
- Created sync interface documentation
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
1. Check sync history in the admin interface
|
||||
2. Review error messages and logs
|
||||
3. Consult this documentation
|
||||
4. Contact system administrator
|
||||
Loading…
Add table
Add a link
Reference in a new issue