wulf-pulse/docs/SYNC_LOGGING_IMPROVEMENTS.md

Sync Logging Improvements

Overview

This document describes the structured logging improvements made to the sync system to address inconsistent logging and improve debugging capabilities.

Changes Made

1. Structured Logging Utility (lib/utils/sync-logger.ts)

Created a comprehensive logging utility with:

• Log Levels: DEBUG, INFO, WARN, ERROR

• Error Categorization: Automatic categorization of errors into types:

  • NETWORK_ERROR: Connection issues, timeouts
  • AUTH_ERROR: Authentication/authorization failures
  • RATE_LIMIT_ERROR: API rate limiting
  • DATABASE_CONSTRAINT_ERROR: Foreign key violations, duplicates
  • DATABASE_ERROR: General database errors
  • API_ERROR: API-specific errors
  • MAPPING_ERROR: Data transformation failures
  • VALIDATION_ERROR: Data validation issues
  • UNKNOWN: Uncategorized errors

• Sync Phases: Track operations through distinct phases:

  • INITIALIZING: Setup and configuration
  • FETCHING: Retrieving data from Autotask API
  • MAPPING: Transforming data to database schema
  • VALIDATING: Checking data integrity
  • UPSERTING: Writing to database
  • DELETING: Soft-deleting missing records
  • COMPLETING: Finalizing sync

• Contextual Logging: Every log entry includes:

  • Timestamp (ISO 8601)
  • Log level
  • Sync ID
  • Entity type
  • Current phase
  • Record counts
  • Duration metrics
  • Error category (for errors)

2. Updated sync-service.ts

Before:

console.log(`Starting ${config.syncType} sync: ${syncId}`);
console.error(`✗ Failed to sync ${entityName}:`);
console.error(`  Error: ${errorMessage}`);

After:

const syncLogger = this.logger.child({ syncId, syncType: config.syncType });
syncLogger.info(`Starting ${config.syncType} sync`, {
  entities: config.entities.map(e => getEntityDisplayName(e)).join(', '),
  triggeredBy: config.triggeredBy,
});

entityLogger.error(`Failed to sync ${entityName}`, {
  errorCategory,
  syncType: config.syncType,
}, err);

Key Improvements:

• Consistent error categorization across all error paths
• Proper error logging for sync history update failures (previously swallowed)
• Structured context in all log statements
• Child loggers for entity-specific operations

3. Updated entity-sync.ts

Before:

console.log(`[${entity}] Starting sync (${isIncremental ? 'incremental' : 'full'})`);
console.error(`[${entity}] API fetch failed:`, errorMessage);
console.warn(`[${entity}] Warning: All records failed mapping validation`);

After:

const entityLogger = this.logger.child({ syncId: trackingId, entityType: entity });
entityLogger.phase(SyncPhase.INITIALIZING);
entityLogger.phase(SyncPhase.FETCHING, 'Fetching records from Autotask API');
entityLogger.error('API fetch failed', { phase: SyncPhase.FETCHING, errorCategory }, err);
entityLogger.warn('All records failed mapping validation', { originalCount: autotaskRecords.length });

Key Improvements:

• Phase tracking throughout sync lifecycle
• Timing information for each operation
• Detailed context for warnings (record counts, sample IDs)
• Consistent error handling with categorization
• Better debugging information (sample records, field keys)

4. Sync Failure Analysis Script

Created scripts/analyze-sync-failures.ts to query and analyze sync history:

Features:

• Query sync history by date range, entity type, or status
• Generate failure summaries by entity
• Categorize errors automatically
• Calculate success rates and statistics
• Display detailed sync records with timing

Usage Examples:

# Show failures from last 7 days
npx tsx scripts/analyze-sync-failures.ts

# Show all syncs from yesterday
npx tsx scripts/analyze-sync-failures.ts --days 1 --all

# Show ticket sync failures from last 30 days
npx tsx scripts/analyze-sync-failures.ts --days 30 --entity tickets

# Show all completed syncs
npx tsx scripts/analyze-sync-failures.ts --status completed --all

Benefits

1. Consistent Logging Format

All sync operations now use the same structured format, making logs easier to parse and analyze.

2. Better Error Categorization

Errors are automatically categorized, making it easier to identify patterns:

• Network issues vs. API errors
• Database constraints vs. mapping failures
• Authentication problems vs. rate limiting

3. Phase Tracking

Know exactly where a sync failed:

• Did it fail during API fetch?
• Was it a mapping error?
• Did the database upsert fail?

4. Improved Debugging

• Sample data logged for complex entities
• Timing information for performance analysis
• Record counts at each stage
• Detailed context for warnings

5. No More Swallowed Errors

Previously, sync history update failures were caught but not logged. Now they're properly tracked.

6. Easy Failure Analysis

The analysis script provides:

• Quick overview of sync health
• Failure trends by entity
• Common error patterns
• Success rate metrics

Log Output Examples

Successful Sync

[2026-01-23T12:21:00.000Z] [INFO] [syncId=sync_1234, entity=tickets, phase=initializing] Starting tickets sync (full)
[2026-01-23T12:21:00.100Z] [INFO] [syncId=sync_1234, entity=tickets, phase=fetching] Fetching records from Autotask API
[2026-01-23T12:21:05.200Z] [INFO] [syncId=sync_1234, entity=tickets, records=150] Fetched records from Autotask
[2026-01-23T12:21:05.300Z] [INFO] [syncId=sync_1234, entity=tickets, phase=mapping] Mapping 150 records to database schema
[2026-01-23T12:21:05.500Z] [INFO] [syncId=sync_1234, entity=tickets, mappedCount=148] Successfully mapped records
[2026-01-23T12:21:05.600Z] [INFO] [syncId=sync_1234, entity=tickets, phase=upserting] Upserting records to PostgreSQL
[2026-01-23T12:21:06.800Z] [INFO] [syncId=sync_1234, entity=tickets, upsertedCount=148] Upserted records to PostgreSQL
[2026-01-23T12:21:06.900Z] [INFO] [syncId=sync_1234, entity=tickets, duration=6900ms] Completed tickets sync

Failed Sync with Error Details

[2026-01-23T12:25:00.000Z] [INFO] [syncId=sync_5678, entity=contacts, phase=initializing] Starting contacts sync (full)
[2026-01-23T12:25:00.100Z] [INFO] [syncId=sync_5678, entity=contacts, phase=fetching] Fetching records from Autotask API
[2026-01-23T12:25:10.200Z] [ERROR] [syncId=sync_5678, entity=contacts, phase=fetching, errorCategory=NETWORK_ERROR] API fetch failed
  Error: ETIMEDOUT: Connection timed out
  Category: NETWORK_ERROR
  Stack: Error: ETIMEDOUT...
[2026-01-23T12:25:10.300Z] [ERROR] [syncId=sync_5678, entity=contacts, duration=10300ms, errorCategory=NETWORK_ERROR] Failed contacts sync

Migration Guide

For Developers

If you're adding new sync operations or modifying existing ones:

1. Import the logger:

   import { createSyncLogger, SyncPhase, categorizeError } from '../utils/sync-logger';
   

2. Create a logger instance:

   const logger = createSyncLogger({ component: 'MyComponent' });
   

3. Use child loggers for operations:

   const operationLogger = logger.child({ syncId, entityType });
   

4. Track phases:

   operationLogger.phase(SyncPhase.FETCHING, 'Fetching data');
   

5. Log with context:

   operationLogger.info('Operation completed', { recordCount: 100 });
   operationLogger.error('Operation failed', { phase: SyncPhase.MAPPING }, error);
   

6. Use timing helpers:

   const startTime = logger.start('My operation');
   // ... do work ...
   logger.complete('My operation', startTime, { recordCount: 100 });
   // or on error:
   logger.fail('My operation', startTime, error);
   

Analyzing Yesterday's Sync

To investigate the sync issues from yesterday:

# Show all failures from yesterday
npx tsx scripts/analyze-sync-failures.ts --days 1

# Show all syncs (including successful) from yesterday
npx tsx scripts/analyze-sync-failures.ts --days 1 --all

# Check specific entity
npx tsx scripts/analyze-sync-failures.ts --days 1 --entity tickets

The script will show:

• Which entities failed
• Error categories
• Timing information
• Success rates
• Common error patterns

Future Enhancements

Potential improvements for consideration:

1. Log Aggregation: Send logs to a centralized logging service (e.g., Datadog, CloudWatch)
2. Alerting: Automatic alerts on consecutive failures
3. Metrics Dashboard: Real-time sync health monitoring
4. Log Rotation: Automatic cleanup of old logs
5. JSON Output: Option for machine-readable JSON logs
6. Performance Profiling: Detailed timing breakdown for optimization

Related Files

• lib/utils/sync-logger.ts - Structured logging utility
• lib/services/sync-service.ts - Main sync orchestration
• lib/services/entity-sync.ts - Entity-specific sync logic
• scripts/analyze-sync-failures.ts - Failure analysis tool
• lib/types/sync.ts - Sync type definitions