wulf-pulse/docs/SYNC_LOGGING_IMPROVEMENTS.md

# Sync Logging Improvements

## Overview

This document describes the structured logging improvements made to the sync system to address inconsistent logging and improve debugging capabilities.

## Changes Made

### 1. Structured Logging Utility (`lib/utils/sync-logger.ts`)

Created a comprehensive logging utility with:

- **Log Levels**: DEBUG, INFO, WARN, ERROR
- **Error Categorization**: Automatic categorization of errors into types:
  - `NETWORK_ERROR`: Connection issues, timeouts
  - `AUTH_ERROR`: Authentication/authorization failures
  - `RATE_LIMIT_ERROR`: API rate limiting
  - `DATABASE_CONSTRAINT_ERROR`: Foreign key violations, duplicates
  - `DATABASE_ERROR`: General database errors
  - `API_ERROR`: API-specific errors
  - `MAPPING_ERROR`: Data transformation failures
  - `VALIDATION_ERROR`: Data validation issues
  - `UNKNOWN`: Uncategorized errors

- **Sync Phases**: Track operations through distinct phases:
  - `INITIALIZING`: Setup and configuration
  - `FETCHING`: Retrieving data from Autotask API
  - `MAPPING`: Transforming data to database schema
  - `VALIDATING`: Checking data integrity
  - `UPSERTING`: Writing to database
  - `DELETING`: Soft-deleting missing records
  - `COMPLETING`: Finalizing sync

- **Contextual Logging**: Every log entry includes:
  - Timestamp (ISO 8601)
  - Log level
  - Sync ID
  - Entity type
  - Current phase
  - Record counts
  - Duration metrics
  - Error category (for errors)

### 2. Updated `sync-service.ts`

**Before:**
```typescript
console.log(`Starting ${config.syncType} sync: ${syncId}`);
console.error(`✗ Failed to sync ${entityName}:`);
console.error(`  Error: ${errorMessage}`);
```

**After:**
```typescript
const syncLogger = this.logger.child({ syncId, syncType: config.syncType });
syncLogger.info(`Starting ${config.syncType} sync`, {
  entities: config.entities.map(e => getEntityDisplayName(e)).join(', '),
  triggeredBy: config.triggeredBy,
});

entityLogger.error(`Failed to sync ${entityName}`, {
  errorCategory,
  syncType: config.syncType,
}, err);
```

**Key Improvements:**
- Consistent error categorization across all error paths
- Proper error logging for sync history update failures (previously swallowed)
- Structured context in all log statements
- Child loggers for entity-specific operations

### 3. Updated `entity-sync.ts`

**Before:**
```typescript
console.log(`[${entity}] Starting sync (${isIncremental ? 'incremental' : 'full'})`);
console.error(`[${entity}] API fetch failed:`, errorMessage);
console.warn(`[${entity}] Warning: All records failed mapping validation`);
```

**After:**
```typescript
const entityLogger = this.logger.child({ syncId: trackingId, entityType: entity });
entityLogger.phase(SyncPhase.INITIALIZING);
entityLogger.phase(SyncPhase.FETCHING, 'Fetching records from Autotask API');
entityLogger.error('API fetch failed', { phase: SyncPhase.FETCHING, errorCategory }, err);
entityLogger.warn('All records failed mapping validation', { originalCount: autotaskRecords.length });
```

**Key Improvements:**
- Phase tracking throughout sync lifecycle
- Timing information for each operation
- Detailed context for warnings (record counts, sample IDs)
- Consistent error handling with categorization
- Better debugging information (sample records, field keys)

### 4. Sync Failure Analysis Script

Created `scripts/analyze-sync-failures.ts` to query and analyze sync history:

**Features:**
- Query sync history by date range, entity type, or status
- Generate failure summaries by entity
- Categorize errors automatically
- Calculate success rates and statistics
- Display detailed sync records with timing

**Usage Examples:**
```bash
# Show failures from last 7 days
npx tsx scripts/analyze-sync-failures.ts

# Show all syncs from yesterday
npx tsx scripts/analyze-sync-failures.ts --days 1 --all

# Show ticket sync failures from last 30 days
npx tsx scripts/analyze-sync-failures.ts --days 30 --entity tickets

# Show all completed syncs
npx tsx scripts/analyze-sync-failures.ts --status completed --all
```

## Benefits

### 1. **Consistent Logging Format**
All sync operations now use the same structured format, making logs easier to parse and analyze.

### 2. **Better Error Categorization**
Errors are automatically categorized, making it easier to identify patterns:
- Network issues vs. API errors
- Database constraints vs. mapping failures
- Authentication problems vs. rate limiting

### 3. **Phase Tracking**
Know exactly where a sync failed:
- Did it fail during API fetch?
- Was it a mapping error?
- Did the database upsert fail?

### 4. **Improved Debugging**
- Sample data logged for complex entities
- Timing information for performance analysis
- Record counts at each stage
- Detailed context for warnings

### 5. **No More Swallowed Errors**
Previously, sync history update failures were caught but not logged. Now they're properly tracked.

### 6. **Easy Failure Analysis**
The analysis script provides:
- Quick overview of sync health
- Failure trends by entity
- Common error patterns
- Success rate metrics

## Log Output Examples

### Successful Sync
```
[2026-01-23T12:21:00.000Z] [INFO] [syncId=sync_1234, entity=tickets, phase=initializing] Starting tickets sync (full)
[2026-01-23T12:21:00.100Z] [INFO] [syncId=sync_1234, entity=tickets, phase=fetching] Fetching records from Autotask API
[2026-01-23T12:21:05.200Z] [INFO] [syncId=sync_1234, entity=tickets, records=150] Fetched records from Autotask
[2026-01-23T12:21:05.300Z] [INFO] [syncId=sync_1234, entity=tickets, phase=mapping] Mapping 150 records to database schema
[2026-01-23T12:21:05.500Z] [INFO] [syncId=sync_1234, entity=tickets, mappedCount=148] Successfully mapped records
[2026-01-23T12:21:05.600Z] [INFO] [syncId=sync_1234, entity=tickets, phase=upserting] Upserting records to PostgreSQL
[2026-01-23T12:21:06.800Z] [INFO] [syncId=sync_1234, entity=tickets, upsertedCount=148] Upserted records to PostgreSQL
[2026-01-23T12:21:06.900Z] [INFO] [syncId=sync_1234, entity=tickets, duration=6900ms] Completed tickets sync
```

### Failed Sync with Error Details
```
[2026-01-23T12:25:00.000Z] [INFO] [syncId=sync_5678, entity=contacts, phase=initializing] Starting contacts sync (full)
[2026-01-23T12:25:00.100Z] [INFO] [syncId=sync_5678, entity=contacts, phase=fetching] Fetching records from Autotask API
[2026-01-23T12:25:10.200Z] [ERROR] [syncId=sync_5678, entity=contacts, phase=fetching, errorCategory=NETWORK_ERROR] API fetch failed
  Error: ETIMEDOUT: Connection timed out
  Category: NETWORK_ERROR
  Stack: Error: ETIMEDOUT...
[2026-01-23T12:25:10.300Z] [ERROR] [syncId=sync_5678, entity=contacts, duration=10300ms, errorCategory=NETWORK_ERROR] Failed contacts sync
```

## Migration Guide

### For Developers

If you're adding new sync operations or modifying existing ones:

1. **Import the logger:**
   ```typescript
   import { createSyncLogger, SyncPhase, categorizeError } from '../utils/sync-logger';
   ```

2. **Create a logger instance:**
   ```typescript
   const logger = createSyncLogger({ component: 'MyComponent' });
   ```

3. **Use child loggers for operations:**
   ```typescript
   const operationLogger = logger.child({ syncId, entityType });
   ```

4. **Track phases:**
   ```typescript
   operationLogger.phase(SyncPhase.FETCHING, 'Fetching data');
   ```

5. **Log with context:**
   ```typescript
   operationLogger.info('Operation completed', { recordCount: 100 });
   operationLogger.error('Operation failed', { phase: SyncPhase.MAPPING }, error);
   ```

6. **Use timing helpers:**
   ```typescript
   const startTime = logger.start('My operation');
   // ... do work ...
   logger.complete('My operation', startTime, { recordCount: 100 });
   // or on error:
   logger.fail('My operation', startTime, error);
   ```

## Analyzing Yesterday's Sync

To investigate the sync issues from yesterday:

```bash
# Show all failures from yesterday
npx tsx scripts/analyze-sync-failures.ts --days 1

# Show all syncs (including successful) from yesterday
npx tsx scripts/analyze-sync-failures.ts --days 1 --all

# Check specific entity
npx tsx scripts/analyze-sync-failures.ts --days 1 --entity tickets
```

The script will show:
- Which entities failed
- Error categories
- Timing information
- Success rates
- Common error patterns

## Future Enhancements

Potential improvements for consideration:

1. **Log Aggregation**: Send logs to a centralized logging service (e.g., Datadog, CloudWatch)
2. **Alerting**: Automatic alerts on consecutive failures
3. **Metrics Dashboard**: Real-time sync health monitoring
4. **Log Rotation**: Automatic cleanup of old logs
5. **JSON Output**: Option for machine-readable JSON logs
6. **Performance Profiling**: Detailed timing breakdown for optimization

## Related Files

- `lib/utils/sync-logger.ts` - Structured logging utility
- `lib/services/sync-service.ts` - Main sync orchestration
- `lib/services/entity-sync.ts` - Entity-specific sync logic
- `scripts/analyze-sync-failures.ts` - Failure analysis tool
- `lib/types/sync.ts` - Sync type definitions