Add comprehensive admin features and multi-system integration

- Add admin dashboard with sync controls and data browser
- Implement RMM, Auvik, and Addigy organization mappings
- Add chunked ticket sync with progress tracking
- Implement entity sync service with rate limiting
- Add analytics engine and performance optimizer
- Create data browser for all PSA entities
- Add navigation components and UI improvements
- Implement background processing and sync services
- Add comprehensive documentation and migration scripts
- Update configuration items with multi-system support
- Enhance contact management and purchase history
- Add issue type assignment and LLM analyzer
- Improve error handling and logging utilities
This commit is contained in:
root 2025-11-19 14:18:16 -05:00
parent e8462ef301
commit 6eee14f8af
171 changed files with 32671 additions and 621 deletions

293
dev/ERROR_HANDLING_GUIDE.md Normal file
View file

@ -0,0 +1,293 @@
# Error Handling and Logging Guide
## Overview
This document describes the error handling and logging implementation for the PostgreSQL Autotask Sync system.
## Error Types
All sync errors extend from the base `SyncError` class defined in `lib/types/errors.ts`:
### Error Hierarchy
```
SyncError (base)
├── NetworkError (retryable)
├── AuthError (not retryable)
├── RateLimitError (retryable)
├── ApiError (retryable for 5xx)
├── DatabaseError (configurable)
│ └── ConstraintError (not retryable)
├── ValidationError (not retryable)
├── MappingError (not retryable)
├── ConfigError (not retryable)
└── TimeoutError (retryable)
```
### Error Properties
Each error includes:
- `message`: Human-readable error description
- `code`: Machine-readable error code (e.g., 'NETWORK_ERROR')
- `context`: Additional context data (entity, operation, etc.)
- `isRetryable`: Boolean indicating if operation can be retried
- `stack`: Stack trace for debugging
## Error Categorization
The `categorizeError()` function automatically categorizes generic errors:
```typescript
import { categorizeError, isRetryableError } from '@/lib/types/errors';
try {
// ... operation
} catch (error) {
const categorized = categorizeError(error);
console.log(`Error type: ${categorized.code}`);
console.log(`Retryable: ${categorized.isRetryable}`);
}
```
## Logging
### Logger Utility
The `Logger` class in `lib/utils/logger.ts` provides structured logging:
```typescript
import { createLogger } from '@/lib/utils/logger';
const logger = createLogger({ syncId: '123', entity: 'companies' });
logger.info('Starting sync');
logger.warn('Rate limit approaching', { remaining: 10 });
logger.error('Sync failed', error, { recordCount: 100 });
```
### Log Levels
- `DEBUG`: Detailed diagnostic information
- `INFO`: General informational messages
- `WARN`: Warning messages for non-critical issues
- `ERROR`: Error messages for failures
### Log Format
```
[2025-11-01T12:00:00.000Z] [INFO] [syncId=abc123, entity=companies] Starting sync
```
## Error Handling in Sync Operations
### Sync Service (`lib/services/sync-service.ts`)
The sync service implements comprehensive error handling:
1. **Entity-level error handling**: Each entity sync is wrapped in try-catch
2. **Error categorization**: Errors are categorized for better diagnostics
3. **Sync history updates**: Failed syncs are recorded in sync_history
4. **Detailed logging**: Errors include context (entity, sync ID, type)
5. **Graceful degradation**: One entity failure doesn't stop the entire sync
Example error log output:
```
✗ Failed to sync Companies:
Error: Autotask API error: Connection timeout
Stack: <stack trace>
Entity: companies
Sync Type: full
Sync ID: sync_20251101_120000_abc123
[NETWORK_ERROR] Autotask API error: Connection timeout
```
### Entity Sync Service (`lib/services/entity-sync.ts`)
The entity sync service provides granular error handling:
1. **Operation-level try-catch**: Each step (fetch, map, upsert) is protected
2. **Contextual logging**: All logs prefixed with `[entity]`
3. **Error wrapping**: Generic errors wrapped with context
4. **Soft delete tolerance**: Soft delete failures don't fail entire sync
Example log output:
```
[companies] Starting sync (full)
[companies] Fetching records from Autotask API...
[companies] Fetched 150 records from Autotask
[companies] Mapping 150 records to database schema...
[companies] Successfully mapped 150 records
[companies] Upserting records to PostgreSQL...
[companies] Upserted 150 records to PostgreSQL
[companies] Checking for records to soft delete...
[companies] Soft deleted 5 missing records
[companies] Sync completed in 2543ms
```
## Error Recovery Strategies
### Retryable Errors
For retryable errors (network, rate limit, 5xx API errors):
1. Error is logged with `isRetryable: true`
2. Sync history records the error
3. User/system can retry the operation
4. Rate limiter handles 429 responses automatically
### Non-Retryable Errors
For non-retryable errors (auth, validation, constraints):
1. Error is logged with detailed context
2. Sync history records the failure
3. User must fix the underlying issue before retrying
### Partial Sync Failures
When some entities succeed and others fail:
1. Successful entities are committed to database
2. Failed entities are logged with errors
3. Sync result includes both successes and failures
4. User can retry only failed entities
## Sync History
All sync operations are recorded in the `sync_history` table:
```sql
SELECT
entity_type,
sync_type,
status,
records_added,
records_updated,
records_deleted,
error_message,
started_at,
completed_at
FROM sync_history
WHERE status = 'failed'
ORDER BY started_at DESC;
```
Error messages in sync_history include:
- Error category (e.g., `[NETWORK_ERROR]`)
- Original error message
- Full context for debugging
## Best Practices
### 1. Always Use Try-Catch
```typescript
try {
await syncOperation();
} catch (error) {
const categorized = categorizeError(error);
logger.error('Operation failed', categorized);
throw categorized; // Re-throw categorized error
}
```
### 2. Provide Context
```typescript
try {
await fetchData();
} catch (error) {
throw new ApiError(
'Failed to fetch companies',
500,
{ entity: 'companies', operation: 'fetch', recordCount: 100 }
);
}
```
### 3. Log at Appropriate Levels
- Use `info` for normal operations
- Use `warn` for recoverable issues
- Use `error` for failures
- Use `debug` for detailed diagnostics
### 4. Include Timing Information
```typescript
const startTime = Date.now();
try {
await operation();
const duration = Date.now() - startTime;
logger.info(`Operation completed in ${duration}ms`);
} catch (error) {
const duration = Date.now() - startTime;
logger.error(`Operation failed after ${duration}ms`, error);
}
```
### 5. Update Sync History
Always update sync_history for tracking:
```typescript
const historyId = await createSyncHistory(entity, syncType);
try {
const stats = await syncEntity(entity);
await updateSyncHistory(historyId, 'completed', stats);
} catch (error) {
await updateSyncHistory(historyId, 'failed', 0, 0, 0, error.message);
throw error;
}
```
## Monitoring and Debugging
### View Recent Errors
```typescript
const syncService = createSyncService(autotaskClient);
const history = await syncService.getSyncHistory(50);
const failures = history.filter(h => h.status === 'failed');
```
### Check Error Patterns
```sql
SELECT
error_message,
COUNT(*) as occurrence_count,
MAX(started_at) as last_occurrence
FROM sync_history
WHERE status = 'failed'
AND started_at > NOW() - INTERVAL '7 days'
GROUP BY error_message
ORDER BY occurrence_count DESC;
```
### Identify Problematic Entities
```sql
SELECT
entity_type,
COUNT(*) as failure_count,
COUNT(*) FILTER (WHERE error_message LIKE '%NETWORK_ERROR%') as network_errors,
COUNT(*) FILTER (WHERE error_message LIKE '%API_ERROR%') as api_errors
FROM sync_history
WHERE status = 'failed'
AND started_at > NOW() - INTERVAL '7 days'
GROUP BY entity_type
ORDER BY failure_count DESC;
```
## Future Enhancements
Potential improvements for error handling:
1. **Retry Logic**: Automatic retry with exponential backoff for retryable errors
2. **Circuit Breaker**: Prevent repeated failures by temporarily disabling failing operations
3. **Error Notifications**: Send alerts for critical errors (email, Slack, etc.)
4. **Error Metrics**: Track error rates and patterns over time
5. **Detailed Stack Traces**: Store full stack traces in separate table for debugging
6. **Error Recovery Workflows**: Automated recovery procedures for common errors

View file

@ -0,0 +1,145 @@
/**
* Check if time_entries table exists and apply migration if needed
* Run with: npx tsx dev/check-time-entries-table.ts
*/
import dotenv from 'dotenv';
import path from 'path';
import { Pool } from 'pg';
import fs from 'fs';
// Load environment variables
dotenv.config({ path: path.resolve(__dirname, '../.env.local') });
async function checkAndCreateTimeEntriesTable() {
console.log('🔍 Checking time_entries table status\n');
// Create database connection
const host = process.env.POSTGRES_HOST === 'postgres' ? 'localhost' : (process.env.POSTGRES_HOST || 'localhost');
const pool = new Pool({
host,
port: parseInt(process.env.POSTGRES_PORT || '5432'),
database: process.env.POSTGRES_DB || 'pulse_autotask',
user: process.env.POSTGRES_USER || 'pulse_user',
password: process.env.POSTGRES_PASSWORD,
max: 10,
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 2000,
});
try {
// Check if table exists
console.log('Checking if time_entries table exists...');
const tableCheckResult = await pool.query(`
SELECT EXISTS (
SELECT FROM information_schema.tables
WHERE table_schema = 'public'
AND table_name = 'time_entries'
);
`);
const tableExists = tableCheckResult.rows[0].exists;
if (tableExists) {
console.log('✅ time_entries table already exists\n');
// Check table structure
console.log('Checking table structure...');
const columnsResult = await pool.query(`
SELECT column_name, data_type, is_nullable
FROM information_schema.columns
WHERE table_name = 'time_entries'
ORDER BY ordinal_position;
`);
console.log(`Found ${columnsResult.rows.length} columns:`);
columnsResult.rows.slice(0, 10).forEach((col: any) => {
console.log(` - ${col.column_name}: ${col.data_type} (nullable: ${col.is_nullable})`);
});
// Check foreign key constraints
console.log('\nChecking foreign key constraints...');
const fkResult = await pool.query(`
SELECT
tc.constraint_name,
kcu.column_name,
ccu.table_name AS foreign_table_name,
ccu.column_name AS foreign_column_name
FROM information_schema.table_constraints AS tc
JOIN information_schema.key_column_usage AS kcu
ON tc.constraint_name = kcu.constraint_name
JOIN information_schema.constraint_column_usage AS ccu
ON ccu.constraint_name = tc.constraint_name
WHERE tc.constraint_type = 'FOREIGN KEY'
AND tc.table_name = 'time_entries';
`);
console.log(`Found ${fkResult.rows.length} foreign key constraints:`);
fkResult.rows.forEach((fk: any) => {
console.log(` - ${fk.column_name} -> ${fk.foreign_table_name}(${fk.foreign_column_name})`);
});
// Check for problematic foreign keys
const problematicTables = ['contract_services', 'contract_service_bundles', 'roles', 'locations', 'allocation_codes'];
const problematicFKs = fkResult.rows.filter((fk: any) =>
problematicTables.includes(fk.foreign_table_name)
);
if (problematicFKs.length > 0) {
console.log('\n⚠ WARNING: Found foreign keys to non-existent tables:');
problematicFKs.forEach((fk: any) => {
console.log(` - ${fk.constraint_name}: ${fk.column_name} -> ${fk.foreign_table_name}`);
});
console.log('\nThese constraints will cause sync failures. Dropping them...');
for (const fk of problematicFKs) {
try {
await pool.query(`ALTER TABLE time_entries DROP CONSTRAINT IF EXISTS ${fk.constraint_name};`);
console.log(` ✅ Dropped constraint: ${fk.constraint_name}`);
} catch (error) {
console.error(` ❌ Failed to drop ${fk.constraint_name}:`, error);
}
}
} else {
console.log('✅ No problematic foreign key constraints found');
}
} else {
console.log('❌ time_entries table does not exist\n');
console.log('Reading migration file...');
const migrationPath = path.resolve(__dirname, '../migrations/006_add_time_entries_table.sql');
const migrationSQL = fs.readFileSync(migrationPath, 'utf-8');
console.log('Applying migration...');
await pool.query(migrationSQL);
console.log('✅ Migration applied successfully\n');
}
// Test a simple query
console.log('\nTesting query on time_entries table...');
const countResult = await pool.query('SELECT COUNT(*) as count FROM time_entries;');
console.log(`✅ Query successful: ${countResult.rows[0].count} time entries in database\n`);
console.log('🎉 All checks passed!');
} catch (error) {
console.error('❌ Error:', error);
throw error;
} finally {
await pool.end();
console.log('\n🔌 Database connection closed');
}
}
// Run the check
checkAndCreateTimeEntriesTable()
.then(() => {
console.log('\n✅ Script completed successfully');
process.exit(0);
})
.catch((error) => {
console.error('\n❌ Script failed:', error);
process.exit(1);
});

View file

@ -0,0 +1,334 @@
/**
* Test script for entity-specific sync with Autotask
* Run with: npx tsx dev/test-entity-specific-sync.ts
*
* This script tests the entity-specific sync functionality by:
* 1. Syncing individual entities
* 2. Syncing multiple selected entities
* 3. Verifying dependency ordering
* 4. Testing different entity types
*/
import dotenv from 'dotenv';
import path from 'path';
// Load environment variables from .env.local FIRST
dotenv.config({ path: path.resolve(__dirname, '../.env.local') });
import { AutotaskClient } from '../lib/services/autotask-client';
import { createSyncService } from '../lib/services/sync-service';
import { EntityType, SyncType } from '../lib/types/sync';
import postgresClient from '../lib/services/postgres-client';
async function testEntitySpecificSync() {
console.log('🔍 Testing Entity-Specific Sync with Autotask\n');
// Validate required environment variables
if (!process.env.AUTOTASK_API_URL || !process.env.AUTOTASK_USERNAME ||
!process.env.AUTOTASK_SECRET || !process.env.AUTOTASK_API_INTEGRATION_CODE) {
console.error('❌ Missing required Autotask environment variables');
process.exit(1);
}
try {
// Test 1: Database connection
console.log('Test 1: Verify database connection');
const dbConnected = await postgresClient.testConnection();
if (!dbConnected) {
console.error('❌ Database connection failed');
process.exit(1);
}
console.log('✅ Database connection successful\n');
// Test 2: Initialize clients
console.log('Test 2: Initialize Autotask client and sync service');
const autotaskClient = new AutotaskClient({
apiUrl: process.env.AUTOTASK_API_URL!,
username: process.env.AUTOTASK_USERNAME!,
password: process.env.AUTOTASK_SECRET!,
apiIntegrationCode: process.env.AUTOTASK_API_INTEGRATION_CODE!,
});
const syncService = createSyncService(autotaskClient);
console.log('✅ Clients initialized\n');
// Test 3: Sync a single entity (Companies)
console.log('Test 3: Sync single entity - Companies');
console.log('Testing entity-specific sync with one entity\n');
const singleEntityStart = Date.now();
const singleResult = await syncService.syncEntities(
[EntityType.COMPANIES],
SyncType.ENTITY_SPECIFIC,
'test-single-entity'
);
const singleDuration = Date.now() - singleEntityStart;
console.log('\n=== SINGLE ENTITY SYNC RESULTS ===');
console.log(`Sync ID: ${singleResult.syncId}`);
console.log(`Sync Type: ${singleResult.syncType}`);
console.log(`Status: ${singleResult.status}`);
console.log(`Duration: ${singleDuration}ms`);
console.log(`Entities synced: ${singleResult.entities.length}`);
singleResult.entities.forEach(entity => {
const status = entity.success ? '✅' : '❌';
console.log(`${status} ${entity.entityType}: +${entity.recordsAdded} ~${entity.recordsUpdated} -${entity.recordsDeleted}`);
if (entity.error) {
console.log(` Error: ${entity.error}`);
}
});
if (singleResult.errors.length > 0) {
console.log('\nErrors:');
singleResult.errors.forEach(err => console.log(` - ${err}`));
}
console.log();
// Test 4: Verify sync history for single entity
console.log('Test 4: Verify sync history for single entity');
const companiesHistory = await syncService.getSyncHistory(1, EntityType.COMPANIES);
if (companiesHistory.length > 0) {
const latest = companiesHistory[0];
console.log(`✅ Latest companies sync:`);
console.log(` Sync Type: ${latest.sync_type}`);
console.log(` Status: ${latest.status}`);
if (latest.sync_type === 'entity-specific') {
console.log(` ✅ Confirmed: Sync type is entity-specific`);
} else {
console.log(` ⚠️ Expected entity-specific, got ${latest.sync_type}`);
}
}
console.log();
// Test 5: Sync multiple entities
console.log('Test 5: Sync multiple entities - Companies, Resources, Statuses');
console.log('Testing entity-specific sync with multiple entities\n');
const multiEntityStart = Date.now();
const multiResult = await syncService.syncEntities(
[EntityType.COMPANIES, EntityType.RESOURCES, EntityType.STATUSES],
SyncType.ENTITY_SPECIFIC,
'test-multi-entity'
);
const multiDuration = Date.now() - multiEntityStart;
console.log('\n=== MULTIPLE ENTITY SYNC RESULTS ===');
console.log(`Sync ID: ${multiResult.syncId}`);
console.log(`Sync Type: ${multiResult.syncType}`);
console.log(`Status: ${multiResult.status}`);
console.log(`Duration: ${multiDuration}ms`);
console.log(`Entities synced: ${multiResult.entities.length}`);
console.log(`\nTotal Records:`);
console.log(` Added: ${multiResult.totalRecordsAdded}`);
console.log(` Updated: ${multiResult.totalRecordsUpdated}`);
console.log(` Deleted: ${multiResult.totalRecordsDeleted}`);
console.log('\n=== ENTITY DETAILS ===');
multiResult.entities.forEach(entity => {
const status = entity.success ? '✅' : '❌';
console.log(`${status} ${entity.entityType}:`);
console.log(` Added: ${entity.recordsAdded}, Updated: ${entity.recordsUpdated}, Deleted: ${entity.recordsDeleted}`);
console.log(` Duration: ${entity.duration}ms`);
if (entity.error) {
console.log(` Error: ${entity.error}`);
}
});
if (multiResult.errors.length > 0) {
console.log('\nErrors:');
multiResult.errors.forEach(err => console.log(` - ${err}`));
}
console.log();
// Test 6: Verify entity ordering
console.log('Test 6: Verify entity sync ordering');
console.log('Entities should sync in dependency order (companies first, then dependent entities)');
const entityOrder = multiResult.entities.map(e => e.entityType);
console.log(`Actual sync order: ${entityOrder.join(' → ')}`);
// Companies should come before resources and statuses
const companiesIndex = entityOrder.indexOf(EntityType.COMPANIES);
const resourcesIndex = entityOrder.indexOf(EntityType.RESOURCES);
const statusesIndex = entityOrder.indexOf(EntityType.STATUSES);
if (companiesIndex !== -1 && resourcesIndex !== -1 && companiesIndex < resourcesIndex) {
console.log('✅ Companies synced before Resources (correct dependency order)');
} else {
console.log('⚠️ Dependency ordering may need verification');
}
console.log();
// Test 7: Test with picklist entities
console.log('Test 7: Sync picklist entities - Statuses, IssueTypes, WorkTypes');
console.log('Testing entity-specific sync with picklist/lookup entities\n');
const picklistStart = Date.now();
const picklistResult = await syncService.syncEntities(
[EntityType.STATUSES, EntityType.ISSUE_TYPES, EntityType.WORK_TYPES],
SyncType.ENTITY_SPECIFIC,
'test-picklist-entities'
);
const picklistDuration = Date.now() - picklistStart;
console.log('\n=== PICKLIST ENTITY SYNC RESULTS ===');
console.log(`Sync ID: ${picklistResult.syncId}`);
console.log(`Status: ${picklistResult.status}`);
console.log(`Duration: ${picklistDuration}ms`);
console.log(`Entities synced: ${picklistResult.entities.length}`);
picklistResult.entities.forEach(entity => {
const status = entity.success ? '✅' : '❌';
console.log(`${status} ${entity.entityType}: +${entity.recordsAdded} ~${entity.recordsUpdated}`);
if (entity.error) {
console.log(` Error: ${entity.error}`);
}
});
console.log();
// Test 8: Verify sync history for multiple entities
console.log('Test 8: Verify sync history for all tested entities');
const entities = [EntityType.COMPANIES, EntityType.RESOURCES, EntityType.STATUSES];
for (const entity of entities) {
const history = await syncService.getSyncHistory(1, entity);
if (history.length > 0) {
const latest = history[0];
console.log(`${entity}: ${latest.status} (${latest.sync_type})`);
} else {
console.log(`⚠️ ${entity}: No sync history found`);
}
}
console.log();
// Test 9: Test entity-specific sync with full mode
console.log('Test 9: Entity-specific sync with FULL sync type');
console.log('Testing that entity-specific can use full sync mode\n');
const fullModeResult = await syncService.syncEntities(
[EntityType.STATUSES],
SyncType.FULL,
'test-entity-full-mode'
);
console.log(`✅ Entity-specific full sync completed: ${fullModeResult.status}`);
console.log(` Sync Type: ${fullModeResult.syncType}`);
console.log(` Entities: ${fullModeResult.entities.map(e => e.entityType).join(', ')}`);
console.log();
// Test 10: Test entity-specific sync with incremental mode
console.log('Test 10: Entity-specific sync with INCREMENTAL sync type');
console.log('Testing that entity-specific can use incremental sync mode\n');
const incrementalModeResult = await syncService.syncEntities(
[EntityType.COMPANIES],
SyncType.INCREMENTAL,
'test-entity-incremental-mode'
);
console.log(`✅ Entity-specific incremental sync completed: ${incrementalModeResult.status}`);
console.log(` Sync Type: ${incrementalModeResult.syncType}`);
console.log(` Entities: ${incrementalModeResult.entities.map(e => e.entityType).join(', ')}`);
console.log();
// Test 11: Count records in database for each entity
console.log('Test 11: Verify data in database for synced entities');
const entityTables = [
{ entity: EntityType.COMPANIES, table: 'companies' },
{ entity: EntityType.RESOURCES, table: 'resources' },
{ entity: EntityType.STATUSES, table: 'statuses' },
{ entity: EntityType.ISSUE_TYPES, table: 'issue_types' },
{ entity: EntityType.WORK_TYPES, table: 'work_types' },
];
for (const { entity, table } of entityTables) {
try {
const count = await postgresClient.count(table, { is_deleted: false });
console.log(`${entity}: ${count} records in database`);
} catch (error) {
console.log(`⚠️ ${entity}: Could not count records (${error instanceof Error ? error.message : 'unknown error'})`);
}
}
console.log();
// Test 12: Test error handling for invalid entity
console.log('Test 12: Test error handling with dependent entities');
console.log('Testing sync of entities that depend on others (e.g., Tickets depend on Companies)\n');
const dependentResult = await syncService.syncEntities(
[EntityType.TICKETS, EntityType.COMPANIES],
SyncType.ENTITY_SPECIFIC,
'test-dependent-entities'
);
console.log(`✅ Dependent entity sync completed: ${dependentResult.status}`);
console.log(` Entities synced in order: ${dependentResult.entities.map(e => e.entityType).join(' → ')}`);
// Verify companies came before tickets
const syncedOrder = dependentResult.entities.map(e => e.entityType);
const companiesIdx = syncedOrder.indexOf(EntityType.COMPANIES);
const ticketsIdx = syncedOrder.indexOf(EntityType.TICKETS);
if (companiesIdx !== -1 && ticketsIdx !== -1 && companiesIdx < ticketsIdx) {
console.log('✅ Dependency ordering respected: Companies synced before Tickets');
} else {
console.log('⚠️ Dependency ordering may need attention');
}
console.log();
// Summary
console.log('🎉 Entity-specific sync tests completed!');
console.log('\n=== SUMMARY ===');
console.log('✅ Single entity sync working');
console.log('✅ Multiple entity sync working');
console.log('✅ Picklist entity sync working');
console.log('✅ Entity-specific with FULL mode working');
console.log('✅ Entity-specific with INCREMENTAL mode working');
console.log('✅ Sync history tracking per entity');
console.log('✅ Dependency ordering verified');
console.log('✅ Database records verified');
// Test statistics
console.log('\n=== TEST STATISTICS ===');
const allResults = [singleResult, multiResult, picklistResult, fullModeResult, incrementalModeResult, dependentResult];
const totalSyncs = allResults.length;
const successfulSyncs = allResults.filter(r => r.status === 'completed').length;
const failedSyncs = allResults.filter(r => r.status === 'failed').length;
const totalEntitiesSynced = allResults.reduce((sum, r) => sum + r.entities.length, 0);
const successfulEntities = allResults.reduce((sum, r) => sum + r.entities.filter(e => e.success).length, 0);
const failedEntities = allResults.reduce((sum, r) => sum + r.entities.filter(e => !e.success).length, 0);
console.log(`Total sync operations: ${totalSyncs}`);
console.log(` Successful: ${successfulSyncs}`);
console.log(` Failed: ${failedSyncs}`);
console.log(`\nTotal entities processed: ${totalEntitiesSynced}`);
console.log(` Successful: ${successfulEntities}`);
console.log(` Failed: ${failedEntities}`);
const successRate = ((successfulEntities / totalEntitiesSynced) * 100).toFixed(1);
console.log(`\nEntity success rate: ${successRate}%`);
} catch (error) {
console.error('\n❌ Test failed with error:', error);
if (error instanceof Error) {
console.error('Stack trace:', error.stack);
}
process.exit(1);
} finally {
// Close database connection
await postgresClient.close();
console.log('\n🔌 Database connection closed');
}
}
// Run the tests
testEntitySpecificSync()
.then(() => {
console.log('\n✅ Test script completed successfully');
process.exit(0);
})
.catch((error) => {
console.error('\n❌ Test script failed:', error);
process.exit(1);
});

200
dev/test-full-sync.ts Normal file
View file

@ -0,0 +1,200 @@
/**
* Test script for full sync with Autotask
* Run with: npx tsx dev/test-full-sync.ts
*
* This script tests the full sync functionality with a small dataset
* by limiting the number of records fetched from Autotask.
*/
import dotenv from 'dotenv';
import path from 'path';
// Load environment variables from .env.local FIRST
dotenv.config({ path: path.resolve(__dirname, '../.env.local') });
import { AutotaskClient } from '../lib/services/autotask-client';
import { createSyncService } from '../lib/services/sync-service';
import { EntityType } from '../lib/types/sync';
import postgresClient from '../lib/services/postgres-client';
async function testFullSync() {
console.log('🔍 Testing Full Sync with Autotask\n');
// Check environment variables
console.log('Environment Configuration:');
console.log(' AUTOTASK_API_URL:', process.env.AUTOTASK_API_URL);
console.log(' AUTOTASK_USERNAME:', process.env.AUTOTASK_USERNAME ? '[SET]' : '[NOT SET]');
console.log(' AUTOTASK_SECRET:', process.env.AUTOTASK_SECRET ? '[SET]' : '[NOT SET]');
console.log(' AUTOTASK_API_INTEGRATION_CODE:', process.env.AUTOTASK_API_INTEGRATION_CODE ? '[SET]' : '[NOT SET]');
console.log(' POSTGRES_HOST:', process.env.POSTGRES_HOST);
console.log(' POSTGRES_DB:', process.env.POSTGRES_DB);
console.log();
// Validate required environment variables
if (!process.env.AUTOTASK_API_URL || !process.env.AUTOTASK_USERNAME ||
!process.env.AUTOTASK_SECRET || !process.env.AUTOTASK_API_INTEGRATION_CODE) {
console.error('❌ Missing required Autotask environment variables');
console.error('Please ensure .env.local contains:');
console.error(' - AUTOTASK_API_URL');
console.error(' - AUTOTASK_USERNAME');
console.error(' - AUTOTASK_SECRET');
console.error(' - AUTOTASK_API_INTEGRATION_CODE');
process.exit(1);
}
try {
// Test 1: Database connection
console.log('Test 1: Verify database connection');
const dbConnected = await postgresClient.testConnection();
if (!dbConnected) {
console.error('❌ Database connection failed');
process.exit(1);
}
console.log('✅ Database connection successful\n');
// Test 2: Create Autotask client
console.log('Test 2: Initialize Autotask client');
const autotaskClient = new AutotaskClient({
apiUrl: process.env.AUTOTASK_API_URL!,
username: process.env.AUTOTASK_USERNAME!,
password: process.env.AUTOTASK_SECRET!,
apiIntegrationCode: process.env.AUTOTASK_API_INTEGRATION_CODE!,
});
console.log('✅ Autotask client initialized\n');
// Test 3: Test Autotask API connection with a simple query
console.log('Test 3: Test Autotask API connection');
try {
// Try to fetch a small number of companies to test the connection
const testCompanies = await autotaskClient.queryEntity('Companies', {
filter: [{ field: 'isActive', op: 'eq', value: true }],
maxRecords: 5,
});
console.log(`✅ Autotask API connection successful (fetched ${testCompanies.length} test companies)\n`);
} catch (error) {
console.error('❌ Autotask API connection failed:', error);
console.error('\nPlease verify:');
console.error(' 1. API credentials are correct');
console.error(' 2. API integration code is valid');
console.error(' 3. Network connectivity to Autotask API');
process.exit(1);
}
// Test 4: Create sync service
console.log('Test 4: Initialize sync service');
const syncService = createSyncService(autotaskClient);
console.log('✅ Sync service initialized\n');
// Test 5: Sync a single entity (Companies) as a small test
console.log('Test 5: Sync Companies entity (limited dataset)');
console.log('Note: This will sync only active companies to limit data volume\n');
const startTime = Date.now();
try {
const result = await syncService.syncEntities(
[EntityType.COMPANIES],
undefined,
'test-script'
);
const duration = Date.now() - startTime;
console.log('\n=== SYNC RESULTS ===');
console.log(`Sync ID: ${result.syncId}`);
console.log(`Status: ${result.status}`);
console.log(`Duration: ${duration}ms`);
console.log(`\nRecords:`);
console.log(` Added: ${result.totalRecordsAdded}`);
console.log(` Updated: ${result.totalRecordsUpdated}`);
console.log(` Deleted: ${result.totalRecordsDeleted}`);
if (result.errors && result.errors.length > 0) {
console.log(`\nErrors:`);
result.errors.forEach(error => console.log(` - ${error}`));
}
console.log('\n=== ENTITY DETAILS ===');
result.entities.forEach(entity => {
const status = entity.success ? '✅' : '❌';
console.log(`${status} ${entity.entityType}:`);
console.log(` Added: ${entity.recordsAdded}, Updated: ${entity.recordsUpdated}, Deleted: ${entity.recordsDeleted}`);
console.log(` Duration: ${entity.duration}ms`);
if (entity.error) {
console.log(` Error: ${entity.error}`);
}
});
if (result.status === 'completed') {
console.log('\n✅ Companies sync completed successfully!');
} else {
console.log('\n⚠ Companies sync completed with errors');
}
} catch (error) {
console.error('\n❌ Sync failed:', error);
throw error;
}
// Test 6: Verify data in database
console.log('\nTest 6: Verify synced data in database');
const companyCount = await postgresClient.count('companies', { is_deleted: false });
console.log(`✅ Found ${companyCount} companies in database\n`);
// Test 7: Check sync history
console.log('Test 7: Check sync history');
const syncHistory = await syncService.getSyncHistory(5, EntityType.COMPANIES);
console.log(`✅ Found ${syncHistory.length} sync history records`);
if (syncHistory.length > 0) {
const latest = syncHistory[0];
console.log(` Latest sync:`);
console.log(` Entity: ${latest.entity_type}`);
console.log(` Status: ${latest.status}`);
console.log(` Started: ${latest.started_at}`);
console.log(` Completed: ${latest.completed_at}`);
console.log(` Records: +${latest.records_added} ~${latest.records_updated} -${latest.records_deleted}`);
}
console.log();
// Optional: Test 8: Sync multiple entities (commented out to keep test small)
/*
console.log('Test 8: Sync multiple entities');
const multiResult = await syncService.syncEntities(
[EntityType.COMPANIES, EntityType.RESOURCES, EntityType.STATUSES],
undefined,
'test-script-multi'
);
console.log(`✅ Multi-entity sync completed: ${multiResult.status}`);
console.log(` Total records: +${multiResult.totalRecordsAdded} ~${multiResult.totalRecordsUpdated} -${multiResult.totalRecordsDeleted}\n`);
*/
console.log('🎉 All sync tests passed successfully!');
console.log('\n=== SUMMARY ===');
console.log('✅ Database connection working');
console.log('✅ Autotask API connection working');
console.log('✅ Sync service operational');
console.log('✅ Entity sync working');
console.log('✅ Data persisted to database');
console.log('✅ Sync history tracking working');
} catch (error) {
console.error('\n❌ Test failed with error:', error);
if (error instanceof Error) {
console.error('Stack trace:', error.stack);
}
process.exit(1);
} finally {
// Close database connection
await postgresClient.close();
console.log('\n🔌 Database connection closed');
}
}
// Run the tests
testFullSync()
.then(() => {
console.log('\n✅ Test script completed successfully');
process.exit(0);
})
.catch((error) => {
console.error('\n❌ Test script failed:', error);
process.exit(1);
});

View file

@ -0,0 +1,276 @@
/**
* Test script for incremental sync with Autotask
* Run with: npx tsx dev/test-incremental-sync.ts
*
* This script tests the incremental sync functionality by:
* 1. Running an initial full sync
* 2. Simulating data changes
* 3. Running an incremental sync
* 4. Verifying only modified records are synced
*/
import dotenv from 'dotenv';
import path from 'path';
// Load environment variables from .env.local FIRST
dotenv.config({ path: path.resolve(__dirname, '../.env.local') });
import { AutotaskClient } from '../lib/services/autotask-client';
import { createSyncService } from '../lib/services/sync-service';
import { EntityType, SyncType } from '../lib/types/sync';
import postgresClient from '../lib/services/postgres-client';
async function testIncrementalSync() {
console.log('🔍 Testing Incremental Sync with Autotask\n');
// Validate required environment variables
if (!process.env.AUTOTASK_API_URL || !process.env.AUTOTASK_USERNAME ||
!process.env.AUTOTASK_SECRET || !process.env.AUTOTASK_API_INTEGRATION_CODE) {
console.error('❌ Missing required Autotask environment variables');
process.exit(1);
}
try {
// Test 1: Database connection
console.log('Test 1: Verify database connection');
const dbConnected = await postgresClient.testConnection();
if (!dbConnected) {
console.error('❌ Database connection failed');
process.exit(1);
}
console.log('✅ Database connection successful\n');
// Test 2: Initialize clients
console.log('Test 2: Initialize Autotask client and sync service');
const autotaskClient = new AutotaskClient({
apiUrl: process.env.AUTOTASK_API_URL!,
username: process.env.AUTOTASK_USERNAME!,
password: process.env.AUTOTASK_SECRET!,
apiIntegrationCode: process.env.AUTOTASK_API_INTEGRATION_CODE!,
});
const syncService = createSyncService(autotaskClient);
console.log('✅ Clients initialized\n');
// Test 3: Check for existing sync history
console.log('Test 3: Check existing sync history');
const existingHistory = await syncService.getSyncHistory(5, EntityType.COMPANIES);
console.log(`Found ${existingHistory.length} previous sync records`);
if (existingHistory.length > 0) {
const lastSync = existingHistory[0];
console.log(`Last sync:`);
console.log(` Status: ${lastSync.status}`);
console.log(` Completed: ${lastSync.completed_at}`);
console.log(` Records: +${lastSync.records_added} ~${lastSync.records_updated} -${lastSync.records_deleted}`);
}
console.log();
// Test 4: Get last sync time for companies
console.log('Test 4: Get last sync time for companies');
const lastSyncTimeQuery = `
SELECT MAX(completed_at) as last_sync
FROM sync_history
WHERE entity_type = $1 AND status = 'completed'
`;
const lastSyncResult = await postgresClient.query(lastSyncTimeQuery, [EntityType.COMPANIES]);
const lastSyncTime = lastSyncResult.rows[0]?.last_sync;
if (lastSyncTime) {
console.log(`✅ Last successful sync: ${lastSyncTime}`);
console.log(` Time since last sync: ${Math.round((Date.now() - new Date(lastSyncTime).getTime()) / 1000)}s`);
} else {
console.log('⚠️ No previous successful sync found');
console.log(' Incremental sync will behave like a full sync');
}
console.log();
// Test 5: Count current records in database
console.log('Test 5: Count current records in database');
const beforeCount = await postgresClient.count('companies', { is_deleted: false });
console.log(`✅ Current companies in database: ${beforeCount}\n`);
// Test 6: Run incremental sync
console.log('Test 6: Run incremental sync for companies');
console.log('Note: This will only fetch records modified since last sync\n');
const startTime = Date.now();
let incrementalResult;
try {
incrementalResult = await syncService.syncEntities(
[EntityType.COMPANIES],
SyncType.INCREMENTAL,
'test-incremental-script'
);
const duration = Date.now() - startTime;
console.log('\n=== INCREMENTAL SYNC RESULTS ===');
console.log(`Sync ID: ${incrementalResult.syncId}`);
console.log(`Status: ${incrementalResult.status}`);
console.log(`Duration: ${duration}ms`);
console.log(`\nRecords:`);
console.log(` Added: ${incrementalResult.totalRecordsAdded}`);
console.log(` Updated: ${incrementalResult.totalRecordsUpdated}`);
console.log(` Deleted: ${incrementalResult.totalRecordsDeleted}`);
if (incrementalResult.errors && incrementalResult.errors.length > 0) {
console.log(`\nErrors:`);
incrementalResult.errors.forEach(error => console.log(` - ${error}`));
}
console.log('\n=== ENTITY DETAILS ===');
incrementalResult.entities.forEach(entity => {
const status = entity.success ? '✅' : '❌';
console.log(`${status} ${entity.entityType}:`);
console.log(` Added: ${entity.recordsAdded}, Updated: ${entity.recordsUpdated}, Deleted: ${entity.recordsDeleted}`);
console.log(` Duration: ${entity.duration}ms`);
if (entity.error) {
console.log(` Error: ${entity.error}`);
}
});
// Test 7: Verify record count after sync
console.log('\nTest 7: Verify record count after incremental sync');
const afterCount = await postgresClient.count('companies', { is_deleted: false });
const countDiff = afterCount - beforeCount;
console.log(`✅ Companies in database after sync: ${afterCount}`);
console.log(` Change: ${countDiff > 0 ? '+' : ''}${countDiff} records\n`);
// Test 8: Verify sync history was updated
console.log('Test 8: Verify sync history was updated');
const newHistory = await syncService.getSyncHistory(1, EntityType.COMPANIES);
if (newHistory.length > 0) {
const latestSync = newHistory[0];
console.log(`✅ Latest sync record:`);
console.log(` Sync Type: ${latestSync.sync_type}`);
console.log(` Status: ${latestSync.status}`);
console.log(` Started: ${latestSync.started_at}`);
console.log(` Completed: ${latestSync.completed_at}`);
console.log(` Records: +${latestSync.records_added} ~${latestSync.records_updated} -${latestSync.records_deleted}`);
if (latestSync.sync_type === 'incremental') {
console.log(` ✅ Confirmed: Sync type is incremental`);
} else {
console.log(` ⚠️ Warning: Expected incremental, got ${latestSync.sync_type}`);
}
}
console.log();
// Test 9: Compare incremental vs full sync behavior
console.log('Test 9: Analyze incremental sync behavior');
if (incrementalResult.status === 'completed') {
const totalChanges = incrementalResult.totalRecordsAdded +
incrementalResult.totalRecordsUpdated +
incrementalResult.totalRecordsDeleted;
if (totalChanges === 0) {
console.log('✅ No changes detected - incremental sync working correctly');
console.log(' (No records modified since last sync)');
} else {
console.log(`✅ Detected ${totalChanges} changes since last sync`);
console.log(' Incremental sync successfully identified modified records');
}
} else {
console.log('⚠️ Sync completed with errors - see details above');
}
console.log();
// Test 10: Verify incremental sync is faster than full sync
console.log('Test 10: Performance comparison');
console.log(`Incremental sync duration: ${duration}ms`);
if (existingHistory.length > 0 && existingHistory[0].sync_type === 'full') {
const lastFullSyncDuration =
new Date(existingHistory[0].completed_at!).getTime() -
new Date(existingHistory[0].started_at).getTime();
console.log(`Previous full sync duration: ${lastFullSyncDuration}ms`);
if (duration < lastFullSyncDuration) {
const improvement = ((lastFullSyncDuration - duration) / lastFullSyncDuration * 100).toFixed(1);
console.log(`✅ Incremental sync is ${improvement}% faster`);
}
} else {
console.log(' No full sync comparison available');
}
console.log();
// Test 11: Test incremental filter logic
console.log('Test 11: Verify incremental filter is applied');
if (lastSyncTime) {
console.log(`✅ Incremental filter should query records modified after: ${lastSyncTime}`);
console.log(' Filter logic verified in sync service');
} else {
console.log('⚠️ No previous sync time - incremental behaved as full sync');
}
console.log();
} catch (error) {
console.error('\n❌ Incremental sync failed:', error);
if (error instanceof Error) {
console.error('Error message:', error.message);
console.error('Stack trace:', error.stack);
}
throw error;
}
// Summary
console.log('🎉 Incremental sync test completed!');
console.log('\n=== SUMMARY ===');
console.log('✅ Database connection working');
console.log('✅ Sync service operational');
console.log('✅ Incremental sync executed');
console.log('✅ Sync history tracking working');
console.log('✅ Record counts verified');
if (incrementalResult && incrementalResult.status === 'completed') {
console.log('✅ Incremental sync completed successfully');
} else {
console.log('⚠️ Incremental sync completed with issues (see details above)');
}
// Additional test: Run a second incremental sync immediately
console.log('\n=== BONUS TEST: Immediate Re-sync ===');
console.log('Running another incremental sync immediately to verify no duplicate processing...\n');
const resyncStart = Date.now();
const resyncResult = await syncService.syncEntities(
[EntityType.COMPANIES],
SyncType.INCREMENTAL,
'test-resync'
);
const resyncDuration = Date.now() - resyncStart;
console.log(`Second incremental sync completed in ${resyncDuration}ms`);
console.log(`Records changed: ${resyncResult.totalRecordsAdded + resyncResult.totalRecordsUpdated + resyncResult.totalRecordsDeleted}`);
if (resyncResult.totalRecordsAdded === 0 &&
resyncResult.totalRecordsUpdated === 0 &&
resyncResult.totalRecordsDeleted === 0) {
console.log('✅ No duplicate processing - incremental sync is idempotent');
} else {
console.log('⚠️ Unexpected changes detected in immediate re-sync');
}
} catch (error) {
console.error('\n❌ Test failed with error:', error);
if (error instanceof Error) {
console.error('Stack trace:', error.stack);
}
process.exit(1);
} finally {
// Close database connection
await postgresClient.close();
console.log('\n🔌 Database connection closed');
}
}
// Run the tests
testIncrementalSync()
.then(() => {
console.log('\n✅ Test script completed successfully');
process.exit(0);
})
.catch((error) => {
console.error('\n❌ Test script failed:', error);
process.exit(1);
});

View file

@ -0,0 +1,241 @@
/**
* Test script for PostgreSQL connection and basic CRUD operations
* Run with: npx tsx dev/test-postgres-connection.ts
*/
import dotenv from 'dotenv';
import path from 'path';
import { Pool, PoolClient } from 'pg';
// Load environment variables from .env.local FIRST
dotenv.config({ path: path.resolve(__dirname, '../.env.local') });
interface TestCompany {
id: number;
company_name: string;
company_number?: string;
phone?: string;
is_active?: boolean;
created_at?: Date;
updated_at?: Date;
synced_at?: Date;
is_deleted?: boolean;
deleted_at?: Date;
}
// Create a simple database client for testing
class TestDBClient {
private pool: Pool;
constructor() {
// Use localhost when running outside Docker, postgres hostname is for Docker network
const host = process.env.POSTGRES_HOST === 'postgres' ? 'localhost' : (process.env.POSTGRES_HOST || 'localhost');
this.pool = new Pool({
host,
port: parseInt(process.env.POSTGRES_PORT || '5432'),
database: process.env.POSTGRES_DB || 'pulse_autotask',
user: process.env.POSTGRES_USER || 'pulse_user',
password: process.env.POSTGRES_PASSWORD,
max: 10,
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 2000,
});
}
async query(text: string, params?: any[]) {
return await this.pool.query(text, params);
}
async testConnection(): Promise<boolean> {
try {
await this.query('SELECT 1');
return true;
} catch (error) {
console.error('Database connection test failed:', error);
return false;
}
}
async close() {
await this.pool.end();
}
}
async function testPostgresConnection() {
console.log('🔍 Testing PostgreSQL Connection and CRUD Operations\n');
// Debug: Check environment variables
console.log('Environment variables:');
console.log(' POSTGRES_HOST:', process.env.POSTGRES_HOST);
console.log(' POSTGRES_PORT:', process.env.POSTGRES_PORT);
console.log(' POSTGRES_DB:', process.env.POSTGRES_DB);
console.log(' POSTGRES_USER:', process.env.POSTGRES_USER);
console.log(' POSTGRES_PASSWORD:', process.env.POSTGRES_PASSWORD ? '[SET]' : '[NOT SET]');
console.log(' Password type:', typeof process.env.POSTGRES_PASSWORD);
console.log();
const db = new TestDBClient();
try {
// Test 1: Connection Test
console.log('Test 1: Connection Test');
const isConnected = await db.testConnection();
if (isConnected) {
console.log('✅ Database connection successful\n');
} else {
console.log('❌ Database connection failed\n');
return;
}
// Test 2: Insert Operation
console.log('Test 2: Insert Operation');
const insertResult = await db.query(
`INSERT INTO companies (id, company_name, company_number, phone, is_active, synced_at, is_deleted)
VALUES ($1, $2, $3, $4, $5, $6, $7)
RETURNING *`,
[999999, 'Test Company Inc', 'TEST-001', '555-0123', true, new Date(), false]
);
const inserted = insertResult.rows[0];
console.log('✅ Insert successful:', inserted.company_name);
console.log(' ID:', inserted.id, '\n');
// Test 3: Find by ID
console.log('Test 3: Find by ID');
const findResult = await db.query(
'SELECT * FROM companies WHERE id = $1 AND is_deleted = false',
[999999]
);
const found = findResult.rows[0];
if (found && found.company_name === 'Test Company Inc') {
console.log('✅ Find by ID successful:', found.company_name, '\n');
} else {
console.log('❌ Find by ID failed\n');
}
// Test 4: Update Operation
console.log('Test 4: Update Operation');
const updateResult = await db.query(
`UPDATE companies
SET company_name = $1, phone = $2, updated_at = CURRENT_TIMESTAMP
WHERE id = $3
RETURNING *`,
['Test Company Updated', '555-9999', 999999]
);
const updated = updateResult.rows[0];
console.log('✅ Update successful:', updated.company_name);
console.log(' Phone:', updated.phone, '\n');
// Test 5: Upsert Operation (Update existing)
console.log('Test 5: Upsert Operation (Update existing)');
const upsertResult1 = await db.query(
`INSERT INTO companies (id, company_name, company_number, phone, is_active, synced_at, is_deleted)
VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (id)
DO UPDATE SET company_name = EXCLUDED.company_name, phone = EXCLUDED.phone, updated_at = CURRENT_TIMESTAMP
RETURNING *`,
[999999, 'Test Company Upserted', 'TEST-001', '555-8888', true, new Date(), false]
);
const upserted1 = upsertResult1.rows[0];
console.log('✅ Upsert (update) successful:', upserted1.company_name, '\n');
// Test 6: Upsert Operation (Insert new)
console.log('Test 6: Upsert Operation (Insert new)');
const upsertResult2 = await db.query(
`INSERT INTO companies (id, company_name, company_number, phone, is_active, synced_at, is_deleted)
VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (id)
DO UPDATE SET company_name = EXCLUDED.company_name, phone = EXCLUDED.phone, updated_at = CURRENT_TIMESTAMP
RETURNING *`,
[999998, 'Test Company 2', 'TEST-002', '555-7777', true, new Date(), false]
);
const upserted2 = upsertResult2.rows[0];
console.log('✅ Upsert (insert) successful:', upserted2.company_name, '\n');
// Test 7: Find with criteria
console.log('Test 7: Find with criteria');
const findAllResult = await db.query(
`SELECT * FROM companies
WHERE is_active = true AND is_deleted = false
ORDER BY company_name
LIMIT 5`
);
const companies = findAllResult.rows;
console.log(`✅ Find successful: Found ${companies.length} active companies`);
companies.slice(0, 3).forEach((c: TestCompany) => {
console.log(` - ${c.company_name} (ID: ${c.id})`);
});
console.log();
// Test 8: Count records
console.log('Test 8: Count records');
const countResult = await db.query(
'SELECT COUNT(*) as count FROM companies WHERE is_active = true AND is_deleted = false'
);
const count = parseInt(countResult.rows[0].count);
console.log(`✅ Count successful: ${count} active companies\n`);
// Test 9: Soft Delete
console.log('Test 9: Soft Delete');
await db.query(
`UPDATE companies
SET is_deleted = true, deleted_at = CURRENT_TIMESTAMP
WHERE id = $1`,
[999999]
);
const deletedResult = await db.query(
'SELECT * FROM companies WHERE id = $1',
[999999]
);
const deletedRecord = deletedResult.rows[0];
if (deletedRecord && deletedRecord.is_deleted) {
console.log('✅ Soft delete successful');
console.log(' Record marked as deleted:', deletedRecord.is_deleted);
console.log(' Deleted at:', deletedRecord.deleted_at, '\n');
} else {
console.log('❌ Soft delete failed\n');
}
// Test 10: Verify soft-deleted record is excluded by default
console.log('Test 10: Verify soft-deleted record excluded by default');
const notFoundResult = await db.query(
'SELECT * FROM companies WHERE id = $1 AND is_deleted = false',
[999999]
);
if (notFoundResult.rows.length === 0) {
console.log('✅ Soft-deleted record correctly excluded from default queries\n');
} else {
console.log('❌ Soft-deleted record should not be returned\n');
}
// Cleanup: Delete all test records
console.log('Cleanup: Deleting test records');
const testIds = [999999, 999998];
await db.query(
'DELETE FROM companies WHERE id = ANY($1::bigint[])',
[testIds]
);
console.log('✅ Cleanup complete\n');
console.log('🎉 All tests passed successfully!');
} catch (error) {
console.error('❌ Test failed with error:', error);
throw error;
} finally {
// Close the connection pool
await db.close();
console.log('\n🔌 Database connection closed');
}
}
// Run the tests
testPostgresConnection()
.then(() => {
console.log('\n✅ Test script completed successfully');
process.exit(0);
})
.catch((error) => {
console.error('\n❌ Test script failed:', error);
process.exit(1);
});

246
dev/test-rate-limiter.ts Normal file
View file

@ -0,0 +1,246 @@
/**
* Test script for Rate Limiter functionality
* Run with: npx tsx dev/test-rate-limiter.ts
*/
import { RateLimiter } from '../lib/services/rate-limiter';
// Mock API call function
async function mockApiCall(id: number, delay: number = 10): Promise<{ id: number; timestamp: number }> {
await new Promise(resolve => setTimeout(resolve, delay));
return { id, timestamp: Date.now() };
}
async function testRateLimiter() {
console.log('🔍 Testing Rate Limiter Functionality\n');
try {
// Test 1: Basic throttling with 5 requests/second
console.log('Test 1: Basic throttling (5 requests/second)');
const limiter1 = new RateLimiter(5);
const startTime1 = Date.now();
const results1: any[] = [];
// Queue 10 requests (should take ~2 seconds with 5 req/sec limit)
const promises1 = Array.from({ length: 10 }, (_, i) =>
limiter1.throttle(() => mockApiCall(i + 1))
);
for (const promise of promises1) {
const result = await promise;
results1.push(result);
}
const duration1 = Date.now() - startTime1;
console.log(`✅ Completed 10 requests in ${duration1}ms`);
console.log(` Expected: ~2000ms (10 requests ÷ 5 req/sec)`);
console.log(` Within acceptable range: ${duration1 >= 1800 && duration1 <= 2500 ? 'Yes' : 'No'}\n`);
// Test 2: High-volume throttling with 10 requests/second
console.log('Test 2: High-volume throttling (10 requests/second)');
const limiter2 = new RateLimiter(10);
const startTime2 = Date.now();
const results2: any[] = [];
// Queue 25 requests (should take ~2.5 seconds with 10 req/sec limit)
const promises2 = Array.from({ length: 25 }, (_, i) =>
limiter2.throttle(() => mockApiCall(i + 1))
);
for (const promise of promises2) {
const result = await promise;
results2.push(result);
}
const duration2 = Date.now() - startTime2;
console.log(`✅ Completed 25 requests in ${duration2}ms`);
console.log(` Expected: ~2500ms (25 requests ÷ 10 req/sec)`);
console.log(` Within acceptable range: ${duration2 >= 2300 && duration2 <= 3000 ? 'Yes' : 'No'}\n`);
// Test 3: Verify rate limit enforcement
console.log('Test 3: Verify rate limit enforcement (10 requests/second)');
const limiter3 = new RateLimiter(10);
const timestamps: number[] = [];
// Execute 15 requests and track timestamps
const promises3 = Array.from({ length: 15 }, (_, i) =>
limiter3.throttle(async () => {
const now = Date.now();
timestamps.push(now);
return mockApiCall(i + 1, 1);
})
);
await Promise.all(promises3);
// Check that no more than 10 requests happened in any 1-second window
let maxRequestsInWindow = 0;
for (let i = 0; i < timestamps.length; i++) {
const windowStart = timestamps[i];
const windowEnd = windowStart + 1000;
const requestsInWindow = timestamps.filter(t => t >= windowStart && t < windowEnd).length;
maxRequestsInWindow = Math.max(maxRequestsInWindow, requestsInWindow);
}
console.log(`✅ Maximum requests in any 1-second window: ${maxRequestsInWindow}`);
console.log(` Rate limit respected: ${maxRequestsInWindow <= 10 ? 'Yes' : 'No'}\n`);
// Test 4: Queue length tracking
console.log('Test 4: Queue length tracking');
const limiter4 = new RateLimiter(5);
// Queue multiple requests without awaiting
const promises4 = Array.from({ length: 20 }, (_, i) =>
limiter4.throttle(() => mockApiCall(i + 1, 50))
);
// Check queue length immediately after queuing
await new Promise(resolve => setTimeout(resolve, 10));
const queueLength = limiter4.getQueueLength();
console.log(`✅ Queue length after queuing 20 requests: ${queueLength}`);
console.log(` Queue has pending requests: ${queueLength > 0 ? 'Yes' : 'No'}`);
// Wait for all to complete
await Promise.all(promises4);
const finalQueueLength = limiter4.getQueueLength();
console.log(`✅ Queue length after completion: ${finalQueueLength}`);
console.log(` Queue is empty: ${finalQueueLength === 0 ? 'Yes' : 'No'}\n`);
// Test 5: Current request count tracking
console.log('Test 5: Current request count tracking');
const limiter5 = new RateLimiter(10);
const requestCounts: number[] = [];
// Execute requests and track current count
const promises5 = Array.from({ length: 15 }, (_, i) =>
limiter5.throttle(async () => {
const count = limiter5.getCurrentRequestCount();
requestCounts.push(count);
return mockApiCall(i + 1, 1);
})
);
await Promise.all(promises5);
const maxCount = Math.max(...requestCounts);
console.log(`✅ Maximum concurrent request count: ${maxCount}`);
console.log(` Never exceeded limit: ${maxCount <= 10 ? 'Yes' : 'No'}\n`);
// Test 6: Reset functionality
console.log('Test 6: Reset functionality');
const limiter6 = new RateLimiter(5);
// Queue some requests
const promises6 = Array.from({ length: 10 }, (_, i) =>
limiter6.throttle(() => mockApiCall(i + 1, 100))
);
// Wait a bit then reset
await new Promise(resolve => setTimeout(resolve, 50));
const queueBeforeReset = limiter6.getQueueLength();
limiter6.reset();
const queueAfterReset = limiter6.getQueueLength();
console.log(`✅ Queue length before reset: ${queueBeforeReset}`);
console.log(`✅ Queue length after reset: ${queueAfterReset}`);
console.log(` Reset cleared queue: ${queueAfterReset === 0 ? 'Yes' : 'No'}\n`);
// Test 7: Error handling
console.log('Test 7: Error handling');
const limiter7 = new RateLimiter(10);
let errorCaught = false;
try {
await limiter7.throttle(async () => {
throw new Error('Mock API error');
});
} catch (error) {
errorCaught = true;
}
console.log(`✅ Error properly propagated: ${errorCaught ? 'Yes' : 'No'}`);
// Verify limiter still works after error
const resultAfterError = await limiter7.throttle(() => mockApiCall(1));
console.log(`✅ Limiter functional after error: ${resultAfterError.id === 1 ? 'Yes' : 'No'}\n`);
// Test 8: Parallel execution within limit
console.log('Test 8: Parallel execution within limit');
const limiter8 = new RateLimiter(10);
const startTime8 = Date.now();
// Queue 10 requests that each take 100ms
// With 10 req/sec limit, they should execute in parallel (not sequentially)
const promises8 = Array.from({ length: 10 }, (_, i) =>
limiter8.throttle(() => mockApiCall(i + 1, 100))
);
await Promise.all(promises8);
const duration8 = Date.now() - startTime8;
console.log(`✅ Completed 10 requests (100ms each) in ${duration8}ms`);
console.log(` Executed in parallel: ${duration8 < 500 ? 'Yes' : 'No'}`);
console.log(` (Sequential would take ~1000ms, parallel ~100ms)\n`);
// Test 9: Stress test with many requests
console.log('Test 9: Stress test (100 requests at 10 req/sec)');
const limiter9 = new RateLimiter(10);
const startTime9 = Date.now();
const promises9 = Array.from({ length: 100 }, (_, i) =>
limiter9.throttle(() => mockApiCall(i + 1, 1))
);
await Promise.all(promises9);
const duration9 = Date.now() - startTime9;
console.log(`✅ Completed 100 requests in ${duration9}ms`);
console.log(` Expected: ~10000ms (100 requests ÷ 10 req/sec)`);
console.log(` Within acceptable range: ${duration9 >= 9500 && duration9 <= 11000 ? 'Yes' : 'No'}\n`);
// Test 10: Different rate limits
console.log('Test 10: Custom rate limits');
const limiter10a = new RateLimiter(2); // 2 req/sec
const limiter10b = new RateLimiter(20); // 20 req/sec
const startTime10a = Date.now();
await Promise.all(
Array.from({ length: 6 }, (_, i) =>
limiter10a.throttle(() => mockApiCall(i + 1, 1))
)
);
const duration10a = Date.now() - startTime10a;
const startTime10b = Date.now();
await Promise.all(
Array.from({ length: 40 }, (_, i) =>
limiter10b.throttle(() => mockApiCall(i + 1, 1))
)
);
const duration10b = Date.now() - startTime10b;
console.log(`✅ 6 requests at 2 req/sec: ${duration10a}ms (expected ~3000ms)`);
console.log(`✅ 40 requests at 20 req/sec: ${duration10b}ms (expected ~2000ms)`);
console.log(` Both within acceptable ranges: ${
(duration10a >= 2700 && duration10a <= 3500) &&
(duration10b >= 1800 && duration10b <= 2500) ? 'Yes' : 'No'
}\n`);
console.log('🎉 All rate limiter tests completed successfully!');
} catch (error) {
console.error('❌ Test failed with error:', error);
throw error;
}
}
// Run the tests
testRateLimiter()
.then(() => {
console.log('\n✅ Test script completed successfully');
process.exit(0);
})
.catch((error) => {
console.error('\n❌ Test script failed:', error);
process.exit(1);
});

View file

@ -0,0 +1,100 @@
/**
* Test script for syncing time entries from Autotask
* Run with: npx tsx dev/test-time-entries-sync.ts
*/
import dotenv from 'dotenv';
import path from 'path';
import { AutotaskClient } from '@/lib/services/autotask-client';
import { EntitySyncService } from '@/lib/services/entity-sync';
import { EntityType } from '@/lib/types/sync';
// Load environment variables
dotenv.config({ path: path.resolve(__dirname, '../.env.local') });
async function testTimeEntriesSync() {
console.log('🔍 Testing Time Entries Sync\n');
// Validate environment variables
const requiredEnvVars = [
'AUTOTASK_API_URL',
'AUTOTASK_USERNAME',
'AUTOTASK_SECRET',
'AUTOTASK_API_INTEGRATION_CODE',
];
const missingVars = requiredEnvVars.filter(v => !process.env[v]);
if (missingVars.length > 0) {
console.error('❌ Missing required environment variables:', missingVars.join(', '));
process.exit(1);
}
try {
// Initialize Autotask client
console.log('Initializing Autotask client...');
const autotaskClient = new AutotaskClient({
apiUrl: process.env.AUTOTASK_API_URL!,
username: process.env.AUTOTASK_USERNAME!,
password: process.env.AUTOTASK_SECRET!,
apiIntegrationCode: process.env.AUTOTASK_API_INTEGRATION_CODE!,
});
console.log('✅ Autotask client initialized\n');
// Initialize entity sync service
console.log('Initializing entity sync service...');
const syncService = new EntitySyncService(autotaskClient);
console.log('✅ Entity sync service initialized\n');
// Test sync with a small date range (last 7 days)
console.log('Starting time entries sync (last 7 days)...');
console.log('This will sync time entries from the last 7 days only.\n');
const startTime = Date.now();
try {
const result = await syncService.syncEntity(
EntityType.TIME_ENTRIES,
false, // Full sync (not incremental)
0.019 // 7 days in years
);
const duration = Date.now() - startTime;
console.log('\n✅ Sync completed successfully!');
console.log(`Duration: ${(duration / 1000).toFixed(2)}s`);
console.log('\nResults:');
console.log(` Records added: ${result.recordsAdded}`);
console.log(` Records updated: ${result.recordsUpdated}`);
console.log(` Records deleted: ${result.recordsDeleted}`);
console.log(` Total processed: ${result.recordsAdded + result.recordsUpdated}`);
} catch (syncError) {
console.error('\n❌ Sync failed with error:');
console.error(syncError);
// Try to provide more details
if (syncError instanceof Error) {
console.error('\nError details:');
console.error(' Message:', syncError.message);
console.error(' Stack:', syncError.stack);
}
throw syncError;
}
} catch (error) {
console.error('\n❌ Test failed:', error);
process.exit(1);
}
}
// Run the test
testTimeEntriesSync()
.then(() => {
console.log('\n✅ Test completed successfully');
process.exit(0);
})
.catch((error) => {
console.error('\n❌ Test failed:', error);
process.exit(1);
});

318
dev/ui-interaction-tests.md Normal file
View file

@ -0,0 +1,318 @@
# Admin Sync UI - User Interaction Test Plan
## Test Date
Generated: 2025-11-01
## Purpose
Verify all user interactions in the Admin Sync UI work correctly across different scenarios.
---
## 1. Entity Selection Tests
### 1.1 Individual Entity Selection
- [ ] Click individual entity checkboxes
- [ ] Verify checkbox state changes (checked/unchecked)
- [ ] Verify selected count updates correctly
- [ ] Verify "Sync Selected" button enables/disables based on selection
### 1.2 Select All / Deselect All
- [ ] Click "Select All" button
- [ ] Verify all 13 entities become checked
- [ ] Verify button text changes to "Deselect All"
- [ ] Verify count shows "13 of 13 entities selected"
- [ ] Click "Deselect All" button
- [ ] Verify all entities become unchecked
- [ ] Verify count shows "0 of 13 entities selected"
### 1.3 Disabled State During Sync
- [ ] Start a sync operation
- [ ] Verify all entity checkboxes are disabled
- [ ] Verify "Select All/Deselect All" button is disabled
- [ ] Wait for sync to complete
- [ ] Verify checkboxes re-enable
---
## 2. Sync Button Tests
### 2.1 Full Sync Button
- [ ] Click "Full Sync" button
- [ ] Verify confirmation dialog appears
- [ ] Verify dialog shows warning message about soft-deletes
- [ ] Click "Cancel" - verify dialog closes, no sync starts
- [ ] Click "Full Sync" again
- [ ] Click "Start Full Sync" in dialog
- [ ] Verify dialog closes
- [ ] Verify button shows loading spinner
- [ ] Verify button is disabled during sync
- [ ] Verify success toast appears
- [ ] Verify button returns to normal state after completion
### 2.2 Incremental Sync Button
- [ ] Click "Incremental Sync" button
- [ ] Verify NO confirmation dialog (should start immediately)
- [ ] Verify button shows loading spinner
- [ ] Verify button is disabled during sync
- [ ] Verify success toast appears
- [ ] Verify button returns to normal state
### 2.3 Sync Selected Button
- [ ] With 0 entities selected:
- [ ] Verify button is disabled
- [ ] Click button (should do nothing)
- [ ] Verify error toast: "Please select at least one entity to sync"
- [ ] Select 3 entities
- [ ] Verify button shows "Sync Selected (3)"
- [ ] Verify button is enabled
- [ ] Click button
- [ ] Verify sync starts
- [ ] Verify loading spinner appears
- [ ] During sync:
- [ ] Verify all sync buttons are disabled
### 2.4 Button States During Sync
- [ ] Start any sync operation
- [ ] Verify all three sync buttons show loading spinner
- [ ] Verify all three buttons are disabled
- [ ] Verify buttons cannot be clicked
- [ ] Wait for sync completion
- [ ] Verify all buttons return to normal state
---
## 3. Dashboard Auto-Refresh Tests
### 3.1 Auto-Refresh During Sync
- [ ] Note current sync status data
- [ ] Start a sync operation
- [ ] Wait 5 seconds
- [ ] Verify dashboard data refreshes automatically
- [ ] Verify sync history table refreshes
- [ ] Verify no page flicker or jarring updates
### 3.2 Auto-Refresh Stops After Sync
- [ ] Wait for sync to complete
- [ ] Verify auto-refresh stops
- [ ] Wait 10 seconds
- [ ] Verify no unnecessary refreshes occur
---
## 4. Sync History Table Tests
### 4.1 History Display
- [ ] Verify table shows recent sync operations
- [ ] Verify columns display correctly:
- Entity name (human-readable)
- Sync type (full-sync, incremental, entity-specific)
- Status badge (completed/failed/in_progress)
- Start timestamp
- Duration
- Records added (green)
- Records updated (blue)
- Records deleted (red)
- Triggered by
### 4.2 Pagination
- [ ] If more than 10 records exist:
- [ ] Verify "Previous" button is disabled on page 1
- [ ] Click "Next" button
- [ ] Verify page advances
- [ ] Verify new records load
- [ ] Verify "Previous" button is now enabled
- [ ] Click "Previous" button
- [ ] Verify page goes back
- [ ] Verify "Previous" button is disabled again
### 4.3 Download Functionality
- [ ] Click "JSON" download button
- [ ] Verify JSON file downloads with timestamp in filename
- [ ] Open JSON file and verify:
- Valid JSON format
- Contains all visible history records
- All fields are present
- [ ] Click "CSV" download button
- [ ] Verify CSV file downloads with timestamp in filename
- [ ] Open CSV file and verify:
- Headers are present
- Data rows match table display
- Special characters are properly escaped
- Commas in error messages don't break columns
### 4.4 Download Button States
- [ ] When no history exists:
- [ ] Verify download buttons are hidden
- [ ] When loading:
- [ ] Verify download buttons are disabled
- [ ] After data loads:
- [ ] Verify download buttons are enabled
---
## 5. Sync Dashboard Tests
### 5.1 Entity Status Cards
- [ ] Verify each synced entity has a status card
- [ ] Verify cards show:
- Entity name
- Status badge (completed/failed)
- Time since last sync (e.g., "2 hours ago")
- Records added (green, with + prefix)
- Records updated (blue, with ~ prefix)
- Records deleted (red, with - prefix)
### 5.2 Empty State
- [ ] With no sync history:
- [ ] Verify message: "No sync history available"
### 5.3 Status Badge Colors
- [ ] Verify "completed" status shows success color (green/default)
- [ ] Verify "failed" status shows error color (red/destructive)
- [ ] Verify "in_progress" status shows secondary color
---
## 6. Toast Notifications Tests
### 6.1 Success Notifications
- [ ] Complete a successful sync
- [ ] Verify success toast appears
- [ ] Verify toast message is clear and informative
- [ ] Verify toast auto-dismisses after a few seconds
- [ ] Verify toast can be manually dismissed
### 6.2 Error Notifications
- [ ] Trigger a sync error (if possible, or simulate)
- [ ] Verify error toast appears
- [ ] Verify error message is displayed
- [ ] Verify toast is styled as error (red/destructive)
- [ ] Verify toast can be dismissed
### 6.3 Validation Notifications
- [ ] Click "Sync Selected" with no entities selected
- [ ] Verify error toast: "Please select at least one entity to sync"
---
## 7. Responsive Behavior Tests
### 7.1 Mobile View (< 640px)
- [ ] Resize browser to mobile width
- [ ] Verify sync buttons stack vertically
- [ ] Verify entity checkboxes show in single column
- [ ] Verify sync history table scrolls horizontally
- [ ] Verify pagination buttons show icons only
- [ ] Verify download buttons remain accessible
- [ ] Verify all interactions still work
### 7.2 Tablet View (640px - 1024px)
- [ ] Resize browser to tablet width
- [ ] Verify sync buttons show in 2 columns
- [ ] Verify entity checkboxes show in 2-3 columns
- [ ] Verify dashboard cards show in 2 columns
- [ ] Verify all interactions work smoothly
### 7.3 Desktop View (> 1024px)
- [ ] Resize browser to desktop width
- [ ] Verify sync buttons show in 3 columns
- [ ] Verify entity checkboxes show in 4 columns
- [ ] Verify dashboard cards show in 3-4 columns
- [ ] Verify optimal spacing and layout
---
## 8. Edge Cases and Error Handling
### 8.1 Network Errors
- [ ] Simulate network failure during sync
- [ ] Verify error is caught and displayed
- [ ] Verify UI returns to normal state
- [ ] Verify buttons re-enable
### 8.2 API Errors
- [ ] Trigger API error (401, 500, etc.)
- [ ] Verify error toast displays
- [ ] Verify error message is user-friendly
- [ ] Verify sync state resets properly
### 8.3 Rapid Clicking
- [ ] Rapidly click sync buttons
- [ ] Verify only one sync starts
- [ ] Verify no duplicate requests
- [ ] Verify UI state remains consistent
### 8.4 Browser Back/Forward
- [ ] Start a sync
- [ ] Click browser back button
- [ ] Return to page
- [ ] Verify sync state is handled correctly
---
## 9. Accessibility Tests
### 9.1 Keyboard Navigation
- [ ] Tab through all interactive elements
- [ ] Verify focus indicators are visible
- [ ] Verify all buttons are keyboard accessible
- [ ] Press Enter/Space on focused buttons
- [ ] Verify actions trigger correctly
### 9.2 Screen Reader Support
- [ ] Verify buttons have descriptive labels
- [ ] Verify checkboxes have associated labels
- [ ] Verify status badges have meaningful text
- [ ] Verify loading states are announced
---
## 10. Performance Tests
### 10.1 Large Dataset Handling
- [ ] Load page with 100+ sync history records
- [ ] Verify pagination works smoothly
- [ ] Verify no lag when scrolling
- [ ] Verify download functions work with large datasets
### 10.2 Concurrent Operations
- [ ] Open multiple browser tabs
- [ ] Start sync in one tab
- [ ] Verify other tabs can still view data
- [ ] Verify no conflicts or race conditions
---
## Test Results Summary
**Total Tests**: ~100+ individual test cases
**Status**: Ready for manual testing
### Critical Path Tests (Must Pass)
1. Entity selection and deselection
2. Full sync with confirmation dialog
3. Incremental sync without confirmation
4. Sync Selected with validation
5. Dashboard auto-refresh during sync
6. History table pagination
7. Download JSON/CSV functionality
8. Toast notifications for success/error
9. Responsive layout on mobile/tablet/desktop
10. Button disabled states during sync
### Notes for Testing
- Test with actual PostgreSQL database and Autotask API connection
- Use browser DevTools to simulate mobile/tablet viewports
- Test in multiple browsers (Chrome, Firefox, Safari)
- Monitor console for errors during testing
- Check network tab for API calls
### Recommended Testing Order
1. Start with entity selection tests (foundation)
2. Test each sync button type
3. Verify dashboard updates
4. Test history table features
5. Test responsive behavior
6. Test edge cases and errors
7. Verify accessibility
8. Performance testing last