Add comprehensive admin features and multi-system integration
- Add admin dashboard with sync controls and data browser - Implement RMM, Auvik, and Addigy organization mappings - Add chunked ticket sync with progress tracking - Implement entity sync service with rate limiting - Add analytics engine and performance optimizer - Create data browser for all PSA entities - Add navigation components and UI improvements - Implement background processing and sync services - Add comprehensive documentation and migration scripts - Update configuration items with multi-system support - Enhance contact management and purchase history - Add issue type assignment and LLM analyzer - Improve error handling and logging utilities
This commit is contained in:
root
parent
e8462ef301
commit
6eee14f8af
171 changed files with 32671 additions and 621 deletions
133
docs/CHUNKED_SYNC_IMPLEMENTATION.md
Normal file
133
docs/CHUNKED_SYNC_IMPLEMENTATION.md
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
# Chunked Ticket Sync Implementation
|
||||
|
||||
## Overview
|
||||
This implementation adds monthly chunking for ticket synchronization to prevent timeouts and API failures when syncing large date ranges.
|
||||
|
||||
## Problem Solved
|
||||
- **Previous Issue**: Syncing tickets over large date ranges (e.g., 2+ years) would often timeout or fail, causing the entire sync to fail
|
||||
- **Solution**: Break the sync into monthly chunks, process each chunk independently, and continue even if individual chunks fail
|
||||
|
||||
## Key Features
|
||||
|
||||
### 1. Monthly Chunking Logic
|
||||
- **Location**: `/opt/stacks/pulse/lib/services/entity-sync.ts`
|
||||
- **Method**: `syncTicketsChunked()`
|
||||
- Automatically calculates monthly date ranges based on `yearsBack` parameter
|
||||
- Processes each month independently with its own API request
|
||||
- Continues processing even if individual chunks fail
|
||||
- Aggregates results across all chunks
|
||||
|
||||
### 2. Progress Tracking
|
||||
- **Location**: `/opt/stacks/pulse/lib/types/sync.ts`
|
||||
- Added `ChunkProgress` interface for detailed tracking
|
||||
- Extended `SyncProgress` interface with chunk-specific fields:
|
||||
- `currentChunk`: Current chunk being processed
|
||||
- `totalChunks`: Total number of chunks
|
||||
- `chunkDescription`: Human-readable description (e.g., "Jan 2024")
|
||||
|
||||
### 3. Animated Progress Component
|
||||
- **Location**: `/opt/stacks/pulse/components/admin/ChunkedSyncProgress.tsx`
|
||||
- Displays real-time progress with animated progress bar
|
||||
- Shows current chunk being processed
|
||||
- Lists failed chunks with error messages
|
||||
- Provides completion summary
|
||||
|
||||
### 4. API Endpoint
|
||||
- **Location**: `/opt/stacks/pulse/app/api/sync/tickets-chunked/route.ts`
|
||||
- **Endpoint**: `POST /api/sync/tickets-chunked`
|
||||
- **Parameters**:
|
||||
- `yearsBack`: Number of years to sync (default: 2)
|
||||
- `triggeredBy`: User identifier (default: 'api')
|
||||
|
||||
### 5. UI Integration
|
||||
- **Location**: `/opt/stacks/pulse/components/admin/SyncControlPanel.tsx`
|
||||
- Added "Chunked Tickets" button with distinctive blue styling
|
||||
- Integrated progress component that appears during sync
|
||||
- Disabled other sync buttons while chunked sync is running
|
||||
|
||||
## Usage
|
||||
|
||||
### From UI
|
||||
1. Navigate to the Admin Sync page
|
||||
2. Select desired date range (e.g., "Last 2 Years")
|
||||
3. Click "Chunked Tickets" button
|
||||
4. Monitor progress in the animated progress card
|
||||
5. View completion summary or failed chunks
|
||||
|
||||
### From API
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/sync/tickets-chunked \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"yearsBack": 2, "triggeredBy": "admin"}'
|
||||
```
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Chunking Algorithm
|
||||
```typescript
|
||||
// Splits date range into monthly chunks
|
||||
private calculateMonthlyChunks(yearsBack: number) {
|
||||
const now = new Date();
|
||||
const startDate = new Date(now);
|
||||
startDate.setFullYear(now.getFullYear() - yearsBack);
|
||||
|
||||
// Creates array of {startDate, endDate} for each month
|
||||
// Example for 2 years: ~24 chunks
|
||||
}
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
- Each chunk is wrapped in try-catch
|
||||
- Failed chunks are logged but don't stop the sync
|
||||
- Failed chunk descriptions are collected and displayed
|
||||
- Partial success is possible (some chunks succeed, others fail)
|
||||
|
||||
### Date Filtering
|
||||
- Uses Autotask API `createDate` field
|
||||
- Filters: `createDate >= chunkStart AND createDate < chunkEnd`
|
||||
- Ensures no overlap or gaps between chunks
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Reliability**: Individual chunk failures don't break entire sync
|
||||
2. **Progress Visibility**: Users can see exactly which months are being processed
|
||||
3. **Timeout Prevention**: Smaller API requests are less likely to timeout
|
||||
4. **Partial Recovery**: Can resume from failed chunks without re-syncing everything
|
||||
5. **Better UX**: Animated progress bar provides feedback during long operations
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Recommended Improvements
|
||||
1. **WebSocket/SSE Integration**: Real-time progress updates instead of simulated progress
|
||||
2. **Chunk Retry Logic**: Automatically retry failed chunks with exponential backoff
|
||||
3. **Configurable Chunk Size**: Allow users to choose weekly, monthly, or quarterly chunks
|
||||
4. **Resume Capability**: Save progress and resume from last successful chunk
|
||||
5. **Parallel Processing**: Process multiple chunks concurrently (with rate limiting)
|
||||
6. **Database Tracking**: Store chunk progress in database for persistence
|
||||
|
||||
### Code Locations for Future Work
|
||||
- **WebSocket Handler**: Create `/app/api/sync/tickets-chunked/stream/route.ts`
|
||||
- **Progress Store**: Add Redis or database table for chunk progress
|
||||
- **Retry Logic**: Enhance `syncTicketsChunked()` method in `entity-sync.ts`
|
||||
|
||||
## Testing
|
||||
|
||||
### Manual Testing Steps
|
||||
1. Set date range to "Last 2 Years" or "Last 5 Years"
|
||||
2. Click "Chunked Tickets" button
|
||||
3. Verify progress bar animates smoothly
|
||||
4. Check console logs for chunk-by-chunk progress
|
||||
5. Verify sync history shows completed records
|
||||
6. Test with intentional API failures to verify error handling
|
||||
|
||||
### Expected Behavior
|
||||
- Progress bar should animate from 0% to 100%
|
||||
- Each chunk should log: `[tickets] Processing chunk X/Y: Month Year`
|
||||
- Failed chunks should be listed in red error box
|
||||
- Completion should show total records processed
|
||||
|
||||
## Notes
|
||||
- Current implementation uses simulated progress updates (5-second timeout)
|
||||
- For production use, implement real-time progress tracking via WebSocket or polling
|
||||
- Chunked sync is independent of regular sync operations
|
||||
- Can be run alongside other entity syncs
|
||||
Loading…
Add table
Add a link
Reference in a new issue