Add comprehensive admin features and multi-system integration
- Add admin dashboard with sync controls and data browser - Implement RMM, Auvik, and Addigy organization mappings - Add chunked ticket sync with progress tracking - Implement entity sync service with rate limiting - Add analytics engine and performance optimizer - Create data browser for all PSA entities - Add navigation components and UI improvements - Implement background processing and sync services - Add comprehensive documentation and migration scripts - Update configuration items with multi-system support - Enhance contact management and purchase history - Add issue type assignment and LLM analyzer - Improve error handling and logging utilities
This commit is contained in:
root
parent
e8462ef301
commit
6eee14f8af
171 changed files with 32671 additions and 621 deletions
332
docs/SYNC_PROGRESS_TRACKING.md
Normal file
332
docs/SYNC_PROGRESS_TRACKING.md
Normal file
|
|
@ -0,0 +1,332 @@
|
|||
# Sync Progress Tracking Implementation
|
||||
|
||||
## Overview
|
||||
|
||||
Implemented real-time progress tracking for entity sync operations with persistent state that allows users to navigate away and return to see accurate progress.
|
||||
|
||||
## Problem Solved
|
||||
|
||||
1. **100-Page Limit Removed** - Previously, syncs were capped at 50,000 records (100 pages × 500 records/page)
|
||||
2. **No Progress Visibility** - Users couldn't see sync progress or know when operations would complete
|
||||
3. **Lost Progress on Navigation** - Navigating away from the sync page lost all progress information
|
||||
|
||||
## Solution Architecture
|
||||
|
||||
### Backend Components
|
||||
|
||||
#### 1. **SyncProgressTracker** (`lib/services/sync-progress-tracker.ts`)
|
||||
Global singleton service that tracks sync progress in memory.
|
||||
|
||||
**Features:**
|
||||
- Tracks multiple concurrent syncs
|
||||
- Stores progress state with phases (fetching, mapping, upserting, deleting)
|
||||
- Persists across API calls (in-memory during app lifetime)
|
||||
- Auto-cleanup of old sync records (keeps last 10 per entity)
|
||||
|
||||
**Key Methods:**
|
||||
```typescript
|
||||
startSync(syncId, entityType) // Initialize tracking
|
||||
updateProgress(syncId, updates) // Update progress
|
||||
completeSync(syncId, totalRecords) // Mark complete
|
||||
failSync(syncId, error) // Mark failed
|
||||
getProgress(syncId) // Get specific sync
|
||||
getLatestSync(entityType) // Get latest for entity
|
||||
```
|
||||
|
||||
#### 2. **Entity Sync Service Updates** (`lib/services/entity-sync.ts`)
|
||||
Integrated progress tracking at key phases:
|
||||
|
||||
```typescript
|
||||
async syncEntity(entity, isIncremental, yearsBack, syncId?) {
|
||||
const trackingId = syncId || `${entity}_${Date.now()}`;
|
||||
|
||||
syncProgressTracker.startSync(trackingId, entity);
|
||||
|
||||
// Phase 1: Fetching
|
||||
syncProgressTracker.updateProgress(trackingId, { phase: 'fetching' });
|
||||
|
||||
// Phase 2: Mapping
|
||||
syncProgressTracker.updateProgress(trackingId, {
|
||||
totalRecords: count,
|
||||
phase: 'mapping'
|
||||
});
|
||||
|
||||
// Phase 3: Upserting
|
||||
syncProgressTracker.updateProgress(trackingId, { phase: 'upserting' });
|
||||
|
||||
// Phase 4: Deleting (full sync only)
|
||||
syncProgressTracker.updateProgress(trackingId, { phase: 'deleting' });
|
||||
|
||||
// Complete
|
||||
syncProgressTracker.completeSync(trackingId, totalRecords);
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. **Progress API Endpoint** (`app/api/sync/progress/route.ts`)
|
||||
RESTful endpoint for polling progress:
|
||||
|
||||
```bash
|
||||
# Get specific sync
|
||||
GET /api/sync/progress?syncId=time_entries_1730000000
|
||||
|
||||
# Get latest sync for entity
|
||||
GET /api/sync/progress?entityType=time_entries
|
||||
|
||||
# Get all active syncs
|
||||
GET /api/sync/progress
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"progress": {
|
||||
"syncId": "time_entries_1730000000",
|
||||
"entityType": "time_entries",
|
||||
"status": "running",
|
||||
"currentPage": 0,
|
||||
"totalRecords": 75000,
|
||||
"startTime": 1730000000000,
|
||||
"phase": "upserting"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Frontend Components
|
||||
|
||||
#### 1. **EntitySyncProgress Component** (`components/admin/EntitySyncProgress.tsx`)
|
||||
Reusable progress display following shadcn/ui best practices.
|
||||
|
||||
**Features:**
|
||||
- ✅ Animated progress bar (smooth transitions)
|
||||
- ✅ Phase indicators with icons
|
||||
- ✅ Real-time polling (every 2 seconds)
|
||||
- ✅ Persistent across navigation (polls by syncId or entityType)
|
||||
- ✅ Dark mode support
|
||||
- ✅ Accessibility (ARIA labels)
|
||||
- ✅ Auto-cleanup on completion/failure
|
||||
|
||||
**Usage:**
|
||||
```tsx
|
||||
<EntitySyncProgress
|
||||
entityType="time_entries"
|
||||
syncId="time_entries_1730000000"
|
||||
onComplete={() => console.log('Sync done!')}
|
||||
onError={(error) => console.error(error)}
|
||||
/>
|
||||
```
|
||||
|
||||
**Visual States:**
|
||||
- **Running** - Blue badge, spinning loader, animated progress
|
||||
- **Completed** - Green badge, checkmark, success message
|
||||
- **Failed** - Red badge, X icon, error message
|
||||
|
||||
**Progress Calculation:**
|
||||
```typescript
|
||||
fetching → 25%
|
||||
mapping → 50%
|
||||
upserting → 75%
|
||||
deleting → 90%
|
||||
completed → 100%
|
||||
```
|
||||
|
||||
#### 2. **SyncControlPanel Integration** (`components/admin/SyncControlPanel.tsx`)
|
||||
Integrated progress tracking for entity-specific syncs.
|
||||
|
||||
**Auto-tracking:**
|
||||
- Detects single-entity syncs
|
||||
- Generates unique syncId
|
||||
- Shows progress component
|
||||
- Auto-hides on completion
|
||||
|
||||
**Multi-entity syncs:**
|
||||
- Still supported
|
||||
- No individual progress (would need separate implementation)
|
||||
|
||||
## shadcn/ui Best Practices Applied
|
||||
|
||||
### 1. **Component Composition**
|
||||
```tsx
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>Entity Sync</CardTitle>
|
||||
<CardDescription>Phase description</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<Progress value={animatedProgress} />
|
||||
</CardContent>
|
||||
</Card>
|
||||
```
|
||||
|
||||
### 2. **Smooth Animations**
|
||||
```typescript
|
||||
// Gradual progress updates
|
||||
const step = (targetProgress - animatedProgress) / 10;
|
||||
const interval = setInterval(() => {
|
||||
setAnimatedProgress(prev => prev + step);
|
||||
}, 50);
|
||||
```
|
||||
|
||||
### 3. **Dark Mode Support**
|
||||
```tsx
|
||||
className="dark:bg-gray-700 dark:border-gray-700"
|
||||
```
|
||||
|
||||
### 4. **Accessibility**
|
||||
```tsx
|
||||
<Progress
|
||||
value={progress}
|
||||
aria-label={`Sync progress: ${Math.round(progress)}%`}
|
||||
/>
|
||||
```
|
||||
|
||||
### 5. **Loading States**
|
||||
```tsx
|
||||
{status === 'running' && (
|
||||
<Loader2 className="h-5 w-5 animate-spin text-blue-500" />
|
||||
)}
|
||||
```
|
||||
|
||||
### 6. **Responsive Design**
|
||||
```tsx
|
||||
<div className="grid grid-cols-2 gap-4">
|
||||
{/* Stats */}
|
||||
</div>
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### 1. Sync Time Entries with Progress
|
||||
```typescript
|
||||
// From UI
|
||||
1. Navigate to /admin/sync
|
||||
2. Select "Time Entries" entity
|
||||
3. Click "Sync Selected Entities"
|
||||
4. Watch real-time progress
|
||||
5. Navigate away (progress persists)
|
||||
6. Return to see updated progress
|
||||
```
|
||||
|
||||
### 2. Programmatic Sync with Tracking
|
||||
```typescript
|
||||
import { syncProgressTracker } from '@/lib/services/sync-progress-tracker';
|
||||
|
||||
const syncId = `time_entries_${Date.now()}`;
|
||||
|
||||
// Start sync with tracking
|
||||
await entitySyncService.syncEntity(
|
||||
EntityType.TIME_ENTRIES,
|
||||
false,
|
||||
1,
|
||||
syncId
|
||||
);
|
||||
|
||||
// Poll progress
|
||||
const progress = syncProgressTracker.getProgress(syncId);
|
||||
console.log(progress.phase, progress.totalRecords);
|
||||
```
|
||||
|
||||
### 3. Monitor from API
|
||||
```bash
|
||||
# Start sync
|
||||
curl -X POST http://localhost:3000/api/sync/entity \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"entities": ["time_entries"], "yearsBack": 1}'
|
||||
|
||||
# Poll progress
|
||||
while true; do
|
||||
curl http://localhost:3000/api/sync/progress?entityType=time_entries
|
||||
sleep 2
|
||||
done
|
||||
```
|
||||
|
||||
## Key Improvements
|
||||
|
||||
### Before
|
||||
- ❌ 50,000 record limit
|
||||
- ❌ No progress visibility
|
||||
- ❌ Lost progress on navigation
|
||||
- ❌ No phase information
|
||||
- ❌ No error details
|
||||
|
||||
### After
|
||||
- ✅ Unlimited records (removed page limit)
|
||||
- ✅ Real-time progress tracking
|
||||
- ✅ Persistent across navigation
|
||||
- ✅ Detailed phase indicators
|
||||
- ✅ Comprehensive error reporting
|
||||
- ✅ Animated progress bar
|
||||
- ✅ Dark mode support
|
||||
- ✅ Accessibility compliant
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Polling Frequency
|
||||
- **2 seconds** - Good balance between responsiveness and server load
|
||||
- Stops polling when sync completes/fails
|
||||
- Cleanup interval prevents memory leaks
|
||||
|
||||
### Memory Management
|
||||
- Keeps last 10 syncs per entity type
|
||||
- Auto-cleanup on completion
|
||||
- In-memory storage (resets on app restart)
|
||||
|
||||
### Future Enhancements
|
||||
- **WebSocket support** - Real-time push instead of polling
|
||||
- **Persistent storage** - Redis/database for cross-instance tracking
|
||||
- **Page-level progress** - Track individual API pages during fetch
|
||||
- **Estimated time remaining** - Calculate based on current rate
|
||||
|
||||
## Testing
|
||||
|
||||
### Manual Test
|
||||
```bash
|
||||
# 1. Start a time entries sync (1 year)
|
||||
# 2. Observe progress phases:
|
||||
# - Fetching (0-25%)
|
||||
# - Mapping (25-50%)
|
||||
# - Upserting (50-75%)
|
||||
# - Deleting (75-90%)
|
||||
# - Completed (100%)
|
||||
# 3. Navigate to another page
|
||||
# 4. Return to sync page
|
||||
# 5. Verify progress is still visible and accurate
|
||||
```
|
||||
|
||||
### API Test
|
||||
```bash
|
||||
# Terminal 1: Start sync
|
||||
curl -X POST http://localhost:3000/api/sync/entity \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"entities": ["time_entries"], "yearsBack": 1}'
|
||||
|
||||
# Terminal 2: Monitor progress
|
||||
watch -n 2 'curl -s http://localhost:3000/api/sync/progress?entityType=time_entries | jq'
|
||||
```
|
||||
|
||||
## Files Modified
|
||||
|
||||
### Backend
|
||||
- `/lib/services/autotask-client.ts` - Removed 100-page limit
|
||||
- `/lib/services/entity-sync.ts` - Added progress tracking
|
||||
- `/lib/services/sync-progress-tracker.ts` - New progress tracker service
|
||||
- `/app/api/sync/progress/route.ts` - New progress API endpoint
|
||||
|
||||
### Frontend
|
||||
- `/components/admin/EntitySyncProgress.tsx` - New progress component
|
||||
- `/components/admin/SyncControlPanel.tsx` - Integrated progress tracking
|
||||
|
||||
### Documentation
|
||||
- `/docs/SYNC_PROGRESS_TRACKING.md` - This file
|
||||
- `/docs/TIME_ENTRY_FIELD_MAPPING.md` - Field mapping analysis
|
||||
- `/docs/TIME_ENTRIES_SORTING_FIX.md` - Sorting implementation
|
||||
|
||||
## Conclusion
|
||||
|
||||
The sync progress tracking system provides:
|
||||
1. **Visibility** - Users see exactly what's happening
|
||||
2. **Persistence** - Progress survives navigation
|
||||
3. **Scalability** - No record limits
|
||||
4. **UX** - Beautiful, accessible, responsive UI
|
||||
5. **Reliability** - Error handling and recovery
|
||||
|
||||
This implementation follows shadcn/ui best practices and provides a production-ready solution for long-running sync operations.
|
||||
Loading…
Add table
Add a link
Reference in a new issue