feat: add scheduled sync system with admin UI

Implements comprehensive scheduled sync system using node-cron with
full admin interface for configuration and monitoring.

Features:
- Configurable sync schedules with cron expressions
- Enable/disable schedules without deletion
- Manual trigger for testing
- Status monitoring (last run, next run, success/failure)
- Error tracking and display
- Incremental and full sync support
- Multiple concurrent schedules
- Admin UI with schedule management

Components:

1. Sync Scheduler Service (lib/services/sync-scheduler.ts)
   - node-cron integration for scheduling
   - Database-backed schedule configuration
   - Automatic initialization on startup
   - Prevents concurrent runs of same schedule
   - Calculates next run times
   - Tracks execution status and errors

2. Database Schema (sync_schedules table)
   - Schedule configuration storage
   - Execution history tracking
   - Last run status and errors
   - Next run calculation

3. API Endpoints
   - GET /api/sync/schedules - List all schedules
   - POST /api/sync/schedules - Create schedule
   - GET /api/sync/schedules/[id] - Get schedule
   - PATCH /api/sync/schedules/[id] - Update schedule
   - DELETE /api/sync/schedules/[id] - Delete schedule
   - POST /api/sync/schedules/[id]/trigger - Manual trigger

4. Admin UI (components/admin/SyncScheduler.tsx)
   - View all schedules with status
   - Create/edit/delete schedules
   - Enable/disable toggle
   - Manual trigger button
   - Cron expression presets
   - Real-time status updates
   - Error message display
   - Next run countdown

5. Default Schedules (created on first startup, disabled)
   - Daily Incremental: 2 AM daily (0 2 * * *)
   - Weekly Full: 3 AM Sunday (0 3 * * 0)

Admin Interface:
- New 'Schedules' tab in sync page
- Schedule cards with status badges
- Enable/disable with play/pause button
- Manual trigger with clock button
- Edit dialog with cron presets
- Create dialog for new schedules
- Real-time status (running, next run, last run)
- Success/failure indicators
- Error message alerts

Cron Features:
- Full cron expression support
- Validation before saving
- Common presets (daily, weekly, hourly)
- Next run time calculation
- Automatic schedule restart on config change

Monitoring:
- Last run timestamp
- Next run countdown (e.g., 'in 2h 15m')
- Success/failure status with icons
- Error messages for failed syncs
- Running indicator (animated badge)
- Schedule validity check

Dependencies:
- node-cron: ^3.0.3
- @types/node-cron: ^3.0.11

UI Components:
- Alert component added (components/ui/alert.tsx)
- Integrated into sync page tabs
- Responsive design

Documentation:
- Complete guide (docs/SCHEDULED_SYNCS.md)
- Cron expression reference
- Best practices
- Troubleshooting guide
- API reference
- Database schema

Use Cases:
1. Daily incremental sync for recent changes
2. Weekly full sync for data integrity
3. Custom schedules for specific needs
4. Off-peak hour automation
5. Backup for webhook failures

Benefits:
- No manual intervention required
- Consistent data freshness
- Flexible scheduling
- Easy monitoring
- Error tracking
- Manual override available

Next Steps:
1. Restart application to initialize scheduler
2. Navigate to Admin → Sync → Schedules tab
3. Enable default schedules or create custom ones
4. Monitor first runs for success
5. Adjust schedules as needed

Files Added/Modified:
- lib/services/sync-scheduler.ts (new)
- app/api/sync/schedules/route.ts (new)
- app/api/sync/schedules/[id]/route.ts (new)
- app/api/sync/schedules/[id]/trigger/route.ts (new)
- components/admin/SyncScheduler.tsx (new)
- components/ui/alert.tsx (new)
- app/admin/sync/page.tsx (modified - added Schedules tab)
- docs/SCHEDULED_SYNCS.md (new)
- package.json (node-cron added)
This commit is contained in:
root 2026-01-26 10:24:58 -05:00
parent a093f8787c
commit f117210c9d
10 changed files with 5860 additions and 320 deletions

616
docs/SCHEDULED_SYNCS.md Normal file
View file

@ -0,0 +1,616 @@
# Scheduled Syncs Guide
## Overview
Pulse supports automatic scheduled syncs using node-cron. This allows you to configure daily, weekly, or custom sync schedules without manual intervention.
---
## Features
- ✅ **Configurable Schedules** - Create multiple sync schedules with different frequencies
- ✅ **Cron Expressions** - Full cron syntax support for flexible scheduling
- ✅ **Admin UI** - Manage schedules through the web interface
- ✅ **Enable/Disable** - Toggle schedules on/off without deleting them
- ✅ **Manual Trigger** - Run any schedule immediately for testing
- ✅ **Status Monitoring** - View last run, next run, and success/failure status
- ✅ **Error Tracking** - See error messages for failed syncs
- ✅ **Incremental or Full** - Choose sync type per schedule
---
## Default Schedules
Two default schedules are created on first startup (disabled by default):
### **1. Daily Incremental Sync**
- **Schedule:** Every day at 2 AM (`0 2 * * *`)
- **Type:** Incremental (last 24 hours)
- **Purpose:** Keep data up-to-date with daily changes
- **Status:** Disabled (enable via UI)
### **2. Weekly Full Sync**
- **Schedule:** Every Sunday at 3 AM (`0 3 * * 0`)
- **Type:** Full (2 years back)
- **Purpose:** Ensure data integrity with complete sync
- **Status:** Disabled (enable via UI)
---
## Managing Schedules
### **Via Admin UI**
1. **Navigate to Sync Page**
- Go to Admin → Sync
- Click on "Schedules" tab
2. **View Schedules**
- See all configured schedules
- Check status (enabled/disabled, running, last run, next run)
- View error messages for failed syncs
3. **Enable/Disable Schedule**
- Click the Play/Pause button
- Schedule starts/stops immediately
4. **Manually Trigger Schedule**
- Click the Clock button
- Sync runs immediately (doesn't affect schedule)
5. **Edit Schedule**
- Click "Edit" button
- Modify name, description, cron expression, sync type
- Changes take effect immediately
6. **Create New Schedule**
- Click "New Schedule" button
- Fill in details:
- **ID:** Unique identifier (e.g., `hourly-incremental`)
- **Name:** Display name (e.g., `Hourly Incremental Sync`)
- **Description:** What this schedule does
- **Sync Type:** Incremental or Full
- **Cron Expression:** When to run (use presets or custom)
- **Enable:** Start immediately or leave disabled
7. **Delete Schedule**
- Click Trash button
- Confirm deletion
- Schedule is permanently removed
### **Via API**
**Get All Schedules:**
```bash
curl http://localhost:3100/api/sync/schedules
```
**Get Specific Schedule:**
```bash
curl http://localhost:3100/api/sync/schedules/daily-incremental
```
**Create Schedule:**
```bash
curl -X POST http://localhost:3100/api/sync/schedules \
-H "Content-Type: application/json" \
-d '{
"id": "hourly-tickets",
"name": "Hourly Ticket Sync",
"description": "Sync tickets every hour",
"cron_expression": "0 * * * *",
"sync_type": "incremental",
"is_enabled": true
}'
```
**Update Schedule:**
```bash
curl -X PATCH http://localhost:3100/api/sync/schedules/daily-incremental \
-H "Content-Type: application/json" \
-d '{
"is_enabled": true,
"cron_expression": "0 3 * * *"
}'
```
**Delete Schedule:**
```bash
curl -X DELETE http://localhost:3100/api/sync/schedules/hourly-tickets
```
**Trigger Schedule Manually:**
```bash
curl -X POST http://localhost:3100/api/sync/schedules/daily-incremental/trigger
```
---
## Cron Expression Guide
Cron format: `minute hour day month weekday`
### **Common Patterns**
| Expression | Description |
|------------|-------------|
| `0 2 * * *` | Every day at 2 AM |
| `0 3 * * 0` | Every Sunday at 3 AM |
| `0 */6 * * *` | Every 6 hours |
| `0 0 1 * *` | First day of every month at midnight |
| `30 4 * * 1-5` | 4:30 AM on weekdays |
| `0 */2 * * *` | Every 2 hours |
| `15 14 1 * *` | 2:15 PM on the first of every month |
### **Field Values**
- **Minute:** 0-59
- **Hour:** 0-23 (0 = midnight, 12 = noon)
- **Day:** 1-31
- **Month:** 1-12
- **Weekday:** 0-7 (0 or 7 = Sunday)
### **Special Characters**
- `*` - Any value
- `,` - List (e.g., `1,15` = 1st and 15th)
- `-` - Range (e.g., `1-5` = Monday through Friday)
- `/` - Step (e.g., `*/2` = every 2 units)
### **Examples**
```
0 2 * * * # 2:00 AM every day
0 3 * * 0 # 3:00 AM every Sunday
0 */4 * * * # Every 4 hours
30 9 * * 1-5 # 9:30 AM on weekdays
0 0 1,15 * * # Midnight on 1st and 15th
```
---
## Recommended Schedules
### **Small Organization (< 100 tickets/day)**
```
Daily Incremental: 0 2 * * * (2 AM daily)
Weekly Full: 0 3 * * 0 (3 AM Sunday)
```
### **Medium Organization (100-500 tickets/day)**
```
Incremental: 0 */6 * * * (Every 6 hours)
Weekly Full: 0 3 * * 0 (3 AM Sunday)
```
### **Large Organization (> 500 tickets/day)**
```
Incremental: 0 */2 * * * (Every 2 hours)
Daily Full: 0 3 * * * (3 AM daily)
```
### **With Webhooks**
If using webhooks for real-time updates:
```
Daily Incremental: 0 2 * * * (Backup for missed webhooks)
Weekly Full: 0 3 * * 0 (Data integrity check)
```
---
## Best Practices
### **1. Stagger Schedules**
Don't run multiple syncs at the same time:
```
Daily Incremental: 0 2 * * * (2 AM)
Weekly Full: 0 3 * * 0 (3 AM Sunday)
```
### **2. Off-Peak Hours**
Schedule syncs during low-usage periods:
- ✅ 2-4 AM (recommended)
- ✅ Late evening (10 PM - midnight)
- ❌ Business hours (9 AM - 5 PM)
### **3. Start Disabled**
Create new schedules disabled, test manually first:
```json
{
"is_enabled": false
}
```
Then enable after verifying it works.
### **4. Monitor First Runs**
After enabling a schedule:
1. Wait for first scheduled run
2. Check sync history for success
3. Review any error messages
4. Adjust schedule if needed
### **5. Incremental + Full Strategy**
Combine both for best results:
- **Incremental:** Daily or more frequent (fast, recent changes)
- **Full:** Weekly or monthly (slow, ensures data integrity)
### **6. Test with Manual Trigger**
Before enabling a schedule:
1. Create schedule (disabled)
2. Click "Trigger" button to run manually
3. Verify sync completes successfully
4. Enable schedule
---
## Monitoring
### **Schedule Status**
Each schedule shows:
- **Enabled/Disabled** - Current state
- **Running** - Currently executing (animated badge)
- **Next Run** - When it will run next (e.g., "in 2h 15m")
- **Last Run** - When it last executed
- **Last Status** - Success ✓ or Failed ✗
- **Error Message** - Details if failed
### **Check Logs**
View application logs for scheduler activity:
```bash
# Docker logs
docker logs pulse-app | grep SCHEDULER
# Recent scheduler events
docker logs pulse-app --tail 100 | grep SCHEDULER
```
### **Database Queries**
```sql
-- View all schedules
SELECT * FROM sync_schedules ORDER BY id;
-- View enabled schedules
SELECT id, name, cron_expression, next_run
FROM sync_schedules
WHERE is_enabled = true;
-- View failed schedules
SELECT id, name, last_run, last_error
FROM sync_schedules
WHERE last_status = 'failed';
-- View schedule history
SELECT
s.name,
s.last_run,
s.last_status,
h.records_added,
h.records_updated
FROM sync_schedules s
LEFT JOIN sync_history h ON h.started_at = s.last_run
WHERE s.last_run IS NOT NULL
ORDER BY s.last_run DESC;
```
---
## Troubleshooting
### **Schedule Not Running**
**Check 1: Is it enabled?**
- Look for "Enabled" badge
- If disabled, click Play button
**Check 2: Is cron expression valid?**
- Look for "Invalid Cron" badge
- Edit schedule and fix expression
**Check 3: Check next run time**
- Ensure next run is in the future
- If "Calculating...", wait a moment and refresh
**Check 4: Application running?**
- Scheduler only works when app is running
- Check `docker ps` or process status
**Check 5: Check logs**
```bash
docker logs pulse-app | grep "SCHEDULER.*daily-incremental"
```
### **Schedule Failing**
**Check Error Message:**
- View in UI under schedule card
- Shows last error from failed sync
**Common Errors:**
1. **"A sync operation is already in progress"**
- Another sync is running
- Wait for it to complete
- Consider adjusting schedule times
2. **"Failed to connect to Autotask API"**
- Check API credentials
- Verify network connectivity
- Check Autotask API status
3. **"Database connection error"**
- Check PostgreSQL is running
- Verify database credentials
- Check disk space
**View Sync History:**
- Go to "Sync History" tab
- Filter by entity type
- Check error details
### **Schedule Running Too Long**
If a sync takes longer than expected:
1. **Check current sync status:**
```bash
curl http://localhost:3100/api/sync/status
```
2. **Review sync history:**
- Look at duration of previous syncs
- Identify slow entities
3. **Consider splitting:**
- Create separate schedules for slow entities
- Run them at different times
### **Missed Schedules**
If application was down during scheduled time:
- Schedule will NOT run retroactively
- Next run will be at next scheduled time
- Consider manual trigger if data is critical
---
## Database Schema
### **sync_schedules Table**
```sql
CREATE TABLE sync_schedules (
id VARCHAR(50) PRIMARY KEY,
name VARCHAR(100) NOT NULL,
description TEXT,
cron_expression VARCHAR(50) NOT NULL,
sync_type VARCHAR(20) NOT NULL, -- 'incremental' or 'full'
years_back INTEGER DEFAULT 2,
is_enabled BOOLEAN NOT NULL DEFAULT true,
last_run TIMESTAMP,
next_run TIMESTAMP,
last_status VARCHAR(20), -- 'success' or 'failed'
last_error TEXT,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
```
---
## Advanced Configuration
### **Environment Variables**
Currently, schedules are managed via database and UI. Future versions may support:
```env
# Example (not yet implemented)
ENABLE_SCHEDULED_SYNC=true
DEFAULT_INCREMENTAL_CRON=0 2 * * *
DEFAULT_FULL_CRON=0 3 * * 0
```
### **Custom Schedules**
Create specialized schedules for specific needs:
**High-Priority Entities:**
```json
{
"id": "hourly-tickets",
"name": "Hourly Ticket Sync",
"cron_expression": "0 * * * *",
"sync_type": "incremental"
}
```
**Monthly Reports:**
```json
{
"id": "monthly-full",
"name": "Monthly Full Sync",
"cron_expression": "0 4 1 * *",
"sync_type": "full",
"years_back": 5
}
```
**Business Hours Only:**
```json
{
"id": "business-hours",
"name": "Business Hours Sync",
"cron_expression": "0 9-17 * * 1-5",
"sync_type": "incremental"
}
```
---
## Performance Considerations
### **Concurrent Syncs**
- Scheduler prevents concurrent runs of the SAME schedule
- Different schedules CAN run concurrently
- Sync service prevents multiple syncs system-wide
### **Resource Usage**
**During Sync:**
- CPU: Moderate (data processing)
- Memory: Moderate (batch processing)
- Network: High (API calls)
- Database: Moderate (bulk inserts)
**Recommendations:**
- Schedule during off-peak hours
- Monitor server resources
- Adjust frequency based on load
### **API Rate Limits**
Autotask has rate limits:
- Be mindful of sync frequency
- Incremental syncs use fewer API calls
- Full syncs can hit rate limits on large datasets
---
## Migration from Manual Syncs
### **Step 1: Document Current Process**
- How often do you sync manually?
- Which entities do you sync?
- What time of day?
### **Step 2: Create Equivalent Schedules**
- Daily manual sync → Daily incremental schedule
- Weekly manual sync → Weekly full schedule
### **Step 3: Test Schedules**
- Create schedules (disabled)
- Trigger manually to test
- Verify results in sync history
### **Step 4: Enable Gradually**
- Enable one schedule at a time
- Monitor for 1 week
- Adjust as needed
### **Step 5: Stop Manual Syncs**
- Once confident in automated syncs
- Keep manual option for emergencies
---
## Security
### **Access Control**
Currently, schedule management requires:
- Access to admin interface
- No authentication implemented yet
**Future considerations:**
- Role-based access control
- Audit logging for schedule changes
- API key authentication
### **Schedule Validation**
- Cron expressions validated before saving
- Invalid expressions rejected
- Prevents malicious or broken schedules
---
## Backup and Recovery
### **Backup Schedules**
Export schedule configuration:
```sql
-- Export schedules
COPY (
SELECT id, name, description, cron_expression, sync_type, years_back, is_enabled
FROM sync_schedules
) TO '/tmp/schedules_backup.csv' CSV HEADER;
```
### **Restore Schedules**
Import schedule configuration:
```sql
-- Import schedules
COPY sync_schedules (id, name, description, cron_expression, sync_type, years_back, is_enabled)
FROM '/tmp/schedules_backup.csv' CSV HEADER;
```
Or use API to recreate schedules.
---
## FAQ
**Q: Can I have multiple schedules running at once?**
A: Different schedules can run concurrently, but the sync service prevents multiple syncs system-wide. If one schedule is running, others will wait.
**Q: What happens if the app restarts during a scheduled sync?**
A: The sync will be interrupted. The schedule will run again at the next scheduled time.
**Q: Can I change a schedule while it's running?**
A: Yes, but changes won't affect the current run. They'll apply to the next scheduled run.
**Q: Do schedules run if the app is stopped?**
A: No. Schedules only run when the application is running. Consider using systemd or Docker restart policies.
**Q: Can I schedule specific entities?**
A: Not yet. Schedules sync all entities. This may be added in a future version.
**Q: What timezone are schedules in?**
A: Schedules use the server's timezone. Check with `date` command on the server.
**Q: Can I get notifications when syncs fail?**
A: Not yet. Check the UI or logs. Notifications may be added in a future version.
---
## Summary
**Scheduled syncs provide:**
- ✅ Automatic data synchronization
- ✅ Flexible scheduling with cron expressions
- ✅ Easy management via admin UI
- ✅ Status monitoring and error tracking
- ✅ Manual trigger for testing
- ✅ Multiple schedules for different needs
**Recommended setup:**
1. Enable daily incremental sync (2 AM)
2. Enable weekly full sync (Sunday 3 AM)
3. Monitor for first week
4. Adjust as needed based on your usage
**Best combined with:**
- Webhooks for real-time updates
- Manual syncs for immediate needs
- Regular monitoring of sync history