wulf-pulse/docs/SCHEDULED_SYNCS.md

# Scheduled Syncs Guide

## Overview

Pulse supports automatic scheduled syncs using node-cron. This allows you to configure daily, weekly, or custom sync schedules without manual intervention.

---

## Features

- ✅ **Configurable Schedules** - Create multiple sync schedules with different frequencies
- ✅ **Cron Expressions** - Full cron syntax support for flexible scheduling
- ✅ **Admin UI** - Manage schedules through the web interface
- ✅ **Enable/Disable** - Toggle schedules on/off without deleting them
- ✅ **Manual Trigger** - Run any schedule immediately for testing
- ✅ **Status Monitoring** - View last run, next run, and success/failure status
- ✅ **Error Tracking** - See error messages for failed syncs
- ✅ **Incremental or Full** - Choose sync type per schedule

---

## Default Schedules

Two default schedules are created on first startup (disabled by default):

### **1. Daily Incremental Sync**
- **Schedule:** Every day at 2 AM (`0 2 * * *`)
- **Type:** Incremental (last 24 hours)
- **Purpose:** Keep data up-to-date with daily changes
- **Status:** Disabled (enable via UI)

### **2. Weekly Full Sync**
- **Schedule:** Every Sunday at 3 AM (`0 3 * * 0`)
- **Type:** Full (2 years back)
- **Purpose:** Ensure data integrity with complete sync
- **Status:** Disabled (enable via UI)

---

## Managing Schedules

### **Via Admin UI**

1. **Navigate to Sync Page**
   - Go to Admin → Sync
   - Click on "Schedules" tab

2. **View Schedules**
   - See all configured schedules
   - Check status (enabled/disabled, running, last run, next run)
   - View error messages for failed syncs

3. **Enable/Disable Schedule**
   - Click the Play/Pause button
   - Schedule starts/stops immediately

4. **Manually Trigger Schedule**
   - Click the Clock button
   - Sync runs immediately (doesn't affect schedule)

5. **Edit Schedule**
   - Click "Edit" button
   - Modify name, description, cron expression, sync type
   - Changes take effect immediately

6. **Create New Schedule**
   - Click "New Schedule" button
   - Fill in details:
     - **ID:** Unique identifier (e.g., `hourly-incremental`)
     - **Name:** Display name (e.g., `Hourly Incremental Sync`)
     - **Description:** What this schedule does
     - **Sync Type:** Incremental or Full
     - **Cron Expression:** When to run (use presets or custom)
     - **Enable:** Start immediately or leave disabled

7. **Delete Schedule**
   - Click Trash button
   - Confirm deletion
   - Schedule is permanently removed

### **Via API**

**Get All Schedules:**
```bash
curl http://localhost:3100/api/sync/schedules
```

**Get Specific Schedule:**
```bash
curl http://localhost:3100/api/sync/schedules/daily-incremental
```

**Create Schedule:**
```bash
curl -X POST http://localhost:3100/api/sync/schedules \
  -H "Content-Type: application/json" \
  -d '{
    "id": "hourly-tickets",
    "name": "Hourly Ticket Sync",
    "description": "Sync tickets every hour",
    "cron_expression": "0 * * * *",
    "sync_type": "incremental",
    "is_enabled": true
  }'
```

**Update Schedule:**
```bash
curl -X PATCH http://localhost:3100/api/sync/schedules/daily-incremental \
  -H "Content-Type: application/json" \
  -d '{
    "is_enabled": true,
    "cron_expression": "0 3 * * *"
  }'
```

**Delete Schedule:**
```bash
curl -X DELETE http://localhost:3100/api/sync/schedules/hourly-tickets
```

**Trigger Schedule Manually:**
```bash
curl -X POST http://localhost:3100/api/sync/schedules/daily-incremental/trigger
```

---

## Cron Expression Guide

Cron format: `minute hour day month weekday`

### **Common Patterns**

| Expression | Description |
|------------|-------------|
| `0 2 * * *` | Every day at 2 AM |
| `0 3 * * 0` | Every Sunday at 3 AM |
| `0 */6 * * *` | Every 6 hours |
| `0 0 1 * *` | First day of every month at midnight |
| `30 4 * * 1-5` | 4:30 AM on weekdays |
| `0 */2 * * *` | Every 2 hours |
| `15 14 1 * *` | 2:15 PM on the first of every month |

### **Field Values**

- **Minute:** 0-59
- **Hour:** 0-23 (0 = midnight, 12 = noon)
- **Day:** 1-31
- **Month:** 1-12
- **Weekday:** 0-7 (0 or 7 = Sunday)

### **Special Characters**

- `*` - Any value
- `,` - List (e.g., `1,15` = 1st and 15th)
- `-` - Range (e.g., `1-5` = Monday through Friday)
- `/` - Step (e.g., `*/2` = every 2 units)

### **Examples**

```
0 2 * * *       # 2:00 AM every day
0 3 * * 0       # 3:00 AM every Sunday
0 */4 * * *     # Every 4 hours
30 9 * * 1-5    # 9:30 AM on weekdays
0 0 1,15 * *    # Midnight on 1st and 15th
```

---

## Recommended Schedules

### **Small Organization (< 100 tickets/day)**

```
Daily Incremental:  0 2 * * *   (2 AM daily)
Weekly Full:        0 3 * * 0   (3 AM Sunday)
```

### **Medium Organization (100-500 tickets/day)**

```
Incremental:        0 */6 * * *  (Every 6 hours)
Weekly Full:        0 3 * * 0    (3 AM Sunday)
```

### **Large Organization (> 500 tickets/day)**

```
Incremental:        0 */2 * * *  (Every 2 hours)
Daily Full:         0 3 * * *    (3 AM daily)
```

### **With Webhooks**

If using webhooks for real-time updates:

```
Daily Incremental:  0 2 * * *   (Backup for missed webhooks)
Weekly Full:        0 3 * * 0   (Data integrity check)
```

---

## Best Practices

### **1. Stagger Schedules**

Don't run multiple syncs at the same time:
```
Daily Incremental:  0 2 * * *   (2 AM)
Weekly Full:        0 3 * * 0   (3 AM Sunday)
```

### **2. Off-Peak Hours**

Schedule syncs during low-usage periods:
- ✅ 2-4 AM (recommended)
- ✅ Late evening (10 PM - midnight)
- ❌ Business hours (9 AM - 5 PM)

### **3. Start Disabled**

Create new schedules disabled, test manually first:
```json
{
  "is_enabled": false
}
```

Then enable after verifying it works.

### **4. Monitor First Runs**

After enabling a schedule:
1. Wait for first scheduled run
2. Check sync history for success
3. Review any error messages
4. Adjust schedule if needed

### **5. Incremental + Full Strategy**

Combine both for best results:
- **Incremental:** Daily or more frequent (fast, recent changes)
- **Full:** Weekly or monthly (slow, ensures data integrity)

### **6. Test with Manual Trigger**

Before enabling a schedule:
1. Create schedule (disabled)
2. Click "Trigger" button to run manually
3. Verify sync completes successfully
4. Enable schedule

---

## Monitoring

### **Schedule Status**

Each schedule shows:
- **Enabled/Disabled** - Current state
- **Running** - Currently executing (animated badge)
- **Next Run** - When it will run next (e.g., "in 2h 15m")
- **Last Run** - When it last executed
- **Last Status** - Success ✓ or Failed ✗
- **Error Message** - Details if failed

### **Check Logs**

View application logs for scheduler activity:

```bash
# Docker logs
docker logs pulse-app | grep SCHEDULER

# Recent scheduler events
docker logs pulse-app --tail 100 | grep SCHEDULER
```

### **Database Queries**

```sql
-- View all schedules
SELECT * FROM sync_schedules ORDER BY id;

-- View enabled schedules
SELECT id, name, cron_expression, next_run 
FROM sync_schedules 
WHERE is_enabled = true;

-- View failed schedules
SELECT id, name, last_run, last_error 
FROM sync_schedules 
WHERE last_status = 'failed';

-- View schedule history
SELECT 
  s.name,
  s.last_run,
  s.last_status,
  h.records_added,
  h.records_updated
FROM sync_schedules s
LEFT JOIN sync_history h ON h.started_at = s.last_run
WHERE s.last_run IS NOT NULL
ORDER BY s.last_run DESC;
```

---

## Troubleshooting

### **Schedule Not Running**

**Check 1: Is it enabled?**
- Look for "Enabled" badge
- If disabled, click Play button

**Check 2: Is cron expression valid?**
- Look for "Invalid Cron" badge
- Edit schedule and fix expression

**Check 3: Check next run time**
- Ensure next run is in the future
- If "Calculating...", wait a moment and refresh

**Check 4: Application running?**
- Scheduler only works when app is running
- Check `docker ps` or process status

**Check 5: Check logs**
```bash
docker logs pulse-app | grep "SCHEDULER.*daily-incremental"
```

### **Schedule Failing**

**Check Error Message:**
- View in UI under schedule card
- Shows last error from failed sync

**Common Errors:**

1. **"A sync operation is already in progress"**
   - Another sync is running
   - Wait for it to complete
   - Consider adjusting schedule times

2. **"Failed to connect to Autotask API"**
   - Check API credentials
   - Verify network connectivity
   - Check Autotask API status

3. **"Database connection error"**
   - Check PostgreSQL is running
   - Verify database credentials
   - Check disk space

**View Sync History:**
- Go to "Sync History" tab
- Filter by entity type
- Check error details

### **Schedule Running Too Long**

If a sync takes longer than expected:

1. **Check current sync status:**
   ```bash
   curl http://localhost:3100/api/sync/status
   ```

2. **Review sync history:**
   - Look at duration of previous syncs
   - Identify slow entities

3. **Consider splitting:**
   - Create separate schedules for slow entities
   - Run them at different times

### **Missed Schedules**

If application was down during scheduled time:
- Schedule will NOT run retroactively
- Next run will be at next scheduled time
- Consider manual trigger if data is critical

---

## Database Schema

### **sync_schedules Table**

```sql
CREATE TABLE sync_schedules (
  id VARCHAR(50) PRIMARY KEY,
  name VARCHAR(100) NOT NULL,
  description TEXT,
  cron_expression VARCHAR(50) NOT NULL,
  sync_type VARCHAR(20) NOT NULL,  -- 'incremental' or 'full'
  years_back INTEGER DEFAULT 2,
  is_enabled BOOLEAN NOT NULL DEFAULT true,
  last_run TIMESTAMP,
  next_run TIMESTAMP,
  last_status VARCHAR(20),         -- 'success' or 'failed'
  last_error TEXT,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
```

---

## Advanced Configuration

### **Environment Variables**

Currently, schedules are managed via database and UI. Future versions may support:

```env
# Example (not yet implemented)
ENABLE_SCHEDULED_SYNC=true
DEFAULT_INCREMENTAL_CRON=0 2 * * *
DEFAULT_FULL_CRON=0 3 * * 0
```

### **Custom Schedules**

Create specialized schedules for specific needs:

**High-Priority Entities:**
```json
{
  "id": "hourly-tickets",
  "name": "Hourly Ticket Sync",
  "cron_expression": "0 * * * *",
  "sync_type": "incremental"
}
```

**Monthly Reports:**
```json
{
  "id": "monthly-full",
  "name": "Monthly Full Sync",
  "cron_expression": "0 4 1 * *",
  "sync_type": "full",
  "years_back": 5
}
```

**Business Hours Only:**
```json
{
  "id": "business-hours",
  "name": "Business Hours Sync",
  "cron_expression": "0 9-17 * * 1-5",
  "sync_type": "incremental"
}
```

---

## Performance Considerations

### **Concurrent Syncs**

- Scheduler prevents concurrent runs of the SAME schedule
- Different schedules CAN run concurrently
- Sync service prevents multiple syncs system-wide

### **Resource Usage**

**During Sync:**
- CPU: Moderate (data processing)
- Memory: Moderate (batch processing)
- Network: High (API calls)
- Database: Moderate (bulk inserts)

**Recommendations:**
- Schedule during off-peak hours
- Monitor server resources
- Adjust frequency based on load

### **API Rate Limits**

Autotask has rate limits:
- Be mindful of sync frequency
- Incremental syncs use fewer API calls
- Full syncs can hit rate limits on large datasets

---

## Migration from Manual Syncs

### **Step 1: Document Current Process**
- How often do you sync manually?
- Which entities do you sync?
- What time of day?

### **Step 2: Create Equivalent Schedules**
- Daily manual sync → Daily incremental schedule
- Weekly manual sync → Weekly full schedule

### **Step 3: Test Schedules**
- Create schedules (disabled)
- Trigger manually to test
- Verify results in sync history

### **Step 4: Enable Gradually**
- Enable one schedule at a time
- Monitor for 1 week
- Adjust as needed

### **Step 5: Stop Manual Syncs**
- Once confident in automated syncs
- Keep manual option for emergencies

---

## Security

### **Access Control**

Currently, schedule management requires:
- Access to admin interface
- No authentication implemented yet

**Future considerations:**
- Role-based access control
- Audit logging for schedule changes
- API key authentication

### **Schedule Validation**

- Cron expressions validated before saving
- Invalid expressions rejected
- Prevents malicious or broken schedules

---

## Backup and Recovery

### **Backup Schedules**

Export schedule configuration:

```sql
-- Export schedules
COPY (
  SELECT id, name, description, cron_expression, sync_type, years_back, is_enabled
  FROM sync_schedules
) TO '/tmp/schedules_backup.csv' CSV HEADER;
```

### **Restore Schedules**

Import schedule configuration:

```sql
-- Import schedules
COPY sync_schedules (id, name, description, cron_expression, sync_type, years_back, is_enabled)
FROM '/tmp/schedules_backup.csv' CSV HEADER;
```

Or use API to recreate schedules.

---

## FAQ

**Q: Can I have multiple schedules running at once?**
A: Different schedules can run concurrently, but the sync service prevents multiple syncs system-wide. If one schedule is running, others will wait.

**Q: What happens if the app restarts during a scheduled sync?**
A: The sync will be interrupted. The schedule will run again at the next scheduled time.

**Q: Can I change a schedule while it's running?**
A: Yes, but changes won't affect the current run. They'll apply to the next scheduled run.

**Q: Do schedules run if the app is stopped?**
A: No. Schedules only run when the application is running. Consider using systemd or Docker restart policies.

**Q: Can I schedule specific entities?**
A: Not yet. Schedules sync all entities. This may be added in a future version.

**Q: What timezone are schedules in?**
A: Schedules use the server's timezone. Check with `date` command on the server.

**Q: Can I get notifications when syncs fail?**
A: Not yet. Check the UI or logs. Notifications may be added in a future version.

---

## Summary

**Scheduled syncs provide:**
- ✅ Automatic data synchronization
- ✅ Flexible scheduling with cron expressions
- ✅ Easy management via admin UI
- ✅ Status monitoring and error tracking
- ✅ Manual trigger for testing
- ✅ Multiple schedules for different needs

**Recommended setup:**
1. Enable daily incremental sync (2 AM)
2. Enable weekly full sync (Sunday 3 AM)
3. Monitor for first week
4. Adjust as needed based on your usage

**Best combined with:**
- Webhooks for real-time updates
- Manual syncs for immediate needs
- Regular monitoring of sync history