wulf-pulse/docs/WEBHOOK_SETUP.md

# Autotask Webhook Setup Guide

## Overview

This guide explains how to configure Autotask webhooks for real-time data synchronization with Pulse. Webhooks provide near-instant updates when entities change in Autotask, reducing API calls and keeping your data fresh.

---

## Table of Contents

- [Benefits of Webhooks](#benefits-of-webhooks)
- [Supported Entities](#supported-entities)
- [Prerequisites](#prerequisites)
- [Setup Steps](#setup-steps)
- [Webhook Endpoint](#webhook-endpoint)
- [Testing Webhooks](#testing-webhooks)
- [Monitoring](#monitoring)
- [Troubleshooting](#troubleshooting)
- [Recommended Sync Strategy](#recommended-sync-strategy)

---

## Benefits of Webhooks

### **Real-Time Updates**
- Instant notification when entities are created or updated in Autotask
- No polling delay - changes appear within seconds

### **Reduced API Usage**
- Fewer API calls compared to frequent polling
- Lower risk of hitting API rate limits
- More efficient use of resources

### **Better Data Freshness**
- Combined with daily incremental syncs for redundancy
- Ensures no events are missed
- Automatic recovery from webhook failures

---

## Supported Entities

The following entities support webhook notifications:

| Entity | Events | Notes |
|--------|--------|-------|
| **Companies** | Create, Update | Real-time company changes |
| **Tickets** | Create, Update | New tickets and status updates |
| **Tasks** | Create, Update | Task creation and modifications |
| **Projects** | Create, Update | Project changes |
| **Time Entries** | Create, Update | New time entries |
| **Contacts** | Create, Update | Contact information changes |
| **Contracts** | Create, Update | Contract updates |
| **Configuration Items** | Create, Update | Asset changes |

**Note:** Delete events are supported but less common in Autotask workflows.

---

## Prerequisites

Before setting up webhooks, ensure you have:

1. **Autotask API Access**
   - API user with appropriate permissions
   - API integration enabled in Autotask

2. **Public Webhook Endpoint**
   - Your Pulse instance must be accessible from the internet
   - HTTPS endpoint (required by Autotask)
   - Example: `https://your-pulse-instance.com/api/webhooks/autotask`

3. **Database Migration**
   - Run migration `004_webhook_support.sql` to create webhook tables
   ```bash
   docker exec pulse-postgres psql -U pulse_user -d pulse_autotask -f /migrations/004_webhook_support.sql
   ```

---

## Setup Steps

### **Step 1: Configure Webhook Endpoint in Autotask**

1. Log in to Autotask as an administrator
2. Navigate to **Admin** → **Features & Settings** → **API & Integrations**
3. Click on **Webhooks**
4. Click **New Webhook**

### **Step 2: Create Webhook for Each Entity**

For each entity you want to track (e.g., Tickets):

1. **Webhook Name:** `Pulse - Tickets`
2. **Endpoint URL:** `https://your-pulse-instance.com/api/webhooks/autotask`
3. **Entity Type:** Select the entity (e.g., `Tickets`)
4. **Events:** Select events to track:
   - ✅ Create
   - ✅ Update
   - ⬜ Delete (optional)
5. **Include Entity Data:** ✅ **Enabled** (recommended)
   - This includes full entity data in the webhook payload
   - Reduces need for additional API calls
6. **Active:** ✅ **Enabled**
7. Click **Save**

### **Step 3: Record Webhook IDs**

After creating each webhook, Autotask will provide a Webhook ID. Record these for reference:

```
Companies: webhook_12345
Tickets: webhook_12346
Tasks: webhook_12347
Projects: webhook_12348
TimeEntries: webhook_12349
Contacts: webhook_12350
```

### **Step 4: Verify Webhook Configuration**

Check that webhooks are configured in the database:

```sql
SELECT entity_type, is_active, autotask_webhook_id
FROM webhook_configs
ORDER BY entity_type;
```

Update webhook IDs if needed:

```sql
UPDATE webhook_configs
SET autotask_webhook_id = 'webhook_12345'
WHERE entity_type = 'Companies';
```

---

## Webhook Endpoint

### **Endpoint Details**

- **URL:** `https://your-domain.com/api/webhooks/autotask`
- **Method:** `POST`
- **Content-Type:** `application/json`
- **Authentication:** None (Autotask doesn't support webhook authentication)
  - Secure via IP whitelisting or network-level security

### **Payload Structure**

Autotask sends webhooks with this structure:

```json
{
  "eventId": "evt_abc123",
  "eventType": "create",
  "entityType": "Tickets",
  "entityId": 12345,
  "eventTimestamp": "2026-01-24T10:00:00Z",
  "entity": {
    "id": 12345,
    "title": "New Ticket",
    "status": 1,
    "companyID": 67890,
    ...
  }
}
```

### **Response**

The endpoint returns:

```json
{
  "success": true,
  "eventId": "evt_abc123",
  "action": "created",
  "processingTime": 45
}
```

---

## Testing Webhooks

### **1. Health Check**

Verify the webhook endpoint is accessible:

```bash
curl https://your-pulse-instance.com/api/webhooks/autotask
```

Expected response:
```json
{
  "status": "active",
  "endpoint": "/api/webhooks/autotask",
  "message": "Autotask webhook receiver is ready"
}
```

### **2. Test Webhook from Autotask**

1. In Autotask, go to the webhook configuration
2. Click **Test Webhook**
3. Autotask will send a test event
4. Check webhook logs in Pulse

### **3. Create Test Entity**

Create a test ticket or company in Autotask and verify:

1. Webhook is received (check logs)
2. Entity appears in database
3. Processing time is reasonable (<1 second)

### **4. View Webhook Logs**

```bash
# Via API
curl https://your-pulse-instance.com/api/webhooks/logs?limit=10

# Via Database
docker exec pulse-postgres psql -U pulse_user -d pulse_autotask -c \
  "SELECT event_id, entity_type, entity_id, status, processing_time_ms 
   FROM webhook_logs 
   ORDER BY received_at DESC 
   LIMIT 10;"
```

---

## Monitoring

### **Webhook Statistics**

View webhook statistics via API:

```bash
curl https://your-pulse-instance.com/api/webhooks/stats?hours=24
```

Response:
```json
{
  "success": true,
  "stats": {
    "total": 150,
    "processed": 148,
    "failed": 2,
    "pending": 0,
    "byEntityType": {
      "Tickets": 75,
      "Companies": 25,
      "Tasks": 30,
      "TimeEntries": 20
    }
  },
  "period": "24 hours"
}
```

### **Failed Webhooks**

Check for failed webhook processing:

```sql
SELECT event_id, entity_type, entity_id, error_message, received_at
FROM webhook_logs
WHERE status = 'failed'
ORDER BY received_at DESC
LIMIT 20;
```

### **Processing Performance**

Monitor webhook processing times:

```sql
SELECT 
  entity_type,
  COUNT(*) as total,
  AVG(processing_time_ms) as avg_ms,
  MAX(processing_time_ms) as max_ms
FROM webhook_logs
WHERE status = 'processed'
  AND received_at >= NOW() - INTERVAL '24 hours'
GROUP BY entity_type
ORDER BY avg_ms DESC;
```

---

## Troubleshooting

### **Webhooks Not Received**

**Check 1: Endpoint Accessibility**
```bash
# From external network
curl https://your-pulse-instance.com/api/webhooks/autotask
```

**Check 2: Firewall/Network**
- Ensure port 443 (HTTPS) is open
- Check firewall rules allow Autotask IPs
- Verify SSL certificate is valid

**Check 3: Autotask Configuration**
- Verify webhook is Active in Autotask
- Check endpoint URL is correct
- Ensure entity type matches

### **Webhooks Failing to Process**

**Check Logs:**
```sql
SELECT event_id, entity_type, error_message, payload
FROM webhook_logs
WHERE status = 'failed'
ORDER BY received_at DESC
LIMIT 5;
```

**Common Issues:**

1. **Missing Entity Data**
   - Enable "Include Entity Data" in Autotask webhook config
   - Without this, webhook only includes entity ID

2. **Foreign Key Violations**
   - Run full sync for dependent entities first
   - Example: Sync Companies before Tickets

3. **Invalid Data**
   - Check entity mapper for missing field mappings
   - Review error_message in webhook_logs

### **High Failure Rate**

If >5% of webhooks fail:

1. **Run Full Sync** for affected entities
2. **Check Dependencies** - ensure parent entities are synced
3. **Review Error Patterns** - look for common error messages
4. **Contact Support** - if issues persist

---

## Recommended Sync Strategy

### **Optimal Configuration**

Combine webhooks with scheduled syncs for best results:

#### **1. Webhooks (Real-Time)**
- **Enabled for:** Companies, Tickets, Tasks, Projects, Time Entries, Contacts
- **Events:** Create, Update
- **Purpose:** Instant updates for active entities

#### **2. Daily Incremental Sync (Scheduled)**
- **Time:** 2:00 AM daily
- **Entities:** All entities
- **Purpose:** 
  - Catch any missed webhooks
  - Sync entities without webhooks (Contracts, Config Items, Picklists)
  - Ensure data consistency

#### **3. Weekly Full Sync (Scheduled)**
- **Time:** Sunday 3:00 AM
- **Entities:** All entities
- **Date Range:** Last 2 years
- **Purpose:**
  - Complete data refresh
  - Verify data integrity
  - Sync historical changes

### **Cron Schedule Example**

```bash
# Daily incremental sync at 2 AM
0 2 * * * curl -X POST https://your-pulse-instance.com/api/sync/incremental

# Weekly full sync on Sunday at 3 AM
0 3 * * 0 curl -X POST https://your-pulse-instance.com/api/sync/full
```

### **Expected Data Freshness**

| Entity | Freshness | Method |
|--------|-----------|--------|
| Tickets | <1 minute | Webhook |
| Tasks | <1 minute | Webhook |
| Companies | <1 minute | Webhook |
| Projects | <1 minute | Webhook |
| Time Entries | <1 minute | Webhook |
| Contacts | <1 minute | Webhook |
| Contracts | 24 hours | Daily sync |
| Config Items | 24 hours | Daily sync |
| Picklists | 7 days | Weekly sync |

---

## Security Considerations

### **1. Network Security**

**Option A: IP Whitelisting**
- Restrict webhook endpoint to Autotask IP ranges
- Configure in firewall or reverse proxy

**Option B: VPN/Private Network**
- Use VPN tunnel for webhook traffic
- More secure but complex setup

### **2. Payload Validation**

The webhook service validates:
- Required fields (eventId, eventType, entityType, entityId)
- Event ID uniqueness (prevents duplicate processing)
- Entity type is supported

### **3. Rate Limiting**

Consider adding rate limiting to prevent abuse:
```nginx
# Nginx example
limit_req_zone $binary_remote_addr zone=webhook:10m rate=100r/m;

location /api/webhooks/autotask {
    limit_req zone=webhook burst=20;
    proxy_pass http://pulse-app:3100;
}
```

---

## Database Schema

### **webhook_logs Table**

Tracks all incoming webhook events:

```sql
CREATE TABLE webhook_logs (
  id SERIAL PRIMARY KEY,
  event_id VARCHAR(255) NOT NULL UNIQUE,
  entity_type VARCHAR(100) NOT NULL,
  entity_id INTEGER NOT NULL,
  event_type VARCHAR(50) NOT NULL,
  status VARCHAR(50) NOT NULL DEFAULT 'pending',
  error_message TEXT,
  received_at TIMESTAMP NOT NULL DEFAULT NOW(),
  processed_at TIMESTAMP,
  processing_time_ms INTEGER,
  payload JSONB NOT NULL
);
```

### **webhook_configs Table**

Manages webhook configurations:

```sql
CREATE TABLE webhook_configs (
  id SERIAL PRIMARY KEY,
  entity_type VARCHAR(100) NOT NULL UNIQUE,
  event_types JSONB NOT NULL,
  is_active BOOLEAN NOT NULL DEFAULT true,
  autotask_webhook_id VARCHAR(255)
);
```

---

## API Reference

### **Receive Webhook**

```
POST /api/webhooks/autotask
Content-Type: application/json

{
  "eventId": "evt_abc123",
  "eventType": "create",
  "entityType": "Tickets",
  "entityId": 12345,
  "entity": { ... }
}
```

### **Get Webhook Logs**

```
GET /api/webhooks/logs?limit=100&entityType=Tickets
```

### **Get Webhook Statistics**

```
GET /api/webhooks/stats?hours=24
```

---

## Support

For issues or questions:
1. Check webhook logs for error messages
2. Verify Autotask webhook configuration
3. Review this documentation
4. Contact system administrator

---

## Related Documentation

- [Sync Behavior Documentation](./SYNC_BEHAVIOR.md)
- [Sync Interface Guide](./SYNC_INTERFACE_GUIDE.md)
- [Autotask Webhook Documentation](https://autotask.net/help/DeveloperHelp/Content/APIs/Webhooks/WEBHOOKS.htm)