feat: add webhook support for real-time Autotask updates

Implements comprehensive webhook infrastructure to receive and process
real-time entity updates from Autotask, reducing API calls and improving
data freshness.

Features:
- Webhook receiver endpoint: POST /api/webhooks/autotask
- Automatic entity mapping and upsert to PostgreSQL
- Event logging and tracking in webhook_logs table
- Duplicate event prevention via unique event_id
- Failed event tracking with error messages
- Statistics and monitoring APIs
- Support for 8 entity types: Companies, Tickets, Tasks, Projects,
  Time Entries, Contacts, Contracts, Configuration Items

Architecture:
- WebhookService: Core processing logic
- Database tables: webhook_logs, webhook_configs
- API endpoints: /autotask (receiver), /logs, /stats
- Automatic data mapping using existing entity-mapper

Benefits:
- Near real-time updates (<1 minute vs 24 hours)
- Reduced API usage (webhooks vs polling)
- Complements daily incremental sync for redundancy
- Automatic recovery from webhook failures

Files Added:
- lib/types/webhook.ts - TypeScript types and interfaces
- lib/services/webhook-service.ts - Webhook processing service
- app/api/webhooks/autotask/route.ts - Webhook receiver
- app/api/webhooks/logs/route.ts - Logs API
- app/api/webhooks/stats/route.ts - Statistics API
- migrations/004_webhook_support.sql - Database schema
- docs/WEBHOOK_SETUP.md - Complete setup guide (47 sections)
- docs/WEBHOOKS_README.md - Quick start guide

Next Steps:
1. Run database migration
2. Configure webhooks in Autotask
3. Test endpoint and monitor logs

See docs/WEBHOOK_SETUP.md for detailed setup instructions.
This commit is contained in:
root 2026-01-24 10:07:20 -05:00
parent 31c2d94a1b
commit 1f83456199
8 changed files with 1317 additions and 0 deletions

127
docs/WEBHOOKS_README.md Normal file
View file

@ -0,0 +1,127 @@
# Webhook Integration - Quick Start
## What's Been Implemented
✅ **Webhook Infrastructure**
- Webhook receiver endpoint: `/api/webhooks/autotask`
- Database tables for tracking webhook events
- Webhook processing service with automatic entity upsert
- Monitoring APIs for logs and statistics
✅ **Supported Entities**
- Companies, Tickets, Tasks, Projects, Time Entries, Contacts, Contracts, Configuration Items
✅ **Features**
- Real-time entity updates from Autotask
- Automatic data mapping and upsert to PostgreSQL
- Event logging and error tracking
- Duplicate event prevention
- Performance monitoring
## Quick Setup
### 1. Run Database Migration
```bash
docker exec pulse-postgres psql -U pulse_user -d pulse_autotask -f /app/migrations/004_webhook_support.sql
```
### 2. Configure Webhooks in Autotask
For each entity (Companies, Tickets, Tasks, etc.):
1. Go to Autotask → Admin → Webhooks
2. Create new webhook:
- **URL:** `https://your-domain.com/api/webhooks/autotask`
- **Entity:** Select entity type
- **Events:** Create, Update
- **Include Entity Data:** ✅ Enabled
3. Save and note the webhook ID
### 3. Test the Endpoint
```bash
# Health check
curl https://your-domain.com/api/webhooks/autotask
# View recent webhooks
curl https://your-domain.com/api/webhooks/logs?limit=10
# View statistics
curl https://your-domain.com/api/webhooks/stats?hours=24
```
## Recommended Sync Strategy
**Webhooks (Real-Time)** + **Daily Incremental Sync** + **Weekly Full Sync**
This combination ensures:
- ✅ Near-instant updates via webhooks
- ✅ Backup sync catches missed events
- ✅ Weekly refresh ensures data integrity
## Files Created
```
/lib/types/webhook.ts - TypeScript types
/lib/services/webhook-service.ts - Webhook processing logic
/app/api/webhooks/autotask/route.ts - Webhook receiver endpoint
/app/api/webhooks/logs/route.ts - Logs API
/app/api/webhooks/stats/route.ts - Statistics API
/migrations/004_webhook_support.sql - Database schema
/docs/WEBHOOK_SETUP.md - Complete setup guide
```
## Next Steps
1. **Deploy the changes** - Rebuild and restart the application
2. **Run migration** - Create webhook tables in database
3. **Configure Autotask** - Set up webhooks for desired entities
4. **Monitor** - Check logs and statistics to verify webhooks are working
## Full Documentation
See [WEBHOOK_SETUP.md](./WEBHOOK_SETUP.md) for complete setup instructions, troubleshooting, and best practices.
## Architecture
```
Autotask → Webhook Event
/api/webhooks/autotask (Receiver)
WebhookService.processWebhook()
1. Log event to webhook_logs
2. Validate entity type is active
3. Map Autotask data to PostgreSQL schema
4. Upsert to appropriate table
5. Update log status (processed/failed)
Real-time data in PostgreSQL ✅
```
## Monitoring Queries
```sql
-- Recent webhooks
SELECT event_id, entity_type, entity_id, status, processing_time_ms
FROM webhook_logs
ORDER BY received_at DESC
LIMIT 20;
-- Failed webhooks
SELECT event_id, entity_type, error_message
FROM webhook_logs
WHERE status = 'failed'
ORDER BY received_at DESC;
-- Statistics by entity
SELECT entity_type, COUNT(*) as total,
COUNT(*) FILTER (WHERE status = 'processed') as processed,
COUNT(*) FILTER (WHERE status = 'failed') as failed,
AVG(processing_time_ms) as avg_ms
FROM webhook_logs
WHERE received_at >= NOW() - INTERVAL '24 hours'
GROUP BY entity_type;
```

524
docs/WEBHOOK_SETUP.md Normal file
View file

@ -0,0 +1,524 @@
# Autotask Webhook Setup Guide
## Overview
This guide explains how to configure Autotask webhooks for real-time data synchronization with Pulse. Webhooks provide near-instant updates when entities change in Autotask, reducing API calls and keeping your data fresh.
---
## Table of Contents
- [Benefits of Webhooks](#benefits-of-webhooks)
- [Supported Entities](#supported-entities)
- [Prerequisites](#prerequisites)
- [Setup Steps](#setup-steps)
- [Webhook Endpoint](#webhook-endpoint)
- [Testing Webhooks](#testing-webhooks)
- [Monitoring](#monitoring)
- [Troubleshooting](#troubleshooting)
- [Recommended Sync Strategy](#recommended-sync-strategy)
---
## Benefits of Webhooks
### **Real-Time Updates**
- Instant notification when entities are created or updated in Autotask
- No polling delay - changes appear within seconds
### **Reduced API Usage**
- Fewer API calls compared to frequent polling
- Lower risk of hitting API rate limits
- More efficient use of resources
### **Better Data Freshness**
- Combined with daily incremental syncs for redundancy
- Ensures no events are missed
- Automatic recovery from webhook failures
---
## Supported Entities
The following entities support webhook notifications:
| Entity | Events | Notes |
|--------|--------|-------|
| **Companies** | Create, Update | Real-time company changes |
| **Tickets** | Create, Update | New tickets and status updates |
| **Tasks** | Create, Update | Task creation and modifications |
| **Projects** | Create, Update | Project changes |
| **Time Entries** | Create, Update | New time entries |
| **Contacts** | Create, Update | Contact information changes |
| **Contracts** | Create, Update | Contract updates |
| **Configuration Items** | Create, Update | Asset changes |
**Note:** Delete events are supported but less common in Autotask workflows.
---
## Prerequisites
Before setting up webhooks, ensure you have:
1. **Autotask API Access**
- API user with appropriate permissions
- API integration enabled in Autotask
2. **Public Webhook Endpoint**
- Your Pulse instance must be accessible from the internet
- HTTPS endpoint (required by Autotask)
- Example: `https://your-pulse-instance.com/api/webhooks/autotask`
3. **Database Migration**
- Run migration `004_webhook_support.sql` to create webhook tables
```bash
docker exec pulse-postgres psql -U pulse_user -d pulse_autotask -f /migrations/004_webhook_support.sql
```
---
## Setup Steps
### **Step 1: Configure Webhook Endpoint in Autotask**
1. Log in to Autotask as an administrator
2. Navigate to **Admin****Features & Settings** → **API & Integrations**
3. Click on **Webhooks**
4. Click **New Webhook**
### **Step 2: Create Webhook for Each Entity**
For each entity you want to track (e.g., Tickets):
1. **Webhook Name:** `Pulse - Tickets`
2. **Endpoint URL:** `https://your-pulse-instance.com/api/webhooks/autotask`
3. **Entity Type:** Select the entity (e.g., `Tickets`)
4. **Events:** Select events to track:
- ✅ Create
- ✅ Update
- ⬜ Delete (optional)
5. **Include Entity Data:****Enabled** (recommended)
- This includes full entity data in the webhook payload
- Reduces need for additional API calls
6. **Active:** ✅ **Enabled**
7. Click **Save**
### **Step 3: Record Webhook IDs**
After creating each webhook, Autotask will provide a Webhook ID. Record these for reference:
```
Companies: webhook_12345
Tickets: webhook_12346
Tasks: webhook_12347
Projects: webhook_12348
TimeEntries: webhook_12349
Contacts: webhook_12350
```
### **Step 4: Verify Webhook Configuration**
Check that webhooks are configured in the database:
```sql
SELECT entity_type, is_active, autotask_webhook_id
FROM webhook_configs
ORDER BY entity_type;
```
Update webhook IDs if needed:
```sql
UPDATE webhook_configs
SET autotask_webhook_id = 'webhook_12345'
WHERE entity_type = 'Companies';
```
---
## Webhook Endpoint
### **Endpoint Details**
- **URL:** `https://your-domain.com/api/webhooks/autotask`
- **Method:** `POST`
- **Content-Type:** `application/json`
- **Authentication:** None (Autotask doesn't support webhook authentication)
- Secure via IP whitelisting or network-level security
### **Payload Structure**
Autotask sends webhooks with this structure:
```json
{
"eventId": "evt_abc123",
"eventType": "create",
"entityType": "Tickets",
"entityId": 12345,
"eventTimestamp": "2026-01-24T10:00:00Z",
"entity": {
"id": 12345,
"title": "New Ticket",
"status": 1,
"companyID": 67890,
...
}
}
```
### **Response**
The endpoint returns:
```json
{
"success": true,
"eventId": "evt_abc123",
"action": "created",
"processingTime": 45
}
```
---
## Testing Webhooks
### **1. Health Check**
Verify the webhook endpoint is accessible:
```bash
curl https://your-pulse-instance.com/api/webhooks/autotask
```
Expected response:
```json
{
"status": "active",
"endpoint": "/api/webhooks/autotask",
"message": "Autotask webhook receiver is ready"
}
```
### **2. Test Webhook from Autotask**
1. In Autotask, go to the webhook configuration
2. Click **Test Webhook**
3. Autotask will send a test event
4. Check webhook logs in Pulse
### **3. Create Test Entity**
Create a test ticket or company in Autotask and verify:
1. Webhook is received (check logs)
2. Entity appears in database
3. Processing time is reasonable (<1 second)
### **4. View Webhook Logs**
```bash
# Via API
curl https://your-pulse-instance.com/api/webhooks/logs?limit=10
# Via Database
docker exec pulse-postgres psql -U pulse_user -d pulse_autotask -c \
"SELECT event_id, entity_type, entity_id, status, processing_time_ms
FROM webhook_logs
ORDER BY received_at DESC
LIMIT 10;"
```
---
## Monitoring
### **Webhook Statistics**
View webhook statistics via API:
```bash
curl https://your-pulse-instance.com/api/webhooks/stats?hours=24
```
Response:
```json
{
"success": true,
"stats": {
"total": 150,
"processed": 148,
"failed": 2,
"pending": 0,
"byEntityType": {
"Tickets": 75,
"Companies": 25,
"Tasks": 30,
"TimeEntries": 20
}
},
"period": "24 hours"
}
```
### **Failed Webhooks**
Check for failed webhook processing:
```sql
SELECT event_id, entity_type, entity_id, error_message, received_at
FROM webhook_logs
WHERE status = 'failed'
ORDER BY received_at DESC
LIMIT 20;
```
### **Processing Performance**
Monitor webhook processing times:
```sql
SELECT
entity_type,
COUNT(*) as total,
AVG(processing_time_ms) as avg_ms,
MAX(processing_time_ms) as max_ms
FROM webhook_logs
WHERE status = 'processed'
AND received_at >= NOW() - INTERVAL '24 hours'
GROUP BY entity_type
ORDER BY avg_ms DESC;
```
---
## Troubleshooting
### **Webhooks Not Received**
**Check 1: Endpoint Accessibility**
```bash
# From external network
curl https://your-pulse-instance.com/api/webhooks/autotask
```
**Check 2: Firewall/Network**
- Ensure port 443 (HTTPS) is open
- Check firewall rules allow Autotask IPs
- Verify SSL certificate is valid
**Check 3: Autotask Configuration**
- Verify webhook is Active in Autotask
- Check endpoint URL is correct
- Ensure entity type matches
### **Webhooks Failing to Process**
**Check Logs:**
```sql
SELECT event_id, entity_type, error_message, payload
FROM webhook_logs
WHERE status = 'failed'
ORDER BY received_at DESC
LIMIT 5;
```
**Common Issues:**
1. **Missing Entity Data**
- Enable "Include Entity Data" in Autotask webhook config
- Without this, webhook only includes entity ID
2. **Foreign Key Violations**
- Run full sync for dependent entities first
- Example: Sync Companies before Tickets
3. **Invalid Data**
- Check entity mapper for missing field mappings
- Review error_message in webhook_logs
### **High Failure Rate**
If >5% of webhooks fail:
1. **Run Full Sync** for affected entities
2. **Check Dependencies** - ensure parent entities are synced
3. **Review Error Patterns** - look for common error messages
4. **Contact Support** - if issues persist
---
## Recommended Sync Strategy
### **Optimal Configuration**
Combine webhooks with scheduled syncs for best results:
#### **1. Webhooks (Real-Time)**
- **Enabled for:** Companies, Tickets, Tasks, Projects, Time Entries, Contacts
- **Events:** Create, Update
- **Purpose:** Instant updates for active entities
#### **2. Daily Incremental Sync (Scheduled)**
- **Time:** 2:00 AM daily
- **Entities:** All entities
- **Purpose:**
- Catch any missed webhooks
- Sync entities without webhooks (Contracts, Config Items, Picklists)
- Ensure data consistency
#### **3. Weekly Full Sync (Scheduled)**
- **Time:** Sunday 3:00 AM
- **Entities:** All entities
- **Date Range:** Last 2 years
- **Purpose:**
- Complete data refresh
- Verify data integrity
- Sync historical changes
### **Cron Schedule Example**
```bash
# Daily incremental sync at 2 AM
0 2 * * * curl -X POST https://your-pulse-instance.com/api/sync/incremental
# Weekly full sync on Sunday at 3 AM
0 3 * * 0 curl -X POST https://your-pulse-instance.com/api/sync/full
```
### **Expected Data Freshness**
| Entity | Freshness | Method |
|--------|-----------|--------|
| Tickets | <1 minute | Webhook |
| Tasks | <1 minute | Webhook |
| Companies | <1 minute | Webhook |
| Projects | <1 minute | Webhook |
| Time Entries | <1 minute | Webhook |
| Contacts | <1 minute | Webhook |
| Contracts | 24 hours | Daily sync |
| Config Items | 24 hours | Daily sync |
| Picklists | 7 days | Weekly sync |
---
## Security Considerations
### **1. Network Security**
**Option A: IP Whitelisting**
- Restrict webhook endpoint to Autotask IP ranges
- Configure in firewall or reverse proxy
**Option B: VPN/Private Network**
- Use VPN tunnel for webhook traffic
- More secure but complex setup
### **2. Payload Validation**
The webhook service validates:
- Required fields (eventId, eventType, entityType, entityId)
- Event ID uniqueness (prevents duplicate processing)
- Entity type is supported
### **3. Rate Limiting**
Consider adding rate limiting to prevent abuse:
```nginx
# Nginx example
limit_req_zone $binary_remote_addr zone=webhook:10m rate=100r/m;
location /api/webhooks/autotask {
limit_req zone=webhook burst=20;
proxy_pass http://pulse-app:3100;
}
```
---
## Database Schema
### **webhook_logs Table**
Tracks all incoming webhook events:
```sql
CREATE TABLE webhook_logs (
id SERIAL PRIMARY KEY,
event_id VARCHAR(255) NOT NULL UNIQUE,
entity_type VARCHAR(100) NOT NULL,
entity_id INTEGER NOT NULL,
event_type VARCHAR(50) NOT NULL,
status VARCHAR(50) NOT NULL DEFAULT 'pending',
error_message TEXT,
received_at TIMESTAMP NOT NULL DEFAULT NOW(),
processed_at TIMESTAMP,
processing_time_ms INTEGER,
payload JSONB NOT NULL
);
```
### **webhook_configs Table**
Manages webhook configurations:
```sql
CREATE TABLE webhook_configs (
id SERIAL PRIMARY KEY,
entity_type VARCHAR(100) NOT NULL UNIQUE,
event_types JSONB NOT NULL,
is_active BOOLEAN NOT NULL DEFAULT true,
autotask_webhook_id VARCHAR(255)
);
```
---
## API Reference
### **Receive Webhook**
```
POST /api/webhooks/autotask
Content-Type: application/json
{
"eventId": "evt_abc123",
"eventType": "create",
"entityType": "Tickets",
"entityId": 12345,
"entity": { ... }
}
```
### **Get Webhook Logs**
```
GET /api/webhooks/logs?limit=100&entityType=Tickets
```
### **Get Webhook Statistics**
```
GET /api/webhooks/stats?hours=24
```
---
## Support
For issues or questions:
1. Check webhook logs for error messages
2. Verify Autotask webhook configuration
3. Review this documentation
4. Contact system administrator
---
## Related Documentation
- [Sync Behavior Documentation](./SYNC_BEHAVIOR.md)
- [Sync Interface Guide](./SYNC_INTERFACE_GUIDE.md)
- [Autotask Webhook Documentation](https://autotask.net/help/DeveloperHelp/Content/APIs/Webhooks/WEBHOOKS.htm)