wulf-pulse/migrations/004_webhook_support.sql

84 lines
3.5 KiB
MySQL
Raw Normal View History

feat: add webhook support for real-time Autotask updates Implements comprehensive webhook infrastructure to receive and process real-time entity updates from Autotask, reducing API calls and improving data freshness. Features: - Webhook receiver endpoint: POST /api/webhooks/autotask - Automatic entity mapping and upsert to PostgreSQL - Event logging and tracking in webhook_logs table - Duplicate event prevention via unique event_id - Failed event tracking with error messages - Statistics and monitoring APIs - Support for 8 entity types: Companies, Tickets, Tasks, Projects, Time Entries, Contacts, Contracts, Configuration Items Architecture: - WebhookService: Core processing logic - Database tables: webhook_logs, webhook_configs - API endpoints: /autotask (receiver), /logs, /stats - Automatic data mapping using existing entity-mapper Benefits: - Near real-time updates (<1 minute vs 24 hours) - Reduced API usage (webhooks vs polling) - Complements daily incremental sync for redundancy - Automatic recovery from webhook failures Files Added: - lib/types/webhook.ts - TypeScript types and interfaces - lib/services/webhook-service.ts - Webhook processing service - app/api/webhooks/autotask/route.ts - Webhook receiver - app/api/webhooks/logs/route.ts - Logs API - app/api/webhooks/stats/route.ts - Statistics API - migrations/004_webhook_support.sql - Database schema - docs/WEBHOOK_SETUP.md - Complete setup guide (47 sections) - docs/WEBHOOKS_README.md - Quick start guide Next Steps: 1. Run database migration 2. Configure webhooks in Autotask 3. Test endpoint and monitor logs See docs/WEBHOOK_SETUP.md for detailed setup instructions.
2026-01-24 10:07:20 -05:00
-- Migration: Add webhook support tables
-- Description: Tables for tracking webhook events and configurations
-- Webhook logs table - tracks all incoming webhook events
CREATE TABLE IF NOT EXISTS webhook_logs (
id SERIAL PRIMARY KEY,
event_id VARCHAR(255) NOT NULL UNIQUE,
entity_type VARCHAR(100) NOT NULL,
entity_id INTEGER NOT NULL,
event_type VARCHAR(50) NOT NULL, -- create, update, delete
status VARCHAR(50) NOT NULL DEFAULT 'pending', -- pending, processed, failed
error_message TEXT,
feat: add IP address logging to webhooks for whitelisting Implements comprehensive IP logging for webhook requests to enable IP whitelisting and security monitoring. Features: - Capture source IP from webhook requests (x-forwarded-for, x-real-ip) - Capture user agent for identification - Store in webhook_logs table - New API endpoint: GET /api/webhooks/ips - View unique IPs with request counts and statistics - Identify Autotask IPs for whitelisting Database Changes: - Added source_ip column (VARCHAR 45) to webhook_logs - Added user_agent column (TEXT) to webhook_logs - Added index on source_ip for efficient queries - Migration 005 for existing installations API Endpoints: - GET /api/webhooks/ips?hours=168&entityType=Tickets Returns unique IPs with: * Request counts (total, successful, failed) * First/last seen timestamps * Entity types accessed * User agent strings Use Cases: 1. Identify Autotask webhook IPs 2. Configure IP whitelist in nginx/Pangolin/Cloudflare 3. Monitor for unauthorized webhook attempts 4. Audit webhook sources 5. Detect IP changes from Autotask Security Benefits: - Enable IP whitelisting for webhook endpoint - Block unauthorized webhook attempts - Monitor for suspicious activity - Audit trail of webhook sources Documentation: - Complete IP whitelisting guide (WEBHOOK_IP_WHITELISTING.md) - Configuration examples for nginx, Pangolin, Cloudflare - Monitoring queries and best practices - Troubleshooting guide Files Modified: - migrations/004_webhook_support.sql - Added IP columns - migrations/005_add_webhook_ip_logging.sql - Migration for existing installs - lib/types/webhook.ts - Added IP fields to WebhookLog - lib/services/webhook-service.ts - Capture and log IPs - app/api/webhooks/autotask/route.ts - Extract IP from headers - app/api/webhooks/ips/route.ts - New IP viewing endpoint - docs/WEBHOOK_IP_WHITELISTING.md - Complete guide Next Steps: 1. Run migration (004 for new, 005 for existing) 2. Deploy updated code 3. Receive webhooks from Autotask 4. View IPs via /api/webhooks/ips 5. Configure IP whitelist in proxy/tunnel
2026-01-24 17:28:56 -05:00
source_ip VARCHAR(45), -- IPv4 or IPv6 address
user_agent TEXT, -- User agent string from request
feat: add webhook support for real-time Autotask updates Implements comprehensive webhook infrastructure to receive and process real-time entity updates from Autotask, reducing API calls and improving data freshness. Features: - Webhook receiver endpoint: POST /api/webhooks/autotask - Automatic entity mapping and upsert to PostgreSQL - Event logging and tracking in webhook_logs table - Duplicate event prevention via unique event_id - Failed event tracking with error messages - Statistics and monitoring APIs - Support for 8 entity types: Companies, Tickets, Tasks, Projects, Time Entries, Contacts, Contracts, Configuration Items Architecture: - WebhookService: Core processing logic - Database tables: webhook_logs, webhook_configs - API endpoints: /autotask (receiver), /logs, /stats - Automatic data mapping using existing entity-mapper Benefits: - Near real-time updates (<1 minute vs 24 hours) - Reduced API usage (webhooks vs polling) - Complements daily incremental sync for redundancy - Automatic recovery from webhook failures Files Added: - lib/types/webhook.ts - TypeScript types and interfaces - lib/services/webhook-service.ts - Webhook processing service - app/api/webhooks/autotask/route.ts - Webhook receiver - app/api/webhooks/logs/route.ts - Logs API - app/api/webhooks/stats/route.ts - Statistics API - migrations/004_webhook_support.sql - Database schema - docs/WEBHOOK_SETUP.md - Complete setup guide (47 sections) - docs/WEBHOOKS_README.md - Quick start guide Next Steps: 1. Run database migration 2. Configure webhooks in Autotask 3. Test endpoint and monitor logs See docs/WEBHOOK_SETUP.md for detailed setup instructions.
2026-01-24 10:07:20 -05:00
received_at TIMESTAMP NOT NULL DEFAULT NOW(),
processed_at TIMESTAMP,
processing_time_ms INTEGER,
payload JSONB NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Webhook configurations table - tracks which webhooks are configured
CREATE TABLE IF NOT EXISTS webhook_configs (
id SERIAL PRIMARY KEY,
entity_type VARCHAR(100) NOT NULL,
event_types JSONB NOT NULL, -- array of event types: ["create", "update", "delete"]
is_active BOOLEAN NOT NULL DEFAULT true,
autotask_webhook_id VARCHAR(255), -- ID from Autotask webhook registration
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
UNIQUE(entity_type)
);
-- Indexes for webhook_logs
CREATE INDEX IF NOT EXISTS idx_webhook_logs_event_id ON webhook_logs(event_id);
CREATE INDEX IF NOT EXISTS idx_webhook_logs_entity ON webhook_logs(entity_type, entity_id);
CREATE INDEX IF NOT EXISTS idx_webhook_logs_status ON webhook_logs(status);
CREATE INDEX IF NOT EXISTS idx_webhook_logs_received_at ON webhook_logs(received_at DESC);
CREATE INDEX IF NOT EXISTS idx_webhook_logs_entity_type ON webhook_logs(entity_type);
feat: add IP address logging to webhooks for whitelisting Implements comprehensive IP logging for webhook requests to enable IP whitelisting and security monitoring. Features: - Capture source IP from webhook requests (x-forwarded-for, x-real-ip) - Capture user agent for identification - Store in webhook_logs table - New API endpoint: GET /api/webhooks/ips - View unique IPs with request counts and statistics - Identify Autotask IPs for whitelisting Database Changes: - Added source_ip column (VARCHAR 45) to webhook_logs - Added user_agent column (TEXT) to webhook_logs - Added index on source_ip for efficient queries - Migration 005 for existing installations API Endpoints: - GET /api/webhooks/ips?hours=168&entityType=Tickets Returns unique IPs with: * Request counts (total, successful, failed) * First/last seen timestamps * Entity types accessed * User agent strings Use Cases: 1. Identify Autotask webhook IPs 2. Configure IP whitelist in nginx/Pangolin/Cloudflare 3. Monitor for unauthorized webhook attempts 4. Audit webhook sources 5. Detect IP changes from Autotask Security Benefits: - Enable IP whitelisting for webhook endpoint - Block unauthorized webhook attempts - Monitor for suspicious activity - Audit trail of webhook sources Documentation: - Complete IP whitelisting guide (WEBHOOK_IP_WHITELISTING.md) - Configuration examples for nginx, Pangolin, Cloudflare - Monitoring queries and best practices - Troubleshooting guide Files Modified: - migrations/004_webhook_support.sql - Added IP columns - migrations/005_add_webhook_ip_logging.sql - Migration for existing installs - lib/types/webhook.ts - Added IP fields to WebhookLog - lib/services/webhook-service.ts - Capture and log IPs - app/api/webhooks/autotask/route.ts - Extract IP from headers - app/api/webhooks/ips/route.ts - New IP viewing endpoint - docs/WEBHOOK_IP_WHITELISTING.md - Complete guide Next Steps: 1. Run migration (004 for new, 005 for existing) 2. Deploy updated code 3. Receive webhooks from Autotask 4. View IPs via /api/webhooks/ips 5. Configure IP whitelist in proxy/tunnel
2026-01-24 17:28:56 -05:00
CREATE INDEX IF NOT EXISTS idx_webhook_logs_source_ip ON webhook_logs(source_ip);
feat: add webhook support for real-time Autotask updates Implements comprehensive webhook infrastructure to receive and process real-time entity updates from Autotask, reducing API calls and improving data freshness. Features: - Webhook receiver endpoint: POST /api/webhooks/autotask - Automatic entity mapping and upsert to PostgreSQL - Event logging and tracking in webhook_logs table - Duplicate event prevention via unique event_id - Failed event tracking with error messages - Statistics and monitoring APIs - Support for 8 entity types: Companies, Tickets, Tasks, Projects, Time Entries, Contacts, Contracts, Configuration Items Architecture: - WebhookService: Core processing logic - Database tables: webhook_logs, webhook_configs - API endpoints: /autotask (receiver), /logs, /stats - Automatic data mapping using existing entity-mapper Benefits: - Near real-time updates (<1 minute vs 24 hours) - Reduced API usage (webhooks vs polling) - Complements daily incremental sync for redundancy - Automatic recovery from webhook failures Files Added: - lib/types/webhook.ts - TypeScript types and interfaces - lib/services/webhook-service.ts - Webhook processing service - app/api/webhooks/autotask/route.ts - Webhook receiver - app/api/webhooks/logs/route.ts - Logs API - app/api/webhooks/stats/route.ts - Statistics API - migrations/004_webhook_support.sql - Database schema - docs/WEBHOOK_SETUP.md - Complete setup guide (47 sections) - docs/WEBHOOKS_README.md - Quick start guide Next Steps: 1. Run database migration 2. Configure webhooks in Autotask 3. Test endpoint and monitor logs See docs/WEBHOOK_SETUP.md for detailed setup instructions.
2026-01-24 10:07:20 -05:00
-- Indexes for webhook_configs
CREATE INDEX IF NOT EXISTS idx_webhook_configs_entity_type ON webhook_configs(entity_type);
CREATE INDEX IF NOT EXISTS idx_webhook_configs_is_active ON webhook_configs(is_active);
-- Function to update updated_at timestamp
CREATE OR REPLACE FUNCTION update_webhook_updated_at()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = NOW();
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
-- Triggers for updated_at
CREATE TRIGGER webhook_logs_updated_at
BEFORE UPDATE ON webhook_logs
FOR EACH ROW
EXECUTE FUNCTION update_webhook_updated_at();
CREATE TRIGGER webhook_configs_updated_at
BEFORE UPDATE ON webhook_configs
FOR EACH ROW
EXECUTE FUNCTION update_webhook_updated_at();
-- Insert default webhook configurations for key entities
INSERT INTO webhook_configs (entity_type, event_types, is_active) VALUES
('Companies', '["create", "update"]', true),
('Tickets', '["create", "update"]', true),
('Tasks', '["create", "update"]', true),
('Projects', '["create", "update"]', true),
('TimeEntries', '["create", "update"]', true),
('Contacts', '["create", "update"]', true)
ON CONFLICT (entity_type) DO NOTHING;
-- Comments
COMMENT ON TABLE webhook_logs IS 'Tracks all incoming webhook events from Autotask';
COMMENT ON TABLE webhook_configs IS 'Configuration for which webhooks are enabled';
COMMENT ON COLUMN webhook_logs.event_id IS 'Unique identifier from Autotask webhook event';
COMMENT ON COLUMN webhook_logs.payload IS 'Full webhook payload as JSON';
COMMENT ON COLUMN webhook_logs.processing_time_ms IS 'Time taken to process the webhook in milliseconds';
COMMENT ON COLUMN webhook_configs.autotask_webhook_id IS 'Webhook ID returned by Autotask API when webhook was registered';