Fixed critical logic bug where Companies and Resources were not getting
proper filters applied when falling back from incremental to full sync.
Root Cause:
- During scheduled incremental sync, Companies and Resources correctly
identify they don't support incremental sync
- However, the filter-building logic was in an else block that only
executed for non-incremental syncs
- This caused Companies and Resources to sync with NO filters at all
- Autotask API now rejects queries without filters, returning:
'Value cannot be null. Parameter name: filters'
Fix:
- Restructured logic so filter building happens for BOTH:
1. Non-incremental (full) syncs
2. Incremental syncs that fall back to full sync
- Companies and Resources now get active filters applied even during
scheduled incremental syncs
This resolves the scheduled sync failures for Companies, Resources,
and all other entities that were failing due to the cascading effect
of early failures.
Added 10 critical missing fields to Tickets entity:
Database Migration (016):
- billing_code_id: Billing code assignment
- configuration_item_id: Primary asset/CI
- creator_resource_id: Ticket creator
- creator_type: Creator type (resource/contact)
- problem_ticket_id: Link to problem ticket
- rma_status: RMA status tracking
- rma_type: RMA type classification
- service_level_agreement_paused_next_event_hours: SLA pause tracking
- is_assigned_to_comanaged: Co-managed assignment flag
- is_visible_to_comanaged: Co-managed visibility flag
Entity Mapper Updates:
- Added all new fields with correct camelCase mapping
- Ensures all Autotask Tickets API fields are captured
This completes the Tickets entity to match the full Autotask API
specification and should resolve issues with missing ticket data.
Fixed field names to use proper camelCase from Autotask API instead of
snake_case. This should resolve the issue where assigned_resource_id
was NULL because the mapper was looking for incorrect field names.
Changed mappings:
- firstResponseDateTime (was first_response_date_time)
- resolutionPlanDateTime (was resolution_plan_date_time)
- resolvedDateTime (was resolved_date_time)
- firstResponseAssignedResourceID (was first_response_assigned_resource_id)
- firstResponseInitiatingResourceID (was first_response_initiating_resource_id)
- projectID, opportunityID, contractID, monitorID (were snake_case)
- All change approval and ticket fields converted to camelCase
- All service thermometer fields converted to camelCase
This fix should populate assigned_resource_id and other fields correctly
on the next ticket sync.
Updated Resources data browser page to display all 40+ fields in the
detail modal including:
- Name fields (prefix, middle initial, suffix)
- All contact information (3 email addresses, home phone)
- Employment details (payroll, accounting, internal cost)
- Location and availability
- System preferences (date/time/number formats)
- Demographics and security (gender, license, security level)
- Survey ratings
Fields are organized into logical groups with comments for clarity.
Autotask API returns string values like 'PRIMARY' for some integer
fields (e.g., emailTypeCode). Added safeInt() helper function to:
- Parse numeric values correctly
- Return null for non-numeric strings
- Handle null/undefined/empty values
This prevents database errors when syncing Resources with non-standard
field values from Autotask API.
Add comprehensive field support for Resources entity including:
- Name fields: middleInitial, namePrefix, nameSuffix
- Contact: emailAddress2, emailAddress3, homePhone
- Employment: accountingReferenceID, payrollType, internalCost
- System: emailTypeCode, numberFormat, timeFormat, dateFormat
- Demographics: gender
- Security: licenseType, securityLevel
- Location: defaultServiceDeskRoleID
Changes:
- Migration 015: Add new columns to resources table
- Updated entity mapper to map all Autotask Resource fields
- Expanded TypeScript Resource interface with all fields
- Maintains backward compatibility with existing field names
This ensures all data exposed by the Autotask Resources API is
now captured and stored in the database.
Document that Companies and Resources entities do not support
incremental sync in the Autotask API and automatically fall back
to full sync when incremental sync is requested.
This clarifies expected behavior and prevents confusion about why
these entities always perform full syncs even during incremental
sync operations.
Companies and Resources entities in Autotask API do not support
date-based filtering for incremental syncs. When attempting incremental
sync, Autotask returns errors:
- Companies: 'Unable to find lastTrackedModificationDateTime'
- Resources: 'Unable to find lastModifiedDate'
Solution:
- Skip incremental sync for these entities
- Always perform full sync for Companies and Resources
- Log informational message when falling back to full sync
- Other entities continue to support incremental sync normally
This prevents sync failures while maintaining data freshness for
Companies and Resources through full syncs.
Fixed Autotask API errors for Billing Items and Time Entries during
incremental syncs by using the correct field names:
- Billing Items: Changed from 'createDate' to 'itemDate'
- Time Entries: Changed from 'lastModifiedDate' to 'dateWorked'
These field names match what's used in full sync filters and are
accepted by the Autotask API.
Errors fixed:
- 'Unable to find createDate in the BillingItem Entity'
- 'Unable to find lastModifiedDate in the TimeEntry Entity'
This allows incremental syncs to work properly for these entities.
Fixed TypeScript errors in sync-scheduler:
- Added constructor to initialize SyncService with AutotaskClient
- Fixed AutotaskConfig property names (password, apiIntegrationCode)
- Changed method calls to match SyncService API (incrementalSync, fullSync)
- Fixed syncService references to use this.syncService
This resolves build errors preventing Docker image creation.
Implements comprehensive IP logging for webhook requests to enable
IP whitelisting and security monitoring.
Features:
- Capture source IP from webhook requests (x-forwarded-for, x-real-ip)
- Capture user agent for identification
- Store in webhook_logs table
- New API endpoint: GET /api/webhooks/ips
- View unique IPs with request counts and statistics
- Identify Autotask IPs for whitelisting
Database Changes:
- Added source_ip column (VARCHAR 45) to webhook_logs
- Added user_agent column (TEXT) to webhook_logs
- Added index on source_ip for efficient queries
- Migration 005 for existing installations
API Endpoints:
- GET /api/webhooks/ips?hours=168&entityType=Tickets
Returns unique IPs with:
* Request counts (total, successful, failed)
* First/last seen timestamps
* Entity types accessed
* User agent strings
Use Cases:
1. Identify Autotask webhook IPs
2. Configure IP whitelist in nginx/Pangolin/Cloudflare
3. Monitor for unauthorized webhook attempts
4. Audit webhook sources
5. Detect IP changes from Autotask
Security Benefits:
- Enable IP whitelisting for webhook endpoint
- Block unauthorized webhook attempts
- Monitor for suspicious activity
- Audit trail of webhook sources
Documentation:
- Complete IP whitelisting guide (WEBHOOK_IP_WHITELISTING.md)
- Configuration examples for nginx, Pangolin, Cloudflare
- Monitoring queries and best practices
- Troubleshooting guide
Files Modified:
- migrations/004_webhook_support.sql - Added IP columns
- migrations/005_add_webhook_ip_logging.sql - Migration for existing installs
- lib/types/webhook.ts - Added IP fields to WebhookLog
- lib/services/webhook-service.ts - Capture and log IPs
- app/api/webhooks/autotask/route.ts - Extract IP from headers
- app/api/webhooks/ips/route.ts - New IP viewing endpoint
- docs/WEBHOOK_IP_WHITELISTING.md - Complete guide
Next Steps:
1. Run migration (004 for new, 005 for existing)
2. Deploy updated code
3. Receive webhooks from Autotask
4. View IPs via /api/webhooks/ips
5. Configure IP whitelist in proxy/tunnel
User confirmed they use Pangolin (similar to Cloudflare Tunnel), so
created a comprehensive Pangolin-specific configuration guide.
Complete guide includes:
- Pangolin agent installation
- Tunnel configuration with path-based access control
- DNS setup
- Systemd service configuration
- Security settings (rate limiting, IP whitelisting)
- Testing procedures
- Troubleshooting guide
- Performance optimization
- High availability setup
Configuration features:
- Only /api/webhooks/autotask exposed
- Rate limiting: 100 req/min, burst 20
- Automatic SSL/TLS via Pangolin
- All other paths return 404
- No firewall changes needed
- No open ports required
Benefits over other approaches:
- No public IP needed
- Zero Trust security model
- Automatic DDoS protection
- Built-in load balancing
- Simple configuration
File: docs/WEBHOOK_PANGOLIN_SETUP.md
More robust fix based on user feedback: instead of hardcoding specific
entity types, now dynamically checks if ANY filters were applied during
the sync and skips soft deletes accordingly.
Previous approach (hardcoded):
- Maintained list of specific entities to skip
- Easy to miss new entities (like billing_items bug)
- Required manual updates when adding new filters
New approach (dynamic):
- Tracks whether params.filter has any values
- Applies to ALL entities with ANY filter type:
* Date filters (tickets, tasks, time_entries, billing_items)
* Status filters (contracts, projects)
* Active filters (companies, resources, contacts, etc.)
- Future-proof: automatically handles new filtered entities
Logic:
- hasAppliedFilters = params.filter && params.filter.length > 0
- if (!isIncremental && !hasAppliedFilters) → perform soft deletes
- if (!isIncremental && hasAppliedFilters) → skip soft deletes
This prevents deleting records that exist outside any filter criteria,
not just date ranges.
CRITICAL BUG FIX: Billing items were being incorrectly soft-deleted during
full sync because BILLING_ITEMS was missing from the hasDateFilter check.
The Issue:
- Billing items have a date filter (itemDate >= now - yearsBack)
- Full sync with 90-day range fetched only recent billing items
- Soft delete logic saw 89,714 older items not in fetched set
- Incorrectly marked them as deleted (outside sync window)
The Fix:
- Added EntityType.BILLING_ITEMS to hasDateFilter check
- Now billing items skip soft deletes (like tickets, tasks, time entries)
- Prevents deletion of records outside the sync date range
Recovery:
- Restored all 89,714 incorrectly deleted billing items
- UPDATE billing_items SET is_deleted = false WHERE is_deleted = true
This is the same pattern used for tickets, tasks, and time entries which
also have date filters and skip soft deletes.
Previously, all picklist syncs (Statuses, Issue Types, Sub-Issue Types, Work Types)
incorrectly reported all records as 'added' even on subsequent syncs.
The issue was that bulkUpsert returns total rowCount without distinguishing
between inserts and updates.
Fix:
- Query existing values before upserting
- Calculate recordsAdded = new values not in existing set
- Calculate recordsUpdated = values already in existing set
- Applied to all 4 picklist sync methods
This resolves the confusing sync history where Sub-Issue Types showed
+805 on every sync instead of ~0 added, ~805 updated.
Created detailed documentation covering:
- Sync types (Full, Incremental, Sync Selected)
- Date range filtering and entity-specific behaviors
- Database operations (UPSERT, soft deletes)
- Foreign key validation mechanisms
- Performance considerations and optimization tips
- Entity-specific filters and requirements
- Troubleshooting guide and best practices
- Workflow examples and API reference
This complements the existing SYNC_INTERFACE_GUIDE.md with
technical implementation details and behavioral specifications.
Tasks also reference resources via assigned_resource_id and other fields.
Added validation for all task resource foreign keys:
- assigned_resource_id
- creator_resource_id
- completed_by_resource_id
- last_activity_resource_id
This completes the fix for tasks foreign key constraint violations.
- Add buildBillingItemsFilter() to provide required itemDate filter
BillingItems API requires a filter parameter and returns 500 error without it
- Add validation for task project_id foreign key references
Tasks with invalid project_id now have the field nullified instead of failing
- Add getValidProjectIds() method to fetch valid project IDs from database
Fixes:
- Billing Items: 'Value cannot be null. Parameter name: filters' error
- Tasks: 'violates foreign key constraint tasks_project_id_fkey' error
Both entities should now sync successfully.
- Separate Sync Status and Sync History into tabs with icons
- Fix pagination in sync history by adding offset parameter
- Update getSyncHistory to support offset for proper pagination
- Update API endpoint to pass offset parameter
- Previous/Next buttons now work correctly to navigate pages
This improves UX by organizing the sync page into logical sections
and enables users to browse through historical sync records.
The previous fix created the picklist sync methods but forgot to add
routing in the syncEntity() method. This caused the sync service to
still call the generic entity sync path instead of the picklist methods.
Added routing for:
- EntityType.STATUSES -> syncStatuses()
- EntityType.WORK_TYPES -> syncWorkTypes()
This completes the fix for the 404 errors on these picklist entities.
Problem:
- syncStatuses() and syncWorkTypes() were trying to query /Statuses/query
and /WorkTypes/query endpoints which don't exist in Autotask API
- This caused 404 errors: 'File or directory not found'
- These are picklist values, not queryable entities
Solution:
- Changed syncStatuses() to use getPicklistValues('Tickets', 'status')
- Changed syncWorkTypes() to use getPicklistValues('TimeEntries', 'workType')
- Now follows same pattern as syncIssueTypes() and syncSubIssueTypes()
- Uses proper picklist API endpoint: /entity/entityInformation/fields
This fixes today's sync failures for statuses and work_types entities.
- Explain all 4 sync options (Full, Incremental, Chunked, Selected)
- Detail when to use each sync type with real-world examples
- Document date range selector and its impact on performance
- List all entities with their dependencies
- Provide best practices for daily, weekly, and monthly syncs
- Include troubleshooting guide for common sync issues
- Add quick reference table for common scenarios
- Explain sync results (added/updated/deleted counts)
- Document API limits and performance considerations
- Reference analysis script for investigating failures
This guide clarifies the sync interface to help users understand what each option does and how to use it effectively.
- Add admin dashboard with sync controls and data browser
- Implement RMM, Auvik, and Addigy organization mappings
- Add chunked ticket sync with progress tracking
- Implement entity sync service with rate limiting
- Add analytics engine and performance optimizer
- Create data browser for all PSA entities
- Add navigation components and UI improvements
- Implement background processing and sync services
- Add comprehensive documentation and migration scripts
- Update configuration items with multi-system support
- Enhance contact management and purchase history
- Add issue type assignment and LLM analyzer
- Improve error handling and logging utilities
- Remove woff2 font files and validator.ts from repository
- Add *.woff2, validator.ts, and routes.ts to .gitignore
- These are Next.js build artifacts that should be generated during build
- Renamed project from PSA-Utils to Pulse
- Moved all app files from autotask-app/ to root
- Updated package.json name to 'pulse'
- Updated Docker container names to pulse-app and pulse-redis
- Updated Docker network name to pulse-network
- Implemented complete Addigy API v2 client with authentication via x-api-key
- Added device and policy endpoints with automatic org ID resolution
- Created field mapping from snake_case to Title Case for UI compatibility
- Handles nested 'facts' response structure from Addigy devices API
- Added comprehensive API documentation in ADDIGY_API_GUIDE.md
- Multi-stage Dockerfile with optimized production build
- Custom ports: App on 3100, Redis on 6380 (avoids conflicts)
- Docker Compose orchestration with health checks
- Standalone Next.js output for smaller container images
- Non-root user execution for security
- Implemented Redis caching layer for API responses
- 5-minute TTL with graceful fallback if Redis unavailable
- Cache key structure: service:entity:filter1:filter2
- Applied to Addigy devices endpoint with cache hit/miss logging
- Fixed TypeScript strict mode errors for production builds
- Added null safety checks with optional chaining throughout API routes
- Wrapped useSearchParams in Suspense boundary for Next.js 15+ compatibility
- Fixed type assertions for dynamic API responses
- Corrected Set<string> type mismatches in device comparison logic
- Created DOCKER_README.md with complete deployment guide
- Updated ADDIGY_API_GUIDE.md with real-world API patterns
- Documented response structures, field mappings, and troubleshooting
- Next.js 16.0.0 with Turbopack
- Redis 7 with AOF persistence
- Podman/Docker compatible
- TypeScript strict mode compliant
