Commit graph

25 commits

Author SHA1 Message Date
1ecaefe85a fix(quick-260521-foj-01): widen Companies full sync to fetch all companies
- Add buildCompaniesFilter() returning id > 0 in lib/utils/sync-helpers.ts
- Route COMPANIES through buildCompaniesFilter on full sync instead of
  the generic buildActiveFilter (which applied isActive=true and missed
  inactive companies with tickets, causing tickets_company_id_fkey on
  weekly-full and full syncs since 2026-05-15)
- hasAppliedFilters stays true (filter is non-empty), so soft-delete of
  companies is not triggered
2026-05-21 11:21:48 -04:00
8f8b5ab7be feat: AI ticket analyzer (phases 1-6)
Multi-stage LLM pipeline that produces structured analyses of Autotask
tickets from local Postgres. Migration 069 + Zod schemas, Stage 0
preprocessor, IT Glue redaction + search, Anthropic SDK wrapper, Stages
1/3/4 (Haiku/Sonnet/Opus), pipeline + cost circuit breaker, job worker
(opt-in autostart), 6 API routes, 3 frontend pages, share-row
persistence (email send deferred to phase 7). 128 vitest tests, tsc
clean. Build journal in docs/wulf-pulse-ticket-analyzer-build-notes.md.

Sync: adds syncTicketNotes() + ticket_notes to ordered/date-filtered
entities so the analyzer's local mirror stays current via scheduler.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 10:59:40 -04:00
ea3471d38d feat: Veeam RPO analysis, comparison, ticket analysis + company teams table
- Add Veeam RPO analysis page (/veeam-analysis) and comparison page (/veeam-comparison)
- Add API routes: /api/veeam/rpo-analyze, rpo-comparison, rpo-offline-log, ticket-analysis
- Add veeam-rpo-service.ts enhancements (RPO logic, offline detection, comparison)
- Add veeam-analysis-state.ts and rmm-device-resolver.ts services
- Add migrations 065-068: company_teams, veeam_rpo_offline_log, rpo_comparison_tables, veeam_ticket_analysis
- Add backup-status page updates and nav links for new Veeam pages
- Add scripts: deactivate-cis-for-inactive-companies, workstation category updates
- Add docs: mimecast-api-guide, veeam-backup-alerting-recommendation, workstation-backup-overview, ticket-analyzer-prompt
- Minor: webhook-service, entity-sync, entity-mapper, sync-helpers, sync.ts, middleware.ts updates
2026-04-29 09:16:46 -04:00
07067bef19 feat: Display Settings UI + Company Category/Type sync
- Add /admin/display-settings page with Kiosk and Mobile sections
- Company category checkbox filter + excluded companies searchable multi-select
- New DB tables: company_categories, company_types (migration 064)
- Sync COMPANY_CATEGORIES via CompanyCategories entity (id/name/isActive)
- Sync COMPANY_TYPES via Companies.companyType picklist
- Add to EntityType, ENTITY_DEPENDENCIES, sync-helpers, entity-mapper, entity-sync
- New API routes: /api/admin/display-settings (GET/POST), /api/data/company-categories, /api/data/companies-list
- Update all 4 routes (kiosk/stats, kiosk/activity, mobile/tickets, mobile/dashboard)
  to filter by kiosk_settings company_category_ids + excluded_company_ids
- Add Display Settings nav link (SlidersHorizontal icon) to Admin menu
- Seed kiosk_settings: kiosk_company_category_ids=1, mobile_company_category_ids=1
2026-04-06 09:03:19 -04:00
0e8eb4871e fix: prevent sync from nullifying assigned_resource_id on tickets
- bulkUpsert now accepts preserveExistingOnNull column list, using
  COALESCE(EXCLUDED.col, table.col) so null incoming values never
  overwrite existing non-null DB values
- bulkUpsertRecords passes resource ID columns as preserve-on-null
  for TICKETS and TASKS entities
- getValidResourceIds now throws on DB error instead of returning
  empty set (which would nullify every resource reference)
- Fix mimecast mailbox-remediate fetch handlers to check res.ok and
  content-type before calling res.json(), preventing JSON parse crash
  on 502 Bad Gateway responses
2026-04-05 08:55:41 -04:00
e3aba93857 feat: Duo Security integration — full data sync from Accounts + Admin API
Duo API Client (lib/services/duo-client.ts):
- HMAC-SHA1 request signing, GET/POST, automatic pagination
- Rate-limit handling (429 + Retry-After), configurable timeout
- Accounts API: listAccounts() via POST /accounts/v1/account/list
- Admin API: getUsers, getPhones, getGroups, getIntegrations, getAuthLogs
- Child account access: parent creds signed against child api_hostname + account_id
- Factory helpers: getDuoAccountsClient(), getDuoAdminClient()

Database (migration 058):
- 6 tables: duo_accounts, duo_users, duo_phones, duo_auth_logs, duo_groups, duo_integrations
- All with proper FKs, indexes, JSONB fields for capabilities/location/groups

Sync Service (lib/services/duo-sync-service.ts):
- syncAll() orchestration, per-child sequential sync, incremental auth logs
- Company matching: exact then case-insensitive containment (30/32 = 94% matched)
- Non-blocking with sync ID tracking

API Routes:
- POST/GET /api/duo/sync — trigger sync / check status
- GET /api/duo/accounts — list all accounts with stats + matched company
- GET /api/duo/accounts/[id]/users — users for a specific account
- POST /api/openclaw/sync/duo — OpenClaw trigger with API key auth

Verified data: 33 accounts, 832 users, 925 phones, 5927 auth logs, 46 groups, 78 integrations

Also: entity-mapper company fields update, task list marked complete
2026-03-27 11:10:30 -04:00
9a448d111c feat: contacts sync — add all missing fields + UDFs
Migration 057:
- ADD user_defined_fields JSONB (GIN indexed)
- ADD middle_initial, note, external_id, country_id, company_location_id
- ADD create_date, impersonator_creator_resource_id
- ADD is_opted_out_from_bulk_email, bulk_email_opt_out_time
- ADD solicitation_opt_out_time, survey_opt_out
- ADD receives_email_notifications, billing_contact

entity-mapper.ts mapContact():
- Fix broken snake_case field refs → correct camelCase API names:
  alternate_phone → alternatePhone, mobile_phone → mobilePhone
  name_prefix/suffix → namePrefix/nameSuffix
  facebook/twitter/linkedin_url → facebookUrl/twitterUrl/linkedInUrl
  primary_contact → primaryContact, solicitation_opt_out → solicitationOptOut
  last_activity/modified_date → lastActivityDate/lastModifiedDate
  api_vendor_id → apiVendorID, is_active → isActive, is_deleted → isDeleted
- Add UDF conversion: userDefinedFields[] → JSONB {name: value}
- 17 UDFs stored: UserID, Birthday, O365License, VIP User, Department,
  Password, Email Password, Archive Email, User System Profile, etc.

Results: 4234 contacts synced, 3318 with UDFs, 4140 with create_date
2026-03-26 13:16:13 -04:00
dd4cf68def feat: add project_phases entity sync with task project_id backfill
The Autotask Tasks bulk API does not return projectID in its response,
causing all tasks.project_id to be NULL. This fixes it by:

- Adding project_phases as a synced entity (Autotask endpoint: /Phases)
- Migration 059: project_phases table with project_id, phase_number,
  estimated_hours, start/due dates, parent_phase_id, is_scheduled
- EntityType.PROJECT_PHASES added to all sync maps and dependency graph
  (depends on PROJECTS, runs before TASKS in sync order)
- buildProjectPhasesFilter: Phases endpoint requires a filter (id > 0)
- mapProjectPhase: maps Autotask field names to DB columns
- Post-sync backfill in syncEntity: after each project_phases sync,
  UPDATE tasks SET project_id = pp.project_id FROM project_phases pp
  JOIN projects p WHERE tasks.phase_id = pp.id
  Only backfills where the project exists in our DB (FK constraint on
  tasks.project_id; archived projects are skipped gracefully)

Result: 2,455 of 4,966 tasks now have project_id populated. Tasks
belonging to archived/completed projects have phase_id resolvable via
project_phases even when project_id remains NULL.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-24 13:57:33 -04:00
ff9e34cafe feat: add Autotask tags sync (tag groups, tags, ticket tag associations)
- Migration 057: autotask_tag_groups, autotask_tags, and junction tables
  (ticket_tags, company_tags, configuration_item_tags, contact_tags)
- Add TAG_GROUPS and TAGS to EntityType enum and dependency map
- Add mapTagGroup() and mapTag() entity mapper functions
- Add syncTagGroups(), syncTags(), syncTicketTagAssociations() methods
- Wire TicketTagAssociations bulk sync into full/incremental sync flow
- Add 'exist' operator to QueryFilter type
- No FK on ticket_id (tagged tickets may be outside 2yr sync window)

Synced: 26 tag groups, 7299 tags, 10141 ticket-tag associations
2026-03-20 09:22:40 -04:00
c8be128c5c feat: companies sync — add UDFs and fix all field mappings
Migration 056:
- ADD user_defined_fields JSONB (GIN indexed)
- ADD parent_company_id, web_address, create_date, company_category_id
- ADD is_client_portal_active, is_task_fire_active, is_comanaged
- ADD country_id, bill_to_country_id, bill_to_address_to_use
- ADD invoice_email_message_id, purchase_order_template_id
- ADD additional_address_information, sic_code, stock_market, etc.

entity-mapper.ts mapCompany():
- Fix broken snake_case field refs (was data.invoice_method, data.last_activity_date etc.)
- All corrected to actual Autotask camelCase API names (invoiceMethod, lastActivityDate, etc.)
- was data.webSiteURL → data.webAddress
- was data.taxExempt → data.isTaxExempt
- was data.lastTrackedModifiedDateTime (various) → data.lastTrackedModifiedDateTime
- Add UDF conversion: userDefinedFields[] array → JSONB {name: value} object
- 38 UDFs now stored: PassportalID, MimecastID, MSP Service Model, Short Name,
  Workstation Lifecycle Plan, Seats (End Users), CSProfileUID, UniFiID, etc.
2026-03-17 23:56:40 -04:00
5be7b22953 feat: OpenClaw external agent sync API at /api/openclaw/*
- API key auth via x-openclaw-key header (OPENCLAW_API_KEY env var)
- GET  /api/openclaw/sync/status
- POST /api/openclaw/sync/autotask/incremental
- POST /api/openclaw/sync/autotask/full
- POST /api/openclaw/sync/autotask/entity   { entities: [...] }
- POST /api/openclaw/sync/datto-rmm          { syncType: full|incremental }
- POST /api/openclaw/sync/sentinelone
- POST /api/openclaw/sync/veeam              { syncType: full|incremental }
- POST /api/openclaw/sync/zoom
- POST /api/openclaw/sync/engagement
- POST /api/openclaw/sync/qbo                { syncType: full|incremental }
- POST /api/openclaw/sync/zabbix
- POST /api/openclaw/sync/itglue
All routes bypass Better Auth middleware, delegate to existing sync services
2026-03-17 23:26:44 -04:00
7ce06803dc fix: time entries incremental sync uses lastModifiedDateTime not dateWorked
Previously the incremental filter compared dateWorked >= lastSyncTime,
meaning edits to existing entries (hours adjusted, notes changed, PTO
blocks entered in advance) were never picked up after initial ingestion.

Switch both the lastModifiedField mapping and buildTimeEntriesFilter to
use lastModifiedDateTime so any created or modified entry is captured
on the next scheduled sync.
2026-03-17 23:01:18 -04:00
cb293ea843 fix: configuration_items sync — correct camelCase field mapping, store all UDFs in udfs JSONB
- Fix mapConfigurationItem() to use Autotask camelCase field names (was using
  snake_case keys that never matched, leaving costs/Datto/RMM fields empty)
- Map all 60+ fields explicitly: costs, Datto device fields, RMM audit fields,
  location, vendor, SNMP, backup dates, source cost, etc.
- Store entire userDefinedFields array as udfs JSONB (keyed by UDF name)
- Retain backup_type_udf VARCHAR column for backward compat
- Migration 052: add udfs JSONB column + GIN index to configuration_items
2026-03-17 15:39:42 -04:00
c518eefdb2 feat: Morning NOC Summary adaptive card for Teams
- Add MorningSummaryService with Zabbix aggregation and adaptive card builder
- Add webhook delivery system with Teams incoming webhooks
- Add admin UI at /admin/morning-summary for webhook/config management
- Add API routes: /send, /test, /webhooks, /webhooks/[id], /config, /history
- Register morning-summary cron job in SyncScheduler (Mon-Fri 6:30 AM)
- Add outages_only filter (Unavailable triggers only)
- Fix host resolution: use getTriggerEnabledHosts to exclude disabled hosts
- Fix resolved events: event.get value:1 scoped to window with r_eventid filter
- Remove emojis from fact rows and section headers in card
- Remove Open Zabbix button (duplicate of View Problems)
- Add migrations: morning_summary_config + morning_summaries tables
- Add outages_only column to morning_summary_config
2026-03-11 09:34:51 -04:00
d7c3dc7168 feat: Autotask webhook integration, TicketNotes, Datto RMM, workflow engine, Veeam agents/alarms, AI triage, misc improvements 2026-02-20 10:28:15 -05:00
5dc7a7e66b feat: Veeam VSPC backup integration - sync, compliance, UI
- Database: 7 Veeam tables + backup_type_udf column on configuration_items
- API Client: VSPC REST API v3 client with pagination, rate limiting, Bearer auth
- Sync Service: full/incremental sync for orgs, servers, repos, jobs, agent jobs, workloads
- Scheduler: veeam-incremental (30min) and veeam-full (daily 2AM) schedules
- Compliance Engine: cross-references Autotask config items vs Veeam workloads
- API Endpoints: backup-status, companies, workloads, jobs, repos, compliance, sync
- UI: Backup Status page with Overview + Contract Compliance tabs
- Navigation: added Backup Status link with HardDrive icon
- Docker: added VEEAM_VSPC_URL and VEEAM_VSPC_API_KEY env vars to compose
2026-02-11 21:04:28 -05:00
root
8405c813a3 feat: add missing Tickets entity fields from Autotask API
Added 10 critical missing fields to Tickets entity:

Database Migration (016):
- billing_code_id: Billing code assignment
- configuration_item_id: Primary asset/CI
- creator_resource_id: Ticket creator
- creator_type: Creator type (resource/contact)
- problem_ticket_id: Link to problem ticket
- rma_status: RMA status tracking
- rma_type: RMA type classification
- service_level_agreement_paused_next_event_hours: SLA pause tracking
- is_assigned_to_comanaged: Co-managed assignment flag
- is_visible_to_comanaged: Co-managed visibility flag

Entity Mapper Updates:
- Added all new fields with correct camelCase mapping
- Ensures all Autotask Tickets API fields are captured

This completes the Tickets entity to match the full Autotask API
specification and should resolve issues with missing ticket data.
2026-01-26 15:26:34 -05:00
root
cc16d9b32d fix: correct Tickets entity field name mappings from Autotask API
Fixed field names to use proper camelCase from Autotask API instead of
snake_case. This should resolve the issue where assigned_resource_id
was NULL because the mapper was looking for incorrect field names.

Changed mappings:
- firstResponseDateTime (was first_response_date_time)
- resolutionPlanDateTime (was resolution_plan_date_time)
- resolvedDateTime (was resolved_date_time)
- firstResponseAssignedResourceID (was first_response_assigned_resource_id)
- firstResponseInitiatingResourceID (was first_response_initiating_resource_id)
- projectID, opportunityID, contractID, monitorID (were snake_case)
- All change approval and ticket fields converted to camelCase
- All service thermometer fields converted to camelCase

This fix should populate assigned_resource_id and other fields correctly
on the next ticket sync.
2026-01-26 15:25:03 -05:00
root
bc01a6aafb fix: handle non-numeric values in Resources integer fields
Autotask API returns string values like 'PRIMARY' for some integer
fields (e.g., emailTypeCode). Added safeInt() helper function to:
- Parse numeric values correctly
- Return null for non-numeric strings
- Handle null/undefined/empty values

This prevents database errors when syncing Resources with non-standard
field values from Autotask API.
2026-01-26 14:48:37 -05:00
root
8e0ca0a63b feat: expand Resources entity to include all Autotask API fields
Add comprehensive field support for Resources entity including:
- Name fields: middleInitial, namePrefix, nameSuffix
- Contact: emailAddress2, emailAddress3, homePhone
- Employment: accountingReferenceID, payrollType, internalCost
- System: emailTypeCode, numberFormat, timeFormat, dateFormat
- Demographics: gender
- Security: licenseType, securityLevel
- Location: defaultServiceDeskRoleID

Changes:
- Migration 015: Add new columns to resources table
- Updated entity mapper to map all Autotask Resource fields
- Expanded TypeScript Resource interface with all fields
- Maintains backward compatibility with existing field names

This ensures all data exposed by the Autotask Resources API is
now captured and stored in the database.
2026-01-26 14:28:51 -05:00
root
43e8c74074 fix: correct Autotask API field names for incremental sync
Fixed Autotask API errors for Billing Items and Time Entries during
incremental syncs by using the correct field names:

- Billing Items: Changed from 'createDate' to 'itemDate'
- Time Entries: Changed from 'lastModifiedDate' to 'dateWorked'

These field names match what's used in full sync filters and are
accepted by the Autotask API.

Errors fixed:
- 'Unable to find createDate in the BillingItem Entity'
- 'Unable to find lastModifiedDate in the TimeEntry Entity'

This allows incremental syncs to work properly for these entities.
2026-01-26 10:37:10 -05:00
root
f1037d7964 fix: add required filters for billing items and validate task project references
- Add buildBillingItemsFilter() to provide required itemDate filter
  BillingItems API requires a filter parameter and returns 500 error without it
- Add validation for task project_id foreign key references
  Tasks with invalid project_id now have the field nullified instead of failing
- Add getValidProjectIds() method to fetch valid project IDs from database

Fixes:
- Billing Items: 'Value cannot be null. Parameter name: filters' error
- Tasks: 'violates foreign key constraint tasks_project_id_fkey' error

Both entities should now sync successfully.
2026-01-24 08:09:10 -05:00
root
e25eb3fa5e feat: implement structured logging for sync operations
- Add comprehensive structured logging utility (lib/utils/sync-logger.ts)
  - Log levels: DEBUG, INFO, WARN, ERROR
  - Automatic error categorization (NETWORK_ERROR, AUTH_ERROR, DATABASE_ERROR, etc.)
  - Phase tracking (INITIALIZING, FETCHING, MAPPING, VALIDATING, UPSERTING, DELETING, COMPLETING)
  - Contextual metadata (syncId, entityType, phase, duration, record counts)
  - Timing helpers for operations

- Update sync-service.ts with structured logging
  - Replace console.log/error with structured logger
  - Consistent error categorization across all error paths
  - Fix swallowed errors in sync history updates
  - Add child loggers for entity-specific operations

- Update entity-sync.ts with structured logging
  - Complete phase tracking throughout sync lifecycle
  - Detailed context for warnings and errors
  - Update chunked sync methods with structured logging
  - Update picklist sync methods (issue types, sub-issue types)
  - Better debugging info with timing and sample data

- Add sync failure analysis script (scripts/analyze-sync-failures.ts)
  - Query sync history by date range, entity type, or status
  - Generate failure summaries by entity
  - Automatic error categorization
  - Calculate success rates and statistics
  - Display detailed sync records with timing

- Add comprehensive documentation (docs/SYNC_LOGGING_IMPROVEMENTS.md)
  - Usage guide and examples
  - Migration guide for developers
  - Before/after comparisons

This addresses inconsistent logging and improves debugging capabilities for sync operations.
2026-01-23 08:02:02 -05:00
root
6eee14f8af Add comprehensive admin features and multi-system integration
- Add admin dashboard with sync controls and data browser
- Implement RMM, Auvik, and Addigy organization mappings
- Add chunked ticket sync with progress tracking
- Implement entity sync service with rate limiting
- Add analytics engine and performance optimizer
- Create data browser for all PSA entities
- Add navigation components and UI improvements
- Implement background processing and sync services
- Add comprehensive documentation and migration scripts
- Update configuration items with multi-system support
- Enhance contact management and purchase history
- Add issue type assignment and LLM analyzer
- Improve error handling and logging utilities
2025-11-19 14:18:16 -05:00
3c3124d8c9 Restructure: rename to Pulse and move app to root
- Renamed project from PSA-Utils to Pulse
- Moved all app files from autotask-app/ to root
- Updated package.json name to 'pulse'
- Updated Docker container names to pulse-app and pulse-redis
- Updated Docker network name to pulse-network
2025-10-28 23:08:54 -04:00