feat: QuickBooks Online integration

- Add QBO OAuth2 client with token refresh (lib/services/qbo-client.ts)
- Add QBO sync service for invoices, payments, deposits, purchases, journal entries, reports (lib/services/qbo-sync-service.ts)
- Add QBO types (lib/types/qbo.ts)
- Add API routes: /api/qbo/auth, /api/qbo/sync, /api/qbo/disconnect
- Add /admin/qbo status and sync management page
- Add legal pages: /legal/eula, /legal/privacy (Intuit app assessment)
- Add QBO nav link under Admin
- Fix reports: remove invalid summarize_column_by, add accounting_method from Preferences API, add showrows=all&showcols=all
- Add CashFlow report type alongside P&L and BalanceSheet
- Add NoReportData check to skip empty report months
- Add intuit_tid capture in error messages
- Add redirect: follow for cluster routing
- Migration 051: qbo_tokens, qbo_invoices, qbo_payments, qbo_deposits, qbo_transactions, qbo_reports tables

Also includes earlier work:
- Ping flap suppression pipeline step
- Ticket digest reports with LLM analysis
- Zabbix WAN monitor and gap analysis
- Kiosk is_deleted filter fixes
- Datto RMM ping target enrichment
- Entity sync soft-delete detection
This commit is contained in:
lorentz 2026-03-17 07:39:55 -04:00
parent c518eefdb2
commit b98c67482a
40 changed files with 6223 additions and 15 deletions

525
PULSE_DATABASE_SKILL.md Normal file
View file

@ -0,0 +1,525 @@
# Pulse Database Skill — Query Reference
> **Purpose:** This document describes the PostgreSQL database behind **Pulse**, an MSP operations platform built by Wulf Consulting. Use it to query Autotask PSA data, RMM alerts, security agents, backup status, IT documentation, engagement metrics, and more.
## Connection
- **Host:** `pulse-postgres` (Docker) or `localhost:5432`
- **Database:** `pulse_autotask`
- **User:** `pulse_user`
- **Read-only queries only** — no INSERT/UPDATE/DELETE
---
## Data Domains at a Glance
| Domain | Key Tables | Approx Rows | Description |
|---|---|---|---|
| **Autotask PSA** | tickets, time_entries, companies, contacts, resources, configuration_items, contracts, projects, tasks, ticket_notes | 109K tickets, 154K time entries, 7K CIs | Service desk, billing, contracts, clients |
| **Datto RMM** | datto_rmm_alerts, datto_rmm_devices, datto_rmm_sites | 4.3K alerts, 3.6K devices | Remote monitoring & management |
| **SentinelOne** | s1_agents, s1_threats, s1_sites | 2.8K agents, 4.1K threats | Endpoint security |
| **Veeam** | veeam_organizations, veeam_backup_jobs, veeam_backup_agents, veeam_alarms, veeam_protected_workloads, veeam_repositories, veeam_backup_servers | ~2.5K total | Backup & disaster recovery |
| **IT Glue** | itg_organizations, itg_configurations, itg_passwords, itg_flexible_assets, itg_contacts, itg_documents, itg_expirations, itg_domains, itg_locations | 14.7K configs, 7.1K contacts | IT documentation |
| **Microsoft 365** | graph_users, teams_meetings, teams_meeting_attendees, engagement_snapshots | 14.3K meetings | Teams meetings, activity reports |
| **Zoom** | zoom_users, zoom_meetings, zoom_meeting_participants, zoom_calls | 2.8K calls, meetings | Zoom calls and meetings |
| **Billing** | billing_items | 95K items | Invoice line items tied to tickets/projects/tasks |
---
## 1. Autotask PSA — Core Service Desk
### tickets (~109K rows, 68 columns)
The central table. Each row is a service ticket.
**Key columns:**
- `id` (bigint PK) — Autotask ticket ID
- `title` (varchar) — ticket subject line
- `description` (text) — full body/description
- `status` (int) — FK to `statuses.value`
- `priority` (int) — FK to `priorities.value`
- `queue_id` (int) — FK to `queues.value`
- `source` (int) — how the ticket was created (see Source Codes below)
- `company_id` (bigint) — FK to `companies.id`
- `contact_id` (bigint) — FK to `contacts.id`
- `assigned_resource_id` (bigint) — FK to `resources.id`
- `configuration_item_id` (bigint) — FK to `configuration_items.id`
- `contract_id` (bigint) — FK to `contracts.id`
- `project_id` (bigint) — FK to `projects.id`
- `issue_type` (int), `sub_issue_type` (int) — classification
- `ticket_type` (int) — 1=Incident, 2=Service Request, 5=Alert
- `create_date` (timestamp) — when opened
- `due_date_time` (timestamp) — SLA due
- `completed_date` (timestamp) — when closed
- `resolved_date_time` (timestamp) — when resolved
- `first_response_date_time` (timestamp) — first response SLA timestamp
- `last_activity_date` (timestamp) — most recent update
- `monitor_id` (bigint) — RMM monitor that created this ticket (if source=8)
- `monitor_type_id` (int) — type of monitor
- `is_deleted` (boolean) — soft delete flag
**Source codes (tickets.source):**
| Value | Meaning | Count |
|---|---|---|
| 8 | Monitoring Alert (RMM/Datto) | 80,736 |
| 4 | Email | 14,872 |
| 21 | Portal | 2,500 |
| 2 | Phone/Voice | 1,781 |
| -1 | Insourced | 1,092 |
| -2 | Outsourced | 1,088 |
| 35 | Phish Alert | 726 |
| 6 | API | 488 |
| 17 | Internal Alert | 453 |
**Ticket types:**
| Value | Meaning | Count |
|---|---|---|
| 1 | Incident | 8,409 |
| 2 | Service Request | 3,217 |
| 5 | Alert | 29,885 |
| NULL | Unclassified | 67,737 |
### companies (~4K rows, 41 columns)
Client/customer organizations.
**Key columns:**
- `id` (bigint PK)
- `company_name` (varchar) — display name
- `company_number` (varchar) — short code
- `is_active` (boolean)
- `company_type` (int) — 1=Customer, 2=Lead, 3=Prospect, 4=Dead, 6=Cancellation, 7=Vendor, etc.
- `owner_resource_id` (bigint) — account manager, FK to `resources.id`
- `classification` (varchar) — e.g. "Platinum", "Gold", etc.
- Address fields: `address1`, `city`, `state`, `postal_code`
- `last_activity_date` (timestamp)
### contacts (~4.2K rows, 37 columns)
People at client companies.
**Key columns:**
- `id` (bigint PK)
- `first_name`, `last_name`, `email_address`, `phone` (varchar)
- `company_id` (bigint) — FK to `companies.id`
- `is_active` (boolean)
- `title` (varchar) — job title
### resources (~40 columns)
Internal staff / technicians.
**Key columns:**
- `id` (bigint PK)
- `first_name`, `last_name`, `email` (varchar)
- `email_address` (varchar) — primary email
- `is_active` (boolean)
- `resource_type` (varchar)
- `default_service_desk_role_id` (bigint)
- `hire_date` (date)
- `location_id` (bigint)
### time_entries (~154K rows, 45 columns)
Work logged against tickets, tasks, or projects.
**Key columns:**
- `id` (bigint PK)
- `resource_id` (bigint) — who did the work, FK to `resources.id`
- `ticket_id` (bigint) — FK to `tickets.id` (NULL if task/project entry)
- `task_id` (bigint) — FK to `tasks.id`
- `project_id` (bigint) — FK to `projects.id`
- `company_id` (bigint) — FK to `companies.id`
- `entry_date` (timestamp) — date of work
- `hours_worked` (numeric) — actual hours
- `hours_to_bill` (numeric) — billable hours
- `start_date_time`, `end_date_time` (timestamp) — clock in/out
- `title` (varchar), `notes` (text), `internal_notes` (text)
- `billable` (boolean), `non_billable` (boolean)
- `billing_rate` (numeric), `cost_rate` (numeric), `revenue` (numeric)
- `contract_id` (bigint), `contract_service_id` (bigint)
- `role_id` (bigint)
- `is_deleted` (boolean)
### ticket_notes (~30K rows, 14 columns)
Notes/comments on tickets.
**Key columns:**
- `id` (bigint PK)
- `ticket_id` (bigint) — FK to `tickets.id`
- `title` (varchar), `description` (text) — note content
- `note_type` (int) — internal, external, etc.
- `publish` (int) — visibility
- `creator_resource_id` (bigint) — who wrote it
- `create_date_time` (timestamptz)
### configuration_items (~7K rows, 95 columns)
Devices/assets tracked in Autotask.
**Key columns:**
- `id` (bigint PK)
- `reference_title` (varchar) — device name (e.g. "DT037", "SRV-DC01")
- `reference_number` (varchar) — often a GUID from RMM
- `serial_number` (varchar)
- `company_id` (bigint) — FK to `companies.id`
- `contact_id` (bigint) — FK to `contacts.id`
- `is_active` (boolean)
- `device_type` (varchar)
**Note:** `reference_title` follows a naming convention per client (e.g. DT037 exists at multiple companies as separate CIs). Always filter by both `reference_title` AND `company_id` when searching.
### contracts (~44 columns)
Service agreements with clients.
**Key columns:**
- `id` (bigint PK)
- `company_id` (bigint) — FK to `companies.id`
- `contract_name` (varchar), `contract_number` (varchar)
- `contract_type` (int), `status` (int)
- `start_date`, `end_date` (date)
- `estimated_hours` (numeric), `estimated_revenue` (numeric)
### contract_services (~8.2K rows)
Line items on contracts.
- `contract_id``contracts.id`
- `company_id``companies.id`
- `service_name` (text), `unit_price`, `quantity`
### projects (~291 rows, 36 columns)
**Key columns:**
- `id`, `company_id`, `project_name`, `status`, `type`
- `project_lead_resource_id``resources.id`
- `start_date_time`, `end_date_time`, `actual_hours`, `estimated_time`
### tasks (~4.3K rows, 34 columns)
Tasks on tickets or projects.
- `ticket_id``tickets.id`
- `project_id``projects.id`
- `assigned_resource_id``resources.id`
- `status`, `priority`, `estimated_hours`, `remaining_hours`
### billing_items (~96K rows)
Invoice line items linked to tickets, tasks, or projects.
- `ticket_id``tickets.id`, `task_id``tasks.id`, `project_id``projects.id`
- `company_id``companies.id`
- `quantity`, `rate`, `total_amount`, `unit_cost`, `unit_price`
---
## 2. Lookup / Picklist Tables
These map integer codes to human-readable labels. Join on `value`.
### statuses (ticket statuses)
| Value | Label |
|---|---|
| 1 | New |
| 5 | Complete |
| 7 | Waiting Customer |
| 8 | In Progress |
| 10 | Dispatched |
| 11 | Escalate |
| 12 | Waiting Vendor |
| 13 | Waiting Approval |
| 14 | Resource Assigned |
| 16 | Reopened |
| 19 | End User Note Added |
| 25 | On Hold |
| 48 | Escalate to MC |
| 54 | Resolved \<CSAT Survey\> |
| 57 | Escalate to Wulf |
### priorities
| Value | Label |
|---|---|
| 1 | Standard |
| 2 | Medium |
| 4 | Critical |
| 6 | High |
| 7 | Security Event |
| 8 | Minor Service |
| 9 | Major Service |
| 11 | Fast Track |
### queues (46 active)
Major queues include:
- `29682833` Level 1 Support
- `29682969` Level 2 Support
- `29703428` Level 3 Support
- `29749490` Client Success
- `8` Monitoring Alert
- `29832283` Operations Triage
- `5` Client Triage
- `29853700` Deployment
- `29853698` Project Delivery
- `29853701` IT Operations
- `29853699` Mission Control
- Client-specific queues: TTG, LEC, PER, VCF, Premier Automation, TNT Pizza, Trivium Packaging, Glunt
---
## 3. Datto RMM
### datto_rmm_alerts (~4.3K rows, 61 columns)
- `id` (int PK), `uid` (text) — alert identifiers
- `alert_category`, `alert_type`, `alert_message_en` — what triggered
- `priority` (text) — Critical, High, Moderate, Low, Information
- `resolved` (boolean), `resolved_on` (timestamptz)
- `muted` (boolean)
- `ticket_number` (text) — linked Autotask ticket
- `device_hostname`, `device_ip`, `device_os`, `device_id`
- `site_id` (text) — FK to `datto_rmm_sites`
- `timestamp` (timestamptz) — when alert fired
### datto_rmm_devices (~3.6K rows, 42 columns)
- `id` (int PK), `uid` (text), `hostname`
- `device_type_category` (text) — Server, Desktop, Laptop, Network Device
- `operating_system`, `domain`, `int_ip_address`, `ext_ip_address`
- `online` (boolean), `last_seen` (timestamptz)
- `last_logged_in_user` (text)
- `antivirus_product`, `antivirus_status`, `patch_status`
- `site_id` (int) — FK to `datto_rmm_sites.id`
- `udf` (jsonb) — custom fields
### datto_rmm_sites (~16 columns)
- `id` (int PK), `uid`, `name`
- `autotask_company_id` (int) — **FK to `companies.id`** (links RMM sites to Autotask clients)
- `autotask_company_name`
- `number_of_devices`, `number_of_online_devices`
**Join pattern:** `datto_rmm_sites.autotask_company_id = companies.id`
---
## 4. SentinelOne
### s1_agents (~2.8K rows, 42 columns)
Endpoint security agents.
- `id` (varchar PK) — S1 agent ID
- `computer_name`, `os_name`, `os_type`
- `site_id``s1_sites.id`, `site_name`
- `is_active`, `is_decommissioned`
- `infected` (boolean), `active_threats` (int)
- `network_status`, `mitigation_mode`, `detection_state`
- `external_ip`, `last_active_date`, `last_logged_in_user_name`
- `firewall_enabled` (boolean)
### s1_threats (~4.1K rows, 25 columns)
Detected threats.
- `id` (varchar PK)
- `threat_name`, `classification`, `confidence_level`
- `mitigation_status`, `analyst_verdict`, `incident_status`
- `agent_id``s1_agents.id`
- `agent_computer_name`, `agent_os_name`
- `site_id``s1_sites.id`
### s1_sites (~22 columns)
- `id` (varchar PK), `name`, `account_name`
- `health_status`, `active_licenses`, `total_licenses`
### s1_company_mappings
Maps S1 sites to Autotask companies for cross-referencing.
---
## 5. Veeam Backup
### veeam_organizations (~17 columns)
- `instance_uid` (PK), `name`, `company_id`
- All other Veeam tables FK to `veeam_organizations.instance_uid`
### veeam_backup_jobs (~234 rows)
- `instance_uid`, `name`, `type`, `status`, `last_run`, `next_run`
- `organization_uid``veeam_organizations`
- `backup_server_uid``veeam_backup_servers`
### veeam_backup_agents (~727 rows)
- Backup agents installed on endpoints
- `organization_uid``veeam_organizations`
### veeam_alarms (~581 rows)
- Active alarms/alerts
- `organization_uid``veeam_organizations`
### veeam_protected_workloads, veeam_repositories, veeam_backup_servers
Supporting tables for backup infrastructure.
---
## 6. IT Glue Documentation
### itg_organizations (~330 rows)
- `id` (bigint PK), `name`, `short_name`, `organization_type_name`
- `psa_id` (varchar) — Autotask company ID (string). Join: `itg_organizations.psa_id::bigint = companies.id`
### itg_configurations (~14.7K rows)
Hardware/software assets documented in IT Glue.
- `id`, `organization_id``itg_organizations.id`
- `name`, `hostname`, `serial_number`, `asset_tag`
- `configuration_type_name`, `configuration_status_name`
- `primary_ip`, `mac_address`, `operating_system`
- `warranty_expires_at`, `installed_at`
### itg_passwords (~17 columns)
- `id`, `organization_id`, `name`, `username`, `password_category_name`
- `url`, `notes`
### itg_flexible_assets (~3.2K rows)
Custom documentation (e.g. Backup configs, Email configs, LAN/VLAN, Voice/PBX).
- `id`, `organization_id`, `flexible_asset_type_id`, `flexible_asset_type_name`
- `traits` (jsonb) — all custom field values
### itg_contacts, itg_documents, itg_expirations, itg_domains, itg_locations
Supporting IT documentation tables.
---
## 7. Engagement & Communications
### graph_users
Microsoft 365 users synced from Azure AD.
- `id` (varchar PK), `display_name`, `email`, `job_title`, `department`
- `account_enabled` (boolean)
### teams_meetings (~14.3K rows)
Teams calendar events / meetings.
- `id` (int PK), `user_email`, `subject`
- `start_time`, `end_time` (timestamptz), `duration_minutes`
- `attendee_count`, `client_attendee_count`, `has_client_attendees` (boolean)
### teams_meeting_attendees (~12.6K rows)
- `meeting_id``teams_meetings.id`
- `attendee_email`, `attendee_name`
- `matched_contact_id``contacts.id`
- `matched_company_id``companies.id`
### engagement_snapshots (~1K rows)
Weekly/monthly aggregates of M365 activity per user.
- `user_email`, `period_type` (D7, D30, D90, D180)
- `teams_chat_messages`, `teams_calls`, `teams_meetings_attended`, `teams_meetings_organized`
- `emails_sent`, `emails_received`, `emails_read`
### zoom_meetings, zoom_meeting_participants
- `host_email`, `topic`, `start_time`, `end_time`, `duration_minutes`
- Participants with `matched_contact_id``contacts.id`, `matched_company_id``companies.id`
- `is_internal` (boolean) — internal vs external attendee
### zoom_calls (~2.8K rows)
- `resource_email`, `direction` (inbound/outbound), `call_status`
- `other_party_number`, `other_party_name`
- `matched_contact_id`, `matched_company_id`
---
## 8. Common Join Patterns
```sql
-- Ticket with company, resource, and status label
SELECT t.id, t.title, c.company_name,
r.first_name || ' ' || r.last_name AS technician,
s.label AS status_label, p.label AS priority_label
FROM tickets t
LEFT JOIN companies c ON c.id = t.company_id
LEFT JOIN resources r ON r.id = t.assigned_resource_id
LEFT JOIN statuses s ON s.value = t.status
LEFT JOIN priorities p ON p.value = t.priority
WHERE t.is_deleted IS NOT TRUE;
-- Time entries for a ticket
SELECT te.entry_date, te.hours_worked, te.notes,
r.first_name || ' ' || r.last_name AS technician
FROM time_entries te
JOIN resources r ON r.id = te.resource_id
WHERE te.ticket_id = $1 AND te.is_deleted IS NOT TRUE;
-- RMM device → Autotask company
SELECT d.hostname, d.device_type_category, d.operating_system,
s.autotask_company_name, d.online, d.last_seen
FROM datto_rmm_devices d
JOIN datto_rmm_sites s ON s.id = d.site_id;
-- Config item lookup (always filter by company too)
SELECT ci.id, ci.reference_title, ci.serial_number, c.company_name
FROM configuration_items ci
JOIN companies c ON c.id = ci.company_id
WHERE ci.reference_title = 'DT037' AND ci.company_id = $1;
-- IT Glue org → Autotask company
SELECT ig.name, ig.id AS itg_org_id, c.id AS autotask_company_id, c.company_name
FROM itg_organizations ig
JOIN companies c ON ig.psa_id::bigint = c.id;
-- Meetings with client attendees
SELECT tm.subject, tm.start_time, tm.duration_minutes,
tma.attendee_name, c.company_name
FROM teams_meetings tm
JOIN teams_meeting_attendees tma ON tma.meeting_id = tm.id
LEFT JOIN companies c ON c.id = tma.matched_company_id
WHERE tm.has_client_attendees = true;
```
---
## 9. Important Notes
1. **Soft deletes:** Most Autotask tables have `is_deleted` (boolean) and `deleted_at`. Always add `WHERE is_deleted IS NOT TRUE` unless you want deleted records.
2. **Picklist joins:** `status`, `priority`, `queue_id`, `source` on tickets are integer codes. Join to `statuses`, `priorities`, `queues` on `.value` for labels.
3. **Configuration item names are NOT unique globally.** Names like "DT037" are a per-client naming convention. Always pair with `company_id`.
4. **Timestamps:** Most Autotask timestamps are `timestamp without time zone` stored in UTC. Teams/Zoom timestamps are `timestamp with time zone`.
5. **Monitor tickets:** `tickets.source = 8` indicates RMM-generated tickets. `monitor_id` links to the specific Datto RMM monitor. These represent ~74% of all tickets.
6. **Cross-platform linking:**
- RMM → Autotask: `datto_rmm_sites.autotask_company_id = companies.id`
- IT Glue → Autotask: `itg_organizations.psa_id::bigint = companies.id`
- S1 → Autotask: via `s1_company_mappings`
- Zoom/Teams → Contacts: `matched_contact_id` / `matched_company_id` columns
- Config Items → RMM: `configuration_items.reference_number` sometimes matches RMM device UIDs
7. **Row counts** (as of March 2026): tickets 109K, time_entries 154K, billing_items 96K, ticket_notes 30K, teams_meetings 14K, itg_configurations 15K, configuration_items 7K, companies 4K, contacts 4.2K, datto_rmm_devices 3.6K, datto_rmm_alerts 4.3K, s1_agents 2.8K, s1_threats 4.1K, zoom_calls 2.8K.