feat: Mimecast multi-tenant held mail viewer

- migration 062: mimecast_tenants table (company_id, client_id/secret, account_code) - Seed Wulf (CUSA13A95) + Seubert (CUSA96A181) tenants - MimecastClient.getHeldMessages(): full pagination via meta.pagination.next cursor (API always returns 10/page regardless of pageSize param, totalCount in meta) - getMimecastClientForTenant() factory for per-tenant instantiation - GET /api/mimecast/held?tenantId=&recipient= — fetches all tenants in parallel, merges + sorts by date, returns per-tenant counts + combined messages[] - Held Mail tab on /admin/sync/mimecast (on-demand load, recipient filter, tenant badges, policy filter dropdown, DMARC/impersonation highlighted red)
2026-03-31 22:38:22 -04:00 · 2026-03-31 22:38:22 -04:00 · fcdec8e38b
commit fcdec8e38b
parent a98c0daf15
12 changed files with 2208 additions and 27 deletions
--- a/docs/DATTO_RMM_API_GUIDE.md
+++ b/docs/DATTO_RMM_API_GUIDE.md
@ -0,0 +1,280 @@
+# Datto RMM API Guide for openclaw
+
+Everything learned from building the Datto RMM integration in Pulse. Use this as a reference for any future work touching the Datto RMM API.
+
+---
+
+## Authentication
+
+Datto RMM API v2 uses **OAuth2 with password grant type** — not a simple API key header.
+
+- **Auth endpoint**: `https://concord-api.centrastage.net/auth/oauth/token`
+- **Base API URL**: `https://concord-api.centrastage.net/api/v2`
+- **Client credentials**: always `public-client:public` (this is the public OAuth client — not your credentials)
+- **Your credentials**: API Access Key = `username`, API Secret Key = `password`
+
+```http
+POST https://concord-api.centrastage.net/auth/oauth/token
+Authorization: Basic cHVibGljLWNsaWVudDpwdWJsaWM=   (base64 of "public-client:public")
+Content-Type: application/x-www-form-urlencoded
+
+grant_type=password&username=YOUR_API_KEY&password=YOUR_API_SECRET
+```
+
+Response returns `access_token` (bearer token). Tokens last ~1 hour; refresh proactively at 50 minutes.
+
+**Environment variables used in Pulse:**
+```
+DATTO_RMM_API_URL=https://concord-api.centrastage.net
+DATTO_RMM_API_KEY=<your API access key>
+DATTO_RMM_API_SECRET=<your API secret key>
+DATTO_RMM_WEBHOOK_SECRET=<shared secret for webhook validation>
+```
+
+---
+
+## Making API Calls
+
+All requests go to `https://concord-api.centrastage.net/api/v2{endpoint}` with:
+
+```http
+Authorization: Bearer {access_token}
+Content-Type: application/json
+Accept: application/json
+```
+
+The response body is always JSON. Always parse it — the API never returns empty 200s (except for write operations where you should guard with `text ? JSON.parse(text) : {}`).
+
+---
+
+## Key Endpoints
+
+### Sites
+```
+GET  /account/sites                  → { sites: [...], pageDetails: {...} }
+GET  /site/{siteUid}/devices         → { devices: [...] }
+```
+
+Sites map to Autotask companies via `autotaskCompanyId` / `autotaskCompanyName` fields on the site object. This is the primary join key between the two systems.
+
+### Devices
+```
+GET  /account/devices                → { devices: [...], pageDetails: {...} }
+GET  /devices/{deviceId}             → { item: DattoRMMDevice }
+GET  /device/{deviceId}/auditdata    → detailed hardware info (bios, processors, memory, disks)
+```
+
+Default page size is 250. Always paginate using `pageDetails.nextPageUrl` — follow it until null.
+
+### Alerts
+```
+GET  /account/alerts/open            → { alerts: [...], pageDetails: {...} }
+GET  /account/alerts/resolved        → { alerts: [...], pageDetails: {...} }
+GET  /alert/{alertUid}               → single alert with full alertContext
+```
+
+Resolved alerts can be huge — cap page fetches (e.g. 4 pages) rather than fetching all.
+
+For PING alerts, `alertContext['@class'] === 'ping_ctx'` and `alertContext.instanceName` holds the ping target hostname.
+
+### Components & Quick Jobs
+```
+GET  /account/components             → { components: [...] }   (automation scripts/tasks)
+PUT  /device/{deviceUid}/quickjob    → run a component on a device
+GET  /job/{jobUid}/results/{deviceUid} → job output/results
+```
+
+Quick job payload:
+```json
+{
+  "jobName": "My Job Name",
+  "jobComponent": {
+    "componentUid": "abc-123-...",
+    "variables": [
+      { "name": "VAR_NAME", "value": "value" }
+    ]
+  }
+}
+```
+
+---
+
+## Pagination Pattern
+
+The API uses `pageDetails.nextPageUrl` for cursor-based pagination. Always follow it:
+
+```typescript
+let url: string | null = `https://concord-api.centrastage.net/api/v2/account/sites?pageSize=250`;
+while (url) {
+  const resp = await fetch(url, { headers: { Authorization: `Bearer ${token}` } });
+  const body = await resp.json();
+  items.push(...(body.sites || []));
+  url = body.pageDetails?.nextPageUrl ?? null;
+}
+```
+
+Some older endpoints use `?page=1&pageSize=250` query params — check which pattern each endpoint uses. The `fetchAllPages` helper in `datto-rmm-client.ts` handles the `nextPageUrl` style.
+
+---
+
+## Data Model (PostgreSQL Tables)
+
+### `datto_rmm_sites`
+| Column | Notes |
+|--------|-------|
+| `id` | Datto integer ID (PK) |
+| `uid` | Datto GUID string (unique) |
+| `name` | Site display name |
+| `autotask_company_id` | FK → `companies(id)` — join key |
+| `autotask_company_name` | Denormalized for display |
+| `number_of_devices` | Online/offline counts from API |
+| `portal_url` | Deep link to Datto portal |
+
+### `datto_rmm_devices`
+| Column | Notes |
+|--------|-------|
+| `id` | Datto integer ID (PK) |
+| `uid` | Datto GUID (unique, used for API calls and quick jobs) |
+| `site_id` | FK → `datto_rmm_sites(id)` |
+| `hostname` | Device hostname |
+| `online` / `suspended` / `deleted` | Status flags |
+| `device_type_category` | e.g. "Desktop", "Server", "Laptop" |
+| `operating_system` | OS string |
+| `antivirus_product` / `antivirus_status` | AV info |
+| `patch_status` | Patch management summary |
+| `patches_approved_pending` | Count of pending patches |
+| `last_seen` | Timestamp (ms epoch from API, stored as TIMESTAMPTZ) |
+| `web_remote_url` | Direct remote session URL |
+| `udf` | JSONB — User Defined Fields 1–10 from API |
+
+### `datto_rmm_alerts`
+| Column | Notes |
+|--------|-------|
+| `alert_uid` | PK (Datto GUID string) |
+| `device_uid` / `device_name` | Source device |
+| `site_uid` / `site_name` | Source site |
+| `priority` | "Critical", "High", "Moderate", "Low", "Information" |
+| `alert_context` | JSONB — varies by alert type, contains `@class` discriminator |
+| `resolved` | Boolean |
+| `resolved_on` | Timestamp |
+| `muted` | Boolean |
+| `ticket_number` | Linked Autotask ticket if any |
+| `alert_category` | e.g. "Patch Management" |
+| `alert_type` | e.g. "PING", "DISK", "CPU" |
+| `alert_message_en` | Human-readable alert message |
+| `device_udf1`–`device_udf29` | All 29 UDF fields from webhook payloads |
+| `ping_target` | Resolved from API for PING alerts |
+| `triggered` | Raw webhook "True"/"False" string |
+
+### `datto_rmm_webhook_logs`
+Raw capture table — stores every incoming webhook payload verbatim for inspection before processing.
+
+---
+
+## Webhooks
+
+Datto RMM can POST alert events to your endpoint when alerts fire or resolve.
+
+**Webhook receiver**: `POST /api/webhooks/datto-rmm`
+
+**Authentication**: Datto sends a shared secret in the `X-Datto-Webhook-Secret` header. Validate it against `DATTO_RMM_WEBHOOK_SECRET` env var.
+
+**Alert webhook payload shape** (flat JSON, not nested like the REST API):
+```json
+{
+  "alert_uid": "abc-123-...",
+  "triggered": "True",          // "True" = alert fired, "False" = resolved
+  "alert_type": "PING",
+  "alert_category": "Networking",
+  "alert_priority": "Critical",
+  "alert_message_en": "Ping monitor failed for ...",
+  "device_uid": "...",
+  "device_hostname": "SERVER01",
+  "device_ip": "10.0.0.1",
+  "device_os": "Windows Server 2019",
+  "device_description": "...",
+  "device_id": "12345",
+  "site_uid": "...",
+  "site_name": "ACME Corp",
+  "site_id": "678",
+  "platform": "Windows",
+  "last_user": "DOMAIN\\user",
+  "device_udf1": "...",
+  // ... device_udf2 through device_udf29
+}
+```
+
+Key gotcha: `triggered === "True"` means the alert is **active** (not resolved). `triggered === "False"` means it **resolved**. Map `triggered === "False"` → `resolved = true`.
+
+**Always return HTTP 200** even on errors — Datto will disable your webhook endpoint if it receives repeated non-200 responses.
+
+---
+
+## Sync Architecture in Pulse
+
+The sync pipeline runs in order: **sites → devices → open_alerts → resolved_alerts**
+
+Sites must sync before devices (FK constraint). The sync service guards against orphaned FK refs by pre-fetching known IDs and setting FK fields to null when the parent doesn't exist yet.
+
+Timestamps from the Datto API come as **millisecond epoch integers**. Convert with `new Date(milliseconds)` before storing in PostgreSQL.
+
+The factory singleton (`datto-rmm-factory.ts`) is the standard way to get a client instance in API routes:
+```typescript
+import { getDattoRMMClient } from '@/lib/services/datto-rmm-factory';
+const client = getDattoRMMClient();
+```
+
+---
+
+## Common Gotchas
+
+1. **Auth URL is different from API URL** — token is fetched from `concord-api.centrastage.net/auth/...`, API calls go to `concord-api.centrastage.net/api/v2/...`
+
+2. **Parse auth response carefully** — if auth fails, the server may return HTML (an error page) instead of JSON. Always try/catch the JSON.parse and log the raw text on failure.
+
+3. **`uid` vs `id`** — devices and sites have both an integer `id` and a GUID `uid`. The REST API uses `uid` in paths for most operations. Quick jobs require `deviceUid` (the GUID), not the integer ID.
+
+4. **UDFs in webhooks vs REST API** — The REST API returns UDFs as `udf: { udf1: "...", udf2: "..." }` (nested object, up to 10). Webhooks flatten them to `device_udf1` through `device_udf29` as top-level fields (29 total).
+
+5. **Alert context varies by type** — always check `alertContext['@class']` to know what fields are available. For PING alerts, fetch `/alert/{uid}` to get `instanceName` (the ping target) since it's not in the bulk alert list response.
+
+6. **Page size 250** is the effective maximum — don't request more.
+
+7. **Resolved alerts grow without bound** — never fetch all resolved alerts in production. Limit to recent pages (e.g. 4 pages = ~1000 most recent).
+
+8. **Site → Company mapping** — `site.autotaskCompanyId` is a string from the API even though it's an integer ID. Always `parseInt()` it and validate `> 0` and `!isNaN()` before using as a FK.
+
+---
+
+## Quick Reference: Alert Priority Values
+
+- `Critical`
+- `High`
+- `Moderate`
+- `Low`
+- `Information`
+
+## Quick Reference: Common Alert Types
+
+- `PING` — ping monitor failure
+- `DISK` — disk space/health
+- `CPU` — CPU utilization
+- `MEMORY` — RAM utilization
+- `SERVICE` — Windows service down
+- `EVENTLOG` — Windows event log match
+- `PATCH` — patch management
+- `ANTIVIRUS` — AV status
+
+---
+
+## Pipeline Integration
+
+In Pulse, incoming Datto RMM webhooks can trigger the **pipeline engine** (fire-and-forget):
+
+```typescript
+pipelineEngine.processTrigger('datto_rmm', payload).catch(err =>
+  console.error('[DATTO-RMM-WEBHOOK] Pipeline processing error:', err)
+);
+```
+
+The trigger type `'datto_rmm'` matches pipeline rules configured in the admin UI. Pipelines can take actions like creating Autotask tickets, sending notifications, etc.
--- a/docs/clista-changes-2026-03-26.md
+++ b/docs/clista-changes-2026-03-26.md
@ -0,0 +1,104 @@
+# Clista Electric — User List Changes Since January 2026
+
+_Comparison of "Clista Approved - User List and Classification.xlsx" against Autotask/M365 as of 2026-03-26. Enriched with ticket history._
+
+---
+
+## New Users Added Since January
+
+| Name | Email | Added | Classification (from tickets) |
+|---|---|---|---|
+| **Brett Tokarski** | btokarski@clistaelectric.com | Jan 6 | Likely **BRONZE** — iPad + Bluebeam Complete (estimator/PM profile), no laptop deployed |
+| **Joe Glassbrenner** | jglassbrenner@clistaelectric.com | Jan 6 | Likely **EMAIL ONLY** — iPad setup only, no workstation |
+| **Daniel Archer** | darcher@clistaelectric.com | Mar 3 | Likely **EMAIL ONLY** — mailbox + Foreman distribution list + iPad; field foreman |
+
+New device added: **SPRO009** (Jan 15, assigned to Josh Miller — "Activate SPRO009 and install Wulf tools")
+
+---
+
+## Deleted Since January
+
+- **Joe Laplace** (jlaplace@clistaelectric.com) — removed Jan 6. His laptop (**LT049**) was reassigned to Bryan Detweiler (ticket T20260119.0181).
+
+---
+
+## Users in Autotask NOT on the Spreadsheet
+
+Ticket history used to determine likely classification:
+
+| Name | Email | Likely Classification | Evidence |
+|---|---|---|---|
+| **Bryan Detweiler** | bdetweiler@clistaelectric.com | **BRONZE USER** | Received LT049 (Joe Laplace's repurposed laptop, Jan 13). iPad email setup Oct 2025. |
+| **Nicholas Flinko** | nflinko@clistaelectric.com | **BRONZE USER** | Has LT032 (multiple patch/disk/Teams tickets on LT032 under his name). Spreadsheet incorrectly lists LT032 under Jared Boger. |
+| **Tanner Lacher** | tlacher@clistaelectric.com | **EMAIL ONLY** | "New iPad and Email Setup Request" Oct 2025 — iPad only, no workstation. |
+| **Randy Nocleg** | rnocleg@clistaelectric.com | **EMAIL ONLY** | "Mobile email set up" Jun 2025 — mobile only. |
+| **John Pronko** | jpronko@clistaelectric.com | **EMAIL ONLY** | "Duo and Outlook Setup on iPad" Nov 2025 — iPad only. |
+| **Bill Shindledecker** | bShindledecker@clistaelectric.com | **EMAIL ONLY** | "Email Setup for iPad" Oct 2025 — iPad only. |
+| **Jeff Gatto** | jgatto@clistaelectric.com | **Unknown** | No tickets found. |
+| **Rob Gerhart** | rgerhart@clistaelectric.com | **Unknown** | No tickets found. |
+| **Mike Jr.** | mikejr@clistaelectric.com | **Likely duplicate** | Only 2 tickets (unusual sign-in alerts, Oct 2025). Likely a legacy/stale account for Mike Clista Jr. (mclistajr@). |
+| **safety unknown** | safety@clistaelectric.com | **DEVICE EMAIL or GENERAL** | Generic safety mailbox — no user-specific tickets. |
+
+---
+
+## Flagged REMOVE — Still Have Active Licenses in M365
+
+Both are still licensed in the M365 export:
+
+- **CAD1** — still has Microsoft 365 Business Standard
+- **Sam Cocola** — still has Microsoft 365 Basic
+
+---
+
+## Device Reassignments Found in Tickets
+
+Devices that moved or were reassigned since the spreadsheet was made:
+
+| Device | Old Assignment (spreadsheet) | New Assignment (from tickets) | Notes |
+|---|---|---|---|
+| **LT049** | (not on spreadsheet) | Bryan Detweiler | Reassigned from Joe Laplace when he left (Jan 13) |
+| **LT074** | (not on spreadsheet) | Mark Blum | Repurposed deployment Mar 4; onboarding check-in Mar 11. **Mark Blum should be reclassified — he was EMAIL ONLY but now has a workstation.** |
+| **LT078** | (not on spreadsheet) | Brendon Bittel | Patch alerts Dec 2025. **Bittel was EMAIL ONLY but now has a laptop.** |
+| **LT079** | (not on spreadsheet) | Travis Lenhart | Patch failure Mar 2026. Lenhart was EMAIL ONLY with MIX003 (now inactive). **Upgrade to BRONZE.** |
+| **LT080** | (not on spreadsheet) | Andrew Holzworth | Docking station ticket Feb 2026. LT040 (his old device) was repurposed for field use Jan 20. |
+| **LT076** | (not on spreadsheet) | Roseann March | VPN confirmation ticket Nov 2025. She already had WL-LT004 + DT034; LT076 may be a replacement. |
+| **LT077** | (not on spreadsheet) | Roseann March | Patch failure Dec 2025 — second device or replacement for WL-LT004. |
+| **LT032** | Jared Boger (EMAIL ONLY) | Nicholas Flinko | Flinko is the contact on all LT032 alerts — Boger classification may be stale. |
+
+---
+
+## Users Whose Classification Should Be Updated
+
+Based on device deployments found in tickets:
+
+| Name | Current Classification | Suggested Update | Reason |
+|---|---|---|---|
+| **Mark Blum** | EMAIL ONLY | **BRONZE USER** | Received workstation LT074 (Mar 2026) |
+| **Brendon Bittel** | EMAIL ONLY | **BRONZE USER** | Has laptop LT078 (added Dec 2025) |
+| **Travis Lenhart** | EMAIL ONLY | **BRONZE USER** | Has laptop LT079; MIX003 retired |
+| **Bryan Detweiler** | Not on list | **BRONZE USER** | Add to list — has LT049 |
+| **Jared Boger** | EMAIL ONLY (LT032) | Review — LT032 now under Flinko | Boger may have no device now |
+
+---
+
+## Devices on Spreadsheet Now Inactive in DB
+
+| Device | Assigned To (per spreadsheet) | Status | Likely Replacement |
+|---|---|---|---|
+| LT017 | Justin Klosky | Inactive | Unknown |
+| LT028 | Anthony Laskey | Inactive | Unknown |
+| LT032 | Jared Boger | Inactive | Now under Nicholas Flinko |
+| MIX003 | Travis Lenhart | Inactive | LT079 |
+| WL-LT004 | Roseann March | Inactive | LT076 or LT077 |
+| LT040 | Andrew Holzworth | Repurposed | LT080 — "repurposed for field use" Jan 20 |
+
+---
+
+## Data Quality Issues
+
+- **Dominic Edwards** — email typo in Autotask: `dedwards@clistaelectric.om` (missing the 'c')
+- **Ron Marangoni** — spreadsheet says `marangoni@`, M365 and Autotask both show `rmarangoni@`
+- **Sonny Stewart** — spreadsheet lists "Microsoft 365" license but M365 export shows no license assigned
+- **Michael Skibinski** — duplicate contact record created Feb 13, 2026 (two records, same email)
+- **Dave Warywoda**, **Grant Hoffman**, **Dominic Edwards**, **Donald Maraugha** — on rachel_list but absent from M365 export entirely
+- **LT073 and LT075** — active devices with no ticket history found; assignment unknown