# Feature: Datto RMM Overshell Evidence Pipeline **Status:** shipped (Phase 4.2 of the AI Ticket Analyzer) **Migration:** `077_rmm_overshell.sql` **Build notes:** `docs/wulf-pulse-ticket-analyzer-build-notes.md` → Phase 4.2 **Related:** `docs/itglue-asset-audit-spec.md` (where evidence feeds into audits) --- ## What it is Pulse dispatches a curated library of read-only PowerShell scripts via the Datto RMM **Overshell** component, captures the structured output, and makes it available to the IT Glue audit pipeline as authoritative live evidence. Every dispatch is recorded with full audit trail; admins trigger; results land within 30–180 seconds; subsequent audits cite the fresh data with high confidence. The motivating example: an AD health summary at Hynes Industries (replication, dcdiag, named services, IP conflicts) — captured by openclaw on 2026-04-25 — was useful for incident response but stayed in chat. With Phase 4.2 the same intel can be reproduced from Pulse directly, persisted, and cited by the LLM next time we audit a Hynes Configuration. --- ## How it fits ``` Audit pipeline Phase 4.2 → live state ───────────────── ───────────────────── Ticket history ─┐ ▲ ├─► AssetAuditContext ─► LLM ─► Suggestions │ IT Glue schema ─┤ │ │ │ Peer exemplars ─┤ │ │ │ Live RMM evidence ◄─── rmm_executions ◄─── Worker ◄──── Datto RMM (this phase) (poller) ▲ │ rmm_settings ┌───┘ (component_uid)│ │ POST /api/rmm/executions │ user clicks "Run discovery" ``` --- ## User-facing flows ### Site-anchored discovery — `/analyzer/itglue/sites/[companyId]` Admin opens a client's site page, picks a script from the dropdown (filtered to `target_type='site_anchor'`), and clicks. Pulse: 1. Resolves the WNP endpoint via `datto_rmm_devices` matching `LLLCCCWNPNN`. 2. Inserts a pending row in `rmm_executions`. 3. Calls `runQuickJob` with the Overshell `component_uid` and the script body as a variable. 4. The worker polls every 5s; finalizes when stdout/stderr arrive. 5. The page lists the run + parsed evidence inline. Site-anchored scripts: `get-ad-health`, `get-dhcp-scopes`, `get-dns-zones`, `get-network-discovery`. ### Asset-specific discovery — Configuration detail page When viewing an IT Glue Configuration that maps to a Datto RMM device (via `itg_configurations.rmm_id` or hostname match), the picker offers asset-self scripts: `get-services`, `get-installed-software`, `get-event-log-recent`. Output is captured against the asset; subsequent audits on that asset see it. ### Application detail page Application records belong to a client, not a specific server. The picker on the Application page offers **site-anchored** scripts targeting the client's WNP — same as the site page, just discoverable in-context. ### Admin settings — `/admin/rmm-overshell` - Current Overshell `component_uid` + name - Variable name (default `CommandLine`; editable per tenant) - "Re-discover" button (forces a full component scan) - 24-hour activity counters (total / running / failed) - Recent execution log --- ## Permissions | Action | Permission | |---|---| | List scripts / list executions / view execution detail | `requireAuth()` | | Trigger an execution | `requirePermission('rmm','execute')` | | Edit Overshell settings, force discovery | `requirePermission('admin','access')` | `rmm.execute` is granted to `admin` + `super-admin` only (see `lib/permissions.ts`). --- ## Script library Each script is a TypeScript module exporting an `RmmScript`: ```ts { id: 'get-services', name: 'Running services', description: 'Snapshot of all running Windows services …', target_type: 'asset_self', expected_runtime_seconds: 15, version: 1, body: '', parseOutput: (stdout) => parseJsonOutput(stdout), } ``` Convention: scripts end with `ConvertTo-Json -Depth N -Compress` so parsing is `JSON.parse`. The body file in `lib/services/rmm/scripts/` is the source of truth — DB never stores executable code. Adding a new script: 1. Create `lib/services/rmm/scripts/.ts` exporting an `RmmScript`. 2. Import + add to `_all` in `lib/services/rmm/scripts/index.ts`. 3. Bump `version` if you change body or output shape. 4. `npm test` — registry validates uniqueness, presence of required fields, and absence of credential-shaped patterns. ### Current library | Script | Target | Notes | |---|---|---| | `get-services` | asset_self | Get-Service running list (Name/DisplayName/StartType/ServiceType). | | `get-installed-software` | asset_self | Win32 + WoW64 uninstall registry. | | `get-event-log-recent` | asset_self | Last 24h Errors+Warnings from System+Application logs (cap 50). | | `get-ad-health` | site_anchor | Per-DC replication, services, dcdiag pass/fail, recent Netlogon/DNS errors. | | `get-dhcp-scopes` | site_anchor | All authorized DHCP servers, scopes, statistics, reservations. | | `get-dns-zones` | site_anchor | Zones + forwarders + conditional forwarders per DC. | | `get-network-discovery` | site_anchor | Local NIC config + ARP table + IP-conflict detection (catches the proven test case). | --- ## Target resolution ### Site-anchored 1. `companies → datto_rmm_sites` via `autotask_company_id`. 2. `datto_rmm_devices WHERE site_id = $1 AND hostname ~* '^[A-Z]{3}[A-Z]{3}WNP[0-9]{2}$'`. 3. Strict regex check in JS (PostgreSQL regex is permissive; JS pins the shape exactly). 4. Sort: online first, then ascending suffix number. 5. Returns `{ device_uid, hostname, online }` or null. ### Asset-self 1. `IT Glue itg_configurations.rmm_id` → `datto_rmm_devices.uid`. 2. Fallback: hostname match on `datto_rmm_devices.hostname`. If neither resolves, the picker is disabled with the tooltip "No Datto device id." --- ## Execution lifecycle ``` queued ─► running ─► complete | failed | timeout ▲ │ │ │ └──────── markExecutionRunning(jobUid) after runQuickJob success ``` - **queued**: row inserted, `runQuickJob` not yet returned. - **running**: `job_uid` stored, worker is polling. Bounded by `timeout_at` (5 minutes from queue). - **complete**: terminal status from Datto, exit_code 0. - **failed**: terminal status with non-zero exit code, OR runQuickJob threw, OR Datto returned no job_uid. - **timeout**: `timeout_at < NOW()` and worker hasn't seen completion. The worker also writes generic `audit_log` rows on every state change: `rmm.execute` (queue), `rmm.execute.complete`, `rmm.execute.failed`, `rmm.execute.timeout`. --- ## Audit-pipeline integration `buildAssetAuditContext` (in `lib/services/analyzer/asset-audit/data-builder.ts`) loads `rmm_evidence` as a 7th LLM context arm: ``` loadRmmEvidence(itglueOrgId, assetType, assetId) ├─ map IT Glue org → Autotask company (case-insensitive name) ├─ listLatestEvidenceForCompany(companyId, 7) // site-anchored, last 7 days └─ listLatestEvidenceForAsset(assetType, assetId) // asset-self, all-time ``` The prompt renders a new section: ``` === LIVE RMM EVIDENCE (most recent successful Overshell runs; AUTHORITATIVE current state) === [ {execution_id, script_id, target_type, target_hostname, captured_at, parsed}, … ] ``` System prompt rule (in `prompt.ts`): > *"Treat parsed contents as authoritative current state … Use it to > justify suggested values with high confidence — e.g. if Get-Services > lists 'BartenderProcessService' running on the target and a ticket > asked about BarTender printing, suggest adding that service name to > operating_system_notes with confidence=high. Cite execution_id > alongside ticket numbers in evidence_ticket_numbers."* Trim priority on overflow: `peer_global → ticket_evidence → rmm_evidence` (rmm last — highest-value section). --- ## API contract | Method | Route | Auth | Purpose | |---|---|---|---| | GET | `/api/admin/rmm/settings` | admin | Settings + 24h counts | | PATCH | `/api/admin/rmm/settings` | admin | Update `overshellVariableName` | | POST | `/api/admin/rmm/settings/discover` | admin | Force component scan | | GET | `/api/rmm/scripts` | auth | Library catalog (no bodies) | | GET | `/api/rmm/executions` | auth | List (filters: companyId, scriptId, status, assetType, assetId) | | POST | `/api/rmm/executions` | rmm.execute | Queue a fresh execution | | GET | `/api/rmm/executions/[id]` | auth | Single execution detail | | GET | `/api/analyzer/itglue/sites/[companyId]` | auth | Site discovery summary | POST body: ```ts { scriptId: string, target: | { type: 'site_anchor', companyId: number | string } | { type: 'asset_self', deviceUid: string, hostname?, companyId?, assetType?, assetId? }, triggeredByAuditId?: string, } ``` --- ## Safety ### Three-layer audit trail | Table | What it captures | |---|---| | `rmm_executions` | Every dispatch — full lifecycle, raw output (redacted), parsed evidence, target, audit/asset linkage. Forever-retained. | | `analyzer_cost_audit` | Rate-limit decision row per request — approved / blocked. Same view as LLM cost decisions. | | Generic `audit_log` | `rmm.execute`, `rmm.execute.complete`, `rmm.execute.failed`, `rmm.execute.timeout`. Surfaces in `/admin/audit-log`. | ### Rate limit 50 executions per user per 24-hour rolling window. Trips before `runQuickJob` is called; logs the block to `analyzer_cost_audit` with `decision='blocked'`. ### Hard timeout 5 minutes per execution. The worker sweeps `timeout_at < NOW()` rows on every tick and marks them `timeout` (with an `error_message` recording the cause). ### Output redaction Every `raw_stdout` and `raw_stderr` passes through `lib/services/analyzer/itglue-redact.ts:redact()` before persistence and again before the audit prompt sees it. Strips any key matching `/password|secret|key|token|credential|api[_-]?key/i`. Belt-and-braces defense even though the curated library has no credential-handling scripts. ### Registry gating Only ids in the in-code `SCRIPTS` registry can run. The endpoint rejects unknown ids with 400 before any Datto API call. There's no UI to paste ad-hoc PowerShell — everything goes through the typed `RmmScript` interface. --- ## Operational SQL ### Most-recent successful run per (company, script) ```sql SELECT DISTINCT ON (target_company_id, script_id) target_company_id, script_id, target_hostname, completed_at, exit_code FROM rmm_executions WHERE status = 'complete' AND completed_at >= NOW() - INTERVAL '14 days' ORDER BY target_company_id, script_id, completed_at DESC; ``` ### Stuck (queued/running past timeout) ```sql SELECT id, script_id, target_hostname, status, queued_at, timeout_at FROM rmm_executions WHERE status IN ('queued','running') AND timeout_at <= NOW() ORDER BY queued_at; ``` (The worker should sweep these on the next tick.) ### Per-user daily activity ```sql SELECT performed_by_user_id, COUNT(*) FILTER (WHERE queued_at >= NOW() - INTERVAL '24 hours') AS last_24h, COUNT(*) FILTER (WHERE status = 'complete' AND queued_at >= NOW() - INTERVAL '24 hours') AS completed_24h FROM rmm_executions WHERE performed_by_user_id IS NOT NULL GROUP BY performed_by_user_id ORDER BY last_24h DESC; ``` ### Verify a particular script's parser ```sql SELECT id, status, parse_error, jsonb_pretty(parsed_evidence) AS parsed FROM rmm_executions WHERE script_id = 'get-ad-health' ORDER BY queued_at DESC LIMIT 5; ``` --- ## Limitations & non-goals - **Forward-only capture.** No backfill of historical Datto Overshell jobs (per user direction). - **WNP-only target resolution.** Direct-to-DC role detection is a fast-follow if AD scripts that need native DC execution become important. - **Read-only scripts only.** No Overshell-driven configuration changes. Apply documentation updates via the IT Glue Phase 4 path. - **No ad-hoc PowerShell input from the UI.** Registry-listed scripts only. - **No audit-driven auto-execution in v1.** Admin clicks the button. A future version could let the audit panel propose "Run Get-Services to fill this gap" with one-click confirm. - **Credential output never enters the audit prompt.** Three-layer refusal at script-library curation, output redaction, and prompt-side rules. --- ## Files **New:** - `migrations/077_rmm_overshell.sql` - `lib/services/rmm/settings.ts` - `lib/services/rmm/persistence.ts` - `lib/services/rmm/target-resolver.ts` + test - `lib/services/rmm/executor.ts` - `lib/services/rmm/worker.ts` + test - `lib/services/rmm/scripts/types.ts` - `lib/services/rmm/scripts/{get-services,get-installed-software,get-event-log-recent,get-ad-health,get-dhcp-scopes,get-dns-zones,get-network-discovery}.ts` - `lib/services/rmm/scripts/index.ts` + registry test - `app/api/admin/rmm/settings/route.ts` - `app/api/admin/rmm/settings/discover/route.ts` - `app/api/rmm/scripts/route.ts` - `app/api/rmm/executions/route.ts` - `app/api/rmm/executions/[id]/route.ts` - `app/api/analyzer/itglue/sites/[companyId]/route.ts` - `app/admin/rmm-overshell/page.tsx` - `app/analyzer/itglue/sites/[companyId]/page.tsx` - `components/rmm/rmm-script-picker.tsx` - `components/rmm/rmm-execution-stream.tsx` **Modified:** - `lib/permissions.ts` — `rmm: ['read','execute']` - `lib/services/datto-rmm-client.ts` — `findOvershellComponent` - `lib/services/analyzer/asset-audit/data-builder.ts` — 7th evidence arm - `lib/services/analyzer/asset-audit/prompt.ts` — RMM evidence section + system rule - `app/api/analyzer/itglue/applications/[id]/route.ts` — surface `autotaskCompanyId` - `app/api/analyzer/itglue/configurations/[id]/route.ts` — surface `dattoDeviceUid` + `autotaskCompanyId` - `app/analyzer/itglue/applications/[id]/page.tsx` — embedded `` - `app/analyzer/itglue/configurations/[id]/page.tsx` — embedded `` - `components/navigation/app-navigation.tsx` — Admin → "RMM Overshell" ## Phase 4.3: LogLift transport variant The same `rmm_executions` table also stores LogLift event-log uploads (`transport='b2_upload'`). Different transport, same evidence pipeline: - The Datto RMM-registered LogLift component handles its own PowerShell collection + B2 upload. Pulse never sees the script body. - Pulse dispatches via `runQuickJob` with `RunId`, `ClientId`, `WebhookUrl`, `WebhookSecret` variables (same factory + same audit trail as Overshell). - The collector POSTs to `/api/rmm/loglift/upload` when the upload finishes. Pulse downloads from B2, slims, and persists. - The audit-context `rmm_evidence` arm picks the LogLift row up by `script_id='loglift-eventlogs'` automatically — same `listLatestEvidenceForAsset` join as any other asset-self script. See `docs/loglift-eventlog-pipeline-spec.md` for the full webhook contract + slim-shape spec.