wulf-pulse/docs/rmm-overshell-evidence-spec.md

Feature: Datto RMM Overshell Evidence Pipeline

Status: shipped (Phase 4.2 of the AI Ticket Analyzer) Migration: 077_rmm_overshell.sql Build notes: docs/wulf-pulse-ticket-analyzer-build-notes.md → Phase 4.2 Related: docs/itglue-asset-audit-spec.md (where evidence feeds into audits)

---

What it is

Pulse dispatches a curated library of read-only PowerShell scripts via the Datto RMM Overshell component, captures the structured output, and makes it available to the IT Glue audit pipeline as authoritative live evidence. Every dispatch is recorded with full audit trail; admins trigger; results land within 30–180 seconds; subsequent audits cite the fresh data with high confidence.

The motivating example: an AD health summary at Hynes Industries (replication, dcdiag, named services, IP conflicts) — captured by openclaw on 2026-04-25 — was useful for incident response but stayed in chat. With Phase 4.2 the same intel can be reproduced from Pulse directly, persisted, and cited by the LLM next time we audit a Hynes Configuration.

---

How it fits

                      Audit pipeline                Phase 4.2 → live state
                     ─────────────────              ─────────────────────
  Ticket history ─┐                                          ▲
                  ├─► AssetAuditContext ─► LLM ─► Suggestions │
  IT Glue schema ─┤                                           │
                  │                                           │
  Peer exemplars ─┤                                           │
                  │                                           │
  Live RMM evidence ◄─── rmm_executions ◄─── Worker ◄──── Datto RMM
  (this phase)                              (poller)         ▲
                                                             │
                                          rmm_settings   ┌───┘
                                          (component_uid)│
                                                         │
                                              POST /api/rmm/executions
                                                         │
                                                  user clicks "Run discovery"

---

User-facing flows

Site-anchored discovery — /analyzer/itglue/sites/[companyId]

Admin opens a client's site page, picks a script from the dropdown (filtered to target_type='site_anchor'), and clicks. Pulse:

1. Resolves the WNP endpoint via datto_rmm_devices matching LLLCCCWNPNN.
2. Inserts a pending row in rmm_executions.
3. Calls runQuickJob with the Overshell component_uid and the script body as a variable.
4. The worker polls every 5s; finalizes when stdout/stderr arrive.
5. The page lists the run + parsed evidence inline.

Site-anchored scripts: get-ad-health, get-dhcp-scopes, get-dns-zones, get-network-discovery.

Asset-specific discovery — Configuration detail page

When viewing an IT Glue Configuration that maps to a Datto RMM device (via itg_configurations.rmm_id or hostname match), the picker offers asset-self scripts: get-services, get-installed-software, get-event-log-recent. Output is captured against the asset; subsequent audits on that asset see it.

Application detail page

Application records belong to a client, not a specific server. The picker on the Application page offers site-anchored scripts targeting the client's WNP — same as the site page, just discoverable in-context.

Admin settings — /admin/rmm-overshell

• Current Overshell component_uid + name
• Variable name (default CommandLine; editable per tenant)
• "Re-discover" button (forces a full component scan)
• 24-hour activity counters (total / running / failed)
• Recent execution log

---

Permissions

Action	Permission
List scripts / list executions / view execution detail	requireAuth()
Trigger an execution	requirePermission('rmm','execute')
Edit Overshell settings, force discovery	requirePermission('admin','access')

rmm.execute is granted to admin + super-admin only (see lib/permissions.ts).

---

Script library

Each script is a TypeScript module exporting an RmmScript:

{
  id: 'get-services',
  name: 'Running services',
  description: 'Snapshot of all running Windows services …',
  target_type: 'asset_self',
  expected_runtime_seconds: 15,
  version: 1,
  body: '<PowerShell>',
  parseOutput: (stdout) => parseJsonOutput(stdout),
}

Convention: scripts end with ConvertTo-Json -Depth N -Compress so parsing is JSON.parse. The body file in lib/services/rmm/scripts/ is the source of truth — DB never stores executable code.

Adding a new script:

1. Create lib/services/rmm/scripts/<id>.ts exporting an RmmScript.
2. Import + add to _all in lib/services/rmm/scripts/index.ts.
3. Bump version if you change body or output shape.
4. npm test — registry validates uniqueness, presence of required fields, and absence of credential-shaped patterns.

Current library

Script	Target	Notes
get-services	asset_self	Get-Service running list (Name/DisplayName/StartType/ServiceType).
get-installed-software	asset_self	Win32 + WoW64 uninstall registry.
get-event-log-recent	asset_self	Last 24h Errors+Warnings from System+Application logs (cap 50).
get-ad-health	site_anchor	Per-DC replication, services, dcdiag pass/fail, recent Netlogon/DNS errors.
get-dhcp-scopes	site_anchor	All authorized DHCP servers, scopes, statistics, reservations.
get-dns-zones	site_anchor	Zones + forwarders + conditional forwarders per DC.
get-network-discovery	site_anchor	Local NIC config + ARP table + IP-conflict detection (catches the proven test case).

---

Target resolution

Site-anchored

1. companies → datto_rmm_sites via autotask_company_id.
2. datto_rmm_devices WHERE site_id = $1 AND hostname ~* '^[A-Z]{3}[A-Z]{3}WNP[0-9]{2}$'.
3. Strict regex check in JS (PostgreSQL regex is permissive; JS pins the shape exactly).
4. Sort: online first, then ascending suffix number.
5. Returns { device_uid, hostname, online } or null.

Asset-self

1. IT Glue itg_configurations.rmm_id → datto_rmm_devices.uid.
2. Fallback: hostname match on datto_rmm_devices.hostname.

If neither resolves, the picker is disabled with the tooltip "No Datto device id."

---

Execution lifecycle

queued ─► running ─► complete | failed | timeout
        ▲           │
        │           │
        └──────── markExecutionRunning(jobUid)
                    after runQuickJob success

• queued: row inserted, runQuickJob not yet returned.
• running: job_uid stored, worker is polling. Bounded by timeout_at (5 minutes from queue).
• complete: terminal status from Datto, exit_code 0.
• failed: terminal status with non-zero exit code, OR runQuickJob threw, OR Datto returned no job_uid.
• timeout: timeout_at < NOW() and worker hasn't seen completion.

The worker also writes generic audit_log rows on every state change: rmm.execute (queue), rmm.execute.complete, rmm.execute.failed, rmm.execute.timeout.

---

Audit-pipeline integration

buildAssetAuditContext (in lib/services/analyzer/asset-audit/data-builder.ts) loads rmm_evidence as a 7th LLM context arm:

loadRmmEvidence(itglueOrgId, assetType, assetId)
  ├─ map IT Glue org → Autotask company (case-insensitive name)
  ├─ listLatestEvidenceForCompany(companyId, 7)  // site-anchored, last 7 days
  └─ listLatestEvidenceForAsset(assetType, assetId)  // asset-self, all-time

The prompt renders a new section:

=== LIVE RMM EVIDENCE (most recent successful Overshell runs; AUTHORITATIVE current state) ===
[ {execution_id, script_id, target_type, target_hostname, captured_at, parsed}, … ]

System prompt rule (in prompt.ts):

"Treat parsed contents as authoritative current state … Use it to justify suggested values with high confidence — e.g. if Get-Services lists 'BartenderProcessService' running on the target and a ticket asked about BarTender printing, suggest adding that service name to operating_system_notes with confidence=high. Cite execution_id alongside ticket numbers in evidence_ticket_numbers."

Trim priority on overflow: peer_global → ticket_evidence → rmm_evidence (rmm last — highest-value section).

---

API contract

Method	Route	Auth	Purpose
GET	/api/admin/rmm/settings	admin	Settings + 24h counts
PATCH	/api/admin/rmm/settings	admin	Update overshellVariableName
POST	/api/admin/rmm/settings/discover	admin	Force component scan
GET	/api/rmm/scripts	auth	Library catalog (no bodies)
GET	/api/rmm/executions	auth	List (filters: companyId, scriptId, status, assetType, assetId)
POST	/api/rmm/executions	rmm.execute	Queue a fresh execution
GET	/api/rmm/executions/[id]	auth	Single execution detail
GET	/api/analyzer/itglue/sites/[companyId]	auth	Site discovery summary

POST body:

{
  scriptId: string,
  target:
    | { type: 'site_anchor', companyId: number | string }
    | { type: 'asset_self', deviceUid: string, hostname?, companyId?, assetType?, assetId? },
  triggeredByAuditId?: string,
}

---

Safety

Three-layer audit trail

Table	What it captures
rmm_executions	Every dispatch — full lifecycle, raw output (redacted), parsed evidence, target, audit/asset linkage. Forever-retained.
analyzer_cost_audit	Rate-limit decision row per request — approved / blocked. Same view as LLM cost decisions.
Generic audit_log	rmm.execute, rmm.execute.complete, rmm.execute.failed, rmm.execute.timeout. Surfaces in /admin/audit-log.

Rate limit

50 executions per user per 24-hour rolling window. Trips before runQuickJob is called; logs the block to analyzer_cost_audit with decision='blocked'.

Hard timeout

5 minutes per execution. The worker sweeps timeout_at < NOW() rows on every tick and marks them timeout (with an error_message recording the cause).

Output redaction

Every raw_stdout and raw_stderr passes through lib/services/analyzer/itglue-redact.ts:redact() before persistence and again before the audit prompt sees it. Strips any key matching /password|secret|key|token|credential|api[_-]?key/i. Belt-and-braces defense even though the curated library has no credential-handling scripts.

Registry gating

Only ids in the in-code SCRIPTS registry can run. The endpoint rejects unknown ids with 400 before any Datto API call. There's no UI to paste ad-hoc PowerShell — everything goes through the typed RmmScript interface.

---

Operational SQL

Most-recent successful run per (company, script)

SELECT DISTINCT ON (target_company_id, script_id)
       target_company_id, script_id, target_hostname,
       completed_at, exit_code
  FROM rmm_executions
 WHERE status = 'complete'
   AND completed_at >= NOW() - INTERVAL '14 days'
 ORDER BY target_company_id, script_id, completed_at DESC;

Stuck (queued/running past timeout)

SELECT id, script_id, target_hostname, status, queued_at, timeout_at
  FROM rmm_executions
 WHERE status IN ('queued','running')
   AND timeout_at <= NOW()
 ORDER BY queued_at;

(The worker should sweep these on the next tick.)

Per-user daily activity

SELECT performed_by_user_id,
       COUNT(*) FILTER (WHERE queued_at >= NOW() - INTERVAL '24 hours') AS last_24h,
       COUNT(*) FILTER (WHERE status = 'complete' AND queued_at >= NOW() - INTERVAL '24 hours') AS completed_24h
  FROM rmm_executions
 WHERE performed_by_user_id IS NOT NULL
 GROUP BY performed_by_user_id
 ORDER BY last_24h DESC;

Verify a particular script's parser

SELECT id, status, parse_error, jsonb_pretty(parsed_evidence) AS parsed
  FROM rmm_executions
 WHERE script_id = 'get-ad-health'
 ORDER BY queued_at DESC
 LIMIT 5;

---

Limitations & non-goals

• Forward-only capture. No backfill of historical Datto Overshell jobs (per user direction).
• WNP-only target resolution. Direct-to-DC role detection is a fast-follow if AD scripts that need native DC execution become important.
• Read-only scripts only. No Overshell-driven configuration changes. Apply documentation updates via the IT Glue Phase 4 path.
• No ad-hoc PowerShell input from the UI. Registry-listed scripts only.
• No audit-driven auto-execution in v1. Admin clicks the button. A future version could let the audit panel propose "Run Get-Services to fill this gap" with one-click confirm.
• Credential output never enters the audit prompt. Three-layer refusal at script-library curation, output redaction, and prompt-side rules.

---

Files

New:

• migrations/077_rmm_overshell.sql
• lib/services/rmm/settings.ts
• lib/services/rmm/persistence.ts
• lib/services/rmm/target-resolver.ts + test
• lib/services/rmm/executor.ts
• lib/services/rmm/worker.ts + test
• lib/services/rmm/scripts/types.ts
• lib/services/rmm/scripts/{get-services,get-installed-software,get-event-log-recent,get-ad-health,get-dhcp-scopes,get-dns-zones,get-network-discovery}.ts
• lib/services/rmm/scripts/index.ts + registry test
• app/api/admin/rmm/settings/route.ts
• app/api/admin/rmm/settings/discover/route.ts
• app/api/rmm/scripts/route.ts
• app/api/rmm/executions/route.ts
• app/api/rmm/executions/[id]/route.ts
• app/api/analyzer/itglue/sites/[companyId]/route.ts
• app/admin/rmm-overshell/page.tsx
• app/analyzer/itglue/sites/[companyId]/page.tsx
• components/rmm/rmm-script-picker.tsx
• components/rmm/rmm-execution-stream.tsx

Modified:

• lib/permissions.ts — rmm: ['read','execute']
• lib/services/datto-rmm-client.ts — findOvershellComponent
• lib/services/analyzer/asset-audit/data-builder.ts — 7th evidence arm
• lib/services/analyzer/asset-audit/prompt.ts — RMM evidence section + system rule
• app/api/analyzer/itglue/applications/[id]/route.ts — surface autotaskCompanyId
• app/api/analyzer/itglue/configurations/[id]/route.ts — surface dattoDeviceUid + autotaskCompanyId
• app/analyzer/itglue/applications/[id]/page.tsx — embedded <RmmScriptPicker filter='site_anchor'/>
• app/analyzer/itglue/configurations/[id]/page.tsx — embedded <RmmScriptPicker filter='asset_self'/>
• components/navigation/app-navigation.tsx — Admin → "RMM Overshell"

Phase 4.3: LogLift transport variant

The same rmm_executions table also stores LogLift event-log uploads (transport='b2_upload'). Different transport, same evidence pipeline:

• The Datto RMM-registered LogLift component handles its own PowerShell collection + B2 upload. Pulse never sees the script body.
• Pulse dispatches via runQuickJob with RunId, ClientId, WebhookUrl, WebhookSecret variables (same factory + same audit trail as Overshell).
• The collector POSTs to /api/rmm/loglift/upload when the upload finishes. Pulse downloads from B2, slims, and persists.
• The audit-context rmm_evidence arm picks the LogLift row up by script_id='loglift-eventlogs' automatically — same listLatestEvidenceForAsset join as any other asset-self script.

See docs/loglift-eventlog-pipeline-spec.md for the full webhook contract + slim-shape spec.