wulf-pulse/docs/rmm-overshell-evidence-spec.md

# Feature: Datto RMM Overshell Evidence Pipeline

**Status:** shipped (Phase 4.2 of the AI Ticket Analyzer)
**Migration:** `077_rmm_overshell.sql`
**Build notes:** `docs/wulf-pulse-ticket-analyzer-build-notes.md` → Phase 4.2
**Related:** `docs/itglue-asset-audit-spec.md` (where evidence feeds into audits)

---

## What it is

Pulse dispatches a curated library of read-only PowerShell scripts via the
Datto RMM **Overshell** component, captures the structured output, and
makes it available to the IT Glue audit pipeline as authoritative live
evidence. Every dispatch is recorded with full audit trail; admins
trigger; results land within 30–180 seconds; subsequent audits cite the
fresh data with high confidence.

The motivating example: an AD health summary at Hynes Industries
(replication, dcdiag, named services, IP conflicts) — captured by openclaw
on 2026-04-25 — was useful for incident response but stayed in chat. With
Phase 4.2 the same intel can be reproduced from Pulse directly, persisted,
and cited by the LLM next time we audit a Hynes Configuration.

---

## How it fits

```
                      Audit pipeline                Phase 4.2 → live state
                     ─────────────────              ─────────────────────
  Ticket history ─┐                                          ▲
                  ├─► AssetAuditContext ─► LLM ─► Suggestions │
  IT Glue schema ─┤                                           │
                  │                                           │
  Peer exemplars ─┤                                           │
                  │                                           │
  Live RMM evidence ◄─── rmm_executions ◄─── Worker ◄──── Datto RMM
  (this phase)                              (poller)         ▲
                                                             │
                                          rmm_settings   ┌───┘
                                          (component_uid)│
                                                         │
                                              POST /api/rmm/executions
                                                         │
                                                  user clicks "Run discovery"
```

---

## User-facing flows

### Site-anchored discovery — `/analyzer/itglue/sites/[companyId]`

Admin opens a client's site page, picks a script from the dropdown
(filtered to `target_type='site_anchor'`), and clicks. Pulse:

1. Resolves the WNP endpoint via `datto_rmm_devices` matching `LLLCCCWNPNN`.
2. Inserts a pending row in `rmm_executions`.
3. Calls `runQuickJob` with the Overshell `component_uid` and the script
   body as a variable.
4. The worker polls every 5s; finalizes when stdout/stderr arrive.
5. The page lists the run + parsed evidence inline.

Site-anchored scripts: `get-ad-health`, `get-dhcp-scopes`, `get-dns-zones`,
`get-network-discovery`.

### Asset-specific discovery — Configuration detail page

When viewing an IT Glue Configuration that maps to a Datto RMM device
(via `itg_configurations.rmm_id` or hostname match), the picker offers
asset-self scripts: `get-services`, `get-installed-software`,
`get-event-log-recent`. Output is captured against the asset; subsequent
audits on that asset see it.

### Application detail page

Application records belong to a client, not a specific server. The picker
on the Application page offers **site-anchored** scripts targeting the
client's WNP — same as the site page, just discoverable in-context.

### Admin settings — `/admin/rmm-overshell`

- Current Overshell `component_uid` + name
- Variable name (default `CommandLine`; editable per tenant)
- "Re-discover" button (forces a full component scan)
- 24-hour activity counters (total / running / failed)
- Recent execution log

---

## Permissions

| Action | Permission |
|---|---|
| List scripts / list executions / view execution detail | `requireAuth()` |
| Trigger an execution | `requirePermission('rmm','execute')` |
| Edit Overshell settings, force discovery | `requirePermission('admin','access')` |

`rmm.execute` is granted to `admin` + `super-admin` only (see
`lib/permissions.ts`).

---

## Script library

Each script is a TypeScript module exporting an `RmmScript`:

```ts
{
  id: 'get-services',
  name: 'Running services',
  description: 'Snapshot of all running Windows services …',
  target_type: 'asset_self',
  expected_runtime_seconds: 15,
  version: 1,
  body: '<PowerShell>',
  parseOutput: (stdout) => parseJsonOutput(stdout),
}
```

Convention: scripts end with `ConvertTo-Json -Depth N -Compress` so
parsing is `JSON.parse`. The body file in `lib/services/rmm/scripts/` is
the source of truth — DB never stores executable code.

Adding a new script:

1. Create `lib/services/rmm/scripts/<id>.ts` exporting an `RmmScript`.
2. Import + add to `_all` in `lib/services/rmm/scripts/index.ts`.
3. Bump `version` if you change body or output shape.
4. `npm test` — registry validates uniqueness, presence of required
   fields, and absence of credential-shaped patterns.

### Current library

| Script | Target | Notes |
|---|---|---|
| `get-services` | asset_self | Get-Service running list (Name/DisplayName/StartType/ServiceType). |
| `get-installed-software` | asset_self | Win32 + WoW64 uninstall registry. |
| `get-event-log-recent` | asset_self | Last 24h Errors+Warnings from System+Application logs (cap 50). |
| `get-ad-health` | site_anchor | Per-DC replication, services, dcdiag pass/fail, recent Netlogon/DNS errors. |
| `get-dhcp-scopes` | site_anchor | All authorized DHCP servers, scopes, statistics, reservations. |
| `get-dns-zones` | site_anchor | Zones + forwarders + conditional forwarders per DC. |
| `get-network-discovery` | site_anchor | Local NIC config + ARP table + IP-conflict detection (catches the proven test case). |

---

## Target resolution

### Site-anchored

1. `companies → datto_rmm_sites` via `autotask_company_id`.
2. `datto_rmm_devices WHERE site_id = $1 AND hostname ~* '^[A-Z]{3}[A-Z]{3}WNP[0-9]{2}$'`.
3. Strict regex check in JS (PostgreSQL regex is permissive; JS pins the
   shape exactly).
4. Sort: online first, then ascending suffix number.
5. Returns `{ device_uid, hostname, online }` or null.

### Asset-self

1. `IT Glue itg_configurations.rmm_id` → `datto_rmm_devices.uid`.
2. Fallback: hostname match on `datto_rmm_devices.hostname`.

If neither resolves, the picker is disabled with the tooltip "No Datto
device id."

---

## Execution lifecycle

```
queued ─► running ─► complete | failed | timeout
        ▲           │
        │           │
        └──────── markExecutionRunning(jobUid)
                    after runQuickJob success
```

- **queued**: row inserted, `runQuickJob` not yet returned.
- **running**: `job_uid` stored, worker is polling. Bounded by `timeout_at`
  (5 minutes from queue).
- **complete**: terminal status from Datto, exit_code 0.
- **failed**: terminal status with non-zero exit code, OR runQuickJob
  threw, OR Datto returned no job_uid.
- **timeout**: `timeout_at < NOW()` and worker hasn't seen completion.

The worker also writes generic `audit_log` rows on every state change:
`rmm.execute` (queue), `rmm.execute.complete`, `rmm.execute.failed`,
`rmm.execute.timeout`.

---

## Audit-pipeline integration

`buildAssetAuditContext` (in `lib/services/analyzer/asset-audit/data-builder.ts`)
loads `rmm_evidence` as a 7th LLM context arm:

```
loadRmmEvidence(itglueOrgId, assetType, assetId)
  ├─ map IT Glue org → Autotask company (case-insensitive name)
  ├─ listLatestEvidenceForCompany(companyId, 7)  // site-anchored, last 7 days
  └─ listLatestEvidenceForAsset(assetType, assetId)  // asset-self, all-time
```

The prompt renders a new section:

```
=== LIVE RMM EVIDENCE (most recent successful Overshell runs; AUTHORITATIVE current state) ===
[ {execution_id, script_id, target_type, target_hostname, captured_at, parsed}, … ]
```

System prompt rule (in `prompt.ts`):

> *"Treat parsed contents as authoritative current state … Use it to
> justify suggested values with high confidence — e.g. if Get-Services
> lists 'BartenderProcessService' running on the target and a ticket
> asked about BarTender printing, suggest adding that service name to
> operating_system_notes with confidence=high. Cite execution_id
> alongside ticket numbers in evidence_ticket_numbers."*

Trim priority on overflow: `peer_global → ticket_evidence → rmm_evidence`
(rmm last — highest-value section).

---

## API contract

| Method | Route | Auth | Purpose |
|---|---|---|---|
| GET | `/api/admin/rmm/settings` | admin | Settings + 24h counts |
| PATCH | `/api/admin/rmm/settings` | admin | Update `overshellVariableName` |
| POST | `/api/admin/rmm/settings/discover` | admin | Force component scan |
| GET | `/api/rmm/scripts` | auth | Library catalog (no bodies) |
| GET | `/api/rmm/executions` | auth | List (filters: companyId, scriptId, status, assetType, assetId) |
| POST | `/api/rmm/executions` | rmm.execute | Queue a fresh execution |
| GET | `/api/rmm/executions/[id]` | auth | Single execution detail |
| GET | `/api/analyzer/itglue/sites/[companyId]` | auth | Site discovery summary |

POST body:

```ts
{
  scriptId: string,
  target:
    | { type: 'site_anchor', companyId: number | string }
    | { type: 'asset_self', deviceUid: string, hostname?, companyId?, assetType?, assetId? },
  triggeredByAuditId?: string,
}
```

---

## Safety

### Three-layer audit trail

| Table | What it captures |
|---|---|
| `rmm_executions` | Every dispatch — full lifecycle, raw output (redacted), parsed evidence, target, audit/asset linkage. Forever-retained. |
| `analyzer_cost_audit` | Rate-limit decision row per request — approved / blocked. Same view as LLM cost decisions. |
| Generic `audit_log` | `rmm.execute`, `rmm.execute.complete`, `rmm.execute.failed`, `rmm.execute.timeout`. Surfaces in `/admin/audit-log`. |

### Rate limit

50 executions per user per 24-hour rolling window. Trips before
`runQuickJob` is called; logs the block to `analyzer_cost_audit` with
`decision='blocked'`.

### Hard timeout

5 minutes per execution. The worker sweeps `timeout_at < NOW()` rows on
every tick and marks them `timeout` (with an `error_message` recording
the cause).

### Output redaction

Every `raw_stdout` and `raw_stderr` passes through
`lib/services/analyzer/itglue-redact.ts:redact()` before persistence and
again before the audit prompt sees it. Strips any key matching
`/password|secret|key|token|credential|api[_-]?key/i`. Belt-and-braces
defense even though the curated library has no credential-handling
scripts.

### Registry gating

Only ids in the in-code `SCRIPTS` registry can run. The endpoint rejects
unknown ids with 400 before any Datto API call. There's no UI to paste
ad-hoc PowerShell — everything goes through the typed `RmmScript`
interface.

---

## Operational SQL

### Most-recent successful run per (company, script)

```sql
SELECT DISTINCT ON (target_company_id, script_id)
       target_company_id, script_id, target_hostname,
       completed_at, exit_code
  FROM rmm_executions
 WHERE status = 'complete'
   AND completed_at >= NOW() - INTERVAL '14 days'
 ORDER BY target_company_id, script_id, completed_at DESC;
```

### Stuck (queued/running past timeout)

```sql
SELECT id, script_id, target_hostname, status, queued_at, timeout_at
  FROM rmm_executions
 WHERE status IN ('queued','running')
   AND timeout_at <= NOW()
 ORDER BY queued_at;
```

(The worker should sweep these on the next tick.)

### Per-user daily activity

```sql
SELECT performed_by_user_id,
       COUNT(*) FILTER (WHERE queued_at >= NOW() - INTERVAL '24 hours') AS last_24h,
       COUNT(*) FILTER (WHERE status = 'complete' AND queued_at >= NOW() - INTERVAL '24 hours') AS completed_24h
  FROM rmm_executions
 WHERE performed_by_user_id IS NOT NULL
 GROUP BY performed_by_user_id
 ORDER BY last_24h DESC;
```

### Verify a particular script's parser

```sql
SELECT id, status, parse_error, jsonb_pretty(parsed_evidence) AS parsed
  FROM rmm_executions
 WHERE script_id = 'get-ad-health'
 ORDER BY queued_at DESC
 LIMIT 5;
```

---

## Limitations & non-goals

- **Forward-only capture.** No backfill of historical Datto Overshell
  jobs (per user direction).
- **WNP-only target resolution.** Direct-to-DC role detection is a
  fast-follow if AD scripts that need native DC execution become
  important.
- **Read-only scripts only.** No Overshell-driven configuration changes.
  Apply documentation updates via the IT Glue Phase 4 path.
- **No ad-hoc PowerShell input from the UI.** Registry-listed scripts only.
- **No audit-driven auto-execution in v1.** Admin clicks the button. A
  future version could let the audit panel propose "Run Get-Services to
  fill this gap" with one-click confirm.
- **Credential output never enters the audit prompt.** Three-layer refusal
  at script-library curation, output redaction, and prompt-side rules.

---

## Files

**New:**
- `migrations/077_rmm_overshell.sql`
- `lib/services/rmm/settings.ts`
- `lib/services/rmm/persistence.ts`
- `lib/services/rmm/target-resolver.ts` + test
- `lib/services/rmm/executor.ts`
- `lib/services/rmm/worker.ts` + test
- `lib/services/rmm/scripts/types.ts`
- `lib/services/rmm/scripts/{get-services,get-installed-software,get-event-log-recent,get-ad-health,get-dhcp-scopes,get-dns-zones,get-network-discovery}.ts`
- `lib/services/rmm/scripts/index.ts` + registry test
- `app/api/admin/rmm/settings/route.ts`
- `app/api/admin/rmm/settings/discover/route.ts`
- `app/api/rmm/scripts/route.ts`
- `app/api/rmm/executions/route.ts`
- `app/api/rmm/executions/[id]/route.ts`
- `app/api/analyzer/itglue/sites/[companyId]/route.ts`
- `app/admin/rmm-overshell/page.tsx`
- `app/analyzer/itglue/sites/[companyId]/page.tsx`
- `components/rmm/rmm-script-picker.tsx`
- `components/rmm/rmm-execution-stream.tsx`

**Modified:**
- `lib/permissions.ts` — `rmm: ['read','execute']`
- `lib/services/datto-rmm-client.ts` — `findOvershellComponent`
- `lib/services/analyzer/asset-audit/data-builder.ts` — 7th evidence arm
- `lib/services/analyzer/asset-audit/prompt.ts` — RMM evidence section + system rule
- `app/api/analyzer/itglue/applications/[id]/route.ts` — surface `autotaskCompanyId`
- `app/api/analyzer/itglue/configurations/[id]/route.ts` — surface `dattoDeviceUid` + `autotaskCompanyId`
- `app/analyzer/itglue/applications/[id]/page.tsx` — embedded `<RmmScriptPicker filter='site_anchor'/>`
- `app/analyzer/itglue/configurations/[id]/page.tsx` — embedded `<RmmScriptPicker filter='asset_self'/>`
- `components/navigation/app-navigation.tsx` — Admin → "RMM Overshell"

## Phase 4.3: LogLift transport variant

The same `rmm_executions` table also stores LogLift event-log uploads
(`transport='b2_upload'`). Different transport, same evidence pipeline:

- The Datto RMM-registered LogLift component handles its own PowerShell
  collection + B2 upload. Pulse never sees the script body.
- Pulse dispatches via `runQuickJob` with `RunId`, `ClientId`,
  `WebhookUrl`, `WebhookSecret` variables (same factory + same audit
  trail as Overshell).
- The collector POSTs to `/api/rmm/loglift/upload` when the upload
  finishes. Pulse downloads from B2, slims, and persists.
- The audit-context `rmm_evidence` arm picks the LogLift row up by
  `script_id='loglift-eventlogs'` automatically — same
  `listLatestEvidenceForAsset` join as any other asset-self script.

See `docs/loglift-eventlog-pipeline-spec.md` for the full webhook
contract + slim-shape spec.