272 lines
11 KiB
Markdown
272 lines
11 KiB
Markdown
|
# LogLift Event-Log Pipeline (Phase 4.3)
|
||
|
|
|
||
|
|
End-to-end spec for the LogLift evidence path: a Datto RMM-deployed
|
||
|
|
PowerShell collector captures a Windows endpoint's event logs + system
|
||
|
|
context, gzips the JSON, uploads it to Backblaze B2, then POSTs metadata
|
||
|
|
to Pulse. Pulse downloads the gzip, slims it, persists it as
|
||
|
|
`rmm_executions` evidence, and (when the hostname uniquely matches an IT
|
||
|
|
Glue Configuration) auto-triggers an asset-first audit.
|
||
|
|
|
||
|
|
## Why this exists
|
||
|
|
|
||
|
|
Phase 4.2 wired Datto RMM Overshell PowerShell evidence into the audit
|
||
|
|
pipeline, but Overshell stdout is capped at ~50KB practical — too small
|
||
|
|
for full event logs. Wulf already runs a richer evidence path through
|
||
|
|
n8n: collector → B2 → n8n decompress + LLM → Telegram. Phase 4.3 makes
|
||
|
|
Pulse the receiver instead of n8n so:
|
||
|
|
|
||
|
|
- LogLift evidence lands in the same `rmm_executions` table.
|
||
|
|
- The audit pipeline's `rmm_evidence` arm picks it up automatically.
|
||
|
|
- Admins can dispatch a LogLift run from the Configuration page (Datto
|
||
|
|
Quick Job into the registered LogLift component).
|
||
|
|
- Successful uploads matched to a unique IT Glue Configuration auto-fire
|
||
|
|
an asset-first audit so documentation suggestions surface immediately.
|
||
|
|
|
||
|
|
## Components
|
||
|
|
|
||
|
|
```
|
||
|
|
┌──────────────────────────┐ ┌────────────────────────┐
|
||
|
|
│ Windows endpoint │ │ Datto RMM │
|
||
|
|
│ ─ collector PowerShell │ ◀───── │ ─ LogLift component │
|
||
|
|
│ ─ gzip event logs │ │ (job dispatched │
|
||
|
|
│ ─ upload to B2 │ ────▶ │ by Pulse) │
|
||
|
|
│ ─ POST webhook to Pulse │ │ │
|
||
|
|
└──────────────────────────┘ └────────────────────────┘
|
||
|
|
│ ▲
|
||
|
|
│ B2 PUT │ runQuickJob
|
||
|
|
▼ │
|
||
|
|
┌──────────────────────────┐ ┌────────────────────────┐
|
||
|
|
│ Backblaze B2 │ ◀─SigV4─│ Pulse │
|
||
|
|
│ bucket: wulf-audits │ │ /api/rmm/loglift/ │
|
||
|
|
│ region: us-west-002 │ GET │ upload │
|
||
|
|
│ │ ────▶ │ ─ download + slim │
|
||
|
|
│ │ │ ─ persist evidence │
|
||
|
|
└──────────────────────────┘ │ ─ auto-audit (single │
|
||
|
|
│ match only) │
|
||
|
|
└────────────────────────┘
|
||
|
|
```
|
||
|
|
|
||
|
|
## Object-key convention
|
||
|
|
|
||
|
|
```
|
||
|
|
{datto_site_uid}/{computer_name}/eventlogs_{YYYYMMDD_HHMMSS}.json.gz
|
||
|
|
```
|
||
|
|
|
||
|
|
Pulse rejects anything not matching:
|
||
|
|
|
||
|
|
```
|
||
|
|
^[A-Za-z0-9_-]+/[A-Za-z0-9_.-]+/eventlogs_[0-9_]+\.json\.gz$
|
||
|
|
```
|
||
|
|
|
||
|
|
## Webhook contract
|
||
|
|
|
||
|
|
`POST /api/rmm/loglift/upload`
|
||
|
|
|
||
|
|
Auth header: `x-openclaw-key: <OPENCLAW_API_KEY>` (constant-time match).
|
||
|
|
|
||
|
|
Request body:
|
||
|
|
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"runId": "pulse_a1b2c3_1714672800000",
|
||
|
|
"clientId": "f7a8b9c0-…",
|
||
|
|
"computerName": "YNGHYNWNP01",
|
||
|
|
"deviceUid": "f7a8b9c0-… (optional)",
|
||
|
|
"summary": {
|
||
|
|
"totalEvents": 4012,
|
||
|
|
"criticalEvents": 0,
|
||
|
|
"errorCount": 23,
|
||
|
|
"warningCount": 187,
|
||
|
|
"timeRange": "Last 24 hours"
|
||
|
|
},
|
||
|
|
"objectKey": "f7a8b9c0/YNGHYNWNP01/eventlogs_20260502_120000.json.gz",
|
||
|
|
"collectedAt": "2026-05-02T12:00:03.124Z",
|
||
|
|
"rmmContext": {
|
||
|
|
"siteName": "Hynes — Youngstown",
|
||
|
|
"siteUid": "f7a8b9c0-…",
|
||
|
|
"accountUid": "…"
|
||
|
|
},
|
||
|
|
"issueDescription": "(optional, free-text)",
|
||
|
|
"ticketNumber": "T20260502.0019"
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
Response (200):
|
||
|
|
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"executionId": "0193…",
|
||
|
|
"matched": {
|
||
|
|
"datto_site_id": 42,
|
||
|
|
"datto_device_uid": "f7a8b9c0-…",
|
||
|
|
"autotask_company_id": "29861375",
|
||
|
|
"configuration_id": "12345",
|
||
|
|
"configuration_single_match": true
|
||
|
|
},
|
||
|
|
"parsed": {
|
||
|
|
"total_events": 4012,
|
||
|
|
"critical_events": 0,
|
||
|
|
"error_count": 23,
|
||
|
|
"warning_count": 187,
|
||
|
|
"time_range": "Last 24 hours"
|
||
|
|
},
|
||
|
|
"auditId": "0193…"
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
## Pulse-driven dispatch
|
||
|
|
|
||
|
|
Configuration page → "Run discovery" picker → LogLift entry. The picker
|
||
|
|
filters by `target_type='asset_self'`. Dispatch path:
|
||
|
|
|
||
|
|
1. Resolve the device's Datto site uid (`datto_rmm_sites.uid`) → use as
|
||
|
|
`ClientId`.
|
||
|
|
2. Resolve the LogLift component uid from `rmm_settings`. If not cached,
|
||
|
|
the executor calls `discoverLogliftComponent()` which scans the Datto
|
||
|
|
API for components matching `/loglift|eventlog/i`.
|
||
|
|
3. Generate `runId = pulse_<hex8>_<ms>`.
|
||
|
|
4. Insert `rmm_executions` row with `transport='b2_upload'`, `run_id`,
|
||
|
|
`status='queued'`. The row's `variables` column stores the
|
||
|
|
non-secret variables (RunId, ClientId, WebhookUrl) — the secret is
|
||
|
|
stripped before persistence.
|
||
|
|
5. `runQuickJob` with the LogLift component_uid + variables:
|
||
|
|
- `RunId` — webhook correlation
|
||
|
|
- `ClientId` — Datto site uid
|
||
|
|
- `WebhookUrl` — `${BETTER_AUTH_URL}/api/rmm/loglift/upload`
|
||
|
|
- `WebhookSecret` — `OPENCLAW_API_KEY`
|
||
|
|
6. On Quick Job ack: flip to `running` + persist `job_uid`.
|
||
|
|
7. Worker skips `b2_upload` rows during stdout-poll. Webhook is the
|
||
|
|
completion event. The 5-minute timeout sweep still applies — stuck
|
||
|
|
rows get marked `timeout`.
|
||
|
|
|
||
|
|
## Out-of-band (collector-driven) ingest
|
||
|
|
|
||
|
|
If the LogLift collector fires from its own schedule (e.g. n8n still
|
||
|
|
runs in parallel during cutover), the webhook receiver inserts a fresh
|
||
|
|
`rmm_executions` row with `triggered_by_user_id=NULL`,
|
||
|
|
`status='running'`, then completes it in the same handler.
|
||
|
|
|
||
|
|
`run_id` has a unique index; replays of the same upload no-op cleanly.
|
||
|
|
|
||
|
|
## Slim-evidence shape
|
||
|
|
|
||
|
|
Stored in `rmm_executions.parsed_evidence` (JSONB). The full gzip stays
|
||
|
|
in B2 forever (forensic replay via the presigned-GET helper):
|
||
|
|
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"schema_version": 1,
|
||
|
|
"transport": "b2_upload",
|
||
|
|
"object_key": "…",
|
||
|
|
"collected_at": "…",
|
||
|
|
"rmm_context": {…},
|
||
|
|
"issue_description": "…",
|
||
|
|
"ticket_number": "…",
|
||
|
|
"webhook_summary": {…}, // raw counts from the webhook
|
||
|
|
"metadata": {…}, // from inside the gzip
|
||
|
|
"system_context": { // OS, hardware, disks, updates …
|
||
|
|
"OS": {…},
|
||
|
|
"Hardware": {…},
|
||
|
|
"Memory": {…},
|
||
|
|
"Disks": [{…}],
|
||
|
|
"RecentUpdates": [{…}],
|
||
|
|
"Uptime": "…",
|
||
|
|
"LastBoot": "…",
|
||
|
|
"PendingReboot": false,
|
||
|
|
"RebootReasons": []
|
||
|
|
},
|
||
|
|
"summary": { // from inside the gzip
|
||
|
|
"TotalEvents": 4012,
|
||
|
|
"CriticalEvents": 0,
|
||
|
|
"ByLevel": {"Error": 23, "Warning": 187},
|
||
|
|
"TimeRange": "Last 24 hours",
|
||
|
|
"TopEventIds": [{"Id": 7036, "Count": 145}]
|
||
|
|
},
|
||
|
|
"top_events": [ // top 100 by severity then recency
|
||
|
|
{"TimeCreated":"…", "LevelDisplayName":"Error", "Id":7036, "Source":"Service Control Manager", "Message":"…"}
|
||
|
|
],
|
||
|
|
"event_count_total": 4012,
|
||
|
|
"top_events_truncated": true
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
Output runs through `redact()` before persistence.
|
||
|
|
|
||
|
|
## Auto-audit hook
|
||
|
|
|
||
|
|
When the resolved IT Glue Configuration is **single-match** (exactly
|
||
|
|
one row matches the company + hostname), Pulse fires
|
||
|
|
`runAssetAudit({ assetType: 'configuration', assetId, generatedByUserId: null, provider: 'anthropic' })`.
|
||
|
|
|
||
|
|
The new audit row appears on the Configuration's audit page with
|
||
|
|
`triggered_by_user_id=null`. Multi-match Configurations are logged but
|
||
|
|
skipped — auditing the wrong asset is worse than no audit.
|
||
|
|
|
||
|
|
`audit_log` actions:
|
||
|
|
|
||
|
|
| Action | Resource | When |
|
||
|
|
| ----------------------------- | --------------------- | ------------------------------------------ |
|
||
|
|
| `rmm.loglift.dispatched` | datto_device | Pulse-driven Quick Job accepted |
|
||
|
|
| `rmm.loglift.received` | datto_device | Webhook landed + evidence persisted |
|
||
|
|
| `rmm.loglift.matched` | itg_configuration | Configuration matched + single |
|
||
|
|
| `rmm.loglift.audit_triggered` | itg_configuration | Auto-audit completed |
|
||
|
|
|
||
|
|
## Safety + cost guards
|
||
|
|
|
||
|
|
- `OBJECT_KEY_REGEX` — path-traversal guard.
|
||
|
|
- B2 download cap: 25MB hard.
|
||
|
|
- Decompress cap: refuse if gzip ISIZE > 100MB (zip-bomb defense),
|
||
|
|
re-checked after inflation.
|
||
|
|
- `redact()` runs on the slim payload before storage.
|
||
|
|
- Auto-audit only on single-match Configurations.
|
||
|
|
- `analyzer_cost_audit` records on the dispatch path (rate-limited the
|
||
|
|
same as Overshell). Inbound webhooks are NOT rate-limited — the
|
||
|
|
agent decides cadence; we trust the agent.
|
||
|
|
- The collector's own B2 credentials never travel through Pulse. Pulse
|
||
|
|
uses its own `B2_KEY_ID` / `B2_APP_KEY` to download.
|
||
|
|
|
||
|
|
## Environment variables
|
||
|
|
|
||
|
|
Required for dispatch + receive:
|
||
|
|
|
||
|
|
| Var | Default | Notes |
|
||
|
|
| -------------------- | -------------------------------- | ---------------------------------- |
|
||
|
|
| `B2_KEY_ID` | — | B2 application key id |
|
||
|
|
| `B2_APP_KEY` | — | B2 application key secret |
|
||
|
|
| `B2_BUCKET` | `wulf-audits` | matches existing n8n config |
|
||
|
|
| `B2_REGION` | `us-west-002` | |
|
||
|
|
| `B2_ENDPOINT` | `s3.us-west-002.backblazeb2.com` | no scheme |
|
||
|
|
| `OPENCLAW_API_KEY` | — | webhook auth + collector variable |
|
||
|
|
| `BETTER_AUTH_URL` | — | base URL the collector POSTs to |
|
||
|
|
|
||
|
|
## Verification checklist
|
||
|
|
|
||
|
|
1. `\d rmm_executions` shows `transport`, `evidence_object_key`, `run_id`.
|
||
|
|
2. `B2 client.test.ts` passes (SigV4 fixture + path-traversal rejection).
|
||
|
|
3. Webhook auth: missing/bad `x-openclaw-key` → 401; good key + valid
|
||
|
|
body → 200.
|
||
|
|
4. `/admin/rmm-overshell` → "Re-discover LogLift" populates
|
||
|
|
`rmm_settings.loglift_component_uid`.
|
||
|
|
5. Configuration page → "Run discovery" → LogLift selection → row goes
|
||
|
|
`running` (`transport='b2_upload'`, `run_id` set).
|
||
|
|
6. Webhook handler downloads from B2, flips row to `complete`,
|
||
|
|
`parsed_evidence` populated.
|
||
|
|
7. Single-match Configuration → new `itglue_asset_audits` row with
|
||
|
|
`triggered_by_user_id=null`.
|
||
|
|
8. Asset-audit prompt's `=== LIVE RMM EVIDENCE ===` block contains the
|
||
|
|
slim LogLift payload (system_context + summary + top_events).
|
||
|
|
9. Non-admin direct webhook POST without the openclaw key returns 401.
|
||
|
|
|
||
|
|
## Non-goals
|
||
|
|
|
||
|
|
- No Telegram summary / non-technical second LLM pass (notification,
|
||
|
|
not data path).
|
||
|
|
- No replacement of the existing collector PowerShell. The Datto
|
||
|
|
component is registered by name; the dispatch path passes `RunId`,
|
||
|
|
`ClientId`, `WebhookUrl`, `WebhookSecret` as variables.
|
||
|
|
- No bulk-replay of historical B2 objects. Manual replay can be added
|
||
|
|
later via an admin endpoint that takes an `objectKey`.
|
||
|
|
- No HMAC signature on the webhook body — `x-openclaw-key` is the auth
|
||
|
|
boundary. Adding HMAC is a fast follow if we expand external
|
||
|
|
integrations.
|