wulf-pulse/docs/loglift-eventlog-pipeline-spec.md

LogLift Event-Log Pipeline (Phase 4.3)

End-to-end spec for the LogLift evidence path: a Datto RMM-deployed PowerShell collector captures a Windows endpoint's event logs + system context, gzips the JSON, uploads it to Backblaze B2, then POSTs metadata to Pulse. Pulse downloads the gzip, slims it, persists it as rmm_executions evidence, and (when the hostname uniquely matches an IT Glue Configuration) auto-triggers an asset-first audit.

Why this exists

Phase 4.2 wired Datto RMM Overshell PowerShell evidence into the audit pipeline, but Overshell stdout is capped at ~50KB practical — too small for full event logs. Wulf already runs a richer evidence path through n8n: collector → B2 → n8n decompress + LLM → Telegram. Phase 4.3 makes Pulse the receiver instead of n8n so:

• LogLift evidence lands in the same rmm_executions table.
• The audit pipeline's rmm_evidence arm picks it up automatically.
• Admins can dispatch a LogLift run from the Configuration page (Datto Quick Job into the registered LogLift component).
• Successful uploads matched to a unique IT Glue Configuration auto-fire an asset-first audit so documentation suggestions surface immediately.

Components

┌──────────────────────────┐         ┌────────────────────────┐
│  Windows endpoint        │         │  Datto RMM             │
│  ─ collector PowerShell  │ ◀─────  │  ─ LogLift component   │
│  ─ gzip event logs       │         │    (job dispatched     │
│  ─ upload to B2          │  ────▶  │     by Pulse)          │
│  ─ POST webhook to Pulse │         │                        │
└──────────────────────────┘         └────────────────────────┘
       │                                       ▲
       │ B2 PUT                                │ runQuickJob
       ▼                                       │
┌──────────────────────────┐         ┌────────────────────────┐
│  Backblaze B2            │ ◀─SigV4─│  Pulse                 │
│  bucket: wulf-audits     │         │  /api/rmm/loglift/     │
│  region: us-west-002     │   GET   │     upload             │
│                          │  ────▶  │  ─ download + slim     │
│                          │         │  ─ persist evidence    │
└──────────────────────────┘         │  ─ auto-audit (single  │
                                     │     match only)        │
                                     └────────────────────────┘

Object-key convention

{datto_site_uid}/{computer_name}/eventlogs_{YYYYMMDD_HHMMSS}.json.gz

Pulse rejects anything not matching:

^[A-Za-z0-9_-]+/[A-Za-z0-9_.-]+/eventlogs_[0-9_]+\.json\.gz$

Webhook contract

POST /api/rmm/loglift/upload

Auth header: x-openclaw-key: <OPENCLAW_API_KEY> (constant-time match).

Request body:

{
  "runId": "pulse_a1b2c3_1714672800000",
  "clientId": "f7a8b9c0-…",
  "computerName": "YNGHYNWNP01",
  "deviceUid": "f7a8b9c0-… (optional)",
  "summary": {
    "totalEvents": 4012,
    "criticalEvents": 0,
    "errorCount": 23,
    "warningCount": 187,
    "timeRange": "Last 24 hours"
  },
  "objectKey": "f7a8b9c0/YNGHYNWNP01/eventlogs_20260502_120000.json.gz",
  "collectedAt": "2026-05-02T12:00:03.124Z",
  "rmmContext": {
    "siteName": "Hynes — Youngstown",
    "siteUid": "f7a8b9c0-…",
    "accountUid": "…"
  },
  "issueDescription": "(optional, free-text)",
  "ticketNumber": "T20260502.0019"
}

Response (200):

{
  "executionId": "0193…",
  "matched": {
    "datto_site_id": 42,
    "datto_device_uid": "f7a8b9c0-…",
    "autotask_company_id": "29861375",
    "configuration_id": "12345",
    "configuration_single_match": true
  },
  "parsed": {
    "total_events": 4012,
    "critical_events": 0,
    "error_count": 23,
    "warning_count": 187,
    "time_range": "Last 24 hours"
  },
  "auditId": "0193…"
}

Pulse-driven dispatch

Configuration page → "Run discovery" picker → LogLift entry. The picker filters by target_type='asset_self'. Dispatch path:

1. Resolve the device's Datto site uid (datto_rmm_sites.uid) → use as ClientId.
2. Resolve the LogLift component uid from rmm_settings. If not cached, the executor calls discoverLogliftComponent() which scans the Datto API for components matching /loglift|eventlog/i.
3. Generate runId = pulse_<hex8>_<ms>.
4. Insert rmm_executions row with transport='b2_upload', run_id, status='queued'. The row's variables column stores the non-secret variables (RunId, ClientId, WebhookUrl) — the secret is stripped before persistence.
5. runQuickJob with the LogLift component_uid + variables:
   • RunId — webhook correlation
   • ClientId — Datto site uid
   • WebhookUrl — ${BETTER_AUTH_URL}/api/rmm/loglift/upload
   • WebhookSecret — OPENCLAW_API_KEY
6. On Quick Job ack: flip to running + persist job_uid.
7. Worker skips b2_upload rows during stdout-poll. Webhook is the completion event. The 5-minute timeout sweep still applies — stuck rows get marked timeout.

Out-of-band (collector-driven) ingest

If the LogLift collector fires from its own schedule (e.g. n8n still runs in parallel during cutover), the webhook receiver inserts a fresh rmm_executions row with triggered_by_user_id=NULL, status='running', then completes it in the same handler.

run_id has a unique index; replays of the same upload no-op cleanly.

Slim-evidence shape

Stored in rmm_executions.parsed_evidence (JSONB). The full gzip stays in B2 forever (forensic replay via the presigned-GET helper):

{
  "schema_version": 1,
  "transport": "b2_upload",
  "object_key": "…",
  "collected_at": "…",
  "rmm_context": {…},
  "issue_description": "…",
  "ticket_number": "…",
  "webhook_summary": {…},          // raw counts from the webhook
  "metadata": {…},                  // from inside the gzip
  "system_context": {                // OS, hardware, disks, updates …
    "OS": {…},
    "Hardware": {…},
    "Memory": {…},
    "Disks": [{…}],
    "RecentUpdates": [{…}],
    "Uptime": "…",
    "LastBoot": "…",
    "PendingReboot": false,
    "RebootReasons": []
  },
  "summary": {                       // from inside the gzip
    "TotalEvents": 4012,
    "CriticalEvents": 0,
    "ByLevel": {"Error": 23, "Warning": 187},
    "TimeRange": "Last 24 hours",
    "TopEventIds": [{"Id": 7036, "Count": 145}]
  },
  "top_events": [                    // top 100 by severity then recency
    {"TimeCreated":"…", "LevelDisplayName":"Error", "Id":7036, "Source":"Service Control Manager", "Message":"…"}
  ],
  "event_count_total": 4012,
  "top_events_truncated": true
}

Output runs through redact() before persistence.

Auto-audit hook

When the resolved IT Glue Configuration is single-match (exactly one row matches the company + hostname), Pulse fires runAssetAudit({ assetType: 'configuration', assetId, generatedByUserId: null, provider: 'anthropic' }).

The new audit row appears on the Configuration's audit page with triggered_by_user_id=null. Multi-match Configurations are logged but skipped — auditing the wrong asset is worse than no audit.

audit_log actions:

Action	Resource	When
rmm.loglift.dispatched	datto_device	Pulse-driven Quick Job accepted
rmm.loglift.received	datto_device	Webhook landed + evidence persisted
rmm.loglift.matched	itg_configuration	Configuration matched + single
rmm.loglift.audit_triggered	itg_configuration	Auto-audit completed

Safety + cost guards

• OBJECT_KEY_REGEX — path-traversal guard.
• B2 download cap: 25MB hard.
• Decompress cap: refuse if gzip ISIZE > 100MB (zip-bomb defense), re-checked after inflation.
• redact() runs on the slim payload before storage.
• Auto-audit only on single-match Configurations.
• analyzer_cost_audit records on the dispatch path (rate-limited the same as Overshell). Inbound webhooks are NOT rate-limited — the agent decides cadence; we trust the agent.
• The collector's own B2 credentials never travel through Pulse. Pulse uses its own B2_KEY_ID / B2_APP_KEY to download.

Environment variables

Required for dispatch + receive:

Var	Default	Notes
B2_KEY_ID	—	B2 application key id
B2_APP_KEY	—	B2 application key secret
B2_BUCKET	wulf-audits	matches existing n8n config
B2_REGION	us-west-002
B2_ENDPOINT	s3.us-west-002.backblazeb2.com	no scheme
OPENCLAW_API_KEY	—	webhook auth + collector variable
BETTER_AUTH_URL	—	base URL the collector POSTs to

Verification checklist

1. \d rmm_executions shows transport, evidence_object_key, run_id.
2. B2 client.test.ts passes (SigV4 fixture + path-traversal rejection).
3. Webhook auth: missing/bad x-openclaw-key → 401; good key + valid body → 200.
4. /admin/rmm-overshell → "Re-discover LogLift" populates rmm_settings.loglift_component_uid.
5. Configuration page → "Run discovery" → LogLift selection → row goes running (transport='b2_upload', run_id set).
6. Webhook handler downloads from B2, flips row to complete, parsed_evidence populated.
7. Single-match Configuration → new itglue_asset_audits row with triggered_by_user_id=null.
8. Asset-audit prompt's === LIVE RMM EVIDENCE === block contains the slim LogLift payload (system_context + summary + top_events).
9. Non-admin direct webhook POST without the openclaw key returns 401.

Non-goals

• No Telegram summary / non-technical second LLM pass (notification, not data path).
• No replacement of the existing collector PowerShell. The Datto component is registered by name; the dispatch path passes RunId, ClientId, WebhookUrl, WebhookSecret as variables.
• No bulk-replay of historical B2 objects. Manual replay can be added later via an admin endpoint that takes an objectKey.
• No HMAC signature on the webhook body — x-openclaw-key is the auth boundary. Adding HMAC is a fast follow if we expand external integrations.