feat: RMM Overshell, IT Glue audit/write-back, LogLift, link-aware bundles, dashboard overhaul
- RMM Overshell (migration 077): admin page, dispatch UI, executor/worker, target resolver, script registry (AD/DHCP/DNS/event-log/services/software/network/loglift) - LogLift evidence pipeline (migration 078): upload webhook, B2 storage client, receiver/matcher, EventLogCollector PowerShell script - IT Glue audit + write-back (migrations 075, 076): asset-audit runner, ticket xrefs, applications/configurations browse pages + apply/revert/audit endpoints - Link-aware analyzer bundles (migration 073) + provider toggle (migration 074): link-discovery service, OpenRouter LLM provider, related-tickets/itglue-suggestion panels, analyze-bundle endpoint - Endpoint data model + device-link reconciliation (migrations 079, 080): conflicts admin page, reconciler service, resolve endpoints - Dashboard overhaul: integration-health service + alerts, overview/health endpoints - Permissions: add itglue + rmm scopes; middleware: public /api/rmm/loglift route Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
378e68ad8a
commit
1112a06afe
132 changed files with 21352 additions and 743 deletions
362
docs/LogLift Review.json
Normal file
362
docs/LogLift Review.json
Normal file
File diff suppressed because one or more lines are too long
604
docs/itglue-asset-audit-spec.md
Normal file
604
docs/itglue-asset-audit-spec.md
Normal file
|
|
@ -0,0 +1,604 @@
|
|||
# Feature: IT Glue Asset Audit & Documentation Write-back
|
||||
|
||||
**Status:** shipped (Phases 4 + 4.1 of the AI Ticket Analyzer)
|
||||
**Migrations:** `075_itglue_audit.sql`, `076_itglue_ticket_xrefs.sql`
|
||||
**Build notes:** `docs/wulf-pulse-ticket-analyzer-build-notes.md` → Phase 4 + 4.1
|
||||
**Operator runbook:** `docs/wulf-pulse-ticket-analyzer-runbook.md` → "IT Glue asset audits"
|
||||
|
||||
---
|
||||
|
||||
## What it is, in one paragraph
|
||||
|
||||
For each IT Glue **flexible-asset Application record** *and* **Configuration
|
||||
record** (server, workstation, network device), Pulse runs an LLM audit
|
||||
that compares the record's current contents against (a) the field schema
|
||||
with hints, (b) other well-filled records at the same client, (c) recent
|
||||
ticket history that mentions the asset. It surfaces **field gaps** ("Wulf
|
||||
Application Champion is empty — Jake Hammel is the de-facto SME per
|
||||
T20260502.0033"), **note promotions** (free text in Notes that belongs in a
|
||||
structured field), and **contradictions** (Notes say "2-3 VMs" but
|
||||
Application-on-Device tags only 1). Admins can apply a suggestion with one
|
||||
click — Pulse PATCHes IT Glue and records every change with full
|
||||
before/after diff and revert capability.
|
||||
|
||||
Two complementary entry points:
|
||||
|
||||
- **Asset-first** (Phase 4) — admin browses lowest-scoring Application or
|
||||
Configuration records and runs an audit against all-time history. Useful
|
||||
for periodic backlog sweeps.
|
||||
- **Ticket-first** (Phase 4.1) — every analyzed ticket gets a "Check IT
|
||||
Glue documentation" button on its analysis page. Click → matched IT Glue
|
||||
records appear → audit per record uses *just this ticket's evidence*.
|
||||
Findings frame as "what did this ticket teach us that the documentation
|
||||
doesn't say?" Drives a forward-only **cross-reference index**
|
||||
(`itglue_ticket_xrefs`) of which tickets referenced or updated which
|
||||
documentation, which is also the lookup index for a future RAG
|
||||
automation.
|
||||
|
||||
---
|
||||
|
||||
## Why we built it
|
||||
|
||||
The analyzer already extracts `documentation_gaps_observed` per ticket
|
||||
(Phase 2.4 fingerprint). Until Phase 4 nothing acted on the signal. The
|
||||
catalyzing example was T20260502.0033 (Hynes — tags not printing from
|
||||
Simple Shop Floor), resolved by Collin + Jake identifying a stopped Windows
|
||||
service on MISYS-SQL processing BarTender scan-folder text files. The IT
|
||||
Glue Application record `MISYS 6.3` had **5/17 fields filled**, missing:
|
||||
|
||||
- Wulf Application Champion (Jake is the SME, but only by tribal knowledge)
|
||||
- Vendor Maintenance/Support (escalation contact "Steve Cianflone" buried
|
||||
in free-text Notes)
|
||||
- The integration architecture (Simple Shop Floor → MISYS-SQL → BarTender
|
||||
scan folder)
|
||||
- The named services on each VM
|
||||
|
||||
The next tech who hits this issue would re-discover everything. Phase 4
|
||||
turns the analyzer's passive "documentation_gaps_observed" output into an
|
||||
actionable backlog with a write-back path.
|
||||
|
||||
---
|
||||
|
||||
## User-facing flows
|
||||
|
||||
### List view — `/analyzer/itglue/applications`
|
||||
|
||||
Shows every Application record across all clients, sorted by **lowest audit
|
||||
score first** (so unaudited and worst-scored assets bubble up). Each row:
|
||||
asset name, client, populated-field count, last-audit timestamp +
|
||||
provider, score badge.
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────────┐
|
||||
│ MISYS 6.3 ▌ 55% │
|
||||
│ Hynes Industries · 5 fields populated · last audited today (Claude)│
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Detail view — `/analyzer/itglue/applications/[id]`
|
||||
|
||||
Three regions:
|
||||
|
||||
1. **Header** — asset name, client, score badge, "Open in IT Glue" link, the
|
||||
`<ProviderToggle>` (Claude vs DeepSeek), and a primary **Run audit** button.
|
||||
2. **Audit findings** (visible after a run) —
|
||||
- **Field gaps** — severity-toned cards (red/amber/blue by confidence),
|
||||
each with: field name, why-it-matters, suggested value (when the LLM
|
||||
has evidence), evidence ticket links, and an **Apply** button.
|
||||
- **Promote from Notes** — quoted substring → target field → suggested
|
||||
structured value, with **Apply**.
|
||||
- **Contradictions** — pure observations (no Apply, by design).
|
||||
3. **Current fields** — every field rendered in IT Glue's order; populated
|
||||
shown normally, empty shown muted with the field's hint inline as
|
||||
guidance.
|
||||
4. **Write history** — every Apply/Revert for this asset, with inline
|
||||
before/after diffs and a **Revert** button on committed writes.
|
||||
5. **Audit history** — score over time across runs.
|
||||
|
||||
### Admin view — `/admin/itglue-writes`
|
||||
|
||||
Cross-asset write log, status filters (pending / committed / failed /
|
||||
reverted), full diff per row, ordered most-recent-first.
|
||||
|
||||
---
|
||||
|
||||
## Permissions
|
||||
|
||||
| Action | Permission |
|
||||
|---|---|
|
||||
| Read audit / View asset detail | `requireAuth()` (any signed-in user) |
|
||||
| Run a fresh audit | `requireAuth()` |
|
||||
| Apply a suggestion (PATCH IT Glue) | `requirePermission('itglue', 'write')` |
|
||||
| Revert a previous write | `requirePermission('itglue', 'write')` |
|
||||
| `/admin/itglue-writes` cross-asset feed | `requirePermission('admin', 'access')` |
|
||||
|
||||
`itglue.write` is granted to `admin` and `super-admin` roles only (see
|
||||
`lib/permissions.ts`). The Apply / Revert buttons render on the page for
|
||||
non-admins but are **disabled** with a tooltip explaining why.
|
||||
|
||||
---
|
||||
|
||||
## Pipeline
|
||||
|
||||
Single LLM call, provider-aware via `stageModelsFor(provider).deep_analysis`
|
||||
(Claude Sonnet for Anthropic; DeepSeek V4 Pro for OpenRouter).
|
||||
|
||||
### Inputs (six context arms)
|
||||
|
||||
1. **Asset snapshot** — the asset's current `traits` JSONB, redacted via
|
||||
`lib/services/analyzer/itglue-redact.ts` to strip any password/secret/
|
||||
key/token/credential-keyed values.
|
||||
2. **Field schema with hints** — every field on the asset type, in IT
|
||||
Glue's display order, with the `hint` text IT Glue surfaces in its own
|
||||
editor (e.g. *"Tag any application servers or Devices it's installed
|
||||
on"*).
|
||||
3. **Peer exemplars from same client** — top 5 most-filled flexible
|
||||
assets of the same type at the same client (ranked by populated trait
|
||||
keys), redacted.
|
||||
4. **Best-in-class peers across all clients** — top 3 most-filled
|
||||
instances of the same asset type globally, redacted.
|
||||
5. **Per-field fill-rate stats** — the % of (a) this client's assets and
|
||||
(b) all assets of this type that have field X populated. Pure SQL
|
||||
aggregate; no token cost. Lets the LLM rank gaps by "unusual to be
|
||||
missing" vs "usually missing anyway."
|
||||
6. **Ticket evidence** — up to 20 most-recent complete analyses for the
|
||||
client whose summary or fingerprint mentions the asset name, with
|
||||
their `aggregate_fingerprint` payload included in full.
|
||||
|
||||
### Output schema (`AssetAuditResponse` in `lib/types/analyzer.ts`)
|
||||
|
||||
```ts
|
||||
{
|
||||
field_gaps: Array<{
|
||||
field_name: string, // must exist on the asset type
|
||||
why_missing_matters: string,
|
||||
suggested_value: string | null, // null if no concrete evidence
|
||||
evidence_ticket_numbers: string[],
|
||||
confidence: 'high' | 'medium' | 'low',
|
||||
}>,
|
||||
notes_promotions: Array<{
|
||||
quoted_note_text: string, // exact substring of Notes
|
||||
target_field: string,
|
||||
suggested_value: string,
|
||||
confidence: 'high' | 'medium' | 'low',
|
||||
}>,
|
||||
contradictions: Array<{
|
||||
description: string,
|
||||
evidence: string,
|
||||
}>,
|
||||
overall_score: number, // 0..1 self-rated completeness
|
||||
}
|
||||
```
|
||||
|
||||
### System-prompt rules (highlights)
|
||||
|
||||
- Never invent a field name not present in the schema.
|
||||
- Never suggest a value you cannot point to evidence for — use `null`
|
||||
instead.
|
||||
- Don't suggest password/secret/key/token/credential-shaped fields. (The
|
||||
Apply endpoint also blocks these defensively.)
|
||||
- Use fill-rate stats: a gap that's empty here but populated >80%
|
||||
elsewhere is a stronger gap than one that's empty 80% of the time
|
||||
globally.
|
||||
|
||||
### Pipeline files
|
||||
|
||||
- `lib/services/analyzer/asset-audit/data-builder.ts` — collects all six
|
||||
inputs.
|
||||
- `lib/services/analyzer/asset-audit/prompt.ts` — system prompt + payload
|
||||
builder (with 80KB cap; trims peer_global first, oldest tickets next).
|
||||
- `lib/services/analyzer/asset-audit/runner.ts` — single `callLLMStage`
|
||||
call, persistence on success or failure.
|
||||
- `lib/services/analyzer/asset-audit/persistence.ts` — typed read/write of
|
||||
both DB tables.
|
||||
|
||||
---
|
||||
|
||||
## Data model
|
||||
|
||||
### `itglue_asset_audits` (one row per audit run)
|
||||
|
||||
| Column | Notes |
|
||||
|---|---|
|
||||
| `id` UUID | PK |
|
||||
| `asset_type` TEXT | Currently `'flexible_asset'` only |
|
||||
| `asset_id` BIGINT | IT Glue resource id |
|
||||
| `asset_type_id`, `organization_id` BIGINT | Denormalized for fast filters |
|
||||
| `generated_by_user_id` TEXT | FK `user(id)`, nullable on user delete |
|
||||
| `generated_at` TIMESTAMPTZ | |
|
||||
| `provider`, `model_used` | Which LLM produced the analysis |
|
||||
| `asset_snapshot` JSONB | Redacted traits at audit time |
|
||||
| `ticket_count` INT | How many fingerprints fed in |
|
||||
| `field_gaps`, `notes_promotions`, `contradictions` JSONB | LLM output |
|
||||
| `overall_score` NUMERIC(3,2) | |
|
||||
| `estimated_cost_usd`, `total_input_tokens`, `total_output_tokens` | Cost telemetry |
|
||||
| `status` | `pending` / `running` / `complete` / `failed` |
|
||||
| `error_message` TEXT | |
|
||||
|
||||
### `itglue_writes` (one row per write attempt)
|
||||
|
||||
| Column | Notes |
|
||||
|---|---|
|
||||
| `id` UUID | PK |
|
||||
| `audit_id` UUID | FK to the audit that prompted the change (nullable) |
|
||||
| `asset_type`, `asset_id` | What was written |
|
||||
| `field_name` TEXT | Human field name (mapped to trait key on apply) |
|
||||
| `before_value`, `after_value` JSONB | Pre/post diff |
|
||||
| `performed_by_user_id` TEXT | FK `user(id)` |
|
||||
| `performed_at` TIMESTAMPTZ | |
|
||||
| `status` | `pending` → `committed` / `failed` / `reverted` |
|
||||
| `itglue_response` JSONB | Raw API response for forensics |
|
||||
| `error_message` TEXT | |
|
||||
| `source_evidence` JSONB | `{ ticket_numbers, gap_description }` or `{ reverts_write_id }` |
|
||||
|
||||
### Generic `audit_log` (existing — also written for every change)
|
||||
|
||||
`audit.log()` is called on every successful Apply/Revert with action
|
||||
`itglue.write` or `itglue.revert`, resource `flexible_asset`, resourceId =
|
||||
asset_id, and details = `{ field_name, before, after, audit_id }`. Surfaces
|
||||
in `/admin/audit-log` next to every other admin action.
|
||||
|
||||
---
|
||||
|
||||
## API contract
|
||||
|
||||
All routes under `/api/analyzer/itglue/`.
|
||||
|
||||
### `GET /applications`
|
||||
List all Application records joined to their latest audit.
|
||||
Returns: `{ applications: [{ id, name, organizationId, organizationName, traitCount, latestAudit }] }`.
|
||||
|
||||
### `GET /applications/[id]`
|
||||
Asset detail + field schema for rendering.
|
||||
Returns: `{ asset, fields }`.
|
||||
|
||||
### `GET /applications/[id]/audit?history=1`
|
||||
Latest audit (and optionally history of last 20).
|
||||
Returns: `{ audit: AssetAuditRow | null, history?: AssetAuditRow[] }`.
|
||||
|
||||
### `POST /applications/[id]/audit`
|
||||
Body: `{ provider?: 'anthropic' | 'openrouter' }`. Runs a fresh audit.
|
||||
Cost-guard records `action='itglue_audit'` in `analyzer_cost_audit`.
|
||||
Returns: `{ audit: AssetAuditRow }`.
|
||||
|
||||
### `POST /applications/[id]/apply` *(admin)*
|
||||
Body: `{ auditId, fieldName, suggestedValue, sourceEvidence? }`.
|
||||
Inserts pending `itglue_writes` row → calls `updateFlexibleAsset` on the IT
|
||||
Glue client → marks committed/failed → refreshes the local mirror →
|
||||
writes generic `audit_log`.
|
||||
Returns: `{ writeId, status: 'committed', asset }` or 502 on IT Glue error.
|
||||
|
||||
### `POST /applications/[id]/revert/[writeId]` *(admin)*
|
||||
Re-applies the original `before_value`. Inserts a new write row with
|
||||
swapped before/after; original row → `status='reverted'`.
|
||||
Returns: `{ writeId, revertedWriteId, status: 'committed', asset }`.
|
||||
|
||||
### `GET /applications/[id]/writes`
|
||||
Per-asset write history.
|
||||
Returns: `{ writes: AssetWriteRow[] }`.
|
||||
|
||||
### `GET /writes` *(admin)*
|
||||
Cross-asset write log with status filter.
|
||||
Query: `?status=committed&limit=100&offset=0`.
|
||||
Returns: `{ writes: AssetWriteRow[] }`.
|
||||
|
||||
---
|
||||
|
||||
## Safeguards
|
||||
|
||||
### Three-layer credential refusal
|
||||
|
||||
1. **Prompt** — system prompt instructs the LLM not to suggest password,
|
||||
secret, key, token, or credential fields.
|
||||
2. **Redaction** — `redact()` from `lib/services/analyzer/itglue-redact.ts`
|
||||
strips matching keys from the asset snapshot, peer traits, and all
|
||||
payloads before they reach the LLM.
|
||||
3. **Endpoint** — `POST /apply` regex-blocks any `field_name` matching
|
||||
`/(password|secret|key|token|credential)/i` and returns 400 even if a
|
||||
compromised payload made it through the first two layers.
|
||||
|
||||
### Three-layer audit trail
|
||||
|
||||
| Layer | Captures | Retention |
|
||||
|---|---|---|
|
||||
| `itglue_asset_audits` | Every audit run with full LLM context (asset snapshot, model used, cost, ticket count) | Forever |
|
||||
| `itglue_writes` | Every PATCH attempt with before/after diff, status (pending → committed \| failed \| reverted), audit provenance, raw IT Glue API response, source_evidence | Forever |
|
||||
| `audit_log` (generic) | One row per Apply/Revert in the format admins are used to | Per existing policy |
|
||||
|
||||
### Revert chain semantics
|
||||
|
||||
A revert produces a **new** `itglue_writes` row whose `before_value` and
|
||||
`after_value` are swapped from the original. The new row carries
|
||||
`source_evidence = { reverts_write_id }`. The original row's status flips
|
||||
to `'reverted'`. To trace any state, walk the `audit_id` and `reverts_write_id`
|
||||
graph — it's always complete.
|
||||
|
||||
### Per-record sync after every write
|
||||
|
||||
`refreshFlexibleAssetById()` runs after a successful PATCH (or a
|
||||
successful revert) so `itg_flexible_assets.traits` reflects the change
|
||||
immediately. The route also returns the freshly-PATCH'd asset in the
|
||||
response so the UI can update without waiting for the mirror.
|
||||
|
||||
---
|
||||
|
||||
## Cost & latency
|
||||
|
||||
| Provider | Per-audit cost (typical) | Per-audit latency |
|
||||
|---|---|---|
|
||||
| Anthropic (Sonnet) | ~$0.10 | 30–60s |
|
||||
| OpenRouter (DeepSeek V4 Pro) | ~$0.01 | 60–180s |
|
||||
|
||||
Apply is one IT Glue PATCH + one local upsert + one audit_log insert —
|
||||
typically < 1s.
|
||||
|
||||
A full audit + 5 applies on a typical Application record: ~$0.10 (Claude)
|
||||
or ~$0.01 (DeepSeek), 3–5 minutes including click-through.
|
||||
|
||||
---
|
||||
|
||||
## Limitations & non-goals
|
||||
|
||||
- **Applications only in v1.** The schema generalizes (`asset_type` is a
|
||||
column), but Configurations / Procedures / Domains / Passwords each have
|
||||
different prompt nuances. Adding more asset types is straightforward —
|
||||
duplicate the data-builder and prompt files, add a new asset detail
|
||||
page.
|
||||
- **No bulk-apply.** Admin clicks each suggestion individually. If a
|
||||
single audit produces 10 gaps that's 10 clicks. Bulk-apply is an easy
|
||||
fast-follow once we trust quality.
|
||||
- **No two-step approval workflow.** Admin-direct write per the design
|
||||
decision; the audit log is the safety net. The existing `approval_requests`
|
||||
table from the pipeline engine is available if we later want to require
|
||||
tech-proposes / admin-approves.
|
||||
- **No auto-create of new asset records.** Apply only updates existing
|
||||
assets. If an audit reveals a totally missing record, the suggestion is
|
||||
shown but Apply is disabled with an explainer; admins create the stub
|
||||
manually in IT Glue.
|
||||
- **No inline editor for suggested values.** Admin sees the LLM's
|
||||
suggestion verbatim; if they want to tweak, they edit the value in IT
|
||||
Glue afterward. v2 could add an inline editor.
|
||||
- **No password / secret / key / token / credential writes through this
|
||||
surface — ever.** Refused at three layers (above).
|
||||
|
||||
---
|
||||
|
||||
## Inspecting state (operational SQL)
|
||||
|
||||
### Audits, lowest score first
|
||||
|
||||
```sql
|
||||
SELECT a.asset_id,
|
||||
fa.name AS application_name, fa.organization_name,
|
||||
a.overall_score,
|
||||
jsonb_array_length(a.field_gaps) AS gap_count,
|
||||
jsonb_array_length(a.notes_promotions) AS promo_count,
|
||||
a.provider, a.estimated_cost_usd,
|
||||
a.generated_at
|
||||
FROM itglue_asset_audits a
|
||||
JOIN itg_flexible_assets fa ON fa.id = a.asset_id::bigint
|
||||
WHERE a.status = 'complete'
|
||||
ORDER BY a.generated_at DESC, a.overall_score ASC;
|
||||
```
|
||||
|
||||
### Writes in the last 7 days
|
||||
|
||||
```sql
|
||||
SELECT performed_at, field_name, status,
|
||||
before_value, after_value,
|
||||
performed_by_user_id, audit_id
|
||||
FROM itglue_writes
|
||||
WHERE performed_at >= NOW() - INTERVAL '7 days'
|
||||
ORDER BY performed_at DESC;
|
||||
```
|
||||
|
||||
### Find what's still waiting on which write
|
||||
|
||||
```sql
|
||||
SELECT id, field_name, before_value, after_value,
|
||||
source_evidence ->> 'reverts_write_id' AS reverts_id,
|
||||
status, performed_at
|
||||
FROM itglue_writes
|
||||
WHERE asset_id = '17096940'
|
||||
ORDER BY performed_at;
|
||||
```
|
||||
|
||||
### Audit cost-guard decisions
|
||||
|
||||
```sql
|
||||
SELECT created_at, user_id, action, estimated_cost,
|
||||
decision, decision_reason
|
||||
FROM analyzer_cost_audit
|
||||
WHERE action = 'itglue_audit'
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 50;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How a typical session looks
|
||||
|
||||
1. Tech opens `/analyzer/itglue/applications`. MISYS 6.3 is at the top with
|
||||
no audit yet.
|
||||
2. Tech clicks the row → lands on the detail page. They see 12/17 fields
|
||||
are empty.
|
||||
3. Tech clicks **Run audit** (DeepSeek selected for cost). 60–180s later,
|
||||
the audit panel populates: 4 high-confidence field gaps, 1 notes
|
||||
promotion, 1 contradiction, score 0.55.
|
||||
4. Tech (admin) clicks **Apply** on `Wulf Application Champion` →
|
||||
suggested value "Jake Hammel". A PATCH lands on IT Glue, the local
|
||||
mirror refreshes, the field shows the new value, a write row appears
|
||||
in the History section.
|
||||
5. Tech clicks **Apply** on the Vendor Maintenance/Support promotion
|
||||
("Steve Cianflone" extracted from Notes).
|
||||
6. The next time someone opens this asset in IT Glue, the documentation
|
||||
reflects what tickets have been telling us all along.
|
||||
|
||||
---
|
||||
|
||||
## Files (quick reference)
|
||||
|
||||
**Schema**: `migrations/075_itglue_audit.sql`
|
||||
|
||||
**Pipeline**: `lib/services/analyzer/asset-audit/`
|
||||
- `data-builder.ts`, `prompt.ts`, `runner.ts`, `persistence.ts`, `runner.test.ts`
|
||||
|
||||
**API**: `app/api/analyzer/itglue/applications/`
|
||||
- `route.ts` (list)
|
||||
- `[id]/route.ts` (detail)
|
||||
- `[id]/audit/route.ts` (GET/POST audit)
|
||||
- `[id]/apply/route.ts` (admin write)
|
||||
- `[id]/revert/[writeId]/route.ts` (admin revert)
|
||||
- `[id]/writes/route.ts` (per-asset history)
|
||||
|
||||
Plus `app/api/analyzer/itglue/writes/route.ts` (admin cross-asset feed).
|
||||
|
||||
**UI**:
|
||||
- `app/analyzer/itglue/applications/page.tsx` (list)
|
||||
- `app/analyzer/itglue/applications/[id]/page.tsx` (detail)
|
||||
- `app/admin/itglue-writes/page.tsx` (admin cross-asset)
|
||||
|
||||
**Modified**:
|
||||
- `lib/services/itglue-client.ts` — `updateFlexibleAsset`, `refreshFlexibleAsset`,
|
||||
`getRawSingle`, `isITGlueConfigured`, internal `patch`
|
||||
- `lib/services/itglue-sync-service.ts` — `refreshFlexibleAssetById`
|
||||
- `lib/permissions.ts` — `itglue: ['read', 'write']`
|
||||
- `lib/types/analyzer.ts` — `AssetAuditResponse` schema + request types
|
||||
- `components/navigation/app-navigation.tsx` — nav entry
|
||||
|
||||
---
|
||||
|
||||
## Phase 4.1 additions
|
||||
|
||||
### Ticket-first capture flow
|
||||
|
||||
Trigger: opt-in. The user clicks **"Check IT Glue documentation"** on an
|
||||
analysis detail page (`/analyzer/analysis/[id]`).
|
||||
|
||||
Pipeline:
|
||||
1. `GET /api/analyzer/analyses/[id]/itglue-suggestions` — runs
|
||||
`matchAssetsForAnalysis(id)` and returns matched flexible_assets +
|
||||
configurations (top 5 each by score) along with any existing
|
||||
ticket-scoped audits.
|
||||
2. User clicks **"Audit for this ticket"** on a matched asset → `POST` to
|
||||
the same endpoint with `{ assetType, assetId, provider }`.
|
||||
3. The runner builds context with `ticketScopeAnalysisId` set, so ticket
|
||||
evidence is exactly the one analysis the user clicked from. The system
|
||||
prompt picks up a ticket-scoped suffix instructing the LLM to frame
|
||||
findings as "what *this ticket* taught us."
|
||||
4. The audit row is persisted with `triggered_by_ticket_number` +
|
||||
`triggered_by_analysis_id` populated.
|
||||
5. Apply works from the same per-asset routes as the asset-first flow.
|
||||
When the audit is ticket-scoped, the resulting `itglue_writes` row also
|
||||
carries `triggered_by_ticket_number`, and the apply path inserts an
|
||||
xref row with `relationship='updated'`.
|
||||
|
||||
Asset matching is loose: substring + word-boundary match between the
|
||||
ticket's `aggregate_fingerprint.{applications_involved, device_classes,
|
||||
vendors_involved}` and `itg_flexible_assets.name` (Application type only)
|
||||
+ `itg_configurations.{name, hostname}` for the same client. Score 3
|
||||
(exact) > 2 (word-boundary) > 1 (substring). Top 5 per kind.
|
||||
|
||||
### Configuration support
|
||||
|
||||
Same audit/apply/revert pattern as Applications, with these differences:
|
||||
|
||||
- **Flat schema** — Configurations have ~17 editable top-level columns
|
||||
rather than a `traits` JSONB blob. The data-builder synthesizes a
|
||||
trait-style map for prompt consistency, then the apply route maps
|
||||
field name → IT Glue dash-case attribute (`primary_ip` →
|
||||
`'primary-ip'`).
|
||||
- **Hand-curated field hints** — IT Glue Configurations don't expose a
|
||||
`_fields` table; the 16 hand-written hints live in
|
||||
`lib/services/analyzer/asset-audit/data-builder.ts`
|
||||
(`CONFIGURATION_FIELDS` constant). Each hint is the per-field
|
||||
documentation the LLM sees.
|
||||
- **Configuration-flavored prompt** — focuses on hostname/FQDN
|
||||
consistency, OS version currency, named services capture (in
|
||||
`operating_system_notes`), IP/MAC hygiene, contact ownership.
|
||||
- **Column allowlist on apply** — even though the audit's output schema
|
||||
doesn't restrict `field_name`, the configuration apply route refuses
|
||||
anything outside `name | hostname | primary_ip | mac_address |
|
||||
serial_number | asset_tag | position | notes |
|
||||
operating_system_notes`. FK-shaped fields (manufacturer_id, model_id,
|
||||
operating_system_id, contact_id, location_id) are read-only via this
|
||||
surface for v1.
|
||||
- **Per-record sync helper** — `refreshConfigurationById(id)` mirrors
|
||||
`refreshFlexibleAssetById(id)`; called after every successful Apply.
|
||||
|
||||
### Cross-reference index — `itglue_ticket_xrefs`
|
||||
|
||||
```
|
||||
ticket_number | analysis_id | asset_type | asset_id | relationship | source | confidence | details | created_at
|
||||
```
|
||||
|
||||
Three relationship types:
|
||||
|
||||
| Relationship | Meaning | Source |
|
||||
|---|---|---|
|
||||
| `referenced` | The analyzer cited this asset/doc when analyzing the ticket. Comes from `analyzer_analyses.itglue_docs_referenced`. | `analyzer_referenced` |
|
||||
| `updated` | A ticket-driven audit produced a write on this asset. | `audit_write` |
|
||||
| `should_have_referenced` | Reserved — gap text suggests we should have found this asset/doc but didn't. Not auto-populated yet. | `manual` (future) |
|
||||
|
||||
Populated by:
|
||||
- **Post-analysis hook** in `lib/services/analyzer/worker.ts` —
|
||||
`insertReferencedXrefsFromAnalysis` runs after every successful
|
||||
analysis insertion. Maps each `ITGlueDocReference.doc_type` →
|
||||
asset_type. Best-effort.
|
||||
- **Post-apply hook** in the apply routes — `insertUpdatedXref` runs
|
||||
after a successful PATCH if the audit was ticket-scoped. Best-effort.
|
||||
- **No backfill** — table fills forward. The unique index
|
||||
`(ticket_number, analysis_id, asset_type, asset_id, relationship,
|
||||
source)` makes ingestion idempotent.
|
||||
|
||||
Two views consume it:
|
||||
|
||||
1. **Asset detail page** — "Tickets that touched this asset" section
|
||||
under the audit panel, with sub-sections for `Referenced by` (with
|
||||
the LLM's relevance reason) and `Updated by` (with the field name and
|
||||
write history link).
|
||||
2. **`GET /api/analyzer/tickets/[ticketNumber]/itglue-xrefs`** —
|
||||
ticket-side view; available for future ticket-page surfacing.
|
||||
|
||||
This table is also the lookup index a future RAG automation will use:
|
||||
given a new ticket's entities, fetch historically-referenced docs as
|
||||
warm candidates for the analyzer's IT Glue retrieval stage.
|
||||
|
||||
### Ticket-linkage columns
|
||||
|
||||
Both audit and write tables carry denormalized ticket linkage:
|
||||
|
||||
- `itglue_asset_audits.triggered_by_ticket_number` (and `triggered_by_analysis_id`)
|
||||
— set on ticket-scoped audits. Null on asset-first audits.
|
||||
- `itglue_writes.triggered_by_ticket_number` — copied forward from the
|
||||
audit on Apply, so "every write a given ticket drove" is a one-query
|
||||
lookup. Null when the audit was asset-first.
|
||||
|
||||
### Files added in 4.1
|
||||
|
||||
- `migrations/076_itglue_ticket_xrefs.sql`
|
||||
- `lib/services/analyzer/asset-audit/xrefs.ts`
|
||||
- `lib/services/analyzer/asset-audit/asset-matcher.ts`
|
||||
- `app/api/analyzer/analyses/[id]/itglue-suggestions/route.ts`
|
||||
- Configuration parallel route tree under `app/api/analyzer/itglue/configurations/`
|
||||
- xref endpoints: `app/api/analyzer/itglue/applications/[id]/xrefs/route.ts`,
|
||||
`app/api/analyzer/itglue/configurations/[id]/xrefs/route.ts`,
|
||||
`app/api/analyzer/tickets/[ticketNumber]/itglue-xrefs/route.ts`
|
||||
- `app/analyzer/itglue/configurations/page.tsx` (list)
|
||||
- `app/analyzer/itglue/configurations/[id]/page.tsx` (detail)
|
||||
- `components/analyzer/itglue-suggestions-panel.tsx`
|
||||
|
||||
### Files modified in 4.1
|
||||
|
||||
- `lib/services/itglue-client.ts` — `updateConfiguration`, `refreshConfiguration`
|
||||
- `lib/services/itglue-sync-service.ts` — `refreshConfigurationById`
|
||||
- `lib/services/analyzer/asset-audit/data-builder.ts` — assetType dispatch + ticket-scope mode + Configuration field schema
|
||||
- `lib/services/analyzer/asset-audit/prompt.ts` — Configuration prompt + ticket-scoped suffix + assetType-aware payload
|
||||
- `lib/services/analyzer/asset-audit/runner.ts` — accepts `assetType` and `ticketScopeAnalysisId`; persists triggered_by_*
|
||||
- `lib/services/analyzer/asset-audit/persistence.ts` — asset_type union extended; `getLatestTicketScopedAudit`; `createPendingWrite` accepts asset_type + ticket linkage
|
||||
- `lib/services/analyzer/worker.ts` — post-analysis xref ingestion hook
|
||||
- `app/api/analyzer/itglue/applications/[id]/apply/route.ts` — ticket linkage + xref insert on Apply
|
||||
- `app/api/analyzer/itglue/applications/[id]/revert/[writeId]/route.ts` — asset_type now passed to createPendingWrite
|
||||
- `app/analyzer/analysis/[id]/page.tsx` — `<ItglueSuggestionsPanel/>` rendered
|
||||
- `app/analyzer/itglue/applications/[id]/page.tsx` — "Tickets that touched this asset" section
|
||||
- `components/navigation/app-navigation.tsx` — split entries
|
||||
271
docs/loglift-eventlog-pipeline-spec.md
Normal file
271
docs/loglift-eventlog-pipeline-spec.md
Normal file
|
|
@ -0,0 +1,271 @@
|
|||
# LogLift Event-Log Pipeline (Phase 4.3)
|
||||
|
||||
End-to-end spec for the LogLift evidence path: a Datto RMM-deployed
|
||||
PowerShell collector captures a Windows endpoint's event logs + system
|
||||
context, gzips the JSON, uploads it to Backblaze B2, then POSTs metadata
|
||||
to Pulse. Pulse downloads the gzip, slims it, persists it as
|
||||
`rmm_executions` evidence, and (when the hostname uniquely matches an IT
|
||||
Glue Configuration) auto-triggers an asset-first audit.
|
||||
|
||||
## Why this exists
|
||||
|
||||
Phase 4.2 wired Datto RMM Overshell PowerShell evidence into the audit
|
||||
pipeline, but Overshell stdout is capped at ~50KB practical — too small
|
||||
for full event logs. Wulf already runs a richer evidence path through
|
||||
n8n: collector → B2 → n8n decompress + LLM → Telegram. Phase 4.3 makes
|
||||
Pulse the receiver instead of n8n so:
|
||||
|
||||
- LogLift evidence lands in the same `rmm_executions` table.
|
||||
- The audit pipeline's `rmm_evidence` arm picks it up automatically.
|
||||
- Admins can dispatch a LogLift run from the Configuration page (Datto
|
||||
Quick Job into the registered LogLift component).
|
||||
- Successful uploads matched to a unique IT Glue Configuration auto-fire
|
||||
an asset-first audit so documentation suggestions surface immediately.
|
||||
|
||||
## Components
|
||||
|
||||
```
|
||||
┌──────────────────────────┐ ┌────────────────────────┐
|
||||
│ Windows endpoint │ │ Datto RMM │
|
||||
│ ─ collector PowerShell │ ◀───── │ ─ LogLift component │
|
||||
│ ─ gzip event logs │ │ (job dispatched │
|
||||
│ ─ upload to B2 │ ────▶ │ by Pulse) │
|
||||
│ ─ POST webhook to Pulse │ │ │
|
||||
└──────────────────────────┘ └────────────────────────┘
|
||||
│ ▲
|
||||
│ B2 PUT │ runQuickJob
|
||||
▼ │
|
||||
┌──────────────────────────┐ ┌────────────────────────┐
|
||||
│ Backblaze B2 │ ◀─SigV4─│ Pulse │
|
||||
│ bucket: wulf-audits │ │ /api/rmm/loglift/ │
|
||||
│ region: us-west-002 │ GET │ upload │
|
||||
│ │ ────▶ │ ─ download + slim │
|
||||
│ │ │ ─ persist evidence │
|
||||
└──────────────────────────┘ │ ─ auto-audit (single │
|
||||
│ match only) │
|
||||
└────────────────────────┘
|
||||
```
|
||||
|
||||
## Object-key convention
|
||||
|
||||
```
|
||||
{datto_site_uid}/{computer_name}/eventlogs_{YYYYMMDD_HHMMSS}.json.gz
|
||||
```
|
||||
|
||||
Pulse rejects anything not matching:
|
||||
|
||||
```
|
||||
^[A-Za-z0-9_-]+/[A-Za-z0-9_.-]+/eventlogs_[0-9_]+\.json\.gz$
|
||||
```
|
||||
|
||||
## Webhook contract
|
||||
|
||||
`POST /api/rmm/loglift/upload`
|
||||
|
||||
Auth header: `x-openclaw-key: <OPENCLAW_API_KEY>` (constant-time match).
|
||||
|
||||
Request body:
|
||||
|
||||
```json
|
||||
{
|
||||
"runId": "pulse_a1b2c3_1714672800000",
|
||||
"clientId": "f7a8b9c0-…",
|
||||
"computerName": "YNGHYNWNP01",
|
||||
"deviceUid": "f7a8b9c0-… (optional)",
|
||||
"summary": {
|
||||
"totalEvents": 4012,
|
||||
"criticalEvents": 0,
|
||||
"errorCount": 23,
|
||||
"warningCount": 187,
|
||||
"timeRange": "Last 24 hours"
|
||||
},
|
||||
"objectKey": "f7a8b9c0/YNGHYNWNP01/eventlogs_20260502_120000.json.gz",
|
||||
"collectedAt": "2026-05-02T12:00:03.124Z",
|
||||
"rmmContext": {
|
||||
"siteName": "Hynes — Youngstown",
|
||||
"siteUid": "f7a8b9c0-…",
|
||||
"accountUid": "…"
|
||||
},
|
||||
"issueDescription": "(optional, free-text)",
|
||||
"ticketNumber": "T20260502.0019"
|
||||
}
|
||||
```
|
||||
|
||||
Response (200):
|
||||
|
||||
```json
|
||||
{
|
||||
"executionId": "0193…",
|
||||
"matched": {
|
||||
"datto_site_id": 42,
|
||||
"datto_device_uid": "f7a8b9c0-…",
|
||||
"autotask_company_id": "29861375",
|
||||
"configuration_id": "12345",
|
||||
"configuration_single_match": true
|
||||
},
|
||||
"parsed": {
|
||||
"total_events": 4012,
|
||||
"critical_events": 0,
|
||||
"error_count": 23,
|
||||
"warning_count": 187,
|
||||
"time_range": "Last 24 hours"
|
||||
},
|
||||
"auditId": "0193…"
|
||||
}
|
||||
```
|
||||
|
||||
## Pulse-driven dispatch
|
||||
|
||||
Configuration page → "Run discovery" picker → LogLift entry. The picker
|
||||
filters by `target_type='asset_self'`. Dispatch path:
|
||||
|
||||
1. Resolve the device's Datto site uid (`datto_rmm_sites.uid`) → use as
|
||||
`ClientId`.
|
||||
2. Resolve the LogLift component uid from `rmm_settings`. If not cached,
|
||||
the executor calls `discoverLogliftComponent()` which scans the Datto
|
||||
API for components matching `/loglift|eventlog/i`.
|
||||
3. Generate `runId = pulse_<hex8>_<ms>`.
|
||||
4. Insert `rmm_executions` row with `transport='b2_upload'`, `run_id`,
|
||||
`status='queued'`. The row's `variables` column stores the
|
||||
non-secret variables (RunId, ClientId, WebhookUrl) — the secret is
|
||||
stripped before persistence.
|
||||
5. `runQuickJob` with the LogLift component_uid + variables:
|
||||
- `RunId` — webhook correlation
|
||||
- `ClientId` — Datto site uid
|
||||
- `WebhookUrl` — `${BETTER_AUTH_URL}/api/rmm/loglift/upload`
|
||||
- `WebhookSecret` — `OPENCLAW_API_KEY`
|
||||
6. On Quick Job ack: flip to `running` + persist `job_uid`.
|
||||
7. Worker skips `b2_upload` rows during stdout-poll. Webhook is the
|
||||
completion event. The 5-minute timeout sweep still applies — stuck
|
||||
rows get marked `timeout`.
|
||||
|
||||
## Out-of-band (collector-driven) ingest
|
||||
|
||||
If the LogLift collector fires from its own schedule (e.g. n8n still
|
||||
runs in parallel during cutover), the webhook receiver inserts a fresh
|
||||
`rmm_executions` row with `triggered_by_user_id=NULL`,
|
||||
`status='running'`, then completes it in the same handler.
|
||||
|
||||
`run_id` has a unique index; replays of the same upload no-op cleanly.
|
||||
|
||||
## Slim-evidence shape
|
||||
|
||||
Stored in `rmm_executions.parsed_evidence` (JSONB). The full gzip stays
|
||||
in B2 forever (forensic replay via the presigned-GET helper):
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"transport": "b2_upload",
|
||||
"object_key": "…",
|
||||
"collected_at": "…",
|
||||
"rmm_context": {…},
|
||||
"issue_description": "…",
|
||||
"ticket_number": "…",
|
||||
"webhook_summary": {…}, // raw counts from the webhook
|
||||
"metadata": {…}, // from inside the gzip
|
||||
"system_context": { // OS, hardware, disks, updates …
|
||||
"OS": {…},
|
||||
"Hardware": {…},
|
||||
"Memory": {…},
|
||||
"Disks": [{…}],
|
||||
"RecentUpdates": [{…}],
|
||||
"Uptime": "…",
|
||||
"LastBoot": "…",
|
||||
"PendingReboot": false,
|
||||
"RebootReasons": []
|
||||
},
|
||||
"summary": { // from inside the gzip
|
||||
"TotalEvents": 4012,
|
||||
"CriticalEvents": 0,
|
||||
"ByLevel": {"Error": 23, "Warning": 187},
|
||||
"TimeRange": "Last 24 hours",
|
||||
"TopEventIds": [{"Id": 7036, "Count": 145}]
|
||||
},
|
||||
"top_events": [ // top 100 by severity then recency
|
||||
{"TimeCreated":"…", "LevelDisplayName":"Error", "Id":7036, "Source":"Service Control Manager", "Message":"…"}
|
||||
],
|
||||
"event_count_total": 4012,
|
||||
"top_events_truncated": true
|
||||
}
|
||||
```
|
||||
|
||||
Output runs through `redact()` before persistence.
|
||||
|
||||
## Auto-audit hook
|
||||
|
||||
When the resolved IT Glue Configuration is **single-match** (exactly
|
||||
one row matches the company + hostname), Pulse fires
|
||||
`runAssetAudit({ assetType: 'configuration', assetId, generatedByUserId: null, provider: 'anthropic' })`.
|
||||
|
||||
The new audit row appears on the Configuration's audit page with
|
||||
`triggered_by_user_id=null`. Multi-match Configurations are logged but
|
||||
skipped — auditing the wrong asset is worse than no audit.
|
||||
|
||||
`audit_log` actions:
|
||||
|
||||
| Action | Resource | When |
|
||||
| ----------------------------- | --------------------- | ------------------------------------------ |
|
||||
| `rmm.loglift.dispatched` | datto_device | Pulse-driven Quick Job accepted |
|
||||
| `rmm.loglift.received` | datto_device | Webhook landed + evidence persisted |
|
||||
| `rmm.loglift.matched` | itg_configuration | Configuration matched + single |
|
||||
| `rmm.loglift.audit_triggered` | itg_configuration | Auto-audit completed |
|
||||
|
||||
## Safety + cost guards
|
||||
|
||||
- `OBJECT_KEY_REGEX` — path-traversal guard.
|
||||
- B2 download cap: 25MB hard.
|
||||
- Decompress cap: refuse if gzip ISIZE > 100MB (zip-bomb defense),
|
||||
re-checked after inflation.
|
||||
- `redact()` runs on the slim payload before storage.
|
||||
- Auto-audit only on single-match Configurations.
|
||||
- `analyzer_cost_audit` records on the dispatch path (rate-limited the
|
||||
same as Overshell). Inbound webhooks are NOT rate-limited — the
|
||||
agent decides cadence; we trust the agent.
|
||||
- The collector's own B2 credentials never travel through Pulse. Pulse
|
||||
uses its own `B2_KEY_ID` / `B2_APP_KEY` to download.
|
||||
|
||||
## Environment variables
|
||||
|
||||
Required for dispatch + receive:
|
||||
|
||||
| Var | Default | Notes |
|
||||
| -------------------- | -------------------------------- | ---------------------------------- |
|
||||
| `B2_KEY_ID` | — | B2 application key id |
|
||||
| `B2_APP_KEY` | — | B2 application key secret |
|
||||
| `B2_BUCKET` | `wulf-audits` | matches existing n8n config |
|
||||
| `B2_REGION` | `us-west-002` | |
|
||||
| `B2_ENDPOINT` | `s3.us-west-002.backblazeb2.com` | no scheme |
|
||||
| `OPENCLAW_API_KEY` | — | webhook auth + collector variable |
|
||||
| `BETTER_AUTH_URL` | — | base URL the collector POSTs to |
|
||||
|
||||
## Verification checklist
|
||||
|
||||
1. `\d rmm_executions` shows `transport`, `evidence_object_key`, `run_id`.
|
||||
2. `B2 client.test.ts` passes (SigV4 fixture + path-traversal rejection).
|
||||
3. Webhook auth: missing/bad `x-openclaw-key` → 401; good key + valid
|
||||
body → 200.
|
||||
4. `/admin/rmm-overshell` → "Re-discover LogLift" populates
|
||||
`rmm_settings.loglift_component_uid`.
|
||||
5. Configuration page → "Run discovery" → LogLift selection → row goes
|
||||
`running` (`transport='b2_upload'`, `run_id` set).
|
||||
6. Webhook handler downloads from B2, flips row to `complete`,
|
||||
`parsed_evidence` populated.
|
||||
7. Single-match Configuration → new `itglue_asset_audits` row with
|
||||
`triggered_by_user_id=null`.
|
||||
8. Asset-audit prompt's `=== LIVE RMM EVIDENCE ===` block contains the
|
||||
slim LogLift payload (system_context + summary + top_events).
|
||||
9. Non-admin direct webhook POST without the openclaw key returns 401.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No Telegram summary / non-technical second LLM pass (notification,
|
||||
not data path).
|
||||
- No replacement of the existing collector PowerShell. The Datto
|
||||
component is registered by name; the dispatch path passes `RunId`,
|
||||
`ClientId`, `WebhookUrl`, `WebhookSecret` as variables.
|
||||
- No bulk-replay of historical B2 objects. Manual replay can be added
|
||||
later via an admin endpoint that takes an `objectKey`.
|
||||
- No HMAC signature on the webhook body — `x-openclaw-key` is the auth
|
||||
boundary. Adding HMAC is a fast follow if we expand external
|
||||
integrations.
|
||||
412
docs/rmm-overshell-evidence-spec.md
Normal file
412
docs/rmm-overshell-evidence-spec.md
Normal file
|
|
@ -0,0 +1,412 @@
|
|||
# Feature: Datto RMM Overshell Evidence Pipeline
|
||||
|
||||
**Status:** shipped (Phase 4.2 of the AI Ticket Analyzer)
|
||||
**Migration:** `077_rmm_overshell.sql`
|
||||
**Build notes:** `docs/wulf-pulse-ticket-analyzer-build-notes.md` → Phase 4.2
|
||||
**Related:** `docs/itglue-asset-audit-spec.md` (where evidence feeds into audits)
|
||||
|
||||
---
|
||||
|
||||
## What it is
|
||||
|
||||
Pulse dispatches a curated library of read-only PowerShell scripts via the
|
||||
Datto RMM **Overshell** component, captures the structured output, and
|
||||
makes it available to the IT Glue audit pipeline as authoritative live
|
||||
evidence. Every dispatch is recorded with full audit trail; admins
|
||||
trigger; results land within 30–180 seconds; subsequent audits cite the
|
||||
fresh data with high confidence.
|
||||
|
||||
The motivating example: an AD health summary at Hynes Industries
|
||||
(replication, dcdiag, named services, IP conflicts) — captured by openclaw
|
||||
on 2026-04-25 — was useful for incident response but stayed in chat. With
|
||||
Phase 4.2 the same intel can be reproduced from Pulse directly, persisted,
|
||||
and cited by the LLM next time we audit a Hynes Configuration.
|
||||
|
||||
---
|
||||
|
||||
## How it fits
|
||||
|
||||
```
|
||||
Audit pipeline Phase 4.2 → live state
|
||||
───────────────── ─────────────────────
|
||||
Ticket history ─┐ ▲
|
||||
├─► AssetAuditContext ─► LLM ─► Suggestions │
|
||||
IT Glue schema ─┤ │
|
||||
│ │
|
||||
Peer exemplars ─┤ │
|
||||
│ │
|
||||
Live RMM evidence ◄─── rmm_executions ◄─── Worker ◄──── Datto RMM
|
||||
(this phase) (poller) ▲
|
||||
│
|
||||
rmm_settings ┌───┘
|
||||
(component_uid)│
|
||||
│
|
||||
POST /api/rmm/executions
|
||||
│
|
||||
user clicks "Run discovery"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## User-facing flows
|
||||
|
||||
### Site-anchored discovery — `/analyzer/itglue/sites/[companyId]`
|
||||
|
||||
Admin opens a client's site page, picks a script from the dropdown
|
||||
(filtered to `target_type='site_anchor'`), and clicks. Pulse:
|
||||
|
||||
1. Resolves the WNP endpoint via `datto_rmm_devices` matching `LLLCCCWNPNN`.
|
||||
2. Inserts a pending row in `rmm_executions`.
|
||||
3. Calls `runQuickJob` with the Overshell `component_uid` and the script
|
||||
body as a variable.
|
||||
4. The worker polls every 5s; finalizes when stdout/stderr arrive.
|
||||
5. The page lists the run + parsed evidence inline.
|
||||
|
||||
Site-anchored scripts: `get-ad-health`, `get-dhcp-scopes`, `get-dns-zones`,
|
||||
`get-network-discovery`.
|
||||
|
||||
### Asset-specific discovery — Configuration detail page
|
||||
|
||||
When viewing an IT Glue Configuration that maps to a Datto RMM device
|
||||
(via `itg_configurations.rmm_id` or hostname match), the picker offers
|
||||
asset-self scripts: `get-services`, `get-installed-software`,
|
||||
`get-event-log-recent`. Output is captured against the asset; subsequent
|
||||
audits on that asset see it.
|
||||
|
||||
### Application detail page
|
||||
|
||||
Application records belong to a client, not a specific server. The picker
|
||||
on the Application page offers **site-anchored** scripts targeting the
|
||||
client's WNP — same as the site page, just discoverable in-context.
|
||||
|
||||
### Admin settings — `/admin/rmm-overshell`
|
||||
|
||||
- Current Overshell `component_uid` + name
|
||||
- Variable name (default `CommandLine`; editable per tenant)
|
||||
- "Re-discover" button (forces a full component scan)
|
||||
- 24-hour activity counters (total / running / failed)
|
||||
- Recent execution log
|
||||
|
||||
---
|
||||
|
||||
## Permissions
|
||||
|
||||
| Action | Permission |
|
||||
|---|---|
|
||||
| List scripts / list executions / view execution detail | `requireAuth()` |
|
||||
| Trigger an execution | `requirePermission('rmm','execute')` |
|
||||
| Edit Overshell settings, force discovery | `requirePermission('admin','access')` |
|
||||
|
||||
`rmm.execute` is granted to `admin` + `super-admin` only (see
|
||||
`lib/permissions.ts`).
|
||||
|
||||
---
|
||||
|
||||
## Script library
|
||||
|
||||
Each script is a TypeScript module exporting an `RmmScript`:
|
||||
|
||||
```ts
|
||||
{
|
||||
id: 'get-services',
|
||||
name: 'Running services',
|
||||
description: 'Snapshot of all running Windows services …',
|
||||
target_type: 'asset_self',
|
||||
expected_runtime_seconds: 15,
|
||||
version: 1,
|
||||
body: '<PowerShell>',
|
||||
parseOutput: (stdout) => parseJsonOutput(stdout),
|
||||
}
|
||||
```
|
||||
|
||||
Convention: scripts end with `ConvertTo-Json -Depth N -Compress` so
|
||||
parsing is `JSON.parse`. The body file in `lib/services/rmm/scripts/` is
|
||||
the source of truth — DB never stores executable code.
|
||||
|
||||
Adding a new script:
|
||||
|
||||
1. Create `lib/services/rmm/scripts/<id>.ts` exporting an `RmmScript`.
|
||||
2. Import + add to `_all` in `lib/services/rmm/scripts/index.ts`.
|
||||
3. Bump `version` if you change body or output shape.
|
||||
4. `npm test` — registry validates uniqueness, presence of required
|
||||
fields, and absence of credential-shaped patterns.
|
||||
|
||||
### Current library
|
||||
|
||||
| Script | Target | Notes |
|
||||
|---|---|---|
|
||||
| `get-services` | asset_self | Get-Service running list (Name/DisplayName/StartType/ServiceType). |
|
||||
| `get-installed-software` | asset_self | Win32 + WoW64 uninstall registry. |
|
||||
| `get-event-log-recent` | asset_self | Last 24h Errors+Warnings from System+Application logs (cap 50). |
|
||||
| `get-ad-health` | site_anchor | Per-DC replication, services, dcdiag pass/fail, recent Netlogon/DNS errors. |
|
||||
| `get-dhcp-scopes` | site_anchor | All authorized DHCP servers, scopes, statistics, reservations. |
|
||||
| `get-dns-zones` | site_anchor | Zones + forwarders + conditional forwarders per DC. |
|
||||
| `get-network-discovery` | site_anchor | Local NIC config + ARP table + IP-conflict detection (catches the proven test case). |
|
||||
|
||||
---
|
||||
|
||||
## Target resolution
|
||||
|
||||
### Site-anchored
|
||||
|
||||
1. `companies → datto_rmm_sites` via `autotask_company_id`.
|
||||
2. `datto_rmm_devices WHERE site_id = $1 AND hostname ~* '^[A-Z]{3}[A-Z]{3}WNP[0-9]{2}$'`.
|
||||
3. Strict regex check in JS (PostgreSQL regex is permissive; JS pins the
|
||||
shape exactly).
|
||||
4. Sort: online first, then ascending suffix number.
|
||||
5. Returns `{ device_uid, hostname, online }` or null.
|
||||
|
||||
### Asset-self
|
||||
|
||||
1. `IT Glue itg_configurations.rmm_id` → `datto_rmm_devices.uid`.
|
||||
2. Fallback: hostname match on `datto_rmm_devices.hostname`.
|
||||
|
||||
If neither resolves, the picker is disabled with the tooltip "No Datto
|
||||
device id."
|
||||
|
||||
---
|
||||
|
||||
## Execution lifecycle
|
||||
|
||||
```
|
||||
queued ─► running ─► complete | failed | timeout
|
||||
▲ │
|
||||
│ │
|
||||
└──────── markExecutionRunning(jobUid)
|
||||
after runQuickJob success
|
||||
```
|
||||
|
||||
- **queued**: row inserted, `runQuickJob` not yet returned.
|
||||
- **running**: `job_uid` stored, worker is polling. Bounded by `timeout_at`
|
||||
(5 minutes from queue).
|
||||
- **complete**: terminal status from Datto, exit_code 0.
|
||||
- **failed**: terminal status with non-zero exit code, OR runQuickJob
|
||||
threw, OR Datto returned no job_uid.
|
||||
- **timeout**: `timeout_at < NOW()` and worker hasn't seen completion.
|
||||
|
||||
The worker also writes generic `audit_log` rows on every state change:
|
||||
`rmm.execute` (queue), `rmm.execute.complete`, `rmm.execute.failed`,
|
||||
`rmm.execute.timeout`.
|
||||
|
||||
---
|
||||
|
||||
## Audit-pipeline integration
|
||||
|
||||
`buildAssetAuditContext` (in `lib/services/analyzer/asset-audit/data-builder.ts`)
|
||||
loads `rmm_evidence` as a 7th LLM context arm:
|
||||
|
||||
```
|
||||
loadRmmEvidence(itglueOrgId, assetType, assetId)
|
||||
├─ map IT Glue org → Autotask company (case-insensitive name)
|
||||
├─ listLatestEvidenceForCompany(companyId, 7) // site-anchored, last 7 days
|
||||
└─ listLatestEvidenceForAsset(assetType, assetId) // asset-self, all-time
|
||||
```
|
||||
|
||||
The prompt renders a new section:
|
||||
|
||||
```
|
||||
=== LIVE RMM EVIDENCE (most recent successful Overshell runs; AUTHORITATIVE current state) ===
|
||||
[ {execution_id, script_id, target_type, target_hostname, captured_at, parsed}, … ]
|
||||
```
|
||||
|
||||
System prompt rule (in `prompt.ts`):
|
||||
|
||||
> *"Treat parsed contents as authoritative current state … Use it to
|
||||
> justify suggested values with high confidence — e.g. if Get-Services
|
||||
> lists 'BartenderProcessService' running on the target and a ticket
|
||||
> asked about BarTender printing, suggest adding that service name to
|
||||
> operating_system_notes with confidence=high. Cite execution_id
|
||||
> alongside ticket numbers in evidence_ticket_numbers."*
|
||||
|
||||
Trim priority on overflow: `peer_global → ticket_evidence → rmm_evidence`
|
||||
(rmm last — highest-value section).
|
||||
|
||||
---
|
||||
|
||||
## API contract
|
||||
|
||||
| Method | Route | Auth | Purpose |
|
||||
|---|---|---|---|
|
||||
| GET | `/api/admin/rmm/settings` | admin | Settings + 24h counts |
|
||||
| PATCH | `/api/admin/rmm/settings` | admin | Update `overshellVariableName` |
|
||||
| POST | `/api/admin/rmm/settings/discover` | admin | Force component scan |
|
||||
| GET | `/api/rmm/scripts` | auth | Library catalog (no bodies) |
|
||||
| GET | `/api/rmm/executions` | auth | List (filters: companyId, scriptId, status, assetType, assetId) |
|
||||
| POST | `/api/rmm/executions` | rmm.execute | Queue a fresh execution |
|
||||
| GET | `/api/rmm/executions/[id]` | auth | Single execution detail |
|
||||
| GET | `/api/analyzer/itglue/sites/[companyId]` | auth | Site discovery summary |
|
||||
|
||||
POST body:
|
||||
|
||||
```ts
|
||||
{
|
||||
scriptId: string,
|
||||
target:
|
||||
| { type: 'site_anchor', companyId: number | string }
|
||||
| { type: 'asset_self', deviceUid: string, hostname?, companyId?, assetType?, assetId? },
|
||||
triggeredByAuditId?: string,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Safety
|
||||
|
||||
### Three-layer audit trail
|
||||
|
||||
| Table | What it captures |
|
||||
|---|---|
|
||||
| `rmm_executions` | Every dispatch — full lifecycle, raw output (redacted), parsed evidence, target, audit/asset linkage. Forever-retained. |
|
||||
| `analyzer_cost_audit` | Rate-limit decision row per request — approved / blocked. Same view as LLM cost decisions. |
|
||||
| Generic `audit_log` | `rmm.execute`, `rmm.execute.complete`, `rmm.execute.failed`, `rmm.execute.timeout`. Surfaces in `/admin/audit-log`. |
|
||||
|
||||
### Rate limit
|
||||
|
||||
50 executions per user per 24-hour rolling window. Trips before
|
||||
`runQuickJob` is called; logs the block to `analyzer_cost_audit` with
|
||||
`decision='blocked'`.
|
||||
|
||||
### Hard timeout
|
||||
|
||||
5 minutes per execution. The worker sweeps `timeout_at < NOW()` rows on
|
||||
every tick and marks them `timeout` (with an `error_message` recording
|
||||
the cause).
|
||||
|
||||
### Output redaction
|
||||
|
||||
Every `raw_stdout` and `raw_stderr` passes through
|
||||
`lib/services/analyzer/itglue-redact.ts:redact()` before persistence and
|
||||
again before the audit prompt sees it. Strips any key matching
|
||||
`/password|secret|key|token|credential|api[_-]?key/i`. Belt-and-braces
|
||||
defense even though the curated library has no credential-handling
|
||||
scripts.
|
||||
|
||||
### Registry gating
|
||||
|
||||
Only ids in the in-code `SCRIPTS` registry can run. The endpoint rejects
|
||||
unknown ids with 400 before any Datto API call. There's no UI to paste
|
||||
ad-hoc PowerShell — everything goes through the typed `RmmScript`
|
||||
interface.
|
||||
|
||||
---
|
||||
|
||||
## Operational SQL
|
||||
|
||||
### Most-recent successful run per (company, script)
|
||||
|
||||
```sql
|
||||
SELECT DISTINCT ON (target_company_id, script_id)
|
||||
target_company_id, script_id, target_hostname,
|
||||
completed_at, exit_code
|
||||
FROM rmm_executions
|
||||
WHERE status = 'complete'
|
||||
AND completed_at >= NOW() - INTERVAL '14 days'
|
||||
ORDER BY target_company_id, script_id, completed_at DESC;
|
||||
```
|
||||
|
||||
### Stuck (queued/running past timeout)
|
||||
|
||||
```sql
|
||||
SELECT id, script_id, target_hostname, status, queued_at, timeout_at
|
||||
FROM rmm_executions
|
||||
WHERE status IN ('queued','running')
|
||||
AND timeout_at <= NOW()
|
||||
ORDER BY queued_at;
|
||||
```
|
||||
|
||||
(The worker should sweep these on the next tick.)
|
||||
|
||||
### Per-user daily activity
|
||||
|
||||
```sql
|
||||
SELECT performed_by_user_id,
|
||||
COUNT(*) FILTER (WHERE queued_at >= NOW() - INTERVAL '24 hours') AS last_24h,
|
||||
COUNT(*) FILTER (WHERE status = 'complete' AND queued_at >= NOW() - INTERVAL '24 hours') AS completed_24h
|
||||
FROM rmm_executions
|
||||
WHERE performed_by_user_id IS NOT NULL
|
||||
GROUP BY performed_by_user_id
|
||||
ORDER BY last_24h DESC;
|
||||
```
|
||||
|
||||
### Verify a particular script's parser
|
||||
|
||||
```sql
|
||||
SELECT id, status, parse_error, jsonb_pretty(parsed_evidence) AS parsed
|
||||
FROM rmm_executions
|
||||
WHERE script_id = 'get-ad-health'
|
||||
ORDER BY queued_at DESC
|
||||
LIMIT 5;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Limitations & non-goals
|
||||
|
||||
- **Forward-only capture.** No backfill of historical Datto Overshell
|
||||
jobs (per user direction).
|
||||
- **WNP-only target resolution.** Direct-to-DC role detection is a
|
||||
fast-follow if AD scripts that need native DC execution become
|
||||
important.
|
||||
- **Read-only scripts only.** No Overshell-driven configuration changes.
|
||||
Apply documentation updates via the IT Glue Phase 4 path.
|
||||
- **No ad-hoc PowerShell input from the UI.** Registry-listed scripts only.
|
||||
- **No audit-driven auto-execution in v1.** Admin clicks the button. A
|
||||
future version could let the audit panel propose "Run Get-Services to
|
||||
fill this gap" with one-click confirm.
|
||||
- **Credential output never enters the audit prompt.** Three-layer refusal
|
||||
at script-library curation, output redaction, and prompt-side rules.
|
||||
|
||||
---
|
||||
|
||||
## Files
|
||||
|
||||
**New:**
|
||||
- `migrations/077_rmm_overshell.sql`
|
||||
- `lib/services/rmm/settings.ts`
|
||||
- `lib/services/rmm/persistence.ts`
|
||||
- `lib/services/rmm/target-resolver.ts` + test
|
||||
- `lib/services/rmm/executor.ts`
|
||||
- `lib/services/rmm/worker.ts` + test
|
||||
- `lib/services/rmm/scripts/types.ts`
|
||||
- `lib/services/rmm/scripts/{get-services,get-installed-software,get-event-log-recent,get-ad-health,get-dhcp-scopes,get-dns-zones,get-network-discovery}.ts`
|
||||
- `lib/services/rmm/scripts/index.ts` + registry test
|
||||
- `app/api/admin/rmm/settings/route.ts`
|
||||
- `app/api/admin/rmm/settings/discover/route.ts`
|
||||
- `app/api/rmm/scripts/route.ts`
|
||||
- `app/api/rmm/executions/route.ts`
|
||||
- `app/api/rmm/executions/[id]/route.ts`
|
||||
- `app/api/analyzer/itglue/sites/[companyId]/route.ts`
|
||||
- `app/admin/rmm-overshell/page.tsx`
|
||||
- `app/analyzer/itglue/sites/[companyId]/page.tsx`
|
||||
- `components/rmm/rmm-script-picker.tsx`
|
||||
- `components/rmm/rmm-execution-stream.tsx`
|
||||
|
||||
**Modified:**
|
||||
- `lib/permissions.ts` — `rmm: ['read','execute']`
|
||||
- `lib/services/datto-rmm-client.ts` — `findOvershellComponent`
|
||||
- `lib/services/analyzer/asset-audit/data-builder.ts` — 7th evidence arm
|
||||
- `lib/services/analyzer/asset-audit/prompt.ts` — RMM evidence section + system rule
|
||||
- `app/api/analyzer/itglue/applications/[id]/route.ts` — surface `autotaskCompanyId`
|
||||
- `app/api/analyzer/itglue/configurations/[id]/route.ts` — surface `dattoDeviceUid` + `autotaskCompanyId`
|
||||
- `app/analyzer/itglue/applications/[id]/page.tsx` — embedded `<RmmScriptPicker filter='site_anchor'/>`
|
||||
- `app/analyzer/itglue/configurations/[id]/page.tsx` — embedded `<RmmScriptPicker filter='asset_self'/>`
|
||||
- `components/navigation/app-navigation.tsx` — Admin → "RMM Overshell"
|
||||
|
||||
## Phase 4.3: LogLift transport variant
|
||||
|
||||
The same `rmm_executions` table also stores LogLift event-log uploads
|
||||
(`transport='b2_upload'`). Different transport, same evidence pipeline:
|
||||
|
||||
- The Datto RMM-registered LogLift component handles its own PowerShell
|
||||
collection + B2 upload. Pulse never sees the script body.
|
||||
- Pulse dispatches via `runQuickJob` with `RunId`, `ClientId`,
|
||||
`WebhookUrl`, `WebhookSecret` variables (same factory + same audit
|
||||
trail as Overshell).
|
||||
- The collector POSTs to `/api/rmm/loglift/upload` when the upload
|
||||
finishes. Pulse downloads from B2, slims, and persists.
|
||||
- The audit-context `rmm_evidence` arm picks the LogLift row up by
|
||||
`script_id='loglift-eventlogs'` automatically — same
|
||||
`listLatestEvidenceForAsset` join as any other asset-self script.
|
||||
|
||||
See `docs/loglift-eventlog-pipeline-spec.md` for the full webhook
|
||||
contract + slim-shape spec.
|
||||
|
|
@ -794,3 +794,764 @@ one Phase 2 push.
|
|||
| 2.6 | 128 | clean | aggregate reports (071, runner, 3 endpoints, 3 pages) |
|
||||
| 2.7 | 128 | clean | cost guards (072, audit log, threshold gating) |
|
||||
| 2.8 | 128 | clean | runbook + build notes |
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Link-aware bundle analysis
|
||||
|
||||
**Why**
|
||||
|
||||
Single-ticket analysis misses the bigger picture for master/problem tickets,
|
||||
which are explicitly aggregator records — a "Master problem ticket" with a
|
||||
`RELATED TICKETS:` block in its description naming the constituent
|
||||
incidents. Aggregate reports already existed (Phase 2.6) but required the
|
||||
user to pre-analyze every constituent and hand-pick them on
|
||||
`/analyzer/reports/new`. Phase 3 closes the gap: one click on a problem
|
||||
ticket fans out individual analyses for each linked ticket and chains them
|
||||
into an aggregate report.
|
||||
|
||||
**Delivered**
|
||||
|
||||
- `migrations/073_analyzer_link_aware_bundles.sql`:
|
||||
- `analyzer_aggregate_reports.expected_ticket_numbers TEXT[]` — the full
|
||||
set of ticket numbers a bundle is waiting on.
|
||||
- `analyzer_aggregate_reports.triggered_by_ticket_number TEXT` — the
|
||||
master ticket the bundle was launched from.
|
||||
- Status check extended to include `'pending_analyses'` (waiting for
|
||||
individual analyses) before transitioning to `'pending'` (ready for
|
||||
aggregate-reduce).
|
||||
- Replaced the partial pending-status index to cover the new state; added
|
||||
a GIN index on `expected_ticket_numbers` filtered to
|
||||
`pending_analyses` for the worker chain-trigger lookup.
|
||||
- `lib/services/analyzer/link-discovery.ts`:
|
||||
- **Explicit arm** (no LLM, deterministic): regex scan over the ticket
|
||||
description and each retained note for `T\d{8}\.\d{4}` references,
|
||||
detection of the structured `RELATED TICKETS:` block (refs inside it
|
||||
flagged `confidence: 'high'`), and resolution of
|
||||
`tickets.problem_ticket_id` to a ticket number. Self-references and
|
||||
refs not present in the local mirror are dropped silently. Capped at
|
||||
`MAX_EXPLICIT_LINKS = 15`.
|
||||
- **Suggested arm** (Haiku, opt-in): one LLM pass over recent
|
||||
same-company tickets (±30 days, capped at 50 candidates). Returns up
|
||||
to 5 candidates with one-sentence reasons. Hallucination-guarded —
|
||||
drops any number not in the candidate list.
|
||||
- `detectProblemTicket()` returns boolean + signal list; UI uses signals
|
||||
to decide whether to highlight the bundle CTA as the primary action.
|
||||
- `app/api/analyzer/tickets/[ticketNumber]/links/route.ts`:
|
||||
- `GET` returns the explicit arm only (cheap, called on page load).
|
||||
- `POST { includeSuggested: true }` runs both arms.
|
||||
- `app/api/analyzer/tickets/[ticketNumber]/analyze-bundle/route.ts`:
|
||||
- Validates master + linked tickets exist locally (single SQL roundtrip).
|
||||
- Runs per-ticket idempotency: existing complete analyses with matching
|
||||
content hash short-circuit; missing tickets are queued via the existing
|
||||
`queueJob()` helper.
|
||||
- Cost guard runs against the **new work only** — already-complete
|
||||
analyses don't add cost. Per-ticket estimate is a flat $0.15
|
||||
(Sonnet-tier pessimistic) plus `estimateAggregateReportCost()` for the
|
||||
reduce step.
|
||||
- Creates one `analyzer_aggregate_reports` row in
|
||||
`'pending_analyses'` (or straight to `'pending'` and fires
|
||||
`runAggregateReport()` if everything was already complete).
|
||||
- Bundle cap: `MAX_BUNDLE_SIZE = 25`.
|
||||
- `lib/services/analyzer/aggregate-persistence.ts`:
|
||||
- `createAggregateReport` accepts `expectedTicketNumbers` and
|
||||
`triggeredByTicketNumber`. When set, status starts as
|
||||
`'pending_analyses'`.
|
||||
- `chainTriggerForCompletedAnalysis(ticketNumber, analysisId)` — called
|
||||
by the worker after each successful job. Atomically appends the
|
||||
analysis_id to every pending_analyses bundle expecting that ticket
|
||||
(deduped via `analysis_ids @> ARRAY[…]` guard) and re-checks whether
|
||||
the full set is now satisfied. Returns `readyReportIds` for the worker
|
||||
to fire `runAggregateReport()` on.
|
||||
- `lib/services/analyzer/worker.ts` — chain-trigger fires from both
|
||||
the success branch and the idempotent-short-circuit branch (the bundle
|
||||
endpoint's idempotency check happens at submit time, but a parallel
|
||||
analysis can complete between then and when the worker picks the job
|
||||
up). Failures here are logged but never fail the underlying job.
|
||||
- `components/analyzer/related-tickets-panel.tsx` — renders above the
|
||||
existing `<AnalyzeButton>` on `/analyzer/ticket/[ticketNumber]`:
|
||||
- Cheap GET on mount populates the panel only when refs exist or the
|
||||
ticket looks like a problem ticket — otherwise the component renders
|
||||
nothing.
|
||||
- Pre-checked checkboxes for explicit refs; Switch toggle to load
|
||||
AI-suggested refs (additive, suggestions show with a badge,
|
||||
unchecked by default).
|
||||
- Primary CTA is bundle ("Analyze with N linked tickets") highlighted
|
||||
when `isProblemTicket=true`. Single-ticket flow is preserved
|
||||
untouched on the existing AnalyzeButton in the parent header.
|
||||
- Polls `GET /api/analyzer/aggregate-reports/:id` every 3s after submit;
|
||||
routes to `/analyzer/reports/:id` on completion.
|
||||
- `lib/services/analyzer/link-discovery.test.ts` — 18 new tests covering
|
||||
the regex, RELATED TICKETS section bounds, problem-ticket signal
|
||||
detection, dedup/self-skip, mirror filtering, `MAX_EXPLICIT_LINKS` cap,
|
||||
and confidence-based sorting.
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Bundle is opt-in via the panel, not auto.** A ticket that mentions
|
||||
another ticket once in passing (e.g. "see T20260101.0001 for context")
|
||||
shouldn't quietly trigger 2× the LLM cost on every analysis. The panel
|
||||
is the consent surface — pre-checked when explicit refs exist, but the
|
||||
user explicitly picks the CTA.
|
||||
- **`RELATED TICKETS:` is a strong signal, not a parser-required format.**
|
||||
The regex catches T-numbers anywhere; the structured block just
|
||||
promotes them to high confidence and acts as a problem-ticket signal.
|
||||
No new format is imposed on whoever writes the master ticket.
|
||||
- **Suggested arm uses Haiku, not Sonnet.** ~$0.005 per call against the
|
||||
$5 per-request confirmation threshold — never trips the modal. We never
|
||||
fail the whole call if the suggestion arm throws (logged-and-suppressed
|
||||
via try/catch in `discoverLinks`).
|
||||
- **Per-ticket cost estimate is flat $0.15.** We could compute it from the
|
||||
preprocessed event count, but at the bundle's typical size (3-10
|
||||
tickets) that's $0.45 – $1.50 — far below the $5 confirmation
|
||||
threshold. Worth revisiting if we see real false-positive blocks.
|
||||
- **`pending_analyses` is the new status, distinct from `'pending'`.**
|
||||
Explicit two-step state lets the runner stay simple — it never has to
|
||||
ask "are all my analyses ready?" — that gate is the chain-trigger's
|
||||
job. Existing manual-multi-select reports continue to start at
|
||||
`'pending'`; their flow is untouched.
|
||||
- **`expected_ticket_numbers` matches via array containment, not a
|
||||
separate join table.** Postgres GIN gives us O(log n) lookup and the
|
||||
data lives where it's used — no new table, no foreign-key cascade
|
||||
decisions to make.
|
||||
- **Self-references and unknown tickets are dropped silently.** The user
|
||||
isn't asked to pick from a list; they get a clean "you have N linked
|
||||
tickets" panel. A ghost reference (T-number that doesn't exist in the
|
||||
mirror) is a sync gap, not a bundle decision.
|
||||
- **Fixture migration: `T20260424.0045.input.json` updated** to include
|
||||
`problem_ticket_id: null` so the type-strict load through the
|
||||
preprocessor still parses. The column is nullable in the data-access
|
||||
query and on the row type.
|
||||
|
||||
**Deliberately left out**
|
||||
|
||||
- No retroactive linking for already-completed analyses. If a master
|
||||
ticket got a single-ticket analysis before this shipped, the user
|
||||
re-runs from the panel to bundle.
|
||||
- No editing the bundle composition after submit. Re-run with a different
|
||||
selection if you want a different scope.
|
||||
- No time-window auto-correlation arm (e.g. "all tickets at this client in
|
||||
the last 6 hours"). At Hynes Industries on 2026-05-01 we observed 84
|
||||
tickets in one day — auto-grouping by time would have been useless
|
||||
noise. Same-company time-window ranking is what the Haiku suggested arm
|
||||
is for.
|
||||
- No UI for the `/analyzer/reports/[id]` page to flag itself as a "bundle"
|
||||
vs a manual report. The new fields are surfaced in the API response but
|
||||
the page renders the same regardless.
|
||||
|
||||
## Status after Phase 3
|
||||
|
||||
| Phase | Tests | tsc | Notes |
|
||||
|---|---|---|---|
|
||||
| 3 | 146 | clean | link discovery, bundle endpoint, chain-trigger, panel (073) |
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — IT Glue asset audit + documentation write-back
|
||||
|
||||
**Why**
|
||||
|
||||
Single-ticket analysis already extracts `documentation_gaps_observed` per
|
||||
ticket (in `aggregate_fingerprint`), but nothing acts on them. The companion
|
||||
direction — audit IT Glue records *against* ticket history — converts a
|
||||
passive output into an actionable backlog and offers direct write-back.
|
||||
Concrete proof: T20260502.0033 (Hynes — tags not printing) was resolved by
|
||||
identifying a stopped Windows service on MISYS-SQL processing BarTender
|
||||
scan-folder text files. The IT Glue record `MISYS 6.3` (asset 17096940) had
|
||||
12/17 fields empty — including Wulf Application Champion and Vendor
|
||||
Maintenance/Support — so the next tech with this issue would re-discover
|
||||
everything.
|
||||
|
||||
**Delivered**
|
||||
|
||||
- `migrations/075_itglue_audit.sql` — two new tables:
|
||||
- `itglue_asset_audits` — one row per audit run with full LLM context
|
||||
snapshot (asset traits at audit time, redacted), the gaps/promotions/
|
||||
contradictions output, and cost.
|
||||
- `itglue_writes` — one row per PATCH attempt; before/after diff, who, when,
|
||||
status pending → committed | failed | reverted, audit_id provenance,
|
||||
raw IT Glue API response, source_evidence (tickets that prompted the gap).
|
||||
- `lib/services/itglue-client.ts`:
|
||||
- `updateFlexibleAsset(id, traits)` — PATCH /flexible_assets/:id with
|
||||
JSON:API body.
|
||||
- `refreshFlexibleAsset(id)` thin wrapper around getFlexibleAsset.
|
||||
- `getRawSingle(path, params)` — for callers that need raw attributes
|
||||
(created-at/updated-at) for upserts.
|
||||
- `isITGlueConfigured()` helper.
|
||||
- Internal `patch(path, body)` mirrors the existing `request` pattern.
|
||||
- `lib/services/itglue-sync-service.ts`:
|
||||
- `refreshFlexibleAssetById(id)` per-record sync helper. Avoids running
|
||||
the full 27-entity `fullSync()` after every write.
|
||||
- `lib/permissions.ts`:
|
||||
- New `itglue: ['read', 'write']` permission. Admin + super-admin get write;
|
||||
user gets read-only.
|
||||
- `lib/services/analyzer/asset-audit/`:
|
||||
- `data-builder.ts` — collects all six context arms: asset snapshot,
|
||||
field schema with hints, peer exemplars same-client (top 5 by trait
|
||||
fill count), peer exemplars across all clients (top 3), per-field
|
||||
fill-rate stats (per-client + global), recent ticket fingerprints
|
||||
matching the asset's name. Redacts asset traits + every peer.
|
||||
- `prompt.ts` — single Sonnet/V4 Pro call; system prompt categorizes
|
||||
findings into field_gaps / notes_promotions / contradictions; refuses
|
||||
to suggest credential-shaped fields. Payload cap 80KB; trims peer_global
|
||||
first, then oldest tickets.
|
||||
- `runner.ts` — `runAssetAudit({assetId, generatedByUserId, provider})`.
|
||||
Provider-aware via `stageModelsFor(provider).deep_analysis`. Persists
|
||||
`itglue_asset_audits` row (or `failed` row on throw).
|
||||
- `persistence.ts` — typed read/write of both tables; `fieldNameToTraitKey()`
|
||||
helper matches IT Glue's `lower-hyphen-strip` convention.
|
||||
- `runner.test.ts` — 12 unit tests covering fillCount semantics, trait-key
|
||||
conversion, AssetAuditResponse Zod validation, payload trimming.
|
||||
- API routes:
|
||||
- `GET /api/analyzer/itglue/applications` — list of all Application records
|
||||
joined to latest audit, ordered worst-score-first.
|
||||
- `GET /api/analyzer/itglue/applications/[id]` — asset detail with field
|
||||
schema (the renderer uses field order + populated state).
|
||||
- `GET /api/analyzer/itglue/applications/[id]/audit?history=1` — latest
|
||||
audit + history.
|
||||
- `POST /api/analyzer/itglue/applications/[id]/audit` — runs a fresh audit;
|
||||
cost-guard via `recordCostAuditDecision({action:'itglue_audit'})`.
|
||||
- `POST /api/analyzer/itglue/applications/[id]/apply` — admin-only; inserts
|
||||
pending row, calls IT Glue PATCH, marks committed/failed, refreshes
|
||||
mirror, writes generic audit_log row.
|
||||
- `POST /api/analyzer/itglue/applications/[id]/revert/[writeId]` — admin-only;
|
||||
inserts a new write row with reversed before/after, applies, marks
|
||||
original status='reverted'.
|
||||
- `GET /api/analyzer/itglue/applications/[id]/writes` — auth-only per-asset
|
||||
history.
|
||||
- `GET /api/analyzer/itglue/writes` — admin-only cross-asset write log.
|
||||
- UI:
|
||||
- `/analyzer/itglue/applications` — list with score badges + filter input.
|
||||
- `/analyzer/itglue/applications/[id]` — asset header (with link to IT Glue),
|
||||
audit panel (gaps with severity-toned cards, notes promotions,
|
||||
contradictions), current fields rendered with hints for empty ones,
|
||||
write history with revert button, audit history with score timeline.
|
||||
`<ProviderToggle/>` reused from Phase 3.
|
||||
- `/admin/itglue-writes` — admin-only global write log with status filters.
|
||||
- Navigation: new "IT Glue audit" entry under the Analyzer dropdown.
|
||||
|
||||
**Audit-trail design**
|
||||
|
||||
Three layers of trail, all permanent:
|
||||
|
||||
1. `itglue_asset_audits` — every audit run with full LLM context.
|
||||
2. `itglue_writes` — every write attempt. Before/after, who, when, status,
|
||||
audit provenance. Reverts produce a new row with reversed diff; original
|
||||
row → `status='reverted'`. The chain is always traceable.
|
||||
3. Generic `audit_log` (existing migration 014) — written in parallel via
|
||||
`audit.log()`. Action `itglue.write` / `itglue.revert`, resource
|
||||
`flexible_asset`, resourceId = asset_id, details = `{ field_name, before,
|
||||
after, audit_id }`. Surfaces in `/admin/audit-log` next to every other
|
||||
admin action.
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Two domain-specific tables, not one generic events table.** Audits
|
||||
carry full LLM context (heavy, infrequent). Writes are atomic per-field
|
||||
decisions with hard-typed before/after diffs. The generic `audit_log`'s
|
||||
free-form JSONB doesn't model the diff cleanly — but we still write to it
|
||||
so admins see a unified feed.
|
||||
- **Admin-direct write, no two-step approval.** Per the user's call. The
|
||||
audit log is the safety net. We left `approval_requests` as a possible v2
|
||||
if mistakes start happening.
|
||||
- **Per-record sync helper instead of `fullSync()` after every write.**
|
||||
`refreshFlexibleAssetById()` does one GET + one upsert. Keeping
|
||||
`fullSync()` available for ops + scheduler; bypassing it on the write
|
||||
path keeps Apply latency under a second after the IT Glue PATCH lands.
|
||||
- **Credential refusal at three layers.** Prompt instructs the LLM not to
|
||||
suggest password/secret/key/token/credential fields. The Apply endpoint
|
||||
also regex-blocks any field name matching that pattern. The `redact()`
|
||||
utility from `itglue-redact.ts` strips matching keys from any payload
|
||||
flowing into the LLM in the first place.
|
||||
- **Trait-key derivation in code, not hard-coded.** IT Glue's convention is
|
||||
field-name lowercased, non-alphanum → single hyphen, stripped. We compute
|
||||
this from each field's `name` (verified against the live Hynes MISYS 6.3
|
||||
trait map). If a future field doesn't match the rule, it'd surface as an
|
||||
unmatched fill-rate / value-not-applied — easy to spot.
|
||||
- **Fill-rate computation in JS, not SQL.** Avoids one query per field. At
|
||||
the example data volume (~92 active Hynes assets, ~few-thousand globally
|
||||
per type) this stays sub-100ms; can revisit if it grows.
|
||||
- **Peer-exemplar ranking by populated-key count.** Cheap proxy for
|
||||
"well-documented." Doesn't penalize asset types whose fields are
|
||||
legitimately optional. If the LLM starts producing strange suggestions
|
||||
we can refine to require/expected-field weighting.
|
||||
- **Asset detail page renders from local mirror, not IT Glue API.** Means a
|
||||
user could see a stale value for a few seconds between Apply and the
|
||||
per-record sync landing. Acceptable for a read view; the API response
|
||||
from Apply returns the freshly-PATCH'd asset so the UI can immediately
|
||||
reflect the new state.
|
||||
|
||||
**Deliberately left out**
|
||||
|
||||
- v1 covers Applications only (`flexible_asset_type_id = 3790`). The schema
|
||||
generalizes (`asset_type` is a column), but Configurations / Procedures /
|
||||
Domains have different shapes and prompts. One type at a time.
|
||||
- No bulk-apply. Admin clicks each suggestion. If a single audit produces 10
|
||||
gaps that's 10 clicks — fine for v1; bulk-apply is an easy follow-on.
|
||||
- No two-step approval workflow.
|
||||
- No auto-create of new asset records — Apply only updates existing.
|
||||
- No inline editor for suggested values — admin sees the LLM's suggestion
|
||||
verbatim and clicks Apply, then edits in IT Glue if they want to tweak.
|
||||
- Passwords / Secrets / Keys / Tokens / Credentials: never written via
|
||||
this surface, ever. Refused at prompt + endpoint + redaction layers.
|
||||
|
||||
## Status after Phase 4
|
||||
|
||||
| Phase | Tests | tsc | Notes |
|
||||
|---|---|---|---|
|
||||
| 4 | 158 | clean | IT Glue asset audit (075), runner + 6 endpoints + 3 pages, write-back with revert |
|
||||
|
||||
---
|
||||
|
||||
## Phase 4.1 — Ticket-first capture + Configurations + cross-reference index
|
||||
|
||||
**Why**
|
||||
|
||||
Phase 4 was asset-first (admin browses worst-scoring records). Phase 4.1
|
||||
flips perspective: every time we analyze a ticket, learn whether *this
|
||||
ticket* taught us something documentable. Plus extends write-back to IT
|
||||
Glue **Configurations** (servers, workstations, devices) — the prior
|
||||
flexible-asset-only scope missed records like MISYS-SQL where the
|
||||
T20260502.0033 root cause actually lived. Plus a cross-reference table so
|
||||
both perspectives become one-query lookups, and the future RAG automation
|
||||
has its lookup index.
|
||||
|
||||
**Delivered**
|
||||
|
||||
- `migrations/076_itglue_ticket_xrefs.sql`:
|
||||
- `triggered_by_ticket_number` + `triggered_by_analysis_id` on
|
||||
`itglue_asset_audits`; `triggered_by_ticket_number` on `itglue_writes`
|
||||
(denormalized per user's call so "every write a ticket drove" is a
|
||||
direct query).
|
||||
- asset_type CHECK extended to include `'configuration'` on both audit +
|
||||
write tables.
|
||||
- New `itglue_ticket_xrefs` table — ticket↔asset linkage with
|
||||
relationship type (`referenced` | `updated` | `should_have_referenced`),
|
||||
source (`analyzer_referenced` | `audit_write` | `manual`), unique
|
||||
constraint preventing dup ingestion.
|
||||
- `lib/services/itglue-client.ts`:
|
||||
- `updateConfiguration(id, attributes)` — PATCH /configurations/:id with
|
||||
flat JSON:API attributes (no traits blob).
|
||||
- `refreshConfiguration(id)` thin wrapper.
|
||||
- `lib/services/itglue-sync-service.ts`:
|
||||
- `refreshConfigurationById(id)` per-record refresh (mirrors the bulk
|
||||
syncConfigurations 33-column upsert).
|
||||
- `lib/services/analyzer/asset-audit/`:
|
||||
- `data-builder.ts` generalized: dispatches on `assetType`, supports
|
||||
`ticketScopeAnalysisId` for ticket-first audits. Configurations get a
|
||||
hand-curated 16-field schema with hints (since IT Glue Configurations
|
||||
don't have a `_fields` table). Synthesizes a "traits" map from flat
|
||||
columns so the prompt stays type-agnostic.
|
||||
- `prompt.ts` — two system prompts (Application-flavored vs
|
||||
Configuration-flavored, the latter focused on hostname/FQDN, OS
|
||||
currency, named services, IP/MAC, contact ownership). Ticket-scoped
|
||||
suffix when an analysis is the source so the LLM frames findings as
|
||||
"what did this ticket teach us?"
|
||||
- `runner.ts` accepts `assetType`, `ticketScopeAnalysisId`; persists
|
||||
`triggered_by_*` columns.
|
||||
- `persistence.ts` — `asset_type` union extended; new `getLatestTicketScopedAudit`
|
||||
helper for the analysis-page panel; `createPendingWrite` accepts asset_type
|
||||
+ triggered_by_ticket_number.
|
||||
- `xrefs.ts` (new) — bulk-insert helpers: `insertReferencedXrefsFromAnalysis`
|
||||
(post-analysis hook), `insertUpdatedXref` (post-apply hook), with
|
||||
listXrefsForAsset / listXrefsForTicket queries.
|
||||
- `asset-matcher.ts` (new) — given an analysis_id, returns matched
|
||||
flexible_assets + configurations for the ticket's client based on
|
||||
fingerprint terms (applications_involved, device_classes, vendors_involved).
|
||||
Score = exact (3) > word-boundary (2) > substring (1); top 5 per kind.
|
||||
- `lib/services/analyzer/worker.ts` — post-analysis hook calls
|
||||
`insertReferencedXrefsFromAnalysis` for every doc the LLM cited;
|
||||
best-effort, never fails the job.
|
||||
- API routes:
|
||||
- `GET /api/analyzer/analyses/[id]/itglue-suggestions` — match assets +
|
||||
return any existing ticket-scoped audits keyed by (assetType, assetId).
|
||||
- `POST /api/analyzer/analyses/[id]/itglue-suggestions` — body
|
||||
`{ assetType, assetId, provider }`, runs a ticket-scoped audit.
|
||||
- Full Configurations route tree mirroring Applications: list, detail,
|
||||
audit (GET/POST), apply (admin), revert (admin), writes, xrefs.
|
||||
- `GET /api/analyzer/applications/[id]/xrefs` (new) and
|
||||
`GET /api/analyzer/tickets/[ticketNumber]/itglue-xrefs` (new).
|
||||
- UI:
|
||||
- `<ItglueSuggestionsPanel/>` rendered on the analysis detail page.
|
||||
Opt-in trigger ("Check IT Glue documentation"); shows matched
|
||||
Applications + Configurations grouped, per-asset ticket-scoped audit
|
||||
buttons, inline gap cards with Apply (admin-only), score badges.
|
||||
Reuses `<ProviderToggle/>` from Phase 3.
|
||||
- `/analyzer/itglue/configurations` — list mirroring Applications.
|
||||
- `/analyzer/itglue/configurations/[id]` — detail mirroring Applications,
|
||||
plus the new "Tickets that touched this configuration" section.
|
||||
- Application detail page — added the same xref section.
|
||||
- Navigation split into "IT Glue — Applications" + "IT Glue — Configurations".
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Two separate Configuration write methods + two route trees instead of
|
||||
one polymorphic surface.** The codebase has no other `[assetType]`-style
|
||||
polymorphism; existing patterns favor parallel resource paths. Added
|
||||
~80 LOC duplication on the page side, but each surface is independently
|
||||
testable + obvious in URL routing.
|
||||
- **Configuration field schema is hand-curated**, not loaded from IT Glue.
|
||||
IT Glue exposes flexible-asset field metadata via `/flexible_asset_fields`
|
||||
but Configurations have no equivalent endpoint. The 16 hand-written
|
||||
hints (in `data-builder.ts`) are what the LLM sees as field documentation.
|
||||
Versioned in code; PR review is the change control.
|
||||
- **Apply on Configurations is column-allowlisted.** Even with the audit
|
||||
pipeline picking `field_name`, the apply route refuses anything outside
|
||||
`name | hostname | primary_ip | mac_address | serial_number | asset_tag |
|
||||
position | notes | operating_system_notes`. Stops the LLM from suggesting
|
||||
edits to read-only/derived fields like `manufacturer_id` (which is an FK
|
||||
resolved by IT Glue, not a free-text field).
|
||||
- **xref ingestion happens in the worker after a successful analysis**,
|
||||
not as a separate batch job. Best-effort wrap means a transient DB
|
||||
hiccup never fails the analysis itself. The unique index on the xref
|
||||
table ensures retries are idempotent.
|
||||
- **ticketScopeAnalysisId narrows ticket evidence to one row.** This is
|
||||
the key prompt-shaping decision for Phase 4.1: the LLM sees just the
|
||||
one analysis the user clicked from, plus the asset state + schema + peer
|
||||
exemplars. Findings frame as "what *this ticket* revealed" rather than
|
||||
all-time history.
|
||||
- **No backfill of existing analyses.** Per user's call. The xref table
|
||||
fills forward; backfill is a future opt-in script if needed.
|
||||
- **Asset matching is loose** — substring + word-boundary. A ticket
|
||||
mentioning "MISYS" matches both `MISYS 6.3` (the Application) and
|
||||
`MISYS-SQL` (the Configuration), and the user picks per-asset which to
|
||||
audit. Less false-negative-y than strict matching; user controls
|
||||
confirmation.
|
||||
- **Configuration audits don't write to manufacturer/model/OS-name/contact/location**
|
||||
— those are FK fields IT Glue resolves by id, not free-text. The
|
||||
audit prompt can suggest changes but Apply blocks them. Future iteration
|
||||
could resolve names → ids via the IT Glue manufacturers/models endpoints.
|
||||
|
||||
**Deliberately left out**
|
||||
|
||||
- **Datto RMM script execution** (Phase 4.2 — separate plan). Ability to
|
||||
run PowerShell via Datto RMM Overshell on Wulf Nurse endpoints to gather
|
||||
fresh evidence (DHCP scopes, DNS zones, AD info, named services) and
|
||||
feed it into the audit pipeline. Decisions logged: generic Overshell
|
||||
component + Pulse-managed scripts; admin-direct with audit log; new
|
||||
`rmm.execute` permission.
|
||||
- No backfill of the xref table.
|
||||
- No bulk-apply across multiple gaps; admin clicks each one.
|
||||
- No two-step approval workflow. Audit log + role gating remain the
|
||||
safety net.
|
||||
- No Configuration write for FK-shaped fields (manufacturer, model, OS,
|
||||
contact, location) — only flat editable columns.
|
||||
|
||||
## Status after Phase 4.1
|
||||
|
||||
| Phase | Tests | tsc | Notes |
|
||||
|---|---|---|---|
|
||||
| 4.1 | 160 | clean | xref table (076), Configurations parity, ticket-first capture, analysis-page panel |
|
||||
|
||||
---
|
||||
|
||||
## Phase 4.2 — Datto RMM Overshell evidence pipeline
|
||||
|
||||
**Why**
|
||||
|
||||
Phase 4.1 wires ticket history + IT Glue field schemas into LLM-driven
|
||||
documentation suggestions. The next leverage point is **fresh evidence
|
||||
from the live environment** — service lists, AD health, DHCP scopes, DNS
|
||||
zones, event logs — that ticket history can't surface. Without it, audits
|
||||
flag "the named service that processes BarTender scan-folder text files
|
||||
isn't documented" but can't suggest the actual service name. With it, we
|
||||
suggest the literal value pulled from the running server seconds ago.
|
||||
|
||||
The proven test case: openclaw produced an AD health summary at Hynes
|
||||
around 2026-04-25 (IP conflicts, ZR006 missing trust account, DNS
|
||||
forwarders timing out, Hendricks site missing site-links) by orchestrating
|
||||
Datto RMM Overshell. Phase 4.2 lets Pulse produce the same intel directly
|
||||
from a button on the asset page, store it, and feed it back into audits.
|
||||
|
||||
**Delivered**
|
||||
|
||||
- `migrations/077_rmm_overshell.sql`:
|
||||
- `rmm_settings` singleton — caches the discovered Overshell `component_uid`,
|
||||
`component_name`, `variable_name` (default `CommandLine`).
|
||||
- `rmm_executions` — full lifecycle row per dispatch: queued → running →
|
||||
complete | failed | timeout. Captures `target_device_uid`,
|
||||
`target_hostname`, `target_company_id`, optional audit/asset linkage,
|
||||
`job_uid`, raw stdout/stderr (redacted), `parsed_evidence`, exit code,
|
||||
`timeout_at`. 8 indexes covering all query paths the audit pipeline +
|
||||
UI need.
|
||||
- `lib/services/rmm/scripts/`:
|
||||
- 7 v1 scripts, all read-only. Each is a typed `RmmScript` exporting
|
||||
`body` (PowerShell), `target_type`, `parseOutput`, `expected_runtime_seconds`,
|
||||
`version`. Bodies end with `ConvertTo-Json -Depth … -Compress` so the
|
||||
parser is just `JSON.parse`. Registry validates uniqueness at load.
|
||||
- asset_self: `get-services`, `get-installed-software`, `get-event-log-recent`.
|
||||
- site_anchor: `get-ad-health` (mirrors the openclaw test case),
|
||||
`get-dhcp-scopes`, `get-dns-zones`, `get-network-discovery` (catches
|
||||
the IP-conflict pattern from the proven test case).
|
||||
- `lib/services/rmm/target-resolver.ts`:
|
||||
- `resolveSiteAnchorTarget(companyId)` — looks up `datto_rmm_sites`
|
||||
where `autotask_company_id = $1`, finds devices matching
|
||||
`^[A-Z]{3}[A-Z]{3}WNP\d{2}$`, picks online + lowest numeric suffix.
|
||||
- `resolveAssetSelfTarget(deviceUid)` — direct lookup.
|
||||
- `resolveDeviceByHostname(hostname)` — fallback when an IT Glue
|
||||
Configuration's `rmm_id` doesn't resolve cleanly.
|
||||
- `lib/services/rmm/settings.ts`:
|
||||
- `discoverOvershellComponent()` — calls
|
||||
`client.findOvershellComponent(/overshell/i)` and persists the uid.
|
||||
- `resolveOvershellComponent()` — read-cache-or-discover; throws if
|
||||
nothing matches.
|
||||
- `lib/services/rmm/executor.ts`:
|
||||
- `queueExecution({ scriptId, target, performedByUserId, triggeredByAuditId? })`
|
||||
— validates registry, resolves target, runs cost-guard rate limit
|
||||
(50/user/24h, decision logged to `analyzer_cost_audit`), inserts
|
||||
pending row, calls `runQuickJob`, captures `jobUid`, flips to
|
||||
`running`. Generic `audit.log` entry on success.
|
||||
- `lib/services/rmm/worker.ts`:
|
||||
- 5-second poll loop, self-init pattern matching `analyzerWorker`.
|
||||
- Sweeps timed-out rows first (status → `timeout`).
|
||||
- Polls `running` rows via `client.getJobResults` per `target_device_uid`.
|
||||
On terminal status: redacts stdout/stderr, runs script's `parseOutput`,
|
||||
persists. Parse errors are non-fatal — raw output still kept.
|
||||
- `lib/services/rmm/persistence.ts`:
|
||||
- Typed `RmmExecutionRow` + status helpers, plus the audit-pipeline
|
||||
queries `listLatestEvidenceForCompany(companyId, days)` and
|
||||
`listLatestEvidenceForAsset(assetType, assetId)`.
|
||||
- `lib/services/datto-rmm-client.ts` — added `findOvershellComponent(pattern)`.
|
||||
- `lib/services/analyzer/asset-audit/data-builder.ts`:
|
||||
- 7th LLM context arm `rmm_evidence` populated from
|
||||
`listLatestEvidenceForCompany` (site-anchored, last 7 days) +
|
||||
`listLatestEvidenceForAsset` (asset-self, all-time).
|
||||
- Joins via `companies → itg_organizations` on case-insensitive
|
||||
`company_name` match (same join the ticket-evidence loader uses).
|
||||
- `lib/services/analyzer/asset-audit/prompt.ts`:
|
||||
- New `=== LIVE RMM EVIDENCE ===` section emits when
|
||||
`ctx.rmm_evidence.length > 0`. Trim path drops it last (highest-value
|
||||
section).
|
||||
- `LIVE_EVIDENCE_NOTE` injected into the system prompt: *"Treat parsed
|
||||
contents as authoritative current state … Cite execution_id alongside
|
||||
ticket numbers."*
|
||||
- `lib/permissions.ts` — new `rmm: ['read','execute']`. Admin + super-admin
|
||||
get both; user gets read.
|
||||
- API routes:
|
||||
- `GET/PATCH /api/admin/rmm/settings` — admin-only, view + edit variable name.
|
||||
- `POST /api/admin/rmm/settings/discover` — admin-only, force component scan.
|
||||
- `GET /api/rmm/scripts` — auth, library catalog (no bodies).
|
||||
- `GET/POST /api/rmm/executions` — list (auth) + queue (`rmm.execute`).
|
||||
- `GET /api/rmm/executions/[id]` — auth, polls one execution.
|
||||
- `GET /api/analyzer/itglue/sites/[companyId]` — site-discovery summary.
|
||||
- UI:
|
||||
- `<RmmScriptPicker filter='site_anchor'|'asset_self' …/>` — popover
|
||||
listing applicable scripts, dispatches on click, disables for
|
||||
non-admins.
|
||||
- `<RmmExecutionStream/>` — polls every 3s, shows status + parsed
|
||||
evidence + raw stdout (collapsible).
|
||||
- `/admin/rmm-overshell` — settings + recent execution log.
|
||||
- `/analyzer/itglue/sites/[companyId]` — site-discovery view.
|
||||
- Embedded picker on Application detail (site-anchor with parent
|
||||
client) and Configuration detail (asset-self if `rmm_id` resolves to
|
||||
a Datto device).
|
||||
- Nav entry: Admin → "RMM Overshell".
|
||||
|
||||
**Decisions worth flagging**
|
||||
|
||||
- **Component discovery is automatic and cached.** Pulse scans for any
|
||||
component matching `/overshell/i` on first dispatch, persists the uid,
|
||||
and never re-scans unless an admin clicks "Re-discover". The variable
|
||||
name defaults to `CommandLine` (Datto's "Run Command" component). If
|
||||
Wulf's Overshell uses a different variable, admin sets it once via
|
||||
`/admin/rmm-overshell`.
|
||||
- **Script bodies live in code, not the DB.** Three reasons: PR review is
|
||||
the change-control mechanism; nothing in the database is treated as
|
||||
executable PowerShell; the 7 scripts are already curated and we don't
|
||||
need (or want) ad-hoc paste-a-script UX.
|
||||
- **WNP-only target resolution.** Site-anchored scripts hit the Wulf
|
||||
Nurse Production endpoint (`LLLCCCWNPNN`); PowerShell uses native AD
|
||||
cmdlets to reach the DC over the network. Direct-to-DC role detection
|
||||
is a fast-follow.
|
||||
- **5-minute hard timeout + 50/user/24h rate limit.** Both enforced
|
||||
server-side in the executor. The cost-guard rows in `analyzer_cost_audit`
|
||||
give admins a unified view of LLM and RMM activity per user.
|
||||
- **Output is redacted before persistence.** Same `redact()` from the
|
||||
IT Glue redaction module; strips any password/secret/key/token/credential
|
||||
keyed values from stdout/stderr before the parser sees them.
|
||||
- **Live RMM evidence trims last.** When the audit prompt overflows the
|
||||
80KB cap, peer_global → ticket_evidence → rmm_evidence (in that order).
|
||||
Live evidence is the most novel signal; it's worth keeping.
|
||||
- **Worker is in-process, not a separate service.** Same auto-start
|
||||
pattern as `analyzerWorker`. `RMM_WORKER_AUTOSTART=1` opt-in for dev.
|
||||
Multiple Next.js workers are safe — each row's `jobUid` is set once and
|
||||
the poll loop is idempotent.
|
||||
- **`getJobResults` response shape is variable across Datto tenants.** The
|
||||
worker handles both top-level and `results[*]` payloads, picks the
|
||||
per-device result when present, and falls back to the first array entry.
|
||||
|
||||
**Deliberately left out**
|
||||
|
||||
- **No backfill of existing Overshell jobs.** Per user's call. Pulse
|
||||
starts capturing from the first dispatch.
|
||||
- **No DC-role detection.** WNP-only. Add later if AD scripts that need
|
||||
native DC execution become important.
|
||||
- **No ad-hoc PowerShell paste-in.** Only registry-listed scripts run.
|
||||
- **No openclaw integration.** Phase 4.2 talks directly to Datto RMM.
|
||||
- **No audit-driven auto-execution.** v1 is admin-clicks-button. The
|
||||
audit panel will gain a "Run Get-Services to fill this gap?" prompt in
|
||||
a fast-follow once we trust the safety layers.
|
||||
- **No Overshell write operations.** All scripts are read-only / discovery.
|
||||
Configuration changes happen via IT Glue (Phase 4) or manually.
|
||||
- **No per-script per-user permissions.** Anyone with `rmm.execute` can
|
||||
run any script. Per-script gating is a fast-follow if needed.
|
||||
- **No credential output ever.** Three-layer refusal:
|
||||
1. Script library has no credential-handling scripts; tests verify
|
||||
bodies don't reference `$plaintext` password patterns.
|
||||
2. `redact()` strips matching keys from output before persistence.
|
||||
3. The audit prompt's existing credential refusal applies to anything
|
||||
that does sneak through.
|
||||
|
||||
## Status after Phase 4.2
|
||||
|
||||
| Phase | Tests | tsc | Notes |
|
||||
|---|---|---|---|
|
||||
| 4.2 | 174 | clean | RMM Overshell pipeline (077), 7 scripts, executor + worker, audit-context arm |
|
||||
|
||||
## Phase 4.3 — LogLift event-log ingestion
|
||||
|
||||
### Why
|
||||
|
||||
Overshell stdout caps around ~50KB practical — fine for service lists
|
||||
or installed-software dumps, too small for full Windows event logs
|
||||
across critical/error/warning levels. Wulf already runs a richer
|
||||
collector via n8n: PowerShell on each endpoint gathers logs + system
|
||||
context, gzips it, uploads to a Backblaze B2 bucket
|
||||
(`wulf-audits` / `us-west-002`), then POSTs metadata. n8n decompresses,
|
||||
runs an LLM analysis, and posts a Telegram summary.
|
||||
|
||||
Phase 4.3 makes Pulse the receiver instead of n8n so:
|
||||
|
||||
- LogLift evidence lands in the same `rmm_executions` table 4.2 introduced.
|
||||
- The audit pipeline's `rmm_evidence` arm picks it up automatically.
|
||||
- Admins can dispatch a LogLift run directly from the Configuration page.
|
||||
- Successful uploads matched to a unique IT Glue Configuration auto-fire
|
||||
an asset-first audit so documentation suggestions surface immediately.
|
||||
|
||||
### Shape
|
||||
|
||||
`migrations/078_loglift_uploads.sql` adds three columns to
|
||||
`rmm_executions` (`transport`, `evidence_object_key`, `run_id` — with a
|
||||
unique index on `run_id`), and three to `rmm_settings`
|
||||
(`loglift_component_uid`, `loglift_component_name`,
|
||||
`loglift_discovered_at`). The transport column has a CHECK constraint
|
||||
restricting it to `overshell_stdout` | `b2_upload`.
|
||||
|
||||
`lib/services/b2/client.ts` is a from-scratch SigV4 implementation
|
||||
ported from `docs/LogLift Review.json`: presigned GET + PUT (different
|
||||
expiries), 25MB hard download cap, path-traversal-safe object key regex,
|
||||
and a `B2NotConfiguredError` when env vars are missing. 8 tests cover
|
||||
the regex + signature stability + signing-key derivation.
|
||||
|
||||
`lib/services/rmm/scripts/loglift-eventlogs.ts` registers the script:
|
||||
`target_type='asset_self'`, `transport='b2_upload'`, empty body (the
|
||||
collector PowerShell lives in the Datto-registered LogLift component, not
|
||||
in Pulse). The registry's body-length sanity test skips `b2_upload`
|
||||
scripts.
|
||||
|
||||
### Dispatch path
|
||||
|
||||
`executor.ts` forks on `script.transport`:
|
||||
|
||||
- `overshell_stdout` (default) — unchanged 4.2 path: resolve Overshell
|
||||
component, dispatch with `{Variable: body}`, worker polls for stdout.
|
||||
- `b2_upload` — new fork. Resolves the Datto site uid from the device,
|
||||
resolves the LogLift component (discover-on-demand), generates a
|
||||
`runId` (`pulse_<hex>_<ms>`), inserts an `rmm_executions` row with
|
||||
`transport='b2_upload'`, dispatches the Quick Job with variables
|
||||
`RunId`, `ClientId`, `WebhookUrl`, `WebhookSecret`. The persisted
|
||||
`variables` column strips `WebhookSecret` so admins can read the row
|
||||
without exposing the OPENCLAW key.
|
||||
|
||||
### Receive path
|
||||
|
||||
`POST /api/rmm/loglift/upload` (public per `middleware.ts`,
|
||||
`x-openclaw-key` validated):
|
||||
|
||||
1. Zod validate body + object-key regex.
|
||||
2. Resolve `clientId` (Datto site uid) → `datto_rmm_sites.id` →
|
||||
`autotask_company_id` (FK or name fallback — same as 4.2 multi-site
|
||||
work).
|
||||
3. Resolve `computerName` → Datto device uid (case-insensitive).
|
||||
4. Resolve `computerName` + company → `itg_configurations.id`. Two-pass
|
||||
(count + fetch) sets `single_match=true` only when exactly one
|
||||
Configuration matches.
|
||||
5. Correlate to a Pulse-dispatched execution by `run_id`. If no match
|
||||
(out-of-band collector), insert a fresh `running` row.
|
||||
6. Download from B2 (25MB cap), gunzip with zip-bomb guard
|
||||
(refuse > 100MB inflated, checked via gzip ISIZE before decompression
|
||||
and again after).
|
||||
7. Slim: keep `system_context` + `summary` + top 100 events sorted by
|
||||
severity (Critical → Error → Warning → Info), then recency. Drop the
|
||||
raw `events` array; the full gzip stays in B2 forever.
|
||||
8. `redact()` the slim object, persist with `markExecutionFromB2Upload`.
|
||||
9. Auto-audit hook: if Configuration matched single, fire
|
||||
`runAssetAudit({assetType:'configuration', assetId})` synchronously
|
||||
(still in the webhook handler — the LLM call is the bottleneck but
|
||||
the agent doesn't care about webhook latency past ~30s). On failure,
|
||||
log + continue — webhook still 200s.
|
||||
|
||||
`audit_log` actions: `rmm.loglift.dispatched`, `rmm.loglift.received`,
|
||||
`rmm.loglift.matched`, `rmm.loglift.audit_triggered`.
|
||||
|
||||
### Worker change
|
||||
|
||||
`worker.ts` filters `transport='b2_upload'` rows out of the running
|
||||
poll list — no stdout to fetch. The 5-minute timeout sweep still
|
||||
applies; stuck rows get marked `timeout`.
|
||||
|
||||
### Prompt update
|
||||
|
||||
`LIVE_EVIDENCE_NOTE` extended to teach the LLM about the LogLift slim
|
||||
shape: cite events as `event:<EventId>` or `execution:<id>`,
|
||||
`event_count_total` is the original count (top 100 only in the prompt),
|
||||
and `system_context` is authoritative for OS / hardware / disk / memory
|
||||
facts on the matched Configuration.
|
||||
|
||||
### UI surfaces
|
||||
|
||||
- `/admin/rmm-overshell` gets a second "LogLift component" block with a
|
||||
"Re-discover LogLift" button next to the existing Overshell discovery.
|
||||
`discoverLogliftComponent()` matches `/loglift|eventlog/i`.
|
||||
- Configuration page picker (filtered by `target_type='asset_self'`)
|
||||
surfaces the LogLift entry automatically — Phase 4.2's executor + UI
|
||||
scaffolding handles it through the new dispatch fork.
|
||||
|
||||
### Files
|
||||
|
||||
**New:** `migrations/078_loglift_uploads.sql`, `lib/services/b2/client.ts`
|
||||
(+ test), `lib/services/rmm/scripts/loglift-eventlogs.ts`,
|
||||
`lib/services/rmm/loglift-matcher.ts`,
|
||||
`lib/services/rmm/loglift-receiver.ts`,
|
||||
`app/api/rmm/loglift/upload/route.ts`,
|
||||
`app/api/admin/rmm/settings/discover-loglift/route.ts`,
|
||||
`docs/loglift-eventlog-pipeline-spec.md`.
|
||||
|
||||
**Modified:** `lib/services/datto-rmm-client.ts` (generalized
|
||||
`findOvershellComponent` → `findComponentByName`),
|
||||
`lib/services/rmm/settings.ts` (LogLift discover/resolve),
|
||||
`lib/services/rmm/persistence.ts` (transport + run_id + new
|
||||
`findExecutionByRunId`, `createOutOfBandUploadExecution`,
|
||||
`markExecutionFromB2Upload`), `lib/services/rmm/executor.ts` (b2_upload
|
||||
fork), `lib/services/rmm/worker.ts` (skip b2_upload poll),
|
||||
`lib/services/rmm/scripts/index.ts` (register), `…/scripts/types.ts`
|
||||
(transport field), `…/scripts/registry.test.ts` (8-script expectation +
|
||||
b2_upload body skip), `lib/services/analyzer/asset-audit/prompt.ts`
|
||||
(LIVE_EVIDENCE_NOTE), `app/admin/rmm-overshell/page.tsx`
|
||||
(LogLift block + button), `middleware.ts` (`/api/rmm/loglift` public).
|
||||
|
||||
### Refusals + guards
|
||||
|
||||
1. Object-key regex (`^[A-Za-z0-9_-]+/[A-Za-z0-9_.-]+/eventlogs_[0-9_]+\.json\.gz$`).
|
||||
2. B2 25MB download cap.
|
||||
3. Decompress 100MB cap (gzip ISIZE pre-check + post-inflate re-check).
|
||||
4. `redact()` on slim payload before persistence.
|
||||
5. Auto-audit only on single-match Configurations — multiple matches
|
||||
logged + skipped.
|
||||
6. Webhook secret stripped from persisted `variables` column.
|
||||
|
||||
### Status after Phase 4.3
|
||||
|
||||
| Phase | Tests | tsc | Notes |
|
||||
|---|---|---|---|
|
||||
| 4.3 | TBD (target ~182) | TBD | LogLift pipeline (078), B2 SigV4, b2_upload transport, slim + auto-audit |
|
||||
|
|
|
|||
|
|
@ -485,3 +485,515 @@ either caching the current hash on the tickets row (sync change) or
|
|||
computing it on read for the visible page (slow). The date heuristic
|
||||
gets ~95% of the value at zero compute cost; revisit when there's real
|
||||
load signal.
|
||||
|
||||
---
|
||||
|
||||
## Link-aware bundles (Phase 3)
|
||||
|
||||
A "bundle" is an aggregate report launched from a single ticket page —
|
||||
typically a master/problem ticket that names other tickets in its
|
||||
description. Instead of forcing the user to analyze each constituent
|
||||
manually and then visit `/analyzer/reports/new`, the bundle endpoint
|
||||
fans out per-ticket analyses and chains them into an aggregate report
|
||||
automatically.
|
||||
|
||||
### How it works end-to-end
|
||||
|
||||
1. User loads `/analyzer/ticket/<ticket-number>`.
|
||||
2. `<RelatedTicketsPanel/>` calls `GET /api/analyzer/tickets/:tn/links`
|
||||
(cheap, no LLM) — regex extraction over the description and retained
|
||||
notes for `T\d{8}\.\d{4}` references, plus the structured
|
||||
`RELATED TICKETS:` block detector and `problem_ticket_id` resolution.
|
||||
The panel renders only when refs exist or the ticket looks like a
|
||||
problem ticket.
|
||||
3. (Optional) User flips the "AI-suggest more" Switch — this POSTs the
|
||||
same endpoint with `includeSuggested: true`, runs one Haiku pass over
|
||||
recent same-company tickets (±30 days, capped at 50 candidates), and
|
||||
returns up to 5 suggestions with one-line reasons.
|
||||
4. User clicks **"Analyze with N linked tickets"**. The panel POSTs to
|
||||
`/api/analyzer/tickets/:tn/analyze-bundle` with
|
||||
`linkedTicketNumbers: [...]`.
|
||||
5. The bundle endpoint:
|
||||
- Verifies every ticket exists locally (one SQL round-trip).
|
||||
- Per ticket: idempotency-checks via content hash. If a complete
|
||||
analysis exists, it's reused; otherwise a fresh `analyzer_jobs` row
|
||||
is queued.
|
||||
- Cost-guard runs against **new work only** plus the aggregate-reduce
|
||||
step. $5 confirmation threshold and $50 daily hard block are the
|
||||
same gates as standalone aggregate reports.
|
||||
- Inserts an `analyzer_aggregate_reports` row in `'pending_analyses'`
|
||||
state with `expected_ticket_numbers` populated (or straight to
|
||||
`'pending'` and immediately fires the runner if everything was
|
||||
already complete).
|
||||
6. Worker polls and runs each queued job. After each successful
|
||||
analysis, `chainTriggerForCompletedAnalysis()` looks up bundles
|
||||
waiting on that ticket, appends the new analysis_id, and (if the full
|
||||
set is now satisfied) flips status to `'pending'` and fires
|
||||
`runAggregateReport()`.
|
||||
7. Frontend polls `GET /api/analyzer/aggregate-reports/:id` every 3s and
|
||||
navigates to `/analyzer/reports/:id` on completion.
|
||||
|
||||
### Status state machine
|
||||
|
||||
```
|
||||
pending_analyses ──── all expected analyses complete ────► pending
|
||||
│
|
||||
▼
|
||||
running
|
||||
│
|
||||
▼
|
||||
complete | failed
|
||||
```
|
||||
|
||||
`'pending_analyses'` is the new state Phase 3 introduces.
|
||||
Manual-multi-select reports created via `/analyzer/reports/new` skip it
|
||||
and start at `'pending'` (their analyses must already be complete to
|
||||
even submit).
|
||||
|
||||
### Inspecting a bundle
|
||||
|
||||
```sql
|
||||
SELECT id, status, ticket_count,
|
||||
array_length(expected_ticket_numbers, 1) AS expected,
|
||||
array_length(analysis_ids, 1) AS collected,
|
||||
triggered_by_ticket_number,
|
||||
generated_at
|
||||
FROM analyzer_aggregate_reports
|
||||
WHERE expected_ticket_numbers IS NOT NULL
|
||||
ORDER BY generated_at DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
Find which expected tickets a stuck `pending_analyses` bundle is still
|
||||
waiting on:
|
||||
|
||||
```sql
|
||||
WITH r AS (
|
||||
SELECT id, expected_ticket_numbers, analysis_ids
|
||||
FROM analyzer_aggregate_reports
|
||||
WHERE id = '<report-id>'
|
||||
)
|
||||
SELECT etn.ticket_number,
|
||||
(SELECT bool_or(aa.id = ANY(r.analysis_ids))
|
||||
FROM analyzer_analyses aa
|
||||
WHERE aa.ticket_number = etn.ticket_number
|
||||
AND aa.status = 'complete') AS has_collected_analysis
|
||||
FROM r,
|
||||
LATERAL UNNEST(r.expected_ticket_numbers) AS etn(ticket_number);
|
||||
```
|
||||
|
||||
The `false` rows are the tickets we're still waiting on. Cross-reference
|
||||
with `analyzer_jobs` filtered by those ticket numbers to see whether the
|
||||
job is queued, in-flight, or failed.
|
||||
|
||||
### Cost shape
|
||||
|
||||
For a typical 4-ticket problem bundle on fresh tickets:
|
||||
|
||||
| Step | Model | Approx cost |
|
||||
|---|---|---|
|
||||
| Link discovery (explicit) | none | ~free |
|
||||
| AI-suggested arm (if toggled) | Haiku | ~$0.005 |
|
||||
| Per-ticket pipeline × 4 | Haiku → Sonnet (+ optional Opus) | $0.20 – $1.20 |
|
||||
| Aggregate reduce | Opus | ~$0.50 |
|
||||
| **Total** | | **~$1 – $2** |
|
||||
|
||||
The bundle endpoint's per-ticket estimate is a flat $0.15 (pessimistic
|
||||
Sonnet) used purely for the cost guard. Real spend is captured per row
|
||||
on `analyzer_analyses.estimated_cost_usd` once each pipeline run
|
||||
completes.
|
||||
|
||||
### When the panel doesn't render
|
||||
|
||||
The panel is intentionally invisible on tickets that aren't candidates
|
||||
for bundling:
|
||||
|
||||
- No `T<YYYYMMDD>.<####>` references found in description or retained
|
||||
notes
|
||||
- `tickets.problem_ticket_id` is null
|
||||
- Title contains neither "master problem ticket" nor "problem ticket"
|
||||
|
||||
If a user expects to see the panel and doesn't, the most common reason
|
||||
is that the referenced tickets aren't in our local mirror yet (sync
|
||||
gap) — `discoverExplicitLinks` filters refs against
|
||||
`tickets.ticket_number` to keep ghost links out of the UI. Run the
|
||||
ticket sync and reload.
|
||||
|
||||
### Suggested-arm limits
|
||||
|
||||
The Haiku call drops any suggestion whose ticket_number isn't in the
|
||||
candidate list it was given (hallucination guard). Suggestions are
|
||||
capped at 5 and never auto-included — the user has to tick the
|
||||
checkbox. If the LLM call throws, the failure is logged and the panel
|
||||
still shows the explicit refs (the suggestion arm is opportunistic, not
|
||||
load-bearing).
|
||||
|
||||
---
|
||||
|
||||
## IT Glue asset audits (Phase 4)
|
||||
|
||||
The asset-audit pipeline analyzes one IT Glue Application record at a time
|
||||
against ticket history + IT Glue's own field schema, surfaces documentation
|
||||
gaps and "promote-from-Notes" suggestions, and lets admins push approved
|
||||
changes back to IT Glue. Every change is recorded in three audit layers
|
||||
(see Phase 4 build notes).
|
||||
|
||||
Permissions:
|
||||
|
||||
- Read audit / run audit → any authenticated user (cheap, ~$0.01).
|
||||
- Apply or Revert → `requirePermission('itglue', 'write')` — admin or
|
||||
super-admin only.
|
||||
- `/admin/itglue-writes` → admin-only.
|
||||
|
||||
### Inspecting audits
|
||||
|
||||
```sql
|
||||
-- Latest audit per asset, lowest scoring first
|
||||
SELECT a.asset_id,
|
||||
fa.name AS application_name,
|
||||
fa.organization_name,
|
||||
a.overall_score,
|
||||
a.ticket_count,
|
||||
jsonb_array_length(a.field_gaps) AS gap_count,
|
||||
jsonb_array_length(a.notes_promotions) AS promo_count,
|
||||
a.provider, a.model_used,
|
||||
a.estimated_cost_usd,
|
||||
a.generated_at
|
||||
FROM itglue_asset_audits a
|
||||
JOIN itg_flexible_assets fa ON fa.id = a.asset_id::bigint
|
||||
WHERE a.status = 'complete'
|
||||
ORDER BY a.asset_id, a.generated_at DESC;
|
||||
```
|
||||
|
||||
```sql
|
||||
-- Failed audits (forensics)
|
||||
SELECT id, asset_id, generated_at, provider, model_used,
|
||||
LEFT(error_message, 200) AS error
|
||||
FROM itglue_asset_audits
|
||||
WHERE status = 'failed'
|
||||
ORDER BY generated_at DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
### Inspecting writes
|
||||
|
||||
```sql
|
||||
-- All committed writes in the last 7 days, with provenance
|
||||
SELECT w.performed_at,
|
||||
w.field_name,
|
||||
w.before_value, w.after_value,
|
||||
w.performed_by_user_id,
|
||||
w.audit_id,
|
||||
fa.name AS application_name,
|
||||
fa.organization_name
|
||||
FROM itglue_writes w
|
||||
JOIN itg_flexible_assets fa ON fa.id = w.asset_id::bigint
|
||||
WHERE w.status = 'committed'
|
||||
AND w.performed_at >= NOW() - INTERVAL '7 days'
|
||||
ORDER BY w.performed_at DESC;
|
||||
```
|
||||
|
||||
```sql
|
||||
-- Failed writes (admin should investigate)
|
||||
SELECT id, asset_id, field_name, error_message, performed_at
|
||||
FROM itglue_writes
|
||||
WHERE status = 'failed'
|
||||
ORDER BY performed_at DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
### How a revert works
|
||||
|
||||
Reverts produce a brand-new `itglue_writes` row whose `before_value` /
|
||||
`after_value` are swapped from the original, and mark the original row
|
||||
`status='reverted'`. The chain is always traceable:
|
||||
|
||||
```sql
|
||||
-- Find every write + its revert (if any) for one asset
|
||||
SELECT id, field_name, status,
|
||||
before_value, after_value,
|
||||
performed_at,
|
||||
audit_id,
|
||||
source_evidence ->> 'reverts_write_id' AS reverts_id
|
||||
FROM itglue_writes
|
||||
WHERE asset_id = '17096940'
|
||||
ORDER BY performed_at;
|
||||
```
|
||||
|
||||
### Cost-guard
|
||||
|
||||
Audit runs charge ~$0.01 (DeepSeek) or ~$0.10 (Claude) per call. Same
|
||||
guard primitives as ticket analyses:
|
||||
|
||||
```sql
|
||||
SELECT created_at, user_id, action, estimated_cost,
|
||||
decision, decision_reason
|
||||
FROM analyzer_cost_audit
|
||||
WHERE action = 'itglue_audit'
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 50;
|
||||
```
|
||||
|
||||
### When the data-builder returns no ticket evidence
|
||||
|
||||
`buildAssetAuditContext` joins via `companies.company_name = itg_organizations.name`
|
||||
(case-insensitive) to map IT Glue org → Autotask company id, then matches
|
||||
ticket fingerprints whose summary or fingerprint mention the asset name. Two
|
||||
common reasons for an empty `ticket_evidence` array:
|
||||
|
||||
1. The IT Glue org's `name` doesn't match any `companies.company_name`
|
||||
exactly — fix by aligning the names, or extend the join.
|
||||
2. The asset's `name` is too generic ("Office", "Email") and the ILIKE match
|
||||
pulls nothing distinctive — accept the audit will rely on field schema +
|
||||
peer exemplars only, no per-ticket grounding.
|
||||
|
||||
### Generic audit_log surfaces every change too
|
||||
|
||||
`audit.log()` is called after every successful Apply/Revert with action
|
||||
`itglue.write` or `itglue.revert`. `/admin/audit-log` shows it. So admins
|
||||
have two views: domain-specific at `/admin/itglue-writes` (with diffs +
|
||||
revert button), and the generic admin feed at `/admin/audit-log`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4.1: ticket-first capture, Configurations, xrefs
|
||||
|
||||
### Inspecting the cross-reference index
|
||||
|
||||
```sql
|
||||
-- Every ticket↔asset linkage in the last 7 days
|
||||
SELECT created_at,
|
||||
ticket_number,
|
||||
relationship,
|
||||
asset_type,
|
||||
asset_id,
|
||||
source,
|
||||
confidence,
|
||||
details
|
||||
FROM itglue_ticket_xrefs
|
||||
WHERE created_at >= NOW() - INTERVAL '7 days'
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 100;
|
||||
```
|
||||
|
||||
```sql
|
||||
-- Most-referenced IT Glue assets across all tickets (heat-map for which
|
||||
-- docs the analyzer leans on most)
|
||||
SELECT asset_type,
|
||||
asset_id,
|
||||
COUNT(*) AS reference_count,
|
||||
COUNT(DISTINCT ticket_number) AS distinct_tickets,
|
||||
MAX(created_at) AS last_referenced
|
||||
FROM itglue_ticket_xrefs
|
||||
WHERE relationship = 'referenced'
|
||||
GROUP BY asset_type, asset_id
|
||||
ORDER BY reference_count DESC
|
||||
LIMIT 30;
|
||||
```
|
||||
|
||||
```sql
|
||||
-- Assets that have been UPDATED via ticket-driven audits but never
|
||||
-- REFERENCED — possibly newly-introduced docs that haven't proven their
|
||||
-- worth yet, or docs the analyzer's retrieval stage isn't finding.
|
||||
SELECT u.asset_type, u.asset_id, COUNT(*) AS update_count
|
||||
FROM itglue_ticket_xrefs u
|
||||
WHERE u.relationship = 'updated'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM itglue_ticket_xrefs r
|
||||
WHERE r.asset_type = u.asset_type
|
||||
AND r.asset_id = u.asset_id
|
||||
AND r.relationship = 'referenced'
|
||||
)
|
||||
GROUP BY u.asset_type, u.asset_id
|
||||
ORDER BY update_count DESC;
|
||||
```
|
||||
|
||||
### Ticket-scoped audits
|
||||
|
||||
```sql
|
||||
-- All audits triggered from a specific ticket
|
||||
SELECT a.id, a.asset_type, a.asset_id,
|
||||
fa.name AS asset_name,
|
||||
a.overall_score,
|
||||
jsonb_array_length(a.field_gaps) AS gap_count,
|
||||
a.provider, a.estimated_cost_usd,
|
||||
a.generated_at
|
||||
FROM itglue_asset_audits a
|
||||
LEFT JOIN itg_flexible_assets fa
|
||||
ON fa.id = a.asset_id::bigint AND a.asset_type = 'flexible_asset'
|
||||
WHERE a.triggered_by_ticket_number = 'T20260502.0033'
|
||||
ORDER BY a.generated_at DESC;
|
||||
```
|
||||
|
||||
```sql
|
||||
-- Every write a given ticket drove (denormalized for one-query lookup)
|
||||
SELECT w.performed_at, w.asset_type, w.asset_id, w.field_name,
|
||||
w.before_value, w.after_value, w.status
|
||||
FROM itglue_writes w
|
||||
WHERE w.triggered_by_ticket_number = 'T20260502.0033'
|
||||
ORDER BY w.performed_at;
|
||||
```
|
||||
|
||||
### Configuration audits
|
||||
|
||||
Same shape as Application audits but `asset_type='configuration'`:
|
||||
|
||||
```sql
|
||||
SELECT a.id, c.name, c.hostname, c.configuration_type_name,
|
||||
a.overall_score, jsonb_array_length(a.field_gaps) AS gap_count,
|
||||
a.provider, a.generated_at
|
||||
FROM itglue_asset_audits a
|
||||
JOIN itg_configurations c ON c.id = a.asset_id::bigint
|
||||
WHERE a.asset_type = 'configuration'
|
||||
AND a.status = 'complete'
|
||||
ORDER BY a.generated_at DESC
|
||||
LIMIT 30;
|
||||
```
|
||||
|
||||
### When asset-matcher returns nothing
|
||||
|
||||
`matchAssetsForAnalysis(analysisId)` joins `companies → itg_organizations`
|
||||
on `LOWER(name)`. If a ticket comes back with `flexibleAssets: []` and
|
||||
`configurations: []`, two common causes:
|
||||
|
||||
1. The IT Glue org name doesn't match the Autotask company name (case-
|
||||
insensitive exact). Fix by aligning the names in either system, or
|
||||
extend the join in `lib/services/analyzer/asset-audit/asset-matcher.ts`.
|
||||
2. The fingerprint's `applications_involved` / `device_classes` don't
|
||||
contain any term that substring-matches a real asset name. The audit
|
||||
panel will show "No matches" — the user can still manually navigate
|
||||
to the relevant asset's detail page and run an asset-first audit.
|
||||
|
||||
### Cost shape for ticket-first audits
|
||||
|
||||
Per-asset audit on a ticket-scoped run:
|
||||
|
||||
| Provider | Cost | Latency |
|
||||
|---|---|---|
|
||||
| Anthropic (Sonnet) | ~$0.05 | 30-60s |
|
||||
| OpenRouter (DeepSeek V4 Pro) | ~$0.005 | 60-180s |
|
||||
|
||||
A typical ticket with 1 Application match + 1 Configuration match audited
|
||||
on DeepSeek runs ~$0.01 and ~3 minutes total.
|
||||
|
||||
|
||||
## Phase 4.3: LogLift event-log evidence pipeline
|
||||
|
||||
### Required env vars
|
||||
|
||||
```
|
||||
B2_KEY_ID=… # Backblaze B2 application key id
|
||||
B2_APP_KEY=… # Backblaze B2 application key secret
|
||||
B2_BUCKET=wulf-audits # default; matches existing n8n bucket
|
||||
B2_REGION=us-west-002 # default
|
||||
B2_ENDPOINT=s3.us-west-002.backblazeb2.com # default; no scheme
|
||||
OPENCLAW_API_KEY=… # webhook auth + collector variable
|
||||
BETTER_AUTH_URL=… # base URL the collector POSTs back to
|
||||
```
|
||||
|
||||
### One-time discovery
|
||||
|
||||
```bash
|
||||
# Sign in as admin → /admin/rmm-overshell → click "Re-discover LogLift"
|
||||
# Or via curl with an admin session cookie:
|
||||
curl -X POST "$BETTER_AUTH_URL/api/admin/rmm/settings/discover-loglift" \
|
||||
-H "Cookie: better-auth.session_token=…" -i
|
||||
```
|
||||
|
||||
Confirm:
|
||||
|
||||
```sql
|
||||
SELECT loglift_component_uid, loglift_component_name, loglift_discovered_at
|
||||
FROM rmm_settings WHERE id = true;
|
||||
```
|
||||
|
||||
### Inspecting LogLift uploads
|
||||
|
||||
```sql
|
||||
-- Recent LogLift executions, with match status
|
||||
SELECT e.id, e.run_id, e.target_hostname, e.status,
|
||||
e.evidence_object_key,
|
||||
e.parsed_evidence -> 'event_count_total' AS event_count,
|
||||
e.parsed_evidence -> 'webhook_summary' -> 'criticalEvents' AS critical_events,
|
||||
e.queued_at, e.completed_at
|
||||
FROM rmm_executions e
|
||||
WHERE e.transport = 'b2_upload'
|
||||
ORDER BY e.queued_at DESC
|
||||
LIMIT 25;
|
||||
```
|
||||
|
||||
```sql
|
||||
-- LogLift uploads that didn't match a Configuration (review for hostname
|
||||
-- typos or unmapped Configurations)
|
||||
SELECT e.run_id, e.target_hostname, e.target_company_id,
|
||||
e.queued_at
|
||||
FROM rmm_executions e
|
||||
WHERE e.transport = 'b2_upload'
|
||||
AND e.status = 'complete'
|
||||
AND e.asset_id IS NULL
|
||||
ORDER BY e.queued_at DESC;
|
||||
```
|
||||
|
||||
```sql
|
||||
-- LogLift uploads that auto-fired an audit
|
||||
SELECT a.id AS audit_id, a.asset_id AS configuration_id,
|
||||
a.overall_score, a.generated_at,
|
||||
l.action, l.created_at AS triggered_at
|
||||
FROM audit_log l
|
||||
JOIN itglue_asset_audits a ON a.id::text = (l.details ->> 'audit_id')
|
||||
WHERE l.action = 'rmm.loglift.audit_triggered'
|
||||
ORDER BY l.created_at DESC LIMIT 25;
|
||||
```
|
||||
|
||||
### Manual replay of a B2 object
|
||||
|
||||
If a webhook came in but Pulse was down, you can replay by re-POSTing
|
||||
the original webhook payload (the collector keeps the metadata; if not,
|
||||
build it from the object key + B2's metadata API). The receiver is
|
||||
idempotent on `run_id` — a duplicate replay with the same `run_id` will
|
||||
update the existing row rather than insert a duplicate.
|
||||
|
||||
### Inspecting a stuck b2_upload row
|
||||
|
||||
```sql
|
||||
SELECT id, run_id, target_hostname, status, started_at, timeout_at,
|
||||
evidence_object_key
|
||||
FROM rmm_executions
|
||||
WHERE transport = 'b2_upload'
|
||||
AND status = 'running'
|
||||
ORDER BY started_at;
|
||||
```
|
||||
|
||||
After 5 minutes, the Overshell worker's timeout sweep flips stuck rows
|
||||
to `timeout`. If the upload arrives later, the receiver still updates
|
||||
the same row by `run_id` (the unique index makes this safe).
|
||||
|
||||
### Force a one-off audit replay from existing evidence
|
||||
|
||||
If the auto-audit failed at upload time (e.g. LLM timeout) and the
|
||||
evidence is already in `rmm_executions`, you can re-fire the audit:
|
||||
|
||||
```sql
|
||||
-- Find the configuration_id from the most recent LogLift upload
|
||||
SELECT asset_id::text AS configuration_id
|
||||
FROM rmm_executions
|
||||
WHERE transport = 'b2_upload'
|
||||
AND target_hostname ILIKE 'YNGHYNWNP01'
|
||||
ORDER BY completed_at DESC LIMIT 1;
|
||||
```
|
||||
|
||||
Then trigger via the existing audit endpoint:
|
||||
|
||||
```bash
|
||||
curl -X POST "$BETTER_AUTH_URL/api/itglue/asset-audit/run" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Cookie: better-auth.session_token=…" \
|
||||
-d '{"assetType":"configuration","assetId":"<config_id>","provider":"anthropic"}'
|
||||
```
|
||||
|
||||
### Why no Telegram summary?
|
||||
|
||||
Out of scope for v1 — that flow was a notification, not a data path.
|
||||
Audits show up on the Configuration page automatically. If you want a
|
||||
Slack/Teams ping when an auto-audit completes, hook it off the
|
||||
`rmm.loglift.audit_triggered` audit_log entry.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue