wulf-pulse/docs/itglue-asset-audit-spec.md

Feature: IT Glue Asset Audit & Documentation Write-back

Status: shipped (Phases 4 + 4.1 of the AI Ticket Analyzer) Migrations: 075_itglue_audit.sql, 076_itglue_ticket_xrefs.sql Build notes: docs/wulf-pulse-ticket-analyzer-build-notes.md → Phase 4 + 4.1 Operator runbook: docs/wulf-pulse-ticket-analyzer-runbook.md → "IT Glue asset audits"

---

What it is, in one paragraph

For each IT Glue flexible-asset Application record and Configuration record (server, workstation, network device), Pulse runs an LLM audit that compares the record's current contents against (a) the field schema with hints, (b) other well-filled records at the same client, (c) recent ticket history that mentions the asset. It surfaces field gaps ("Wulf Application Champion is empty — Jake Hammel is the de-facto SME per T20260502.0033"), note promotions (free text in Notes that belongs in a structured field), and contradictions (Notes say "2-3 VMs" but Application-on-Device tags only 1). Admins can apply a suggestion with one click — Pulse PATCHes IT Glue and records every change with full before/after diff and revert capability.

Two complementary entry points:

• Asset-first (Phase 4) — admin browses lowest-scoring Application or Configuration records and runs an audit against all-time history. Useful for periodic backlog sweeps.
• Ticket-first (Phase 4.1) — every analyzed ticket gets a "Check IT Glue documentation" button on its analysis page. Click → matched IT Glue records appear → audit per record uses just this ticket's evidence. Findings frame as "what did this ticket teach us that the documentation doesn't say?" Drives a forward-only cross-reference index (itglue_ticket_xrefs) of which tickets referenced or updated which documentation, which is also the lookup index for a future RAG automation.

---

Why we built it

The analyzer already extracts documentation_gaps_observed per ticket (Phase 2.4 fingerprint). Until Phase 4 nothing acted on the signal. The catalyzing example was T20260502.0033 (Hynes — tags not printing from Simple Shop Floor), resolved by Collin + Jake identifying a stopped Windows service on MISYS-SQL processing BarTender scan-folder text files. The IT Glue Application record MISYS 6.3 had 5/17 fields filled, missing:

• Wulf Application Champion (Jake is the SME, but only by tribal knowledge)
• Vendor Maintenance/Support (escalation contact "Steve Cianflone" buried in free-text Notes)
• The integration architecture (Simple Shop Floor → MISYS-SQL → BarTender scan folder)
• The named services on each VM

The next tech who hits this issue would re-discover everything. Phase 4 turns the analyzer's passive "documentation_gaps_observed" output into an actionable backlog with a write-back path.

---

User-facing flows

List view — /analyzer/itglue/applications

Shows every Application record across all clients, sorted by lowest audit score first (so unaudited and worst-scored assets bubble up). Each row: asset name, client, populated-field count, last-audit timestamp + provider, score badge.

┌────────────────────────────────────────────────────────────────────┐
│ MISYS 6.3                                              ▌ 55%       │
│ Hynes Industries · 5 fields populated · last audited today (Claude)│
└────────────────────────────────────────────────────────────────────┘

Detail view — /analyzer/itglue/applications/[id]

Three regions:

1. Header — asset name, client, score badge, "Open in IT Glue" link, the <ProviderToggle> (Claude vs DeepSeek), and a primary Run audit button.
2. Audit findings (visible after a run) —
   • Field gaps — severity-toned cards (red/amber/blue by confidence), each with: field name, why-it-matters, suggested value (when the LLM has evidence), evidence ticket links, and an Apply button.
   • Promote from Notes — quoted substring → target field → suggested structured value, with Apply.
   • Contradictions — pure observations (no Apply, by design).
3. Current fields — every field rendered in IT Glue's order; populated shown normally, empty shown muted with the field's hint inline as guidance.
4. Write history — every Apply/Revert for this asset, with inline before/after diffs and a Revert button on committed writes.
5. Audit history — score over time across runs.

Admin view — /admin/itglue-writes

Cross-asset write log, status filters (pending / committed / failed / reverted), full diff per row, ordered most-recent-first.

---

Permissions

Action	Permission
Read audit / View asset detail	requireAuth() (any signed-in user)
Run a fresh audit	requireAuth()
Apply a suggestion (PATCH IT Glue)	requirePermission('itglue', 'write')
Revert a previous write	requirePermission('itglue', 'write')
/admin/itglue-writes cross-asset feed	requirePermission('admin', 'access')

itglue.write is granted to admin and super-admin roles only (see lib/permissions.ts). The Apply / Revert buttons render on the page for non-admins but are disabled with a tooltip explaining why.

---

Pipeline

Single LLM call, provider-aware via stageModelsFor(provider).deep_analysis (Claude Sonnet for Anthropic; DeepSeek V4 Pro for OpenRouter).

Inputs (six context arms)

1. Asset snapshot — the asset's current traits JSONB, redacted via lib/services/analyzer/itglue-redact.ts to strip any password/secret/ key/token/credential-keyed values.
2. Field schema with hints — every field on the asset type, in IT Glue's display order, with the hint text IT Glue surfaces in its own editor (e.g. "Tag any application servers or Devices it's installed on").
3. Peer exemplars from same client — top 5 most-filled flexible assets of the same type at the same client (ranked by populated trait keys), redacted.
4. Best-in-class peers across all clients — top 3 most-filled instances of the same asset type globally, redacted.
5. Per-field fill-rate stats — the % of (a) this client's assets and (b) all assets of this type that have field X populated. Pure SQL aggregate; no token cost. Lets the LLM rank gaps by "unusual to be missing" vs "usually missing anyway."
6. Ticket evidence — up to 20 most-recent complete analyses for the client whose summary or fingerprint mentions the asset name, with their aggregate_fingerprint payload included in full.

Output schema (AssetAuditResponse in lib/types/analyzer.ts)

{
  field_gaps: Array<{
    field_name: string,                     // must exist on the asset type
    why_missing_matters: string,
    suggested_value: string | null,         // null if no concrete evidence
    evidence_ticket_numbers: string[],
    confidence: 'high' | 'medium' | 'low',
  }>,
  notes_promotions: Array<{
    quoted_note_text: string,               // exact substring of Notes
    target_field: string,
    suggested_value: string,
    confidence: 'high' | 'medium' | 'low',
  }>,
  contradictions: Array<{
    description: string,
    evidence: string,
  }>,
  overall_score: number,                    // 0..1 self-rated completeness
}

System-prompt rules (highlights)

• Never invent a field name not present in the schema.
• Never suggest a value you cannot point to evidence for — use null instead.
• Don't suggest password/secret/key/token/credential-shaped fields. (The Apply endpoint also blocks these defensively.)
• Use fill-rate stats: a gap that's empty here but populated >80% elsewhere is a stronger gap than one that's empty 80% of the time globally.

Pipeline files

• lib/services/analyzer/asset-audit/data-builder.ts — collects all six inputs.
• lib/services/analyzer/asset-audit/prompt.ts — system prompt + payload builder (with 80KB cap; trims peer_global first, oldest tickets next).
• lib/services/analyzer/asset-audit/runner.ts — single callLLMStage call, persistence on success or failure.
• lib/services/analyzer/asset-audit/persistence.ts — typed read/write of both DB tables.

---

Data model

itglue_asset_audits (one row per audit run)

Column	Notes
id UUID	PK
asset_type TEXT	Currently 'flexible_asset' only
asset_id BIGINT	IT Glue resource id
asset_type_id, organization_id BIGINT	Denormalized for fast filters
generated_by_user_id TEXT	FK user(id), nullable on user delete
generated_at TIMESTAMPTZ
provider, model_used	Which LLM produced the analysis
asset_snapshot JSONB	Redacted traits at audit time
ticket_count INT	How many fingerprints fed in
field_gaps, notes_promotions, contradictions JSONB	LLM output
overall_score NUMERIC(3,2)
estimated_cost_usd, total_input_tokens, total_output_tokens	Cost telemetry
status	pending / running / complete / failed
error_message TEXT

itglue_writes (one row per write attempt)

Column	Notes
id UUID	PK
audit_id UUID	FK to the audit that prompted the change (nullable)
asset_type, asset_id	What was written
field_name TEXT	Human field name (mapped to trait key on apply)
before_value, after_value JSONB	Pre/post diff
performed_by_user_id TEXT	FK user(id)
performed_at TIMESTAMPTZ
status	pending → committed / failed / reverted
itglue_response JSONB	Raw API response for forensics
error_message TEXT
source_evidence JSONB	{ ticket_numbers, gap_description } or { reverts_write_id }

Generic audit_log (existing — also written for every change)

audit.log() is called on every successful Apply/Revert with action itglue.write or itglue.revert, resource flexible_asset, resourceId = asset_id, and details = { field_name, before, after, audit_id }. Surfaces in /admin/audit-log next to every other admin action.

---

API contract

All routes under /api/analyzer/itglue/.

GET /applications

List all Application records joined to their latest audit. Returns: { applications: [{ id, name, organizationId, organizationName, traitCount, latestAudit }] }.

GET /applications/[id]

Asset detail + field schema for rendering. Returns: { asset, fields }.

GET /applications/[id]/audit?history=1

Latest audit (and optionally history of last 20). Returns: { audit: AssetAuditRow | null, history?: AssetAuditRow[] }.

POST /applications/[id]/audit

Body: { provider?: 'anthropic' | 'openrouter' }. Runs a fresh audit. Cost-guard records action='itglue_audit' in analyzer_cost_audit. Returns: { audit: AssetAuditRow }.

POST /applications/[id]/apply (admin)

Body: { auditId, fieldName, suggestedValue, sourceEvidence? }. Inserts pending itglue_writes row → calls updateFlexibleAsset on the IT Glue client → marks committed/failed → refreshes the local mirror → writes generic audit_log. Returns: { writeId, status: 'committed', asset } or 502 on IT Glue error.

POST /applications/[id]/revert/[writeId] (admin)

Re-applies the original before_value. Inserts a new write row with swapped before/after; original row → status='reverted'. Returns: { writeId, revertedWriteId, status: 'committed', asset }.

GET /applications/[id]/writes

Per-asset write history. Returns: { writes: AssetWriteRow[] }.

GET /writes (admin)

Cross-asset write log with status filter. Query: ?status=committed&limit=100&offset=0. Returns: { writes: AssetWriteRow[] }.

---

Safeguards

Three-layer credential refusal

1. Prompt — system prompt instructs the LLM not to suggest password, secret, key, token, or credential fields.
2. Redaction — redact() from lib/services/analyzer/itglue-redact.ts strips matching keys from the asset snapshot, peer traits, and all payloads before they reach the LLM.
3. Endpoint — POST /apply regex-blocks any field_name matching /(password|secret|key|token|credential)/i and returns 400 even if a compromised payload made it through the first two layers.

Three-layer audit trail

Layer	Captures	Retention
itglue_asset_audits	Every audit run with full LLM context (asset snapshot, model used, cost, ticket count)	Forever
itglue_writes	Every PATCH attempt with before/after diff, status (pending → committed | failed | reverted), audit provenance, raw IT Glue API response, source_evidence	Forever
audit_log (generic)	One row per Apply/Revert in the format admins are used to	Per existing policy

Revert chain semantics

A revert produces a new itglue_writes row whose before_value and after_value are swapped from the original. The new row carries source_evidence = { reverts_write_id }. The original row's status flips to 'reverted'. To trace any state, walk the audit_id and reverts_write_id graph — it's always complete.

Per-record sync after every write

refreshFlexibleAssetById() runs after a successful PATCH (or a successful revert) so itg_flexible_assets.traits reflects the change immediately. The route also returns the freshly-PATCH'd asset in the response so the UI can update without waiting for the mirror.

---

Cost & latency

Provider	Per-audit cost (typical)	Per-audit latency
Anthropic (Sonnet)	~$0.10	30–60s
OpenRouter (DeepSeek V4 Pro)	~$0.01	60–180s

Apply is one IT Glue PATCH + one local upsert + one audit_log insert — typically < 1s.

A full audit + 5 applies on a typical Application record: ~$0.10 (Claude) or ~$0.01 (DeepSeek), 3–5 minutes including click-through.

---

Limitations & non-goals

• Applications only in v1. The schema generalizes (asset_type is a column), but Configurations / Procedures / Domains / Passwords each have different prompt nuances. Adding more asset types is straightforward — duplicate the data-builder and prompt files, add a new asset detail page.
• No bulk-apply. Admin clicks each suggestion individually. If a single audit produces 10 gaps that's 10 clicks. Bulk-apply is an easy fast-follow once we trust quality.
• No two-step approval workflow. Admin-direct write per the design decision; the audit log is the safety net. The existing approval_requests table from the pipeline engine is available if we later want to require tech-proposes / admin-approves.
• No auto-create of new asset records. Apply only updates existing assets. If an audit reveals a totally missing record, the suggestion is shown but Apply is disabled with an explainer; admins create the stub manually in IT Glue.
• No inline editor for suggested values. Admin sees the LLM's suggestion verbatim; if they want to tweak, they edit the value in IT Glue afterward. v2 could add an inline editor.
• No password / secret / key / token / credential writes through this surface — ever. Refused at three layers (above).

---

Inspecting state (operational SQL)

Audits, lowest score first

SELECT a.asset_id,
       fa.name AS application_name, fa.organization_name,
       a.overall_score,
       jsonb_array_length(a.field_gaps)        AS gap_count,
       jsonb_array_length(a.notes_promotions)  AS promo_count,
       a.provider, a.estimated_cost_usd,
       a.generated_at
  FROM itglue_asset_audits a
  JOIN itg_flexible_assets fa ON fa.id = a.asset_id::bigint
 WHERE a.status = 'complete'
 ORDER BY a.generated_at DESC, a.overall_score ASC;

Writes in the last 7 days

SELECT performed_at, field_name, status,
       before_value, after_value,
       performed_by_user_id, audit_id
  FROM itglue_writes
 WHERE performed_at >= NOW() - INTERVAL '7 days'
 ORDER BY performed_at DESC;

Find what's still waiting on which write

SELECT id, field_name, before_value, after_value,
       source_evidence ->> 'reverts_write_id' AS reverts_id,
       status, performed_at
  FROM itglue_writes
 WHERE asset_id = '17096940'
 ORDER BY performed_at;

Audit cost-guard decisions

SELECT created_at, user_id, action, estimated_cost,
       decision, decision_reason
  FROM analyzer_cost_audit
 WHERE action = 'itglue_audit'
 ORDER BY created_at DESC
 LIMIT 50;

---

How a typical session looks

1. Tech opens /analyzer/itglue/applications. MISYS 6.3 is at the top with no audit yet.
2. Tech clicks the row → lands on the detail page. They see 12/17 fields are empty.
3. Tech clicks Run audit (DeepSeek selected for cost). 60–180s later, the audit panel populates: 4 high-confidence field gaps, 1 notes promotion, 1 contradiction, score 0.55.
4. Tech (admin) clicks Apply on Wulf Application Champion → suggested value "Jake Hammel". A PATCH lands on IT Glue, the local mirror refreshes, the field shows the new value, a write row appears in the History section.
5. Tech clicks Apply on the Vendor Maintenance/Support promotion ("Steve Cianflone" extracted from Notes).
6. The next time someone opens this asset in IT Glue, the documentation reflects what tickets have been telling us all along.

---

Files (quick reference)

Schema: migrations/075_itglue_audit.sql

Pipeline: lib/services/analyzer/asset-audit/

• data-builder.ts, prompt.ts, runner.ts, persistence.ts, runner.test.ts

API: app/api/analyzer/itglue/applications/

• route.ts (list)
• [id]/route.ts (detail)
• [id]/audit/route.ts (GET/POST audit)
• [id]/apply/route.ts (admin write)
• [id]/revert/[writeId]/route.ts (admin revert)
• [id]/writes/route.ts (per-asset history)

Plus app/api/analyzer/itglue/writes/route.ts (admin cross-asset feed).

UI:

• app/analyzer/itglue/applications/page.tsx (list)
• app/analyzer/itglue/applications/[id]/page.tsx (detail)
• app/admin/itglue-writes/page.tsx (admin cross-asset)

Modified:

• lib/services/itglue-client.ts — updateFlexibleAsset, refreshFlexibleAsset, getRawSingle, isITGlueConfigured, internal patch
• lib/services/itglue-sync-service.ts — refreshFlexibleAssetById
• lib/permissions.ts — itglue: ['read', 'write']
• lib/types/analyzer.ts — AssetAuditResponse schema + request types
• components/navigation/app-navigation.tsx — nav entry

---

Phase 4.1 additions

Ticket-first capture flow

Trigger: opt-in. The user clicks "Check IT Glue documentation" on an analysis detail page (/analyzer/analysis/[id]).

Pipeline:

1. GET /api/analyzer/analyses/[id]/itglue-suggestions — runs matchAssetsForAnalysis(id) and returns matched flexible_assets + configurations (top 5 each by score) along with any existing ticket-scoped audits.
2. User clicks "Audit for this ticket" on a matched asset → POST to the same endpoint with { assetType, assetId, provider }.
3. The runner builds context with ticketScopeAnalysisId set, so ticket evidence is exactly the one analysis the user clicked from. The system prompt picks up a ticket-scoped suffix instructing the LLM to frame findings as "what this ticket taught us."
4. The audit row is persisted with triggered_by_ticket_number + triggered_by_analysis_id populated.
5. Apply works from the same per-asset routes as the asset-first flow. When the audit is ticket-scoped, the resulting itglue_writes row also carries triggered_by_ticket_number, and the apply path inserts an xref row with relationship='updated'.

Asset matching is loose: substring + word-boundary match between the ticket's aggregate_fingerprint.{applications_involved, device_classes, vendors_involved} and itg_flexible_assets.name (Application type only)

• itg_configurations.{name, hostname} for the same client. Score 3 (exact) > 2 (word-boundary) > 1 (substring). Top 5 per kind.

Configuration support

Same audit/apply/revert pattern as Applications, with these differences:

• Flat schema — Configurations have ~17 editable top-level columns rather than a traits JSONB blob. The data-builder synthesizes a trait-style map for prompt consistency, then the apply route maps field name → IT Glue dash-case attribute (primary_ip → 'primary-ip').
• Hand-curated field hints — IT Glue Configurations don't expose a _fields table; the 16 hand-written hints live in lib/services/analyzer/asset-audit/data-builder.ts (CONFIGURATION_FIELDS constant). Each hint is the per-field documentation the LLM sees.
• Configuration-flavored prompt — focuses on hostname/FQDN consistency, OS version currency, named services capture (in operating_system_notes), IP/MAC hygiene, contact ownership.
• Column allowlist on apply — even though the audit's output schema doesn't restrict field_name, the configuration apply route refuses anything outside name | hostname | primary_ip | mac_address | serial_number | asset_tag | position | notes | operating_system_notes. FK-shaped fields (manufacturer_id, model_id, operating_system_id, contact_id, location_id) are read-only via this surface for v1.
• Per-record sync helper — refreshConfigurationById(id) mirrors refreshFlexibleAssetById(id); called after every successful Apply.

Cross-reference index — itglue_ticket_xrefs

ticket_number | analysis_id | asset_type | asset_id | relationship | source | confidence | details | created_at

Three relationship types:

Relationship	Meaning	Source
referenced	The analyzer cited this asset/doc when analyzing the ticket. Comes from analyzer_analyses.itglue_docs_referenced.	analyzer_referenced
updated	A ticket-driven audit produced a write on this asset.	audit_write
should_have_referenced	Reserved — gap text suggests we should have found this asset/doc but didn't. Not auto-populated yet.	manual (future)

Populated by:

• Post-analysis hook in lib/services/analyzer/worker.ts — insertReferencedXrefsFromAnalysis runs after every successful analysis insertion. Maps each ITGlueDocReference.doc_type → asset_type. Best-effort.
• Post-apply hook in the apply routes — insertUpdatedXref runs after a successful PATCH if the audit was ticket-scoped. Best-effort.
• No backfill — table fills forward. The unique index (ticket_number, analysis_id, asset_type, asset_id, relationship, source) makes ingestion idempotent.

Two views consume it:

1. Asset detail page — "Tickets that touched this asset" section under the audit panel, with sub-sections for Referenced by (with the LLM's relevance reason) and Updated by (with the field name and write history link).
2. GET /api/analyzer/tickets/[ticketNumber]/itglue-xrefs — ticket-side view; available for future ticket-page surfacing.

This table is also the lookup index a future RAG automation will use: given a new ticket's entities, fetch historically-referenced docs as warm candidates for the analyzer's IT Glue retrieval stage.

Ticket-linkage columns

Both audit and write tables carry denormalized ticket linkage:

• itglue_asset_audits.triggered_by_ticket_number (and triggered_by_analysis_id) — set on ticket-scoped audits. Null on asset-first audits.
• itglue_writes.triggered_by_ticket_number — copied forward from the audit on Apply, so "every write a given ticket drove" is a one-query lookup. Null when the audit was asset-first.

Files added in 4.1

• migrations/076_itglue_ticket_xrefs.sql
• lib/services/analyzer/asset-audit/xrefs.ts
• lib/services/analyzer/asset-audit/asset-matcher.ts
• app/api/analyzer/analyses/[id]/itglue-suggestions/route.ts
• Configuration parallel route tree under app/api/analyzer/itglue/configurations/
• xref endpoints: app/api/analyzer/itglue/applications/[id]/xrefs/route.ts, app/api/analyzer/itglue/configurations/[id]/xrefs/route.ts, app/api/analyzer/tickets/[ticketNumber]/itglue-xrefs/route.ts
• app/analyzer/itglue/configurations/page.tsx (list)
• app/analyzer/itglue/configurations/[id]/page.tsx (detail)
• components/analyzer/itglue-suggestions-panel.tsx

Files modified in 4.1

• lib/services/itglue-client.ts — updateConfiguration, refreshConfiguration
• lib/services/itglue-sync-service.ts — refreshConfigurationById
• lib/services/analyzer/asset-audit/data-builder.ts — assetType dispatch + ticket-scope mode + Configuration field schema
• lib/services/analyzer/asset-audit/prompt.ts — Configuration prompt + ticket-scoped suffix + assetType-aware payload
• lib/services/analyzer/asset-audit/runner.ts — accepts assetType and ticketScopeAnalysisId; persists triggered_by_*
• lib/services/analyzer/asset-audit/persistence.ts — asset_type union extended; getLatestTicketScopedAudit; createPendingWrite accepts asset_type + ticket linkage
• lib/services/analyzer/worker.ts — post-analysis xref ingestion hook
• app/api/analyzer/itglue/applications/[id]/apply/route.ts — ticket linkage + xref insert on Apply
• app/api/analyzer/itglue/applications/[id]/revert/[writeId]/route.ts — asset_type now passed to createPendingWrite
• app/analyzer/analysis/[id]/page.tsx — <ItglueSuggestionsPanel/> rendered
• app/analyzer/itglue/applications/[id]/page.tsx — "Tickets that touched this asset" section
• components/navigation/app-navigation.tsx — split entries