wulf-pulse/docs/wulf-pulse-ticket-analyzer-prompt.md

Feature: AI Ticket Analyzer (wulf-pulse)

Add an on-demand AI-powered ticket analysis feature to the existing wulf-pulse app at forgejo.wulfconsulting.cloud/lorentz/wulf-pulse.

This is a feature addition, not a new project. Conform to existing wulf-pulse conventions: React + Vite + TailwindCSS + shadcn/ui frontend, Node.js + Express backend, Entra ID OIDC auth, the existing PostgreSQL instance (the one that already has Autotask data syncing into it), Forgejo CI, Pangolin reverse proxy. Do not introduce new frameworks. Match the existing folder layout, error handling style, and route conventions.

Before writing any code, read claude.md, the route registration file, the auth middleware, and the data-access layer for tickets so the feature plugs into the existing patterns. If those files don't exist, ask before guessing.

---

What this feature does

A wulf-pulse user opens a ticket view, clicks Analyze, and the system produces a structured analysis covering:

• A unified chronological timeline (with markers for source type)
• What was actually done vs. what should have been done
• Gaps — including subtle ones like the customer telling us to stop while work continued, status not matching reality, or the original ask never being directly answered
• Recommended next step with rationale
• Post-resolution analysis (if the ticket is resolved)
• Confidence score and human-review flag
• Referenced IT Glue documentation

The analysis is stored versioned by ticket number, can be re-run when new activity arrives, and can be emailed to other Wulf users.

---

Critical: how Autotask notes/entries actually work

The current generic prompt would mis-handle the real Autotask note structure. Here are the distinctions the analyzer must make:

Note types the analyzer must classify in pre-processing (Stage 0)

1. Workflow rule firings — Note | Autotask Administrator with title like Workflow Rule "..." fired. These are pure noise. Filter them out entirely before any model call. Tag each as workflow_noise.

2. Service Desk Notification emails — Note | <tech name> with title Service Desk Notification and a description that's just a list of email addresses. These are auto-generated email send confirmations. Filter out before model calls. Tag as email_notification.

3. Ticket Notes — Ticket Note | <person name>. These are real communications, often from the customer or a forwarded email. Keep. Tag as customer_communication or internal_communication based on the person's email domain.

4. Time Entry Summary Notes — the Summary Notes field of a Time Entry. Customer-visible. Tag as time_entry_summary. Keep.

5. Time Entry Internal Notes — the Internal Notes field of the same Time Entry. Technician-only. Tag as time_entry_internal. Keep — these are usually the highest-signal entries.

A single Time Entry can have BOTH Summary Notes and Internal Notes — they should appear as two separate timeline events with the same timestamp but different visibility markers, OR as a single event with both fields preserved. Choose the latter for cleaner timelines but always render them visually distinct.

The tagging schema

Every retained event in the unified timeline must have:

{
  timestamp: string,           // ISO 8601
  actor: string,               // person name
  actor_type: "wulf_tech" | "client_contact" | "vendor" | "system" | "automation",
  source: "ticket_create" | "ticket_note" | "time_entry" | "status_change" | "resolution",
  visibility: "customer_facing" | "internal_only" | "mixed",  // mixed = time entry with both fields
  summary_notes?: string,      // customer-facing content if present
  internal_notes?: string,     // internal content if present
  hours?: number,              // for time entries
}

Render markers in the analysis output as:

• 🟢 customer-facing
• 🔒 internal-only
• 🔄 mixed (both)

---

Real example to test against

Use this real ticket as a fixture for your tests. The analyzer must catch all four findings listed below — if any are missed, the analysis prompts need refinement.

Ticket T20260424.0045 — "Outmarket AI vendor integration request"

The internal notes reveal:

1. The original ask was narrower than the public summary suggests — Lorentz's email said "I'm gonna need access to that for another integration with the claims department for loss run pro please let me know where that credential is in Passportal."
2. At 04/24 10:34, Lorentz posted a Ticket Note saying "I was able to access the Vertafore Developer portal and determine what is necessary - no need to reach out to Vertafore. I'll take it from here, thank you!"
3. On 04/27 (next business day), the assigned tech took a call from Vertafore anyway and logged ~20 min of additional work after the customer said to stop.
4. Status is still "Waiting Customer" three days after the requestor effectively closed the loop.

The analyzer must flag:

• Gap (high): Work continued after the customer indicated they were taking it from here.
• Gap (medium): Status hasn't been updated to reflect the requestor's resolution.
• Gap (low): The original credential-locator ask was never directly answered before the conversation pivoted.
• Next step: Confirm with requestor whether the Vertafore endpoint info is still useful, then close.

Build a test fixture from this ticket (PDF is available, transcribe the structured fields) and assert these findings appear in the analysis output. The fixture lives in apps/api/test/fixtures/tickets/T20260424.0045.json.

---

Database changes

Add these tables to the wulf-pulse Postgres database. Use the project's existing migration tool. Prefix all new tables with analyzer_ to keep them clearly scoped to this feature.

analyzer_analyses

id uuid pk default gen_random_uuid()
ticket_number text not null
autotask_ticket_id bigint not null
analysis_version int not null            -- monotonic per ticket_number
content_hash_at_analysis text not null   -- sha256 of source data at analysis time
triggered_by_user_id uuid                -- references existing pulse users table
triggered_at timestamptz default now()
status text not null default 'pending'   -- pending|running|complete|failed
completed_at timestamptz

-- model usage
haiku_used boolean default false
sonnet_used boolean default false
opus_used boolean default false
total_input_tokens int default 0
total_output_tokens int default 0
estimated_cost_usd numeric(10,4) default 0

-- structured output
summary text
timeline jsonb                           -- unified, with visibility markers
what_was_done jsonb
what_should_have_been_done jsonb
gaps jsonb                               -- [{description, severity, evidence_timestamps}]
next_step text
next_step_rationale text
post_resolution_analysis text
confidence_score numeric(3,2)
needs_human_review boolean default false
human_review_reasons jsonb

-- IT Glue
itglue_docs_referenced jsonb default '[]'

-- debugging
model_traces jsonb
filtered_noise_count int default 0       -- how many workflow/notification entries were stripped
error_message text

unique (ticket_number, analysis_version)

Indexes: (ticket_number, analysis_version desc), (triggered_at desc), (needs_human_review) where needs_human_review = true.

analyzer_shares

id uuid pk default gen_random_uuid()
analysis_id uuid not null references analyzer_analyses(id) on delete cascade
shared_by_user_id uuid not null
shared_with_email text not null          -- validate against ALLOWED_SHARE_DOMAINS
note text
shared_at timestamptz default now()
viewed_at timestamptz

analyzer_jobs

id uuid pk default gen_random_uuid()
ticket_number text not null
queued_by_user_id uuid
status text not null default 'queued'    -- queued|fetching|triaging|itglue|analyzing|deep_review|complete|failed
result_analysis_id uuid references analyzer_analyses(id)
queued_at timestamptz default now()
started_at timestamptz
finished_at timestamptz
error_message text

If wulf-pulse already uses BullMQ or another job queue, plug into it. If not, a simple Postgres-row-based queue with a worker polling every 2 seconds is acceptable for an on-demand-only feature — discuss with me before pulling in a new dependency.

---

Backend changes

Source-of-truth question (ask me before deciding)

Wulf-pulse already syncs Autotask data to Postgres. Before implementing, look at the existing sync to determine:

1. Does the Pulse sync include ticket notes and time entries, or just ticket headers?
2. How fresh is the sync? Real-time (webhook), minute-level, or hourly?
3. Are Internal Notes synced? (They may be excluded from some syncs for privacy reasons.)

Based on what you find, choose one of:

• (A) Read everything from Pulse Postgres — preferred if notes + internal notes + time entries are all synced and fresh.
• (B) Pull live from Autotask REST at analyze-time — required if the sync is incomplete.
• (C) Hybrid: Pulse for fast list/search, live REST fetch for the full payload at analyze-time — most likely the right answer.

Tell me which one fits before writing the data-access layer.

New routes (mount under existing wulf-pulse API namespace)

POST /api/analyzer/tickets/:ticketNumber/analyze
                                    # body: { force?: boolean }
                                    # returns: { jobId, status, existingAnalysisId? }

GET  /api/analyzer/jobs/:jobId      # poll status
GET  /api/analyzer/analyses/:id     # fetch a specific analysis
GET  /api/analyzer/tickets/:ticketNumber/analyses
                                    # list versions
GET  /api/analyzer/needs-review     # filtered queue

POST /api/analyzer/analyses/:id/share
                                    # body: { recipientEmail, note? }

All routes require existing wulf-pulse auth middleware.

IT Glue client

New module apps/api/src/services/itglue/:

• client.ts — REST client with x-api-key auth, retry/backoff
• redact.ts — strips fields matching /password|secret|key|token|credential|api[_-]?key/i (case-insensitive, recursive) BEFORE any value reaches the LLM or the database. Replace with "[REDACTED]". Include unit tests for nested objects and arrays.
• search.ts — given a client name and search hints, returns sanitized doc snippets capped at 2000 chars per doc, max 10 docs.

The redaction is a security-critical code path. Add a comment explaining why and link to this prompt section. Do not log full doc bodies anywhere — only IDs and names.

Anthropic SDK setup

Add @anthropic-ai/sdk to apps/api/package.json. Create apps/api/src/services/llm/:

• client.ts — singleton SDK instance, reads ANTHROPIC_API_KEY from env
• pricing.ts — per-model input/output rates with a comment to verify against https://docs.claude.com/en/docs/about-claude/pricing quarterly. Don't hardcode rates without comments noting the as-of date.
• models.ts — exports the canonical model IDs:
  • HAIKU = "claude-haiku-4-5"
  • SONNET = "claude-sonnet-4-6"
  • OPUS = "claude-opus-4-7"

Before finalizing those constants, verify each model ID is current and available on the API. If you find newer versions or the IDs are wrong, ask me before substituting.

---

The analysis pipeline

Implemented as apps/api/src/services/analyzer/pipeline.ts. Each stage is a separate function for testability.

Stage 0 — Fetch & Pre-process

1. Resolve ticket via the data-access strategy chosen above.
2. Filter noise: strip Note | Autotask Administrator workflow firings and Service Desk Notification notes. Count them and store in filtered_noise_count for transparency.
3. Tag remaining events per the schema in the "How Autotask notes actually work" section.
4. Sort chronologically.
5. Compute content_hash = sha256(canonical_json({tagged_events, ticket_status, ticket_priority, queue})).
6. Idempotency check: if force=false and a complete analysis exists with the same hash, short-circuit.

Stage 1 — Triage (Haiku)

Model: claude-haiku-4-5

System prompt:

You are a ticket triage assistant for Wulf Consulting, an MSP. You will receive
an Autotask ticket with notes and time entries that have already been pre-filtered
to remove workflow noise and tagged by visibility (customer-facing vs internal).

Extract structured metadata and assess complexity. Pay special attention to
internal-only notes — these often contain the real story.

Respond ONLY with JSON:

{
  "ticket_type": "incident" | "service_request" | "problem" | "change" | "other",
  "category": string,
  "entities": {
    "client_name": string | null,
    "site_name": string | null,
    "devices": string[],
    "users": string[],
    "applications": string[],
    "vendors": string[]            // third parties involved (Vertafore, etc.)
  },
  "is_resolved": boolean,
  "status_matches_reality": boolean,    // does the Autotask status reflect the actual state?
  "complexity_tier": "low" | "medium" | "high",
  "complexity_reasons": string[],
  "itglue_lookup_needed": boolean,
  "itglue_search_hints": string[]
}

Complexity rubric:
- low: single straightforward issue, ≤3 retained events, clear path
- medium: multiple events, some back-and-forth, moderate ambiguity
- high: any of — bounced between techs, conflicting notes, unresolved >5 days,
        customer-vs-internal narrative mismatch, multiple vendors involved,
        or status appears to disagree with the actual state of the work

User message: a structured payload containing:

• Ticket header fields (title, status, priority, queue, account, contact, dates)
• Tagged event list (filtered + tagged from Stage 0)
• Counts: total events, internal-only count, customer-facing count

Cap total payload at ~50KB. If larger, truncate oldest internal-only events first (preserving all customer-facing communications), and add a marker.

Stage 2 — IT Glue Retrieval (conditional)

Run if itglue_lookup_needed === true.

1. Resolve client name → IT Glue org ID. Maintain a small JSON alias map at apps/api/src/services/itglue/aliases.json for known fuzzy mappings (e.g. "Seubert" / "Seubert and Associates" / "S&A" → org id). Document this file in the README.
2. For each search hint, query configurations, flexible_assets, and documents endpoints.
3. Dedupe, cap at 10 docs total.
4. Run each result through redact.ts BEFORE adding to context.
5. Cap each doc body at 2000 chars in the LLM context.

Stage 3 — Deep Analysis (Sonnet)

Model: claude-sonnet-4-6

System prompt (verbatim):

You are a senior MSP technician at Wulf Consulting reviewing a ticket. You will
receive:
1. The full ticket with all retained notes and time entries (already filtered for
   workflow noise; tagged with visibility markers — customer_facing, internal_only,
   or mixed)
2. Triage metadata from a previous pass
3. Optionally, sanitized IT Glue documentation snippets for the client

Be specific and reference events by their timestamp and actor. Do not invent facts.
If something is unclear, say so explicitly.

Pay particular attention to these patterns, which are common failure modes:
- The customer indicates they have resolved the issue or want to take it over,
  but work continues afterward
- The Autotask status does not match the actual state (e.g. "Waiting Customer"
  when the customer has already responded, or "In Progress" with no recent activity)
- The original ask in the requester's first message is different from what the
  ticket pivoted to addressing
- Internal notes contradict or add important context missing from customer-facing
  summary notes
- A vendor case was opened but the customer's direct ask could have been answered
  without it
- Time was billed for work the customer didn't ultimately need

Respond ONLY with JSON:

{
  "summary": string,                    // 2-4 sentences, neutral tone
  "timeline": [
    {
      "timestamp": string,              // ISO 8601
      "actor": string,
      "actor_type": "wulf_tech" | "client_contact" | "vendor" | "system" | "automation",
      "source": "ticket_create" | "ticket_note" | "time_entry" | "status_change" | "resolution",
      "visibility": "customer_facing" | "internal_only" | "mixed",
      "action": string                  // what happened, in plain language
    }
  ],
  "what_was_done": string[],            // concrete actions in order
  "what_should_have_been_done": string[], // ideal actions per MSP best practice
                                          // and any IT Glue docs provided
  "gaps": [
    {
      "description": string,
      "severity": "low" | "medium" | "high",
      "evidence_timestamps": string[]   // which timeline events support this
    }
  ],
  "next_step": string,                  // single concrete next action
  "next_step_rationale": string,
  "post_resolution_analysis": string | null, // only if is_resolved=true
  "confidence_score": number,           // 0.0–1.0
  "needs_human_review": boolean,
  "human_review_reasons": string[],
  "ambiguities_for_opus": string[],     // questions a deeper-reasoning model should resolve
  "itglue_docs_referenced": [
    {
      "id": string,
      "name": string,
      "url": string,
      "doc_type": string,
      "relevance_reason": string
    }
  ]
}

Set needs_human_review=true if any of:
- confidence_score < 0.6
- gaps contain any "high" severity item
- ticket open >7 days with no clear resolution path
- conflicting information between notes
- billed hours appear excessive for the work performed

Stage 4 — Deep Reasoning (Opus, conditional)

Model: claude-opus-4-7

Trigger conditions (any one):

• complexity_tier === "high" from Stage 1
• ambiguities_for_opus.length > 0 from Stage 3
• confidence_score < 0.5 from Stage 3
• Stage 1's status_matches_reality === false

System prompt:

You are a principal-level MSP engineer doing a final review of a complex ticket.
You will be given:
1. The full tagged ticket
2. The Sonnet-tier analysis
3. A list of specific ambiguities or open questions

Address each ambiguity directly with reasoning. Then produce updates ONLY for
fields that should change.

Respond ONLY with JSON:

{
  "opus_notes": string,                 // your reasoning, 1–3 paragraphs
  "updates": {
    // any subset of: next_step, next_step_rationale, gaps,
    // confidence_score, needs_human_review, human_review_reasons,
    // post_resolution_analysis
  }
}

Stage 5 — Persistence

1. Compute next analysis_version for this ticket_number.
2. Insert into analyzer_analyses.
3. Sum tokens, compute estimated cost.
4. Update analyzer_jobs row.

---

Frontend changes

Routes (add to existing wulf-pulse router)

• /analyzer/ticket/:ticketNumber — ticket detail with Analyze button + history of prior analyses
• /analyzer/analysis/:id — full analysis view, printable, with timeline visualization
• /analyzer/queue — needs-human-review queue

Analyze button behavior

1. Click → POST to analyze endpoint.
2. If existingAnalysisId returned and content unchanged, navigate straight to it.
3. Otherwise show progress UI polling job status every 2s with stage labels: "Fetching..." → "Triaging..." → "Searching IT Glue..." → "Analyzing..." → "Deep review..." → "Done".
4. On completion, navigate to the analysis view.

Analysis view layout

Use shadcn/ui components. Sections in order:

1. Header — ticket number + title + Autotask deep link, model tier badges (Haiku/Sonnet/Opus pills), confidence score, total cost, "Share" button
2. Summary — paragraph
3. Next Step — highlighted card with rationale collapsed by default
4. Timeline — vertical timeline with the three visibility markers (🟢 / 🔒 / 🔄), each event clickable to expand full notes
5. What Was Done — bulleted list
6. What Should Have Been Done — bulleted list, side-by-side with #5 on wide screens
7. Gaps — cards colored by severity, with "Evidence:" linking back to timeline events
8. Post-Resolution Analysis — only if present
9. Human Review Flags — only if needs_human_review = true
10. IT Glue References — list of docs with external links

Re-analyze indicator

On the ticket view, if cached content_hash differs from the latest analysis's content_hash_at_analysis, show a banner: "New activity since last analysis · Re-analyze".

Share modal

• Recipient email field (autocomplete from existing wulf-pulse user list if available)
• Validate domain against ALLOWED_SHARE_DOMAINS env var
• Optional note
• Sends via M365 Graph using existing wulf-pulse mail integration if one exists; otherwise use a new module under apps/api/src/services/mail/ and ask before adding new credentials

---

Environment variables (additions)

# IT Glue
ITGLUE_API_KEY=
ITGLUE_API_BASE=https://api.itglue.com

# Anthropic
ANTHROPIC_API_KEY=

# Sharing
ALLOWED_SHARE_DOMAINS=wulfconsulting.com

Add to ~/projects_env/wulf-pulse.env. Do not commit example values.

---

Testing

Required tests (use whatever wulf-pulse already uses for testing):

1. IT Glue redaction — nested objects, arrays of objects, mixed-case field names. Must redact and never allow a password to reach the database.
2. Note pre-processor — given a fixture with all five note types, asserts workflow firings and notification emails are filtered, and remaining events are correctly tagged.
3. Pipeline against the T20260424.0045 fixture — asserts all four required findings appear in the analysis output. This is a regression test for the prompts, not just code.
4. Schema validation — every LLM response is parsed through Zod. Test with deliberately malformed responses to confirm graceful retry then failure.
5. Idempotency — same content_hash with force=false returns the existing analysis without invoking the LLM.

---

Critical correctness notes

• Never store IT Glue secrets/passwords in any database row, log line, or LLM context. Redact before everything.
• Never log full IT Glue document content. Only doc IDs and names.
• Validate every LLM JSON response with Zod. On parse failure, retry once with the prior response and the parse error. After two failures, mark the job failed and store the raw response in error_message.
• Token budget guard: if any single model call would exceed 100k input tokens, truncate oldest internal-only events first while preserving all customer-facing communications, and add a marker. Log a warning.
• Cost circuit breaker: if estimated total cost would exceed $2.00 before the Opus call, skip Opus, set needs_human_review = true, and add reason "cost ceiling reached".
• Idempotency: the analyze endpoint must be idempotent on (ticket_number, content_hash) when force=false.
• Verify model IDs and pricing against https://docs.claude.com before finalizing constants. Models and rates change.
• Ticket Notes from wulfconsulting.com addresses are internal communications, not customer communications. Tag them by domain, not by author.

---

Delivery order

Build and ship in this order, asking me to review between each phase:

1. Database migrations + Zod schemas in packages/shared (or wulf-pulse equivalent).
2. Note pre-processor with unit tests against the T20260424.0045 fixture (PDF → JSON transcription is the first deliverable).
3. IT Glue client + redaction (security-critical, must land before LLM integration).
4. Anthropic SDK setup + per-stage prompt files.
5. Full pipeline + job worker.
6. API routes.
7. Frontend pages.
8. Share-via-email integration.
9. README updates with operator runbook (how to monitor cost, how to add IT Glue org aliases, how to triage failed analyses).

For phase 1 specifically: confirm the data-access strategy (read from Pulse Postgres / live Autotask REST / hybrid) before writing the migration, since the schema for tickets_cache may or may not be needed depending on which path we take.