feat: Veeam RPO analysis, comparison, ticket analysis + company teams table
- Add Veeam RPO analysis page (/veeam-analysis) and comparison page (/veeam-comparison) - Add API routes: /api/veeam/rpo-analyze, rpo-comparison, rpo-offline-log, ticket-analysis - Add veeam-rpo-service.ts enhancements (RPO logic, offline detection, comparison) - Add veeam-analysis-state.ts and rmm-device-resolver.ts services - Add migrations 065-068: company_teams, veeam_rpo_offline_log, rpo_comparison_tables, veeam_ticket_analysis - Add backup-status page updates and nav links for new Veeam pages - Add scripts: deactivate-cis-for-inactive-companies, workstation category updates - Add docs: mimecast-api-guide, veeam-backup-alerting-recommendation, workstation-backup-overview, ticket-analyzer-prompt - Minor: webhook-service, entity-sync, entity-mapper, sync-helpers, sync.ts, middleware.ts updates
This commit is contained in:
parent
07067bef19
commit
ea3471d38d
36 changed files with 5604 additions and 217 deletions
341
docs/mimecast-api-guide.md
Normal file
341
docs/mimecast-api-guide.md
Normal file
|
|
@ -0,0 +1,341 @@
|
|||
# Mimecast API Guide — Threat Dashboard Integration
|
||||
|
||||
## Overview
|
||||
|
||||
This guide covers authentication and data access for the Mimecast API v2, oriented toward building a threat dashboard in a client portal. All examples use OAuth 2.0 (the current recommended approach).
|
||||
|
||||
---
|
||||
|
||||
## 1. Authentication (OAuth 2.0 — Client Credentials)
|
||||
|
||||
### Setup in Mimecast Console
|
||||
|
||||
1. Navigate to **Administration > Services > API and Platform Integrations** (or Integrations Hub in newer tenants)
|
||||
2. Create a new API 2.0 application
|
||||
3. Assign only the permission scopes you need (see §3 below)
|
||||
4. Record the `client_id` and `client_secret`
|
||||
|
||||
### Getting an Access Token
|
||||
|
||||
```http
|
||||
POST https://api.services.mimecast.com/oauth/token
|
||||
Content-Type: application/x-www-form-urlencoded
|
||||
|
||||
grant_type=client_credentials&client_id=YOUR_CLIENT_ID&client_secret=YOUR_CLIENT_SECRET
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"access_token": "eyJ...",
|
||||
"token_type": "Bearer",
|
||||
"expires_in": 3600
|
||||
}
|
||||
```
|
||||
|
||||
Cache the token and refresh before expiry. Plan for a 401 → refresh → retry flow.
|
||||
|
||||
### Using the Token
|
||||
|
||||
All API requests:
|
||||
|
||||
```http
|
||||
Authorization: Bearer {access_token}
|
||||
Content-Type: application/json
|
||||
x-mc-req-id: {uuid-per-request}
|
||||
```
|
||||
|
||||
> `x-mc-req-id` should be a fresh UUID per call — used for idempotency and support tracing.
|
||||
|
||||
### Region Discovery
|
||||
|
||||
Mimecast routes requests by region. If you get unexpected 401s or 404s, call the discovery endpoint first to resolve the correct base URL for a given account:
|
||||
|
||||
```http
|
||||
GET https://api.services.mimecast.com/oauth/discover
|
||||
Authorization: Bearer {access_token}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Request / Response Structure
|
||||
|
||||
Nearly all Mimecast endpoints use `POST`, even for reads. The body follows a consistent envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"data": [
|
||||
{ /* endpoint-specific params */ }
|
||||
],
|
||||
"meta": {
|
||||
"pagination": {
|
||||
"pageToken": "next-page-token-from-previous-response"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"data": [ /* results */ ],
|
||||
"meta": { /* status info */ },
|
||||
"fail": [ /* per-item errors */ ],
|
||||
"pagination": {
|
||||
"next": "token_for_next_page",
|
||||
"previous": "token_for_prev_page"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Important**: HTTP 200 means the request was received. Always check `fail` array and `meta.status` for functional errors — a 200 can still contain failures.
|
||||
|
||||
---
|
||||
|
||||
## 3. Threat Dashboard Endpoints
|
||||
|
||||
### A. SIEM Logs (Core Email Threat Stream)
|
||||
|
||||
The broadest threat feed — covers all MTA events including blocked senders, malicious URLs, attachment verdicts, spoofing attempts.
|
||||
|
||||
```http
|
||||
POST /api/audit/get-siem-logs
|
||||
|
||||
{
|
||||
"data": [{
|
||||
"type": "MTA",
|
||||
"fileFormat": "json",
|
||||
"compress": false,
|
||||
"token": ""
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
- **Returns**: `application/octet-stream` — newline-delimited JSON events
|
||||
- **Token-based cursor**: response includes `mc-siem-token` header — pass this as `token` on the next call to get the next batch
|
||||
- **Retention**: 7 days only — pull at least every 6 hours, ideally hourly
|
||||
- **Update frequency**: new batches available every ~30 minutes
|
||||
- **Rate limit**: 300 calls/hour for this endpoint
|
||||
- **Required scope**: `Gateway | Tracking | Read`
|
||||
- **Prerequisite**: Enable **Enhanced Logging** in Administration Console > Account Settings
|
||||
|
||||
Key fields in each event:
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `Act` | Action taken (Acc=accepted, Rej=rejected, Hld=held) |
|
||||
| `ThreatDictionary` | Threat categories triggered |
|
||||
| `SpamInfo` | Spam score and verdict |
|
||||
| `URL` | Rewritten URL if TTP URL protection triggered |
|
||||
| `FileName` | Attachment name if scanned |
|
||||
| `SHA256` | Hash of attachment |
|
||||
| `SenderIP` | Originating IP |
|
||||
|
||||
---
|
||||
|
||||
### B. TTP Attachment Protection Logs
|
||||
|
||||
Sandboxing verdicts for email attachments — malware, ransomware, etc.
|
||||
|
||||
```http
|
||||
POST /api/ttp/attachment/get-logs
|
||||
|
||||
{
|
||||
"data": [{
|
||||
"from": "2026-01-01T00:00:00+0000",
|
||||
"to": "2026-01-02T00:00:00+0000",
|
||||
"route": "inbound",
|
||||
"result": "malicious",
|
||||
"pageSize": 100
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
- `route`: `inbound` | `outbound` | `internal` | `all`
|
||||
- `result`: `safe` | `malicious` | `timeout` | `error` | `unsafe` | `all`
|
||||
- **Required scope**: `Monitoring | Attachment Protection | Read`
|
||||
|
||||
Key response fields:
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `result` | Sandbox verdict |
|
||||
| `fileName` | Original attachment filename |
|
||||
| `sha256` | File hash (pivot to threat intel) |
|
||||
| `senderAddress` | From address |
|
||||
| `recipientAddress` | Target mailbox |
|
||||
| `actionTriggered` | What Mimecast did (block, sandbox, etc.) |
|
||||
| `date` | ISO 8601 timestamp |
|
||||
|
||||
---
|
||||
|
||||
### C. TTP URL Protection — Managed URL List
|
||||
|
||||
Retrieve blocked/tracked URLs from the managed threat list:
|
||||
|
||||
```http
|
||||
GET /api/ttp/url/get-managed-url
|
||||
```
|
||||
|
||||
Or decode a rewritten Mimecast URL back to the original for IOC extraction:
|
||||
|
||||
```http
|
||||
POST /api/ttp/url/decode-url
|
||||
|
||||
{
|
||||
"data": [{
|
||||
"url": "https://protect-eu.mimecast.com/s/ABC..."
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### D. Audit Events (Security Configuration Changes)
|
||||
|
||||
Policy changes, admin actions — useful for detecting misconfigurations or insider threats:
|
||||
|
||||
```http
|
||||
POST /api/audit/get-audit-events
|
||||
|
||||
{
|
||||
"data": [{
|
||||
"startDateTime": "2026-01-01T00:00:00+0000",
|
||||
"endDateTime": "2026-01-02T00:00:00+0000",
|
||||
"categories": ["policy", "account"]
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
- **Required scope**: `Account | Logs | Read`
|
||||
|
||||
---
|
||||
|
||||
## 4. Pagination
|
||||
|
||||
All paginated endpoints use cursor tokens:
|
||||
|
||||
```typescript
|
||||
async function fetchAllPages(endpoint: string, basePayload: object): Promise<any[]> {
|
||||
const results: any[] = [];
|
||||
let pageToken: string | undefined;
|
||||
|
||||
do {
|
||||
const body = {
|
||||
...basePayload,
|
||||
meta: pageToken ? { pagination: { pageToken } } : undefined
|
||||
};
|
||||
|
||||
const res = await fetch(`https://api.services.mimecast.com${endpoint}`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
Authorization: `Bearer ${token}`,
|
||||
'Content-Type': 'application/json',
|
||||
'x-mc-req-id': crypto.randomUUID()
|
||||
},
|
||||
body: JSON.stringify(body)
|
||||
});
|
||||
|
||||
const json = await res.json();
|
||||
results.push(...(json.data ?? []));
|
||||
pageToken = json.pagination?.next;
|
||||
} while (pageToken);
|
||||
|
||||
return results;
|
||||
}
|
||||
```
|
||||
|
||||
Default page size: 100. Maximum: 500. Set `pageSize` in the `data` array params.
|
||||
|
||||
---
|
||||
|
||||
## 5. Rate Limiting
|
||||
|
||||
| Header | Meaning |
|
||||
|--------|---------|
|
||||
| `X-RateLimit-Limit` | Total requests allowed in window |
|
||||
| `X-RateLimit-Remaining` | Calls left |
|
||||
| `X-RateLimit-Reset` | Milliseconds until reset |
|
||||
|
||||
On HTTP 429, back off exponentially. SIEM logs endpoint has a hard limit of **300 calls/hour**.
|
||||
|
||||
```typescript
|
||||
async function fetchWithRetry(url: string, options: RequestInit, retries = 3): Promise<Response> {
|
||||
const res = await fetch(url, options);
|
||||
if (res.status === 429 && retries > 0) {
|
||||
const resetMs = parseInt(res.headers.get('X-RateLimit-Reset') ?? '5000');
|
||||
await new Promise(r => setTimeout(r, resetMs));
|
||||
return fetchWithRetry(url, options, retries - 1);
|
||||
}
|
||||
return res;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Multi-Tenant (MSP) Access
|
||||
|
||||
When accessing multiple client accounts with a single credential set, pass `accountCode` in the meta:
|
||||
|
||||
```json
|
||||
{
|
||||
"data": [{ /* params */ }],
|
||||
"meta": {
|
||||
"accountCode": "CLIENT_ACCOUNT_CODE"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Each Mimecast tenant has a unique account code visible in the administration console.
|
||||
|
||||
---
|
||||
|
||||
## 7. Required Permission Scopes by Endpoint
|
||||
|
||||
| Endpoint | Required Scope |
|
||||
|----------|---------------|
|
||||
| SIEM Logs | `Gateway > Tracking > Read` |
|
||||
| Audit Events | `Account > Logs > Read` |
|
||||
| TTP Attachment Logs | `Monitoring > Attachment Protection > Read` |
|
||||
| TTP URL Management | `Gateway > Policies > Read` |
|
||||
|
||||
Request only the scopes needed. Grant least privilege per API application.
|
||||
|
||||
---
|
||||
|
||||
## 8. Dashboard Data Model Suggestion
|
||||
|
||||
For a threat dashboard showing a summary per client:
|
||||
|
||||
```sql
|
||||
-- Suggested local cache table
|
||||
CREATE TABLE mimecast_threat_events (
|
||||
id TEXT PRIMARY KEY,
|
||||
account_code TEXT NOT NULL,
|
||||
event_type TEXT NOT NULL, -- 'siem_mta' | 'ttp_attachment' | 'ttp_url'
|
||||
event_ts TIMESTAMPTZ NOT NULL,
|
||||
sender TEXT,
|
||||
recipient TEXT,
|
||||
sender_ip INET,
|
||||
threat_type TEXT, -- malicious | spam | spoofing | etc.
|
||||
action TEXT, -- blocked | sandboxed | delivered
|
||||
sha256 TEXT,
|
||||
url TEXT,
|
||||
raw JSONB,
|
||||
synced_at TIMESTAMPTZ DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX ON mimecast_threat_events (account_code, event_ts DESC);
|
||||
CREATE INDEX ON mimecast_threat_events (event_type, threat_type);
|
||||
```
|
||||
|
||||
Sync strategy: poll SIEM logs hourly using stored `mc-siem-token` per account. Store the cursor token per `account_code` in a config table.
|
||||
|
||||
---
|
||||
|
||||
## 9. Gotchas
|
||||
|
||||
- **Enhanced Logging must be enabled** in the Mimecast console before SIEM data appears — without it the endpoint returns empty.
|
||||
- **SIEM uses a streaming cursor**, not date ranges — don't skip token management or you'll re-read old events.
|
||||
- **HTTP 200 ≠ success** — always check `fail[]` in the response body.
|
||||
- **POST for reads** — don't expect REST conventions; almost everything is POST.
|
||||
- **Token expiry** — implement proactive refresh (check `expires_in`, refresh at 80% of TTL).
|
||||
- **Region routing** — if a client's tenant is in EU/AU, the base URL differs; use the discovery endpoint per account.
|
||||
182
docs/veeam-backup-alerting-recommendation.md
Normal file
182
docs/veeam-backup-alerting-recommendation.md
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
# Veeam Backup Alerting — Current State & RPO-Based Recommendation
|
||||
|
||||
> **Date:** April 14, 2026
|
||||
> **Author:** Pulse / Cascade
|
||||
> **Status:** Recommendation — not yet implemented
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem: The Current Alerting Pipeline is Noisy
|
||||
|
||||
### How It Works Today
|
||||
|
||||
Every Veeam backup alert that reaches Autotask passes through **Datto RMM** — Veeam does not create Autotask tickets directly. There are 6 Datto RMM monitors that watch for Veeam conditions (via Windows Event Log or script checks on each WNP/endpoint) and auto-create tickets when a condition is true:
|
||||
|
||||
| RMM Monitor ID | Description | Tickets (Last 30 Days) |
|
||||
|---|---|---|
|
||||
| **422325** | Veeam Agent Backup Stalled | 638 |
|
||||
| **770091** | Backup Copy Job Failed | 194 |
|
||||
| **369602** | Veeam Agent Backup Failed | 103 |
|
||||
| **369604** | Veeam Backup Job Missing or Stalled | 61 |
|
||||
| **392069** | Veeam Agent Job Finished with Failed | 58 |
|
||||
| **849483** | Veeam Server Agent Backup Job Stalled | 38 |
|
||||
|
||||
**Total: ~1,100 backup-related tickets per month** from these 6 monitors alone.
|
||||
|
||||
### Monthly Ticket Volume (Last 6 Months)
|
||||
|
||||
| Month | Backup Tickets |
|
||||
|---|---|
|
||||
| October 2025 | 461 |
|
||||
| November 2025 | 1,033 |
|
||||
| December 2025 | 1,008 |
|
||||
| January 2026 | 1,583 |
|
||||
| February 2026 | 728 |
|
||||
| March 2026 | 946 |
|
||||
|
||||
### The Core Defects
|
||||
|
||||
**1. No deduplication — one new ticket per check cycle.**
|
||||
A workstation that misses its backup today gets a new Autotask ticket every time the RMM monitor runs. `DT061` generated **41 individual tickets in 30 days** for a single laptop. In the last 30 days, **23 devices each triggered 5+ alerts**, producing **387 redundant tickets** from a single monitor.
|
||||
|
||||
**2. No schedule awareness.**
|
||||
The monitors check "has Veeam run in X days?" without knowing the job's schedule. A laptop that is legitimately offline over a weekend gets stalled alerts on Saturday and Sunday even though no backup was missed relative to its RPO.
|
||||
|
||||
**3. No auto-resolution.**
|
||||
When the underlying backup issue is fixed and the job succeeds, the open Autotask tickets are not automatically closed. Resolution requires manual action.
|
||||
|
||||
**4. No escalation logic.**
|
||||
A job that has been failing for 3 days looks identical in Autotask to one that missed a single run — both generate the same priority ticket.
|
||||
|
||||
**5. No failure context.**
|
||||
The RMM alerts contain only the device name and a generic "stalled" or "failed" label. The root cause (VBM desync, Wasabi DNS failure, VSS error, license expiry) is not surfaced in the ticket.
|
||||
|
||||
---
|
||||
|
||||
## 2. Recommendation: RPO-Based Alerting via Pulse
|
||||
|
||||
### What Already Exists
|
||||
|
||||
Pulse already contains a fully-built **RPO monitoring service** at `lib/services/veeam-rpo-service.ts` and an API endpoint at `POST /api/veeam/rpo-check`. The service:
|
||||
|
||||
- Reads live job data from the Veeam VSPC sync in PostgreSQL (`veeam_backup_agent_jobs`)
|
||||
- Computes whether each job has breached its **Recovery Point Objective (RPO)** — i.e., the acceptable maximum gap since the last successful backup — based on the job's configured schedule
|
||||
- Creates **one Autotask ticket per breached job** (deduped via the `veeam_rpo_tickets` tracking table)
|
||||
- **Auto-resolves** that ticket the moment a successful backup run is detected
|
||||
- **Escalates** the ticket's priority as the RPO violation ages
|
||||
- **Categorizes** the failure reason from the Veeam failure message
|
||||
|
||||
The `veeam_rpo_tickets` table exists and is ready. It currently has 0 rows because the service has never been run.
|
||||
|
||||
### RPO Threshold Logic
|
||||
|
||||
| Schedule Type | RPO Window | Alert After | Escalate to High | Escalate to Critical |
|
||||
|---|---|---|---|---|
|
||||
| **Daily** | 24h | +4h grace = 28h | +48h | +7 days |
|
||||
| **Weekly** | 168h | +4h grace = 172h | +48h | +7 days |
|
||||
| **Continuous** | 1h | +1h grace = 2h | +4h | +12h |
|
||||
|
||||
The **grace period** (4h for daily jobs) accounts for jobs that run slightly late due to the workstation being offline, slow networks, or queued jobs on the WNP — preventing false positives for on-time jobs with minor delays.
|
||||
|
||||
### Ticket Behavior
|
||||
|
||||
| Event | Current (RMM) | Proposed (RPO) |
|
||||
|---|---|---|
|
||||
| Job misses backup | New ticket every check cycle | One ticket opened, ticket title includes hours overdue |
|
||||
| Job still failing next day | Another new ticket | Same ticket remains open, priority escalated |
|
||||
| Job still failing after 2 days | Another new ticket | Ticket escalated to High |
|
||||
| Job still failing after 7 days | Another new ticket | Ticket escalated to Critical |
|
||||
| Backup succeeds | Tickets stay open, manual close | Ticket automatically resolved |
|
||||
| Machine stale >30 days | Continuous daily alert | Skip ticket creation (likely abandoned/decommissioned machine) |
|
||||
|
||||
### Ticket Content
|
||||
|
||||
RPO tickets are filed to **Operations Triage** queue with:
|
||||
- **Title:** `[Veeam RPO] <JobName> @ <OrgName> — <N>h since last backup`
|
||||
- **Issue Type:** Backups / Veeam Agent for Microsoft Windows
|
||||
- **Description:** Job name, org, schedule, last successful backup timestamp, hours overdue, restore points available, categorized failure reason, raw error
|
||||
|
||||
**Failure categories surfaced automatically:**
|
||||
- License Expired — renew via VSPC
|
||||
- Cloud Gateway Unreachable — check `vcg01.wulfconsulting.com`
|
||||
- Backup Repository Inaccessible
|
||||
- Service Provider Maintenance
|
||||
- Network/Connectivity Error
|
||||
- Backup Job Timeout
|
||||
- VBM desync (raw message)
|
||||
|
||||
---
|
||||
|
||||
## 3. Estimated Impact
|
||||
|
||||
| Metric | Current (RMM) | Projected (RPO) |
|
||||
|---|---|---|
|
||||
| Tickets/month (backup) | ~950–1,100 | **~50–150** (one per new breach, not per check) |
|
||||
| Duplicate tickets for same device | Up to 41/month | **0** (deduped by job UID) |
|
||||
| Manual ticket closures required | All | **0** (auto-resolved on success) |
|
||||
| Failure root cause in ticket | No | **Yes** (categorized + raw message) |
|
||||
| False positives (offline weekend) | Yes | **Minimal** (RPO + grace window) |
|
||||
| Escalation based on severity | No | **Yes** (medium → high → critical) |
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation Steps
|
||||
|
||||
### Step 1 — Enable Veeam Sync
|
||||
|
||||
The RPO service reads from the Veeam VSPC sync tables. The sync schedules already exist but are disabled:
|
||||
|
||||
```sql
|
||||
UPDATE sync_schedules SET is_enabled = true WHERE sync_type IN ('veeam-incremental', 'veeam-full');
|
||||
```
|
||||
|
||||
The 30-minute incremental sync keeps job status current enough for RPO evaluation.
|
||||
|
||||
### Step 2 — Add RPO Check Schedule
|
||||
|
||||
Add a schedule entry to run the RPO check every 2 hours:
|
||||
|
||||
```sql
|
||||
INSERT INTO sync_schedules (id, sync_type, cron_expression, is_enabled, description)
|
||||
VALUES ('veeam-rpo-check', 'veeam-rpo-check', '0 */2 * * *', true, 'Veeam RPO check — creates/escalates/resolves backup tickets');
|
||||
```
|
||||
|
||||
The scheduler needs to handle the `veeam-rpo-check` sync type by calling `POST /api/veeam/rpo-check`.
|
||||
|
||||
### Step 3 — Disable the 6 Datto RMM Backup Monitors
|
||||
|
||||
In Datto RMM, disable ticket creation (or disable the monitors entirely) for the 6 monitors listed in Section 1. This eliminates the duplicate ticket stream. The monitors can remain as alerting events in RMM itself without creating Autotask tickets, if desired.
|
||||
|
||||
> **Do not disable the monitors before the RPO service is confirmed working.** Run both in parallel for at least one week to validate coverage.
|
||||
|
||||
### Step 4 — Dry Run Before Go-Live
|
||||
|
||||
The RPO check endpoint supports a dry-run mode that shows what it *would* do without creating any tickets:
|
||||
|
||||
```bash
|
||||
curl -X POST https://pulse.wulfconsulting.cloud/api/veeam/rpo-check \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"dryRun": true}'
|
||||
```
|
||||
|
||||
This returns `wouldCreate`, `wouldResolve`, `wouldEscalate`, and `wouldSkipTooOld` counts along with the full job status list.
|
||||
|
||||
### Step 5 — Validate & Monitor
|
||||
|
||||
After enabling, confirm via Autotask that:
|
||||
- RPO tickets are appearing with `[Veeam RPO]` prefix in the title
|
||||
- Resolved tickets are auto-closing when backups succeed
|
||||
- Ticket count is trending down from the ~1,100/month baseline
|
||||
|
||||
---
|
||||
|
||||
## 5. What Is NOT Changing
|
||||
|
||||
- **Backup infrastructure is unchanged** — WNPs, Veeam agents, Wasabi, VSPC all remain identical
|
||||
- **Datto RMM monitoring** — RMM continues to monitor Veeam; only the ticket-creation action on those monitors is disabled
|
||||
- **Server/VM backup alerting** — this recommendation is scoped to workstation jobs only; server and VM backup alerting should be evaluated separately
|
||||
- **Restore workflows** — restore requests continue to be filed as standard Autotask tickets by users/staff
|
||||
|
||||
---
|
||||
|
||||
*This document was produced by analyzing Autotask ticket data and the Veeam VSPC sync in the Pulse PostgreSQL database. The RPO service implementation is in `lib/services/veeam-rpo-service.ts` and `app/api/veeam/rpo-check/route.ts`.*
|
||||
350
docs/workstation-backup-overview.md
Normal file
350
docs/workstation-backup-overview.md
Normal file
|
|
@ -0,0 +1,350 @@
|
|||
# Wulf Consulting — Workstation Backup Overview
|
||||
|
||||
> **Generated:** April 10, 2026
|
||||
> **Sources:** Veeam VSPC API (live sync via Pulse), Autotask PSA ticket data, IT Glue documentation
|
||||
> **Scope:** Workstation (laptop and desktop) backups only — server and VM backups are excluded
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
Wulf Consulting protects **1,219 workstation backup jobs** across **59 client organizations** using **Veeam Backup & Replication** managed through the **Veeam Service Provider Console (VSPC)**. Workstations are backed up via **Veeam Agent for Windows** (and 1 Mac agent) deployed to endpoints, with backup policies pushed centrally from on-premise Veeam Backup & Replication servers (WNPs) at each client site. Data flows to two tiers: **local on-prem repositories** and **Wasabi S3-compatible cloud storage** for offsite/immutable copies.
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Total workstation backup jobs | 1,219 |
|
||||
| Active Veeam workstation agents | 688 |
|
||||
| Client organizations covered | 59 |
|
||||
| Current success rate | **83.3%** (1,016 Success) |
|
||||
| Failed jobs | 84 (6.9%) |
|
||||
| Warning jobs | 2 |
|
||||
| Jobs with status "None" (never run / new) | 49 |
|
||||
| Backup-related Autotask tickets (last 12 months) | 13,946 |
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
### 2.1 Platform Stack
|
||||
|
||||
| Component | Details |
|
||||
|---|---|
|
||||
| **Backup Software** | Veeam Backup & Replication v12.3 / v13.0 |
|
||||
| **Management Console** | Veeam Service Provider Console (VSPC) at `vac.wulfconsulting.com:1280` |
|
||||
| **Agent** | Veeam Agent for Windows (687 workstations) + Veeam Agent for Mac (1 workstation) |
|
||||
| **Cloud Connect Server** | `wemvemasp02` — Veeam v13.0.1.1071, role: CloudConnect, status: Healthy |
|
||||
| **Client Backup Servers** | 46 on-premise Windows servers (WNPs) at client sites, all status: Healthy |
|
||||
| **Cloud Storage** | Wasabi S3 (us-east-1) via Veeam Cloud Connect — immutable object lock |
|
||||
| **Monitoring** | VSPC alarms → Datto RMM alerts → Autotask ticket auto-creation |
|
||||
|
||||
### 2.2 On-Premise Backup Servers (WNPs)
|
||||
|
||||
Each managed client site has a dedicated **Workstation Network Proxy (WNP)** — a Windows server running Veeam Backup & Replication that acts as the local backup infrastructure. These WNPs are registered to VSPC as "Client" role backup servers.
|
||||
|
||||
**Top WNPs by job count:**
|
||||
|
||||
| WNP Server | Client | Workstation Jobs |
|
||||
|---|---|---|
|
||||
| `pitseuwnp01` | Seubert and Associates | 274 |
|
||||
| `ynghynwnp01` | Hynes Industries | 254 |
|
||||
| `monprewnp01` | Premier Automation Holdings | 125 |
|
||||
| `geoprewnp01` | Premier Automation Holdings | 125 |
|
||||
| `PITPOHWNP01` | POH+W Architects | 86 |
|
||||
| `pgbvsywnp01` | V-Systems | 57 |
|
||||
| `hpbhrswnp01` | Hergenroeder Rega Ewing Kennedy | 44 |
|
||||
|
||||
All 46 client WNPs are running Veeam v12.3.2.4165 (two upgraded to v13.0.1.1071). All report **Healthy** status.
|
||||
|
||||
### 2.3 Data Flow
|
||||
|
||||
```
|
||||
Workstation (Veeam Agent)
|
||||
├── [1] Local backup → On-prem WNP repository (NAS / local disk)
|
||||
└── [2] Cloud copy → WNP → Cloud Connect (wemvemasp02) → Wasabi S3 (immutable)
|
||||
```
|
||||
|
||||
Each workstation runs a Veeam Agent that executes a daily backup job. Most workstations have **two jobs**: one targeting a local repository on the WNP and one replicating to Wasabi via the VSPC Cloud Connect gateway. The cloud copy is stored in per-machine immutable object-lock repositories (naming convention: `wulf-<client>_<hostname>_Repository`).
|
||||
|
||||
---
|
||||
|
||||
## 3. Backup Policies & Job Configuration
|
||||
|
||||
### 3.1 Standard Policies
|
||||
|
||||
Wulf deploys centrally managed backup policies through VSPC. The two primary workstation policies are:
|
||||
|
||||
| Policy Name | Jobs | Success | Failed | Backup Mode | Target |
|
||||
|---|---|---|---|---|---|
|
||||
| `WulfStdWRKSTFiles` | 519 | 455 | 28 | File-level | Cloud (Wasabi) |
|
||||
| `WulfStdWRKSTFilesWasabi` | 400 | 340 | 36 | File-level | Cloud (Wasabi) |
|
||||
| `Premier_WulfStdWRKSTFilesD_Drive` | 48 | 36 | 5 | File-level | Cloud |
|
||||
| `Premier_WulfStdWRKSTFilesD_Drive_Wasabi` | 35 | 29 | 5 | File-level | Cloud |
|
||||
|
||||
Additional per-site policies exist for clients with specific requirements (e.g., `POH_ATL_Desktops`, `Blake_Desktops`, `Greco_TAR_Desktops`), each targeting the local WNP.
|
||||
|
||||
### 3.2 Backup Modes
|
||||
|
||||
| Mode | Jobs | Avg Backed-Up Size | Total Data |
|
||||
|---|---|---|---|
|
||||
| **File-level** | 1,041 (85%) | 18.88 GB | 18.96 TB |
|
||||
| **Entire Computer** (volume/image) | 178 (15%) | 336.60 GB | 44.38 TB |
|
||||
|
||||
- **File-level** — protects user profile data (Documents, Desktop, Downloads, AppData). This is the standard for most workstations.
|
||||
- **Entire Computer** — full volume image backup. Used for specialized workstations (LOB app machines, CAD/engineering stations, machines with large local data stores).
|
||||
|
||||
### 3.3 Backup Targets
|
||||
|
||||
| Destination | Jobs | Total Data Protected |
|
||||
|---|---|---|
|
||||
| **Wasabi S3 (Cloud)** | 1,043 (86%) | 19.36 TB |
|
||||
| **Local (On-Prem WNP)** | 176 (14%) | 44.38 TB |
|
||||
|
||||
The cloud-targeted jobs use Wasabi S3-compatible storage via the VSPC Cloud Connect gateway (`wemvemasp02`). Local-targeted jobs write to storage attached to the client's WNP server.
|
||||
|
||||
### 3.4 Schedule
|
||||
|
||||
All 1,219 workstation backup jobs are configured with a **Daily** schedule. Per IT Glue documentation, typical trigger windows are:
|
||||
|
||||
- **Morning run:** 6:00 AM, 9:00 AM, or 11:00 AM (varies by client)
|
||||
- **Event-based:** On logoff/lock/restart events, with a minimum interval of 4–6 hours between runs
|
||||
- **Backup window:** 24×7 (jobs can run any time when the workstation is available)
|
||||
|
||||
> *IT Glue reference: Backup flexible asset documentation records per-client schedules. Example — Seubert and Associates desktops: "6am + log off event, not exceeding 6 hours" with Veeam Volume/Image method.*
|
||||
|
||||
### 3.5 Retention
|
||||
|
||||
**Per IT Glue Backup documentation (195 assets across all device types):**
|
||||
|
||||
| Retention Tier | Policy | Count |
|
||||
|---|---|---|
|
||||
| **Local** | Veeam 4 Weeks | 62 |
|
||||
| **Local** | Veeam 1 Week | 49 |
|
||||
| **Local** | Veeam 2 Weeks | 33 |
|
||||
| **Offsite** | GFS - 1 Year | 54 |
|
||||
| **Offsite** | Veeam O365 1 year | 14 |
|
||||
| **Offsite** | GFS - 6 Month | 1 |
|
||||
|
||||
Workstation cloud copies average **18.9 restore points** per job (file-level) and **14.5 restore points** per job (entire computer).
|
||||
|
||||
### 3.6 Immutable / Offsite Storage
|
||||
|
||||
Per IT Glue documentation, **54 backup assets** reference **Wasabi Object Storage** as the offsite immutable location. Immutable object lock ensures that backup data cannot be deleted or modified by ransomware or compromised credentials for the duration of the retention period.
|
||||
|
||||
Cloud repositories follow the naming convention `wulf.<client>.veeam.workstations.immutable` — e.g.:
|
||||
- `wulf.seubert.veeam.workstations.immutable`
|
||||
- `wulf.vsys.veeam.workstations.immutable`
|
||||
- `wulf.tcg.veeam.workstations.immutable`
|
||||
- `wulf.superior.veeam.workstations.immutable`
|
||||
|
||||
---
|
||||
|
||||
## 4. Client Coverage
|
||||
|
||||
### 4.1 Top Clients by Workstation Backup Volume
|
||||
|
||||
| Client | Jobs | Success | Failed | Warning |
|
||||
|---|---|---|---|---|
|
||||
| Seubert and Associates | 274 | 224 | 32 | 0 |
|
||||
| Hynes Industries | 254 | 225 | 14 | 0 |
|
||||
| Premier Automation Holdings | 125 | 101 | 13 | 0 |
|
||||
| POH+W Architects | 86 | 67 | 4 | 1 |
|
||||
| ConnecTel | 58 | 52 | 3 | 0 |
|
||||
| V-Systems | 57 | 53 | 2 | 0 |
|
||||
| Hergenroeder Rega Ewing Kennedy | 44 | 41 | 1 | 0 |
|
||||
| Superior Distributing Co | 40 | 35 | 2 | 0 |
|
||||
| Greco Gas | 35 | 29 | 2 | 0 |
|
||||
| Thoroughbred Construction Group | 32 | 31 | 0 | 0 |
|
||||
| Commonwealth Suburban Title Agency | 25 | 9 | 3 | 0 |
|
||||
| Nordmann Roofing | 17 | 17 | 0 | 0 |
|
||||
| Blake Dentistry | 15 | 15 | 0 | 0 |
|
||||
| TK Plastics Company | 13 | 13 | 0 | 0 |
|
||||
|
||||
### 4.2 IT Glue Backup Documentation
|
||||
|
||||
IT Glue contains **195 Backup flexible assets** across all managed clients, broken down by documented scope:
|
||||
|
||||
| Category | Documented Assets |
|
||||
|---|---|
|
||||
| Server backups | 69 |
|
||||
| Desktop backups | 44 |
|
||||
| Other / mixed | 32 |
|
||||
| Laptop backups | 30 |
|
||||
| M365 backups | 18 |
|
||||
| LOB application backups | 2 |
|
||||
|
||||
Each asset records: backup software, method, frequency, window, local retention, offsite provider, offsite schedule, offsite retention, protected devices (tagged), and restore approval contacts.
|
||||
|
||||
> *IT Glue also maintains 3 "Veeam (auto)" flexible assets that auto-document the VSPC tenant structure including sites, sub-tenants, and company mappings.*
|
||||
|
||||
---
|
||||
|
||||
## 5. Current Health & Failure Analysis
|
||||
|
||||
### 5.1 Job Status Breakdown
|
||||
|
||||
| Status | Count | Percentage |
|
||||
|---|---|---|
|
||||
| ✅ Success | 1,016 | 83.3% |
|
||||
| ❌ Failed | 84 | 6.9% |
|
||||
| ⚠️ Warning | 2 | 0.2% |
|
||||
| ⬜ None (never run) | 49 | 4.0% |
|
||||
| 🔒 Disabled | 0 | 0% |
|
||||
| 📉 Stale (no run in 48h) | 1,219* | — |
|
||||
|
||||
*\*Note: The "stale" count includes all jobs where `last_run` is older than 48 hours or null — this is inflated by the daily schedule; many jobs last ran within 24 hours but are captured by the 48h window. The actual number of concerning stale jobs is the 49 with status "None."*
|
||||
|
||||
### 5.2 Failure Root Causes
|
||||
|
||||
| Failure Pattern | Failed Jobs | Description |
|
||||
|---|---|---|
|
||||
| **VBM Desync (needs rescan)** | 28 | Backup metadata file out of sync with DB; requires manual repository rescan on the WNP |
|
||||
| **Other** | 35 | Mixed errors — agent communication failures, locked files, transient issues |
|
||||
| **Wasabi DNS/Connectivity** | 14 | Cannot resolve `s3.us-east-1.wasabisys.com` — DNS or internet outage at client site |
|
||||
| **VSPC Connectivity** | 6 | "Service provider's infrastructure is not responding" — VSPC gateway temporarily unavailable |
|
||||
| **VSS Error** | 1 | Volume Shadow Copy failure on the workstation |
|
||||
|
||||
The **dominant failure mode** is VBM (Veeam Backup Metadata) desynchronization, which requires a repository rescan on the affected WNP. This is a known Veeam issue that typically resolves after the rescan completes.
|
||||
|
||||
### 5.3 Backup Job Performance
|
||||
|
||||
| Backup Mode | Avg Duration (sec) | Avg Backed-Up Size (GB) |
|
||||
|---|---|---|
|
||||
| Entire Computer | 3,212 (~53 min) | 332.21 GB |
|
||||
| File-level | 1,971 (~33 min) | 17.75 GB |
|
||||
|
||||
---
|
||||
|
||||
## 6. Autotask Ticket Data — Operational Insights
|
||||
|
||||
### 6.1 Backup Ticket Volume (Last 12 Months)
|
||||
|
||||
Wulf's Autotask PSA logged **13,946 backup-related tickets** in the past 12 months. These are predominantly auto-generated alerts from the monitoring pipeline (Datto RMM → Autotask webhook):
|
||||
|
||||
| Category | Tickets | Resolved |
|
||||
|---|---|---|
|
||||
| Backup Failure alerts | 7,181 | 7,177 (99.9%) |
|
||||
| Veeam Agent issues | 5,378 | 5,358 (99.6%) |
|
||||
| Veeam General | 1,036 | 1,031 (99.5%) |
|
||||
| Restore Requests | 223 | 222 (99.6%) |
|
||||
| Backup General | 128 | 125 (97.7%) |
|
||||
|
||||
The vast majority of backup tickets are **automated monitoring alerts** that are triaged and resolved by the operations team. The **99.5%+ resolution rate** indicates an effective remediation workflow.
|
||||
|
||||
### 6.2 Recent Ticket Examples (April 2026)
|
||||
|
||||
**Automated alerts (Priority 6 — Monitoring):**
|
||||
- `T20260410.0054` — *Backup Alert - SP021 (Surface Pro 9) - Veeam Agent Backup Stalled (POH+W Architects - Atlanta)*
|
||||
- `T20260410.0047` — *Backup Alert - LT065 (21MV LENOVO) - Veeam Agent Backup Stalled* — ConnecTel
|
||||
- `T20260410.0036` — *Backup Alert - YNGHYNLT084 (Precision 7680, Dell) - Veeam Agent Backup Stalled* — Hynes Industries
|
||||
- `T20260410.0028` — *Backup Alert - pgbvsywnp01 - Veeam Backup Job Missing or Stalled* — V-Systems
|
||||
|
||||
**Backup copy failures (server-side, affects workstation offsite copies):**
|
||||
- `T20260410.0065` — *Backup Copy Job Failed - FOSSUPWNP01 - Superior-FOS-BackupCopyVMs-WasabiV2* — Superior Distributing Co
|
||||
- `T20260410.0049` — *Backup Copy Job Failed - ynghynwnp01 - Hynes-BackupCopyVMs-WulfWasabiV2* — Hynes Industries
|
||||
|
||||
### 6.3 Restore Requests (Last 6 Months — Sample)
|
||||
|
||||
| Date | Ticket | Description | Client |
|
||||
|---|---|---|---|
|
||||
| 2026-04-06 | T20260406.0227 | Folder Restore | Thrasher Group |
|
||||
| 2026-03-19 | T20260319.0350 | Restore overwritten Excel file from S:\Filestore | Insurance Restoration Consultants |
|
||||
| 2026-03-16 | T20260316.0288 | Restore overwritten Photoshop file from G: drive | POH+W Architects |
|
||||
| 2026-03-04 | T20260304.0210 | Device reimage — restore files via Veeam after wipe | Kuhn's Quality Foods |
|
||||
| 2026-03-03 | T20260303.0189 | Veeam 365 Restore Request — OneDrive recordings | Lighthouse Electric |
|
||||
| 2026-03-01 | T20260301.0108 | Restore archived Outlook calendar items | Hergenroeder Rega Ewing Kennedy |
|
||||
| 2026-02-26 | T20260226.0163 | Restore emptied shared folder from file server | Blackburn's Physicians Pharmacy |
|
||||
|
||||
Restore requests demonstrate the breadth of recovery scenarios handled: accidental file deletion, file overwrites, full device reimaging, and application-level restores (Outlook, OneDrive).
|
||||
|
||||
---
|
||||
|
||||
## 7. IT Glue Documentation Structure
|
||||
|
||||
### 7.1 Backup Flexible Asset Schema
|
||||
|
||||
Each client's workstation backup is documented in IT Glue as a **Backup** flexible asset (type ID: 3791) with the following fields:
|
||||
|
||||
| Field | Type | Purpose |
|
||||
|---|---|---|
|
||||
| Backup Software | Select | Platform (Veeam for all workstations) |
|
||||
| Backup Method | Select | Files, Volume/Image, Office365 Veeam, Other |
|
||||
| Backup Description | Text | What is being backed up |
|
||||
| Backup Frequency | Text | Schedule description (e.g., "6am + log off event") |
|
||||
| Backup Window | Select | When backups can run (typically 24×7) |
|
||||
| Wulf Backup Package | Select | Service tier: All-Desktops, All-Laptops, All-Servers, Selective, etc. |
|
||||
| Local Backup Server(s) | Tag | Tagged configuration items (WNP servers) |
|
||||
| Local Location | Text | Repository path (e.g., `D:\Backup`) |
|
||||
| Local Retention | Select | Veeam 1 Week / 2 Weeks / 4 Weeks |
|
||||
| Offsite Provider | Select | Wulf, Veeam, Datto, Druva, Other |
|
||||
| Offsite Replication Schedule | Select | Frequency of offsite copy |
|
||||
| Offsite Retention | Select | GFS-1 Year, GFS-6 Month, Veeam O365 1 Year |
|
||||
| Offsite Immutable Location | Select | Wasabi Object Storage |
|
||||
| Protected Devices | Tag | Tagged workstation configuration items |
|
||||
| Who Approves Restore Requests? | Tag | Contact(s) authorized to approve restores |
|
||||
| Last Backup Verification | Date | Date of last manual restore test |
|
||||
| Next Verification | Date | Scheduled next restore verification |
|
||||
|
||||
### 7.2 Documented Backup Methods (All Device Types)
|
||||
|
||||
| Method | Assets |
|
||||
|---|---|
|
||||
| Volume/Image | 127 |
|
||||
| Files | 47 |
|
||||
| Office365 Veeam | 16 |
|
||||
| Other | 4 |
|
||||
|
||||
### 7.3 Wulf Backup Packages
|
||||
|
||||
| Package | Assets |
|
||||
|---|---|
|
||||
| All - Servers | 60 |
|
||||
| All - Desktops | 36 |
|
||||
| Selective - See Protected Devices | 34 |
|
||||
| All - Laptops | 24 |
|
||||
| All - O365 | 17 |
|
||||
| TAM - Sell | 6 |
|
||||
|
||||
---
|
||||
|
||||
## 8. Standard Operating Procedure
|
||||
|
||||
Based on the data above, Wulf Consulting's standard workstation backup workflow is:
|
||||
|
||||
1. **Deployment:** Veeam Agent for Windows is installed on each managed workstation during onboarding. The agent is registered to the client's on-premise WNP server.
|
||||
|
||||
2. **Policy Assignment:** A VSPC backup policy (e.g., `WulfStdWRKSTFiles` or `WulfStdWRKSTFilesWasabi`) is assigned to the agent. This determines backup mode (file-level or volume), schedule, and target.
|
||||
|
||||
3. **Daily Execution:** The agent runs on a daily schedule (typically early morning + logoff/lock events). File-level jobs take ~33 minutes on average; full image jobs take ~53 minutes.
|
||||
|
||||
4. **Local Storage:** For clients with on-prem image backups, data is written to the WNP's local repository (NAS or direct-attached storage). Local retention is typically 1–4 weeks.
|
||||
|
||||
5. **Cloud Replication:** A second job (or backup copy job on the WNP) replicates data to Wasabi S3 via the VSPC Cloud Connect gateway (`wemvemasp02`). Cloud copies are stored in per-machine **immutable** repositories with GFS retention (typically 1 year).
|
||||
|
||||
6. **Monitoring:** VSPC monitors job status and raises alarms. These propagate to Datto RMM, which creates Autotask tickets automatically. The operations team triages alerts daily.
|
||||
|
||||
7. **Remediation:** Failed jobs are investigated — common fixes include repository rescans (VBM desync), DNS resolution (Wasabi connectivity), and agent reinstalls.
|
||||
|
||||
8. **Restores:** End users or client contacts request restores via Autotask ticket. Wulf engineers recover files from local or cloud repositories as needed.
|
||||
|
||||
9. **Documentation:** Each client's backup configuration is documented in IT Glue with method, schedule, retention, protected devices, and restore approval contacts.
|
||||
|
||||
---
|
||||
|
||||
## 9. Key Observations & Recommendations
|
||||
|
||||
### Strengths
|
||||
- **Comprehensive coverage:** 688 active workstation agents across 59 clients with centralized VSPC management
|
||||
- **Immutable offsite copies:** Wasabi S3 object lock protects against ransomware and accidental deletion
|
||||
- **Automated monitoring pipeline:** VSPC → Datto RMM → Autotask ensures no backup failure goes unnoticed
|
||||
- **High resolution rate:** 99.5%+ of backup tickets are resolved, indicating effective operational processes
|
||||
- **Standardized policies:** Two primary policies (`WulfStdWRKSTFiles`, `WulfStdWRKSTFilesWasabi`) cover 75% of jobs
|
||||
|
||||
### Areas for Attention
|
||||
- **VBM desync failures (28 jobs):** The most common failure mode. Consider automating repository rescans or upgrading to Veeam v13 which improved metadata handling.
|
||||
- **Wasabi DNS failures (14 jobs):** Client-side DNS resolution issues. May benefit from secondary DNS or direct-IP fallback configuration.
|
||||
- **49 jobs with "None" status:** Jobs that have never run — likely newly deployed agents awaiting first execution or stale configurations.
|
||||
- **84 total failed jobs (6.9%):** While the alert pipeline catches these, the raw failure rate could be improved by addressing the top two root causes (VBM desync + Wasabi DNS).
|
||||
|
||||
---
|
||||
|
||||
*This document was compiled from live VSPC data synced to Pulse, Autotask ticket history, and IT Glue backup documentation. Data reflects the state as of April 10, 2026.*
|
||||
526
docs/wulf-pulse-ticket-analyzer-prompt.md
Normal file
526
docs/wulf-pulse-ticket-analyzer-prompt.md
Normal file
|
|
@ -0,0 +1,526 @@
|
|||
# Feature: AI Ticket Analyzer (wulf-pulse)
|
||||
|
||||
Add an on-demand AI-powered ticket analysis feature to the existing **wulf-pulse** app at `forgejo.wulfconsulting.cloud/lorentz/wulf-pulse`.
|
||||
|
||||
This is a **feature addition**, not a new project. Conform to existing wulf-pulse conventions: React + Vite + TailwindCSS + shadcn/ui frontend, Node.js + Express backend, Entra ID OIDC auth, the existing PostgreSQL instance (the one that already has Autotask data syncing into it), Forgejo CI, Pangolin reverse proxy. Do not introduce new frameworks. Match the existing folder layout, error handling style, and route conventions.
|
||||
|
||||
Before writing any code, read `claude.md`, the route registration file, the auth middleware, and the data-access layer for tickets so the feature plugs into the existing patterns. If those files don't exist, ask before guessing.
|
||||
|
||||
---
|
||||
|
||||
## What this feature does
|
||||
|
||||
A wulf-pulse user opens a ticket view, clicks **Analyze**, and the system produces a structured analysis covering:
|
||||
|
||||
- A unified chronological timeline (with markers for source type)
|
||||
- What was actually done vs. what should have been done
|
||||
- Gaps — including subtle ones like the customer telling us to stop while work continued, status not matching reality, or the original ask never being directly answered
|
||||
- Recommended next step with rationale
|
||||
- Post-resolution analysis (if the ticket is resolved)
|
||||
- Confidence score and human-review flag
|
||||
- Referenced IT Glue documentation
|
||||
|
||||
The analysis is stored versioned by ticket number, can be re-run when new activity arrives, and can be emailed to other Wulf users.
|
||||
|
||||
---
|
||||
|
||||
## Critical: how Autotask notes/entries actually work
|
||||
|
||||
The current generic prompt would mis-handle the real Autotask note structure. Here are the distinctions the analyzer **must** make:
|
||||
|
||||
### Note types the analyzer must classify in pre-processing (Stage 0)
|
||||
|
||||
1. **Workflow rule firings** — `Note | Autotask Administrator` with title like `Workflow Rule "..." fired.` These are **pure noise**. Filter them out entirely before any model call. Tag each as `workflow_noise`.
|
||||
|
||||
2. **Service Desk Notification emails** — `Note | <tech name>` with title `Service Desk Notification` and a description that's just a list of email addresses. These are auto-generated email send confirmations. **Filter out** before model calls. Tag as `email_notification`.
|
||||
|
||||
3. **Ticket Notes** — `Ticket Note | <person name>`. These are real communications, often from the customer or a forwarded email. **Keep.** Tag as `customer_communication` or `internal_communication` based on the person's email domain.
|
||||
|
||||
4. **Time Entry Summary Notes** — the `Summary Notes` field of a Time Entry. **Customer-visible.** Tag as `time_entry_summary`. Keep.
|
||||
|
||||
5. **Time Entry Internal Notes** — the `Internal Notes` field of the same Time Entry. **Technician-only.** Tag as `time_entry_internal`. Keep — these are usually the highest-signal entries.
|
||||
|
||||
A single Time Entry can have BOTH Summary Notes and Internal Notes — they should appear as **two separate timeline events** with the same timestamp but different visibility markers, OR as a single event with both fields preserved. Choose the latter for cleaner timelines but always render them visually distinct.
|
||||
|
||||
### The tagging schema
|
||||
|
||||
Every retained event in the unified timeline must have:
|
||||
```ts
|
||||
{
|
||||
timestamp: string, // ISO 8601
|
||||
actor: string, // person name
|
||||
actor_type: "wulf_tech" | "client_contact" | "vendor" | "system" | "automation",
|
||||
source: "ticket_create" | "ticket_note" | "time_entry" | "status_change" | "resolution",
|
||||
visibility: "customer_facing" | "internal_only" | "mixed", // mixed = time entry with both fields
|
||||
summary_notes?: string, // customer-facing content if present
|
||||
internal_notes?: string, // internal content if present
|
||||
hours?: number, // for time entries
|
||||
}
|
||||
```
|
||||
|
||||
Render markers in the analysis output as:
|
||||
- 🟢 customer-facing
|
||||
- 🔒 internal-only
|
||||
- 🔄 mixed (both)
|
||||
|
||||
---
|
||||
|
||||
## Real example to test against
|
||||
|
||||
Use this real ticket as a fixture for your tests. The analyzer must catch all four findings listed below — if any are missed, the analysis prompts need refinement.
|
||||
|
||||
**Ticket T20260424.0045** — "Outmarket AI vendor integration request"
|
||||
|
||||
The internal notes reveal:
|
||||
1. The original ask was narrower than the public summary suggests — Lorentz's email said "I'm gonna need access to that for another integration with the claims department for loss run pro please let me know where that credential is in Passportal."
|
||||
2. At 04/24 10:34, Lorentz posted a Ticket Note saying "I was able to access the Vertafore Developer portal and determine what is necessary - no need to reach out to Vertafore. I'll take it from here, thank you!"
|
||||
3. On 04/27 (next business day), the assigned tech took a call from Vertafore anyway and logged ~20 min of additional work after the customer said to stop.
|
||||
4. Status is still "Waiting Customer" three days after the requestor effectively closed the loop.
|
||||
|
||||
The analyzer **must** flag:
|
||||
- **Gap (high):** Work continued after the customer indicated they were taking it from here.
|
||||
- **Gap (medium):** Status hasn't been updated to reflect the requestor's resolution.
|
||||
- **Gap (low):** The original credential-locator ask was never directly answered before the conversation pivoted.
|
||||
- **Next step:** Confirm with requestor whether the Vertafore endpoint info is still useful, then close.
|
||||
|
||||
Build a test fixture from this ticket (PDF is available, transcribe the structured fields) and assert these findings appear in the analysis output. The fixture lives in `apps/api/test/fixtures/tickets/T20260424.0045.json`.
|
||||
|
||||
---
|
||||
|
||||
## Database changes
|
||||
|
||||
Add these tables to the wulf-pulse Postgres database. Use the project's existing migration tool. Prefix all new tables with `analyzer_` to keep them clearly scoped to this feature.
|
||||
|
||||
### `analyzer_analyses`
|
||||
|
||||
```sql
|
||||
id uuid pk default gen_random_uuid()
|
||||
ticket_number text not null
|
||||
autotask_ticket_id bigint not null
|
||||
analysis_version int not null -- monotonic per ticket_number
|
||||
content_hash_at_analysis text not null -- sha256 of source data at analysis time
|
||||
triggered_by_user_id uuid -- references existing pulse users table
|
||||
triggered_at timestamptz default now()
|
||||
status text not null default 'pending' -- pending|running|complete|failed
|
||||
completed_at timestamptz
|
||||
|
||||
-- model usage
|
||||
haiku_used boolean default false
|
||||
sonnet_used boolean default false
|
||||
opus_used boolean default false
|
||||
total_input_tokens int default 0
|
||||
total_output_tokens int default 0
|
||||
estimated_cost_usd numeric(10,4) default 0
|
||||
|
||||
-- structured output
|
||||
summary text
|
||||
timeline jsonb -- unified, with visibility markers
|
||||
what_was_done jsonb
|
||||
what_should_have_been_done jsonb
|
||||
gaps jsonb -- [{description, severity, evidence_timestamps}]
|
||||
next_step text
|
||||
next_step_rationale text
|
||||
post_resolution_analysis text
|
||||
confidence_score numeric(3,2)
|
||||
needs_human_review boolean default false
|
||||
human_review_reasons jsonb
|
||||
|
||||
-- IT Glue
|
||||
itglue_docs_referenced jsonb default '[]'
|
||||
|
||||
-- debugging
|
||||
model_traces jsonb
|
||||
filtered_noise_count int default 0 -- how many workflow/notification entries were stripped
|
||||
error_message text
|
||||
|
||||
unique (ticket_number, analysis_version)
|
||||
```
|
||||
|
||||
Indexes: `(ticket_number, analysis_version desc)`, `(triggered_at desc)`, `(needs_human_review) where needs_human_review = true`.
|
||||
|
||||
### `analyzer_shares`
|
||||
|
||||
```sql
|
||||
id uuid pk default gen_random_uuid()
|
||||
analysis_id uuid not null references analyzer_analyses(id) on delete cascade
|
||||
shared_by_user_id uuid not null
|
||||
shared_with_email text not null -- validate against ALLOWED_SHARE_DOMAINS
|
||||
note text
|
||||
shared_at timestamptz default now()
|
||||
viewed_at timestamptz
|
||||
```
|
||||
|
||||
### `analyzer_jobs`
|
||||
|
||||
```sql
|
||||
id uuid pk default gen_random_uuid()
|
||||
ticket_number text not null
|
||||
queued_by_user_id uuid
|
||||
status text not null default 'queued' -- queued|fetching|triaging|itglue|analyzing|deep_review|complete|failed
|
||||
result_analysis_id uuid references analyzer_analyses(id)
|
||||
queued_at timestamptz default now()
|
||||
started_at timestamptz
|
||||
finished_at timestamptz
|
||||
error_message text
|
||||
```
|
||||
|
||||
If wulf-pulse already uses BullMQ or another job queue, plug into it. If not, a simple Postgres-row-based queue with a worker polling every 2 seconds is acceptable for an on-demand-only feature — discuss with me before pulling in a new dependency.
|
||||
|
||||
---
|
||||
|
||||
## Backend changes
|
||||
|
||||
### Source-of-truth question (ask me before deciding)
|
||||
|
||||
Wulf-pulse already syncs Autotask data to Postgres. **Before implementing**, look at the existing sync to determine:
|
||||
|
||||
1. Does the Pulse sync include ticket notes and time entries, or just ticket headers?
|
||||
2. How fresh is the sync? Real-time (webhook), minute-level, or hourly?
|
||||
3. Are Internal Notes synced? (They may be excluded from some syncs for privacy reasons.)
|
||||
|
||||
Based on what you find, choose one of:
|
||||
- **(A) Read everything from Pulse Postgres** — preferred if notes + internal notes + time entries are all synced and fresh.
|
||||
- **(B) Pull live from Autotask REST at analyze-time** — required if the sync is incomplete.
|
||||
- **(C) Hybrid: Pulse for fast list/search, live REST fetch for the full payload at analyze-time** — most likely the right answer.
|
||||
|
||||
Tell me which one fits before writing the data-access layer.
|
||||
|
||||
### New routes (mount under existing wulf-pulse API namespace)
|
||||
|
||||
```
|
||||
POST /api/analyzer/tickets/:ticketNumber/analyze
|
||||
# body: { force?: boolean }
|
||||
# returns: { jobId, status, existingAnalysisId? }
|
||||
|
||||
GET /api/analyzer/jobs/:jobId # poll status
|
||||
GET /api/analyzer/analyses/:id # fetch a specific analysis
|
||||
GET /api/analyzer/tickets/:ticketNumber/analyses
|
||||
# list versions
|
||||
GET /api/analyzer/needs-review # filtered queue
|
||||
|
||||
POST /api/analyzer/analyses/:id/share
|
||||
# body: { recipientEmail, note? }
|
||||
```
|
||||
|
||||
All routes require existing wulf-pulse auth middleware.
|
||||
|
||||
### IT Glue client
|
||||
|
||||
New module `apps/api/src/services/itglue/`:
|
||||
|
||||
- `client.ts` — REST client with `x-api-key` auth, retry/backoff
|
||||
- `redact.ts` — strips fields matching `/password|secret|key|token|credential|api[_-]?key/i` (case-insensitive, recursive) BEFORE any value reaches the LLM or the database. Replace with `"[REDACTED]"`. Include unit tests for nested objects and arrays.
|
||||
- `search.ts` — given a client name and search hints, returns sanitized doc snippets capped at 2000 chars per doc, max 10 docs.
|
||||
|
||||
The redaction is a **security-critical** code path. Add a comment explaining why and link to this prompt section. Do not log full doc bodies anywhere — only IDs and names.
|
||||
|
||||
### Anthropic SDK setup
|
||||
|
||||
Add `@anthropic-ai/sdk` to `apps/api/package.json`. Create `apps/api/src/services/llm/`:
|
||||
|
||||
- `client.ts` — singleton SDK instance, reads `ANTHROPIC_API_KEY` from env
|
||||
- `pricing.ts` — per-model input/output rates with a comment to verify against `https://docs.claude.com/en/docs/about-claude/pricing` quarterly. Don't hardcode rates without comments noting the as-of date.
|
||||
- `models.ts` — exports the canonical model IDs:
|
||||
- `HAIKU = "claude-haiku-4-5"`
|
||||
- `SONNET = "claude-sonnet-4-6"`
|
||||
- `OPUS = "claude-opus-4-7"`
|
||||
|
||||
Before finalizing those constants, verify each model ID is current and available on the API. If you find newer versions or the IDs are wrong, ask me before substituting.
|
||||
|
||||
---
|
||||
|
||||
## The analysis pipeline
|
||||
|
||||
Implemented as `apps/api/src/services/analyzer/pipeline.ts`. Each stage is a separate function for testability.
|
||||
|
||||
### Stage 0 — Fetch & Pre-process
|
||||
|
||||
1. Resolve ticket via the data-access strategy chosen above.
|
||||
2. **Filter noise:** strip `Note | Autotask Administrator` workflow firings and `Service Desk Notification` notes. Count them and store in `filtered_noise_count` for transparency.
|
||||
3. **Tag remaining events** per the schema in the "How Autotask notes actually work" section.
|
||||
4. **Sort chronologically.**
|
||||
5. Compute `content_hash = sha256(canonical_json({tagged_events, ticket_status, ticket_priority, queue}))`.
|
||||
6. **Idempotency check:** if `force=false` and a complete analysis exists with the same hash, short-circuit.
|
||||
|
||||
### Stage 1 — Triage (Haiku)
|
||||
|
||||
**Model:** `claude-haiku-4-5`
|
||||
|
||||
System prompt:
|
||||
```
|
||||
You are a ticket triage assistant for Wulf Consulting, an MSP. You will receive
|
||||
an Autotask ticket with notes and time entries that have already been pre-filtered
|
||||
to remove workflow noise and tagged by visibility (customer-facing vs internal).
|
||||
|
||||
Extract structured metadata and assess complexity. Pay special attention to
|
||||
internal-only notes — these often contain the real story.
|
||||
|
||||
Respond ONLY with JSON:
|
||||
|
||||
{
|
||||
"ticket_type": "incident" | "service_request" | "problem" | "change" | "other",
|
||||
"category": string,
|
||||
"entities": {
|
||||
"client_name": string | null,
|
||||
"site_name": string | null,
|
||||
"devices": string[],
|
||||
"users": string[],
|
||||
"applications": string[],
|
||||
"vendors": string[] // third parties involved (Vertafore, etc.)
|
||||
},
|
||||
"is_resolved": boolean,
|
||||
"status_matches_reality": boolean, // does the Autotask status reflect the actual state?
|
||||
"complexity_tier": "low" | "medium" | "high",
|
||||
"complexity_reasons": string[],
|
||||
"itglue_lookup_needed": boolean,
|
||||
"itglue_search_hints": string[]
|
||||
}
|
||||
|
||||
Complexity rubric:
|
||||
- low: single straightforward issue, ≤3 retained events, clear path
|
||||
- medium: multiple events, some back-and-forth, moderate ambiguity
|
||||
- high: any of — bounced between techs, conflicting notes, unresolved >5 days,
|
||||
customer-vs-internal narrative mismatch, multiple vendors involved,
|
||||
or status appears to disagree with the actual state of the work
|
||||
```
|
||||
|
||||
User message: a structured payload containing:
|
||||
- Ticket header fields (title, status, priority, queue, account, contact, dates)
|
||||
- Tagged event list (filtered + tagged from Stage 0)
|
||||
- Counts: total events, internal-only count, customer-facing count
|
||||
|
||||
Cap total payload at ~50KB. If larger, truncate oldest internal-only events first (preserving all customer-facing communications), and add a marker.
|
||||
|
||||
### Stage 2 — IT Glue Retrieval (conditional)
|
||||
|
||||
Run if `itglue_lookup_needed === true`.
|
||||
|
||||
1. Resolve client name → IT Glue org ID. Maintain a small JSON alias map at `apps/api/src/services/itglue/aliases.json` for known fuzzy mappings (e.g. "Seubert" / "Seubert and Associates" / "S&A" → org id). Document this file in the README.
|
||||
2. For each search hint, query configurations, flexible_assets, and documents endpoints.
|
||||
3. Dedupe, cap at 10 docs total.
|
||||
4. Run each result through `redact.ts` BEFORE adding to context.
|
||||
5. Cap each doc body at 2000 chars in the LLM context.
|
||||
|
||||
### Stage 3 — Deep Analysis (Sonnet)
|
||||
|
||||
**Model:** `claude-sonnet-4-6`
|
||||
|
||||
System prompt (verbatim):
|
||||
```
|
||||
You are a senior MSP technician at Wulf Consulting reviewing a ticket. You will
|
||||
receive:
|
||||
1. The full ticket with all retained notes and time entries (already filtered for
|
||||
workflow noise; tagged with visibility markers — customer_facing, internal_only,
|
||||
or mixed)
|
||||
2. Triage metadata from a previous pass
|
||||
3. Optionally, sanitized IT Glue documentation snippets for the client
|
||||
|
||||
Be specific and reference events by their timestamp and actor. Do not invent facts.
|
||||
If something is unclear, say so explicitly.
|
||||
|
||||
Pay particular attention to these patterns, which are common failure modes:
|
||||
- The customer indicates they have resolved the issue or want to take it over,
|
||||
but work continues afterward
|
||||
- The Autotask status does not match the actual state (e.g. "Waiting Customer"
|
||||
when the customer has already responded, or "In Progress" with no recent activity)
|
||||
- The original ask in the requester's first message is different from what the
|
||||
ticket pivoted to addressing
|
||||
- Internal notes contradict or add important context missing from customer-facing
|
||||
summary notes
|
||||
- A vendor case was opened but the customer's direct ask could have been answered
|
||||
without it
|
||||
- Time was billed for work the customer didn't ultimately need
|
||||
|
||||
Respond ONLY with JSON:
|
||||
|
||||
{
|
||||
"summary": string, // 2-4 sentences, neutral tone
|
||||
"timeline": [
|
||||
{
|
||||
"timestamp": string, // ISO 8601
|
||||
"actor": string,
|
||||
"actor_type": "wulf_tech" | "client_contact" | "vendor" | "system" | "automation",
|
||||
"source": "ticket_create" | "ticket_note" | "time_entry" | "status_change" | "resolution",
|
||||
"visibility": "customer_facing" | "internal_only" | "mixed",
|
||||
"action": string // what happened, in plain language
|
||||
}
|
||||
],
|
||||
"what_was_done": string[], // concrete actions in order
|
||||
"what_should_have_been_done": string[], // ideal actions per MSP best practice
|
||||
// and any IT Glue docs provided
|
||||
"gaps": [
|
||||
{
|
||||
"description": string,
|
||||
"severity": "low" | "medium" | "high",
|
||||
"evidence_timestamps": string[] // which timeline events support this
|
||||
}
|
||||
],
|
||||
"next_step": string, // single concrete next action
|
||||
"next_step_rationale": string,
|
||||
"post_resolution_analysis": string | null, // only if is_resolved=true
|
||||
"confidence_score": number, // 0.0–1.0
|
||||
"needs_human_review": boolean,
|
||||
"human_review_reasons": string[],
|
||||
"ambiguities_for_opus": string[], // questions a deeper-reasoning model should resolve
|
||||
"itglue_docs_referenced": [
|
||||
{
|
||||
"id": string,
|
||||
"name": string,
|
||||
"url": string,
|
||||
"doc_type": string,
|
||||
"relevance_reason": string
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Set needs_human_review=true if any of:
|
||||
- confidence_score < 0.6
|
||||
- gaps contain any "high" severity item
|
||||
- ticket open >7 days with no clear resolution path
|
||||
- conflicting information between notes
|
||||
- billed hours appear excessive for the work performed
|
||||
```
|
||||
|
||||
### Stage 4 — Deep Reasoning (Opus, conditional)
|
||||
|
||||
**Model:** `claude-opus-4-7`
|
||||
|
||||
Trigger conditions (any one):
|
||||
- `complexity_tier === "high"` from Stage 1
|
||||
- `ambiguities_for_opus.length > 0` from Stage 3
|
||||
- `confidence_score < 0.5` from Stage 3
|
||||
- Stage 1's `status_matches_reality === false`
|
||||
|
||||
System prompt:
|
||||
```
|
||||
You are a principal-level MSP engineer doing a final review of a complex ticket.
|
||||
You will be given:
|
||||
1. The full tagged ticket
|
||||
2. The Sonnet-tier analysis
|
||||
3. A list of specific ambiguities or open questions
|
||||
|
||||
Address each ambiguity directly with reasoning. Then produce updates ONLY for
|
||||
fields that should change.
|
||||
|
||||
Respond ONLY with JSON:
|
||||
|
||||
{
|
||||
"opus_notes": string, // your reasoning, 1–3 paragraphs
|
||||
"updates": {
|
||||
// any subset of: next_step, next_step_rationale, gaps,
|
||||
// confidence_score, needs_human_review, human_review_reasons,
|
||||
// post_resolution_analysis
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Stage 5 — Persistence
|
||||
|
||||
1. Compute next `analysis_version` for this `ticket_number`.
|
||||
2. Insert into `analyzer_analyses`.
|
||||
3. Sum tokens, compute estimated cost.
|
||||
4. Update `analyzer_jobs` row.
|
||||
|
||||
---
|
||||
|
||||
## Frontend changes
|
||||
|
||||
### Routes (add to existing wulf-pulse router)
|
||||
|
||||
- `/analyzer/ticket/:ticketNumber` — ticket detail with Analyze button + history of prior analyses
|
||||
- `/analyzer/analysis/:id` — full analysis view, printable, with timeline visualization
|
||||
- `/analyzer/queue` — needs-human-review queue
|
||||
|
||||
### Analyze button behavior
|
||||
|
||||
1. Click → POST to analyze endpoint.
|
||||
2. If `existingAnalysisId` returned and content unchanged, navigate straight to it.
|
||||
3. Otherwise show progress UI polling job status every 2s with stage labels: "Fetching..." → "Triaging..." → "Searching IT Glue..." → "Analyzing..." → "Deep review..." → "Done".
|
||||
4. On completion, navigate to the analysis view.
|
||||
|
||||
### Analysis view layout
|
||||
|
||||
Use shadcn/ui components. Sections in order:
|
||||
|
||||
1. **Header** — ticket number + title + Autotask deep link, model tier badges (Haiku/Sonnet/Opus pills), confidence score, total cost, "Share" button
|
||||
2. **Summary** — paragraph
|
||||
3. **Next Step** — highlighted card with rationale collapsed by default
|
||||
4. **Timeline** — vertical timeline with the three visibility markers (🟢 / 🔒 / 🔄), each event clickable to expand full notes
|
||||
5. **What Was Done** — bulleted list
|
||||
6. **What Should Have Been Done** — bulleted list, side-by-side with #5 on wide screens
|
||||
7. **Gaps** — cards colored by severity, with "Evidence:" linking back to timeline events
|
||||
8. **Post-Resolution Analysis** — only if present
|
||||
9. **Human Review Flags** — only if `needs_human_review = true`
|
||||
10. **IT Glue References** — list of docs with external links
|
||||
|
||||
### Re-analyze indicator
|
||||
|
||||
On the ticket view, if cached `content_hash` differs from the latest analysis's `content_hash_at_analysis`, show a banner: "New activity since last analysis · Re-analyze".
|
||||
|
||||
### Share modal
|
||||
|
||||
- Recipient email field (autocomplete from existing wulf-pulse user list if available)
|
||||
- Validate domain against `ALLOWED_SHARE_DOMAINS` env var
|
||||
- Optional note
|
||||
- Sends via M365 Graph using existing wulf-pulse mail integration if one exists; otherwise use a new module under `apps/api/src/services/mail/` and ask before adding new credentials
|
||||
|
||||
---
|
||||
|
||||
## Environment variables (additions)
|
||||
|
||||
```
|
||||
# IT Glue
|
||||
ITGLUE_API_KEY=
|
||||
ITGLUE_API_BASE=https://api.itglue.com
|
||||
|
||||
# Anthropic
|
||||
ANTHROPIC_API_KEY=
|
||||
|
||||
# Sharing
|
||||
ALLOWED_SHARE_DOMAINS=wulfconsulting.com
|
||||
```
|
||||
|
||||
Add to `~/projects_env/wulf-pulse.env`. Do not commit example values.
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
Required tests (use whatever wulf-pulse already uses for testing):
|
||||
|
||||
1. **IT Glue redaction** — nested objects, arrays of objects, mixed-case field names. Must redact and never allow a password to reach the database.
|
||||
2. **Note pre-processor** — given a fixture with all five note types, asserts workflow firings and notification emails are filtered, and remaining events are correctly tagged.
|
||||
3. **Pipeline against the T20260424.0045 fixture** — asserts all four required findings appear in the analysis output. This is a **regression test for the prompts**, not just code.
|
||||
4. **Schema validation** — every LLM response is parsed through Zod. Test with deliberately malformed responses to confirm graceful retry then failure.
|
||||
5. **Idempotency** — same content_hash with `force=false` returns the existing analysis without invoking the LLM.
|
||||
|
||||
---
|
||||
|
||||
## Critical correctness notes
|
||||
|
||||
- **Never store IT Glue secrets/passwords in any database row, log line, or LLM context.** Redact before everything.
|
||||
- **Never log full IT Glue document content.** Only doc IDs and names.
|
||||
- **Validate every LLM JSON response with Zod.** On parse failure, retry once with the prior response and the parse error. After two failures, mark the job failed and store the raw response in `error_message`.
|
||||
- **Token budget guard:** if any single model call would exceed 100k input tokens, truncate oldest internal-only events first while preserving all customer-facing communications, and add a marker. Log a warning.
|
||||
- **Cost circuit breaker:** if estimated total cost would exceed $2.00 before the Opus call, skip Opus, set `needs_human_review = true`, and add reason "cost ceiling reached".
|
||||
- **Idempotency:** the analyze endpoint must be idempotent on `(ticket_number, content_hash)` when `force=false`.
|
||||
- **Verify model IDs and pricing against `https://docs.claude.com` before finalizing constants.** Models and rates change.
|
||||
- **Ticket Notes from `wulfconsulting.com` addresses are internal communications, not customer communications.** Tag them by domain, not by author.
|
||||
|
||||
---
|
||||
|
||||
## Delivery order
|
||||
|
||||
Build and ship in this order, asking me to review between each phase:
|
||||
|
||||
1. Database migrations + Zod schemas in `packages/shared` (or wulf-pulse equivalent).
|
||||
2. Note pre-processor with unit tests against the T20260424.0045 fixture (PDF → JSON transcription is the first deliverable).
|
||||
3. IT Glue client + redaction (security-critical, must land before LLM integration).
|
||||
4. Anthropic SDK setup + per-stage prompt files.
|
||||
5. Full pipeline + job worker.
|
||||
6. API routes.
|
||||
7. Frontend pages.
|
||||
8. Share-via-email integration.
|
||||
9. README updates with operator runbook (how to monitor cost, how to add IT Glue org aliases, how to triage failed analyses).
|
||||
|
||||
For phase 1 specifically: confirm the data-access strategy (read from Pulse Postgres / live Autotask REST / hybrid) before writing the migration, since the schema for `tickets_cache` may or may not be needed depending on which path we take.
|
||||
Loading…
Add table
Add a link
Reference in a new issue