# Veeam Backup Failure Diagnostic Pipeline — Progress & Pickup Guide

**Last updated:** 2026-02-20 5:45pm EST  
**Status:** Code complete, deployed. Pending: B2 credentials + RMM component re-upload.

---

## What Was Built

A 14-step automated diagnostic pipeline that triggers when a Veeam backup failure alert arrives from Datto RMM. Instead of creating a generic "backup failed" ticket, it:

1. Enriches from multiple sources (RMM device data, Autotask company, VSPC backup status, local DB trends)
2. Runs a diagnostic PowerShell script on the affected device via RMM quick job
3. Uploads diagnostic results to Backblaze B2 (avoids RMM StdOut size limits)
4. Downloads the results via presigned S3 URL
5. Feeds everything to AI for root cause analysis
6. Creates a rich Autotask ticket with all findings
7. Notifies the team via Teams

---

## Pipeline Steps (14 total, pipeline_id=2 in DB)

```
 1. transform           — Extract 11 alert fields from webhook payload
 2. enrich_device       — Lookup device details from datto_rmm_devices
 3. enrich_company      — Lookup Autotask company from site name
 4. enrich_vspc         — Query VSPC tables for backup jobs, workloads, alarms
 5. db_query            — Backup failure trend: job status counts over 7 days
 6. db_query            — Recent RMM alerts for this device (pattern detection)
 7. rmm_quick_job       — Run Veeam diagnostic PowerShell script on device
 8. delay               — Wait 60s for script execution
 9. rmm_get_job_results — Retrieve StdOut (contains B2 object key)
10. fetch_b2_result     — Download full diagnostic JSON from B2 via presigned URL
11. ai_analyze          — AI root cause analysis using ALL collected data
12. create_ticket       — Create rich Autotask ticket with diagnostics + AI analysis
13. create_note         — Add AI analysis as internal ticket note
14. notify              — Teams Adaptive Card with summary + ticket link
```

---

## Files Created/Modified

### New Step Executors
- `lib/services/pipeline-steps/enrich-vspc.ts` — Queries local VSPC tables (agent jobs, server jobs, workloads, alarms) by device name or org UID
- `lib/services/pipeline-steps/db-query.ts` — Parameterized read-only SQL against local Postgres (blocks mutations)
- `lib/services/pipeline-steps/fetch-b2-result.ts` — Generates S3v4 presigned URL, downloads JSON from B2, parses into context

### Registration
- `lib/services/pipeline-steps/index.ts` — Added imports for `enrich-vspc`, `db-query`, `fetch-b2-result`

### UI Updates
- `app/admin/workflow/pipelines/[id]/page.tsx` — Added `enrich_vspc`, `db_query`, `fetch_b2_result` to `STEP_TYPES` array + `Data` category color
- `components/admin/pipeline/StepConfigEditor.tsx` — Added:
  - `STEP_OUTPUT_FIELDS` entries for all 3 new types
  - `EnrichVspcEditor` component (lookup_by dropdown + source_field)
  - `DbQueryEditor` component (SQL textarea + params + output_key + single_row toggle)
  - `FetchB2ResultEditor` component (object_key with variable picker + output_key + bucket)
  - `getContextFields()` updated for dynamic output keys from `db_query` and `fetch_b2_result`

### PowerShell Diagnostic Script
- `scripts/rmm-diagnostics/veeam-backup-diagnostic.ps1` — 7 diagnostic checks + B2 upload:
  - Veeam services (11 service names)
  - Backup jobs via VBR PowerShell snap-in
  - Disk space (all fixed drives, flags >90%)
  - Windows Event Logs (Veeam errors, last 48h)
  - Stuck processes (>48h)
  - Network connectivity (SQL server, ports 9392/9419/6180)
  - **B2 upload** via S3v4 signed PUT, outputs object key to StdOut
  - Falls back to inline JSON if B2 creds missing
  - PS 5.1 compatible (fixed `-AsUTC` → `[DateTime]::UtcNow`)
  - Wrapped in try/catch for error visibility

### Infrastructure
- `docker-compose.yml` — Added `B2_KEY_ID`, `B2_APP_KEY`, `B2_BUCKET`, `B2_REGION`, `B2_ENDPOINT` env vars

### Migrations
- `migrations/034_seed_veeam_backup_failure_pipeline.sql` — Seeds the 13-step pipeline (already applied)
- `migrations/035_update_veeam_pipeline_b2_fetch.sql` — Inserts `fetch_b2_result` at step 10, shifts 10-13→11-14, updates `{{context.job_results}}` → `{{context.diagnostic_results}}` in AI and ticket steps (already applied)

### Documentation
- `docs/webhook-pipeline-engine.md` — Updated with `enrich_vspc`, `db_query`, `fetch_b2_result` step docs, full Veeam pipeline example, activation steps, file structure

---

## What's Left To Do

### 1. Add B2 credentials to `.env.local` on the Pulse server

```bash
# Add to /opt/stacks/pulse/.env.local
B2_KEY_ID=<your-backblaze-b2-key-id>
B2_APP_KEY=<your-backblaze-b2-application-key>
```

Then restart the app:
```bash
cd /opt/stacks/pulse && docker compose up -d app
```

### 2. Set B2 credentials in Datto RMM

The PowerShell script reads B2 creds from Datto RMM component/site variables. Set these as **site-level variables** in Datto RMM (so all devices inherit them):

| Variable Name | Value |
|---------------|-------|
| `usrB2KeyId`  | Your B2 key ID |
| `usrB2AppKey` | Your B2 application key |

Or set them as component variables on the VeeamDiagnostic component itself.

### 3. Re-upload the PowerShell script to Datto RMM

The script at `scripts/rmm-diagnostics/veeam-backup-diagnostic.ps1` has been updated with:
- B2 upload capability
- PS 5.1 compatibility fixes
- `Write-Host` for StdOut capture
- try/catch error handling

Upload it to replace the existing `VeeamDiagnostic` component in Datto RMM.

### 4. Verify the component UID

The pipeline step 7 currently has `component_uid: "30941089-23c4-4f3e-be23-4470f1e33650"`. If you re-upload as a new component (rather than editing the existing one), update this UID:

```sql
-- Run against pulse_autotask DB if needed
UPDATE pipeline_steps
SET config = jsonb_set(config, '{component_uid}', '"NEW_COMPONENT_UID_HERE"')
WHERE pipeline_id = 2 AND step_type = 'rmm_quick_job';
```

Or edit it in the UI at `/admin/workflow/pipelines/2`.

### 5. Set up notification channel

Create a Teams webhook notification channel at `/admin/workflow/channels`, then update step 14's `channel_id`:

```sql
UPDATE pipeline_steps
SET config = jsonb_set(config, '{channel_id}', 'YOUR_CHANNEL_ID')
WHERE pipeline_id = 2 AND step_type = 'notify';
```

### 6. Activate the pipeline

The pipeline is seeded as **inactive**. Toggle it active at `/admin/workflow/pipelines` or:

```sql
UPDATE webhook_pipelines SET is_active = true WHERE id = 2;
```

---

## Testing Checklist

- [ ] B2 creds in `.env.local` → restart app
- [ ] B2 creds in Datto RMM site/component variables
- [ ] Updated PS script uploaded to Datto RMM
- [ ] Run diagnostic script manually on a test device → verify B2 upload + object key in StdOut
- [ ] Verify object appears in `wulf-audits` bucket under `diagnostics/{hostname}/{date}/`
- [ ] Test the pipeline with a sample payload via `/admin/workflow/pipelines/2` → Test tab
- [ ] Activate pipeline and trigger with a real Veeam alert

---

## Architecture: How B2 Upload Works

```
Device (PS script)                    Backblaze B2                    Pulse Server
     │                                    │                               │
     │  1. Collect diagnostics            │                               │
     │  2. S3v4 signed PUT ──────────────►│  diagnostics/HOST/DATE/TS.json│
     │  3. Write-Host "object key" ───────┼──────────────────────────────►│
     │                                    │                               │
     │                                    │  4. Presigned GET URL         │
     │                                    │◄──────────────────────────────│
     │                                    │  5. Download JSON ───────────►│
     │                                    │                               │
     │                                    │         6. Parse into context │
     │                                    │         7. Feed to AI         │
     │                                    │         8. Create ticket      │
```

The PS script uploads the full diagnostic JSON (~5-50KB) to B2 and only outputs the object key (~60 chars) via StdOut. This avoids RMM StdOut size limits and encoding issues. The `fetch_b2_result` step on the Pulse side generates a presigned URL and downloads the full payload.

**Fallback:** If B2 credentials are missing on the device, the script falls back to writing the full JSON to StdOut (original behavior).

---

## Key Environment Variables

| Variable | Where | Purpose |
|----------|-------|---------|
| `B2_KEY_ID` | `.env.local` + Datto RMM (`usrB2KeyId`) | B2 auth for both upload and download |
| `B2_APP_KEY` | `.env.local` + Datto RMM (`usrB2AppKey`) | B2 auth for both upload and download |
| `B2_BUCKET` | `.env.local` (default: `wulf-audits`) | Bucket name |
| `B2_REGION` | `.env.local` (default: `us-west-002`) | B2 region |
| `B2_ENDPOINT` | `.env.local` (default: `s3.us-west-002.backblazeb2.com`) | S3-compatible endpoint |
