9 KiB
Veeam Backup Failure Diagnostic Pipeline — Progress & Pickup Guide
Last updated: 2026-02-20 5:45pm EST
Status: Code complete, deployed. Pending: B2 credentials + RMM component re-upload.
What Was Built
A 14-step automated diagnostic pipeline that triggers when a Veeam backup failure alert arrives from Datto RMM. Instead of creating a generic "backup failed" ticket, it:
- Enriches from multiple sources (RMM device data, Autotask company, VSPC backup status, local DB trends)
- Runs a diagnostic PowerShell script on the affected device via RMM quick job
- Uploads diagnostic results to Backblaze B2 (avoids RMM StdOut size limits)
- Downloads the results via presigned S3 URL
- Feeds everything to AI for root cause analysis
- Creates a rich Autotask ticket with all findings
- Notifies the team via Teams
Pipeline Steps (14 total, pipeline_id=2 in DB)
1. transform — Extract 11 alert fields from webhook payload
2. enrich_device — Lookup device details from datto_rmm_devices
3. enrich_company — Lookup Autotask company from site name
4. enrich_vspc — Query VSPC tables for backup jobs, workloads, alarms
5. db_query — Backup failure trend: job status counts over 7 days
6. db_query — Recent RMM alerts for this device (pattern detection)
7. rmm_quick_job — Run Veeam diagnostic PowerShell script on device
8. delay — Wait 60s for script execution
9. rmm_get_job_results — Retrieve StdOut (contains B2 object key)
10. fetch_b2_result — Download full diagnostic JSON from B2 via presigned URL
11. ai_analyze — AI root cause analysis using ALL collected data
12. create_ticket — Create rich Autotask ticket with diagnostics + AI analysis
13. create_note — Add AI analysis as internal ticket note
14. notify — Teams Adaptive Card with summary + ticket link
Files Created/Modified
New Step Executors
lib/services/pipeline-steps/enrich-vspc.ts— Queries local VSPC tables (agent jobs, server jobs, workloads, alarms) by device name or org UIDlib/services/pipeline-steps/db-query.ts— Parameterized read-only SQL against local Postgres (blocks mutations)lib/services/pipeline-steps/fetch-b2-result.ts— Generates S3v4 presigned URL, downloads JSON from B2, parses into context
Registration
lib/services/pipeline-steps/index.ts— Added imports forenrich-vspc,db-query,fetch-b2-result
UI Updates
app/admin/workflow/pipelines/[id]/page.tsx— Addedenrich_vspc,db_query,fetch_b2_resulttoSTEP_TYPESarray +Datacategory colorcomponents/admin/pipeline/StepConfigEditor.tsx— Added:STEP_OUTPUT_FIELDSentries for all 3 new typesEnrichVspcEditorcomponent (lookup_by dropdown + source_field)DbQueryEditorcomponent (SQL textarea + params + output_key + single_row toggle)FetchB2ResultEditorcomponent (object_key with variable picker + output_key + bucket)getContextFields()updated for dynamic output keys fromdb_queryandfetch_b2_result
PowerShell Diagnostic Script
scripts/rmm-diagnostics/veeam-backup-diagnostic.ps1— 7 diagnostic checks + B2 upload:- Veeam services (11 service names)
- Backup jobs via VBR PowerShell snap-in
- Disk space (all fixed drives, flags >90%)
- Windows Event Logs (Veeam errors, last 48h)
- Stuck processes (>48h)
- Network connectivity (SQL server, ports 9392/9419/6180)
- B2 upload via S3v4 signed PUT, outputs object key to StdOut
- Falls back to inline JSON if B2 creds missing
- PS 5.1 compatible (fixed
-AsUTC→[DateTime]::UtcNow) - Wrapped in try/catch for error visibility
Infrastructure
docker-compose.yml— AddedB2_KEY_ID,B2_APP_KEY,B2_BUCKET,B2_REGION,B2_ENDPOINTenv vars
Migrations
migrations/034_seed_veeam_backup_failure_pipeline.sql— Seeds the 13-step pipeline (already applied)migrations/035_update_veeam_pipeline_b2_fetch.sql— Insertsfetch_b2_resultat step 10, shifts 10-13→11-14, updates{{context.job_results}}→{{context.diagnostic_results}}in AI and ticket steps (already applied)
Documentation
docs/webhook-pipeline-engine.md— Updated withenrich_vspc,db_query,fetch_b2_resultstep docs, full Veeam pipeline example, activation steps, file structure
What's Left To Do
1. Add B2 credentials to .env.local on the Pulse server
# Add to /opt/stacks/pulse/.env.local
B2_KEY_ID=<your-backblaze-b2-key-id>
B2_APP_KEY=<your-backblaze-b2-application-key>
Then restart the app:
cd /opt/stacks/pulse && docker compose up -d app
2. Set B2 credentials in Datto RMM
The PowerShell script reads B2 creds from Datto RMM component/site variables. Set these as site-level variables in Datto RMM (so all devices inherit them):
| Variable Name | Value |
|---|---|
usrB2KeyId |
Your B2 key ID |
usrB2AppKey |
Your B2 application key |
Or set them as component variables on the VeeamDiagnostic component itself.
3. Re-upload the PowerShell script to Datto RMM
The script at scripts/rmm-diagnostics/veeam-backup-diagnostic.ps1 has been updated with:
- B2 upload capability
- PS 5.1 compatibility fixes
Write-Hostfor StdOut capture- try/catch error handling
Upload it to replace the existing VeeamDiagnostic component in Datto RMM.
4. Verify the component UID
The pipeline step 7 currently has component_uid: "30941089-23c4-4f3e-be23-4470f1e33650". If you re-upload as a new component (rather than editing the existing one), update this UID:
-- Run against pulse_autotask DB if needed
UPDATE pipeline_steps
SET config = jsonb_set(config, '{component_uid}', '"NEW_COMPONENT_UID_HERE"')
WHERE pipeline_id = 2 AND step_type = 'rmm_quick_job';
Or edit it in the UI at /admin/workflow/pipelines/2.
5. Set up notification channel
Create a Teams webhook notification channel at /admin/workflow/channels, then update step 14's channel_id:
UPDATE pipeline_steps
SET config = jsonb_set(config, '{channel_id}', 'YOUR_CHANNEL_ID')
WHERE pipeline_id = 2 AND step_type = 'notify';
6. Activate the pipeline
The pipeline is seeded as inactive. Toggle it active at /admin/workflow/pipelines or:
UPDATE webhook_pipelines SET is_active = true WHERE id = 2;
Testing Checklist
- B2 creds in
.env.local→ restart app - B2 creds in Datto RMM site/component variables
- Updated PS script uploaded to Datto RMM
- Run diagnostic script manually on a test device → verify B2 upload + object key in StdOut
- Verify object appears in
wulf-auditsbucket underdiagnostics/{hostname}/{date}/ - Test the pipeline with a sample payload via
/admin/workflow/pipelines/2→ Test tab - Activate pipeline and trigger with a real Veeam alert
Architecture: How B2 Upload Works
Device (PS script) Backblaze B2 Pulse Server
│ │ │
│ 1. Collect diagnostics │ │
│ 2. S3v4 signed PUT ──────────────►│ diagnostics/HOST/DATE/TS.json│
│ 3. Write-Host "object key" ───────┼──────────────────────────────►│
│ │ │
│ │ 4. Presigned GET URL │
│ │◄──────────────────────────────│
│ │ 5. Download JSON ───────────►│
│ │ │
│ │ 6. Parse into context │
│ │ 7. Feed to AI │
│ │ 8. Create ticket │
The PS script uploads the full diagnostic JSON (~5-50KB) to B2 and only outputs the object key (~60 chars) via StdOut. This avoids RMM StdOut size limits and encoding issues. The fetch_b2_result step on the Pulse side generates a presigned URL and downloads the full payload.
Fallback: If B2 credentials are missing on the device, the script falls back to writing the full JSON to StdOut (original behavior).
Key Environment Variables
| Variable | Where | Purpose |
|---|---|---|
B2_KEY_ID |
.env.local + Datto RMM (usrB2KeyId) |
B2 auth for both upload and download |
B2_APP_KEY |
.env.local + Datto RMM (usrB2AppKey) |
B2 auth for both upload and download |
B2_BUCKET |
.env.local (default: wulf-audits) |
Bucket name |
B2_REGION |
.env.local (default: us-west-002) |
B2 region |
B2_ENDPOINT |
.env.local (default: s3.us-west-002.backblazeb2.com) |
S3-compatible endpoint |