201 lines
9 KiB
Markdown
201 lines
9 KiB
Markdown
|
# Veeam Backup Failure Diagnostic Pipeline — Progress & Pickup Guide
|
||
|
|
|
||
|
|
**Last updated:** 2026-02-20 5:45pm EST
|
||
|
|
**Status:** Code complete, deployed. Pending: B2 credentials + RMM component re-upload.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## What Was Built
|
||
|
|
|
||
|
|
A 14-step automated diagnostic pipeline that triggers when a Veeam backup failure alert arrives from Datto RMM. Instead of creating a generic "backup failed" ticket, it:
|
||
|
|
|
||
|
|
1. Enriches from multiple sources (RMM device data, Autotask company, VSPC backup status, local DB trends)
|
||
|
|
2. Runs a diagnostic PowerShell script on the affected device via RMM quick job
|
||
|
|
3. Uploads diagnostic results to Backblaze B2 (avoids RMM StdOut size limits)
|
||
|
|
4. Downloads the results via presigned S3 URL
|
||
|
|
5. Feeds everything to AI for root cause analysis
|
||
|
|
6. Creates a rich Autotask ticket with all findings
|
||
|
|
7. Notifies the team via Teams
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Pipeline Steps (14 total, pipeline_id=2 in DB)
|
||
|
|
|
||
|
|
```
|
||
|
|
1. transform — Extract 11 alert fields from webhook payload
|
||
|
|
2. enrich_device — Lookup device details from datto_rmm_devices
|
||
|
|
3. enrich_company — Lookup Autotask company from site name
|
||
|
|
4. enrich_vspc — Query VSPC tables for backup jobs, workloads, alarms
|
||
|
|
5. db_query — Backup failure trend: job status counts over 7 days
|
||
|
|
6. db_query — Recent RMM alerts for this device (pattern detection)
|
||
|
|
7. rmm_quick_job — Run Veeam diagnostic PowerShell script on device
|
||
|
|
8. delay — Wait 60s for script execution
|
||
|
|
9. rmm_get_job_results — Retrieve StdOut (contains B2 object key)
|
||
|
|
10. fetch_b2_result — Download full diagnostic JSON from B2 via presigned URL
|
||
|
|
11. ai_analyze — AI root cause analysis using ALL collected data
|
||
|
|
12. create_ticket — Create rich Autotask ticket with diagnostics + AI analysis
|
||
|
|
13. create_note — Add AI analysis as internal ticket note
|
||
|
|
14. notify — Teams Adaptive Card with summary + ticket link
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Files Created/Modified
|
||
|
|
|
||
|
|
### New Step Executors
|
||
|
|
- `lib/services/pipeline-steps/enrich-vspc.ts` — Queries local VSPC tables (agent jobs, server jobs, workloads, alarms) by device name or org UID
|
||
|
|
- `lib/services/pipeline-steps/db-query.ts` — Parameterized read-only SQL against local Postgres (blocks mutations)
|
||
|
|
- `lib/services/pipeline-steps/fetch-b2-result.ts` — Generates S3v4 presigned URL, downloads JSON from B2, parses into context
|
||
|
|
|
||
|
|
### Registration
|
||
|
|
- `lib/services/pipeline-steps/index.ts` — Added imports for `enrich-vspc`, `db-query`, `fetch-b2-result`
|
||
|
|
|
||
|
|
### UI Updates
|
||
|
|
- `app/admin/workflow/pipelines/[id]/page.tsx` — Added `enrich_vspc`, `db_query`, `fetch_b2_result` to `STEP_TYPES` array + `Data` category color
|
||
|
|
- `components/admin/pipeline/StepConfigEditor.tsx` — Added:
|
||
|
|
- `STEP_OUTPUT_FIELDS` entries for all 3 new types
|
||
|
|
- `EnrichVspcEditor` component (lookup_by dropdown + source_field)
|
||
|
|
- `DbQueryEditor` component (SQL textarea + params + output_key + single_row toggle)
|
||
|
|
- `FetchB2ResultEditor` component (object_key with variable picker + output_key + bucket)
|
||
|
|
- `getContextFields()` updated for dynamic output keys from `db_query` and `fetch_b2_result`
|
||
|
|
|
||
|
|
### PowerShell Diagnostic Script
|
||
|
|
- `scripts/rmm-diagnostics/veeam-backup-diagnostic.ps1` — 7 diagnostic checks + B2 upload:
|
||
|
|
- Veeam services (11 service names)
|
||
|
|
- Backup jobs via VBR PowerShell snap-in
|
||
|
|
- Disk space (all fixed drives, flags >90%)
|
||
|
|
- Windows Event Logs (Veeam errors, last 48h)
|
||
|
|
- Stuck processes (>48h)
|
||
|
|
- Network connectivity (SQL server, ports 9392/9419/6180)
|
||
|
|
- **B2 upload** via S3v4 signed PUT, outputs object key to StdOut
|
||
|
|
- Falls back to inline JSON if B2 creds missing
|
||
|
|
- PS 5.1 compatible (fixed `-AsUTC` → `[DateTime]::UtcNow`)
|
||
|
|
- Wrapped in try/catch for error visibility
|
||
|
|
|
||
|
|
### Infrastructure
|
||
|
|
- `docker-compose.yml` — Added `B2_KEY_ID`, `B2_APP_KEY`, `B2_BUCKET`, `B2_REGION`, `B2_ENDPOINT` env vars
|
||
|
|
|
||
|
|
### Migrations
|
||
|
|
- `migrations/034_seed_veeam_backup_failure_pipeline.sql` — Seeds the 13-step pipeline (already applied)
|
||
|
|
- `migrations/035_update_veeam_pipeline_b2_fetch.sql` — Inserts `fetch_b2_result` at step 10, shifts 10-13→11-14, updates `{{context.job_results}}` → `{{context.diagnostic_results}}` in AI and ticket steps (already applied)
|
||
|
|
|
||
|
|
### Documentation
|
||
|
|
- `docs/webhook-pipeline-engine.md` — Updated with `enrich_vspc`, `db_query`, `fetch_b2_result` step docs, full Veeam pipeline example, activation steps, file structure
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## What's Left To Do
|
||
|
|
|
||
|
|
### 1. Add B2 credentials to `.env.local` on the Pulse server
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Add to /opt/stacks/pulse/.env.local
|
||
|
|
B2_KEY_ID=<your-backblaze-b2-key-id>
|
||
|
|
B2_APP_KEY=<your-backblaze-b2-application-key>
|
||
|
|
```
|
||
|
|
|
||
|
|
Then restart the app:
|
||
|
|
```bash
|
||
|
|
cd /opt/stacks/pulse && docker compose up -d app
|
||
|
|
```
|
||
|
|
|
||
|
|
### 2. Set B2 credentials in Datto RMM
|
||
|
|
|
||
|
|
The PowerShell script reads B2 creds from Datto RMM component/site variables. Set these as **site-level variables** in Datto RMM (so all devices inherit them):
|
||
|
|
|
||
|
|
| Variable Name | Value |
|
||
|
|
|---------------|-------|
|
||
|
|
| `usrB2KeyId` | Your B2 key ID |
|
||
|
|
| `usrB2AppKey` | Your B2 application key |
|
||
|
|
|
||
|
|
Or set them as component variables on the VeeamDiagnostic component itself.
|
||
|
|
|
||
|
|
### 3. Re-upload the PowerShell script to Datto RMM
|
||
|
|
|
||
|
|
The script at `scripts/rmm-diagnostics/veeam-backup-diagnostic.ps1` has been updated with:
|
||
|
|
- B2 upload capability
|
||
|
|
- PS 5.1 compatibility fixes
|
||
|
|
- `Write-Host` for StdOut capture
|
||
|
|
- try/catch error handling
|
||
|
|
|
||
|
|
Upload it to replace the existing `VeeamDiagnostic` component in Datto RMM.
|
||
|
|
|
||
|
|
### 4. Verify the component UID
|
||
|
|
|
||
|
|
The pipeline step 7 currently has `component_uid: "30941089-23c4-4f3e-be23-4470f1e33650"`. If you re-upload as a new component (rather than editing the existing one), update this UID:
|
||
|
|
|
||
|
|
```sql
|
||
|
|
-- Run against pulse_autotask DB if needed
|
||
|
|
UPDATE pipeline_steps
|
||
|
|
SET config = jsonb_set(config, '{component_uid}', '"NEW_COMPONENT_UID_HERE"')
|
||
|
|
WHERE pipeline_id = 2 AND step_type = 'rmm_quick_job';
|
||
|
|
```
|
||
|
|
|
||
|
|
Or edit it in the UI at `/admin/workflow/pipelines/2`.
|
||
|
|
|
||
|
|
### 5. Set up notification channel
|
||
|
|
|
||
|
|
Create a Teams webhook notification channel at `/admin/workflow/channels`, then update step 14's `channel_id`:
|
||
|
|
|
||
|
|
```sql
|
||
|
|
UPDATE pipeline_steps
|
||
|
|
SET config = jsonb_set(config, '{channel_id}', 'YOUR_CHANNEL_ID')
|
||
|
|
WHERE pipeline_id = 2 AND step_type = 'notify';
|
||
|
|
```
|
||
|
|
|
||
|
|
### 6. Activate the pipeline
|
||
|
|
|
||
|
|
The pipeline is seeded as **inactive**. Toggle it active at `/admin/workflow/pipelines` or:
|
||
|
|
|
||
|
|
```sql
|
||
|
|
UPDATE webhook_pipelines SET is_active = true WHERE id = 2;
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Testing Checklist
|
||
|
|
|
||
|
|
- [ ] B2 creds in `.env.local` → restart app
|
||
|
|
- [ ] B2 creds in Datto RMM site/component variables
|
||
|
|
- [ ] Updated PS script uploaded to Datto RMM
|
||
|
|
- [ ] Run diagnostic script manually on a test device → verify B2 upload + object key in StdOut
|
||
|
|
- [ ] Verify object appears in `wulf-audits` bucket under `diagnostics/{hostname}/{date}/`
|
||
|
|
- [ ] Test the pipeline with a sample payload via `/admin/workflow/pipelines/2` → Test tab
|
||
|
|
- [ ] Activate pipeline and trigger with a real Veeam alert
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Architecture: How B2 Upload Works
|
||
|
|
|
||
|
|
```
|
||
|
|
Device (PS script) Backblaze B2 Pulse Server
|
||
|
|
│ │ │
|
||
|
|
│ 1. Collect diagnostics │ │
|
||
|
|
│ 2. S3v4 signed PUT ──────────────►│ diagnostics/HOST/DATE/TS.json│
|
||
|
|
│ 3. Write-Host "object key" ───────┼──────────────────────────────►│
|
||
|
|
│ │ │
|
||
|
|
│ │ 4. Presigned GET URL │
|
||
|
|
│ │◄──────────────────────────────│
|
||
|
|
│ │ 5. Download JSON ───────────►│
|
||
|
|
│ │ │
|
||
|
|
│ │ 6. Parse into context │
|
||
|
|
│ │ 7. Feed to AI │
|
||
|
|
│ │ 8. Create ticket │
|
||
|
|
```
|
||
|
|
|
||
|
|
The PS script uploads the full diagnostic JSON (~5-50KB) to B2 and only outputs the object key (~60 chars) via StdOut. This avoids RMM StdOut size limits and encoding issues. The `fetch_b2_result` step on the Pulse side generates a presigned URL and downloads the full payload.
|
||
|
|
|
||
|
|
**Fallback:** If B2 credentials are missing on the device, the script falls back to writing the full JSON to StdOut (original behavior).
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Key Environment Variables
|
||
|
|
|
||
|
|
| Variable | Where | Purpose |
|
||
|
|
|----------|-------|---------|
|
||
|
|
| `B2_KEY_ID` | `.env.local` + Datto RMM (`usrB2KeyId`) | B2 auth for both upload and download |
|
||
|
|
| `B2_APP_KEY` | `.env.local` + Datto RMM (`usrB2AppKey`) | B2 auth for both upload and download |
|
||
|
|
| `B2_BUCKET` | `.env.local` (default: `wulf-audits`) | Bucket name |
|
||
|
|
| `B2_REGION` | `.env.local` (default: `us-west-002`) | B2 region |
|
||
|
|
| `B2_ENDPOINT` | `.env.local` (default: `s3.us-west-002.backblazeb2.com`) | S3-compatible endpoint |
|