wulf-pulse/docs/veeam-backup-alerting-recommendation.md

Veeam Backup Alerting — Current State & RPO-Based Recommendation

Date: April 14, 2026 Author: Pulse / Cascade Status: Recommendation — not yet implemented

---

1. Problem: The Current Alerting Pipeline is Noisy

How It Works Today

Every Veeam backup alert that reaches Autotask passes through Datto RMM — Veeam does not create Autotask tickets directly. There are 6 Datto RMM monitors that watch for Veeam conditions (via Windows Event Log or script checks on each WNP/endpoint) and auto-create tickets when a condition is true:

RMM Monitor ID	Description	Tickets (Last 30 Days)
422325	Veeam Agent Backup Stalled	638
770091	Backup Copy Job Failed	194
369602	Veeam Agent Backup Failed	103
369604	Veeam Backup Job Missing or Stalled	61
392069	Veeam Agent Job Finished with Failed	58
849483	Veeam Server Agent Backup Job Stalled	38

Total: ~1,100 backup-related tickets per month from these 6 monitors alone.

Monthly Ticket Volume (Last 6 Months)

Month	Backup Tickets
October 2025	461
November 2025	1,033
December 2025	1,008
January 2026	1,583
February 2026	728
March 2026	946

The Core Defects

1. No deduplication — one new ticket per check cycle. A workstation that misses its backup today gets a new Autotask ticket every time the RMM monitor runs. DT061 generated 41 individual tickets in 30 days for a single laptop. In the last 30 days, 23 devices each triggered 5+ alerts, producing 387 redundant tickets from a single monitor.

2. No schedule awareness. The monitors check "has Veeam run in X days?" without knowing the job's schedule. A laptop that is legitimately offline over a weekend gets stalled alerts on Saturday and Sunday even though no backup was missed relative to its RPO.

3. No auto-resolution. When the underlying backup issue is fixed and the job succeeds, the open Autotask tickets are not automatically closed. Resolution requires manual action.

4. No escalation logic. A job that has been failing for 3 days looks identical in Autotask to one that missed a single run — both generate the same priority ticket.

5. No failure context. The RMM alerts contain only the device name and a generic "stalled" or "failed" label. The root cause (VBM desync, Wasabi DNS failure, VSS error, license expiry) is not surfaced in the ticket.

---

2. Recommendation: RPO-Based Alerting via Pulse

What Already Exists

Pulse already contains a fully-built RPO monitoring service at lib/services/veeam-rpo-service.ts and an API endpoint at POST /api/veeam/rpo-check. The service:

• Reads live job data from the Veeam VSPC sync in PostgreSQL (veeam_backup_agent_jobs)
• Computes whether each job has breached its Recovery Point Objective (RPO) — i.e., the acceptable maximum gap since the last successful backup — based on the job's configured schedule
• Creates one Autotask ticket per breached job (deduped via the veeam_rpo_tickets tracking table)
• Auto-resolves that ticket the moment a successful backup run is detected
• Escalates the ticket's priority as the RPO violation ages
• Categorizes the failure reason from the Veeam failure message

The veeam_rpo_tickets table exists and is ready. It currently has 0 rows because the service has never been run.

RPO Threshold Logic

Schedule Type	RPO Window	Alert After	Escalate to High	Escalate to Critical
Daily	24h	+4h grace = 28h	+48h	+7 days
Weekly	168h	+4h grace = 172h	+48h	+7 days
Continuous	1h	+1h grace = 2h	+4h	+12h

The grace period (4h for daily jobs) accounts for jobs that run slightly late due to the workstation being offline, slow networks, or queued jobs on the WNP — preventing false positives for on-time jobs with minor delays.

Ticket Behavior

Event	Current (RMM)	Proposed (RPO)
Job misses backup	New ticket every check cycle	One ticket opened, ticket title includes hours overdue
Job still failing next day	Another new ticket	Same ticket remains open, priority escalated
Job still failing after 2 days	Another new ticket	Ticket escalated to High
Job still failing after 7 days	Another new ticket	Ticket escalated to Critical
Backup succeeds	Tickets stay open, manual close	Ticket automatically resolved
Machine stale >30 days	Continuous daily alert	Skip ticket creation (likely abandoned/decommissioned machine)

Ticket Content

RPO tickets are filed to Operations Triage queue with:

• Title: [Veeam RPO] <JobName> @ <OrgName> — <N>h since last backup
• Issue Type: Backups / Veeam Agent for Microsoft Windows
• Description: Job name, org, schedule, last successful backup timestamp, hours overdue, restore points available, categorized failure reason, raw error

Failure categories surfaced automatically:

• License Expired — renew via VSPC
• Cloud Gateway Unreachable — check vcg01.wulfconsulting.com
• Backup Repository Inaccessible
• Service Provider Maintenance
• Network/Connectivity Error
• Backup Job Timeout
• VBM desync (raw message)

---

3. Estimated Impact

Metric	Current (RMM)	Projected (RPO)
Tickets/month (backup)	~950–1,100	~50–150 (one per new breach, not per check)
Duplicate tickets for same device	Up to 41/month	0 (deduped by job UID)
Manual ticket closures required	All	0 (auto-resolved on success)
Failure root cause in ticket	No	Yes (categorized + raw message)
False positives (offline weekend)	Yes	Minimal (RPO + grace window)
Escalation based on severity	No	Yes (medium → high → critical)

---

4. Implementation Steps

Step 1 — Enable Veeam Sync

The RPO service reads from the Veeam VSPC sync tables. The sync schedules already exist but are disabled:

UPDATE sync_schedules SET is_enabled = true WHERE sync_type IN ('veeam-incremental', 'veeam-full');

The 30-minute incremental sync keeps job status current enough for RPO evaluation.

Step 2 — Add RPO Check Schedule

Add a schedule entry to run the RPO check every 2 hours:

INSERT INTO sync_schedules (id, sync_type, cron_expression, is_enabled, description)
VALUES ('veeam-rpo-check', 'veeam-rpo-check', '0 */2 * * *', true, 'Veeam RPO check — creates/escalates/resolves backup tickets');

The scheduler needs to handle the veeam-rpo-check sync type by calling POST /api/veeam/rpo-check.

Step 3 — Disable the 6 Datto RMM Backup Monitors

In Datto RMM, disable ticket creation (or disable the monitors entirely) for the 6 monitors listed in Section 1. This eliminates the duplicate ticket stream. The monitors can remain as alerting events in RMM itself without creating Autotask tickets, if desired.

Do not disable the monitors before the RPO service is confirmed working. Run both in parallel for at least one week to validate coverage.

Step 4 — Dry Run Before Go-Live

The RPO check endpoint supports a dry-run mode that shows what it would do without creating any tickets:

curl -X POST https://pulse.wulfconsulting.cloud/api/veeam/rpo-check \
  -H "Content-Type: application/json" \
  -d '{"dryRun": true}'

This returns wouldCreate, wouldResolve, wouldEscalate, and wouldSkipTooOld counts along with the full job status list.

Step 5 — Validate & Monitor

After enabling, confirm via Autotask that:

• RPO tickets are appearing with [Veeam RPO] prefix in the title
• Resolved tickets are auto-closing when backups succeed
• Ticket count is trending down from the ~1,100/month baseline

---

5. What Is NOT Changing

• Backup infrastructure is unchanged — WNPs, Veeam agents, Wasabi, VSPC all remain identical
• Datto RMM monitoring — RMM continues to monitor Veeam; only the ticket-creation action on those monitors is disabled
• Server/VM backup alerting — this recommendation is scoped to workstation jobs only; server and VM backup alerting should be evaluated separately
• Restore workflows — restore requests continue to be filed as standard Autotask tickets by users/staff

---

This document was produced by analyzing Autotask ticket data and the Veeam VSPC sync in the Pulse PostgreSQL database. The RPO service implementation is in lib/services/veeam-rpo-service.ts and app/api/veeam/rpo-check/route.ts.