wulf-pulse/docs/veeam-backup-alerting-recommendation.md

# Veeam Backup Alerting — Current State & RPO-Based Recommendation

> **Date:** April 14, 2026
> **Author:** Pulse / Cascade
> **Status:** Recommendation — not yet implemented

---

## 1. Problem: The Current Alerting Pipeline is Noisy

### How It Works Today

Every Veeam backup alert that reaches Autotask passes through **Datto RMM** — Veeam does not create Autotask tickets directly. There are 6 Datto RMM monitors that watch for Veeam conditions (via Windows Event Log or script checks on each WNP/endpoint) and auto-create tickets when a condition is true:

| RMM Monitor ID | Description | Tickets (Last 30 Days) |
|---|---|---|
| **422325** | Veeam Agent Backup Stalled | 638 |
| **770091** | Backup Copy Job Failed | 194 |
| **369602** | Veeam Agent Backup Failed | 103 |
| **369604** | Veeam Backup Job Missing or Stalled | 61 |
| **392069** | Veeam Agent Job Finished with Failed | 58 |
| **849483** | Veeam Server Agent Backup Job Stalled | 38 |

**Total: ~1,100 backup-related tickets per month** from these 6 monitors alone.

### Monthly Ticket Volume (Last 6 Months)

| Month | Backup Tickets |
|---|---|
| October 2025 | 461 |
| November 2025 | 1,033 |
| December 2025 | 1,008 |
| January 2026 | 1,583 |
| February 2026 | 728 |
| March 2026 | 946 |

### The Core Defects

**1. No deduplication — one new ticket per check cycle.**
A workstation that misses its backup today gets a new Autotask ticket every time the RMM monitor runs. `DT061` generated **41 individual tickets in 30 days** for a single laptop. In the last 30 days, **23 devices each triggered 5+ alerts**, producing **387 redundant tickets** from a single monitor.

**2. No schedule awareness.**
The monitors check "has Veeam run in X days?" without knowing the job's schedule. A laptop that is legitimately offline over a weekend gets stalled alerts on Saturday and Sunday even though no backup was missed relative to its RPO.

**3. No auto-resolution.**
When the underlying backup issue is fixed and the job succeeds, the open Autotask tickets are not automatically closed. Resolution requires manual action.

**4. No escalation logic.**
A job that has been failing for 3 days looks identical in Autotask to one that missed a single run — both generate the same priority ticket.

**5. No failure context.**
The RMM alerts contain only the device name and a generic "stalled" or "failed" label. The root cause (VBM desync, Wasabi DNS failure, VSS error, license expiry) is not surfaced in the ticket.

---

## 2. Recommendation: RPO-Based Alerting via Pulse

### What Already Exists

Pulse already contains a fully-built **RPO monitoring service** at `lib/services/veeam-rpo-service.ts` and an API endpoint at `POST /api/veeam/rpo-check`. The service:

- Reads live job data from the Veeam VSPC sync in PostgreSQL (`veeam_backup_agent_jobs`)
- Computes whether each job has breached its **Recovery Point Objective (RPO)** — i.e., the acceptable maximum gap since the last successful backup — based on the job's configured schedule
- Creates **one Autotask ticket per breached job** (deduped via the `veeam_rpo_tickets` tracking table)
- **Auto-resolves** that ticket the moment a successful backup run is detected
- **Escalates** the ticket's priority as the RPO violation ages
- **Categorizes** the failure reason from the Veeam failure message

The `veeam_rpo_tickets` table exists and is ready. It currently has 0 rows because the service has never been run.

### RPO Threshold Logic

| Schedule Type | RPO Window | Alert After | Escalate to High | Escalate to Critical |
|---|---|---|---|---|
| **Daily** | 24h | +4h grace = 28h | +48h | +7 days |
| **Weekly** | 168h | +4h grace = 172h | +48h | +7 days |
| **Continuous** | 1h | +1h grace = 2h | +4h | +12h |

The **grace period** (4h for daily jobs) accounts for jobs that run slightly late due to the workstation being offline, slow networks, or queued jobs on the WNP — preventing false positives for on-time jobs with minor delays.

### Ticket Behavior

| Event | Current (RMM) | Proposed (RPO) |
|---|---|---|
| Job misses backup | New ticket every check cycle | One ticket opened, ticket title includes hours overdue |
| Job still failing next day | Another new ticket | Same ticket remains open, priority escalated |
| Job still failing after 2 days | Another new ticket | Ticket escalated to High |
| Job still failing after 7 days | Another new ticket | Ticket escalated to Critical |
| Backup succeeds | Tickets stay open, manual close | Ticket automatically resolved |
| Machine stale >30 days | Continuous daily alert | Skip ticket creation (likely abandoned/decommissioned machine) |

### Ticket Content

RPO tickets are filed to **Operations Triage** queue with:
- **Title:** `[Veeam RPO] <JobName> @ <OrgName> — <N>h since last backup`
- **Issue Type:** Backups / Veeam Agent for Microsoft Windows
- **Description:** Job name, org, schedule, last successful backup timestamp, hours overdue, restore points available, categorized failure reason, raw error

**Failure categories surfaced automatically:**
- License Expired — renew via VSPC
- Cloud Gateway Unreachable — check `vcg01.wulfconsulting.com`
- Backup Repository Inaccessible
- Service Provider Maintenance
- Network/Connectivity Error
- Backup Job Timeout
- VBM desync (raw message)

---

## 3. Estimated Impact

| Metric | Current (RMM) | Projected (RPO) |
|---|---|---|
| Tickets/month (backup) | ~950–1,100 | **~50–150** (one per new breach, not per check) |
| Duplicate tickets for same device | Up to 41/month | **0** (deduped by job UID) |
| Manual ticket closures required | All | **0** (auto-resolved on success) |
| Failure root cause in ticket | No | **Yes** (categorized + raw message) |
| False positives (offline weekend) | Yes | **Minimal** (RPO + grace window) |
| Escalation based on severity | No | **Yes** (medium → high → critical) |

---

## 4. Implementation Steps

### Step 1 — Enable Veeam Sync

The RPO service reads from the Veeam VSPC sync tables. The sync schedules already exist but are disabled:

```sql
UPDATE sync_schedules SET is_enabled = true WHERE sync_type IN ('veeam-incremental', 'veeam-full');
```

The 30-minute incremental sync keeps job status current enough for RPO evaluation.

### Step 2 — Add RPO Check Schedule

Add a schedule entry to run the RPO check every 2 hours:

```sql
INSERT INTO sync_schedules (id, sync_type, cron_expression, is_enabled, description)
VALUES ('veeam-rpo-check', 'veeam-rpo-check', '0 */2 * * *', true, 'Veeam RPO check — creates/escalates/resolves backup tickets');
```

The scheduler needs to handle the `veeam-rpo-check` sync type by calling `POST /api/veeam/rpo-check`.

### Step 3 — Disable the 6 Datto RMM Backup Monitors

In Datto RMM, disable ticket creation (or disable the monitors entirely) for the 6 monitors listed in Section 1. This eliminates the duplicate ticket stream. The monitors can remain as alerting events in RMM itself without creating Autotask tickets, if desired.

> **Do not disable the monitors before the RPO service is confirmed working.** Run both in parallel for at least one week to validate coverage.

### Step 4 — Dry Run Before Go-Live

The RPO check endpoint supports a dry-run mode that shows what it *would* do without creating any tickets:

```bash
curl -X POST https://pulse.wulfconsulting.cloud/api/veeam/rpo-check \
  -H "Content-Type: application/json" \
  -d '{"dryRun": true}'
```

This returns `wouldCreate`, `wouldResolve`, `wouldEscalate`, and `wouldSkipTooOld` counts along with the full job status list.

### Step 5 — Validate & Monitor

After enabling, confirm via Autotask that:
- RPO tickets are appearing with `[Veeam RPO]` prefix in the title
- Resolved tickets are auto-closing when backups succeed
- Ticket count is trending down from the ~1,100/month baseline

---

## 5. What Is NOT Changing

- **Backup infrastructure is unchanged** — WNPs, Veeam agents, Wasabi, VSPC all remain identical
- **Datto RMM monitoring** — RMM continues to monitor Veeam; only the ticket-creation action on those monitors is disabled
- **Server/VM backup alerting** — this recommendation is scoped to workstation jobs only; server and VM backup alerting should be evaluated separately
- **Restore workflows** — restore requests continue to be filed as standard Autotask tickets by users/staff

---

*This document was produced by analyzing Autotask ticket data and the Veeam VSPC sync in the Pulse PostgreSQL database. The RPO service implementation is in `lib/services/veeam-rpo-service.ts` and `app/api/veeam/rpo-check/route.ts`.*