wulf-pulse/docs/workstation-backup-overview.md

# Wulf Consulting — Workstation Backup Overview

> **Generated:** April 10, 2026
> **Sources:** Veeam VSPC API (live sync via Pulse), Autotask PSA ticket data, IT Glue documentation
> **Scope:** Workstation (laptop and desktop) backups only — server and VM backups are excluded

---

## 1. Executive Summary

Wulf Consulting protects **1,219 workstation backup jobs** across **59 client organizations** using **Veeam Backup & Replication** managed through the **Veeam Service Provider Console (VSPC)**. Workstations are backed up via **Veeam Agent for Windows** (and 1 Mac agent) deployed to endpoints, with backup policies pushed centrally from on-premise Veeam Backup & Replication servers (WNPs) at each client site. Data flows to two tiers: **local on-prem repositories** and **Wasabi S3-compatible cloud storage** for offsite/immutable copies.

| Metric | Value |
|---|---|
| Total workstation backup jobs | 1,219 |
| Active Veeam workstation agents | 688 |
| Client organizations covered | 59 |
| Current success rate | **83.3%** (1,016 Success) |
| Failed jobs | 84 (6.9%) |
| Warning jobs | 2 |
| Jobs with status "None" (never run / new) | 49 |
| Backup-related Autotask tickets (last 12 months) | 13,946 |

---

## 2. Architecture

### 2.1 Platform Stack

| Component | Details |
|---|---|
| **Backup Software** | Veeam Backup & Replication v12.3 / v13.0 |
| **Management Console** | Veeam Service Provider Console (VSPC) at `vac.wulfconsulting.com:1280` |
| **Agent** | Veeam Agent for Windows (687 workstations) + Veeam Agent for Mac (1 workstation) |
| **Cloud Connect Server** | `wemvemasp02` — Veeam v13.0.1.1071, role: CloudConnect, status: Healthy |
| **Client Backup Servers** | 46 on-premise Windows servers (WNPs) at client sites, all status: Healthy |
| **Cloud Storage** | Wasabi S3 (us-east-1) via Veeam Cloud Connect — immutable object lock |
| **Monitoring** | VSPC alarms → Datto RMM alerts → Autotask ticket auto-creation |

### 2.2 On-Premise Backup Servers (WNPs)

Each managed client site has a dedicated **Workstation Network Proxy (WNP)** — a Windows server running Veeam Backup & Replication that acts as the local backup infrastructure. These WNPs are registered to VSPC as "Client" role backup servers.

**Top WNPs by job count:**

| WNP Server | Client | Workstation Jobs |
|---|---|---|
| `pitseuwnp01` | Seubert and Associates | 274 |
| `ynghynwnp01` | Hynes Industries | 254 |
| `monprewnp01` | Premier Automation Holdings | 125 |
| `geoprewnp01` | Premier Automation Holdings | 125 |
| `PITPOHWNP01` | POH+W Architects | 86 |
| `pgbvsywnp01` | V-Systems | 57 |
| `hpbhrswnp01` | Hergenroeder Rega Ewing Kennedy | 44 |

All 46 client WNPs are running Veeam v12.3.2.4165 (two upgraded to v13.0.1.1071). All report **Healthy** status.

### 2.3 Data Flow

```
Workstation (Veeam Agent)
    ├── [1] Local backup → On-prem WNP repository (NAS / local disk)
    └── [2] Cloud copy  → WNP → Cloud Connect (wemvemasp02) → Wasabi S3 (immutable)
```

Each workstation runs a Veeam Agent that executes a daily backup job. Most workstations have **two jobs**: one targeting a local repository on the WNP and one replicating to Wasabi via the VSPC Cloud Connect gateway. The cloud copy is stored in per-machine immutable object-lock repositories (naming convention: `wulf-<client>_<hostname>_Repository`).

---

## 3. Backup Policies & Job Configuration

### 3.1 Standard Policies

Wulf deploys centrally managed backup policies through VSPC. The two primary workstation policies are:

| Policy Name | Jobs | Success | Failed | Backup Mode | Target |
|---|---|---|---|---|---|
| `WulfStdWRKSTFiles` | 519 | 455 | 28 | File-level | Cloud (Wasabi) |
| `WulfStdWRKSTFilesWasabi` | 400 | 340 | 36 | File-level | Cloud (Wasabi) |
| `Premier_WulfStdWRKSTFilesD_Drive` | 48 | 36 | 5 | File-level | Cloud |
| `Premier_WulfStdWRKSTFilesD_Drive_Wasabi` | 35 | 29 | 5 | File-level | Cloud |

Additional per-site policies exist for clients with specific requirements (e.g., `POH_ATL_Desktops`, `Blake_Desktops`, `Greco_TAR_Desktops`), each targeting the local WNP.

### 3.2 Backup Modes

| Mode | Jobs | Avg Backed-Up Size | Total Data |
|---|---|---|---|
| **File-level** | 1,041 (85%) | 18.88 GB | 18.96 TB |
| **Entire Computer** (volume/image) | 178 (15%) | 336.60 GB | 44.38 TB |

- **File-level** — protects user profile data (Documents, Desktop, Downloads, AppData). This is the standard for most workstations.
- **Entire Computer** — full volume image backup. Used for specialized workstations (LOB app machines, CAD/engineering stations, machines with large local data stores).

### 3.3 Backup Targets

| Destination | Jobs | Total Data Protected |
|---|---|---|
| **Wasabi S3 (Cloud)** | 1,043 (86%) | 19.36 TB |
| **Local (On-Prem WNP)** | 176 (14%) | 44.38 TB |

The cloud-targeted jobs use Wasabi S3-compatible storage via the VSPC Cloud Connect gateway (`wemvemasp02`). Local-targeted jobs write to storage attached to the client's WNP server.

### 3.4 Schedule

All 1,219 workstation backup jobs are configured with a **Daily** schedule. Per IT Glue documentation, typical trigger windows are:

- **Morning run:** 6:00 AM, 9:00 AM, or 11:00 AM (varies by client)
- **Event-based:** On logoff/lock/restart events, with a minimum interval of 4–6 hours between runs
- **Backup window:** 24×7 (jobs can run any time when the workstation is available)

> *IT Glue reference: Backup flexible asset documentation records per-client schedules. Example — Seubert and Associates desktops: "6am + log off event, not exceeding 6 hours" with Veeam Volume/Image method.*

### 3.5 Retention

**Per IT Glue Backup documentation (195 assets across all device types):**

| Retention Tier | Policy | Count |
|---|---|---|
| **Local** | Veeam 4 Weeks | 62 |
| **Local** | Veeam 1 Week | 49 |
| **Local** | Veeam 2 Weeks | 33 |
| **Offsite** | GFS - 1 Year | 54 |
| **Offsite** | Veeam O365 1 year | 14 |
| **Offsite** | GFS - 6 Month | 1 |

Workstation cloud copies average **18.9 restore points** per job (file-level) and **14.5 restore points** per job (entire computer).

### 3.6 Immutable / Offsite Storage

Per IT Glue documentation, **54 backup assets** reference **Wasabi Object Storage** as the offsite immutable location. Immutable object lock ensures that backup data cannot be deleted or modified by ransomware or compromised credentials for the duration of the retention period.

Cloud repositories follow the naming convention `wulf.<client>.veeam.workstations.immutable` — e.g.:
- `wulf.seubert.veeam.workstations.immutable`
- `wulf.vsys.veeam.workstations.immutable`
- `wulf.tcg.veeam.workstations.immutable`
- `wulf.superior.veeam.workstations.immutable`

---

## 4. Client Coverage

### 4.1 Top Clients by Workstation Backup Volume

| Client | Jobs | Success | Failed | Warning |
|---|---|---|---|---|
| Seubert and Associates | 274 | 224 | 32 | 0 |
| Hynes Industries | 254 | 225 | 14 | 0 |
| Premier Automation Holdings | 125 | 101 | 13 | 0 |
| POH+W Architects | 86 | 67 | 4 | 1 |
| ConnecTel | 58 | 52 | 3 | 0 |
| V-Systems | 57 | 53 | 2 | 0 |
| Hergenroeder Rega Ewing Kennedy | 44 | 41 | 1 | 0 |
| Superior Distributing Co | 40 | 35 | 2 | 0 |
| Greco Gas | 35 | 29 | 2 | 0 |
| Thoroughbred Construction Group | 32 | 31 | 0 | 0 |
| Commonwealth Suburban Title Agency | 25 | 9 | 3 | 0 |
| Nordmann Roofing | 17 | 17 | 0 | 0 |
| Blake Dentistry | 15 | 15 | 0 | 0 |
| TK Plastics Company | 13 | 13 | 0 | 0 |

### 4.2 IT Glue Backup Documentation

IT Glue contains **195 Backup flexible assets** across all managed clients, broken down by documented scope:

| Category | Documented Assets |
|---|---|
| Server backups | 69 |
| Desktop backups | 44 |
| Other / mixed | 32 |
| Laptop backups | 30 |
| M365 backups | 18 |
| LOB application backups | 2 |

Each asset records: backup software, method, frequency, window, local retention, offsite provider, offsite schedule, offsite retention, protected devices (tagged), and restore approval contacts.

> *IT Glue also maintains 3 "Veeam (auto)" flexible assets that auto-document the VSPC tenant structure including sites, sub-tenants, and company mappings.*

---

## 5. Current Health & Failure Analysis

### 5.1 Job Status Breakdown

| Status | Count | Percentage |
|---|---|---|
| ✅ Success | 1,016 | 83.3% |
| ❌ Failed | 84 | 6.9% |
| ⚠️ Warning | 2 | 0.2% |
| ⬜ None (never run) | 49 | 4.0% |
| 🔒 Disabled | 0 | 0% |
| 📉 Stale (no run in 48h) | 1,219* | — |

*\*Note: The "stale" count includes all jobs where `last_run` is older than 48 hours or null — this is inflated by the daily schedule; many jobs last ran within 24 hours but are captured by the 48h window. The actual number of concerning stale jobs is the 49 with status "None."*

### 5.2 Failure Root Causes

| Failure Pattern | Failed Jobs | Description |
|---|---|---|
| **VBM Desync (needs rescan)** | 28 | Backup metadata file out of sync with DB; requires manual repository rescan on the WNP |
| **Other** | 35 | Mixed errors — agent communication failures, locked files, transient issues |
| **Wasabi DNS/Connectivity** | 14 | Cannot resolve `s3.us-east-1.wasabisys.com` — DNS or internet outage at client site |
| **VSPC Connectivity** | 6 | "Service provider's infrastructure is not responding" — VSPC gateway temporarily unavailable |
| **VSS Error** | 1 | Volume Shadow Copy failure on the workstation |

The **dominant failure mode** is VBM (Veeam Backup Metadata) desynchronization, which requires a repository rescan on the affected WNP. This is a known Veeam issue that typically resolves after the rescan completes.

### 5.3 Backup Job Performance

| Backup Mode | Avg Duration (sec) | Avg Backed-Up Size (GB) |
|---|---|---|
| Entire Computer | 3,212 (~53 min) | 332.21 GB |
| File-level | 1,971 (~33 min) | 17.75 GB |

---

## 6. Autotask Ticket Data — Operational Insights

### 6.1 Backup Ticket Volume (Last 12 Months)

Wulf's Autotask PSA logged **13,946 backup-related tickets** in the past 12 months. These are predominantly auto-generated alerts from the monitoring pipeline (Datto RMM → Autotask webhook):

| Category | Tickets | Resolved |
|---|---|---|
| Backup Failure alerts | 7,181 | 7,177 (99.9%) |
| Veeam Agent issues | 5,378 | 5,358 (99.6%) |
| Veeam General | 1,036 | 1,031 (99.5%) |
| Restore Requests | 223 | 222 (99.6%) |
| Backup General | 128 | 125 (97.7%) |

The vast majority of backup tickets are **automated monitoring alerts** that are triaged and resolved by the operations team. The **99.5%+ resolution rate** indicates an effective remediation workflow.

### 6.2 Recent Ticket Examples (April 2026)

**Automated alerts (Priority 6 — Monitoring):**
- `T20260410.0054` — *Backup Alert - SP021 (Surface Pro 9) - Veeam Agent Backup Stalled (POH+W Architects - Atlanta)*
- `T20260410.0047` — *Backup Alert - LT065 (21MV LENOVO) - Veeam Agent Backup Stalled* — ConnecTel
- `T20260410.0036` — *Backup Alert - YNGHYNLT084 (Precision 7680, Dell) - Veeam Agent Backup Stalled* — Hynes Industries
- `T20260410.0028` — *Backup Alert - pgbvsywnp01 - Veeam Backup Job Missing or Stalled* — V-Systems

**Backup copy failures (server-side, affects workstation offsite copies):**
- `T20260410.0065` — *Backup Copy Job Failed - FOSSUPWNP01 - Superior-FOS-BackupCopyVMs-WasabiV2* — Superior Distributing Co
- `T20260410.0049` — *Backup Copy Job Failed - ynghynwnp01 - Hynes-BackupCopyVMs-WulfWasabiV2* — Hynes Industries

### 6.3 Restore Requests (Last 6 Months — Sample)

| Date | Ticket | Description | Client |
|---|---|---|---|
| 2026-04-06 | T20260406.0227 | Folder Restore | Thrasher Group |
| 2026-03-19 | T20260319.0350 | Restore overwritten Excel file from S:\Filestore | Insurance Restoration Consultants |
| 2026-03-16 | T20260316.0288 | Restore overwritten Photoshop file from G: drive | POH+W Architects |
| 2026-03-04 | T20260304.0210 | Device reimage — restore files via Veeam after wipe | Kuhn's Quality Foods |
| 2026-03-03 | T20260303.0189 | Veeam 365 Restore Request — OneDrive recordings | Lighthouse Electric |
| 2026-03-01 | T20260301.0108 | Restore archived Outlook calendar items | Hergenroeder Rega Ewing Kennedy |
| 2026-02-26 | T20260226.0163 | Restore emptied shared folder from file server | Blackburn's Physicians Pharmacy |

Restore requests demonstrate the breadth of recovery scenarios handled: accidental file deletion, file overwrites, full device reimaging, and application-level restores (Outlook, OneDrive).

---

## 7. IT Glue Documentation Structure

### 7.1 Backup Flexible Asset Schema

Each client's workstation backup is documented in IT Glue as a **Backup** flexible asset (type ID: 3791) with the following fields:

| Field | Type | Purpose |
|---|---|---|
| Backup Software | Select | Platform (Veeam for all workstations) |
| Backup Method | Select | Files, Volume/Image, Office365 Veeam, Other |
| Backup Description | Text | What is being backed up |
| Backup Frequency | Text | Schedule description (e.g., "6am + log off event") |
| Backup Window | Select | When backups can run (typically 24×7) |
| Wulf Backup Package | Select | Service tier: All-Desktops, All-Laptops, All-Servers, Selective, etc. |
| Local Backup Server(s) | Tag | Tagged configuration items (WNP servers) |
| Local Location | Text | Repository path (e.g., `D:\Backup`) |
| Local Retention | Select | Veeam 1 Week / 2 Weeks / 4 Weeks |
| Offsite Provider | Select | Wulf, Veeam, Datto, Druva, Other |
| Offsite Replication Schedule | Select | Frequency of offsite copy |
| Offsite Retention | Select | GFS-1 Year, GFS-6 Month, Veeam O365 1 Year |
| Offsite Immutable Location | Select | Wasabi Object Storage |
| Protected Devices | Tag | Tagged workstation configuration items |
| Who Approves Restore Requests? | Tag | Contact(s) authorized to approve restores |
| Last Backup Verification | Date | Date of last manual restore test |
| Next Verification | Date | Scheduled next restore verification |

### 7.2 Documented Backup Methods (All Device Types)

| Method | Assets |
|---|---|
| Volume/Image | 127 |
| Files | 47 |
| Office365 Veeam | 16 |
| Other | 4 |

### 7.3 Wulf Backup Packages

| Package | Assets |
|---|---|
| All - Servers | 60 |
| All - Desktops | 36 |
| Selective - See Protected Devices | 34 |
| All - Laptops | 24 |
| All - O365 | 17 |
| TAM - Sell | 6 |

---

## 8. Standard Operating Procedure

Based on the data above, Wulf Consulting's standard workstation backup workflow is:

1. **Deployment:** Veeam Agent for Windows is installed on each managed workstation during onboarding. The agent is registered to the client's on-premise WNP server.

2. **Policy Assignment:** A VSPC backup policy (e.g., `WulfStdWRKSTFiles` or `WulfStdWRKSTFilesWasabi`) is assigned to the agent. This determines backup mode (file-level or volume), schedule, and target.

3. **Daily Execution:** The agent runs on a daily schedule (typically early morning + logoff/lock events). File-level jobs take ~33 minutes on average; full image jobs take ~53 minutes.

4. **Local Storage:** For clients with on-prem image backups, data is written to the WNP's local repository (NAS or direct-attached storage). Local retention is typically 1–4 weeks.

5. **Cloud Replication:** A second job (or backup copy job on the WNP) replicates data to Wasabi S3 via the VSPC Cloud Connect gateway (`wemvemasp02`). Cloud copies are stored in per-machine **immutable** repositories with GFS retention (typically 1 year).

6. **Monitoring:** VSPC monitors job status and raises alarms. These propagate to Datto RMM, which creates Autotask tickets automatically. The operations team triages alerts daily.

7. **Remediation:** Failed jobs are investigated — common fixes include repository rescans (VBM desync), DNS resolution (Wasabi connectivity), and agent reinstalls.

8. **Restores:** End users or client contacts request restores via Autotask ticket. Wulf engineers recover files from local or cloud repositories as needed.

9. **Documentation:** Each client's backup configuration is documented in IT Glue with method, schedule, retention, protected devices, and restore approval contacts.

---

## 9. Key Observations & Recommendations

### Strengths
- **Comprehensive coverage:** 688 active workstation agents across 59 clients with centralized VSPC management
- **Immutable offsite copies:** Wasabi S3 object lock protects against ransomware and accidental deletion
- **Automated monitoring pipeline:** VSPC → Datto RMM → Autotask ensures no backup failure goes unnoticed
- **High resolution rate:** 99.5%+ of backup tickets are resolved, indicating effective operational processes
- **Standardized policies:** Two primary policies (`WulfStdWRKSTFiles`, `WulfStdWRKSTFilesWasabi`) cover 75% of jobs

### Areas for Attention
- **VBM desync failures (28 jobs):** The most common failure mode. Consider automating repository rescans or upgrading to Veeam v13 which improved metadata handling.
- **Wasabi DNS failures (14 jobs):** Client-side DNS resolution issues. May benefit from secondary DNS or direct-IP fallback configuration.
- **49 jobs with "None" status:** Jobs that have never run — likely newly deployed agents awaiting first execution or stale configurations.
- **84 total failed jobs (6.9%):** While the alert pipeline catches these, the raw failure rate could be improved by addressing the top two root causes (VBM desync + Wasabi DNS).

---

*This document was compiled from live VSPC data synced to Pulse, Autotask ticket history, and IT Glue backup documentation. Data reflects the state as of April 10, 2026.*