wulf-pulse/docs/workstation-backup-overview.md
lorentz ea3471d38d feat: Veeam RPO analysis, comparison, ticket analysis + company teams table
- Add Veeam RPO analysis page (/veeam-analysis) and comparison page (/veeam-comparison)
- Add API routes: /api/veeam/rpo-analyze, rpo-comparison, rpo-offline-log, ticket-analysis
- Add veeam-rpo-service.ts enhancements (RPO logic, offline detection, comparison)
- Add veeam-analysis-state.ts and rmm-device-resolver.ts services
- Add migrations 065-068: company_teams, veeam_rpo_offline_log, rpo_comparison_tables, veeam_ticket_analysis
- Add backup-status page updates and nav links for new Veeam pages
- Add scripts: deactivate-cis-for-inactive-companies, workstation category updates
- Add docs: mimecast-api-guide, veeam-backup-alerting-recommendation, workstation-backup-overview, ticket-analyzer-prompt
- Minor: webhook-service, entity-sync, entity-mapper, sync-helpers, sync.ts, middleware.ts updates
2026-04-29 09:16:46 -04:00

17 KiB
Raw Blame History

Wulf Consulting — Workstation Backup Overview

Generated: April 10, 2026 Sources: Veeam VSPC API (live sync via Pulse), Autotask PSA ticket data, IT Glue documentation Scope: Workstation (laptop and desktop) backups only — server and VM backups are excluded


1. Executive Summary

Wulf Consulting protects 1,219 workstation backup jobs across 59 client organizations using Veeam Backup & Replication managed through the Veeam Service Provider Console (VSPC). Workstations are backed up via Veeam Agent for Windows (and 1 Mac agent) deployed to endpoints, with backup policies pushed centrally from on-premise Veeam Backup & Replication servers (WNPs) at each client site. Data flows to two tiers: local on-prem repositories and Wasabi S3-compatible cloud storage for offsite/immutable copies.

Metric Value
Total workstation backup jobs 1,219
Active Veeam workstation agents 688
Client organizations covered 59
Current success rate 83.3% (1,016 Success)
Failed jobs 84 (6.9%)
Warning jobs 2
Jobs with status "None" (never run / new) 49
Backup-related Autotask tickets (last 12 months) 13,946

2. Architecture

2.1 Platform Stack

Component Details
Backup Software Veeam Backup & Replication v12.3 / v13.0
Management Console Veeam Service Provider Console (VSPC) at vac.wulfconsulting.com:1280
Agent Veeam Agent for Windows (687 workstations) + Veeam Agent for Mac (1 workstation)
Cloud Connect Server wemvemasp02 — Veeam v13.0.1.1071, role: CloudConnect, status: Healthy
Client Backup Servers 46 on-premise Windows servers (WNPs) at client sites, all status: Healthy
Cloud Storage Wasabi S3 (us-east-1) via Veeam Cloud Connect — immutable object lock
Monitoring VSPC alarms → Datto RMM alerts → Autotask ticket auto-creation

2.2 On-Premise Backup Servers (WNPs)

Each managed client site has a dedicated Workstation Network Proxy (WNP) — a Windows server running Veeam Backup & Replication that acts as the local backup infrastructure. These WNPs are registered to VSPC as "Client" role backup servers.

Top WNPs by job count:

WNP Server Client Workstation Jobs
pitseuwnp01 Seubert and Associates 274
ynghynwnp01 Hynes Industries 254
monprewnp01 Premier Automation Holdings 125
geoprewnp01 Premier Automation Holdings 125
PITPOHWNP01 POH+W Architects 86
pgbvsywnp01 V-Systems 57
hpbhrswnp01 Hergenroeder Rega Ewing Kennedy 44

All 46 client WNPs are running Veeam v12.3.2.4165 (two upgraded to v13.0.1.1071). All report Healthy status.

2.3 Data Flow

Workstation (Veeam Agent)
    ├── [1] Local backup → On-prem WNP repository (NAS / local disk)
    └── [2] Cloud copy  → WNP → Cloud Connect (wemvemasp02) → Wasabi S3 (immutable)

Each workstation runs a Veeam Agent that executes a daily backup job. Most workstations have two jobs: one targeting a local repository on the WNP and one replicating to Wasabi via the VSPC Cloud Connect gateway. The cloud copy is stored in per-machine immutable object-lock repositories (naming convention: wulf-<client>_<hostname>_Repository).


3. Backup Policies & Job Configuration

3.1 Standard Policies

Wulf deploys centrally managed backup policies through VSPC. The two primary workstation policies are:

Policy Name Jobs Success Failed Backup Mode Target
WulfStdWRKSTFiles 519 455 28 File-level Cloud (Wasabi)
WulfStdWRKSTFilesWasabi 400 340 36 File-level Cloud (Wasabi)
Premier_WulfStdWRKSTFilesD_Drive 48 36 5 File-level Cloud
Premier_WulfStdWRKSTFilesD_Drive_Wasabi 35 29 5 File-level Cloud

Additional per-site policies exist for clients with specific requirements (e.g., POH_ATL_Desktops, Blake_Desktops, Greco_TAR_Desktops), each targeting the local WNP.

3.2 Backup Modes

Mode Jobs Avg Backed-Up Size Total Data
File-level 1,041 (85%) 18.88 GB 18.96 TB
Entire Computer (volume/image) 178 (15%) 336.60 GB 44.38 TB
  • File-level — protects user profile data (Documents, Desktop, Downloads, AppData). This is the standard for most workstations.
  • Entire Computer — full volume image backup. Used for specialized workstations (LOB app machines, CAD/engineering stations, machines with large local data stores).

3.3 Backup Targets

Destination Jobs Total Data Protected
Wasabi S3 (Cloud) 1,043 (86%) 19.36 TB
Local (On-Prem WNP) 176 (14%) 44.38 TB

The cloud-targeted jobs use Wasabi S3-compatible storage via the VSPC Cloud Connect gateway (wemvemasp02). Local-targeted jobs write to storage attached to the client's WNP server.

3.4 Schedule

All 1,219 workstation backup jobs are configured with a Daily schedule. Per IT Glue documentation, typical trigger windows are:

  • Morning run: 6:00 AM, 9:00 AM, or 11:00 AM (varies by client)
  • Event-based: On logoff/lock/restart events, with a minimum interval of 46 hours between runs
  • Backup window: 24×7 (jobs can run any time when the workstation is available)

IT Glue reference: Backup flexible asset documentation records per-client schedules. Example — Seubert and Associates desktops: "6am + log off event, not exceeding 6 hours" with Veeam Volume/Image method.

3.5 Retention

Per IT Glue Backup documentation (195 assets across all device types):

Retention Tier Policy Count
Local Veeam 4 Weeks 62
Local Veeam 1 Week 49
Local Veeam 2 Weeks 33
Offsite GFS - 1 Year 54
Offsite Veeam O365 1 year 14
Offsite GFS - 6 Month 1

Workstation cloud copies average 18.9 restore points per job (file-level) and 14.5 restore points per job (entire computer).

3.6 Immutable / Offsite Storage

Per IT Glue documentation, 54 backup assets reference Wasabi Object Storage as the offsite immutable location. Immutable object lock ensures that backup data cannot be deleted or modified by ransomware or compromised credentials for the duration of the retention period.

Cloud repositories follow the naming convention wulf.<client>.veeam.workstations.immutable — e.g.:

  • wulf.seubert.veeam.workstations.immutable
  • wulf.vsys.veeam.workstations.immutable
  • wulf.tcg.veeam.workstations.immutable
  • wulf.superior.veeam.workstations.immutable

4. Client Coverage

4.1 Top Clients by Workstation Backup Volume

Client Jobs Success Failed Warning
Seubert and Associates 274 224 32 0
Hynes Industries 254 225 14 0
Premier Automation Holdings 125 101 13 0
POH+W Architects 86 67 4 1
ConnecTel 58 52 3 0
V-Systems 57 53 2 0
Hergenroeder Rega Ewing Kennedy 44 41 1 0
Superior Distributing Co 40 35 2 0
Greco Gas 35 29 2 0
Thoroughbred Construction Group 32 31 0 0
Commonwealth Suburban Title Agency 25 9 3 0
Nordmann Roofing 17 17 0 0
Blake Dentistry 15 15 0 0
TK Plastics Company 13 13 0 0

4.2 IT Glue Backup Documentation

IT Glue contains 195 Backup flexible assets across all managed clients, broken down by documented scope:

Category Documented Assets
Server backups 69
Desktop backups 44
Other / mixed 32
Laptop backups 30
M365 backups 18
LOB application backups 2

Each asset records: backup software, method, frequency, window, local retention, offsite provider, offsite schedule, offsite retention, protected devices (tagged), and restore approval contacts.

IT Glue also maintains 3 "Veeam (auto)" flexible assets that auto-document the VSPC tenant structure including sites, sub-tenants, and company mappings.


5. Current Health & Failure Analysis

5.1 Job Status Breakdown

Status Count Percentage
Success 1,016 83.3%
Failed 84 6.9%
⚠️ Warning 2 0.2%
None (never run) 49 4.0%
🔒 Disabled 0 0%
📉 Stale (no run in 48h) 1,219*

*Note: The "stale" count includes all jobs where last_run is older than 48 hours or null — this is inflated by the daily schedule; many jobs last ran within 24 hours but are captured by the 48h window. The actual number of concerning stale jobs is the 49 with status "None."

5.2 Failure Root Causes

Failure Pattern Failed Jobs Description
VBM Desync (needs rescan) 28 Backup metadata file out of sync with DB; requires manual repository rescan on the WNP
Other 35 Mixed errors — agent communication failures, locked files, transient issues
Wasabi DNS/Connectivity 14 Cannot resolve s3.us-east-1.wasabisys.com — DNS or internet outage at client site
VSPC Connectivity 6 "Service provider's infrastructure is not responding" — VSPC gateway temporarily unavailable
VSS Error 1 Volume Shadow Copy failure on the workstation

The dominant failure mode is VBM (Veeam Backup Metadata) desynchronization, which requires a repository rescan on the affected WNP. This is a known Veeam issue that typically resolves after the rescan completes.

5.3 Backup Job Performance

Backup Mode Avg Duration (sec) Avg Backed-Up Size (GB)
Entire Computer 3,212 (~53 min) 332.21 GB
File-level 1,971 (~33 min) 17.75 GB

6. Autotask Ticket Data — Operational Insights

6.1 Backup Ticket Volume (Last 12 Months)

Wulf's Autotask PSA logged 13,946 backup-related tickets in the past 12 months. These are predominantly auto-generated alerts from the monitoring pipeline (Datto RMM → Autotask webhook):

Category Tickets Resolved
Backup Failure alerts 7,181 7,177 (99.9%)
Veeam Agent issues 5,378 5,358 (99.6%)
Veeam General 1,036 1,031 (99.5%)
Restore Requests 223 222 (99.6%)
Backup General 128 125 (97.7%)

The vast majority of backup tickets are automated monitoring alerts that are triaged and resolved by the operations team. The 99.5%+ resolution rate indicates an effective remediation workflow.

6.2 Recent Ticket Examples (April 2026)

Automated alerts (Priority 6 — Monitoring):

  • T20260410.0054Backup Alert - SP021 (Surface Pro 9) - Veeam Agent Backup Stalled (POH+W Architects - Atlanta)
  • T20260410.0047Backup Alert - LT065 (21MV LENOVO) - Veeam Agent Backup Stalled — ConnecTel
  • T20260410.0036Backup Alert - YNGHYNLT084 (Precision 7680, Dell) - Veeam Agent Backup Stalled — Hynes Industries
  • T20260410.0028Backup Alert - pgbvsywnp01 - Veeam Backup Job Missing or Stalled — V-Systems

Backup copy failures (server-side, affects workstation offsite copies):

  • T20260410.0065Backup Copy Job Failed - FOSSUPWNP01 - Superior-FOS-BackupCopyVMs-WasabiV2 — Superior Distributing Co
  • T20260410.0049Backup Copy Job Failed - ynghynwnp01 - Hynes-BackupCopyVMs-WulfWasabiV2 — Hynes Industries

6.3 Restore Requests (Last 6 Months — Sample)

Date Ticket Description Client
2026-04-06 T20260406.0227 Folder Restore Thrasher Group
2026-03-19 T20260319.0350 Restore overwritten Excel file from S:\Filestore Insurance Restoration Consultants
2026-03-16 T20260316.0288 Restore overwritten Photoshop file from G: drive POH+W Architects
2026-03-04 T20260304.0210 Device reimage — restore files via Veeam after wipe Kuhn's Quality Foods
2026-03-03 T20260303.0189 Veeam 365 Restore Request — OneDrive recordings Lighthouse Electric
2026-03-01 T20260301.0108 Restore archived Outlook calendar items Hergenroeder Rega Ewing Kennedy
2026-02-26 T20260226.0163 Restore emptied shared folder from file server Blackburn's Physicians Pharmacy

Restore requests demonstrate the breadth of recovery scenarios handled: accidental file deletion, file overwrites, full device reimaging, and application-level restores (Outlook, OneDrive).


7. IT Glue Documentation Structure

7.1 Backup Flexible Asset Schema

Each client's workstation backup is documented in IT Glue as a Backup flexible asset (type ID: 3791) with the following fields:

Field Type Purpose
Backup Software Select Platform (Veeam for all workstations)
Backup Method Select Files, Volume/Image, Office365 Veeam, Other
Backup Description Text What is being backed up
Backup Frequency Text Schedule description (e.g., "6am + log off event")
Backup Window Select When backups can run (typically 24×7)
Wulf Backup Package Select Service tier: All-Desktops, All-Laptops, All-Servers, Selective, etc.
Local Backup Server(s) Tag Tagged configuration items (WNP servers)
Local Location Text Repository path (e.g., D:\Backup)
Local Retention Select Veeam 1 Week / 2 Weeks / 4 Weeks
Offsite Provider Select Wulf, Veeam, Datto, Druva, Other
Offsite Replication Schedule Select Frequency of offsite copy
Offsite Retention Select GFS-1 Year, GFS-6 Month, Veeam O365 1 Year
Offsite Immutable Location Select Wasabi Object Storage
Protected Devices Tag Tagged workstation configuration items
Who Approves Restore Requests? Tag Contact(s) authorized to approve restores
Last Backup Verification Date Date of last manual restore test
Next Verification Date Scheduled next restore verification

7.2 Documented Backup Methods (All Device Types)

Method Assets
Volume/Image 127
Files 47
Office365 Veeam 16
Other 4

7.3 Wulf Backup Packages

Package Assets
All - Servers 60
All - Desktops 36
Selective - See Protected Devices 34
All - Laptops 24
All - O365 17
TAM - Sell 6

8. Standard Operating Procedure

Based on the data above, Wulf Consulting's standard workstation backup workflow is:

  1. Deployment: Veeam Agent for Windows is installed on each managed workstation during onboarding. The agent is registered to the client's on-premise WNP server.

  2. Policy Assignment: A VSPC backup policy (e.g., WulfStdWRKSTFiles or WulfStdWRKSTFilesWasabi) is assigned to the agent. This determines backup mode (file-level or volume), schedule, and target.

  3. Daily Execution: The agent runs on a daily schedule (typically early morning + logoff/lock events). File-level jobs take ~33 minutes on average; full image jobs take ~53 minutes.

  4. Local Storage: For clients with on-prem image backups, data is written to the WNP's local repository (NAS or direct-attached storage). Local retention is typically 14 weeks.

  5. Cloud Replication: A second job (or backup copy job on the WNP) replicates data to Wasabi S3 via the VSPC Cloud Connect gateway (wemvemasp02). Cloud copies are stored in per-machine immutable repositories with GFS retention (typically 1 year).

  6. Monitoring: VSPC monitors job status and raises alarms. These propagate to Datto RMM, which creates Autotask tickets automatically. The operations team triages alerts daily.

  7. Remediation: Failed jobs are investigated — common fixes include repository rescans (VBM desync), DNS resolution (Wasabi connectivity), and agent reinstalls.

  8. Restores: End users or client contacts request restores via Autotask ticket. Wulf engineers recover files from local or cloud repositories as needed.

  9. Documentation: Each client's backup configuration is documented in IT Glue with method, schedule, retention, protected devices, and restore approval contacts.


9. Key Observations & Recommendations

Strengths

  • Comprehensive coverage: 688 active workstation agents across 59 clients with centralized VSPC management
  • Immutable offsite copies: Wasabi S3 object lock protects against ransomware and accidental deletion
  • Automated monitoring pipeline: VSPC → Datto RMM → Autotask ensures no backup failure goes unnoticed
  • High resolution rate: 99.5%+ of backup tickets are resolved, indicating effective operational processes
  • Standardized policies: Two primary policies (WulfStdWRKSTFiles, WulfStdWRKSTFilesWasabi) cover 75% of jobs

Areas for Attention

  • VBM desync failures (28 jobs): The most common failure mode. Consider automating repository rescans or upgrading to Veeam v13 which improved metadata handling.
  • Wasabi DNS failures (14 jobs): Client-side DNS resolution issues. May benefit from secondary DNS or direct-IP fallback configuration.
  • 49 jobs with "None" status: Jobs that have never run — likely newly deployed agents awaiting first execution or stale configurations.
  • 84 total failed jobs (6.9%): While the alert pipeline catches these, the raw failure rate could be improved by addressing the top two root causes (VBM desync + Wasabi DNS).

This document was compiled from live VSPC data synced to Pulse, Autotask ticket history, and IT Glue backup documentation. Data reflects the state as of April 10, 2026.