wulf-pulse/dev/pulse-morning-summary-architecture.md

Architecture: Pulse Morning NOC Summary

Context

Wulf Consulting (MSP) needs a daily morning summary sent to management showing overnight activity and open issues across client infrastructure. Pulse (Node.js/TypeScript) is the internal web app with existing API integrations to Zabbix, PSA, RMM, Veeam, Zoom, and MS Graph. It already has a job scheduler (bull/agenda/node-cron). Apprise is running on the monitoring server for notification routing.

Architecture Overview

┌──────────────┐   schedule    ┌──────────────────────┐
│  Job Scheduler├─────────────►│  Summary Aggregator  │
│  (existing)  │   6:30 AM    │  Service             │
└──────────────┘               └──────┬───────────────┘
                                      │ parallel queries
                    ┌─────────────────┼─────────────────┐
                    ▼                 ▼                 ▼
              ┌──────────┐    ┌──────────┐     ┌──────────┐
              │ Zabbix   │    │  PSA     │     │  Veeam   │
              │ API      │    │  API     │     │  API     │
              └────┬─────┘    └────┬─────┘     └────┬─────┘
                   │               │                │
                   └───────┬───────┘                │
                           ▼                        │
                  ┌─────────────────┐               │
                  │  Data Correlator │◄──────────────┘
                  │  & Formatter     │
                  └────────┬────────┘
                           │
               ┌───────────┼───────────┐
               ▼           ▼           ▼
         ┌──────────┐ ┌────────┐ ┌─────────┐
         │  Teams   │ │ Apprise│ │  Pulse  │
         │  MS Graph│ │ (ntfy, │ │  DB +   │
         │  Adaptive│ │  email)│ │  Widget │
         │  Card    │ │        │ │         │
         └──────────┘ └────────┘ └─────────┘

Hybrid Notification Strategy

Channel	Method	Why
Teams Adaptive Card	Direct via MS Graph	Rich formatting, action buttons, inline rendering — can't get this through Apprise
Email (HTML)	Via Apprise	Apprise handles SMTP config, templating is simpler for email
ntfy	Via Apprise	Already configured, Apprise knows the topic/auth
Pulse Dashboard	Direct DB write	Store the summary as a record, render as a widget

---

Data Model

1. Summary Aggregator Service

Single service class/module: MorningSummaryService

interface MorningSummary {
  generatedAt: Date;
  reportWindow: { from: Date; to: Date };  // e.g. 6pm → 6:30am

  openProblems: Problem[];       // currently active — most important section
  resolvedOvernight: Problem[];  // resolved during the window
  backupFailures: BackupJob[];   // from Veeam API
  unmatchedAlerts: Problem[];    // Zabbix problems with no PSA ticket (action needed!)

  stats: {
    totalIncidents: number;
    resolved: number;
    stillOpen: number;
    mttrMinutes: number;         // mean time to resolve (overnight only)
    clientsAffected: string[];
  };
}

interface Problem {
  host: string;
  client: string;               // from Zabbix host group "Clients/..."
  triggerName: string;           // "Host Unreachable", "High Packet Loss", etc.
  severity: string;
  startedAt: Date;
  resolvedAt?: Date;
  duration: string;
  psaTicketId?: string;         // correlated from PSA
  psaTicketUrl?: string;
  acknowledged: boolean;
}

interface BackupJob {
  client: string;
  server: string;
  jobName: string;
  status: string;
  lastRun: Date;
  message?: string;
}

2. Zabbix API Queries Needed

Open problems:

{
  "method": "problem.get",
  "params": {
    "recent": true,
    "sortfield": ["eventid"],
    "sortorder": "DESC"
  }
}

Resolved overnight:

{
  "method": "event.get",
  "params": {
    "source": 0,
    "object": 0,
    "value": 0,
    "time_from": "<6pm_yesterday_unix>",
    "time_to": "<6:30am_today_unix>",
    "selectHosts": ["name"],
    "selectRelatedObject": ["description"]
  }
}

Host → Client mapping:

{
  "method": "host.get",
  "params": {
    "selectHostGroups": ["name"]
  }
}

Filter groups starting with Clients/ to determine the client name per host.

Host groups available for reference:

• Clients/Kuhn's Quality Foods, Clients/ADM Signs, Clients/Seubert and Associates, etc. (50+ clients)
• ISP/Comcast Cable Communications, LLC, ISP/AT&T Enterprises, LLC, etc. (25+ ISPs)

3. PSA Correlation

For each open Zabbix problem, query the PSA for matching tickets:

• Match on hostname or client name + date range
• Flag any Zabbix problem that has NO corresponding PSA ticket — these are "unmatched" and should be highlighted as needing attention

4. Veeam Backup Failures

Query Veeam API for jobs that ran overnight with status != Success. Include in a separate section of the summary.

---

Teams Adaptive Card Design

{
  "type": "AdaptiveCard",
  "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
  "version": "1.4",
  "body": [
    {
      "type": "TextBlock",
      "text": "☀️ Morning NOC Summary — Mar 12, 2026",
      "weight": "bolder",
      "size": "large"
    },
    {
      "type": "ColumnSet",
      "columns": [
        {
          "type": "Column",
          "width": "auto",
          "items": [
            { "type": "TextBlock", "text": "3", "size": "extraLarge", "color": "attention", "weight": "bolder" },
            { "type": "TextBlock", "text": "Open", "spacing": "none" }
          ]
        },
        {
          "type": "Column",
          "width": "auto",
          "items": [
            { "type": "TextBlock", "text": "5", "size": "extraLarge", "color": "good", "weight": "bolder" },
            { "type": "TextBlock", "text": "Resolved", "spacing": "none" }
          ]
        },
        {
          "type": "Column",
          "width": "auto",
          "items": [
            { "type": "TextBlock", "text": "18m", "size": "extraLarge", "weight": "bolder" },
            { "type": "TextBlock", "text": "Avg MTTR", "spacing": "none" }
          ]
        }
      ]
    },
    {
      "type": "Container",
      "style": "attention",
      "bleed": true,
      "items": [
        { "type": "TextBlock", "text": "🔴 OPEN ISSUES", "weight": "bolder", "spacing": "small" },
        {
          "type": "FactSet",
          "facts": [
            { "title": "Kuhn's / FW-01", "value": "Host Unreachable — 4h 12m" },
            { "title": "ADM / SW-Core", "value": "High Packet Loss — 2h 5m" },
            { "title": "Seubert / DC-01", "value": "⚠️ Backup Failed — 6h (no ticket!)" }
          ]
        }
      ]
    },
    {
      "type": "Container",
      "style": "good",
      "bleed": true,
      "items": [
        { "type": "TextBlock", "text": "🟢 RESOLVED OVERNIGHT", "weight": "bolder", "spacing": "small" },
        {
          "type": "FactSet",
          "facts": [
            { "title": "LWHResTest", "value": "Host Unreachable — resolved in 15m" },
            { "title": "Brodaks / RTR-01", "value": "Slow Response — resolved in 22m" },
            { "title": "+3 more", "value": "All auto-resolved" }
          ]
        }
      ]
    }
  ],
  "actions": [
    { "type": "Action.OpenUrl", "title": "Open Zabbix", "url": "https://zabbix.wulfconsulting.cloud" },
    { "type": "Action.OpenUrl", "title": "Open Pulse", "url": "https://pulse.wulfconsulting.cloud" },
    { "type": "Action.OpenUrl", "title": "Ack All Open", "url": "https://pulse.wulfconsulting.cloud/ack-all" }
  ]
}

Post via MS Graph:

POST https://graph.microsoft.com/v1.0/teams/{teamId}/channels/{channelId}/messages
Content-Type: application/json

{
  "body": {
    "contentType": "html",
    "content": "<attachment id=\"card\"></attachment>"
  },
  "attachments": [{
    "id": "card",
    "contentType": "application/vnd.microsoft.card.adaptive",
    "content": "<adaptive card JSON string>"
  }]
}

---

User Preference System

interface NotificationPreferences {
  userId: string;
  morningSummary: {
    enabled: boolean;
    channels: ('teams' | 'email' | 'ntfy' | 'pulse')[];
    schedule: string;           // cron expression, default "30 6 * * 1-5"
    timezone: string;           // "America/New_York"
    includeBackups: boolean;
    includeResolvedDetail: boolean;  // some execs just want open issues
    severityFilter: number;         // minimum severity to include (default: 2/Warning)
  };
}

Store in Pulse's existing user/settings table. Expose in Pulse UI as a settings page.

---

Job Scheduler Integration

Use the existing scheduler to run the aggregation:

// Register the job
scheduler.register('morning-summary', '30 6 * * 1-5', async () => {
  const users = await getUsersWithMorningSummaryEnabled();

  // Aggregate once (shared data)
  const summary = await morningSummaryService.aggregate();

  // Store for Pulse dashboard widget
  await morningSummaryService.persist(summary);

  // Deliver per user preferences
  for (const user of users) {
    const prefs = user.notificationPreferences.morningSummary;

    if (prefs.channels.includes('teams'))
      await teamsService.postAdaptiveCard(user, summary);

    if (prefs.channels.includes('email'))
      await appriseService.sendEmail(user, formatEmailHtml(summary));

    if (prefs.channels.includes('ntfy'))
      await appriseService.sendNtfy(formatNtfySummary(summary));
  }
});

Aggregator Pseudocode

class MorningSummaryService {
  async aggregate(): Promise<MorningSummary> {
    const now = new Date();
    const windowStart = yesterday6pm(now);
    const windowEnd = now;

    // Run all API calls in parallel
    const [zabbixOpen, zabbixResolved, backups, hostGroups] = await Promise.all([
      this.zabbixApi.getOpenProblems(),
      this.zabbixApi.getResolvedEvents(windowStart, windowEnd),
      this.veeamApi.getOvernightJobs(windowStart, windowEnd),
      this.zabbixApi.getHostGroupMapping()  // cache this, changes rarely
    ]);

    // Map hosts → client names via host groups
    const clientMap = buildClientMap(hostGroups);

    // Correlate with PSA tickets
    const openWithTickets = await this.psaApi.correlateProblems(zabbixOpen, clientMap);

    // Find unmatched (no PSA ticket)
    const unmatched = openWithTickets.filter(p => !p.psaTicketId);

    // Calculate stats
    const mttr = calculateMTTR(zabbixResolved);

    return {
      generatedAt: now,
      reportWindow: { from: windowStart, to: windowEnd },
      openProblems: openWithTickets,
      resolvedOvernight: zabbixResolved.map(e => enrichWithClient(e, clientMap)),
      backupFailures: backups.filter(b => b.status !== 'Success'),
      unmatchedAlerts: unmatched,
      stats: {
        totalIncidents: zabbixOpen.length + zabbixResolved.length,
        resolved: zabbixResolved.length,
        stillOpen: zabbixOpen.length,
        mttrMinutes: mttr,
        clientsAffected: [...new Set(openWithTickets.map(p => p.client))]
      }
    };
  }
}

---

Key Recommendations

1. Aggregate once, deliver many — Don't re-query Zabbix/PSA/Veeam per user. Run the aggregation once, then fan out to each user's preferred channels.

2. "Unmatched alerts" section is the killer feature — Highlighting Zabbix problems that have no PSA ticket is what will make management love this. It shows gaps in the process.

3. Teams adaptive card direct, everything else through Apprise — Adaptive cards need the MS Graph payload format which Apprise can't produce. For simpler formats (email body, ntfy text), Apprise handles routing without Pulse needing SMTP config.

4. Pulse dashboard widget — Persist each summary to the DB. Show the latest on the Pulse home screen so anyone can check it anytime, not just at 6:30 AM.

5. Weekend mode — Consider a different schedule or suppression for weekends. The cron 30 6 * * 1-5 only fires Mon-Fri. But you may want a Monday morning summary that covers the full weekend window (Friday 6pm → Monday 6:30am).

6. Escalation hint — If any problem has been open > 4 hours with no PSA ticket and no acknowledgement, flag it red in the card with "Needs Attention" — gives management actionable signal, not just data.

7. Cache the host → client mapping — The Clients/ host group mapping rarely changes. Cache it in Pulse (refresh every few hours) to avoid an API call on every summary run.

---

Zabbix Host Group Reference

Your host groups are well-organized for this feature:

Client groups (50+): Clients/Kuhn's Quality Foods, Clients/ADM Signs, Clients/Seubert and Associates, Clients/Brodaks, etc.

ISP groups (25+): ISP/Comcast, ISP/AT&T, ISP/Armstrong, ISP/Bigleaf, etc.

The ISP grouping can be used for a future enhancement: "ISP Outage Detection" — if 3+ hosts on the same ISP go down simultaneously, flag it as a likely ISP outage rather than individual site problems.

---

ntfy Summary Format (via Apprise)

For the condensed ntfy version:

☀️ Morning Summary — Mar 12

🔴 3 Open
 · Host Unreachable — Kuhn's / FW-01 (4h)
 · High Packet Loss — ADM / SW-Core (2h)
 · Backup Failed — Seubert / DC-01 (6h)

🟢 5 Resolved overnight (avg 18m)

⚠️ 1 issue with no ticket

---

Email HTML Format (via Apprise)

Use a clean responsive HTML template with:

• Header with date and stats (open/resolved/MTTR)
• Red-bordered table for open issues
• Green-bordered table for resolved
• Yellow callout box for unmatched alerts
• Footer with links to Zabbix and Pulse

Keep it mobile-friendly — management reads email on phones.