wulf-pulse/dev/pulse-morning-summary-architecture.md

# Architecture: Pulse Morning NOC Summary

## Context
Wulf Consulting (MSP) needs a daily morning summary sent to management showing overnight activity
and open issues across client infrastructure. Pulse (Node.js/TypeScript) is the internal web app
with existing API integrations to Zabbix, PSA, RMM, Veeam, Zoom, and MS Graph. It already has a
job scheduler (bull/agenda/node-cron). Apprise is running on the monitoring server for notification routing.

## Architecture Overview

```
┌──────────────┐   schedule    ┌──────────────────────┐
│  Job Scheduler├─────────────►│  Summary Aggregator  │
│  (existing)  │   6:30 AM    │  Service             │
└──────────────┘               └──────┬───────────────┘
                                      │ parallel queries
                    ┌─────────────────┼─────────────────┐
                    ▼                 ▼                 ▼
              ┌──────────┐    ┌──────────┐     ┌──────────┐
              │ Zabbix   │    │  PSA     │     │  Veeam   │
              │ API      │    │  API     │     │  API     │
              └────┬─────┘    └────┬─────┘     └────┬─────┘
                   │               │                │
                   └───────┬───────┘                │
                           ▼                        │
                  ┌─────────────────┐               │
                  │  Data Correlator │◄──────────────┘
                  │  & Formatter     │
                  └────────┬────────┘
                           │
               ┌───────────┼───────────┐
               ▼           ▼           ▼
         ┌──────────┐ ┌────────┐ ┌─────────┐
         │  Teams   │ │ Apprise│ │  Pulse  │
         │  MS Graph│ │ (ntfy, │ │  DB +   │
         │  Adaptive│ │  email)│ │  Widget │
         │  Card    │ │        │ │         │
         └──────────┘ └────────┘ └─────────┘
```

## Hybrid Notification Strategy

| Channel | Method | Why |
|---------|--------|-----|
| **Teams Adaptive Card** | Direct via MS Graph | Rich formatting, action buttons, inline rendering — can't get this through Apprise |
| **Email (HTML)** | Via Apprise | Apprise handles SMTP config, templating is simpler for email |
| **ntfy** | Via Apprise | Already configured, Apprise knows the topic/auth |
| **Pulse Dashboard** | Direct DB write | Store the summary as a record, render as a widget |

---

## Data Model

### 1. Summary Aggregator Service

Single service class/module: `MorningSummaryService`

```typescript
interface MorningSummary {
  generatedAt: Date;
  reportWindow: { from: Date; to: Date };  // e.g. 6pm → 6:30am

  openProblems: Problem[];       // currently active — most important section
  resolvedOvernight: Problem[];  // resolved during the window
  backupFailures: BackupJob[];   // from Veeam API
  unmatchedAlerts: Problem[];    // Zabbix problems with no PSA ticket (action needed!)

  stats: {
    totalIncidents: number;
    resolved: number;
    stillOpen: number;
    mttrMinutes: number;         // mean time to resolve (overnight only)
    clientsAffected: string[];
  };
}

interface Problem {
  host: string;
  client: string;               // from Zabbix host group "Clients/..."
  triggerName: string;           // "Host Unreachable", "High Packet Loss", etc.
  severity: string;
  startedAt: Date;
  resolvedAt?: Date;
  duration: string;
  psaTicketId?: string;         // correlated from PSA
  psaTicketUrl?: string;
  acknowledged: boolean;
}

interface BackupJob {
  client: string;
  server: string;
  jobName: string;
  status: string;
  lastRun: Date;
  message?: string;
}
```

### 2. Zabbix API Queries Needed

**Open problems:**
```json
{
  "method": "problem.get",
  "params": {
    "recent": true,
    "sortfield": ["eventid"],
    "sortorder": "DESC"
  }
}
```

**Resolved overnight:**
```json
{
  "method": "event.get",
  "params": {
    "source": 0,
    "object": 0,
    "value": 0,
    "time_from": "<6pm_yesterday_unix>",
    "time_to": "<6:30am_today_unix>",
    "selectHosts": ["name"],
    "selectRelatedObject": ["description"]
  }
}
```

**Host → Client mapping:**
```json
{
  "method": "host.get",
  "params": {
    "selectHostGroups": ["name"]
  }
}
```
Filter groups starting with `Clients/` to determine the client name per host.

**Host groups available for reference:**
- `Clients/Kuhn's Quality Foods`, `Clients/ADM Signs`, `Clients/Seubert and Associates`, etc. (50+ clients)
- `ISP/Comcast Cable Communications, LLC`, `ISP/AT&T Enterprises, LLC`, etc. (25+ ISPs)

### 3. PSA Correlation

For each open Zabbix problem, query the PSA for matching tickets:
- Match on hostname or client name + date range
- Flag any Zabbix problem that has NO corresponding PSA ticket — these are "unmatched"
  and should be highlighted as needing attention

### 4. Veeam Backup Failures

Query Veeam API for jobs that ran overnight with status != Success.
Include in a separate section of the summary.

---

## Teams Adaptive Card Design

```json
{
  "type": "AdaptiveCard",
  "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
  "version": "1.4",
  "body": [
    {
      "type": "TextBlock",
      "text": "☀️ Morning NOC Summary — Mar 12, 2026",
      "weight": "bolder",
      "size": "large"
    },
    {
      "type": "ColumnSet",
      "columns": [
        {
          "type": "Column",
          "width": "auto",
          "items": [
            { "type": "TextBlock", "text": "3", "size": "extraLarge", "color": "attention", "weight": "bolder" },
            { "type": "TextBlock", "text": "Open", "spacing": "none" }
          ]
        },
        {
          "type": "Column",
          "width": "auto",
          "items": [
            { "type": "TextBlock", "text": "5", "size": "extraLarge", "color": "good", "weight": "bolder" },
            { "type": "TextBlock", "text": "Resolved", "spacing": "none" }
          ]
        },
        {
          "type": "Column",
          "width": "auto",
          "items": [
            { "type": "TextBlock", "text": "18m", "size": "extraLarge", "weight": "bolder" },
            { "type": "TextBlock", "text": "Avg MTTR", "spacing": "none" }
          ]
        }
      ]
    },
    {
      "type": "Container",
      "style": "attention",
      "bleed": true,
      "items": [
        { "type": "TextBlock", "text": "🔴 OPEN ISSUES", "weight": "bolder", "spacing": "small" },
        {
          "type": "FactSet",
          "facts": [
            { "title": "Kuhn's / FW-01", "value": "Host Unreachable — 4h 12m" },
            { "title": "ADM / SW-Core", "value": "High Packet Loss — 2h 5m" },
            { "title": "Seubert / DC-01", "value": "⚠️ Backup Failed — 6h (no ticket!)" }
          ]
        }
      ]
    },
    {
      "type": "Container",
      "style": "good",
      "bleed": true,
      "items": [
        { "type": "TextBlock", "text": "🟢 RESOLVED OVERNIGHT", "weight": "bolder", "spacing": "small" },
        {
          "type": "FactSet",
          "facts": [
            { "title": "LWHResTest", "value": "Host Unreachable — resolved in 15m" },
            { "title": "Brodaks / RTR-01", "value": "Slow Response — resolved in 22m" },
            { "title": "+3 more", "value": "All auto-resolved" }
          ]
        }
      ]
    }
  ],
  "actions": [
    { "type": "Action.OpenUrl", "title": "Open Zabbix", "url": "https://zabbix.wulfconsulting.cloud" },
    { "type": "Action.OpenUrl", "title": "Open Pulse", "url": "https://pulse.wulfconsulting.cloud" },
    { "type": "Action.OpenUrl", "title": "Ack All Open", "url": "https://pulse.wulfconsulting.cloud/ack-all" }
  ]
}
```

Post via MS Graph:
```
POST https://graph.microsoft.com/v1.0/teams/{teamId}/channels/{channelId}/messages
Content-Type: application/json

{
  "body": {
    "contentType": "html",
    "content": "<attachment id=\"card\"></attachment>"
  },
  "attachments": [{
    "id": "card",
    "contentType": "application/vnd.microsoft.card.adaptive",
    "content": "<adaptive card JSON string>"
  }]
}
```

---

## User Preference System

```typescript
interface NotificationPreferences {
  userId: string;
  morningSummary: {
    enabled: boolean;
    channels: ('teams' | 'email' | 'ntfy' | 'pulse')[];
    schedule: string;           // cron expression, default "30 6 * * 1-5"
    timezone: string;           // "America/New_York"
    includeBackups: boolean;
    includeResolvedDetail: boolean;  // some execs just want open issues
    severityFilter: number;         // minimum severity to include (default: 2/Warning)
  };
}
```

Store in Pulse's existing user/settings table. Expose in Pulse UI as a settings page.

---

## Job Scheduler Integration

Use the existing scheduler to run the aggregation:

```typescript
// Register the job
scheduler.register('morning-summary', '30 6 * * 1-5', async () => {
  const users = await getUsersWithMorningSummaryEnabled();

  // Aggregate once (shared data)
  const summary = await morningSummaryService.aggregate();

  // Store for Pulse dashboard widget
  await morningSummaryService.persist(summary);

  // Deliver per user preferences
  for (const user of users) {
    const prefs = user.notificationPreferences.morningSummary;

    if (prefs.channels.includes('teams'))
      await teamsService.postAdaptiveCard(user, summary);

    if (prefs.channels.includes('email'))
      await appriseService.sendEmail(user, formatEmailHtml(summary));

    if (prefs.channels.includes('ntfy'))
      await appriseService.sendNtfy(formatNtfySummary(summary));
  }
});
```

### Aggregator Pseudocode

```typescript
class MorningSummaryService {
  async aggregate(): Promise<MorningSummary> {
    const now = new Date();
    const windowStart = yesterday6pm(now);
    const windowEnd = now;

    // Run all API calls in parallel
    const [zabbixOpen, zabbixResolved, backups, hostGroups] = await Promise.all([
      this.zabbixApi.getOpenProblems(),
      this.zabbixApi.getResolvedEvents(windowStart, windowEnd),
      this.veeamApi.getOvernightJobs(windowStart, windowEnd),
      this.zabbixApi.getHostGroupMapping()  // cache this, changes rarely
    ]);

    // Map hosts → client names via host groups
    const clientMap = buildClientMap(hostGroups);

    // Correlate with PSA tickets
    const openWithTickets = await this.psaApi.correlateProblems(zabbixOpen, clientMap);

    // Find unmatched (no PSA ticket)
    const unmatched = openWithTickets.filter(p => !p.psaTicketId);

    // Calculate stats
    const mttr = calculateMTTR(zabbixResolved);

    return {
      generatedAt: now,
      reportWindow: { from: windowStart, to: windowEnd },
      openProblems: openWithTickets,
      resolvedOvernight: zabbixResolved.map(e => enrichWithClient(e, clientMap)),
      backupFailures: backups.filter(b => b.status !== 'Success'),
      unmatchedAlerts: unmatched,
      stats: {
        totalIncidents: zabbixOpen.length + zabbixResolved.length,
        resolved: zabbixResolved.length,
        stillOpen: zabbixOpen.length,
        mttrMinutes: mttr,
        clientsAffected: [...new Set(openWithTickets.map(p => p.client))]
      }
    };
  }
}
```

---

## Key Recommendations

1. **Aggregate once, deliver many** — Don't re-query Zabbix/PSA/Veeam per user. Run the
   aggregation once, then fan out to each user's preferred channels.

2. **"Unmatched alerts" section is the killer feature** — Highlighting Zabbix problems that
   have no PSA ticket is what will make management love this. It shows gaps in the process.

3. **Teams adaptive card direct, everything else through Apprise** — Adaptive cards need
   the MS Graph payload format which Apprise can't produce. For simpler formats (email body,
   ntfy text), Apprise handles routing without Pulse needing SMTP config.

4. **Pulse dashboard widget** — Persist each summary to the DB. Show the latest on the
   Pulse home screen so anyone can check it anytime, not just at 6:30 AM.

5. **Weekend mode** — Consider a different schedule or suppression for weekends. The cron
   `30 6 * * 1-5` only fires Mon-Fri. But you may want a Monday morning summary that covers
   the full weekend window (Friday 6pm → Monday 6:30am).

6. **Escalation hint** — If any problem has been open > 4 hours with no PSA ticket and no
   acknowledgement, flag it red in the card with "Needs Attention" — gives management
   actionable signal, not just data.

7. **Cache the host → client mapping** — The `Clients/` host group mapping rarely changes.
   Cache it in Pulse (refresh every few hours) to avoid an API call on every summary run.

---

## Zabbix Host Group Reference

Your host groups are well-organized for this feature:

**Client groups (50+):** `Clients/Kuhn's Quality Foods`, `Clients/ADM Signs`, `Clients/Seubert and Associates`, `Clients/Brodaks`, etc.

**ISP groups (25+):** `ISP/Comcast`, `ISP/AT&T`, `ISP/Armstrong`, `ISP/Bigleaf`, etc.

The ISP grouping can be used for a future enhancement: "ISP Outage Detection" — if 3+ hosts
on the same ISP go down simultaneously, flag it as a likely ISP outage rather than individual
site problems.

---

## ntfy Summary Format (via Apprise)

For the condensed ntfy version:

```
☀️ Morning Summary — Mar 12

🔴 3 Open
 · Host Unreachable — Kuhn's / FW-01 (4h)
 · High Packet Loss — ADM / SW-Core (2h)
 · Backup Failed — Seubert / DC-01 (6h)

🟢 5 Resolved overnight (avg 18m)

⚠️ 1 issue with no ticket
```

---

## Email HTML Format (via Apprise)

Use a clean responsive HTML template with:
- Header with date and stats (open/resolved/MTTR)
- Red-bordered table for open issues
- Green-bordered table for resolved
- Yellow callout box for unmatched alerts
- Footer with links to Zabbix and Pulse

Keep it mobile-friendly — management reads email on phones.