feat: Morning NOC Summary adaptive card for Teams
- Add MorningSummaryService with Zabbix aggregation and adaptive card builder - Add webhook delivery system with Teams incoming webhooks - Add admin UI at /admin/morning-summary for webhook/config management - Add API routes: /send, /test, /webhooks, /webhooks/[id], /config, /history - Register morning-summary cron job in SyncScheduler (Mon-Fri 6:30 AM) - Add outages_only filter (Unavailable triggers only) - Fix host resolution: use getTriggerEnabledHosts to exclude disabled hosts - Fix resolved events: event.get value:1 scoped to window with r_eventid filter - Remove emojis from fact rows and section headers in card - Remove Open Zabbix button (duplicate of View Problems) - Add migrations: morning_summary_config + morning_summaries tables - Add outages_only column to morning_summary_config
This commit is contained in:
parent
19605f82aa
commit
c518eefdb2
61 changed files with 11236 additions and 237 deletions
435
dev/pulse-morning-summary-architecture.md
Normal file
435
dev/pulse-morning-summary-architecture.md
Normal file
|
|
@ -0,0 +1,435 @@
|
|||
# Architecture: Pulse Morning NOC Summary
|
||||
|
||||
## Context
|
||||
Wulf Consulting (MSP) needs a daily morning summary sent to management showing overnight activity
|
||||
and open issues across client infrastructure. Pulse (Node.js/TypeScript) is the internal web app
|
||||
with existing API integrations to Zabbix, PSA, RMM, Veeam, Zoom, and MS Graph. It already has a
|
||||
job scheduler (bull/agenda/node-cron). Apprise is running on the monitoring server for notification routing.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌──────────────┐ schedule ┌──────────────────────┐
|
||||
│ Job Scheduler├─────────────►│ Summary Aggregator │
|
||||
│ (existing) │ 6:30 AM │ Service │
|
||||
└──────────────┘ └──────┬───────────────┘
|
||||
│ parallel queries
|
||||
┌─────────────────┼─────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│ Zabbix │ │ PSA │ │ Veeam │
|
||||
│ API │ │ API │ │ API │
|
||||
└────┬─────┘ └────┬─────┘ └────┬─────┘
|
||||
│ │ │
|
||||
└───────┬───────┘ │
|
||||
▼ │
|
||||
┌─────────────────┐ │
|
||||
│ Data Correlator │◄──────────────┘
|
||||
│ & Formatter │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌───────────┼───────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌────────┐ ┌─────────┐
|
||||
│ Teams │ │ Apprise│ │ Pulse │
|
||||
│ MS Graph│ │ (ntfy, │ │ DB + │
|
||||
│ Adaptive│ │ email)│ │ Widget │
|
||||
│ Card │ │ │ │ │
|
||||
└──────────┘ └────────┘ └─────────┘
|
||||
```
|
||||
|
||||
## Hybrid Notification Strategy
|
||||
|
||||
| Channel | Method | Why |
|
||||
|---------|--------|-----|
|
||||
| **Teams Adaptive Card** | Direct via MS Graph | Rich formatting, action buttons, inline rendering — can't get this through Apprise |
|
||||
| **Email (HTML)** | Via Apprise | Apprise handles SMTP config, templating is simpler for email |
|
||||
| **ntfy** | Via Apprise | Already configured, Apprise knows the topic/auth |
|
||||
| **Pulse Dashboard** | Direct DB write | Store the summary as a record, render as a widget |
|
||||
|
||||
---
|
||||
|
||||
## Data Model
|
||||
|
||||
### 1. Summary Aggregator Service
|
||||
|
||||
Single service class/module: `MorningSummaryService`
|
||||
|
||||
```typescript
|
||||
interface MorningSummary {
|
||||
generatedAt: Date;
|
||||
reportWindow: { from: Date; to: Date }; // e.g. 6pm → 6:30am
|
||||
|
||||
openProblems: Problem[]; // currently active — most important section
|
||||
resolvedOvernight: Problem[]; // resolved during the window
|
||||
backupFailures: BackupJob[]; // from Veeam API
|
||||
unmatchedAlerts: Problem[]; // Zabbix problems with no PSA ticket (action needed!)
|
||||
|
||||
stats: {
|
||||
totalIncidents: number;
|
||||
resolved: number;
|
||||
stillOpen: number;
|
||||
mttrMinutes: number; // mean time to resolve (overnight only)
|
||||
clientsAffected: string[];
|
||||
};
|
||||
}
|
||||
|
||||
interface Problem {
|
||||
host: string;
|
||||
client: string; // from Zabbix host group "Clients/..."
|
||||
triggerName: string; // "Host Unreachable", "High Packet Loss", etc.
|
||||
severity: string;
|
||||
startedAt: Date;
|
||||
resolvedAt?: Date;
|
||||
duration: string;
|
||||
psaTicketId?: string; // correlated from PSA
|
||||
psaTicketUrl?: string;
|
||||
acknowledged: boolean;
|
||||
}
|
||||
|
||||
interface BackupJob {
|
||||
client: string;
|
||||
server: string;
|
||||
jobName: string;
|
||||
status: string;
|
||||
lastRun: Date;
|
||||
message?: string;
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Zabbix API Queries Needed
|
||||
|
||||
**Open problems:**
|
||||
```json
|
||||
{
|
||||
"method": "problem.get",
|
||||
"params": {
|
||||
"recent": true,
|
||||
"sortfield": ["eventid"],
|
||||
"sortorder": "DESC"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Resolved overnight:**
|
||||
```json
|
||||
{
|
||||
"method": "event.get",
|
||||
"params": {
|
||||
"source": 0,
|
||||
"object": 0,
|
||||
"value": 0,
|
||||
"time_from": "<6pm_yesterday_unix>",
|
||||
"time_to": "<6:30am_today_unix>",
|
||||
"selectHosts": ["name"],
|
||||
"selectRelatedObject": ["description"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Host → Client mapping:**
|
||||
```json
|
||||
{
|
||||
"method": "host.get",
|
||||
"params": {
|
||||
"selectHostGroups": ["name"]
|
||||
}
|
||||
}
|
||||
```
|
||||
Filter groups starting with `Clients/` to determine the client name per host.
|
||||
|
||||
**Host groups available for reference:**
|
||||
- `Clients/Kuhn's Quality Foods`, `Clients/ADM Signs`, `Clients/Seubert and Associates`, etc. (50+ clients)
|
||||
- `ISP/Comcast Cable Communications, LLC`, `ISP/AT&T Enterprises, LLC`, etc. (25+ ISPs)
|
||||
|
||||
### 3. PSA Correlation
|
||||
|
||||
For each open Zabbix problem, query the PSA for matching tickets:
|
||||
- Match on hostname or client name + date range
|
||||
- Flag any Zabbix problem that has NO corresponding PSA ticket — these are "unmatched"
|
||||
and should be highlighted as needing attention
|
||||
|
||||
### 4. Veeam Backup Failures
|
||||
|
||||
Query Veeam API for jobs that ran overnight with status != Success.
|
||||
Include in a separate section of the summary.
|
||||
|
||||
---
|
||||
|
||||
## Teams Adaptive Card Design
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "AdaptiveCard",
|
||||
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
|
||||
"version": "1.4",
|
||||
"body": [
|
||||
{
|
||||
"type": "TextBlock",
|
||||
"text": "☀️ Morning NOC Summary — Mar 12, 2026",
|
||||
"weight": "bolder",
|
||||
"size": "large"
|
||||
},
|
||||
{
|
||||
"type": "ColumnSet",
|
||||
"columns": [
|
||||
{
|
||||
"type": "Column",
|
||||
"width": "auto",
|
||||
"items": [
|
||||
{ "type": "TextBlock", "text": "3", "size": "extraLarge", "color": "attention", "weight": "bolder" },
|
||||
{ "type": "TextBlock", "text": "Open", "spacing": "none" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "Column",
|
||||
"width": "auto",
|
||||
"items": [
|
||||
{ "type": "TextBlock", "text": "5", "size": "extraLarge", "color": "good", "weight": "bolder" },
|
||||
{ "type": "TextBlock", "text": "Resolved", "spacing": "none" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "Column",
|
||||
"width": "auto",
|
||||
"items": [
|
||||
{ "type": "TextBlock", "text": "18m", "size": "extraLarge", "weight": "bolder" },
|
||||
{ "type": "TextBlock", "text": "Avg MTTR", "spacing": "none" }
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "Container",
|
||||
"style": "attention",
|
||||
"bleed": true,
|
||||
"items": [
|
||||
{ "type": "TextBlock", "text": "🔴 OPEN ISSUES", "weight": "bolder", "spacing": "small" },
|
||||
{
|
||||
"type": "FactSet",
|
||||
"facts": [
|
||||
{ "title": "Kuhn's / FW-01", "value": "Host Unreachable — 4h 12m" },
|
||||
{ "title": "ADM / SW-Core", "value": "High Packet Loss — 2h 5m" },
|
||||
{ "title": "Seubert / DC-01", "value": "⚠️ Backup Failed — 6h (no ticket!)" }
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "Container",
|
||||
"style": "good",
|
||||
"bleed": true,
|
||||
"items": [
|
||||
{ "type": "TextBlock", "text": "🟢 RESOLVED OVERNIGHT", "weight": "bolder", "spacing": "small" },
|
||||
{
|
||||
"type": "FactSet",
|
||||
"facts": [
|
||||
{ "title": "LWHResTest", "value": "Host Unreachable — resolved in 15m" },
|
||||
{ "title": "Brodaks / RTR-01", "value": "Slow Response — resolved in 22m" },
|
||||
{ "title": "+3 more", "value": "All auto-resolved" }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"actions": [
|
||||
{ "type": "Action.OpenUrl", "title": "Open Zabbix", "url": "https://zabbix.wulfconsulting.cloud" },
|
||||
{ "type": "Action.OpenUrl", "title": "Open Pulse", "url": "https://pulse.wulfconsulting.cloud" },
|
||||
{ "type": "Action.OpenUrl", "title": "Ack All Open", "url": "https://pulse.wulfconsulting.cloud/ack-all" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Post via MS Graph:
|
||||
```
|
||||
POST https://graph.microsoft.com/v1.0/teams/{teamId}/channels/{channelId}/messages
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"body": {
|
||||
"contentType": "html",
|
||||
"content": "<attachment id=\"card\"></attachment>"
|
||||
},
|
||||
"attachments": [{
|
||||
"id": "card",
|
||||
"contentType": "application/vnd.microsoft.card.adaptive",
|
||||
"content": "<adaptive card JSON string>"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## User Preference System
|
||||
|
||||
```typescript
|
||||
interface NotificationPreferences {
|
||||
userId: string;
|
||||
morningSummary: {
|
||||
enabled: boolean;
|
||||
channels: ('teams' | 'email' | 'ntfy' | 'pulse')[];
|
||||
schedule: string; // cron expression, default "30 6 * * 1-5"
|
||||
timezone: string; // "America/New_York"
|
||||
includeBackups: boolean;
|
||||
includeResolvedDetail: boolean; // some execs just want open issues
|
||||
severityFilter: number; // minimum severity to include (default: 2/Warning)
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
Store in Pulse's existing user/settings table. Expose in Pulse UI as a settings page.
|
||||
|
||||
---
|
||||
|
||||
## Job Scheduler Integration
|
||||
|
||||
Use the existing scheduler to run the aggregation:
|
||||
|
||||
```typescript
|
||||
// Register the job
|
||||
scheduler.register('morning-summary', '30 6 * * 1-5', async () => {
|
||||
const users = await getUsersWithMorningSummaryEnabled();
|
||||
|
||||
// Aggregate once (shared data)
|
||||
const summary = await morningSummaryService.aggregate();
|
||||
|
||||
// Store for Pulse dashboard widget
|
||||
await morningSummaryService.persist(summary);
|
||||
|
||||
// Deliver per user preferences
|
||||
for (const user of users) {
|
||||
const prefs = user.notificationPreferences.morningSummary;
|
||||
|
||||
if (prefs.channels.includes('teams'))
|
||||
await teamsService.postAdaptiveCard(user, summary);
|
||||
|
||||
if (prefs.channels.includes('email'))
|
||||
await appriseService.sendEmail(user, formatEmailHtml(summary));
|
||||
|
||||
if (prefs.channels.includes('ntfy'))
|
||||
await appriseService.sendNtfy(formatNtfySummary(summary));
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### Aggregator Pseudocode
|
||||
|
||||
```typescript
|
||||
class MorningSummaryService {
|
||||
async aggregate(): Promise<MorningSummary> {
|
||||
const now = new Date();
|
||||
const windowStart = yesterday6pm(now);
|
||||
const windowEnd = now;
|
||||
|
||||
// Run all API calls in parallel
|
||||
const [zabbixOpen, zabbixResolved, backups, hostGroups] = await Promise.all([
|
||||
this.zabbixApi.getOpenProblems(),
|
||||
this.zabbixApi.getResolvedEvents(windowStart, windowEnd),
|
||||
this.veeamApi.getOvernightJobs(windowStart, windowEnd),
|
||||
this.zabbixApi.getHostGroupMapping() // cache this, changes rarely
|
||||
]);
|
||||
|
||||
// Map hosts → client names via host groups
|
||||
const clientMap = buildClientMap(hostGroups);
|
||||
|
||||
// Correlate with PSA tickets
|
||||
const openWithTickets = await this.psaApi.correlateProblems(zabbixOpen, clientMap);
|
||||
|
||||
// Find unmatched (no PSA ticket)
|
||||
const unmatched = openWithTickets.filter(p => !p.psaTicketId);
|
||||
|
||||
// Calculate stats
|
||||
const mttr = calculateMTTR(zabbixResolved);
|
||||
|
||||
return {
|
||||
generatedAt: now,
|
||||
reportWindow: { from: windowStart, to: windowEnd },
|
||||
openProblems: openWithTickets,
|
||||
resolvedOvernight: zabbixResolved.map(e => enrichWithClient(e, clientMap)),
|
||||
backupFailures: backups.filter(b => b.status !== 'Success'),
|
||||
unmatchedAlerts: unmatched,
|
||||
stats: {
|
||||
totalIncidents: zabbixOpen.length + zabbixResolved.length,
|
||||
resolved: zabbixResolved.length,
|
||||
stillOpen: zabbixOpen.length,
|
||||
mttrMinutes: mttr,
|
||||
clientsAffected: [...new Set(openWithTickets.map(p => p.client))]
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Recommendations
|
||||
|
||||
1. **Aggregate once, deliver many** — Don't re-query Zabbix/PSA/Veeam per user. Run the
|
||||
aggregation once, then fan out to each user's preferred channels.
|
||||
|
||||
2. **"Unmatched alerts" section is the killer feature** — Highlighting Zabbix problems that
|
||||
have no PSA ticket is what will make management love this. It shows gaps in the process.
|
||||
|
||||
3. **Teams adaptive card direct, everything else through Apprise** — Adaptive cards need
|
||||
the MS Graph payload format which Apprise can't produce. For simpler formats (email body,
|
||||
ntfy text), Apprise handles routing without Pulse needing SMTP config.
|
||||
|
||||
4. **Pulse dashboard widget** — Persist each summary to the DB. Show the latest on the
|
||||
Pulse home screen so anyone can check it anytime, not just at 6:30 AM.
|
||||
|
||||
5. **Weekend mode** — Consider a different schedule or suppression for weekends. The cron
|
||||
`30 6 * * 1-5` only fires Mon-Fri. But you may want a Monday morning summary that covers
|
||||
the full weekend window (Friday 6pm → Monday 6:30am).
|
||||
|
||||
6. **Escalation hint** — If any problem has been open > 4 hours with no PSA ticket and no
|
||||
acknowledgement, flag it red in the card with "Needs Attention" — gives management
|
||||
actionable signal, not just data.
|
||||
|
||||
7. **Cache the host → client mapping** — The `Clients/` host group mapping rarely changes.
|
||||
Cache it in Pulse (refresh every few hours) to avoid an API call on every summary run.
|
||||
|
||||
---
|
||||
|
||||
## Zabbix Host Group Reference
|
||||
|
||||
Your host groups are well-organized for this feature:
|
||||
|
||||
**Client groups (50+):** `Clients/Kuhn's Quality Foods`, `Clients/ADM Signs`, `Clients/Seubert and Associates`, `Clients/Brodaks`, etc.
|
||||
|
||||
**ISP groups (25+):** `ISP/Comcast`, `ISP/AT&T`, `ISP/Armstrong`, `ISP/Bigleaf`, etc.
|
||||
|
||||
The ISP grouping can be used for a future enhancement: "ISP Outage Detection" — if 3+ hosts
|
||||
on the same ISP go down simultaneously, flag it as a likely ISP outage rather than individual
|
||||
site problems.
|
||||
|
||||
---
|
||||
|
||||
## ntfy Summary Format (via Apprise)
|
||||
|
||||
For the condensed ntfy version:
|
||||
|
||||
```
|
||||
☀️ Morning Summary — Mar 12
|
||||
|
||||
🔴 3 Open
|
||||
· Host Unreachable — Kuhn's / FW-01 (4h)
|
||||
· High Packet Loss — ADM / SW-Core (2h)
|
||||
· Backup Failed — Seubert / DC-01 (6h)
|
||||
|
||||
🟢 5 Resolved overnight (avg 18m)
|
||||
|
||||
⚠️ 1 issue with no ticket
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Email HTML Format (via Apprise)
|
||||
|
||||
Use a clean responsive HTML template with:
|
||||
- Header with date and stats (open/resolved/MTTR)
|
||||
- Red-bordered table for open issues
|
||||
- Green-bordered table for resolved
|
||||
- Yellow callout box for unmatched alerts
|
||||
- Footer with links to Zabbix and Pulse
|
||||
|
||||
Keep it mobile-friendly — management reads email on phones.
|
||||
Loading…
Add table
Add a link
Reference in a new issue