- Add MorningSummaryService with Zabbix aggregation and adaptive card builder - Add webhook delivery system with Teams incoming webhooks - Add admin UI at /admin/morning-summary for webhook/config management - Add API routes: /send, /test, /webhooks, /webhooks/[id], /config, /history - Register morning-summary cron job in SyncScheduler (Mon-Fri 6:30 AM) - Add outages_only filter (Unavailable triggers only) - Fix host resolution: use getTriggerEnabledHosts to exclude disabled hosts - Fix resolved events: event.get value:1 scoped to window with r_eventid filter - Remove emojis from fact rows and section headers in card - Remove Open Zabbix button (duplicate of View Problems) - Add migrations: morning_summary_config + morning_summaries tables - Add outages_only column to morning_summary_config
339 lines
18 KiB
Markdown
339 lines
18 KiB
Markdown
# PRD: Morning NOC Summary — Teams Adaptive Card
|
||
|
||
**Version:** 1.1
|
||
**Last updated:** 2026-03-10
|
||
|
||
---
|
||
|
||
## 1. Introduction / Overview
|
||
|
||
Wulf Consulting management needs a daily morning briefing on overnight infrastructure activity
|
||
across all ~46 client sites monitored by Zabbix. This feature automates that into a rich Teams
|
||
direct message delivered before the workday starts, with configurable recipients and optional
|
||
channel escalation actions directly from the card.
|
||
|
||
**Goal:** Post a Zabbix-sourced morning summary as a Teams Adaptive Card to one or more
|
||
configured Teams channel webhooks. The primary targets are the **On-Call** and/or
|
||
**Outages-Issues** channels in the **Technical** team. The card runs at 6:30 AM ET on
|
||
weekdays (with a Monday extended window option) and can be triggered on-demand from the
|
||
Pulse admin UI. No bot registration required — delivery is via incoming webhook URLs stored
|
||
in Pulse config.
|
||
|
||
---
|
||
|
||
## 2. Goals
|
||
|
||
1. Post a formatted Teams Adaptive Card to configured channel webhook(s) at 6:30 AM Mon–Fri.
|
||
2. Surface currently-open Zabbix problems (client name, trigger name, duration, severity).
|
||
3. Surface problems that resolved overnight (during the report window).
|
||
4. Show headline stats: open count, resolved count, avg MTTR, clients affected.
|
||
5. Highlight open problems active > 4 hours without acknowledgement ("Needs Attention" flag).
|
||
6. Support posting to multiple webhooks (e.g. both On-Call and Outages-Issues simultaneously).
|
||
7. Provide a **manual trigger** in the Pulse admin UI with per-webhook test send capability.
|
||
8. Support configurable weekend suppression; when Monday is the run day, extend the report
|
||
window to cover the full weekend (Friday 6 PM → Monday 6:30 AM).
|
||
9. Persist each generated summary to the Pulse database and show it as a dashboard widget.
|
||
|
||
---
|
||
|
||
## 3. User Stories
|
||
|
||
- **As Tom Carlin / Lorentz Hinrichsen**, I want to see the morning NOC summary posted to
|
||
the Teams channels I already monitor so I get the briefing without checking a separate tool.
|
||
- **As a Pulse admin**, I want to configure which channel webhooks receive the summary without
|
||
touching code or env files.
|
||
- **As a Pulse admin**, I want to send a test post to a specific webhook to verify formatting
|
||
before the scheduled run.
|
||
- **As a Pulse admin**, I want to toggle weekend suppression and Monday extended window.
|
||
- **As any Pulse user**, I want to see the most recent summary on the Pulse dashboard.
|
||
|
||
---
|
||
|
||
## 4. Functional Requirements
|
||
|
||
### 4.1 Data Collection (Zabbix only — v1)
|
||
|
||
1. The system **must** query `problem.get` (with `recent: true`, `selectHosts: extend`,
|
||
`selectAcknowledges: extend`, `severities: [2,3,4,5]`) to retrieve all currently open
|
||
problems at run time.
|
||
2. The system **must** query `event.get` with `value: 0` (recovery events) between
|
||
`time_from` = window start and `time_to` = run time to retrieve resolved problems.
|
||
3. The system **must** query `host.get` with `selectHostGroups: ['name']` once per run to
|
||
build a `hostid → { clientName, hostName }` map using groups prefixed `Clients/`.
|
||
4. The report window **must** be determined as follows:
|
||
- **Tuesday–Friday runs:** window start = previous calendar day at 18:00 ET.
|
||
- **Monday run (weekend mode on):** window start = the preceding Friday at 18:00 ET,
|
||
covering the full Sat + Sun + Mon pre-6:30 AM period.
|
||
- **Monday run (weekend mode off):** same as Tue–Fri (Sunday 18:00 ET → Monday 6:30 AM).
|
||
5. The system **must** calculate:
|
||
- **Open count** — number of active problems meeting severity filter.
|
||
- **Resolved count** — recovery events in the window.
|
||
- **Avg MTTR (minutes)** — mean of `(r_clock - clock)` for resolved events; `null` if none.
|
||
- **Clients affected** — deduplicated client names from open problems.
|
||
6. Any open problem where `clock` is > 4 hours before run time **and** `acknowledged` array
|
||
is empty **must** be flagged `needsAttention: true`.
|
||
|
||
### 4.2 Webhook Configuration — Pulse UI
|
||
|
||
7. The admin UI **must** provide a **Webhooks** section on `/admin/morning-summary` to manage
|
||
the list of Teams incoming webhook URLs that receive the summary.
|
||
8. The admin **must** be able to **add** a webhook by providing:
|
||
- A display label (e.g. `Technical / On-Call`)
|
||
- The incoming webhook URL
|
||
- An enabled toggle
|
||
9. The admin **must** be able to **remove** or **disable** any webhook.
|
||
10. Webhooks **must** be stored in Postgres (new table `morning_summary_webhooks`) with:
|
||
- `id` (serial PK)
|
||
- `label` (text) — human-friendly name
|
||
- `webhook_url` (text)
|
||
- `enabled` (boolean, default true)
|
||
- `last_delivered_at` (timestamptz, nullable)
|
||
- `last_status` (text, nullable) — `success` or `failed`
|
||
- `created_at` (timestamptz)
|
||
11. The admin UI **must** provide a **"Test"** button next to each webhook that posts the
|
||
current summary card to that URL only, independent of the scheduled run.
|
||
12. The test endpoint **must** be `POST /api/notifications/morning-summary/test` with
|
||
body `{ "webhookId": <id> }`.
|
||
13. The UI **must** display per-webhook last delivery status inline.
|
||
|
||
### 4.3 Teams Adaptive Card — Structure
|
||
|
||
14. The card **must** include a header `TextBlock`: `☀️ Morning NOC Summary — {date}`.
|
||
- On Monday with weekend mode: append ` (Weekend Coverage)` to the header.
|
||
15. The card **must** include a stat `ColumnSet`: Open (red), Resolved (green), Avg MTTR.
|
||
16. If there are **open problems**, the card **must** include a red-styled `Container`
|
||
labelled `🔴 OPEN ISSUES` with a `FactSet`:
|
||
- **Title:** `{ClientName} / {HostName}`
|
||
- **Value:** `{TriggerName} — {duration}` (+ ` ⚠️ Needs Attention` if flagged)
|
||
17. If there are **no open problems**, the card **must** show a green "✅ All Clear" container.
|
||
18. If there are **resolved overnight** events, the card **must** include a green-styled
|
||
`Container` labelled `🟢 RESOLVED OVERNIGHT`. If more than 5, show 5 and append
|
||
`+{N} more — all resolved`.
|
||
19. The card **must** include three action buttons:
|
||
- `Open Zabbix` → `https://zabbix.wulfconsulting.cloud`
|
||
- `Open Pulse` → `https://pulse.wulfconsulting.cloud`
|
||
- `View Problems` → `https://zabbix.wulfconsulting.cloud/zabbix.php?action=problem.view`
|
||
20. The card **must** use Adaptive Card schema version `1.4`.
|
||
21. The card payload **must** be wrapped in the Teams incoming webhook envelope:
|
||
`{ "type": "message", "attachments": [{ "contentType": "application/vnd.microsoft.card.adaptive", "content": <card> }] }`
|
||
|
||
### 4.4 Delivery via Incoming Webhooks
|
||
|
||
22. The system **must** POST the Adaptive Card envelope to each enabled webhook URL.
|
||
23. Delivery **must** be attempted for all enabled webhooks regardless of individual failures.
|
||
24. Each delivery result (HTTP status, error message) **must** be recorded in
|
||
`morning_summaries.delivery_status` keyed by webhook ID.
|
||
25. The send endpoint `POST /api/notifications/morning-summary/send` **must** accept an
|
||
optional `{ "webhookIds": [1, 2] }` body to target specific webhooks; if omitted, all
|
||
enabled webhooks are used.
|
||
26. Incoming webhook URLs are obtained from Teams: channel → connectors → "Incoming Webhook".
|
||
The admin pastes the URL into the Pulse webhook config UI.
|
||
|
||
### 4.5 MS Graph Usage (read-only)
|
||
|
||
27. MS Graph is **not** used for delivery in v1 (webhooks handle that).
|
||
28. MS Graph `getUsers()` **may** still be used in the admin UI to resolve display names when
|
||
configuring webhook labels, but is not required for the core send flow.
|
||
29. No `ChatMessage.Send` or `ChannelMessage.Send` Graph permissions are needed for v1.
|
||
|
||
### 4.6 Scheduling & Weekend Mode
|
||
|
||
33. The job **must** be registered in `SyncScheduler` with cron `30 6 * * 1-5`.
|
||
34. A `morning_summary_config` table (single-row settings) **must** store:
|
||
- `weekend_suppression` (boolean) — if true, skip Sat/Sun runs.
|
||
- `monday_extended_window` (boolean) — if true and today is Monday, set window start
|
||
to preceding Friday 18:00 ET.
|
||
- `severity_filter` (int, default 2) — minimum Zabbix severity to include.
|
||
- `updated_at` (timestamptz)
|
||
35. The admin UI **must** expose toggles for `weekend_suppression` and `monday_extended_window`
|
||
with a clear label: _"On Mondays, extend window to cover the full weekend (Fri 6 PM → Mon
|
||
6:30 AM)"_.
|
||
36. The admin UI **must** display the next scheduled run time calculated from the cron
|
||
expression and current settings.
|
||
|
||
### 4.7 Manual Trigger — Pulse UI
|
||
|
||
37. The admin UI at `/admin/morning-summary` **must** provide:
|
||
- **Schedule status:** next run time, last run time, last run result.
|
||
- **"Send Now"** button — fires `POST /api/notifications/morning-summary/send` to all
|
||
enabled webhooks immediately.
|
||
- **Per-webhook "Test"** buttons (see §4.2 req 11).
|
||
- **Weekend mode toggles** (see §4.6 req 35).
|
||
- **Webhooks section** — add/remove/enable-disable webhook entries (see §4.2).
|
||
- **Last card preview** — most recent `card_payload` as formatted JSON (collapsed by default).
|
||
38. The UI **must** show a toast on send and display per-webhook delivery results inline.
|
||
|
||
### 4.8 Dashboard Widget & Persistence
|
||
|
||
39. Each run (scheduled or manual/test) **must** persist to `morning_summaries`:
|
||
- `id` (serial PK)
|
||
- `generated_at` (timestamptz)
|
||
- `window_from` (timestamptz)
|
||
- `window_to` (timestamptz)
|
||
- `open_count` (int)
|
||
- `resolved_count` (int)
|
||
- `mttr_minutes` (int, nullable)
|
||
- `clients_affected` (text[])
|
||
- `is_weekend_window` (boolean)
|
||
- `card_payload` (jsonb)
|
||
- `delivery_status` (jsonb) — `{ [webhookId]: { success: bool, httpStatus?: number, error?: string } }`
|
||
40. The Pulse home/dashboard **must** display a "Morning Summary" widget with: open count,
|
||
resolved count, MTTR, window label, last generated timestamp, and a link to the admin
|
||
page.
|
||
|
||
---
|
||
|
||
## 5. Non-Goals (Out of Scope for v1)
|
||
|
||
- Veeam backup failure data (deferred to v2).
|
||
- PSA/Autotask ticket correlation and "unmatched alerts" (deferred to v2).
|
||
- Direct Teams DMs to individual users (requires bot registration — out of scope).
|
||
- Email or ntfy delivery channels (webhooks only for v1).
|
||
- ISP outage detection / grouping by ISP.
|
||
- Per-channel content customization (all webhooks receive the same card).
|
||
|
||
---
|
||
|
||
## 6. Design Considerations
|
||
|
||
### Adaptive Card Layout
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────┐
|
||
│ ☀️ Morning NOC Summary — Mon Mar 16 (Weekend) │
|
||
├──────────┬────────────┬────────────────────────── │
|
||
│ 3 Open │ 12 Resolved│ 22m Avg MTTR │
|
||
├──────────────────────────────────────────────────┤
|
||
│ 🔴 OPEN ISSUES │
|
||
│ Kuhn's / FW-01 Host Unreachable — 6h 12m ⚠️ │
|
||
│ ADM / SW-Core High Packet Loss — 2h 5m │
|
||
│ Seubert / DC-01 Host Unreachable — 1h 20m │
|
||
├──────────────────────────────────────────────────┤
|
||
│ 🟢 RESOLVED OVERNIGHT │
|
||
│ Brodaks / RTR-01 Slow Response — 22m │
|
||
│ +11 more — all resolved │
|
||
├──────────────────────────────────────────────────┤
|
||
│[Open Zabbix][Open Pulse][View Problems] │
|
||
│[📣 Notify On Call] [🚨 Post to Outages] │
|
||
└──────────────────────────────────────────────────┘
|
||
```
|
||
|
||
- `"style": "attention"` container → red in Teams (open issues).
|
||
- `"style": "good"` container → green in Teams (resolved).
|
||
- Duration: `Xh Ym` for ≥ 1 hour, `Xm` for < 1 hour.
|
||
- Escalation buttons open a Pulse URL in browser; Pulse posts to channel server-side
|
||
and returns a simple confirmation page.
|
||
|
||
### Recipient Management UI (on `/admin/morning-summary`)
|
||
|
||
```
|
||
Recipients
|
||
┌────────────────────────────┬──────────┬────────────────────┐
|
||
│ Name / Email │ Status │ Actions │
|
||
├────────────────────────────┼──────────┼────────────────────┤
|
||
│ Tom Carlin │ ✅ Last: │ [Test] [Remove] │
|
||
│ tom@wulfconsulting.com │ today │ │
|
||
├────────────────────────────┼──────────┼────────────────────┤
|
||
│ Lorentz Hinrichsen │ ✅ Last: │ [Test] [Remove] │
|
||
│ lorentz@wulfconsulting.com │ today │ │
|
||
└────────────────────────────┴──────────┴────────────────────┘
|
||
[+ Add Recipient ▾] ← searchable dropdown of tenant users
|
||
```
|
||
|
||
### Webhook Delivery Flow
|
||
|
||
```
|
||
// For each enabled webhook in morning_summary_webhooks:
|
||
POST {webhook_url}
|
||
Content-Type: application/json
|
||
|
||
{
|
||
"type": "message",
|
||
"attachments": [{
|
||
"contentType": "application/vnd.microsoft.card.adaptive",
|
||
"contentUrl": null,
|
||
"content": { ...adaptive card JSON... }
|
||
}]
|
||
}
|
||
```
|
||
|
||
To get a webhook URL in Teams: go to the target channel → **...** → Connectors →
|
||
**Incoming Webhook** → Configure → copy URL → paste into Pulse admin.
|
||
|
||
---
|
||
|
||
## 7. Technical Considerations
|
||
|
||
### Existing Infrastructure to Reuse
|
||
|
||
- **`ZabbixClient`** (`lib/services/zabbix-client.ts`) — add `getOpenProblems()` and
|
||
`getResolvedEvents(from: Date, to: Date)` methods.
|
||
- **`MsGraphClient`** — no changes needed for v1.
|
||
- **`SyncScheduler`** — add `'morning-summary'` sync type; register in `initialize()`.
|
||
- **Postgres** — three new tables: `morning_summaries`, `morning_summary_webhooks`,
|
||
`morning_summary_config`.
|
||
|
||
### New Files to Create
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `lib/services/morning-summary-service.ts` | Aggregation, card building, webhook delivery |
|
||
| `app/api/notifications/morning-summary/send/route.ts` | `POST` — send to all/selected webhooks |
|
||
| `app/api/notifications/morning-summary/test/route.ts` | `POST` — test send to single webhook |
|
||
| `app/api/notifications/morning-summary/webhooks/route.ts` | `GET`/`POST` — manage webhook list |
|
||
| `app/api/notifications/morning-summary/webhooks/[id]/route.ts` | `PUT`/`DELETE` — update/remove webhook |
|
||
| `app/api/notifications/morning-summary/config/route.ts` | `GET`/`PUT` — read/update settings |
|
||
| `app/admin/morning-summary/page.tsx` | Admin UI: webhooks, schedule, config, preview |
|
||
| `db/migrations/XXXX_morning_summary.sql` | All three new tables |
|
||
|
||
### Zabbix API Notes
|
||
|
||
- `problem.get` returns `acknowledged` as `"0"/"1"` strings and `clock` as Unix timestamp string.
|
||
- Use `selectAcknowledges: 'extend'` — problem is unacknowledged if the array is empty.
|
||
- `event.get` with `value: 0` returns recoveries; duration = `r_clock - clock` (both seconds).
|
||
- Enrich problems with host/client by cross-referencing `objectid` (triggerid) against a
|
||
trigger→host map built from `trigger.get` with `selectHosts: ['hostid', 'name']`, or
|
||
directly via `problem.get` with `selectSuppressionData` + a pre-built hostid→client map.
|
||
|
||
### MS Graph Permissions
|
||
|
||
No Graph permissions are required for v1 delivery. Delivery uses Teams incoming webhook URLs
|
||
(plain HTTPS POST — no auth needed beyond the secret embedded in the URL).
|
||
|
||
Existing granted permissions (`Chat.Create`, `Team.ReadBasic.All`, `Channel.ReadBasic.All`,
|
||
`ChatMessage.Send`, `ChannelMessage.Send`) are unused by this feature but can remain for
|
||
future use.
|
||
|
||
### Timezone
|
||
|
||
- Cron `30 6 * * 1-5` fires at 6:30 AM server time — ensure server is set to ET or adjust cron.
|
||
- Report window computed in ET using `Intl.DateTimeFormat` or `date-fns-tz`.
|
||
- Monday extended window: `startOfDay(subDays(monday, 3)) + 18h` = Friday 18:00 ET.
|
||
|
||
---
|
||
|
||
## 8. Success Metrics
|
||
|
||
- All configured recipients receive the DM at 6:30 AM on weekdays without manual action.
|
||
- Test send from the UI delivers to the selected individual within 30 seconds.
|
||
- Escalation button (On Call or Outages) posts to the correct Teams channel within 10 seconds
|
||
and the user sees a confirmation page.
|
||
- Monday card correctly shows the extended weekend window label and data.
|
||
- Dashboard widget reflects the most recent summary within 1 minute of generation.
|
||
- Zero unhandled errors surfaced to end users; all failures logged and shown in the admin UI.
|
||
|
||
---
|
||
|
||
## 9. Open Questions
|
||
|
||
1. ~~**MS Graph permissions**~~ — Not needed for v1 webhook delivery.
|
||
2. ~~**Team/channel names**~~ — Confirmed: team = `Technical`, channels = `On-Call` and `Outages-Issues`.
|
||
3. ~~**App membership in Technical team**~~ — Not needed; webhook URLs bypass Graph entirely.
|
||
4. **Error alerting** — If the 6:30 AM scheduled job fails entirely, how should that be
|
||
surfaced? (ntfy push, Pulse admin badge, or both?)
|
||
5. **Severity filter default** — Spec defaults to Warning (2) and above. Should
|
||
Information (1) problems be included? Confirm with Tom/Lorentz.
|
||
6. **Webhook URLs** — Lorentz needs to create incoming webhooks in the `On-Call` and
|
||
`Outages-Issues` channels (Teams channel → ... → Connectors → Incoming Webhook → Configure)
|
||
and paste the URLs into the Pulse admin UI once the feature is deployed.
|
||
|