wulf-pulse/dev/windsurf-zabbix-development-guide.md

# Windsurf + Sonnet Development Guide — Zabbix Monitoring System

> **For:** AI-assisted development in Windsurf using Claude Sonnet 4.6
> **Organization:** Wulf Consulting (MSP)
> **Last updated:** 2026-03-11

---

## Table of Contents

1. [System Overview](#1-system-overview)
2. [Infrastructure & Network Topology](#2-infrastructure--network-topology)
3. [Zabbix API Reference](#3-zabbix-api-reference)
4. [Host Organization & Data Model](#4-host-organization--data-model)
5. [Notification Pipeline](#5-notification-pipeline)
6. [Trigger Naming Conventions](#6-trigger-naming-conventions)
7. [Grafana Integration](#7-grafana-integration)
8. [Pulse Integration Architecture](#8-pulse-integration-architecture)
9. [Development Patterns & Gotchas](#9-development-patterns--gotchas)
10. [API Cookbook](#10-api-cookbook)

---

## 1. System Overview

Wulf Consulting is an MSP managing ~46 client sites. The monitoring stack runs on a single
Ubuntu 24.04 server (public IP: 209.166.162.245) with all services containerized in Docker.

### Stack Components

| Service | Container | Image | Purpose |
|---------|-----------|-------|---------|
| **Zabbix Server** | `zabbix-server` | `zabbix/zabbix-server-pgsql:alpine-7.4-latest` | Core monitoring engine |
| **Zabbix Frontend** | `zabbix-frontend` | `zabbix/zabbix-web-nginx-pgsql:alpine-7.4-latest` | Web UI + API endpoint |
| **PostgreSQL** | `zabbix-postgres` | `postgres:17-alpine` | Zabbix database |
| **Grafana** | `grafana` | `grafana/grafana:latest` | Dashboards + Zabbix plugin |
| **ntfy** | `ntfy` | `binwiederhier/ntfy:latest` | Push notification server |
| **Apprise** | `apprise-api` | `caronc/apprise:latest` | Multi-channel notification router |
| **Authentik** | `authentik` | `ghcr.io/goauthentik/server:2025.8.1` | SSO/identity provider |
| **Newt** | `newt` | `fosrl/newt` | Pangolin tunnel agent |

### External Access

All services are exposed through **Pangolin** (reverse proxy/tunnel), not direct port mappings.

| Service | External URL |
|---------|-------------|
| Zabbix | `https://zabbix.wulfconsulting.cloud` |
| Grafana | *(via Pangolin — check Pangolin config for exact URL)* |

SSO is handled by Authentik with SAML integration to Zabbix.

### Software Versions

- **Zabbix:** 7.4.7
- **PostgreSQL:** 17 (Alpine)
- **Grafana:** Latest (with `alexanderzobnin-zabbix-app` 6.2.1)
- **Host OS:** Ubuntu 24.04.4 LTS

---

## 2. Infrastructure & Network Topology

### Docker Networks

```
┌─────────────────────────────────────────────────────────────┐
│                     pangolin (172.18.0.0/16)                │
│  ┌─────────────┐ ┌──────────┐ ┌──────┐ ┌───────────────┐   │
│  │zabbix-front │ │ grafana  │ │ ntfy │ │ apprise-api   │   │
│  │ 172.18.0.4  │ │172.18.0.3│ │.0.7  │ │  172.18.0.5   │   │
│  └──────┬──────┘ └────┬─────┘ └──────┘ └───────────────┘   │
│         │              │                                     │
│  ┌──────┴──────┐       │        ┌──────────┐ ┌───────────┐  │
│  │zabbix-server│       │        │authentik │ │   newt    │  │
│  │ 172.18.0.8  │       │        │172.18.0.6│ │172.18.0.2 │  │
│  └──────┬──────┘       │        └──────────┘ └───────────┘  │
│         │              │                                     │
└─────────┼──────────────┼─────────────────────────────────────┘
          │              │
┌─────────┼──────────────┼──────────────────────┐
│         │   zabbix_zabbix_internal (172.19.0.0/16)         │
│  ┌──────┴──────┐ ┌────┴─────┐ ┌──────────────┐│
│  │zabbix-server│ │ grafana  │ │zabbix-frontend││
│  │ 172.19.0.4  │ │172.19.0.3│ │ 172.19.0.5   ││
│  └─────────────┘ └──────────┘ └──────────────┘│
│                                ┌──────────────┐│
│                                │zabbix-postgres││
│                                │ 172.19.0.2   ││
│                                └──────────────┘│
└─────────────────────────────────────────────────┘
```

### Key Connectivity Facts

- **Zabbix API (internal):** `http://zabbix-frontend:8080/api_jsonrpc.php` — accessible from `zabbix-server`, `grafana`, and anything on `zabbix_zabbix_internal`
- **Zabbix API (external):** `https://zabbix.wulfconsulting.cloud/api_jsonrpc.php` — via Pangolin
- **ntfy (internal):** `http://ntfy:80` — accessible from `zabbix-server` and `zabbix-frontend` via `pangolin` network
- **Apprise API:** `http://apprise-api:8000` (also `0.0.0.0:8000` on host)
- **Zabbix agent port:** `10051` (mapped to host `0.0.0.0:10051`)
- **Grafana** has NO external port mapping — accessed only through Pangolin

### Docker Compose Location

All stack definitions: `/opt/stacks/zabbix/compose.yml`

Environment variables: `/opt/stacks/zabbix/.env`

---

## 3. Zabbix API Reference

### Authentication

Zabbix 7.4 uses Bearer token authentication:

```
Authorization: Bearer <api_token>
```

API tokens are generated in Zabbix UI: **User Settings → API tokens**

> **Important:** The `apiinfo.version` method MUST be called WITHOUT the Authorization header.
> All other methods require it.

### Base Request Format

```json
{
  "jsonrpc": "2.0",
  "method": "<method_name>",
  "params": { ... },
  "id": 1
}
```

### API Endpoint

| Context | URL |
|---------|-----|
| From `zabbix-server` or `grafana` container | `http://zabbix-frontend:8080/api_jsonrpc.php` |
| From the Docker host | `docker exec zabbix-frontend curl -s -X POST http://localhost:8080/api_jsonrpc.php ...` |
| From external / Pulse | `https://zabbix.wulfconsulting.cloud/api_jsonrpc.php` |

### Key API Methods Used

| Method | Purpose | Notes |
|--------|---------|-------|
| `host.get` | List hosts, get groups/templates | Use `selectHostGroups`, `selectParentTemplates` |
| `hostgroup.get` | List host groups | Filter by `Clients/` or `ISP/` prefix |
| `trigger.get` | Get triggers, active problems | `only_true: true` for currently-firing |
| `problem.get` | Get current problems | `recent: true` for unresolved |
| `event.get` | Get events (problems + recoveries) | Use `time_from`/`time_to`, `value: 0` for recovery |
| `mediatype.get` | Get notification media types | `selectMessageTemplates` for templates |
| `mediatype.update` | Update webhook scripts/templates | Include full `script`, `parameters`, `message_templates` |
| `action.get` | Get trigger actions | `selectOperations`, `selectRecoveryOperations` |
| `action.update` | Update actions | Add `recovery_operations` |
| `trigger.update` | Rename triggers, update descriptions | Use `description` (name) and `comments` (description text) |
| `user.get` | Get users and their media | `selectMedias` for notification channels |
| `template.get` | Get templates and their triggers | `selectTriggers` |

### Zabbix API Quirks (Zabbix 7.4)

- **Trigger `description` = trigger name** (not the description text). The description text is in `comments`.
- **Status codes:** `0` = enabled, `1` = disabled for both media types and actions.
- **`templateid` on host triggers:** If `0`, the trigger was created directly on the host (not inherited from a template). Non-zero = the parent trigger ID on the template.
- **Severity levels:** `0`=Not classified, `1`=Information, `2`=Warning, `3`=Average, `4`=High, `5`=Disaster
- **`{EVENT.VALUE}`:** `1` = problem, `0` = OK/recovery
- **`{EVENT.NSEVERITY}`:** Numeric severity (0-5)
- **Webhook scripts** run in Zabbix's built-in Duktape JavaScript engine (ES5 only — no `let`, `const`, arrow functions, template literals, `Array.find`, etc.)

---

## 4. Host Organization & Data Model

### Host Group Hierarchy

Every monitored host is assigned to multiple groups:

```
Host: "Kuhn's Quality Foods"
├── Clients/Kuhn's Quality Foods    ← client identity
├── ISP/Zito Media, L.P.           ← internet provider
└── Datto RMM Sites                ← RMM platform category
```

This triple-grouping enables:
- **Per-client dashboards** — filter by `Clients/` group
- **ISP outage detection** — if 3+ hosts on same ISP go down = likely ISP issue
- **RMM correlation** — cross-reference Zabbix with Datto RMM agent status

### Client Groups (46)

| ID | Name |
|----|------|
| 84 | Clients/3 Rivers Express |
| 90 | Clients/ABC Fire Extinguisher Inc |
| 49 | Clients/ADM Signs |
| 40 | Clients/Advanced Masonry |
| 50 | Clients/Alabek Commercial Roofing Corp. |
| 33 | Clients/All Saints Catholic Church |
| 51 | Clients/Attica Hub/Seneca Publishing |
| 42 | Clients/Bella Diamond LLC |
| 28 | Clients/Blake Dentistry |
| 85 | Clients/Bosak Eyecare & Optical |
| 55 | Clients/Bridges Health Partners Services, LLC |
| 37 | Clients/Brodaks |
| 45 | Clients/Broker's Settlement Services, Inc. |
| 43 | Clients/Brooks Diamonds |
| 57 | Clients/Buffalo Glass Block |
| 60 | Clients/CUMI America |
| 61 | Clients/Chartiers Animal Hospital Ltd |
| 58 | Clients/Cincinnati Glass Block |
| 56 | Clients/Clista Electric Inc. |
| 35 | Clients/ConnecTel, Inc. |
| 26 | Clients/Finn Chiropractic Group |
| 59 | Clients/Frew Plumbing, Heating, & Air |
| 53 | Clients/Greco Gas |
| 65 | Clients/Heart Prints Center for Early Education |
| 47 | Clients/Hergenroeder, Rega, Ewing, & Kennedy, LLC |
| 69 | Clients/Hynes Industries |
| 48 | Clients/Insurance Restoration Consultants, Inc. |
| 32 | Clients/Kuhn's Quality Foods |
| 44 | Clients/Loss Prevention Services |
| 80 | Clients/MDS Energy Development, LLC |
| 88 | Clients/Marsico Financial Group, LLC |
| 62 | Clients/Nordmann Roofing |
| 68 | Clients/North Eastern Uniforms & Equipment Inc |
| 38 | Clients/POH+W Architects |
| 75 | Clients/Penn Energy Resources |
| 86 | Clients/Pittsburgh Financial Consultants |
| 87 | Clients/Premier Automation Holdings, Inc. |
| 23 | Clients/Seubert and Associates |
| 91 | Clients/Superior Distributing Co |
| 30 | Clients/TK Plastics Company, Inc. |
| 70 | Clients/Thoroughbred Construction Group |
| 66 | Clients/Thrasher Group, Inc. |
| 71 | Clients/Universal Plastics |
| 73 | Clients/Universal Plastics Latrobe |
| 54 | Clients/V-Systems |
| 76 | Clients/Vorteq Coil Finishers |

### ISP Groups (24)

| ID | Name |
|----|------|
| 25 | ISP/AT&T Enterprises, LLC |
| 27 | ISP/Armstrong |
| 24 | ISP/Bigleaf Networks, Inc. |
| 34 | ISP/Buckeye Cablevision, Inc. |
| 52 | ISP/Charter Communications Inc |
| 74 | ISP/Citizens Telecommunication Technologies, Inc |
| 67 | ISP/CityNet |
| 31 | ISP/Comcast Cable Communications, LLC |
| 46 | ISP/DQE Communications LLC |
| 63 | ISP/Expedient |
| 77 | ISP/Fidium |
| 89 | ISP/Frontier Communications of America, Inc. |
| 82 | ISP/GeoLinks |
| 78 | ISP/JACKSON ENERGY AUTHORITY |
| 39 | ISP/Level 3 Parent, LLC |
| 92 | ISP/Metalink Technologies, Inc. |
| 72 | ISP/OneCleveland |
| 64 | ISP/Space Exploration Technologies Corporation |
| 83 | ISP/UPMC |
| 79 | ISP/Ultimate Internet Access, Inc |
| 36 | ISP/Verizon Business |
| 81 | ISP/Wave Broadband |
| 41 | ISP/Windstream Communications LLC |
| 29 | ISP/Zito Media, L.P. |

### Other Groups

| ID | Name | Purpose |
|----|------|---------|
| 19 | Applications | Application-level monitoring |
| 20 | Databases | Database servers |
| 22 | Datto RMM Sites | Hosts also managed by Datto RMM |
| 5 | Discovered hosts | Auto-discovered hosts |
| 7 | Hypervisors | Virtualization hosts |
| 2 | Linux servers | Linux-based systems |
| 93 | Studio Imagine | Internal project |
| 6 | Virtual machines | VMs |
| 4 | Zabbix servers | Zabbix infrastructure |

### Host Pattern

All client-site hosts currently use the **ICMP Ping** template (templateid: 10564) for
basic up/down monitoring. A typical host:

```
Name:      "Kuhn's Quality Foods"
Status:    enabled
Groups:    [Clients/Kuhn's Quality Foods, ISP/Zito Media L.P., Datto RMM Sites]
Templates: [ICMP Ping]
```

**77 total hosts** across the system (including Zabbix infrastructure hosts).

### Global Macros

| Macro | Value |
|-------|-------|
| `{$SNMP_COMMUNITY}` | `public` |

---

## 5. Notification Pipeline

### Current Architecture

```
Zabbix Trigger Fires
       │
       ▼
Action: "Send ntfy to NOC" (actionid: 7)
  ├── Problem  → ntfy webhook (mediatypeid: 102)
  └── Recovery → ntfy webhook (same media type, different template)
       │
       ▼
ntfy Webhook Script (Duktape JS)
  ├── Determines problem vs recovery (EVENT.VALUE)
  ├── Sets 🔴 red_circle (problem) or 🟢 green_circle (recovery)
  ├── Sets ntfy priority (high for problems, low for recovery)
  ├── Adds Click URL → Zabbix event page
  ├── For problems: queries Zabbix API for other active issues on same client
  └── POSTs to ntfy → http://ntfy/noc-alerts
       │
       ▼
ntfy Push Notification → user's phone
```

### Action Configuration

**Action:** "Send ntfy to NOC" (actionid: 7)
- **Trigger condition:** Severity >= High (conditiontype 4, operator 5, value 4)
- **Operations:** Send message to user group "Executive" (usrgrpid: 14)
- **Recovery operations:** Notify all involved (operationtype: 11)
- **Uses default messages** (default_msg: 1) — pulls from media type templates

### Users & Media

| User | Role | ntfy Media |
|------|------|------------|
| Lorentz Hinrichsen (`lorentz@wulfconsulting.com`) | Super admin (3) | `noc-alerts` topic (enabled) |
| Tom Carlin (`tom@wulfconsulting.com`) | Super admin (3) | *(removed — was duplicate to same topic)* |
| Admin | Super admin (3) | *(none)* |
| guest | Guest (4) | *(none)* |

### ntfy Webhook Script (Current)

The webhook script runs inside Zabbix Server's Duktape JS engine. It:

1. Parses parameters from Zabbix macros
2. Determines if this is a problem or recovery event
3. For problems: makes 2 internal Zabbix API calls to find related active issues for the same client
4. Formats and sends the ntfy notification with appropriate tags, priority, and click URL

**Full script and parameters** are documented in `/opt/stacks/zabbix/notification-changes.md`

### Message Templates (Current)

| Event | ntfy Tag | Subject | Body |
|-------|----------|---------|------|
| Problem | 🔴 `red_circle` | `{HOST.NAME} — {EVENT.NAME}` | Trigger description + started time + duration + related issues |
| Recovery | 🟢 `green_circle` | `Resolved: {HOST.NAME} — {EVENT.NAME}` | "Host is back online." + downtime + restored time |
| Update | *(inherits)* | `Updated: {HOST.NAME} — {EVENT.NAME}` | Who updated + action + status |

### Notification Example

**Problem:**
```
🔴 Kuhn's Quality Foods — Host Unreachable

Host failed to respond to 3 consecutive ICMP ping
requests. The site may be offline, the network path
disrupted, or the device powered off.

Started:  2026.03.11 at 14:22:10
Duration: 5m 30s

── Other active issues (Kuhn's Quality Foods) ──
 · High Packet Loss — Kuhn's SW-Core
```

**Recovery:**
```
🟢 Resolved: Kuhn's Quality Foods — Host Unreachable

Host is back online.

Downtime: 22m 15s
Restored: 2026.03.11 at 14:44:25
```

### Available but Disabled Media Types

These exist in Zabbix but are disabled. Can be enabled if needed:
- Email, Email (HTML), Gmail, Office365, SMS
- Discord, Slack, MS Teams, MS Teams Workflow, Telegram
- Jira, Jira Service Management, ServiceNow, Zendesk, PagerDuty, Opsgenie
- Many others (40+ webhook integrations available)

---

## 6. Trigger Naming Conventions

### Design Principles (Established 2026-03-11)

Triggers were renamed from Zabbix defaults to be **executive-friendly**:

| Default Zabbix Name | Current Name | Severity |
|---------------------|-------------|----------|
| `ICMP Ping: Unavailable by ICMP ping` | **Host Unreachable** | High (4) |
| `ICMP Ping: High ICMP ping loss` | **High Packet Loss** | Warning (2) |
| `ICMP Ping: High ICMP ping response time` | **Slow Response Time** | Warning (2) |

### Naming Rules

1. **No technical jargon in trigger names** — use impact-based language
2. **Trigger descriptions (comments field)** contain the technical detail: what the check does, thresholds, possible causes
3. **Keep names short** — they appear in ntfy subjects, Teams cards, dashboards
4. **No host name in trigger name** — `{HOST.NAME}` is added by the notification template

### Cisco Triggers (Not Yet Renamed)

The Cisco Catalyst SNMP templates have their own ICMP triggers that still use the old naming:
- `Cisco Catalyst 3750V2-24FS: Unavailable by ICMP ping`
- `Cisco Catalyst 3750V2-24FS: High ICMP ping loss`
- etc.

These are on separate templates (not the "ICMP Ping" template) and have `templateid: 0` on host triggers. Renaming these requires updating each Cisco template individually.

---

## 7. Grafana Integration

### Plugin

- **alexanderzobnin-zabbix-app** v6.2.1 — connects directly to Zabbix API
- Datasource URL (internal): `http://zabbix-frontend:8080/api_jsonrpc.php`

### Planned: MSP Executive Status Board

Design goal: at-a-glance dashboard for management showing:

| Panel | Type | Data Source |
|-------|------|-------------|
| Current problems by severity | Stat panels | Zabbix problems |
| Problems per client | Bar gauge | Zabbix trigger groups |
| Active problem list | Table | Zabbix problems |
| SLA/uptime per client | Zabbix SLA panel | Zabbix SLA |
| Host status grid | Status map plugin | Zabbix hosts |

Filter by `Clients/` host groups. Use Grafana variables for client selection dropdown.

---

## 8. Pulse Integration Architecture

### Overview

**Pulse** is the internal Node.js/TypeScript web app with API access to:
- Zabbix (this system)
- PSA (ticketing)
- Datto RMM
- Veeam (backups)
- Microsoft Graph (Teams, email)
- Zoom
- Apprise (notification routing)

### Planned: Morning NOC Summary

Full architecture document: `/opt/stacks/zabbix/pulse-morning-summary-architecture.md`

**Key points:**
- Scheduled job at 6:30 AM weekdays
- Aggregates data from Zabbix + PSA + Veeam in parallel
- Sends Teams adaptive card (direct via MS Graph), email/ntfy (via Apprise)
- User-configurable delivery preferences
- Killer feature: **unmatched alerts** — Zabbix problems with no PSA ticket

### Zabbix API Queries for Pulse

**Get all open problems with host/client context:**
```typescript
// 1. Get open problems
const problems = await zabbixApi('problem.get', {
  recent: true,
  sortfield: ['eventid'],
  sortorder: 'DESC'
});

// 2. Get host → client mapping (cache this)
const hosts = await zabbixApi('host.get', {
  output: ['hostid', 'host', 'name'],
  selectHostGroups: ['groupid', 'name']
});

// 3. Build client map
const clientMap = {};
for (const host of hosts) {
  const clientGroup = host.hostgroups.find(g => g.name.startsWith('Clients/'));
  if (clientGroup) {
    clientMap[host.hostid] = clientGroup.name.replace('Clients/', '');
  }
}
```

**Get overnight resolved events:**
```typescript
const resolved = await zabbixApi('event.get', {
  source: 0,
  object: 0,
  value: 0,  // recovery events only
  time_from: Math.floor(yesterday6pm.getTime() / 1000),
  time_to: Math.floor(today630am.getTime() / 1000),
  selectHosts: ['name'],
  selectRelatedObject: ['description'],
  sortfield: ['clock'],
  sortorder: 'DESC'
});
```

---

## 9. Development Patterns & Gotchas

### Zabbix Webhook Script Constraints

The webhook script runs in **Duktape** (ES5 JavaScript engine):

**DO:**
```javascript
var x = 'hello';               // var only
for (var i = 0; i < arr.length; i++) { }  // classic for loops
JSON.parse(), JSON.stringify()  // available
new HttpRequest()               // Zabbix's HTTP client
btoa()                          // Base64 encoding available
```

**DON'T:**
```javascript
let x = 'hello';               // NO let/const
const y = () => {};             // NO arrow functions
`template ${literal}`;          // NO template literals
arr.find(x => x.id === 1);     // NO Array.find/includes/map
for (const x of arr) {}        // NO for...of
```

### HttpRequest in Webhooks

```javascript
var req = new HttpRequest();
req.addHeader('Content-Type: application/json');
req.addHeader('Authorization: Bearer TOKEN');

var resp = req.post(url, body);    // POST
var resp = req.put(url, body);     // PUT
var resp = req.get(url);           // GET

var status = req.getStatus();      // HTTP status code
```

Multiple HttpRequest instances CAN be created in the same script (used for the related-problems enrichment that calls Zabbix API before calling ntfy).

### API Update Patterns

When updating a media type, you must include the FULL array for `parameters` and `message_templates` — they're **replaced entirely**, not merged.

```python
# WRONG — this deletes all other parameters
mediatype.update({ parameters: [{"name": "new_param", "value": "x"}] })

# RIGHT — include ALL parameters
mediatype.update({ parameters: [
    {"name": "endpoint", "value": "http://ntfy/noc-alerts"},
    {"name": "username", "value": "monitoring"},
    # ... all existing params ...
    {"name": "new_param", "value": "x"}
] })
```

### Trigger Description vs Comments

This is confusing in the Zabbix API:
- `description` field = **the trigger name** (what you see in the UI)
- `comments` field = **the description text** (the explanation paragraph)

```json
{
  "triggerid": "23176",
  "description": "Host Unreachable",           // ← this is the NAME
  "comments": "Host failed to respond to..."   // ← this is the DESCRIPTION
}
```

### Host Group Filtering Pattern

To get the client name for a host:

```javascript
// Get host groups
var groups = hostData.hostgroups;
var clientName = null;
for (var i = 0; i < groups.length; i++) {
    if (groups[i].name.indexOf("Clients/") === 0) {
        clientName = groups[i].name.replace("Clients/", "");
        break;
    }
}
// clientName = "Kuhn's Quality Foods"
```

### Docker Exec for API Calls

Since the Zabbix frontend doesn't have ports mapped to the host, API calls from the host must go through `docker exec`:

```bash
docker exec zabbix-frontend curl -s -X POST \
  "http://localhost:8080/api_jsonrpc.php" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"jsonrpc":"2.0","method":"...","params":{...},"id":1}'
```

Or use Python for complex payloads (escaping JSON in bash is fragile):

```python
import json, subprocess

payload = json.dumps({...})
result = subprocess.run(
    ["docker", "exec", "zabbix-frontend", "curl", "-s", "-X", "POST",
     "http://localhost:8080/api_jsonrpc.php",
     "-H", "Content-Type: application/json",
     "-H", f"Authorization: Bearer {TOKEN}",
     "-d", payload],
    capture_output=True, text=True
)
data = json.loads(result.stdout)
```

### External API Access (from Pulse or other servers)

```typescript
const response = await fetch('https://zabbix.wulfconsulting.cloud/api_jsonrpc.php', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${ZABBIX_API_TOKEN}`
  },
  body: JSON.stringify({
    jsonrpc: '2.0',
    method: 'problem.get',
    params: { recent: true },
    id: 1
  })
});
```

---

## 10. API Cookbook

### Get All Active Problems

```json
{
  "method": "problem.get",
  "params": {
    "output": "extend",
    "recent": true,
    "sortfield": ["eventid"],
    "sortorder": "DESC"
  }
}
```

### Get Active Problems for a Specific Client

```json
{
  "method": "trigger.get",
  "params": {
    "output": ["triggerid", "description", "lastchange", "priority"],
    "groupids": ["32"],
    "only_true": true,
    "selectHosts": ["name"],
    "skipDependent": true
  }
}
```
*(groupid 32 = Clients/Kuhn's Quality Foods)*

### Get All Hosts with Client + ISP Mapping

```json
{
  "method": "host.get",
  "params": {
    "output": ["hostid", "host", "name", "status"],
    "selectHostGroups": ["groupid", "name"],
    "selectParentTemplates": ["templateid", "name"]
  }
}
```

### Get Recovery Events in a Time Window

```json
{
  "method": "event.get",
  "params": {
    "source": 0,
    "object": 0,
    "value": 0,
    "time_from": 1741647600,
    "time_to": 1741692600,
    "selectHosts": ["name"],
    "selectRelatedObject": ["description"],
    "sortfield": ["clock"],
    "sortorder": "DESC"
  }
}
```

### Update a Trigger Name + Description

```json
{
  "method": "trigger.update",
  "params": {
    "triggerid": "23176",
    "description": "Host Unreachable",
    "comments": "Host failed to respond to 3 consecutive ICMP ping requests."
  }
}
```

### Get Media Type with Full Details

```json
{
  "method": "mediatype.get",
  "params": {
    "output": "extend",
    "selectMessageTemplates": "extend",
    "mediatypeids": ["102"]
  }
}
```

### Test ntfy Notification Manually

```bash
docker exec zabbix-frontend curl -s -X PUT \
  -H "Authorization: Basic $(echo -n 'monitoring:GBigt1231#' | base64)" \
  -H "Title: Test Notification" \
  -H "Priority: 3" \
  -H "Tags: red_circle" \
  "http://ntfy/noc-alerts" \
  -d "This is a test notification from Zabbix"
```

---

## Related Documentation

- **Notification changes (before/after + restoration):** `/opt/stacks/zabbix/notification-changes.md`
- **Pulse morning summary architecture:** `/opt/stacks/zabbix/pulse-morning-summary-architecture.md`
- **Docker Compose:** `/opt/stacks/zabbix/compose.yml`
- **Zabbix SSO setup:** `/opt/stacks/zabbix/ENTRA-AUTHENTIK-ZABBIX-SSO.md`
- **Zabbix 7.4 API docs:** https://www.zabbix.com/documentation/7.4/en/manual/api