wulf-pulse/dev/windsurf-zabbix-development-guide.md

Windsurf + Sonnet Development Guide — Zabbix Monitoring System

For: AI-assisted development in Windsurf using Claude Sonnet 4.6 Organization: Wulf Consulting (MSP) Last updated: 2026-03-11

---

Table of Contents

1. System Overview
2. Infrastructure & Network Topology
3. Zabbix API Reference
4. Host Organization & Data Model
5. Notification Pipeline
6. Trigger Naming Conventions
7. Grafana Integration
8. Pulse Integration Architecture
9. Development Patterns & Gotchas
10. API Cookbook

---

1. System Overview

Wulf Consulting is an MSP managing ~46 client sites. The monitoring stack runs on a single Ubuntu 24.04 server (public IP: 209.166.162.245) with all services containerized in Docker.

Stack Components

Service	Container	Image	Purpose
Zabbix Server	zabbix-server	zabbix/zabbix-server-pgsql:alpine-7.4-latest	Core monitoring engine
Zabbix Frontend	zabbix-frontend	zabbix/zabbix-web-nginx-pgsql:alpine-7.4-latest	Web UI + API endpoint
PostgreSQL	zabbix-postgres	postgres:17-alpine	Zabbix database
Grafana	grafana	grafana/grafana:latest	Dashboards + Zabbix plugin
ntfy	ntfy	binwiederhier/ntfy:latest	Push notification server
Apprise	apprise-api	caronc/apprise:latest	Multi-channel notification router
Authentik	authentik	ghcr.io/goauthentik/server:2025.8.1	SSO/identity provider
Newt	newt	fosrl/newt	Pangolin tunnel agent

External Access

All services are exposed through Pangolin (reverse proxy/tunnel), not direct port mappings.

Service	External URL
Zabbix	https://zabbix.wulfconsulting.cloud
Grafana	(via Pangolin — check Pangolin config for exact URL)

SSO is handled by Authentik with SAML integration to Zabbix.

Software Versions

• Zabbix: 7.4.7
• PostgreSQL: 17 (Alpine)
• Grafana: Latest (with alexanderzobnin-zabbix-app 6.2.1)
• Host OS: Ubuntu 24.04.4 LTS

---

2. Infrastructure & Network Topology

Docker Networks

┌─────────────────────────────────────────────────────────────┐
│                     pangolin (172.18.0.0/16)                │
│  ┌─────────────┐ ┌──────────┐ ┌──────┐ ┌───────────────┐   │
│  │zabbix-front │ │ grafana  │ │ ntfy │ │ apprise-api   │   │
│  │ 172.18.0.4  │ │172.18.0.3│ │.0.7  │ │  172.18.0.5   │   │
│  └──────┬──────┘ └────┬─────┘ └──────┘ └───────────────┘   │
│         │              │                                     │
│  ┌──────┴──────┐       │        ┌──────────┐ ┌───────────┐  │
│  │zabbix-server│       │        │authentik │ │   newt    │  │
│  │ 172.18.0.8  │       │        │172.18.0.6│ │172.18.0.2 │  │
│  └──────┬──────┘       │        └──────────┘ └───────────┘  │
│         │              │                                     │
└─────────┼──────────────┼─────────────────────────────────────┘
          │              │
┌─────────┼──────────────┼──────────────────────┐
│         │   zabbix_zabbix_internal (172.19.0.0/16)         │
│  ┌──────┴──────┐ ┌────┴─────┐ ┌──────────────┐│
│  │zabbix-server│ │ grafana  │ │zabbix-frontend││
│  │ 172.19.0.4  │ │172.19.0.3│ │ 172.19.0.5   ││
│  └─────────────┘ └──────────┘ └──────────────┘│
│                                ┌──────────────┐│
│                                │zabbix-postgres││
│                                │ 172.19.0.2   ││
│                                └──────────────┘│
└─────────────────────────────────────────────────┘

Key Connectivity Facts

• Zabbix API (internal): http://zabbix-frontend:8080/api_jsonrpc.php — accessible from zabbix-server, grafana, and anything on zabbix_zabbix_internal
• Zabbix API (external): https://zabbix.wulfconsulting.cloud/api_jsonrpc.php — via Pangolin
• ntfy (internal): http://ntfy:80 — accessible from zabbix-server and zabbix-frontend via pangolin network
• Apprise API: http://apprise-api:8000 (also 0.0.0.0:8000 on host)
• Zabbix agent port: 10051 (mapped to host 0.0.0.0:10051)
• Grafana has NO external port mapping — accessed only through Pangolin

Docker Compose Location

All stack definitions: /opt/stacks/zabbix/compose.yml

Environment variables: /opt/stacks/zabbix/.env

---

3. Zabbix API Reference

Authentication

Zabbix 7.4 uses Bearer token authentication:

Authorization: Bearer <api_token>

API tokens are generated in Zabbix UI: User Settings → API tokens

Important: The apiinfo.version method MUST be called WITHOUT the Authorization header. All other methods require it.

Base Request Format

{
  "jsonrpc": "2.0",
  "method": "<method_name>",
  "params": { ... },
  "id": 1
}

API Endpoint

Context	URL
From zabbix-server or grafana container	http://zabbix-frontend:8080/api_jsonrpc.php
From the Docker host	docker exec zabbix-frontend curl -s -X POST http://localhost:8080/api_jsonrpc.php ...
From external / Pulse	https://zabbix.wulfconsulting.cloud/api_jsonrpc.php

Key API Methods Used

Method	Purpose	Notes
host.get	List hosts, get groups/templates	Use selectHostGroups, selectParentTemplates
hostgroup.get	List host groups	Filter by Clients/ or ISP/ prefix
trigger.get	Get triggers, active problems	only_true: true for currently-firing
problem.get	Get current problems	recent: true for unresolved
event.get	Get events (problems + recoveries)	Use time_from/time_to, value: 0 for recovery
mediatype.get	Get notification media types	selectMessageTemplates for templates
mediatype.update	Update webhook scripts/templates	Include full script, parameters, message_templates
action.get	Get trigger actions	selectOperations, selectRecoveryOperations
action.update	Update actions	Add recovery_operations
trigger.update	Rename triggers, update descriptions	Use description (name) and comments (description text)
user.get	Get users and their media	selectMedias for notification channels
template.get	Get templates and their triggers	selectTriggers

Zabbix API Quirks (Zabbix 7.4)

• Trigger description = trigger name (not the description text). The description text is in comments.
• Status codes: 0 = enabled, 1 = disabled for both media types and actions.
• templateid on host triggers: If 0, the trigger was created directly on the host (not inherited from a template). Non-zero = the parent trigger ID on the template.
• Severity levels: 0=Not classified, 1=Information, 2=Warning, 3=Average, 4=High, 5=Disaster
• {EVENT.VALUE}: 1 = problem, 0 = OK/recovery
• {EVENT.NSEVERITY}: Numeric severity (0-5)
• Webhook scripts run in Zabbix's built-in Duktape JavaScript engine (ES5 only — no let, const, arrow functions, template literals, Array.find, etc.)

---

4. Host Organization & Data Model

Host Group Hierarchy

Every monitored host is assigned to multiple groups:

Host: "Kuhn's Quality Foods"
├── Clients/Kuhn's Quality Foods    ← client identity
├── ISP/Zito Media, L.P.           ← internet provider
└── Datto RMM Sites                ← RMM platform category

This triple-grouping enables:

• Per-client dashboards — filter by Clients/ group
• ISP outage detection — if 3+ hosts on same ISP go down = likely ISP issue
• RMM correlation — cross-reference Zabbix with Datto RMM agent status

Client Groups (46)

ID	Name
84	Clients/3 Rivers Express
90	Clients/ABC Fire Extinguisher Inc
49	Clients/ADM Signs
40	Clients/Advanced Masonry
50	Clients/Alabek Commercial Roofing Corp.
33	Clients/All Saints Catholic Church
51	Clients/Attica Hub/Seneca Publishing
42	Clients/Bella Diamond LLC
28	Clients/Blake Dentistry
85	Clients/Bosak Eyecare & Optical
55	Clients/Bridges Health Partners Services, LLC
37	Clients/Brodaks
45	Clients/Broker's Settlement Services, Inc.
43	Clients/Brooks Diamonds
57	Clients/Buffalo Glass Block
60	Clients/CUMI America
61	Clients/Chartiers Animal Hospital Ltd
58	Clients/Cincinnati Glass Block
56	Clients/Clista Electric Inc.
35	Clients/ConnecTel, Inc.
26	Clients/Finn Chiropractic Group
59	Clients/Frew Plumbing, Heating, & Air
53	Clients/Greco Gas
65	Clients/Heart Prints Center for Early Education
47	Clients/Hergenroeder, Rega, Ewing, & Kennedy, LLC
69	Clients/Hynes Industries
48	Clients/Insurance Restoration Consultants, Inc.
32	Clients/Kuhn's Quality Foods
44	Clients/Loss Prevention Services
80	Clients/MDS Energy Development, LLC
88	Clients/Marsico Financial Group, LLC
62	Clients/Nordmann Roofing
68	Clients/North Eastern Uniforms & Equipment Inc
38	Clients/POH+W Architects
75	Clients/Penn Energy Resources
86	Clients/Pittsburgh Financial Consultants
87	Clients/Premier Automation Holdings, Inc.
23	Clients/Seubert and Associates
91	Clients/Superior Distributing Co
30	Clients/TK Plastics Company, Inc.
70	Clients/Thoroughbred Construction Group
66	Clients/Thrasher Group, Inc.
71	Clients/Universal Plastics
73	Clients/Universal Plastics Latrobe
54	Clients/V-Systems
76	Clients/Vorteq Coil Finishers

ISP Groups (24)

ID	Name
25	ISP/AT&T Enterprises, LLC
27	ISP/Armstrong
24	ISP/Bigleaf Networks, Inc.
34	ISP/Buckeye Cablevision, Inc.
52	ISP/Charter Communications Inc
74	ISP/Citizens Telecommunication Technologies, Inc
67	ISP/CityNet
31	ISP/Comcast Cable Communications, LLC
46	ISP/DQE Communications LLC
63	ISP/Expedient
77	ISP/Fidium
89	ISP/Frontier Communications of America, Inc.
82	ISP/GeoLinks
78	ISP/JACKSON ENERGY AUTHORITY
39	ISP/Level 3 Parent, LLC
92	ISP/Metalink Technologies, Inc.
72	ISP/OneCleveland
64	ISP/Space Exploration Technologies Corporation
83	ISP/UPMC
79	ISP/Ultimate Internet Access, Inc
36	ISP/Verizon Business
81	ISP/Wave Broadband
41	ISP/Windstream Communications LLC
29	ISP/Zito Media, L.P.

Other Groups

ID	Name	Purpose
19	Applications	Application-level monitoring
20	Databases	Database servers
22	Datto RMM Sites	Hosts also managed by Datto RMM
5	Discovered hosts	Auto-discovered hosts
7	Hypervisors	Virtualization hosts
2	Linux servers	Linux-based systems
93	Studio Imagine	Internal project
6	Virtual machines	VMs
4	Zabbix servers	Zabbix infrastructure

Host Pattern

All client-site hosts currently use the ICMP Ping template (templateid: 10564) for basic up/down monitoring. A typical host:

Name:      "Kuhn's Quality Foods"
Status:    enabled
Groups:    [Clients/Kuhn's Quality Foods, ISP/Zito Media L.P., Datto RMM Sites]
Templates: [ICMP Ping]

77 total hosts across the system (including Zabbix infrastructure hosts).

Global Macros

Macro	Value
{$SNMP_COMMUNITY}	public

---

5. Notification Pipeline

Current Architecture

Zabbix Trigger Fires
       │
       ▼
Action: "Send ntfy to NOC" (actionid: 7)
  ├── Problem  → ntfy webhook (mediatypeid: 102)
  └── Recovery → ntfy webhook (same media type, different template)
       │
       ▼
ntfy Webhook Script (Duktape JS)
  ├── Determines problem vs recovery (EVENT.VALUE)
  ├── Sets 🔴 red_circle (problem) or 🟢 green_circle (recovery)
  ├── Sets ntfy priority (high for problems, low for recovery)
  ├── Adds Click URL → Zabbix event page
  ├── For problems: queries Zabbix API for other active issues on same client
  └── POSTs to ntfy → http://ntfy/noc-alerts
       │
       ▼
ntfy Push Notification → user's phone

Action Configuration

Action: "Send ntfy to NOC" (actionid: 7)

• Trigger condition: Severity >= High (conditiontype 4, operator 5, value 4)
• Operations: Send message to user group "Executive" (usrgrpid: 14)
• Recovery operations: Notify all involved (operationtype: 11)
• Uses default messages (default_msg: 1) — pulls from media type templates

Users & Media

User	Role	ntfy Media
Lorentz Hinrichsen (lorentz@wulfconsulting.com)	Super admin (3)	noc-alerts topic (enabled)
Tom Carlin (tom@wulfconsulting.com)	Super admin (3)	(removed — was duplicate to same topic)
Admin	Super admin (3)	(none)
guest	Guest (4)	(none)

ntfy Webhook Script (Current)

The webhook script runs inside Zabbix Server's Duktape JS engine. It:

1. Parses parameters from Zabbix macros
2. Determines if this is a problem or recovery event
3. For problems: makes 2 internal Zabbix API calls to find related active issues for the same client
4. Formats and sends the ntfy notification with appropriate tags, priority, and click URL

Full script and parameters are documented in /opt/stacks/zabbix/notification-changes.md

Message Templates (Current)

Event	ntfy Tag	Subject	Body
Problem	🔴 red_circle	{HOST.NAME} — {EVENT.NAME}	Trigger description + started time + duration + related issues
Recovery	🟢 green_circle	Resolved: {HOST.NAME} — {EVENT.NAME}	"Host is back online." + downtime + restored time
Update	(inherits)	Updated: {HOST.NAME} — {EVENT.NAME}	Who updated + action + status

Notification Example

Problem:

🔴 Kuhn's Quality Foods — Host Unreachable

Host failed to respond to 3 consecutive ICMP ping
requests. The site may be offline, the network path
disrupted, or the device powered off.

Started:  2026.03.11 at 14:22:10
Duration: 5m 30s

── Other active issues (Kuhn's Quality Foods) ──
 · High Packet Loss — Kuhn's SW-Core

Recovery:

🟢 Resolved: Kuhn's Quality Foods — Host Unreachable

Host is back online.

Downtime: 22m 15s
Restored: 2026.03.11 at 14:44:25

Available but Disabled Media Types

These exist in Zabbix but are disabled. Can be enabled if needed:

• Email, Email (HTML), Gmail, Office365, SMS
• Discord, Slack, MS Teams, MS Teams Workflow, Telegram
• Jira, Jira Service Management, ServiceNow, Zendesk, PagerDuty, Opsgenie
• Many others (40+ webhook integrations available)

---

6. Trigger Naming Conventions

Design Principles (Established 2026-03-11)

Triggers were renamed from Zabbix defaults to be executive-friendly:

Default Zabbix Name	Current Name	Severity
ICMP Ping: Unavailable by ICMP ping	Host Unreachable	High (4)
ICMP Ping: High ICMP ping loss	High Packet Loss	Warning (2)
ICMP Ping: High ICMP ping response time	Slow Response Time	Warning (2)

Naming Rules

1. No technical jargon in trigger names — use impact-based language
2. Trigger descriptions (comments field) contain the technical detail: what the check does, thresholds, possible causes
3. Keep names short — they appear in ntfy subjects, Teams cards, dashboards
4. No host name in trigger name — {HOST.NAME} is added by the notification template

Cisco Triggers (Not Yet Renamed)

The Cisco Catalyst SNMP templates have their own ICMP triggers that still use the old naming:

• Cisco Catalyst 3750V2-24FS: Unavailable by ICMP ping
• Cisco Catalyst 3750V2-24FS: High ICMP ping loss
• etc.

These are on separate templates (not the "ICMP Ping" template) and have templateid: 0 on host triggers. Renaming these requires updating each Cisco template individually.

---

7. Grafana Integration

Plugin

• alexanderzobnin-zabbix-app v6.2.1 — connects directly to Zabbix API
• Datasource URL (internal): http://zabbix-frontend:8080/api_jsonrpc.php

Planned: MSP Executive Status Board

Design goal: at-a-glance dashboard for management showing:

Panel	Type	Data Source
Current problems by severity	Stat panels	Zabbix problems
Problems per client	Bar gauge	Zabbix trigger groups
Active problem list	Table	Zabbix problems
SLA/uptime per client	Zabbix SLA panel	Zabbix SLA
Host status grid	Status map plugin	Zabbix hosts

Filter by Clients/ host groups. Use Grafana variables for client selection dropdown.

---

8. Pulse Integration Architecture

Overview

Pulse is the internal Node.js/TypeScript web app with API access to:

• Zabbix (this system)
• PSA (ticketing)
• Datto RMM
• Veeam (backups)
• Microsoft Graph (Teams, email)
• Zoom
• Apprise (notification routing)

Planned: Morning NOC Summary

Full architecture document: /opt/stacks/zabbix/pulse-morning-summary-architecture.md

Key points:

• Scheduled job at 6:30 AM weekdays
• Aggregates data from Zabbix + PSA + Veeam in parallel
• Sends Teams adaptive card (direct via MS Graph), email/ntfy (via Apprise)
• User-configurable delivery preferences
• Killer feature: unmatched alerts — Zabbix problems with no PSA ticket

Zabbix API Queries for Pulse

Get all open problems with host/client context:

// 1. Get open problems
const problems = await zabbixApi('problem.get', {
  recent: true,
  sortfield: ['eventid'],
  sortorder: 'DESC'
});

// 2. Get host → client mapping (cache this)
const hosts = await zabbixApi('host.get', {
  output: ['hostid', 'host', 'name'],
  selectHostGroups: ['groupid', 'name']
});

// 3. Build client map
const clientMap = {};
for (const host of hosts) {
  const clientGroup = host.hostgroups.find(g => g.name.startsWith('Clients/'));
  if (clientGroup) {
    clientMap[host.hostid] = clientGroup.name.replace('Clients/', '');
  }
}

Get overnight resolved events:

const resolved = await zabbixApi('event.get', {
  source: 0,
  object: 0,
  value: 0,  // recovery events only
  time_from: Math.floor(yesterday6pm.getTime() / 1000),
  time_to: Math.floor(today630am.getTime() / 1000),
  selectHosts: ['name'],
  selectRelatedObject: ['description'],
  sortfield: ['clock'],
  sortorder: 'DESC'
});

---

9. Development Patterns & Gotchas

Zabbix Webhook Script Constraints

The webhook script runs in Duktape (ES5 JavaScript engine):

DO:

var x = 'hello';               // var only
for (var i = 0; i < arr.length; i++) { }  // classic for loops
JSON.parse(), JSON.stringify()  // available
new HttpRequest()               // Zabbix's HTTP client
btoa()                          // Base64 encoding available

DON'T:

let x = 'hello';               // NO let/const
const y = () => {};             // NO arrow functions
`template ${literal}`;          // NO template literals
arr.find(x => x.id === 1);     // NO Array.find/includes/map
for (const x of arr) {}        // NO for...of

HttpRequest in Webhooks

var req = new HttpRequest();
req.addHeader('Content-Type: application/json');
req.addHeader('Authorization: Bearer TOKEN');

var resp = req.post(url, body);    // POST
var resp = req.put(url, body);     // PUT
var resp = req.get(url);           // GET

var status = req.getStatus();      // HTTP status code

Multiple HttpRequest instances CAN be created in the same script (used for the related-problems enrichment that calls Zabbix API before calling ntfy).

API Update Patterns

When updating a media type, you must include the FULL array for parameters and message_templates — they're replaced entirely, not merged.

# WRONG — this deletes all other parameters
mediatype.update({ parameters: [{"name": "new_param", "value": "x"}] })

# RIGHT — include ALL parameters
mediatype.update({ parameters: [
    {"name": "endpoint", "value": "http://ntfy/noc-alerts"},
    {"name": "username", "value": "monitoring"},
    # ... all existing params ...
    {"name": "new_param", "value": "x"}
] })

Trigger Description vs Comments

This is confusing in the Zabbix API:

• description field = the trigger name (what you see in the UI)
• comments field = the description text (the explanation paragraph)

{
  "triggerid": "23176",
  "description": "Host Unreachable",           // ← this is the NAME
  "comments": "Host failed to respond to..."   // ← this is the DESCRIPTION
}

Host Group Filtering Pattern

To get the client name for a host:

// Get host groups
var groups = hostData.hostgroups;
var clientName = null;
for (var i = 0; i < groups.length; i++) {
    if (groups[i].name.indexOf("Clients/") === 0) {
        clientName = groups[i].name.replace("Clients/", "");
        break;
    }
}
// clientName = "Kuhn's Quality Foods"

Docker Exec for API Calls

Since the Zabbix frontend doesn't have ports mapped to the host, API calls from the host must go through docker exec:

docker exec zabbix-frontend curl -s -X POST \
  "http://localhost:8080/api_jsonrpc.php" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"jsonrpc":"2.0","method":"...","params":{...},"id":1}'

Or use Python for complex payloads (escaping JSON in bash is fragile):

import json, subprocess

payload = json.dumps({...})
result = subprocess.run(
    ["docker", "exec", "zabbix-frontend", "curl", "-s", "-X", "POST",
     "http://localhost:8080/api_jsonrpc.php",
     "-H", "Content-Type: application/json",
     "-H", f"Authorization: Bearer {TOKEN}",
     "-d", payload],
    capture_output=True, text=True
)
data = json.loads(result.stdout)

External API Access (from Pulse or other servers)

const response = await fetch('https://zabbix.wulfconsulting.cloud/api_jsonrpc.php', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${ZABBIX_API_TOKEN}`
  },
  body: JSON.stringify({
    jsonrpc: '2.0',
    method: 'problem.get',
    params: { recent: true },
    id: 1
  })
});

---

10. API Cookbook

Get All Active Problems

{
  "method": "problem.get",
  "params": {
    "output": "extend",
    "recent": true,
    "sortfield": ["eventid"],
    "sortorder": "DESC"
  }
}

Get Active Problems for a Specific Client

{
  "method": "trigger.get",
  "params": {
    "output": ["triggerid", "description", "lastchange", "priority"],
    "groupids": ["32"],
    "only_true": true,
    "selectHosts": ["name"],
    "skipDependent": true
  }
}

(groupid 32 = Clients/Kuhn's Quality Foods)

Get All Hosts with Client + ISP Mapping

{
  "method": "host.get",
  "params": {
    "output": ["hostid", "host", "name", "status"],
    "selectHostGroups": ["groupid", "name"],
    "selectParentTemplates": ["templateid", "name"]
  }
}

Get Recovery Events in a Time Window

{
  "method": "event.get",
  "params": {
    "source": 0,
    "object": 0,
    "value": 0,
    "time_from": 1741647600,
    "time_to": 1741692600,
    "selectHosts": ["name"],
    "selectRelatedObject": ["description"],
    "sortfield": ["clock"],
    "sortorder": "DESC"
  }
}

Update a Trigger Name + Description

{
  "method": "trigger.update",
  "params": {
    "triggerid": "23176",
    "description": "Host Unreachable",
    "comments": "Host failed to respond to 3 consecutive ICMP ping requests."
  }
}

Get Media Type with Full Details

{
  "method": "mediatype.get",
  "params": {
    "output": "extend",
    "selectMessageTemplates": "extend",
    "mediatypeids": ["102"]
  }
}

Test ntfy Notification Manually

docker exec zabbix-frontend curl -s -X PUT \
  -H "Authorization: Basic $(echo -n 'monitoring:GBigt1231#' | base64)" \
  -H "Title: Test Notification" \
  -H "Priority: 3" \
  -H "Tags: red_circle" \
  "http://ntfy/noc-alerts" \
  -d "This is a test notification from Zabbix"

---

Related Documentation

• Notification changes (before/after + restoration): /opt/stacks/zabbix/notification-changes.md
• Pulse morning summary architecture: /opt/stacks/zabbix/pulse-morning-summary-architecture.md
• Docker Compose: /opt/stacks/zabbix/compose.yml
• Zabbix SSO setup: /opt/stacks/zabbix/ENTRA-AUTHENTIK-ZABBIX-SSO.md
• Zabbix 7.4 API docs: https://www.zabbix.com/documentation/7.4/en/manual/api