Notification + Monitoring Stack¶

Notification gateway, network monitoring, and traffic analysis pipeline.

Architecture¶

graph TD
    SVC[Services] -->|alerts| APP
    APP["Apprise :8008"] --> TG["Telegram (Homelab group)"]
    MK["MikroTik RB5009"] --> MKTOOLS["mikrotik-tools :8087
9 scheduled jobs"]
    MK -->|IPFIX :12055| NF2NG["netflow2ng"]
    MK -->|Exporter metrics| MKTOOLS
    MK -->|REST API :443/rest| MKTOOLS
    MKTOOLS --> EXP["Exporter → VictoriaMetrics :8428"]
    MKTOOLS --> VMALERT["vmalert → Apprise on alert"]
    MKTOOLS --> NTFYBR["ntfy-bridge → Apprise enriched alerts"]
    NF2NG --> NTOPNG["ntopng :3004
(DPI + flows, bytes pending fix)"]

Components¶

Notification¶

Apprise Config Keys¶

Telegram only — ntfy removed from all apprise config keys.

Sending Notifications¶

# Stateful (preferred)
curl -X POST http://100.126.172.96:8008/notify/all \
  -H "Content-Type: application/json" \
  -d '{"title": "Alert", "body": "message"}'

# Stateless (backward compatible)
curl -X POST http://100.126.172.96:8008/notify \
  -H "Content-Type: application/json" \
  -d '{"urls": "ntfy://...", "title": "...", "body": "..."}'

Network Monitoring¶

Traffic Analysis¶

Two pipelines (parallel):

MikroTik RB5009
  ├─ :12055 → netflow2ng v0.2.2 → ntopng (:3004)     ← UI + DPI
  │    └─ custom_flow_export.lua (embedded in entrypoint)
  │         ↑ scraped by scrape_ntopng.py → DWH
  └─ :12056 → GoFlow2 → Kafka 'network-flows'          ← Raw flow data

Custom Lua endpoint: /lua/custom_flow_export.lua returns JSON with DPI app name, bytes, packets, AS names for all active flows. Works with --disable-login 1 (no auth).

ntopng Community REST API requires Enterprise license — all /lua/rest/v2/ endpoints return 302. Custom Lua endpoint bypasses this limitation.

DWH Data Sources (ntopng scraper)¶

scrape_ntopng.py runs every 5min:

1. Queries http://127.0.0.1:3004/lua/custom_flow_export.lua
2. Enriches flows: nDPI app → service name + category (30+ mappings)
3. Pushes to DWH /api/v1/logs (provider=ntopng, category=network_flow)
4. Metadata includes: ndpi_app, service, service_category, bytes, packets, src_as, dst_as

mikrotik-tools Scheduler (10 jobs)¶

Alert Rules (vmalert)¶

Located in stacks/monitoring/ros-monitoring/alerts.yml:

Notification Format¶

Mobile-optimized for Telegram:

🟡 /dev/sdd spinning 2d5h
ST12000NM0127 | 🌡️45C
📊 I/O:
  • ntfy-bridge ✍️2MB
  • caddy 👁️16KB
Disk never enters standby

VPN Configuration¶

See docs/mikrotik-vpn-routing-playbook.md for full 6-layer checklist.

WireGuard (ROS)¶

• Interface: protonvpn → ProtonVPN VN6
• Routing: WiFi devices (zone-lan + zone-iot) → VPN, servers + PS5 → WAN
• Failover: distance 2 WAN backup + health monitor script

VPN Exclusions (auto)¶

• port-forwarded address-list: auto-synced from NAT rules
• zone-servers (.10-.32, .59): excluded by whitelist
• Tailscale (100.64.0.0/10): excluded in vpn-route table
• PS5 (.100): auto via port-forwarded list

Health Check¶

docker exec mikrotik-tools python3 /app/scripts/health_check.py

12-section verification: RouterOS, ports, WireGuard, VPN routing, firewall, DHCP, Apprise, VM, AdGuard, DWH, scripts, state files.

Pitfalls¶

• Apprise endpoint is /notify (not /) — POST to / returns 405
• Bot token from Komodo Variable includes [SECRET] suffix — strip with grep -oP '= \K\S+'
• cap_drop: ALL requires cap_add: [SETUID, SETGID, CHOWN] for apprise supervisord
• Bridge fast-forward must be OFF for mangle routing marks to work
• MSS clamping required for protonvpn (TCP SYN hangs without it)
• Forward chain must have allow-lan-to-vpn rule before catch-all drop
• ntopng Community can't receive NetFlow directly — use netflow2ng
• ntopng flows show 0 bytes via netflow2ng — DPI works but bytes pending fix
• AdGuard query log retention increased from 6h to 24h