Skip to content

Project Changelog

2026-07-17 — Unified Monitoring Pipeline + VPN Failover + ElastiFlow Decommission

Network Monitoring (ntopng → Grafana)

  • ElastiFlow decommissioned: flow-collector stopped, image removed, indices deleted (5.1 GB reclaimed), 38 dashboards removed. OpenSearch kept for Obsidian/podcasts.
  • ntopng DPI pipeline: custom Lua endpoint → scraper (5min) → Push API → Kafka → RisingWave → Grafana
  • GeoIP enrichment: GeoLite2-City.mmdb, country/city/lat/lon per flow
  • 6 dbt MVs: flows_hourly, top_talkers, conversations, ports, as_traffic, geo, wan_ip_inventory
  • 3 Grafana dashboards: Network Flows (10 panels), Conversations & Detail (11 panels), Chat Analytics (8 panels)
  • Grafana fix: datasource type postgres → grafana-postgresql-datasource (Grafana 13)
  • Push API fix: metadata VARCHAR → JSONB, timestamp epoch → ISO 8601

VPN Failover (3-tier)

  • VN6 → VN9 → WAN: automatic failover chain with state machine
  • Only 1 tunnel enabled at a time (avoids IP conflict 10.2.0.2)
  • PING_COUNT 3, FAIL_THRESHOLD 2 (reduced false-positive failover)
  • Recovery from WAN: try VN6 → VN9
  • Firewall rules added for protonvpn2 (FORWARD + NAT + MSS clamp)
  • VPN monitoring: WAN IP change detect (5min), VPN state push to RW (5min), ProtonVPN IP auto-update (daily), VPN IP detector (10min)

Unified Monitoring (RisingWave)

  • Chat analytics: Telegram 3,839 + Zalo 614 messages → Grafana dashboard
  • Server health: MikroTik CPU/RAM → platform_metrics (2min)
  • Docker events: container start/stop/restart → platform_events (2min)
  • GoFlow2 source: raw NetFlow data in RisingWave
  • Disk I/O monitor: Unraid disk activity checker + notify on high I/O

Security

  • ProtonVPN IP ranges: 188.214.152.0/24 (VN) + 149.50.211.0/24 (SG) in address-lists
  • Auto-detect VPN IPs: scan ntopng for VPN-port traffic from LAN devices
  • Scheduler: 18 jobs running (see scheduler.py)
  • Grafana dashboard review: 31 "No data" → 9 (hardware limitations only)

Infrastructure Cleanup

  • aria2 removed: container stopped, 17 GB torrent files deleted, session cleared, removed from compose.yaml + Komodo pre_deploy
  • mikrotik-tools: Docker socket mounted, 23 scripts, psycopg2 + geoip2 + docker SDK in requirements
  • dbt project: analytics/streaming with 12 MVs managed by dbt (deploy-dbt.sh streaming)

Grafana Dashboards Summary

Dashboard Panels Status
Network Flows (ntopng DPI) 10 ✅
Network Conversations & Detail 11 ✅
Chat Analytics 8 ✅
DP Pipeline Overview — ✅
DP Redpanda Monitor — ✅
ROS Security — ✅
MKTXP MikroTik — ✅
net-client-devices — ✅
mikrotik-mktxp-fixed — ⚠️ 3 (RB5009 hardware limit)
docker-exporter-fixed — ⚠️ 6 (Unraid platform limit)

Commits (25+ in session)

Key: 30eef34b (dbt MVs), e2b04e76 (Grafana datasource fix), 40fe266a (GeoIP + ElastiFlow decommission), 7fa112d3 (3-tier VPN failover), 6132e4d9 (unified monitoring), ddad8312 (docker SDK), 6b166bac (aria2 removal), 40cb3ca7 (Komodo pre_deploy fix)

2026-07-07 — Unraid: Memory Pressure Runbook + 32 GB Swap on Vortex

Incident (severity: high, impact: production)

Unraid server entered page cache thrashing cascade: load avg 308 (12-core box), sys CPU 96–98%, 12 containers unhealthy, NVMe reads 3.5 GB/s. Root cause: 31 GiB RAM exhausted by 60+ containers with 0 swap configured. Kernel evicted page cache → every docker exec (health checks) hit disk → IO amplification to 44 TB cumulative NVMe reads in 65 h.

Mitigation applied

  • Added 32 GB swap on vortex btrfs (/mnt/vortex/@swap/swapfile{,2}) using NOCOW subvolume
  • Persisted swap activation in /boot/config/go (backed up to go.bak.20260706-174058)
  • Verified recovery: load 1–2, sys CPU 1–9%, page cache 3.5 GB → 8.6 GB, 0 unhealthy containers

Documentation (new)

  • docs/runbooks/unraid-memory-pressure.md — diagnostic chain (5-step triage), decision tree, btrfs swap setup procedure, container memory-limit procedure, verification checklist
  • .claude/skills/unraid/SKILL.md — new ## Troubleshooting: Memory / IO Pressure section (symmetric to existing Array Stop Stuck section), with common-cause table and swap setup quick reference

Watch items

  • flowcoll (elastiflow) suspected memory leak — OOM'd once at 5.94 GB RSS, restarted and growing again. Monitor; if leak confirmed, add memory limit via stack TOML.
  • Consider VictoriaMetrics alert: (1 - mem_cache/RAM) < 0.2 AND sys_cpu > 0.5 to catch future thrashing early.

2026-07-07 — Data Platform: Lakehouse Stack Deploy (Gravitino + Trino + RisingWave + Redpanda)

New Category: data-platform (9th category)

Open-source lakehouse pipeline deployed on development server. Replaces broken iceberg-rest-fixture with Apache Gravitino as unified metadata catalog.

Architecture

CDC/Events ──▶ Redpanda ──▶ RisingWave ──▶ Iceberg (on RustFS via Gravitino REST catalog)
                                                   ▲
                                         Trino queries ◀── Superset BI

New Services (3 stacks, 5 containers)

Stack Service Image Port Purpose
data-platform-catalog Gravitino apache/gravitino:1.3.0 8090 Iceberg REST catalog + admin UI
data-platform-query Trino trinodb/trino:482 8091 Distributed SQL over Iceberg tables
data-platform-streaming Redpanda redpandadata/redpanda:v25.3.1 9092/8083/8086 Kafka-compatible streaming (moved from storage)
data-platform-streaming Redpanda Console redpandadata/console:latest 8085 Topic browser UI
data-platform-streaming RisingWave risingwavelabs/risingwave:v2.7.0 4566/5691 Streaming database (materialized views → Iceberg sink)

Infrastructure Changes

  • New resource_sync: data-platform added to komodo/stacks/root_syncs/resource_sync.toml
  • New Komodo Variables: RUSTFS_ICEBERG_ACCESS_KEY, RUSTFS_ICEBERG_SECRET_KEY, GRAVITINO_PG_PASSWORD
  • New Caddy routes: gravitino.home, trino.home, redpanda.home, risingwave.home
  • Pigsty PostgreSQL: new gravitino database + user + DDL schema (39 tables) for Gravitino entity store
  • RustFS: dedicated iceberg bucket for Iceberg data + metadata

Removed / Moved

  • iceberg-rest-fixture (stacks/database/iceberg/) — removed (broken: dead alist:5246 endpoint, in-memory fixture)
  • Redpanda moved from stacks/storage/redpanda/ → stacks/data-platform/compose/40-streaming/

Deferred

  • Authentik (Phase 80) — identity provider deferred per user request
  • Apache Ranger (Phase 90) — dropped from dev (Trino file RBAC sufficient); alternatives documented
  • Bootstrap Procedure (procedures.toml) — Komodo sync rejects file, needs separate debug

Known Limitations

  • RisingWave uses playground mode (in-memory state) — Hummock-on-RustFS deferred
  • Trino runs in no-auth mode (ALLOWALL) — wire-up to Authentik deferred to Phase 80
  • Bind-mounts from /opt/appdata/ fail on Komodo Periphery — configs inlined via entrypoint heredoc workaround

2026-07-04 — Network: FPT → MikroTik Migration + ElastiFlow + Monitoring

Network Infrastructure

FPT → MikroTik Gateway Migration

  • mik-main (RouterOS 7.23.1, x86, MAC 00:F0:4D:00:78:CD) replaces FPT router as gateway at 192.168.100.1
  • PPPoE WAN: 1.54.116.135 (FPT Telecom), MTU/MRU 1492, TCP-MSS clamp
  • ether2 = WAN (PPPoE, MAC clone 54:04:63:15:4E:F1), ether1+3+4 = bridge1 (LAN)
  • Firewall: 7 filter rules (accept-before-drop), NAT masquerade, 4-tier QoS
  • DNS forwarder: 192.168.100.59 (AdGuard) + 1.1.1.1
  • DHCP: pool .100-.199, 11 static leases, lease time 30m
  • FPT router: removed from network (powered off)

IP Planning (192.168.100.0/24)

  • .1 = mik-main (gateway)
  • .10 = Caddy macvlan (*.home reverse proxy)
  • .11 = pve (Proxmox)
  • .30 = MikroTik CHR (VM)
  • .31 = development (dev container)
  • .32 = win11-ltsc
  • .50-.56 = IoT (HP Printer, Xiaomi Camera, Aqara Hub, Xiaomi Gateway, ESP, EcoFlow, Phong-khach)
  • .59 = unraid (kept — 60 compose files reference this IP)
  • .100-.199 = DHCP dynamic pool

QoS (4-tier DSCP-based)

  • Voice (EF/DSCP 46 + DNS/53 + STUN/3478): prio 1, 20M up / 50M down
  • Gaming (AF41/DSCP 34): prio 2, 50M up / 150M down
  • Streaming (AF31/DSCP 26 + Tailscale 41641): prio 3, 100M up / 400M down
  • Bulk (DF/DSCP 0 + catch-all): prio 8, 300M up / 1000M down
  • PCQ per-IP fairness within each tier
  • Mangle: passthrough=no, first-match-wins, DSCP + port-based

Tailscale P2P Optimization

  • Static NAT: UDP 41641 → 192.168.100.59:41641 (unraid Tailscale Docker)
  • UPnP: disabled (security)
  • Before: 185 GB DERP relay traffic through Singapore
  • After: direct P2P established (m2, unraid, zimaos all direct)

SNMP

  • Enabled on mik-main (community: public, addresses: 0.0.0.0/0)

Monitoring

ElastiFlow NetFlow Monitoring

  • flow-collector (elastiflow/flow-collector:7.26.2) on unraid, network_mode: host, UDP 9995
  • Output → OpenSearch (unraid 127.0.0.1:9200, heap 2GB)
  • OpenSearch Dashboards: unraid :5601, 40+ dashboards imported (413 saved objects)
  • GeoIP: MaxMind GeoLite2-City + ASN (from OpenObserve mmdb)
  • Hostname enrichment: user_defined.yml (18 devices, dict format 'IP': 'hostname')
  • DNS reverse lookup: enabled (nameserver 192.168.100.1)
  • Threat intel: 357MB IPDB loaded
  • ISM retention: 180 days (~112 GB raw / ~67 GB compressed)
  • mik-main NetFlow: /ip traffic-flow enabled, dst-address=192.168.100.59 port=9995 version=9
  • Note: RouterOS 7.23 uses /ip/traffic-flow (not /tool/traffic-flow), dst-address param (not address)

mktxp → VictoriaMetrics → Grafana

  • mktxp-exporter connects to mik-main API (192.168.100.1:8728, user lucndm)
  • 59 metrics: CPU, memory, interfaces (rx/tx bytes, packets, errors), QoS queues, firewall counters, DHCP leases, connections, routes
  • otel-collector scrapes mktxp (:49090) every 45s → pushes to VictoriaMetrics via Tailscale
  • Grafana datasource: VictoriaMetrics (uid=mimir, url=100.68.251.84:8428)
  • mktxp config: /mnt/user/appdata/otel-collector/configs/mktxp/mktxp.conf

Key Learnings

  1. RouterOS 7.23 REST quirks: singleton PATCH needs /*ID or script workaround (identity, DNS, NTP, traffic-flow, upnp all need scripts)
  2. traffic-flow path: /ip/traffic-flow in ROS 7.23, NOT /tool/traffic-flow
  3. traffic-flow target param: dst-address (not address)
  4. RouterOS 4 ports (not 5 as plan assumed): ether1+3+4=LAN, ether2=WAN
  5. Tailscale DERP: 185 GB relay traffic when P2P fails — static NAT for port 41641 fixes it
  6. Docker networking: containers on bridge network can't reach host LAN IP — use Tailscale IP or Docker DNS
  7. ElastiFlow user_defined.yml format: 'IP': 'hostname' (dict, not list of objects)
  8. ElastiFlow passthrough=no: first-match-wins for QoS mangle — prevents bulk overwriting DNS/voice marks
  9. VictoriaMetrics: managed by Komodo from git repo — don't recreate container manually
  10. mktxp: already deployed but pointing at old router 10.10.4.1 — updated to 192.168.100.1

Files Modified

  • stacks/database/opensearch/compose.yaml — added opensearch-dashboards, heap 2g
  • stacks/monitoring/elastiflow/compose.yaml — new (flow-collector)
  • plans/260620-1725-fpt-to-mikrotik-migration/plan.md — updated status to Done
  • plans/260704-0530-elastiflow-netflow-deploy/ — new plan (3 phases)
  • mik-main: /ip traffic-flow, /ip upnp, /snmp, /queue tree, /ip firewall mangle, /ip firewall nat, /ip dhcp-server lease, /ip pool, /system ntp client

Pending

  • Password change (minhluc1 leaked — deferred by user)
  • Grafana MikroTik dashboard import (manual, ID 17346)
  • btest service disable (minor)

2026-07-03 — njav: Supabase Platform Features (MV + FTS + JSONB)

2026-07-12: MikroTik Tools Container + Apprise Migration

Added

  • mikrotik-tools container (development server) — centralized MikroTik automation:
  • update_vn_ips.py — weekly APNIC VN IP list → MikroTik address-list (1640 ranges)
  • setup_firewall.py — PSD port scan detection + tarpit + auto-ban for non-VN IPs
  • exporter.py — ros-security-exporter (moved from ros-monitoring, env-var parameterized)
  • web_monitor.py — web dashboard (moved from ros-monitoring, env-var parameterized)
  • scheduler.py — APScheduler for periodic tasks (VN IP weekly, firewall daily)
  • MikroTik firewall: VN IP whitelist + PSD scan detection + tarpit + 14-day auto-ban
  • Apprise Telegram-only: removed ntfy from all apprise config keys

Changed

  • ros-monitoring stack: removed exporter + web-monitor (consolidated into mikrotik-tools)
  • Apprise notification format: mobile-optimized with emoji, I/O rates, enrich per alert type
  • ntfy-bridge: notify on state change only (no 30min repeat), async enrichment

Removed

  • scripts/ntfy-bridge.py (legacy duplicate)
  • ros-monitoring Dockerfile (exporter build now in mikrotik-tools)

2026-07-12/13: MikroTik VPN + DNS→DWH Pipeline

Added — VPN

  • WireGuard tunnel on MikroTik (ProtonVPN VN6) — 4-layer config: routing table + mangle + NAT + MSS clamp + forward filter
  • WiFi VPN routing — all ether4 devices route through protonvpn (whitelist zone-lan + zone-iot only)
  • VPN failover — distance 2 WAN backup + check_vpn_tunnel.py health monitor (ping 8.8.8.8 every 30s)
  • Auto port-forward exclusion — sync_port_forward.py scans dst-nat rules → auto-exclude from VPN
  • Exclusion routes — gluetun endpoint, ROS VPN endpoint, Tailscale subnet, PS5
  • Playbook — docs/mikrotik-vpn-routing-playbook.md (6-layer checklist + common mistakes)

Added — Monitoring & Alerts

  • Port status monitor — check_ports.py alerts on ether1-4 up/down every 2min
  • VPN tunnel health — check_vpn_tunnel.py pings through tunnel, auto-failover + Telegram alert
  • DHCP watch — dhcp_watch.py detects new devices joining network, notifies with vendor OUI
  • VPN route checker — check_vpn_routes.py daily verify no double VPN risk
  • ROS config auto-backup — backup_ros_config.py daily export → git commit (diff, skip unchanged)
  • ASN DB download — download_asn.py daily iptoasn.com (529K ranges, free, offline)

Added — Traffic Analysis (ntopng)

  • ntopng — DPI app recognition (nDPI 300+ protocols) + web UI on port 3004
  • netflow2ng — free NetFlow v9 collector (replaces nProbe €199)
  • MikroTik Traffic Flow — NetFlow v9 export to ntopng
  • Known issue: DPI works but flows show 0 bytes — netflow2ng field mapping pending fix
  • Homepage Network section — 8 network service endpoints
  • Komodo Build for mikrotik-tools — auto-build on commit
  • Health check — health_check.py 12-section verification script
  • WireGuard monitoring — ros_wireguard_up metric + WireguardDown vmalert rule

Changed — Apprise

  • Stack renamed ntfy → apprise (apprise is the notification hub)
  • Security hardening: cap_drop ALL + SETUID/SETGID/CHOWN
  • Mobile-optimized notification format with emoji + I/O rates + per-alert enrichment
  • ntfy removed from all apprise config keys (Telegram only)
  • 3 config keys: all (media), infra (komodo), ros-alerts (ros-alerts)

Removed

  • push_dns_logs.py from scheduler — ntopng DPI replaces DNS guess
  • push_caddy_logs.py from scheduler — ntopng measures all protocols
  • scripts/ntfy-bridge.py (legacy duplicate)
  • ros-monitoring Dockerfile (exporter build now in mikrotik-tools)

Scheduler (9 jobs, was 11)

Job Interval Alert
ros-backup daily 2:00 git commit
asn-download daily 2:50 Telegram
vn-ip-update weekly Sun 3:00 Telegram
port-forward-sync daily 3:30 Telegram
firewall-verify daily 4:00 log
vpn-check daily 4:30 log
vpn-tunnel-monitor every 30s Telegram
dhcp-watch every 1min Telegram
port-monitor every 2min Telegram

Issues Saved

  • plans/issues/protonvpn-extension-conflict.md — WireGuard-in-WireGuard UDP MTU issue
  • plans/issues/ntopng-bytes-zero.md — netflow2ng not forwarding byte counters

Pending Tasks (plans/task-backlog.md)

  • DWH-01: UT1 categories → DWH reference table
  • DWH-02: dns_domain_map table in DWH
  • DWH-04: MikroTik per-service accounting (if ntopng bytes fix fails)
  • Fix ntopng bytes=0 issue
  • Research ProtonVPN extension conflict

2026-07-18 — Edge compute layer (ADR-0002 + ADR-0003)

Added 3 new infra capabilities in a single sprint:

Cloudflare Hyperdrive + Workers (ADR-0002)

  • OpenTofu IaC for CF account in infrastructure/cloudflare/ (provider ~> 5.22 for Workers VPC support)
  • Workers VPC connectivity service (homelab-supabase-pg) — TCP tunnel to supabase-db:5432 via existing cloudflared tunnel, self-signed TLS verify disabled (network path secured by tunnel auth)
  • 2 Hyperdrive configs (homelab-pg-cached 60s TTL + homelab-pg-fresh cache-disabled) — wired to connectivity service
  • Supabase PG hardening: TLS enabled (ssl=on, self-signed cert via host openssl), dedicated hyperdrive role with CONNECTION LIMIT 30 + read-only grants + pg_read_all_stats
  • cloudflared-unraid stack attached to supabase-internal Docker network for Hyperdrive TCP forwarding
  • Workers scaffold in workers/ — wrangler 4 + TypeScript 5.7 + Vitest 2 + pg 8.16.3, with reusable template/ (cached vs fresh binding pattern, AUTH_MODE=public default)
  • Pilots deferred per ADR scope (Phase 1+2 marked DEFERRED; only Phase 0 infrastructure completed)

Cloudflare Workers AI via LiteLLM (ADR-0003)

  • New model: cf-qwen3-embedding (1024 dims, 119 languages, $0.012/M tokens = 1,075 Neurons/M) exposed via LiteLLM
  • Uses openai/ provider prefix + CF OpenAI-compatible /ai/v1/embeddings endpoint (LiteLLM native cloudflare/ provider doesn't support embeddings yet)
  • New credential cloudflare_workers_ai_credential in model_config.yaml, uses shared CLOUDFLARE_API_TOKEN Komodo Variable
  • Compatible with existing qwen3-embedding-small (Ollama) — same 1024 dims, downstream vector stores don't need re-index

Cloudflare Prometheus exporter (monitoring)

  • lablabs/cloudflare_exporter deployed on unraid (port 8081)
  • Auto-scraped by otel-collector → VictoriaMetrics (3 R2 metrics flowing)
  • New Grafana dashboard cloudflare-overview.json (7 panels for Workers metrics + Hyperdrive pg_stat_activity)
  • Token scope issue discovered + fixed: required Account > Workers AI: Run (separate from Workers Scripts: Read). Documented in ADR-0003 risk table.

Grafana Supabase datasource

  • Grafana attached to supabase-internal Docker network (new)
  • New provisioning file datasources/supabase.yaml (uid=supabase, env-var password via ${SUPABASE_DB_PASSWORD} from Komodo Variable HYPERDRIVE_DB_PASSWORD)
  • Enables pg_stat_activity WHERE application_name = 'Cloudflare Hyperdrive' panel

Stacks/service count

  • 61 → 64 services (+cloudflare-exporter, +grafana-mcp previously, +1 monitoring)
  • 63 → 66 Komodo TOML configs
  • 62 → 62 (no new service category, just new ones added to monitoring/networking)

Stats

Metric Before After
ADRs 1 3
TF infra modules 0 1 (infrastructure/cloudflare/)
CF Workers stacks 0 2 (scaffold + template)
CF-managed resources ~17 DNS records 20 (+ connectivity service, 2 Hyperdrive configs)
Edge inference endpoints 0 1 (cf-qwen3-embedding via LiteLLM)

2026-07-18 — Komodo Alerter Fix + Webhook Plan

2026-07-18 — Komodo Alerter → Apprise (unified notification)

Unified notification pipeline

  • Created new alerter apprise (endpoint type: Webhook)
  • URL: http://192.168.100.31:8008/notify/infra
  • Enabled: true
  • Disabled old alerter ntfy-bridge (endpoint type: Ntfy)
  • Was: http://192.168.100.31:9282/komodo
  • Enabled: false (backup)
  • Result: All Komodo alerts now go through Apprise → Telegram
  • Single notification gateway for everything
  • ntfy container (:9282) can be decommissioned when ready

2026-07-18 (cont) — synthnews stack decommissioned

Removed

  • stacks/applications/synthnews/compose.yaml — Docker stack (was zombie: Komodo state=running but container missing on unraid, 0 traffic in 24h)
  • komodo/stacks/applications/synthnews.toml — Komodo config
  • cloudflare_dns_record.synthnews — DNS record synthnews.minhluc.info
  • Tunnel ingress rule for synthnews.${local.domain} in tunnel.tf

Updated

  • ADR-0002: synthnews dropped as Phase 1 pilot candidate. Replacement TBD — likely gmail-webhook-classifier (Workers AI use case per ADR-0003).
  • Stack count: 67 → 66

Reason

  • Review of all 67 Komodo stacks for CF Worker migration candidates revealed synthnews was zombie (Komodo reported running, container absent, 0 traffic)
  • Decommission to clean state before any pilot