Skip to content

Monitoring Pipeline Recovery Runbook

When to use: Grafana dashboards show no data / stale data / alerts not firing. Use this to systematically triage which layer broke.

Estimated time: 5–15 minutes for full triage.

Architecture recap — 2 pipelines

Pipeline 1: Network Flows (ntopng DPI)

1. Source        mik-main (NetFlow v9 :12055 → netflow2ng)
2. DPI engine    ntopng (:3004, custom Lua endpoint)
3. Scraper       scrape_ntopng.py (mikrotik-tools, every 5min)
4. Push API      dp-chat-adapter (:8093) → Kafka platform.logs
5. Streaming     RisingWave (:4566) — 13 MVs
6. Visualization  Grafana (:3002) — 3 dashboards, 29 panels

Pipeline 2: Router Metrics (Prometheus)

1. Source        mik-main (REST :80, API :8728)
2. Collector     mktxp-exporter (:49090, scraped by otel)
3. TSDB          VictoriaMetrics (:8428)
4. Alerting      vmalert (:8880) → apprise-bridge → Apprise (:8008)
5. Meta-monitor  mikrotik-tools scheduler (18 jobs)

Failure at any layer cascades downstream. Identify the lowest broken layer first.

Triage in 4 steps

Step 1 — Identify scope of missing data

Symptom Likely pipeline + layer
ntopng Grafana dashboard empty Pipeline 1: scraper or Push API or RisingWave
Router metrics dashboard empty Pipeline 2: mktxp or VM or otel
Single dashboard empty, others fine Grafana datasource (check type = grafana-postgresql-datasource)
All dashboards empty Grafana down OR both pipelines broken
Apprise silent during outage Pipeline 2 Layer 4: vmalert/apprise-bridge
Apprise flood Alert flapping OR vpn-tunnel-monitor threshold too low

Step 2 — Verify Pipeline 1 (Network Flows)

# Layer 6: Grafana datasource OK?
curl -s http://192.168.100.59:3002/api/datasources/uid/risingwave/health

# Layer 5: RisingWave has data?
PGPASSWORD=postgres psql -h 192.168.100.31 -p 4566 -U root -d dev -c \
  "SELECT count(*) FROM platform_logs WHERE provider='ntopng' AND timestamp > NOW() - INTERVAL '10 min'"

# Layer 4: Push API healthy?
curl -s http://192.168.100.31:8093/api/v1/health

# Layer 3: Scraper running?
docker logs mikrotik-tools 2>&1 | grep "ntopng-scrape" | tail -3

# Layer 2: ntopng has flows?
docker exec ntopng curl -s http://localhost:3004/lua/custom_flow_export.lua | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'flows: {len(d.get(\"flows\",[]))}')"

# Layer 1: MikroTik sending NetFlow?
sshpass -p 'minhluc1' ssh lucndm@192.168.100.1 '/ip traffic-flow target print'

Step 3 — Verify Pipeline 2 (Router Metrics)

# Layer 4: alerts firing?
curl -s http://192.168.100.59:8880/api/v1/alerts | python3 -m json.tool | head -30

# Layer 3: VM responding?
curl -s 'http://192.168.100.59:8428/api/v1/query?query=up'
curl -s 'http://192.168.100.59:8428/api/v1/query?query=mktxp_system_cpu_load'

# Layer 2: exporters running?
docker ps --filter name=mktxp --format '{{.Names}}: {{.Status}}'
docker logs mktxp-exporter --tail 10

# Layer 1: mik-main alive?
ping 192.168.100.1

Step 4 — Match symptom to known failure mode

Symptom Cause Fix
RisingWave MVs show NULL timestamps Scraper sent epoch string instead of ISO 8601 Check scrape_ntopng.py — timestamp must be datetime.isoformat()
Grafana shows "No data" on all panels Datasource type wrong (postgres vs grafana-postgresql-datasource) Fix via Grafana API: see .claude/rules/grafana-testing.md
Push API returns 400 metadata sent as JSON string instead of dict Check schemas.py — metadata must be Dict[str, Any]
Scraper log "Pushed 0 flow logs" ntopng Lua endpoint down or ntopng not receiving NetFlow Check ntopng container + netflow2ng + MikroTik traffic-flow target
mktxp logs "No network connection" API user deleted or API service disabled Verify /user print has mktxp and /ip service has api enabled
VPN failover not working check_vpn_tunnel.py state machine stuck Check /data/vpn_failover_state.json — may need manual reset
VPN route flapping PING_COUNT too low or FAIL_THRESHOLD too low Current: PING_COUNT=3, FAIL_THRESHOLD=2

Common recovery procedures

"ntopng scraper not pushing data"

# Run scraper manually
docker exec mikrotik-tools python3 -u /app/scripts/scrape_ntopng.py

# Check Push API accepted
docker exec dp-chat-adapter tail -5 /tmp/push_api.log

# Check Kafka has data
docker exec dp-chat-adapter python3 -c "
import asyncio
from aiokafka import AIOKafkaConsumer, TopicPartition
async def c():
    con = AIOKafkaConsumer('platform.logs', bootstrap_servers='redpanda:9092')
    await con.start()
    tp = TopicPartition('platform.logs', 0)
    e = await con.end_offsets([tp])
    print(f'HW: {e[tp]}')
    await con.stop()
asyncio.run(c())
"

"RisingWave source shows empty metadata"

-- Source was created with VARCHAR instead of JSONB
DROP SOURCE platform_logs CASCADE;
-- Recreate with JSONB (see risingwave-init.sql)

"VPN failover stuck on WAN"

# Check state
docker exec mikrotik-tools cat /data/vpn_failover_state.json
# If tier=wan, manually trigger recovery:
docker exec mikrotik-tools python3 -c "
import json
with open('/data/vpn_failover_state.json','w') as f:
    json.dump({'tier':'wan','fail_count':0,'recovery_counter':5}, f)
"
# Run check — will try VN6 recovery
docker exec mikrotik-tools python3 -u /app/scripts/check_vpn_tunnel.py

Prevention checklist (post-incident)

  • mikrotik-tools container has restart: unless-stopped + Docker socket mounted
  • Scheduler running (check: docker logs mikrotik-tools | grep "scheduler")
  • Push API healthy (curl http://192.168.100.31:8093/api/v1/health)
  • RisingWave healthy (curl http://192.168.100.31:4566/health)
  • Grafana datasource type = grafana-postgresql-datasource (not postgres)
  • dbt models deployed (scripts/deploy-dbt.sh streaming)
  • VPN failover state not stuck (check /data/vpn_failover_state.json)
  • ProtonVPN IP ranges in address-list (188.214.152.0/24, 149.50.211.0/24)