RouterOS Migration & Network Operations Lessons¶

When to read: before ANY change to mik-main (RouterOS gateway), DHCP, IP planning, NetFlow, QoS, or monitoring integration.

Date: 2026-07-04 (FPT → MikroTik migration + ElastiFlow + monitoring session)

Rule 1: Static Leases BEFORE Pool Change¶

Mistake: Changed DHCP pool .2-.254 → .100-.199 without adding static lease for unraid (.59). unraid renewed DHCP → got .110 → all 60+ compose files, NetFlow, OpenSearch, DNS broke silently. SSH via Tailscale still worked → undetected for hours.

Fix: Add ALL server static leases FIRST, then change pool. Verify each server keeps its IP after pool change.

# CORRECT ORDER:
1. /ip dhcp-server lease add address=.59 mac=XX server=dhcp-lan  # ALL servers first
2. /ip dhcp-server lease add address=.11 mac=XX ...
3. THEN: /ip pool set ranges=.100-.199                           # Pool change LAST
4. Verify: ping each server from mik-main

Check: After ANY DHCP pool change, immediately ping every known server IP from mik-main. If ping fails → IP changed → fix before moving on.

Rule 2: Komodo-Managed Containers — Never docker stop/rm¶

Mistake: docker stop victoriametrics && docker rm victoriametrics then docker run with custom flags. This destroyed Komodo management labels, broke existing scrape pipeline (397M datapoints), and the manual container had different network behavior.

Fix: Update compose file in git → push → Komodo auto-sync. Or SSH to unraid and run docker compose up -d --no-deps <service> from Komodo repo path: /mnt/user/appdata/komodo/komodo-data/repos/docker-compose/...

# CORRECT: deploy from Komodo repo
ssh root@unraid 'cd /mnt/user/appdata/komodo/komodo-data/repos/docker-compose/stacks/monitoring/mimir-loki-tempo && docker compose up -d --no-deps victoriametrics'

NEVER: docker run or docker stop/rm on any container with com.docker.compose.project label.

Rule 3: Tailscale vs LAN vs Docker DNS — When to Use Which¶

Mistake: Used Tailscale IP (100.x) for NetFlow target on mik-main — mik-main has no Tailscale route → packets dropped. Used LAN IP (192.168.100.59) for inter-container communication — Docker bridge can't reach host LAN IP.

Decision tree:

Source → Target	Use
mik-main → unraid/service	LAN IP (192.168.100.x) — mik-main has no Tailscale
Container → Container (same Docker network)	Container name (Docker DNS)
Container → Service on another server	Tailscale IP (100.x)
Dev machine → unraid service	Tailscale IP (100.68.251.84)
mik-main NetFlow/monitoring target	LAN IP (192.168.100.59)

Rule 4: RouterOS REST API Singleton Workaround¶

Pattern: RouterOS 7.x REST API cannot PATCH singletons (identity, DNS, NTP, traffic-flow, upnp, device-mode). Returns missing or invalid resource identifier.

Workaround: Use /system/script create → run → delete:

SRC='/system identity set name=mik-main'  # Any singleton set command
JSON=$(python3 -c 'import json,sys;print(json.dumps({"name":"temp","source":sys.stdin.read()}))' <<< "$SRC")
curl -X PUT "$B/system/script" -d "$JSON"  # Create
SID=$(curl "$B/system/script?name=temp" | jq -r '.[0][".id"]')
curl -X POST "$B/system/script/run" -d "{\".id\":\"$SID\"}"  # Run
curl -X DELETE "$B/system/script/$SID"  # Cleanup

Rule 5: RouterOS 7.x Path Changes (vs old docs)¶

Old docs	RouterOS 7.23 actual
`/tool traffic-flow`	`/ip traffic-flow`
`address=` (target)	`dst-address=`
`cache-entries=32k`	Invalid (use `32768` or omit)
`/tool traffic-flow set enabled=yes`	Works via script only (singleton)

Always: /console/inspect to verify command path before scripting.

Rule 6: Scout Existing Stack Before Building New¶

Mistake: Built SNMP exporter + promscrape pipeline (complex, failed) when mktxp-exporter was already running on unraid with otel-collector scraping it. Only needed: update mktxp config hostname from 10.10.4.1 → 192.168.100.1.

Process:

Query VictoriaMetrics: label/__name__/values → what metrics exist?
Query VM targets: up → what's already being scraped?
Check otel-collector config → what scrape jobs configured?
Only build NEW if nothing existing matches the need.

Rule 7: QoS Mangle — passthrough=no + Correct Order¶

Mistake: Used passthrough=yes on all mangle rules. The bulk catch-all rule (dscp=0) overwrote DNS/voice marks because it came after specific rules with passthrough=yes.

Correct:

passthrough=no on ALL rules (first-match-wins)
Order: voice (dscp=EF) → voice (port=53) → voice (STUN) → gaming → streaming → Tailscale → bulk (catch-all LAST)

Rule 8: Verify After Every Config Change¶

Pattern: Deploy config → move to next task → discover hours later it's broken.

Checklist after ANY mik-main change:

# 1. mik-main reachable?
ping 192.168.100.1
# 2. Internet works?
curl -sI https://example.com
# 3. DNS resolves?
nslookup google.com 192.168.100.1
# 4. DHCP clients OK?
curl -u lucndm:minhluc1 http://192.168.100.1/rest/ip/dhcp-server/lease | python3 -m json.tool | head
# 5. Firewall drops increasing (normal) not flat (broken)?
curl -u lucndm:minhluc1 http://192.168.100.1/rest/ip/firewall/filter | ...
# 6. Affected services still responding?
curl http://192.168.100.59:9200/_cluster/health  # OpenSearch
curl http://192.168.100.59:8428/health            # VictoriaMetrics

Rule 9: User Knows Their Network — Listen¶

Multiple instances:

User said FPT MAC = ...:F1 not ...:F0 → I debated, user was right (WAN vs LAN MAC)
User said ONT already bridged → I dispatched librarian research for VLANs (wasted)
User said use Tailscale IPs → I kept using LAN IPs (broke connectivity)
User said "tôi chắc chắn đã rút FPT" → I kept insisting FPT was still connected

Lesson: When user states a fact with confidence, accept it and investigate WHY the evidence contradicts — don't dismiss the user's statement.

Rule 10: ElastiFlow Specifics¶

Topic	Detail
flow-collector `network_mode`	MUST be `host` (preserves NetFlow source IP)
OpenSearch output address	`127.0.0.1:9200` (localhost, same host)
`user_defined.yml` format	Dict: `'192.168.1.1': 'hostname'` NOT list of objects
DNS enrichment `EF_PROCESSOR_ENRICH_IPADDR_DNS_USERDEF_PATH`	NOT `METADATA_USERDEF_PATH` (different, causes crash)
Permission on flowcoll volume	`chown 1000:1000` — collector runs as uid 1000
Threat collection error	`permission denied` on flowcoll dir → chown fix
ISM policy	ElastiFlow creates own policy (`elastiflow`), update retention there
Dashboard import	`POST /api/saved_objects/_import?overwrite=true` with ndjson file

Rule 11: Long-running container over cron + `docker run --rm`¶

Mistake: Used */1 * * * * docker run --rm ros-exporter ... pattern for monitoring. Image was local-build (ros-exporter:latest, not pushed to registry). Got pruned somehow (cause undetermined; syslog rotated before trace). Cron then failed 1440 times/day silently for 4 hours before user noticed via Grafana gap.

Fix: Long-running container with restart: always, managed by Komodo:

services:
  ros-security-exporter:
    build: . # Komodo auto-rebuild on deploy
    restart: always # auto-recover from crash
    command: python3 -u /app/exporter.py --loop 30

Generic rule: Any service that pushes telemetry must be a long-running container, not a cron-spawned one-shot. Catches both image-prune and crash recovery automatically. Container-level uptime is verified by Uptime Kuma; data freshness is verified by a separate meta-monitor (ros-health-check).

Rule 12: macvlan reachability — test from a non-parent host¶

Mistake: Reported adguard-replica (.229) DOWN based on nc -zv 192.168.100.229 53 returning "No route to host" and ARP showing stale. Actually the test was run from dev (.31), which IS the macvlan parent host. Linux kernel blocks host → own-macvlan- container communication by design.

Verification: From another LAN host (e.g., unraid .59):

dig +short @192.168.100.229 google.com   # works
ping 192.168.100.229                     # works

Rule: For macvlan diagnostics, always test from a host OTHER than the macvlan parent. Testing from the parent itself is a false negative.

Rule 13: Komodo overrides `docker update --restart=no`¶

Mistake: Tried to stop a Komodo-managed container for failure-mode testing. docker update --restart=no && docker stop worked, but container came back within 30s. Concluded "something is wrong with restart policy". Actually Komodo periphery reconcile loop detected container not matching compose desired state and restarted it.

Rule: For testing failure scenarios on Komodo-managed containers, do NOT use docker stop. Instead:

Block network in container namespace: iptables -A OUTPUT -d <target> -j DROP
Or temporarily stop Komodo periphery: docker stop komodo-periphery (affects all)
Or mock the failure (e.g., point exporter at non-existent endpoint)

Side-note: Repeated docker stop/rm/up cycles on a Komodo-managed host trigger concurrent reconcile attempts → dockerd CPU spike. Avoid rapid cycles in production.

Rule 14: mktxp feature flags must match router hardware¶

Mistake: Default mktxp.conf enables poe = True, wireless = True, capsman = True. On mik-main (x86 CHR), these features don't exist → mktxp logs "no such command prefix" errors every scrape cycle. Logs fill, signal-to-noise drops, real issues hidden.

Fix: Disable unsupported features explicitly:

poe = False
wireless = False
wireless_clients = False
capsman = False
capsman_clients = False

Rule: After any router migration, audit mktxp.conf feature flags vs actual hardware capabilities. Also verify hostname and username/password match the new router. | Dark mode | POST /api/opensearch-dashboards/settings/theme:darkMode |