Skip to content

Homelab Deploy/Debug Patterns

When to read: before deploying or debugging ANY stack. Cross-cutting lessons (distilled 2026-06). Stack-specific lessons live alongside (komodo.md, n8n.md, etc.).

Related: verify-stack-workflow.md — 3-layer verification (km container → Loki scan → km-logs) for post-deploy health checks.

Meta-pattern: Silent Config Divergence

The #1 source of bugs: a config change that silently has no effect because the live system reads from a DIFFERENT source than the file you edited. Always identify the live config source before trusting an edit.

After manual SSH edits on a server — pull back to git (MANDATORY)

When you edit a file directly on a server via SSH (Caddy route on unraid, mktxp config, Grafana compose, anything in a Komodo-managed repo), the live state now diverges from git. The next Komodo deploy-stack (triggered by ANY future sync) will silently overwrite your manual edit with the stale git version.

Rule: immediately after any manual config edit on a remote server:

scp root@<server>:<path> <local-repo-path>   # pull back
git add <file> && git commit -m "sync: <what changed>" && git push

Evidence: 3+ occurrences in single session (Caddy route, mktxp config, Grafana compose) — each would have been lost on next deploy.

Edit file Live source Divergence trap
litellm/config/*.yaml (router_settings, litellm_settings, …) DB LiteLLM_Config (overrides YAML) YAML edit ignored if DB row exists — delete the row to govern via YAML. model_list is YAML-only (safe to edit).
Komodo TOML env BOTH TOML + compose.yaml Env in only one = silent failure. [[komodo-env-two-tier-rule]]
nanobot/config/config.json (git) appdata runtime copy (pre_deploy copies) Edit git → deploy (pre_deploy runs) → live. Editing appdata directly is wiped on next deploy.
Service IP may be physical LAN vs Tailscale Physical subnet (e.g. 10.10.5.x) may not route to all hosts; Tailscale 100.x is the reliable cross-server path. Test reachability, don't assume.

Rule: verify live state empirically (curl/psql/spend-logs/health) before trusting config files or docs. Spec ≠ reality (e.g. GLM-5.2 reasoning_effort low/med/high/max all map to "high" except max — only an empirical test reveals it).

Service addressing — IP-resilience (avoid LAN-IP breakage)

Hardcoded LAN IPs (192.168.x / 10.10.5.x) in service-to-service config break on subnet migration (the IP Migration Runbook pain). Address services by stability:

Address type Stable across LAN change? Use for
Tailscale IP (100.x) / MagicDNS ✅ stable cross-server refs (sync→other-host service)
Container name (docker DNS) ✅ stable (IP-independent) same-host, same-network refs
DNS rewrite (svc.home) ✅ if rewrite centralized services reached via Caddy/AdGuard
LAN IP (192.168/10.10.5) ❌ changes ONLY macvlan-assignment + client-DNS config

Rules:

  1. Service-to-service URLs/env → never hardcode LAN IP. Cross-server → Tailscale; same-host → container name (shared bridge net).
  2. LAN IPs only where unavoidable (macvlan ipv4_address, client DNS list) → keep in ONE place.
  3. On subnet change: update the few macvlan-assignment/client-DNS spots; service config is immune.

Case — adguard-sync (2 adguards on 2 servers): sync (dev) talks to home (unraid) + replica (dev). Was ORIGIN=10.10.5.249:3003 + REPLICA=10.10.5.245:3004 (LAN IPs, + replica unreachable: bridge↔macvlan isolation). Fix: ORIGIN=100.68.251.84:3003 (Tailscale, cross-server) + REPLICA=adguard-replica:3000 (container name via shared adguard-mgmt bridge, same-host). Sync config now has zero LAN IPs → subnet-change-immune.

Quick-reference — which address to use (verified 2026-06):

Scenario Correct address Example Why
Container → container, same Docker network Docker DNS name pigsty-postgres:5432 Docker embedded DNS, zero config, always resolves
Container → container, DIFFERENT server Tailscale IP + host-mapped port 100.68.251.84:3100 Cross-server requires routable IP; Tailscale stable across LAN changes
host-network container → localhost 127.0.0.1:<port> 127.0.0.1:3003 host-net shares host network namespace
External client → service Domain via Caddy/Cloudflare kavita.minhluc.info TLS termination + stable public URL
OTel scrape target on same server Docker DNS name litellm-litellm-1:4000 NOT Tailscale IP — avoids port mapping confusion + auth from wrong source IP
OTel scrape with auth Docker DNS + bearer_token litellm-litellm-1:4000 + bearer LiteLLM /metrics requires master key

Common mistakes (each observed in production):

  • container_short_name:port → wrong (Docker DNS uses full container_name, e.g. mktxp-exporter not mktxp)
  • tailscale_ip:container_port → wrong (use host-mapped port, not container-internal)
  • loki:3100 from dev server → fails (Docker DNS doesn't cross servers; use 100.68.251.84:3100)

Gotcha — bridge↔macvlan isolation: a bridge-network container cannot reach a macvlan-network container (macvlan = separate L2). To let them talk, put the macvlan container ALSO on a shared bridge (alongside its macvlan for the bypassed port, e.g. DNS:53). Adding a network needs container recreation → destroy_before_deploy=false can't apply it (flip true once, or docker compose up manually).

Gotcha — Komodo + macvlan deploy: Komodo deploy-stack of a macvlan container often reports FAILED (4s) even though docker compose up works locally + the container runs fine. Treat as a Komodo/macvlan quirk: verify the container actually runs (docker ps, DNS, health) rather than trusting the Komodo status. Redeploy via docker compose up if needed.

Deploy

GitOps one-way (MANDATORY)

edit → commit → push → run-sync <category> → deploy-stack (ALWAYS). run-sync is mandatory before every deploy — even compose-only changes need sync to pull latest repo state to periphery. Skipping run-sync → stale config → deploy uses OLD compose.yaml silently. Evidence: 3+ failed deploys in single session from forgotten run-sync (Kavita compose, Caddyfile, OTel config).

Exception: if only image tag changed (no file change), deploy-stack alone suffices (Komodo re-pulls image on deploy).

km 2.2.0 tooling (verified)

  • No native km logs → km-logs wrapper (~/.local/bin/km-logs; km-logs -S <stack>, km-logs <container> -n 50 -g error). NOT a km subcommand.
  • km exec <c> <shell> needs a TTY (fails non-interactive) → use ssh -o BatchMode=yes root@<tailscale-ip> 'cmd' for one-shot server commands, or read appdata files.
  • km inspect <container> exposes env incl. secrets (LITELLM_MASTER_KEY, service tokens) → useful for direct API testing. Use sparingly (it's a secret leak surface).
  • Direct psql to litellm DB: PGPASSWORD=minhluc1 psql -h 100.68.251.84 -p 5439 -U postgres -d litellm (pigsty port 5439, NOT in litellm container).

Verify after deploy (multi-layer)

  1. Health endpoint (curl …/health).
  2. Spend logs (LiteLLM_SpendLogs table — proves a real model call happened).
  3. Actual API/chat call (not just "deploy SUCCESSFUL").
  4. Log inspection (km-logs). Don't stop at "deploy SUCCESSFUL" — silent misconfig deploys as "successful".

Debug

LiteLLM model/credential issues

  • Models missing → check model_list is YAML + the zai/credential resolves.
  • Fallbacks not working → check router_settings lives in DB (delete row → YAML governs).
  • A model serves but wrong behavior → spend log model column shows what actually ran.

Uptime Kuma socketio (sync-uptime-kuma.py)

  • add acks → use sio.call("add", …) (returns {ok,msg,monitorID}).
  • editMonitor / deleteMonitor do NOT ack → sio.emit(…) fire-and-forget (sio.call times out). Emit works (verified URL/type update).
  • monitorList on connect = monitor CONFIG only (status=None); live status via separate status / importantHeartbeatList events (multi-arg → handle with *args).
  • Orphan monitors: removing a stack does NOT clean its UK monitor. services.yaml is the source of truth; the sync deletes monitors not in it. Wired as pre-commit hook.

Hook conflicts (bd + pre-commit)

  • bd owns core.hooksPath (.beads/hooks) → pre-commit install is REFUSED.
  • Chain a specific pre-commit hook into .beads/hooks/pre-commit AFTER the bd markers (bd regenerates only its markered section; content outside is preserved).
  • Guard with command -v pre-commit so clones without it skip gracefully.

Reasoning models (GLM-5.2)

  • Effort levels: only high (default) and max are distinct. low/medium map to high.
  • Empty reply with no error → reasoning tokens consumed the max_tokens budget (raise it; nanobot uses maxTokens 32768). reasoning_tokens counts toward completion.

AdGuard rewrite API (dedup gotcha)

  • POST /control/rewrite/delete {domain, answer} removes the entire rewrite key (all instances of that domain+answer), NOT one instance. "Dedup by deleting duplicates" deletes the rewrite outright → briefly broke the *.home wildcard (Caddy). To clean duplicates: delete the key, then POST /control/rewrite/add once. Verify resolution after.
  • Duplicates accumulate from adguardhome-sync adding on each run without dedup — harmless (AdGuard returns first match) but noisy. Cleanest: fix at the source.

Quick reference

# LiteLLM live config
PGPASSWORD=minhluc1 psql -h 100.68.251.84 -p 5439 -U postgres -d litellm \
  -c "SELECT param_name FROM \"LiteLLM_Config\" ORDER BY 1;"          # what DB overrides
# Recent model calls (proves routing)
PGPASSWORD=minhluc1 psql -h 100.68.251.84 -p 5439 -U postgres -d litellm \
  -c "SELECT \"startTime\",model,model_group FROM \"LiteLLM_SpendLogs\" ORDER BY \"startTime\" DESC LIMIT 5;"
# Non-interactive server cmd (km exec needs TTY)
ssh -o BatchMode=yes root@100.68.251.84 'docker logs nanobot-gateway --tail 20'
# UK monitor status
uv run scripts/sync-uptime-kuma.py --dry-run     # shows drift (orphans/missing/stale)

Related: [[komodo]] deploy verification, [[replacing-stacks]] pre-replacement checklist, [[n8n]] task-runner gotchas, [[nanobot]] MCP cross-container + workspace rule lifecycle. Memory: [[litellm-db-config-override]], [[uptime-kuma-sync-setup]].