Verify Stack Workflow¶

Compact checklist cho verify: deploy-checklist.md §5. File này là deep reference cho queries/commands cụ thể khi cần debug sâu.

When to read: trước ANY "check stack" / "verify stack" / "kiểm tra" request. Covers both container health and log-level verification across the 2-server homelab (unraid + development).

Related docs:

homelab-deploy-debug-patterns.md — meta patterns for deploy/debug
komodo.md — Komodo CLI usage, km commands
../infra/komodo-api-reference.md — full API endpoints
Grafana Testing Rule — browser-verify panels before reporting done
Monitoring Pipeline Recovery — triage broken monitoring

Principle¶

docker ps healthcheck only says "container is running" — NOT "container is working correctly". Always scan logs for errors too. 2 layers minimum, 3 for deep-dive.

Layer 1: Container healthcheck (quick, 10s)¶

Priority: Komodo CLI first, SSH only as fallback.

1a. km container (preferred — no SSH, both servers)¶

km container -s unraid        # all running containers on unraid
km container -s development   # all running on dev
km container -s unraid -a     # include stopped

Output includes: name, state, server, ports, networks, image.

1b. docker ps (fallback — when km not available or need raw format)¶

ssh root@unraid 'docker ps --format "{{.Names}} {{.Status}}"'
docker ps --format '{{.Names}} {{.Status}}'   # dev server (local, no SSH)

What to check¶

Running count matches expected (compare with last known good)
Any unhealthy, restarting, or exited containers
Uptime — recently restarted containers may indicate crash loops

NOT sufficient alone — "Up healthy" ≠ working correctly.

Layer 2: Loki log scan (thorough, 15s)¶

1 query scans ALL containers across BOTH servers. This catches issues that healthcheck misses (DB connection failures, scrape timeouts, API 404s, OOM warnings).

Query: all errors in last 1 hour¶

curl -s -G 'http://100.68.251.84:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={server=~"unraid|development"} |~ "(?i)(error|fatal|panic|exception|failed|denied|timeout)" !~ "(?i)(no error|error_handler|error_rate|error.*0|OnError)"' \
  --data-urlencode "limit=50" \
  --data-urlencode "start=$(date -d '1 hour ago' +%s)000000000" \
  --data-urlencode "end=$(date +%s)000000000"

Query: errors for specific container¶

curl -s -G 'http://100.68.251.84:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={container="kavita"} |= "error"' \
  --data-urlencode "limit=20" \
  --data-urlencode "start=$(date -d '1 hour ago' +%s)000000000" \
  --data-urlencode "end=$(date +%s)000000000"

Query: errors for specific stack¶

curl -s -G 'http://100.68.251.84:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={compose_project="monitoring"} |= "error"' \
  --data-urlencode "limit=20" \
  --data-urlencode "start=$(date -d '1 hour ago' +%s)000000000" \
  --data-urlencode "end=$(date +%s)000000000"

Grafana UI alternative¶

http://grafana.home/ → Explore → Loki → paste LogQL query. Useful for visual timeline + log context.

Exclude false positives¶

The regex !~ "(?i)(no error|error_handler|error_rate|error.*0|OnError)" filters out common benign matches. Extend as needed per stack.

Layer 3: Deep-dive specific container (only when Layer 2 flags issues)¶

Priority: Komodo CLI first, SSH only as fallback.

3a. km-logs (preferred — no SSH, both servers)¶

km-logs <container> -n 50           # last 50 lines
km-logs <container> -n 100 -t       # with timestamps
km-logs <container> -g error -g fail # grep AND search
km-logs -S <stack> [service] -n 50  # stack-level logs

Auth: reads ~/.config/komodo/komodo.cli.toml.

3b. docker logs (fallback — SSH into server)¶

ssh root@unraid "docker logs <container> --tail 50 2>&1 | grep -i error"
docker logs <container> --tail 50 2>&1 | grep -i error   # dev server (local)

3c. km-exec (non-interactive container exec)¶

km-exec <container> <cmd...>           # auto-detect server
km-exec -s unraid <container> <cmd...> # explicit server
km-exec -s unraid --host <cmd...>      # run on server host (not in container)

Caddy route verification¶

Caddy returns HTTP 200 with an empty body for any unmatched host (no matching site block). So a bare status-code check is ambiguous:

Add: curl … = 200 does not prove the route is live (the catch-all also returns 200).
Remove: curl … ≠ 200 never holds (unmatched hosts still return 200).

Right check — compare against a known-nonexistent host:

for h in <service>.home zz-nonexistent.home; do
  curl -s -o /dev/null -w "$h -> %{http_code}|%{size_download}bytes\n" \
    -H "Host: $h" http://192.168.100.10/
done

Route live: <service>.home differs from zz-nonexistent.home (non-empty body / app headers).
Route gone: both lines identical (200|0bytes).

Source: verified during AMUD removal (2026-06) — amud.home and zz-nonexistent.home both returned 200|0bytes after the route block was deleted from applications.caddyfile.

Observability stack reference¶

Component	Port	Role	Server
VictoriaMetrics	8428	Metrics storage (Prometheus-compatible)	unraid
Loki	3100	Log storage (30d retention)	unraid
Grafana	3002	UI for metrics + logs + SQL dashboards	unraid
RisingWave	4566	Streaming SQL (13 MVs, PostgreSQL wire)	development
Lakekeeper	8181	Iceberg REST catalog (Rust, STS enabled)	development
OPA	8282	Policy enforcement for Trino queries	development
Trino	8091	SQL federation over Iceberg + RisingWave	development
Redpanda	9092	Kafka message broker (platform.logs/events/metrics)	development
Push API	8093	Universal DWH ingestion (dp-chat-adapter)	development
ntopng	3004	DPI traffic analysis (NetFlow → custom Lua endpoint)	development
OTel Collector	4317/4318	Scrapes exporters → VM	unraid
Promtail	—	Docker logs → Loki (label: `server=unraid`)	unraid
Promtail-dev	—	Docker logs → Loki via Tailscale (label: `server=development`)	development

API quick reference¶

# VictoriaMetrics — PromQL query
curl 'http://100.68.251.84:8428/api/v1/query?query=<PROMQL>'

# VictoriaMetrics — list all metric names
curl 'http://100.68.251.84:8428/api/v1/label/__name__/values'

# Loki — LogQL query
curl -G 'http://100.68.251.84:3100/loki/api/v1/query_range' \
  --data-urlencode 'query=<LOGQL>'

# Loki — list all labels
curl 'http://100.68.251.84:3100/loki/api/v1/labels'

# Grafana UI
http://grafana.home/    # internal (LAN)
https://grafana.minhluc.info/  # external (Cloudflare)

# RisingWave — SQL query (PostgreSQL wire)
PGPASSWORD=postgres psql -h 192.168.100.31 -p 4566 -U root -d dev -c "SHOW TABLES"
PGPASSWORD=postgres psql -h 192.168.100.31 -p 4566 -U root -d dev -c "SELECT count(*) FROM platform_logs"

# Push API — health
curl -s http://192.168.100.31:8093/api/v1/health

# Lakekeeper — health + warehouse
curl -s http://192.168.100.31:8181/health
curl -s http://192.168.100.31:8181/management/v1/warehouse

# OPA — health + policy test
curl -s http://192.168.100.31:8282/health
curl -s -X POST http://192.168.100.31:8282/v1/data/trino/allow -H "Content-Type: application/json" -d '{"input":{"action":{"operation":"execute_query"},"resource":{},"identity":{"user":"test"}}}'

# Trino — test query
docker exec dp-trino trino --execute "SELECT count(*) FROM gravitino_iceberg.dev_streaming.platform_metrics"

Grafana dashboard testing¶

MANDATORY: Before reporting a dashboard as complete, test EVERY panel via browser. See .claude/rules/grafana-testing.md for full procedure.

Quick test:

$HOME/.claude/skills/.venv/bin/python3 << 'PYEOF'
from patchright.sync_api import sync_playwright
import time
with sync_playwright() as p:
    browser = p.chromium.launch(args=['--no-sandbox', '--disable-gpu'])
    page = browser.new_page(viewport={'width': 2560, 'height': 1440})
    page.goto('http://192.168.100.59:3002/d/<UID>/<slug>?from=now-24h&to=now', timeout=15000)
    time.sleep(12)
    text = page.inner_text('body')
    no_data = text.count('No data')
    print(f"'No data': {no_data}")
    print(f"{'✅ PASS' if no_data == 0 else '❌ FAIL'}")
    browser.close()
PYEOF

Key gotchas:

Datasource type must be grafana-postgresql-datasource (not postgres) for Grafana 13+
Each panel target needs datasource: {uid: "...", type: "grafana-postgresql-datasource"}

### Scrape targets (OTel Collector)

| Job                 | Target                   | Metrics                                     |
| ------------------- | ------------------------ | ------------------------------------------- |
| adguard_home        | 100.68.251.84:9618       | DNS query stats, processing time            |
| litellm             | 100.68.251.84:4001       | LLM proxy requests, tokens, latency         |
| mktxp               | mktxp:49090              | MikroTik router stats (traffic, interfaces) |
| logporter           | logporter-exporter:9333  | Docker log metrics                          |
| unraid              | 100.68.251.84:9100       | Node exporter (CPU, RAM, disk, network)     |
| crawl4ai            | crawl4ai:11235           | Web crawler metrics                         |
| ha-growatt-exporter | ha-growatt-exporter:9101 | Solar inverter stats                        |

## Decision tree

```mermaid
graph TD
    START["User: check stack / verify stack"]
    START --> L1["Layer 1: km container -s &lt;server&gt;<br/>(or docker ps)"]
    L1 --> L1Q{"Any unhealthy / restarting?"}
    L1Q -->|Yes| L3NOTE["Note for Layer 3"]
    L1Q -->|No| L2["Layer 2: Loki query<br/>(all errors, last 1h)"]
    L3NOTE --> L2
    L2 --> L2Q{"Any errors found?"}
    L2Q -->|Yes| L2NOTE["Note containers + severity"]
    L2Q -->|No| DONE["All clear"]
    L2NOTE --> L3["Layer 3 (if issues found)"]
    L3 --> L3A["km-logs &lt;container&gt; -g error (preferred)"]
    L3 --> L3B["docker logs &lt;container&gt; (fallback via SSH)"]