Skip to content

Deploy Checklist

Single source of truth cho mọi checklist deploy/replace/verify/remove stack. Mọi file khác (stack-reuse-guide, replacing-stacks, verify-stack-workflow, deployment-guide) chỉ giữ reference ngược về đây cho phần checklist thuần túy.

Khi nào dùng

Tình huống Section
Sắp tạo stack mới §1 + §2
Đang định thay thế / migrate một stack có sẵn §3
Vừa add xong một stack mới §4
Vừa deploy xong, cần verify §5
Cần gỡ bỏ một service §6

1. Pre-flight — trước khi tạo stack mới

  • Đi qua decision tree trong stack-reuse-guide.md cho mỗi layer service cần (DB / REST / vector / file storage / queue / auth / proxy / monitoring).
  • Đọc row liên quan trong shared-services catalog (cùng file) — không add capability thứ 2 cho cùng mục đích.
  • Reuse DB đúng cách:
  • Pigsty → tạo dedicated DB + user + GRANT ALL ON SCHEMA public TO {user}, không dùng postgres DB mặc định.
  • Supabase → tạo dedicated schema + role (xem supabase-connection-playbook.md).
  • Service name có prefix — không để rest, auth, db, storage trần trên shared network.
  • Nếu trùng feature với service có sẵn: ghi 1 dòng lý do trong header compose.yaml giải thích vì sao cả 2 phải tồn tại.
  • Không thêm reverse-proxy / vector-db / auth service mới (Caddy + pgvector/Qdrant + Better Auth đã cover).
  • Secret dùng Komodo Variables; thỏa mãn two-tier env rule (xem stack-manager skill).
  • Nếu cần truy cập Tailscale: thêm sidecar tailscale-{service} (pattern trong stack-reuse-guide.md).
  • DNS rule: bridge/custom network MUST có dns: [192.168.100.59, 192.168.100.229, 1.1.1.1].
  • File ownership (unraid): user: "99:100" + chown trong pre_deploy.
  • Nếu service có external dependency (S3/DB/queue): verify endpoint với CLI (mc/rclone/aws/psql) TRƯỚC KHI config service.
  • Inter-container connections (S3, DB, queue): dùng Docker DNS (http://service:port), KHÔNG dùng *.home (Caddy macvlan unreachable từ bridge).
  • Port conflict check: verify port chưa bị chiếm trên target server trước khi deploy. Check existing containers: km container -s <server> | grep -oP ':\d+' | sort -un. Ports thường conflict: 8080, 8088, 9001, 9101 (Prometheus). If unsure, use port > 18000.
  • Config files: KHÔNG bind-mount từ /opt/appdata/ cho config files (Komodo Periphery container rootfs). Dùng entrypoint printf inline hoặc pre_deploy.command copy. See komodo.md § bind-mount.

2. Adding a new stack — steps

Tham khảo đầy đủ commands/TOML template: deployment-guide.md §"Adding a New Stack" và stack-manager skill.

  • mkdir -p stacks/{category}/{service}/ + tạo compose.yaml (services alphabetical, ports double-quoted, security_opt: ["no-new-privileges:true"]).
  • Tạo komodo/stacks/{category}/{service}.toml (copy template, set linked_repo = "docker-compose", run_directory absolute path).
  • Tags: server:{name} + category:{name} (luôn); thêm saas:tailscale / selfhost:{name} / app:{name} nếu áp dụng.
  • Non-YAML config (Caddyfile, grafana provisioning) → dùng pre_deploy.command, không đưa vào file_paths.
  • Deploy one-liner: 
    git add -A && git commit -m "feat({service}): add new stack" && git push && \
  echo "" | km x run-sync {category} && echo "" | km x deploy-stack {service-name}
  • Update docs/codebase-summary.md service inventory counts.
  • Update homepage stacks/applications/homepage/config/services.yaml + chạy scripts/sync-uptime-kuma.py.
  • Homepage URL convention: dùng Tailscale IP (100.x.x.x:port) cho tất cả href và siteMonitor. KHÔNG dùng LAN IP (192.168.100.x) — chỉ work trong LAN. *.home DNS cũng KHÔNG dùng — không resolve từ ngoài LAN.
  • dev server: 100.126.172.96
  • unraid: 100.68.251.84
  • Services dùng Tailscale HTTPS: <name>.tail1137c.ts.net

3. Replacing / migrating a stack — MANDATORY trước khi replace

STOP. Không replace trước khi check hết 4 mục. Anti-pattern: research image/port → replace ngay → phát hiện thiếu feature → revert (xem case MeTube vs Cobalt trong replacing-stacks.md).

  • Feature parity — list toàn bộ feature hiện tại (subscriptions, auto-download, disk save, API...) → verify replacement cover từng cái → nếu thiếu, hỏi user confirm trước.
  • Integration points — volume share với service nào / container-name DNS dependencies / data flow phụ thuộc (vd: Navidrome scan thư mục MeTube ghi).
  • Data flow — input → output, format file, DB schema, state files, config files.
  • Config compatibility — reuse được port / volume path / network hay không.

3.1 Catalog / database migration (nếu thay Iceberg catalog, DB engine, etc.)

  • Register existing data — tables/data trên object storage KHÔNG move. Chỉ cần re-register metadata tại catalog mới (Lakekeeper API: POST /catalog/v1/{wid}/namespaces/{ns}/register).
  • STS configuration — enable sts-enabled: true on warehouse for browser preview (DuckDB WASM).
  • Trino config — update iceberg.rest-catalog.uri + iceberg.rest-catalog.warehouse. Test SELECT before decommissioning old catalog.
  • DB migration to pigsty — create dedicated DB + user + GRANT ALL ON SCHEMA public before running migrate command.
  • Network reachability — Trino and catalog must be on same Docker network OR use Tailscale/LAN IP (not container hostname).

Process: Research → Feature comparison table → Confirm user → Implement → Deploy → Verify.

4. Post-addition review — sau khi add xong (MANDATORY)

  • Feature overlap — stack mới có replace/duplicate service nào không.
  • Integration potential — vd: add Uptime Kuma → phải add monitor cho tất cả service cũ, không chỉ cái mới.
  • Shared resources — consolidate volume/network/config được không.
  • Docs consistency — Homepage dashboard, Uptime Kuma monitors, tất cả docs có reference service cũ.

5. Verify sau deploy — 3 layer

Chi tiết queries/commands: verify-stack-workflow.md.

  • Layer 1 — Container health (10s): km container -s <server>. Running count đúng, không có unhealthy / restarting / exited. Chưa đủ — "Up healthy" ≠ working.
  • Layer 2 — Loki log scan (15s): 1 query quét hết container trên 2 server, filter error|fatal|panic|exception|failed|denied|timeout. Bắt DB conn fail / scrape timeout mà healthcheck miss.
  • Layer 3 — Deep-dive (chỉ khi Layer 2 flag): km-logs <container> -g error -g fail.
  • Caddy route check: so sánh %{http_code}|%{size_download} giữa <service>.home vs zz-nonexistent.home (Caddy trả 200 rỗng cho host không match → status-code đơn thuần không đủ).

6. Removing a service

Procedure đầy đủ với commands: deployment-guide.md §"Removing a Service".

6.1 Core removal

  • stop-stack + destroy-stack.
  • Verify orphan containers: docker ps -a | grep <name> — nếu còn container, docker rm -f <name>. Destroy-stack qua Komodo có thể bỏ sót container vật lý.
  • rm -rf stacks/{category}/{service} + rm komodo/stacks/{category}/{service}.toml.

6.2 Komodo cleanup

  • Check pre_deploy scripts — grep service name in all komodo/stacks/**/*.toml pre_deploy.command blocks. Remove stale mkdir, cp, chown lines. (aria2 lesson: pre_deploy recreated config dir → container respawned after deploy.)
  • Remove backup files — find . -name "*.bak" → delete stale .bak compose/TOML files.
  • Push + run-sync — git push then trigger Komodo sync for affected categories.

6.3 Docker cleanup

  • Remove Docker image: docker rmi <image> — reclaim disk space. Verify: docker images | grep <name> = empty.
  • Remove Docker volumes (if any): docker volume ls | grep <name> → docker volume rm.

6.4 Database cleanup

  • DROP dedicated database on pigsty/supabase: DROP DATABASE <name>; DROP USER <name>;.
  • Verify: psql -c "SELECT datname FROM pg_database WHERE datname='<name>'" = empty.

6.5 Monitoring cleanup

  • Homepage: Remove entry in services.yaml (href + siteMonitor). Deploy homepage.
  • Uptime Kuma: Delete monitor in Uptime Kuma UI (if exists).
  • Grafana: Delete dashboards via API (DELETE /api/dashboards/uid/<uid>). Delete datasources if service-specific. Sync to Unraid provisioning.
  • vmalert rules: Check alerts.yml for stale references.

6.6 Final verification

  • km list stacks — service not listed.
  • docker ps -a | grep <name> — empty.
  • docker images | grep <name> — empty.
  • Homepage — no entry, no dead siteMonitor.
  • Grafana — no stale dashboards.
  • grep -rn "<service>" stacks/ komodo/ docs/ — no stale references (excluding changelog/ADR).

7. Migrating a stack between servers

Full checklist: migrate-stack-between-servers.md.

7.1 Generic migration

  • Pre-flight verify: SSH vào target server, check RAM/disk/connectivity (RAM, ports, network). Say "I don't know" if can't verify.
  • Config changes: Docker DNS (pigsty-postgres, redis) → Tailscale IP (100.68.251.84:5439, 100.68.251.84:6379). TOML server tag → target server.
  • Deploy on target: run-sync + deploy-stack. Stop (NOT destroy) on source for rollback.
  • Verify: container running, health check, DB/Redis connections, functional test.
  • Decommission source: destroy-stack on source (only after verify green).
  • Cleanup: homepage, Caddy, CF tunnel, Uptime Kuma, docs.

7.2 Komodo-managed stacks (Core/Periphery migration)

Komodo has self-management quirks not covered by generic migration.

  • Copy Core private key: core.key + core.pub from source → target. New key = Periphery auth failure (all servers show unreachable). 
    docker exec <periphery> cat /path/to/config/keys/core.key | ssh target "cat > /path/config/keys/core.key"
  • destroy_before_deploy = false on the stack TOML. If true, docker compose down kills the Periphery that Core needs to communicate with → stack shows unhealthy → cascade failure.
  • Docker network alignment: target Periphery must be on reverse-proxy network (or same network as Core). If Periphery is on bridge, connect it: 
    docker network connect reverse-proxy <periphery_container>
  • KOMODO_FIRST_SERVER: use http:// not https:// (Periphery has SSL_ENABLED=false by default). Use container name on same network (e.g., http://komodo-periphery:8120).
  • Compose split: if migrating Core only, update source compose to periphery-only before redeploying. Old compose with core + periphery services → deploy failure on source.
  • FerretDB (if DB also migrated): CREATE EXTENSION documentdb CASCADE required after PG init. Add command: ["postgres", "-c", "cron.database_name=ferretdb"] to compose.
  • Verify both servers online: after Core move, check Komodo UI → Resources → Servers. Both source and target Peripheries must show green.

Common pitfalls (xem đầy đủ trong deployment-guide.md)

  • run_directory phải absolute (/etc/komodo/repos/...), không dùng ./stacks/... (double path bug).
  • linked_repo phải = "docker-compose", không được rỗng.
  • file_paths chỉ chứa YAML; non-YAML đi qua pre_deploy.command.
  • Cross-server: dùng Tailscale IP (100.x), không dùng container name.
  • Không bind port 80/443 trên Tailscale IP (Unraid nginx chiếm).