Lessons: Deploying with Komodo¶

Patterns and pitfalls when deploying stacks via Komodo GitOps workflow.

Source: extracted from AGENTS.md

Lessons Learned: Deploying with Komodo¶

Patterns và pitfalls cụ thể khi deploy stacks qua Komodo GitOps workflow.

`destroy_before_deploy` + Stateful Services¶

destroy_before_deploy = true recreate containers nhưng DB data giữ nguyên (trên external PostgreSQL/volume)
Nếu service store encrypted data trong DB (Fernet key, JWT secret), key PHẢI fixed qua Komodo Variable — không đổi giữa deploys
Key thay đổi → tất cả encrypted data trong DB thành garbage → InvalidToken / Signature verification failed
Rule: Fernet keys, JWT secrets → Komodo Variable, set một lần, KHÔNG bao giờ đổi

`destroy_before_deploy = false` Limitations¶

Stacks dùng false (networking: gluetun, adguard-home/sync/replica; qmd) — không destroy để tránh downtime / break deps (DNS/VPN). Nhưng false KHÔNG apply được changes yêu cầu container recreation:

Network change (add/join network): deploy FAILS fast (~6s, container unchanged). qmd case: thêm reverse-proxy network → deploy failed.
Volume / mount change: không recreate → không apply.
UID / user change: same.
"created" không start: đôi khi container kẹt ở "created" sau deploy (openlist/ariang case) → start thủ công.

Khi cần apply recreation-requiring change trên stack false:

Tạm flip destroy_before_deploy = true cho lần deploy đó (rồi revert), HOẶC
Workaround không recreate — vd Caddy reverse_proxy <host-ip>:<port> thay container name (qmd: 192.168.100.59:8183 thay qmd:8183).

Rule: deploy FAIL fast + container unchanged trên stack false → nghi ngay recreation-required change (network/volume/UID).

`file_paths` Limitation¶

Periphery chỉ fetch files listed trong file_paths — KHÔNG clone full repo
DAGs, plugins, config files KHÔNG được copy tự động
pre_deploy.command chạy trên server nhưng repo files không có trên disk (chỉ file_paths files tồn tại)
Solutions:
scp DAGs/plugins lên host volume manually
Embed trong Docker image (rebuild khi thay đổi)
Git-clone trong pre_deploy.command (không recommend — circular với Gitea)

Volume Ownership After `destroy_before_deploy`¶

Container recreated → host volume mounts giữ ownership cũ
Nếu image chạy non-root user, host dir root:root → Permission denied
Fix: docker run --rm -v /path:/data alpine chown -R <UID>:0 /data
Prevent: Add chown vào pre_deploy.command

Deploy Verification Quirks¶

Komodo có thể report EXECUTION FAILED trong khi containers actually running
Luôn verify qua: docker ps, health check endpoints, km-logs -S <name>
Long-running init containers (DB migration) có thể timeout Komodo deploy — nhưng task vẫn complete

Corrupted Encrypted Data Recovery¶

Khi encrypted data trong DB không decode được (Fernet key mismatch sau redeploy):
CLI commands crash: airflow connections list → InvalidToken
API endpoints crash: /api/v2/connections → 500 Internal Server Error

Fix: Delete corrupted rows trực tiếp qua SQLAlchemy, KHÔNG qua CLI/API:

from airflow import settings
from sqlalchemy import text
session = settings.Session()
session.execute(text('DELETE FROM connection'))
session.commit()

Recreate với current Fernet key

Multi-Process Config Consistency¶

Nhiều containers (scheduler, worker, apiserver) cần cùng signing keys
Nếu config derive random key khi empty (e.g., jwt_secret), mỗi process generate key khác nhau → signature mismatch
Rule: CHO MỌI distributed service, set explicitly tất cả shared secrets/secrets trong compose environment

Debugging Checklist¶

Step	Command	Purpose
0	`km list stacks -n <name> -f json \\| python3 -c "..."`	Xác định server nào đang chạy stack
1	`km container -s <server> -a \\| grep <name>`	Check container state
2	`ssh <server> "docker inspect <name> --format '{{.State.ExitCode}}'"`	Get exit code
3	`ssh <server> "docker logs --tail 100 <name>"`	Full logs (not compose truncated)
4	`ssh <server> "docker run --rm ... <image> --debug"`	Debug mode
5	`ssh <server> "docker exec postgres-... psql -U postgres -d <db> -c '<query>'"`	Check DB data

Server map: | Server | SSH | Tailscale IP | |--------|-----|-------------| | unraid | ssh root@100.68.251.84 | 100.68.251.84 | | development | ssh root@100.126.172.96 | 100.126.172.96 |

`pre_deploy.command` Flattening¶

Komodo flattens pre_deploy.command into bash -c "cmd1 && cmd2 && ..." — single line with && separators
NEVER use: #!/usr/bin/env bash, set -eu pipefail, multi-line if/fi blocks
Use instead: one-line guards [ -d "./dir" ] && cp ... || true
Bash builtins (set, source) cannot be chained with && — causes syntax error near unexpected token '&&'

Pattern:

# CORRECT
pre_deploy.command = """
mkdir -p /path/config
rm -rf /path/config/*
cp -r ./config/* /path/config/
test -d ./extra && cp -r ./extra/* /path/extra || true
ls -lha /path/config/
"""

Stale `file_paths` After Removal¶

When removing a file from file_paths in TOML AND deleting it from repo, Komodo DB retains stale deployed_contents and missing_files
run-sync alone does NOT clear stale entries — the deploy still fails with "Missing files: ..."
Fix sequence: run-sync <category> → destroy-stack <name> → deploy-stack <name>
destroy-stack clears deployed_contents so the next deploy starts fresh
clear-repo-cache + RefreshStackCache API also do NOT fix this

Batch Deploy vs `sync-and-deploy`¶

sync-and-deploy procedure deploys ALL stacks with hash mismatch simultaneously
With 50+ stacks, this can crash Komodo Core (OOM or connection exhaustion)
Prefer: deploy stacks in batches of 5-8 using km x -y deploy-stack <name> loops
After Komodo Core crash, restart via: cd stacks/development/komodo && docker compose up -d

Komodo Core Crash Recovery¶

Komodo Core runs as container komodo-core on development server (local)
Periphery container (periphery) may also exit when Core crashes

Recovery:

cd stacks/development/komodo && docker compose up -d   # restarts both core + periphery

Wait for port 9120 to respond before running any km commands
If Core keeps crashing on deploy, reduce batch size

Never Delete Stacks Without Git History Check¶

A stack without TOML in current repo may have been intentionally removed (check git log --diff-filter=D)
Always verify via git log --all --diff-filter=D --name-only -- "**/stackname*" before declaring "orphaned"
Ask user before deleting stacks via API — DeleteStack is irreversible

LiteLLM Config Include Shallow Merge¶

LiteLLM _process_includes does shallow merge — config[key] = value overwrites, does NOT deep-merge nested dicts
Keys like mcp_servers MUST live at top-level in their own YAML file — proxy_server.py reads config.get("mcp_servers") directly
Nesting mcp_servers under litellm_settings causes silent failure: config loads but MCP manager never sees the servers, API returns empty tools
Rule: When LiteLLM source reads config.get("X"), that key must appear at top-level after all includes are resolved — check source if unsure

Read Library Docs Before Coding (lessons from chat-adapter session, 2026-07)¶

Root cause of 3/6 bugs: Jumping straight to code without reading library API docs. Result: trial-and-error debugging cycle (code → runtime error → fix → repeat) instead of correct-first implementation.

Bugs that docs-seeker / context7 would have prevented:

Bug	Library	What docs would have shown
`offset_id=None` TypeError	Telethon `iter_messages()`	API signature: `offset_id` must be int, not None
`asyncio.run()` event loop reuse	Telethon FAQ	"Event loop must not change after connection" — well-documented
`create_table_if_not_exists` missing	RisingWave Iceberg sink	Source code + docs mention this required parameter

Rule: Before implementing ANY code that calls an external library API:

Check docs-seeker skill or context7 for the specific function/class
Read the API signature, parameter types, and known caveats
Verify Python version compatibility (e.g., kafka-python broken on 3.12)
Only THEN write the implementation

Anti-pattern: # write code based on assumptions → deploy → read error logs → fix → redeploy (cost: 3-5 iterations × build+deploy time)

Pattern: # docs-seeker <library> <function> → verify API → write correct code → deploy once

RisingWave Single-Node Persistent Mode — Memory Requirement (2026-07)¶

Root cause: RisingWave single_node / standalone modes panic at compactor/src/server.rs:122 when container memory < 8GB. The compactor asserts compactor_memory_limit_bytes > 2 × (sstable_size_mb + block_size_kb) after subtracting 128MB meta cache.

The math (why 3-4GB always panics):

single_node allocates compactor = total_memory / 8
3GB → 375MB → ×0.8 proportion → −128MB cache = 172MB → 172 < 512 → PANIC
4GB → 500MB → ×0.8 → −128MB = 272MB → 272 < 512 → PANIC
8GB → 1000MB → ×0.8 → −128MB = 672MB → 672 > 512 → OK

Why playground works: playground forces hummock+memory state store → compactor is skipped entirely → assertion never reached.

Official minimum: 8GB container / 1GB compactor (from docker/README.md memory table).

Fix: mem_limit: 8g + default CMD single_node + volume mount /root/.risingwave. State persists across restarts (embedded SQLite meta + local Hummock SSTs). rw-bootstrap.sh only needed ONCE (first deploy).

Verified: Restart test confirmed sources (5), MVs (6), sinks (5) all survive restart. Source resumes from checkpoint, NOT from earliest offset.

Rule: RisingWave persistent mode requires minimum 8GB RAM container. Playground mode (3GB) is for smoke tests only.

Bind-Mount Invisible on Periphery Container (data-platform deploy, 2026-07)¶

Root cause: Komodo Periphery runs INSIDE a Docker container. pre_deploy.command writes to Periphery container's filesystem, but Docker bind-mounts in compose.yaml resolve against the HOST filesystem. Files written by pre_deploy to /opt/appdata/... exist in Periphery container but NOT on host → deployed container sees empty directory or Docker auto-creates empty dir.

Evidence: 5+ attempts with different paths (./conf/gravitino.conf, /opt/appdata/data-platform/gravitino/conf, /opt/appdata/data-platform/bootstrap) — ALL failed silently. pre_deploy ls shows files present, but container startup fails with "file not found" or "not a directory".

Workaround: Inline config via printf / heredoc in container entrypoint override:

entrypoint: ["/bin/sh", "-c"]
command:
  - >-
    printf '%s\n' 'key1 = value1' 'key2 = value2' > /etc/app/config.conf &&
    exec /original/entrypoint.sh

Rule: Do NOT use volumes: bind-mount for config files when Komodo Periphery runs as a container. Use entrypoint override with inline printf instead. Only use bind-mounts for data volumes (/opt/appdata/.../data) which exist on host before deploy.

Procedure TOML Format Gotchas (data-platform deploy, 2026-07)¶

Root cause: Komodo resource_sync silently rejects procedures.toml with generic "Found file errors. Cannot execute sync." — no detail about which file or which line.

Evidence: Complex procedure with nested [[procedure.config.stage]] + multi-line execution params rejected. Minimal procedure with only [[procedure]] + name + description accepted. Adding stages back in dotted-key format (execution.type = "RunStackService", execution.params.stack = "...") also works.

Rule:

Start with minimal [[procedure]] block (name + description only) to verify sync picks it up
Add stages one at a time, run-sync after each
Use dotted-key format for executions: { execution.type = "...", execution.params.X = "..." }
Do NOT use inline table format: { execution.params = { stack = "..." } } — may cause parse failure
run-sync logs show "adding resources from " — if your file ISN'T listed, it was rejected

Komodo Alerter Endpoint Types — No `Webhook` Variant (2026-07-18)¶

Root cause: Assumed type = "Webhook" was valid for Komodo Alerter config. Actual valid variants (v2.2): Custom, Slack, Discord, Ntfy, Pushover only. There is no Webhook variant.

Cascade impact (severe): One bad Alerter record in FerretDB → Komodo can't deserialize collection Alerter → Failed to load all resources by id cache every 15s → every RunSync execution fails because Komodo checks alerter cache before each sync → all 12 resource syncs display Failed state simultaneously.

Symptom signature:

ERROR Failed to load all resources by id cache | Failed to pull Alerters from mongo:
Kind: unknown variant `Webhook`, expected one of `Custom`, `Slack`, `Discord`, `Ntfy`, `Pushover`

Fix when DB already corrupted (chicken-and-egg): Sync can't run to update the alerter because sync itself errors on the bad alerter. Must delete directly in DB:

# FerretDB stores Mongo data in documentdb_data schema with CHECK(false) — direct SQL writes blocked.
# Must use Mongo wire protocol via temporary mongosh container:
ssh unraid 'docker run --rm --network=host node:22-alpine sh -c "
  npm install -g mongosh >/dev/null 2>&1 && \
  mongosh mongodb://ferretdb:ferretdb@127.0.0.1:27017/komodo --quiet \
    --eval \"db.Alerter.deleteMany({}); db.Alerter.find({}, {name:1}).toArray()\"
"'
# After deletion: fix TOML in repo, push, trigger root-syncs — alerter is recreated from TOML.

Rule: For HTTP POST to arbitrary webhook URL (e.g. Apprise), use type = "Custom":

[[alerter]]
name = "apprise"
[alerter.config]
enabled  = true
endpoint = { type = "Custom", params = { url = "http://apprise:8008/notify/infra" } }

Verify types from source, not docs: Komodo's enum variants live in komodo/core/src/entities/alerty.rs — the AlerterEndpoint::Custom|Slack|Discord|Ntfy|Pushover enum. Always check this file when configuring alerter TOML.

Recovery verification checklist:

docker logs komodo-core --since 1m | grep ERROR — empty = cache load fixed
Trigger root-syncs first (it owns all other syncs) — must show EXECUTION SUCCESSFUL
Verify alerter recreated via mongosh db.Alerter.find()
Note: other syncs may still show Failed state until resource_poll_interval (1h) refreshes their latest Update record — execution success ≠ state refresh if no resources changed