Deployment Guide¶

Operational procedures for deploying and managing services via GitOps.

See DEPLOYMENT.md for the full workflow. See komodo.md for Komodo reference.

GitOps Workflow (MANDATORY)¶

All changes MUST go through git. Never deploy via SSH, Komodo UI, or CLI directly.

Edit local files → git push → Komodo sync → Deploy

Compose-Only Changes¶

When only compose.yaml or compose.env changes (no TOML edits):

git add -A && git commit -m "chore(<stack>): <message>" && git push && echo "" | km x deploy-stack <name>

TOML Changes or New Stacks¶

When Komodo TOML files change or a new stack is added:

git add -A && git commit -m "chore(<stack>): <message>" && git push && echo "" | km x run-sync <category> && echo "" | km x deploy-stack <name>

Rules¶

Always push + deploy together — changes must be live before moving on
TOML changes need sync first — Komodo must pick up new/changed configs
Verify deploy result — check for SUCCESSFUL vs FAILED
Never push without deploying — stranded commits create drift

Adding a New Stack¶

Checklist tổng quan (pre-flight + naming + secrets + docs updates): deploy-checklist.md §1 + §2. Section dưới đây là how-to đầy đủ với commands/TOML template.

Step 1: Create Compose File¶

mkdir -p stacks/{category}/{service}

Create stacks/{category}/{service}/compose.yaml following conventions:

Services listed alphabetically
Ports double-quoted
security_opt: ["no-new-privileges:true"]
Same-network services use container names
GPU access: mount /dev/dri for hardware transcoding

Step 2: Create Komodo TOML¶

Copy from an existing stack as template:

[[stack]]
name = "service-name"
deploy = true
tags = ["server:unraid", "category:{category}"]

[stack.config]
server = "unraid"
poll_for_updates = false
auto_update = true
destroy_before_deploy = true
auto_update_all_services = false
linked_repo = "docker-compose"
run_directory = "/etc/komodo/repos/docker-compose/stacks/{category}/{service}/"
file_paths = ["compose.yaml"]
environment = """
TZ=Asia/Ho_Chi_Minh
"""

Step 3: Add Tags¶

server:{name} — target server (always)
category:{name} — service category (always)
saas:tailscale — if exposed via Tailscale
selfhost:{name} — if replacing a cloud service
app:{name} — application identifier

Step 4: Deploy¶

git add -A && git commit -m "feat({service}): add new stack" && git push && echo "" | km x run-sync {category} && echo "" | km x deploy-stack {service-name}

Removing a Service¶

Checklist compact: deploy-checklist.md §6. Section dưới đây là procedure đầy đủ với commands.

Complete checklist để remove một service khỏi infrastructure.

Step 1: Stop + Destroy Container¶

echo "" | km x stop-stack <name>
echo "" | km x destroy-stack <name>

Step 2: Remove Files¶

rm -rf stacks/{category}/{service}
rm komodo/stacks/{category}/{service}.toml

Step 3: Update Homepage Dashboard¶

Edit stacks/applications/homepage/config/services.yaml — remove service entry từ relevant group.

Step 4: Update Documentation¶

Kiểm tra và cập nhật các file docs:

# Tìm tất cả references
grep -rn "<service-name>" docs/

Files cần update:

docs/codebase-summary.md — remove khỏi Services Inventory table, giảm count
docs/project-overview-pdr.md — remove khỏi Service Registry table, giảm counts (category + total)

Step 5: Commit + Push + Sync + Deploy¶

git add -A && git commit -m "chore: remove <service> stack" && \
git pull --rebase origin main && git push origin main && \
echo "" | km x run-sync {category} && \
echo "" | km x deploy-stack homepage

Lưu ý: run-sync {category} cần thiết để Komodo xóa TOML definition. Homepage deploy để cập nhật dashboard.

Step 6: Verify¶

km list stacks — service không còn trong danh sách
http://100.68.251.84:8088 — Homepage dashboard không còn service entry
Trên unraid: container không còn chạy

Adding a New Category¶

Create stacks/{new_category}/{service}/compose.yaml
Add [[resource_sync]] entry in komodo/stacks/root_syncs/resource_sync.toml:

[[resource_sync]]
name = "new_category"
[resource_sync.config]
linked_repo       = "docker-compose"
resource_path     = ["./komodo/stacks/new_category"]
include_resources = true
pending_alert     = true
webhook_enabled   = false

Deploy with sync:

git add -A && git commit -m "feat(infra): add {new_category} category" && git push && echo "" | km x run-sync root-syncs

Non-YAML Config Files¶

Stacks that need extra config files (Caddyfile, grafana provisioning) must use pre_deploy.command — do NOT add non-YAML files to file_paths (Core parses them as YAML).

pre_deploy.command = """
#!/usr/bin/env bash
set -eu pipefail
mkdir -p /etc/komodo/repos/docker-compose/stacks/{category}/{service}
cp ./Caddyfile /etc/komodo/repos/docker-compose/stacks/{category}/{service}/Caddyfile
"""

Media Stack Deploy Notes¶

Jellyfin (Hardware Transcoding)¶

Requires /dev/dri device mount in compose
Select VAAPI as transcoding method in Jellyfin settings
Transcoding logs: check docker logs for iHD driver errors
If transcoding fails: verify ls /dev/dri/renderD128 exists on host

Navidrome¶

Scans /mnt/user/music/ automatically on startup
No special deploy steps beyond standard compose

Airflow Deploy Notes¶

DAG changes: Push to git → scheduler auto-pulls within 5 min (GitDagBundle)
Compose/TOML changes: Standard run-sync orchestration && deploy-stack airflow
Python package changes: Rebuild image on development server, then deploy
Key requirement: linked_repo must be set — periphery needs repo files for GitDagBundle
PAT token: Stored in Komodo Variable GITEA_AIRFLOW_TOKEN

ntfy Deploy Notes¶

Server: Development (100.126.172.96:9282)
Image: binwiederhier/ntfy:latest
Auth: deny-all default, admin user managed via Komodo Variable
User provisioning: NTFY_AUTH_USERS env var in compose (non-interactive, no setup.sh needed)
Topics: komodo (Komodo alerts + Uptime Kuma), unraid, mikrotik
Subscribe (read): Anonymous — no auth required for clients
Publish (write): Admin only — credentials managed via Komodo Variable

AdGuard HA Deploy Notes¶

Architecture¶

Primary: Unraid (192.168.100.59:53, :3003) — main DNS server
Replica: Development (192.168.100.229:53, :3004) — backup DNS server
Sync: Development (adguard-sync stack) — one-way primary→replica every 30min
Failover chain: Primary → Replica → Cloudflare (1.1.1.1, internet-only)

Sync Configuration¶

Tool: bakito/adguardhome-sync:latest
Direction: One-way (primary → replica only)
Interval: 30 minutes (configured in cron schedule)
Auth: Uses default credentials on replica (local-only access)
Sync items: DNS filters, rewrites, services, query log config

DHCP Configuration¶

Both MikroTik subnets use: 192.168.100.59, 192.168.100.229, 1.1.1.1

Tailscale DNS: Global nameservers set to 192.168.100.59, 192.168.100.229

Deployment Order¶

adguard-home (unraid) — primary must be deployed first
adguard-replica (development) — replica depends on primary API
adguard-sync (development) — sync service connects to both

Recovery Procedures¶

Primary down: Replica takes over automatically (DHCP failover). Changes made during outage will not sync back when primary recovers (one-way sync).

Replica down: Primary continues serving DNS. No action needed until replica is restored.

Sync lag: DNS changes made on primary take up to 30min to appear on replica. Plan changes accordingly when testing filters.

ntfy Topic Management¶

# Publish a test message (requires auth — use Komodo Variable credentials)
curl -u "$NTFY_USER:$NTFY_PASS" -d "test message" http://100.126.172.96:9282/komodo

# Subscribe (anonymous, no auth needed)
curl http://100.126.172.96:9282/komodo

# Create a new topic (just publish to it — ntfy auto-creates)
curl -u "$NTFY_USER:$NTFY_PASS" -d "init" http://100.126.172.96:9282/new-topic-name

# Add a new user (update NTFY_AUTH_USERS in compose.yaml, then redeploy)

Komodo Alerter + Uptime Kuma Integration¶

Komodo alerter (komodo/stacks/root_syncs/alerter.toml) sends to topic: komodo
Uptime Kuma sends all monitor alerts to topic: komodo via ntfy provider
Both use auth URL format: http://[[NTFY_USER]]:[[NTFY_PASS]]@100.126.172.96:9282/komodo (Komodo Variables)
Uptime Kuma needs extra_hosts: host.docker.internal:host-gateway for cross-server access

Recovery Procedures¶

Gitea Is Down¶

Gitea hosts the repo it depends on. Recovery:

SSH to unraid: ssh root@100.68.251.84
Edit files directly: /etc/komodo/repos/docker-compose/stacks/{category}/{service}/
Deploy via Komodo UI (periphery reads local files, no git needed)

Periphery can deploy using the last cloned files on disk. Risk: volume prune or deleting /etc/komodo/repos/.

Periphery Issues¶

Check periphery logs: km-logs komodo-periphery-{server}
Verify connectivity: curl http://{server-ip}:8120/health
Restart periphery: km restart stack komodo-periphery-{server}

Common Pitfalls¶

run_directory Double Path Bug¶

# WRONG — resolves to /etc/komodo/stacks/stacks/...
run_directory = "./stacks/{category}/{service}/"

# CORRECT — absolute path
run_directory = "/etc/komodo/repos/docker-compose/stacks/{category}/{service}/"

Port Conflicts¶

Unraid nginx occupies 80/443 on Tailscale IP — never bind these ports
Use unique ports: 3005, 3002, 2283, 8096, etc.

linked_repo Empty String¶

# WRONG — periphery can't find repo files
linked_repo = ""

# CORRECT — must reference the repo name
linked_repo = "docker-compose"

Cross-Server Communication¶

# WRONG — container names don't work across servers
redis:6379

# CORRECT — use Tailscale IP
100.68.251.84:6379

file_paths Non-YAML Files¶

# WRONG — Core parses ALL file_paths entries as YAML
file_paths = ["compose.yaml", "Caddyfile"]

# CORRECT — only YAML files in file_paths, non-YAML via pre_deploy
file_paths = ["compose.yaml"]

Komodo CLI Quick Reference¶

km list                       # List all resources
km list stacks                # List stacks only
km list servers               # List servers only
km deploy stack <name>        # Deploy a stack
km restart stack <name>       # Restart a stack
km-logs -S <name>          # View stack logs
km server-stats <name>        # Server CPU/memory/disk
km x run-sync <category>      # Trigger resource sync
km x deploy-stack <name>      # Deploy (pipe empty for confirmation)
km run procedure <name>       # Run a procedure

Core address: http://100.126.172.96:9120 (Tailscale IP, not Cloudflare)

DEPLOYMENT.md — Full deployment workflow
komodo.md — Complete Komodo guide
system-architecture.md — Network layout

Deployment Guide¶

GitOps Workflow (MANDATORY)¶

Compose-Only Changes¶

TOML Changes or New Stacks¶

Rules¶

Adding a New Stack¶

Step 1: Create Compose File¶

Step 2: Create Komodo TOML¶

Step 3: Add Tags¶

Step 4: Deploy¶

Removing a Service¶

Step 1: Stop + Destroy Container¶

Step 2: Remove Files¶

Step 3: Update Homepage Dashboard¶

Step 4: Update Documentation¶

Step 5: Commit + Push + Sync + Deploy¶

Step 6: Verify¶

Adding a New Category¶

Non-YAML Config Files¶

Media Stack Deploy Notes¶

Jellyfin (Hardware Transcoding)¶

Navidrome¶

Airflow Deploy Notes¶

ntfy Deploy Notes¶

AdGuard HA Deploy Notes¶

Architecture¶

Sync Configuration¶

DHCP Configuration¶

Deployment Order¶

Recovery Procedures¶

ntfy Topic Management¶

Komodo Alerter + Uptime Kuma Integration¶

Recovery Procedures¶

Gitea Is Down¶

Periphery Issues¶

Common Pitfalls¶

run_directory Double Path Bug¶

Port Conflicts¶

linked_repo Empty String¶

Cross-Server Communication¶

file_paths Non-YAML Files¶

Komodo CLI Quick Reference¶

Related Documentation¶