Skip to content

Airflow & Database Backup

Airflow DAG development workflow, GitDagBundle, podcast pipeline, and Databasus backup management.

Source: extracted from AGENTS.md

Airflow DAG & Plugin Development Workflow

Architecture: GitDagBundle (auto-sync DAGs from Git)

Airflow 3.x GitDagBundle pulls DAGs from GitHub repo (NOT Gitea — Gitea repo is archived). Scheduler auto-refreshes every 5 minutes (refresh_interval: 300).

Bundle config (in compose.yaml env vars):

  • Name: dags-folder (must match Airflow default — task runner expects this name)
  • Repo: https://github.com/minhluc-info/docker-compose.git
  • Subdir: stacks/orchestration/airflow/dags
  • Tracking: main branch
  • Token: Komodo Variable GITHUB_TOKEN (scope: repo)

Server paths:

  • Tracking repo: /tmp/airflow/dag_bundles/dags-folder/tracking_repo/
  • Versions: /tmp/airflow/dag_bundles/dags-folder/versions/{commit_hash}/
  • Bare: /tmp/airflow/dag_bundles/dags-folder/bare/

Deploy methods

What Location Deploy method Restart?
DAGs stacks/orchestration/airflow/dags/*.py Git push → scheduler auto-pulls within 5min ❌
Python packages requirements.txt Rebuild image + push + redeploy ✅
Compose/TOML compose.yaml / .toml km x deploy-stack airflow ✅

Development workflows

1. DAG-only changes (fully automatic — just push):

# Edit DAG file in stacks/orchestration/airflow/dags/
git add -A && git commit -m "feat(airflow): new DAG" && git push origin main
# Done — scheduler auto-pulls within 5 minutes
# To force immediate: docker exec airflow-scheduler airflow dags reserialize

2. Normal deploy (compose/TOML changes):

git add -A && git commit -m "chore(airflow): ..." && git push origin main
echo "" | km x run-sync orchestration          # sync TOML changes
echo "" | km x deploy-stack airflow            # deploy containers

3. New Python package (rebuild image):

# Edit requirements.txt on development server
ssh development
cd /tmp/airflow-build
rsync -av /path/to/stacks/orchestration/airflow/{Dockerfile,requirements.txt} .
docker build -t 100.68.251.84:3005/lucndm/airflow:latest .
docker push 100.68.251.84:3005/lucndm/airflow:latest
exit
echo "" | km x deploy-stack airflow

4. Edit-on-server (fast iteration):

# Edit DAG directly on server for testing
ssh root@100.68.251.84
# Find the latest version dir
ls /tmp/airflow/dag_bundles/dags-folder/versions/
# Edit the DAG file
vi /tmp/airflow/dag_bundles/dags-folder/versions/{hash}/stacks/orchestration/airflow/dags/podcast_download.py
# Force re-parse
docker exec airflow-scheduler airflow dags reserialize
# Test and iterate, then commit to git when done

Podcast Download Pipeline

DAG: podcast_download — Daily automated podcast management pipeline.

Schedule: 0 18 * * * (18:00 UTC = 01:00 UTC+7)

Task graph: discover_and_download → trigger_library_scan → cleanup_podcasts

Architecture:

YouTube channels → yt-dlp (audio + VTT + thumbnail) → ffmpeg (opus 96kbps)
  → metadata.json → mc cp S3 OpenList + local copy → ABS library scan → cleanup

Channels: Tangocquy (23 eps), BVVN (197 eps), Spiderum (~150 eps after Shorts filter)

Config:

  • Audio format: opus 96kbps (YouTube serves opus natively, ~40% smaller than m4a)
  • Shorts filter: duration > 120s at discover + --match-filter at download
  • Subtitles: en,vi VTT format (best effort — YouTube 429 rate limits subtitles)
  • S3 checkpoint: mc stat skips already-processed episodes on retry
  • tmpfs: 1GB /tmp named volume for temp downloads (avoids Unraid FUSE overhead)
  • Local copy: /podcasts/{channel}/{episode_name}/{episode_name}.opus for ABS/Jellyfin
  • S3 path: podcasts/{channel}/{year}/{episode}/ (backup/remote access)

Key env vars (in compose.yaml):

  • S3_ENDPOINT, S3_ACCESS_KEY, S3_SECRET_KEY — OpenList S3 credentials
  • ABS_URL, ABS_TOKEN — Audiobookshelf API for library scan

ABS library: "YouTube Podcasts" (ID 61e735a3-c237-4a3f-b6b7-c772b69a3d4a, type=podcast, path /podcasts)

Jellyfin library: "Podcasts" (ID 608e1ed1e97ee5c4556f3d0299ac9894, Music type, path /data/podcasts)

Key files

File Purpose
stacks/orchestration/airflow/compose.yaml 2 services (apiserver, scheduler), LocalExecutor, UID 99:100, iGPU mount, tmpfs
stacks/orchestration/airflow/Dockerfile Custom image: ffmpeg, mc (MinIO client), yt-dlp, chmod o+rX for UID 99
stacks/orchestration/airflow/requirements.txt Python packages: yt-dlp>=2025.5.0, apache-airflow-providers-git
stacks/orchestration/airflow/dags/podcast_download.py Main DAG: 3 tasks, S3 checkpoint, local copy, Shorts filter, cleanup
komodo/stacks/orchestration/airflow.toml Komodo config: server unraid, S3/ABS env vars via Komodo Variables
Docker image: 100.68.251.84:3005/lucndm/airflow:latest Custom image with podcast deps

Database Backup (Databasus/Postgresus)

Databasus (container name: postgresus) — self-hosted backup management cho PostgreSQL, MySQL, MariaDB, MongoDB.

Detail Value
Web UI http://100.68.251.84:4005
Auth admin / Admin@2026 (reset via DB: clear hashed_password, then /api/v1/users/admin/set-password)
Workspace Homelab (e1541210-7770-42fc-9fd9-0082cf8918a6)
Storage S3 via OpenList (http://100.68.251.84:5246/backups/databasus/)
Schedule Daily at 18:00 UTC (01:00 UTC+7)
Retention 3 months
Encryption None
Notifier ntfy via webhook (http://100.126.172.96:9282/databasus) — fires on scheduled backup completion/failure
Stack stacks/database/postgresus/compose.yaml

Database inventory (43 databases):

Instance Port Type Databases
100.68.251.84 5432 PostgreSQL 17 airflow, alist, anythingllm, blinko, cliproxyapi, coder, devpidb, firefly, firefly_pico, freshrss, gitea, hyperdrive, inboxzero, jellyfin, litellm, mem0db, mempalace, n8n, nanobot_v2, nanobot_v2_test, opencode_metrics, paperless, postgres, quickwit, romm, sftpgo, supabase, wiki, _supabase
100.68.251.84 5437 PostgreSQL 14 immich
100.68.251.84 5438 PostgreSQL 17 affine, lightrag, litellm, mem0db, nanobot, nanobot_test, openwebui, postgres, release_insights
100.68.251.84 3306 MariaDB 11 memos

DB credentials: postgres / minhluc1 (PostgreSQL), root / minhluc1 (MariaDB) — stored in Komodo Variable DB_PASSWORD.

API auth flow:

  1. POST /api/v1/users/signin → {"email":"admin","password":"..."} → JWT token
  2. All API calls: Authorization: Bearer <token> + ?workspace_id=<ws_id>

Password reset (if locked out):

-- Via internal PG (port 5437 inside container)
UPDATE users SET hashed_password=NULL, password_creation_time=NOW() WHERE email='admin';
-- Then: POST /api/v1/users/admin/set-password {"password":"<new>"}

Internal database: PostgreSQL 17 on port 5437 (inside container). Access: docker exec postgresus /usr/lib/postgresql/17/bin/psql -p 5437 -U postgres -d postgresus

Pitfalls:

  • storePeriod enum values: "WEEK", "MONTH", "3_MONTH", "6_MONTH" (NOT "THREE_MONTH")
  • storageId not saved properly via /backup-configs/save API — must update directly in backup_configs table
  • MARIADB type uses key "mariadb" in create payload (NOT "mysql")
  • pgdata permissions: chmod 700 /mnt/user/appdata/postgresus/data/pgdata required after volume creation