System Architecture¶

Infrastructure layout, network topology, and service communication patterns.

High-Level Architecture¶

graph TD
    DEV["Developer Workstation
(Tailscale client, git, km CLI)"]
    DEV -->|git push| GITEA["Gitea
source control
unraid:3005"]
    DEV -->|km CLI| KCORE["Komodo Core
deployment orchestrator
unraid:9120"]
    GITEA <-->|webhook| KCORE
    KCORE -->|localhost| FDB["FerretDB :27017
(MongoDB-compatible)
unraid"]
    KCORE -->|deploy commands| KPERU["Komodo Periphery (unraid)
unraid:8120"]
    KCORE -->|deploy commands| KPERD["Komodo Periphery (development)
development:8120"]
        KPERU --> DCU["Docker Compose — 49 stacks
Unraid NAS"]
        KPERD --> DCD["Docker Compose — 20 stacks
Dev VM"]

Network Topology¶

graph TD
    subgraph TN["Tailscale Tailnet (WireGuard encrypted)"]
        subgraph UNR["unraid — 100.68.251.84"]
            UNRS["Komodo Core :9120 · FerretDB :27017 · Gitea :3005
Grafana :3002 · RustFS :9000 · Homepage :8088
AdGuard Primary :53 · Uptime Kuma :3004
LiteLLM :4001 · Caddy :80 · Pigsty PG :5439
VictoriaMetrics :8428 · ... 30 more"]
        end
        subgraph DEV["dev — 100.126.172.96"]
            DEVS["Komodo Periphery :8120 only
Redpanda :9092 · RisingWave :4566
Trino :8091 · Lakekeeper :8181 · OpenFGA :8082 · OPA :8282
ntopng :3004 · mikrotik-tools · Push API :8093"]
        end
    end
    NOTE["Client devices access via Caddy reverse proxy (*.home)
Internal *.home via Caddy (macvlan 192.168.100.10)
Public: 100.68.251.84 is CGNAT — unreachable"]
    TN -.-> NOTE

Key Network Decisions¶

• All inter-client traffic encrypted by Tailscale WireGuard

Domain Architecture: Public vs Internal¶

graph LR
    subgraph PUB["Public flow"]
        INET[Internet] -->|*.minhluc.info| CF[Cloudflare] -->|Auth0 SSO| CFT["CF Tunnel"] --> SVC1[Service]
    end
    subgraph INT["Internal flow"]
        LAN["LAN / Tailscale"] -->|*.home| AG["AdGuard
192.168.100.59 / 229"] --> CAD["Caddy
192.168.100.10:80"] --> SVC2[Service]
    end
    subgraph EDGE["Edge compute (ADR-0002 + ADR-0003)"]
        WR[Cloudflare Worker] -->|Hyperdrive binding| HDC[Workers VPC connectivity dir] -->|WS tunnel| CDB[supabase-db:5432 on supabase-internal net]
        WR -.->|Workers AI binding (future)| AI[CF Workers AI @ edge GPU]
    end
    AI -.->|embeddings/cache| WR

Edge Compute Layer (added 2026-07)¶

Cloudflare Workers stack integrated via Infrastructure-as-Code + Komodo:

Edge → Internal flow (Worker → Hyperdrive → Supabase PG):

1. Worker A receives HTTP request at CF POP
2. Reads env.HYPERDRIVE.connectionString (synthetic, points to nearby CF pool)
3. Hyperdrive pool opens TCP 5432 to supabase-db via Workers VPC service + cloudflared tunnel
4. PostgreSQL on unraid serves (TLS self-signed, verify disabled)

Why this matters: Workers get edge caching (60s TTL) + connection pooling near homelab origin — eliminates per-request TLS handshake from user → tunnel → PG. Composes cleanly with Litellm gateway (LiteLLM proxies Workers AI similarly for client apps).

• *.home not exposed to internet — Tailscale is the auth layer, no SSO needed
• To expose a service publicly: route via *.minhluc.info → CF tunnel → Auth0
• AdGuard DNS rewrites only affect *.home — public domains resolve via Cloudflare
• Caddy on macvlan (192.168.100.10) listens only on LAN interface, not WAN-accessible

DNS High Availability Architecture¶

graph TD
    CLIENT[Client Devices] -->|query| AG1["AdGuard Primary
192.168.100.59:53"]
    AG1 -->|fallback if unavailable| AG2["AdGuard Replica
192.168.100.229:53"]
    AG2 -->|fallback if unavailable| CF["Cloudflare DNS
1.1.1.1:53
internet-only, *.home=NXDOMAIN"]
    SYNC["AdGuard Sync
every 30min, one-way primary→replica
→ DNS filter changes propagate with up to 30min lag"]

DNS Components¶

DNS Configuration¶

DHCP Configuration (MikroTik):

• Both subnets use: 192.168.100.59, 192.168.100.229, 1.1.1.1
• Primary DNS: AdGuard Primary (unraid)
• Secondary DNS: AdGuard Replica (development)
• Tertiary DNS: Cloudflare fallback (internet only)

Tailscale DNS:

• Global nameservers: 192.168.100.59, 192.168.100.229
• Overrides DNS for all tailnet devices
• Tailnet devices benefit from same failover chain

Sync Configuration¶

• Tool: bakito/adguardhome-sync
• Direction: One-way (primary → replica only)
• Interval: Every 30 minutes
• Sync items: DNS filters, rewrites, services, query log config
• Auth: admin:adminadmin (default, replica only accepts local connections)

Failover Behavior¶

Known limitation: DNS changes made on primary take up to 30min to appear on replica due to sync interval.

Server Layout¶

Unraid (100.68.251.84) — Primary Server¶

• Role: Main NAS and service host
• OS: Unraid OS 7.x
• Services: 49 stacks (media, databases, monitoring, core apps, networking, LLM, Komodo Core + FerretDB)
• GPU: Intel iGPU with VAAPI support (hardware transcoding)
• Storage: NFS mounts at /mnt/user/appdata/{service}/, media at /mnt/user/{music,youtube,tvshows}/
• API: GraphQL at https://unraid.tail1137c.ts.net/graphql

Development (100.126.172.96) — Secondary Server¶

• Role: Development services, Data Platform, LLM/AI workloads, AdGuard HA replica
• Services: 20 stacks (Data Platform, ntopng, mikrotik-tools, AdGuard replica+sync, etc.)
• Storage: Local volumes at /opt/appdata/{service}/
• Key services: Komodo Periphery (:8120), Redpanda (:9092), RisingWave (:4566), Trino (:8091), Lakekeeper (:8181), OpenFGA (:8082), OPA (:8282), ntopng (:3004), mikrotik-tools, Push API (:8093)

Data Platform Architecture¶

The data platform is a lakehouse pipeline on the development server. Gravitino → Lakekeeper migration is COMPLETE (see ADR-0001). The Gravitino container has been REMOVED.

graph TD
    subgraph Ingest["Ingestion (sole entry: Push API :8093)"]
        TG["Telegram Telethon
(dp-chat-adapter)"]
        MK["mikrotik-tools
ntopng scraper"]
        AD["Push API :8093
dp-chat-adapter container
HTTP /api/v1/events, /metrics, /logs, /chat"]
    end

    AD --> RP["Redpanda :9092
tmpfs 2GB RAM disk
zero disk writes"]
    TG --> AD
    MK --> AD

    RP --> RW["RisingWave v3.0.0 :4566
single_node persistent mode
8GB RAM (mem_limit)"]
    RW -->|"materialized views + sinks"| LK["Lakekeeper :8181
Iceberg REST catalog"]
    LK -->|"REST catalog"| RFS["RustFS :9000
s3://iceberg/"]
    LK -->|"warehouse=homelab"| TR["Trino :8091
catalog=gravitino_iceberg (legacy name)"]

    subgraph Authz["Authz stack"]
        FGA["OpenFGA :8082
fine-grained auth (future)"]
        OPA["OPA :8282
policy enforcement for Trino"]
    end
    OPA -.->|allow/deny| TR
    LK -.-> FGA

    subgraph DBT["dbt analytics (analytics/)"]
        STG["staging/
stg_chat_messages
stg_chat_messages_valuable"]
        MV["mvs/
5 materialized views"]
        STG --> MV
    end
    RW <-.->|dbt-risingwave| DBT

    HC["dp-health-check :60s loop
checks Push API, RW, Redpanda,
Lakekeeper :8181, Trino"] -.->|apprise tag:infra| AP["Apprise gateway"]

Components¶

Migration status: Gravitino → Lakekeeper (COMPLETE)¶

• Gravitino container REMOVED (folder 10-catalog/ retains only legacy gravitino.conf reference, not deployed)
• Lakekeeper deployed at 15-lakekeeper/ with OpenFGA auth backend
• OPA deployed at 16-opa/ as Trino access-control proxy
• Trino catalog still named gravitino_iceberg (legacy filename retained), iceberg.rest-catalog.uri=http://192.168.100.31:8181/catalog, iceberg.rest-catalog.warehouse=homelab
• Health checker now hits http://192.168.100.31:8181/management/v1/warehouse (was http://...:8090/api/metalakes/dev)

Push API — sole data entry point¶

dp-chat-adapter container exposes :8093 as the Push API (HTTP). ALL data types enter the platform through this endpoint:

Direct Kafka producers from outside the platform are NOT permitted — push through :8093 for schema validation and idempotency.

dbt staging layer¶

The dbt project at analytics/streaming/models/staging/ contains two staging models that clean raw Kafka data before downstream MVs consume it:

thread_category values:

Owner identification per provider:

• Zalo: msg_sender_name = 'Bạn' (Zalo's convention for "self")
• Telegram: all messages kept (already curated via WHITELIST_THREAD_IDS)
• Others: all kept

Redpanda tmpfs mode¶

Redpanda runs on a tmpfs 2GB RAM disk at /var/lib/redpanda/data. Trade-offs:

• ✅ Zero disk writes — eliminates btrfs CoW write amplification on dev server
• ✅ Faster topic produce/consume (in-memory)
• ⚠️ Data is lost on container restart (topics auto-recreate via auto_create_topics_enabled=true)
• ⚠️ Not suitable for production — dev-only acceptable because upstream Telegram session is replayable

RisingWave persistence¶

RisingWave runs in single_node persistent mode (image default CMD = single_node):

• Version: v3.0.0
• Volume: /mnt/user/appdata/risingwave:/root/.risingwave (persisted)
• Memory: mem_limit: 8g, mem_reservation: 6g
• Sources/MVs/sinks survive restarts — no need to re-bootstrap after restart (unlike the old playground mode)

Service Communication Patterns¶

graph LR
    subgraph SAME["Same Server, Same Network"]
        SA["Service A
redis:6379"] -->|container name| REDIS[(Redis)]
    end
    subgraph CROSS["Cross-Server"]
        CA["Service A
(unraid)"] -->|Tailscale IP:port| CB["Service B
(development)
100.126.172.96"]
    end

Rule: Prefer LAN IP (192.168.100.x) over Tailscale IP for internal service-to-service communication. Only use Tailscale IP when LAN is not reachable (e.g., from outside network). Never use public domains internally.

Docker Networks¶

Media Architecture¶

graph TD
    subgraph NAS["Unraid NAS — Media Stack"]
        MUSIC["/mnt/user/music/"] -->|scan| NAV["Navidrome :4533
(Subsonic)"]
        TV["/mnt/user/tvshows/"] --> JEL["Jellyfin :8096"]
        JEL <-->|/dev/dri VAAPI / iHD| IGPU["Intel iGPU (VAAPI)"]
    end

Media Stack Details¶

Monitoring Architecture¶

graph TD
    APP["Applications (OTLP)"] --> OTEL
    DB["Databases (OTLP)"] --> OTEL
    NET["Networking (Prom)"] --> OTEL
    OTEL["OTel Collector (unraid)"] --> MIMIR[Mimir
metrics]
    OTEL --> LOKI[Loki
logs]
    OTEL --> TEMPO[Tempo
traces]
    MIMIR --> GRAFANA["Grafana :3002"]
    LOKI --> GRAFANA
    TEMPO --> GRAFANA

Monitoring Stack Details¶

• Multi-file overlay: compose.exporters.yaml adds exporters for Unraid-specific services
• COMPOSE_PROFILES does NOT work with Komodo — use file overlay instead
• Uptime Kuma uses SQLite (does NOT support PostgreSQL) — monitors 32 checks across both servers

Notification Architecture¶

Two notification paths: direct to Apprise (Komodo, Uptime Kuma) and via Apprise gateway (monitoring scripts, health checks).

graph TD
    subgraph Alert Sources
        KOM["Komodo alerter"]
        UK["Uptime Kuma :3004"]
        MK["MikroTik scripts
(check_ports, dhcp_watch, VPN)"]
        ROS["ROS freshness-check
(metrics stale >120s)"]
        VMA["vmalert
(VictoriaMetrics rules)"]
        DPH["DP health-check
(data platform 60s)"]
    end

    subgraph "Apprise Gateway :8008"
        AP["Apprise
POST /notify/{tag}"]
    end

    subgraph Delivery
        APPR["Apprise :8008"]
        TG["Telegram DM"]
    end

    C1["Phone (Telegram app)"]
    C2["Browser (Telegram web)"]

    KOM -->|topic: komodo| APPR
    UK -->|topic: komodo| APPR

    MK -->|tag: infra| AP
    ROS -->|tag: ros-alerts| AP
    VMA -->|apprise-bridge :9093| AP
    DPH -->|tag: infra| AP

    AP --> APPR
    AP --> TG

    APPR --> C1
    APPR --> C2

Apprise Gateway¶

Central notification router — receives alerts via REST API and routes to Apprise + Telegram.

Apprise Tags¶

Direct Apprise Path (bypasses gateway)¶

Apprise Server¶

Alert Flow Summary¶

vmalert → apprise-bridge → Apprise¶

vmalert evaluates alerting rules in VictoriaMetrics and sends firing/resolved alerts to apprise-bridge (Alertmanager-compatible API on :9093). apprise-bridge enriches alerts with disk/WAN stats and deduplicates (notify once on fire, once on resolve) before forwarding to Apprise.

Homepage Dashboard¶

Central service dashboard at http://100.68.251.84:8088 (unraid).

• Image: ghcr.io/gethomepage/homepage:latest
• Config: stacks/applications/homepage/config/ (YAML configs copied via pre_deploy.command)
• Verified endpoints across categories for all deployed services
• Integrates with Docker API for live container status

Multi-Server URL Routing¶

Homepage uses Tailscale IPs and *.home domains to reach services on both servers:

Storage Architecture¶

Unraid (NFS):                          Development (Local):
/mnt/user/appdata/{service}/           /mnt/user/appdata/{service}/
├── gitea/                             ├── komodo/
├── immich/                            ├── bigquery-emulator/
├── grafana/                            ├── crawl4ai/
├── postgres/                           ├── clickhouse/
├── nextcloud/                          ├── redpanda/
├── homepage/                           ├── risingwave/
├── uptime-kuma/                        ├── apprise/
└── ...                                 └── ...

Tank mounts (NextCloud on Unraid):
/mnt/user/audiobooks/          → /tank/audiobooks
/mnt/user/podcasts/            → /tank/podcasts
/mnt/user/music/               → /tank/music
/mnt/user/isos/                → /tank/isos

Media libraries (Unraid):
/mnt/user/music/              # Navidrome scans
/mnt/user/downloads/          # Downloads
/mnt/user/tvshows/            # TV content for Jellyfin

File Management Architecture (NextCloud)¶

NextCloud replaces OpenList as the file management and WebDAV solution.

graph TD
    subgraph Unraid
        NC["NextCloud :80
(nextcloud:31-apache)"]
        PG["pigsty-postgres :5439
(nextcloud DB)"]
        REDIS["Redis
(cache)"]
        TANK["Tank mounts
audiobooks, podcasts,
music, isos"]
    end

    TS["Tailscale sidecar
(HTTPS :443)"]
    CADDY["Caddy
nextcloud.home:80"]

    NC --> PG
    NC --> REDIS
    NC -->|bind mount RW| TANK

    CADDY -->|reverse_proxy| NC
    TS -->|serve :443 → :80| NC

    CLIENTS["Tailscale clients:
https://nextcloud.tail1137c.ts.net
http://nextcloud.home"]
    TS -.-> CLIENTS
    CADDY -.-> CLIENTS

NextCloud Details:

Tailscale HTTPS:

NextCloud runs a Tailscale sidecar (tailscale-nextcloud) that serves HTTPS on port 443 via Tailscale Funnel/Serve. The serve.json config maps :443 → container :80.

Caddy route:

nextcloud.home:80 {
    request_body { max_size 10G }
    reverse_proxy nextcloud:80 {
        trusted_proxies private_ranges
        header_up X-Forwarded-Proto {scheme}
        header_up X-Real-IP {remote_host}
    }
}

Legacy: rclone Cloud Storage (Unraid)¶

rclone FUSE mounts may still exist on Unraid for cloud storage access (Google Drive, ProtonDrive). These are NOT exposed through NextCloud — they are standalone mounts on the Unraid host.

• Mount script: /boot/config/scripts/rclone/mount-cloud.sh (auto-start on boot)
• Health check: /boot/config/scripts/rclone/health-check.sh (cron + Apprise alerts)
• rclone config: /boot/config/plugins/rclone/.rclone.conf

Architecture Decision Records¶

ADR-001: Docker Swarm — Không áp dụng¶

Ngày: 2026-05-08 Trạng thái: ❌ REJECTED — Không cân nhắc lại

Lý do không chọn Docker Swarm¶

Khi nào nên reconsider¶

Chỉ khi CẢ HAI điều kiện sau đồng thời đúng:

• Unraid bị thay thế bởi pure Linux servers (3+ nodes)
• Cần zero-downtime rolling updates hoặc cross-node workload scheduling

Network Topology (Updated 2026-07-04)¶

Gateway: mik-main (RouterOS 7.23.1 x86)¶

graph TD
    INET["Internet (FPT PPPoE)"] --> ONT["ONT (bridge mode)"]
    ONT -->|ether2 WAN, MAC clone, PPPoE| MIK
    subgraph MIK["mik-main 192.168.100.1 — RouterOS 7.23.1 x86, 4GB RAM, Intel 4-core"]
        M1["ether2 = WAN PPPoE MTU 1492 MSS clamp
ether1+3+4 = bridge1 LAN 192.168.100.0/24
Services: REST API :80, Winbox :8291, API :8728
DHCP: pool .100-.199, 23 static leases
DNS: forward to AdGuard .59 + 1.1.1.1
QoS: 4-tier + PCQ · Firewall: 13 filter + 18 NAT + 15 mangle
NAT: masquerade + UDP 41641→unraid Tailscale + PS5 Remote Play (16 rules)
NetFlow v9 → 192.168.100.59:9995 · SNMP: public
UPnP: disabled"]
    end
    MIK -->|bridge1| UNR["unraid .59
49 Docker stacks
Komodo Core + Periphery
FerretDB · Grafana · RustFS"]
    MIK -->|bridge1| DEV["dev .31
21 Docker stacks
Komodo Core"]
    MIK -->|bridge1| PVE["pve .11
Proxmox VE
VMs: CHR .30, win11 .32"]

IP Allocation Table¶

Monitoring Stack¶

graph TD
    MIK["mik-main RB5009"] -->|NetFlow v9 :12055| NF2NG["netflow2ng"]
    NF2NG -->|ZMQ| NTOPNG["ntopng :3004
DPI + custom Lua endpoint"]
    NTOPNG -->|5min scrape| SCRAPER["scrape_ntopng.py
(mikrotik-tools)"]
    SCRAPER -->|push logs| PUSHAPI["Push API :8093
(dp-chat-adapter)"]
    PUSHAPI -->|Kafka| RP["Redpanda :9092"]
    RP -->|consume| RW["RisingWave :4566
13 MVs + 6 sources"]
    RW -->|PostgreSQL wire| GRAFANA["Grafana :3002
3 dashboards, 29 panels"]
    RW -->|Iceberg sink| LK["Lakekeeper :8181
(Rust, STS enabled)"]
    MIK -->|API :8728| MKTXP["mktxp-exporter :49090"]
    MKTXP -->|scraped by otel-collector 45s| VM["VictoriaMetrics :8428"]
    VM -->|queried by| GRAFANA
    MIK -->|REST API :80| MTOOLS["mikrotik-tools
18 scheduler jobs"]
    MTOOLS -->|metrics| PUSHAPI
    OPA["OPA :8282
policy enforcement"] -->|allow/deny| TRINO["Trino :8091"]
    LK -->|Iceberg REST| TRINO
    TRINO -->|query| RW

Tailscale Overlay¶

dev      100.126.172.96   ← Komodo Periphery, Data Platform containers
unraid   100.68.251.84    ← Komodo Core + Periphery, FerretDB, NAS
pve      100.114.144.55   ← Proxmox VE
zimaos   100.73.218.119   ← Remote site (113.187.62.194)

P2P: static NAT UDP 41641 → unraid (direct, no DERP relay)

Key Services on unraid (192.168.100.59)¶

Public Inbound Surface (PS5 Remote Play)¶

PS5 at 192.168.100.100 is excluded from VPN routing (direct WAN) AND has 16 inbound port forwards for Remote Play from internet:

These are the only inbound DSTNAT rules besides Tailscale (UDP 41641 → .59). Source restriction is in-interface=pppoe-out1 (any internet source). Document because they are a public attack surface that should be inventoried.

Domain-based VPN Exclusions¶

Some destinations bypass ProtonVPN routing via a mangle accept rule placed before vpn-zone-lan / vpn-zone-iot (the rules that mark traffic for VPN). Each exclusion has a corresponding address-list kept in sync by a scheduler job in mikrotik-tools.

Pattern: script fetches authoritative CIDR blocks (e.g., https://api.github.com/meta blocks web/api/git/github_actions), resolves extra domains (Fastly CDN, GHCR, Copilot) to /24 subnets, and idempotently syncs to MikroTik address-list via REST API. The mangle rule references the list by name, so list updates take effect immediately — no firewall reload needed.

Mangle rule (live):

/ip firewall mangle add chain=prerouting action=accept \
    dst-address-list=github-direct comment="vpn-skip-github" \
    place-before=[find comment="vpn-zone-lan"]

Current exclusions summary (devices + subnets + domains):

RouterOS REST API Notes¶

• Base URL: http://192.168.100.1/rest/
• Auth: Basic lucndm:minhluc1
• Singleton PATCH requires /*ID or script workaround
• Script pattern: create /system/script → run → delete
• network_mode: host required for flow-collector (preserves NetFlow source IP)

Related Documentation¶

• Design Guidelines — Why this architecture was chosen
• Deployment Guide — How to deploy and manage services
• Komodo Guide — Komodo platform documentation
• Migration Plan — FPT → MikroTik
• Lakekeeper Migration ADR — Gravitino → Lakekeeper decision
• MikroTik Tailscale — Tailscale on RouterOS