Skip to content

Data Platform — Bootstrap Playbook

Full rebuild instructions for the data-platform lakehouse stack. Use when: fresh deploy, server rebuild, or pipeline reset.

Migration status: Gravitino → Lakekeeper migration is COMPLETE (see ADR-0001). All Gravitino references below have been replaced with Lakekeeper.

Architecture overview

graph TD
    TG[Telegram Telethon] --> AD
    WC[WeChat future] --> AD
    FB[Facebook future] --> AD
    ZL[Zalo future] --> AD
    MK[mikrotik-tools
ntopng scraper] --> AD
    AD["Push API :8093 (50-adapters)
Sole entry point. HTTP /api/v1/{chat,events,metrics,logs}"]
    AD --> RP["Redpanda (40-streaming)
tmpfs 2GB RAM disk
Topics: chat.messages, cdc.events, platform_*"]
    RP --> RW["RisingWave v3.0.0 (40-streaming)
single_node persistent mode (8GB RAM)
Sources → MVs → Iceberg Sinks"]
    RW --> RFS["RustFS (unraid)
s3://iceberg/"]
    RW --> LK["Lakekeeper (15-lakekeeper) :8181
Iceberg REST catalog
warehouse=homelab"]
    LK -->|REST catalog| RFS
    LK -.->|auth backend| FGA["OpenFGA :8082"]
    OPA["OPA (16-opa) :8282
policy enforcement"] -.->|allow/deny| TR
    RFS --> TR["Trino (30-query) :8091
catalog=gravitino_iceberg (legacy name)"]
    TR --> SS["Superset (BI dashboards)
http://superset.tail1137c..."]
    HC["dp-health-check (60-monitoring)
60s loop"] -.->|apprise tag:infra| AP["Apprise gateway"]

Stacks (6 Komodo-managed)

Stack Folder Containers Purpose
data-platform-catalog 15-lakekeeper/ dp-lakekeeper, dp-lakekeeper-migrate, dp-openfga, dp-openfga-migrate Lakekeeper Iceberg REST catalog + OpenFGA auth
data-platform-opa 16-opa/ dp-opa OPA policy enforcement for Trino
data-platform-query 30-query/ dp-trino SQL query engine
data-platform-streaming 40-streaming/ redpanda, redpanda-console, dp-risingwave Kafka (tmpfs) + streaming DB (single_node persistent)
data-platform-adapters 50-adapters/ dp-chat-adapter (Push API + Telegram listener) Sole data entry point + Telegram ingestion
data-platform-monitoring 60-monitoring/ dp-health-check 60s health loop, Apprise infra tag alerts

10-catalog/ is legacy — only retains conf/gravitino.conf for reference. NOT deployed.

Prerequisites

  • Komodo Core + Periphery running on development (100.126.172.96)
  • RustFS running on unraid (192.168.100.59:9000) with ports 9000 + 19001 published
  • Pigsty PostgreSQL running on unraid (192.168.100.59:5439)
  • Komodo Variables set:
  • RUSTFS_ACCESS_KEY, RUSTFS_SECRET_KEY (existing admin)
  • RUSTFS_ICEBERG_ACCESS_KEY, RUSTFS_ICEBERG_SECRET_KEY (set to admin key values for dev)
  • DB_PASSWORD (pigsty superuser)
  • LAKEKEEPER_PG_PASSWORD (lakekeeper user in pigsty — currently hardcoded lakekeeper for dev)
  • OPENFGA_PG_PASSWORD (openfga user in pigsty — currently hardcoded openfga for dev)
  • TELEGRAM_API_ID, TELEGRAM_API_HASH (from https://my.telegram.org)
  • WHITELIST_THREAD_IDS (Telegram group IDs to ingest; DMs always included)

Phase 1: Deploy containers (GitOps)

# Sync TOMLs
echo "" | km execute run-sync data-platform

# Deploy stacks (order matters — Lakekeeper + OpenFGA must be up before Trino)
echo "" | km execute deploy-stack data-platform-catalog
echo "" | km execute deploy-stack data-platform-opa
echo "" | km execute deploy-stack data-platform-streaming
echo "" | km execute deploy-stack data-platform-query
echo "" | km execute deploy-stack data-platform-adapters
echo "" | km execute deploy-stack data-platform-monitoring

# Verify all running
km container -s development | grep -E "dp-|redpanda"
# Expected: dp-lakekeeper :8181, dp-openfga :8082, dp-opa :8282,
#           dp-trino :8091, redpanda :9092, redpanda-console :8085,
#           dp-risingwave :4566/:5691, dp-chat-adapter :8092/:8093,
#           dp-health-check (no port — host network loop)

Phase 2: Initialize pigsty databases (one-time)

Lakekeeper and OpenFGA each need their own database in pigsty-postgres. Schema migration is handled by the *-migrate init containers in 15-lakekeeper/compose.yaml (lakekeeper-migrate and openfga-migrate). You only need to create the databases

  • users.
# Run via Komodo API (RunStackService on pigsty):
python3 -c "
import os, urllib.request, json

url = 'http://100.126.172.96:9120/execute'
HEADERS = {
    'Content-Type': 'application/json',
    'x-api-key': os.environ['KOMODO_API_KEY'],
    'x-api-secret': os.environ['KOMODO_API_SECRET']
}

cmd = (
    f\"PGPASSWORD='{os.environ[\"DB_PASSWORD\"]}' psql -h pigsty-postgres -U postgres -d postgres -v ON_ERROR_STOP=1 \"
    f\"-c \\\"DO \\$\\$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'lakekeeper') THEN CREATE ROLE lakekeeper WITH LOGIN PASSWORD 'lakekeeper'; END IF; END \\$\\$;\\\" \"
    f\"-c \\\"DO \\$\\$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'openfga') THEN CREATE ROLE openfga WITH LOGIN PASSWORD 'openfga'; END IF; END \\$\\$;\\\" \"
    f\"-c \\\"SELECT 'CREATE DATABASE lakekeeper OWNER lakekeeper' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'lakekeeper')\\\" | grep -q CREATE && psql -h pigsty-postgres -U postgres -c 'CREATE DATABASE lakekeeper OWNER lakekeeper' || true \"
    f\"-c \\\"SELECT 'CREATE DATABASE openfga OWNER openfga' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'openfga')\\\" | grep -q CREATE && psql -h pigsty-postgres -U postgres -c 'CREATE DATABASE openfga OWNER openfga' || true \"
    f\"-c \\\"GRANT ALL ON SCHEMA public TO lakekeeper\\\" \"
    f\"-c \\\"GRANT ALL ON SCHEMA public TO openfga\\\"\"
)
data = {'type': 'RunStackService', 'params': {'stack': 'pigsty', 'service': 'postgres', 'no_deps': True, 'command': ['sh', '-c', cmd]}}
resp = urllib.request.urlopen(urllib.request.Request(url, data=json.dumps(data).encode(), headers=HEADERS), timeout=120)
print('DB init:', json.loads(resp.read()).get('_id', {}).get('\$oid'))
"

After this, restart data-platform-catalog so the migrate init containers re-run against the freshly created databases:

echo "" | km execute destroy-stack data-platform-catalog
echo "" | km execute deploy-stack data-platform-catalog

Verify: dp-lakekeeper logs show listening on :8181 and migrate containers exit 0.

Phase 3: Create RustFS bucket

docker run --rm --network reverse-proxy --entrypoint sh minio/mc:latest -c \
  "mc alias set rustfs http://192.168.100.59:9000 <RUSTFS_ACCESS_KEY> <RUSTFS_SECRET_KEY> --api S3v4 && \
   mc mb rustfs/iceberg 2>&1; mc ls rustfs/"

Phase 4: Bootstrap Lakekeeper warehouse + namespace

Lakekeeper uses a warehouse concept (not Gravitino's metalake). The Trino catalog config declares iceberg.rest-catalog.warehouse=homelab, so Lakekeeper must have a warehouse named homelab with a storage profile pointing at RustFS.

sleep 30  # Wait for Lakekeeper startup

# Create warehouse 'homelab' + namespace via Lakekeeper REST API (no container exec needed)
curl -fsS -X POST http://192.168.100.31:8181/management/v1/warehouse \
  -H 'Content-Type: application/json' \
  -d '{
    "warehouse-name": "homelab",
    "project-id": "00000000-0000-0000-0000-000000000000",
    "storage-profile": {
      "type": "s3",
      "bucket": "iceberg",
      "key-prefix": "homelab",
      "region": "us-east-1",
      "path-style-access": true,
      "endpoint": "http://192.168.100.59:9000",
      "access-key-id": "<RUSTFS_ICEBERG_ACCESS_KEY>",
      "secret-access-key": "<RUSTFS_ICEBERG_SECRET_KEY>"
    },
    "storage-credential": {
      "type": "s3",
      "credential-type": "access-key",
      "access-key-id": "<RUSTFS_ICEBERG_ACCESS_KEY>",
      "secret-access-key": "<RUSTFS_ICEBERG_SECRET_KEY>"
    }
  }'

# Create namespace dev_streaming under warehouse homelab
curl -fsS -X POST 'http://192.168.100.31:8181/catalog/v1/homelab/namespaces' \
  -H 'Content-Type: application/json' \
  -d '{"namespace": ["dev_streaming"]}'

Note: the project-id of all-zeros is Lakekeeper's default project when LAKEKEEPER__AUTHZ_BACKEND=allowall is set (dev mode).

Phase 5: Bootstrap Redpanda topics

Redpanda runs on tmpfs — topics are recreated automatically via auto_create_topics_enabled=true. This step is only needed if you want explicit partition/replica config.

timeout 30 script -qec "km exec redpanda sh" /dev/null << 'EOF'
for topic in cdc.events chat.messages platform_events platform_metrics platform_logs; do
  rpk topic describe "$topic" > /dev/null 2>&1 && echo "$topic exists" || \
  rpk topic create "$topic" --partitions 1 --replicas 1
done
rpk topic list
exit
EOF

tmpfs trade-off note: the 2GB RAM disk at /var/lib/redpanda/data is in-memory only. Topics + their data are lost on container restart. Acceptable for dev because Telegram session can replay and RisingWave re-consumes from the earliest available offset.

Phase 6: Bootstrap RisingWave pipeline

RisingWave runs in single_node persistent mode — sources, MVs, and sinks survive container restarts. You only need to run this phase ONCE per fresh volume, not after every restart.

The init script bootstrap/risingwave-init.sql declares all sources, MVs, and Iceberg sinks. The Iceberg sink config must point at Lakekeeper's REST endpoint (http://192.168.100.31:8181/catalog) and use warehouse=homelab.

# Step 1: Install psql in RisingWave container
timeout 180 script -qec "km exec dp-risingwave bash" /dev/null << 'EOF'
apt-get update -qq > /dev/null 2>&1 && apt-get install -y -qq postgresql-client > /dev/null 2>&1 && echo "psql ready"
exit
EOF

# Step 2: Create all pipelines (sources + MVs + Iceberg sinks via Lakekeeper)
# Source file: stacks/data-platform/compose/bootstrap/risingwave-init.sql
# Substitute ${S3_ACCESS_KEY} / ${S3_SECRET_KEY} with Komodo Variable values,
# and ensure catalog.uri points to Lakekeeper :8181 (NOT Gravitino :8090).
timeout 120 script -qec "km exec dp-risingwave bash" /dev/null << 'OUTER'
AK=$(curl -sS http://100.126.172.96:9120/read ... )  # substitute actual value
SK=$(curl -sS http://100.126.172.96:9120/read ... )  # substitute actual value

cat > /tmp/rw_init.sql << 'INNERSQL'
# Paste contents of risingwave-init.sql here with ${S3_ACCESS_KEY}/${S3_SECRET_KEY} substituted
# AND catalog.uri=http://192.168.100.31:8181/catalog AND warehouse=homelab
INNERSQL

psql -h localhost -p 4566 -U root -d dev -v ON_ERROR_STOP=1 -f /tmp/rw_init.sql 2>&1
echo "RC=$?"
psql -h localhost -p 4566 -U root -d dev -tAc "SELECT 'sources:' || count(*) FROM rw_catalog.rw_sources"
psql -h localhost -p 4566 -U root -d dev -tAc "SELECT 'mvs:' || count(*) FROM rw_catalog.rw_materialized_views"
psql -h localhost -p 4566 -U root -d dev -tAc "SELECT 'sinks:' || count(*) FROM rw_catalog.rw_sinks"
exit
OUTER

# Expected: sources:2, mvs:3+, sinks:2

Phase 7: Telegram login (one-time)

Chat adapter needs authenticated Telegram session.

# Switch to login mode
# Edit compose.yaml: MODE=login
# Redeploy: km execute deploy-stack data-platform-adapters
# Open: http://100.126.172.96:8092
# Enter: phone → verification code → 2FA password
# Switch back: MODE=listen
# Redeploy: km execute deploy-stack data-platform-adapters

Option B: CLI login

timeout 60 script -qec "km exec dp-chat-adapter bash" /dev/null << 'EOF'
python -m adapters.telegram_login
exit
EOF

Phase 8: Verify end-to-end

8a. Send a test message in any Telegram group

Open Telegram → send a message in any group you're a member of.

8b. Check adapter received it

km-logs dp-chat-adapter -s development -n 5
# Should show: no errors, adapter running

8c. Check Redpanda topic

timeout 30 script -qec "km exec redpanda sh" /dev/null << 'EOF'
rpk topic consume chat.messages --num 1
exit
EOF
# Should show JSON message with provider=telegram

8d. Check RisingWave

timeout 30 script -qec "km exec dp-risingwave bash" /dev/null << 'EOF'
psql -h localhost -p 4566 -U root -d dev -tAc \
  "SELECT count(*) FROM chat_messages"
exit
EOF
# Should show >= 1

8e. Check RustFS (Iceberg sink via Lakekeeper)

docker run --rm --network reverse-proxy --entrypoint sh minio/mc:latest -c \
  "mc alias set rustfs http://192.168.100.59:9000 <KEY> <SECRET> --api S3v4 && \
   mc ls rustfs/iceberg/homelab/dev_streaming/chat_messages/data/"
# Should show .parquet files

8f. Check health-check

km-logs dp-health-check -s development -n 5
# Should show: "OK (6 checks)" every 60s
# Checks: push-api, risingwave, redpanda-console, lakekeeper, trino, risingwave-pipeline

Post-restart recovery

Persistence model after Lakekeeper migration:

Component What's lost on restart Recovery needed
Lakekeeper catalog Nothing (pigsty-backed) Auto-recovers
OpenFGA Nothing (pigsty-backed) Auto-recovers
Redpanda (tmpfs) Topics + data (in-memory) Topics auto-recreate; RW re-consumes from start
RisingWave (single_node) Nothing (persisted to disk) Auto-recovers all sources/MVs/sinks
Chat adapter Nothing (session file persisted) Auto-reconnects

Recovery sequence after full server restart:

# 1. Wait for stacks to auto-start (restart: unless-stopped)
sleep 120

# 2. Verify Lakekeeper + OpenFGA migrate containers completed:
km-logs dp-lakekeeper-migrate -s development -n 20
km-logs dp-openfga-migrate -s development -n 20

# 3. Verify Trino can see Iceberg tables via Lakekeeper:
curl -X POST http://100.126.172.96:8091/v1/statement \
  -H "X-Trino-User: user" \
  -d "SHOW TABLES IN gravitino_iceberg.dev_streaming"

# 4. Verify RisingWave pipeline still has sources:
timeout 30 script -qec "km exec dp-risingwave bash" /dev/null << 'EOF'
psql -h localhost -p 4566 -U root -d dev -tAc \
  "SELECT count(*) FROM rw_catalog.rw_sources"
exit
EOF

# 5. Verify health-check (Phase 8f)

Troubleshooting

See README.md § Known limitations + komodo.md lessons.