Skip to content

Playbook: Sync Superset Data Sources

Khi datasources.yaml thay đổi, chạy sync_datasources.py để apply vào Superset.

Trigger

File stacks/applications/superset/datasources.yaml được edit (add/edit/remove database).

Steps

# 1. Preview changes
python3 scripts/superset/sync_datasources.py --dry-run

# 2. Apply (add new DBs, update descriptions, remove deleted DBs)
python3 scripts/superset/sync_datasources.py

Script so sánh YAML declaration vs live Superset state via REST API, rồi apply diff.

Prerequisites

Pigsty read-only role (1-time per new DB)

Khi add database mới vào YAML, cần grant read-only cho superset_reader trên Pigsty:

echo "GRANT USAGE ON SCHEMA public TO superset_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO superset_reader;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO superset_reader;" \
  | ssh root@100.68.251.84 "docker exec -i pigsty-postgres psql -U postgres -d {new_db_name}"

Superset running

  • Container superset on unraid (healthy)
  • Admin login: admin / minhluc1
  • Endpoint: http://superset.home or https://viz.minhluc.info

YAML format

credentials:
  host: pigsty-postgres # Docker DNS
  port: "5432"
  user: superset_reader
  password: d2d3a996a969b7255c396bbb0b0a657f
  driver: postgresql+psycopg2

databases:
  airflow: # key = Pigsty database name
    description: Airflow DAG metadata
    schema: public
    expose_in_sqllab: true
  new_db: # add new → script creates in Superset
    description: New analytics
    expose_in_sqllab: true
  # removed_db:                # remove from YAML → script deletes from Superset

Custom host (non-Pigsty DBs)

supabase_memos:
  description: Memos analytics
  host: 100.121.78.65 # override default host
  port: "5432"
  user: postgres.homelab
  password: <password>
  db_name: postgres # override database name
  driver: postgresql+psycopg2
  expose_in_sqllab: true

What the script does

Action Condition API call
ADD DB in YAML but not in Superset POST /api/v1/database/
EDIT Description or URI changed PUT /api/v1/database/{id}
REMOVE DB in Superset but not in YAML DELETE /api/v1/database/{id}

Verification

After sync, verify in Superset UI:

  • Settings → Database Connections (list should match YAML)
  • SQL Lab → select database → run SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public'

Files

  • stacks/applications/superset/datasources.yaml — declarative state
  • scripts/superset/sync_datasources.py — sync script
  • Komodo Variables: SUPERSET_DB_PASSWORD, SUPERSET_SECRET_KEY