Skip to content

Lessons: RustFS Shared S3 Storage

Deployed 2026-07-03. Standalone RustFS replaces Supabase Storage local filesystem + solves Go SDK S3 incompatibility.

Context

Memos app needs S3 storage. Supabase Storage's S3 protocol endpoint (/storage/v1/s3) has a SigV4 incompatibility with Go SDK — mc, rclone, and Memos (all Go-based) fail with SignatureDoesNotMatch. Python-based clients (boto3, aws CLI) work fine.

Root cause: Go SDK includes accept-encoding;amz-sdk-invocation-id;amz-sdk-request in SigV4 SignedHeaders. Storage's validator computes a different canonical request for these extra headers. Confirmed by testing direct (no proxy) connections and reading Storage source code (src/storage/protocols/s3/signature-v4.ts).

Solution: Deploy RustFS (S3-compatible, serves S3 at root — no path prefix). Go SDK works natively. Supabase Storage uses RustFS as S3 backend (unified storage).

Architecture

RustFS container (:9000)
├── networks: reverse-proxy + supabase-internal
├── disk: /mnt/user/s3-storage (single, user: "99:100")
│
├── bucket: supabase/                    ← Supabase Storage backend
│   └── homelab/{bucket_id}/{object}/    ← tenant path structure
│       ├── memos/assets/...             ← Memos app uploads (unified)
│       └── memos/files.zip/...          ← Supabase REST/Studio uploads
│
└── (future service buckets)

Three access paths, same RustFS instance:

Client Path Bucket
Memos app (Go SDK) http://rustfs:9000 direct (Docker DNS) supabase
Supabase REST/Studio Storage → http://rustfs:9000 (S3 backend) supabase
mc/rclone/aws CLI http://s3.home → Caddy → rustfs:9000 any

Adding a New Service to RustFS

# 1. Create bucket
mc mb rustfs/{service-name}

# 2. Configure service S3:
#    endpoint    = http://rustfs:9000  (Docker DNS, NOT http://s3.home)
#    bucket      = {service-name}
#    region      = us-east-1
#    usePathStyle = true
#    Credentials: Komodo Variables RUSTFS_ACCESS_KEY + RUSTFS_SECRET_KEY (shared)

# 3. Ensure service container is on reverse-proxy network (for Docker DNS)

Gotchas

1. Bridge containers cannot reach Caddy macvlan IP

Symptom: dial tcp 192.168.100.10:80: connect: no route to host

Bridge-network containers cannot reach macvlan IPs (Caddy at 192.168.100.10). Use Docker DNS (http://rustfs:9000) for inter-container communication. Reserve http://s3.home (via Caddy) for browser/mc access from LAN only.

2. Go SDK SigV4 fails with Supabase Storage S3

Symptom: SignatureDoesNotMatch from mc, rclone, Memos (Go SDK).

Go SDK adds extra headers to SigV4 SignedHeaders. Supabase Storage's validator doesn't compute matching canonical request. Use RustFS (serves S3 at root, no path prefix) or Python-based clients (aws CLI, boto3) for Supabase S3 protocol endpoint.

3. Memos attachment payloads store per-attachment S3 config

Each memos.attachment row has its own s3Config (bucket, key, endpoint) in the payload JSONB column. When changing bucket or path prefix, UPDATE both:

  • memos.system_setting (template for new uploads)
  • memos.attachment.payload (existing references)
-- Example: change bucket from memos → supabase
UPDATE memos.attachment
SET payload = jsonb_set(
  jsonb_set(payload::jsonb, '{s3Object,s3Config,bucket}', to_jsonb('supabase'::text)),
  '{s3Object,key}', to_jsonb('homelab/memos/' || (payload::jsonb->'s3Object'->>'key'))
)::text WHERE storage_type = 'S3';

4. Supabase postgres role is NOT superuser

postgres via pooler has rolsuper=false. Use supabase_admin for DB writes:

echo "SQL HERE" | ssh root@100.68.251.84 "docker exec -i supabase-db psql -U supabase_admin -d postgres"

km-exec splits args at spaces — do NOT use km-exec supabase-db psql -c "SQL WITH SPACES". Use the SSH+pipe pattern above.

5. RustFS single-disk mode

Set RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true for single-disk deployments. No erasure coding — standalone mode only. For production with data safety, use 4+ disks.

  • stacks/storage/rustfs/compose.yaml — RustFS stack
  • komodo/stacks/storage/rustfs.toml — Komodo deploy config
  • stacks/database/supabase/compose.yaml — STORAGE_BACKEND=s3, GLOBAL_S3_ENDPOINT=http://rustfs:9000
  • stacks/networking/caddy/conf/applications.caddyfile — s3.home + s3-console.home routes
  • komodo/stacks/database/supabase.toml — RUSTFS_ACCESS_KEY + RUSTFS_SECRET_KEY env passthrough
  • Komodo Variables: RUSTFS_ACCESS_KEY, RUSTFS_SECRET_KEY (shared across services)