Skip to content

btrfs NOCOW for Database Workloads

When to read: before deploying any service that uses SQLite, PostgreSQL, MySQL, or any database on a btrfs filesystem (Unraid cache pool, btrfs array disk).

Related docs:

Problem

btrfs Copy-on-Write (CoW) amplifies small random writes. Each 4-8KB DB page write triggers a full CoW copy of the extent. For workloads with frequent small writes (SQLite WAL, Postgres WAL, Redis AOF), this causes:

  • Write amplification 5-10x
  • Disk I/O saturation under moderate load
  • System load spikes (observed: load 170+ on 12-core Unraid)
  • Cascading failures (all containers slow down, Docker daemon hangs)

Symptoms

  • docker logs <container> shows timeouts or slow queries
  • Unraid load average spikes without obvious CPU usage
  • top shows high %wa (I/O wait) and kworker/btrfs-* threads
  • Kavita/Postgres/SQLite operations take 10-60s instead of <1s
  • Multiple containers become simultaneously unresponsive

Root cause analysis (from 2026-06-24 incident)

  1. Mimir (SQLite metrics DB): TSDB block corruption from CoW write amplification under concurrent API writes. Caused restart loops + data corruption.
  2. Kavita tagging: 50 API calls/min → 50 SQLite writes/min → btrfs CoW thrashing → load 150+. Even with 0.3s delays between calls.
  3. Calibre batch convert: 3 parallel ebook-convert processes → heavy I/O on btrfs → load 170+ → zombie process accumulation.

Fix — Disable CoW on appdata directory

One-time migration (requires downtime)

# 1. Stop ALL containers
docker stop $(docker ps -q)

# 2. Snapshot btrfs pool (safety net, 0GB extra via CoW shared blocks)
btrfs subvolume snapshot -r /mnt/vortex /mnt/vortex/.snapshots/pre-cow-disable-$(date +%Y%m%d-%H%M)

# 3. Rename current appdata
cd /mnt/vortex
mv appdata appdata-cow-bak

# 4. Create NOCOW directory
mkdir appdata
chattr +C appdata
lsattr -d appdata  # verify: should show 'C' flag

# 5. Copy data (forces non-CoW copy)
cp -a --reflink=never appdata-cow-bak/. appdata/

# 6. Verify file counts match
find appdata-cow-bak -type f | wc -l
find appdata -type f | wc -l

# 7. Update Unraid config
sed -i 's/shareCOW="auto"/shareCOW="no"/' /boot/config/shares/appdata.cfg

# 8. Restart containers
docker start $(docker ps -aq --filter "status=exited")

# 9. Verify critical services
docker exec kavita curl -s http://localhost:5000/api/Metadata/tags -H "x-api-key: <key>"
docker exec pigsty-postgres psql -U postgres -c "SELECT 1"
lsattr /mnt/vortex/appdata/kavita/config/kavita.db  # should show 'C'

# 10. Cleanup (after 24h stable)
rm -rf /mnt/vortex/appdata-cow-bak
btrfs subvolume delete /mnt/vortex/.snapshots/pre-cow-disable-*

Verify NOCOW is active

lsattr /mnt/vortex/appdata/<service>/data/<db_file>
# Output must contain 'C' in the flags column:
# ---------------C------ /mnt/vortex/appdata/kavita/config/kavita.db

When to use NOCOW vs keep CoW

Data type CoW? Why
Database files (SQLite, Postgres, MySQL, Redis) NO Random small writes → amplification
VM disk images (qcow2, raw) NO Same as DB — random I/O
Docker overlay layers NO (overlayfs manages own CoW) Already handled
Large append-only logs Optional CoW provides limited benefit
Source code, documents YES Bit-rot protection + snapshots
Photos, music, videos, ebooks YES Archive data — want checksum integrity
Backup targets YES Snapshot-based backup relies on CoW

Tools for diagnosing btrfs I/O issues

# Check if CoW is enabled on a file
lsattr <file>

# Check btrfs filesystem stats
btrfs device stats /mnt/vortex

# Monitor I/O in real-time
iostat -x 1

# Find btrfs-specific kworker activity
ps aux | grep btrfs

# Check write amplification (compare logical vs physical writes)
btrfs filesystem df /mnt/vortex

Lessons learned

  1. Mimir (cluster-oriented) on btrfs single-node = TSDB corruption: Replaced with VictoriaMetrics (single binary, stable, ~50MB RAM).
  2. Kavita SQLite on CoW btrfs = I/O death by 1000 writes: Fixed with NOCOW.
  3. Calibre parallel conversion = I/O bomb: Limit to 2 workers, run off-peak.
  4. Komodo periphery zombie accumulation: 235 zombies caused load 170+. Restart periphery to reap: docker restart komodo-periphery.