Unraid Memory / IO Pressure Runbook¶

When to use: Load average > 10×core count, sys CPU > 50%, NVMe %util near 100, OOM kills in dmesg, containers flipping unhealthy, or docker ps taking > 10 s. These symptoms together almost always indicate page cache thrashing from memory exhaustion, not a misbehaving user process.

Estimated time: 5–10 minutes for full triage + mitigation.

Signature: page cache thrashing (the common case)¶

A specific four-signal fingerprint distinguishes cache thrashing from a genuine disk-bound workload:

If all four match thrashing → root cause is memory pressure, not the disk. Adding swap or freeing RAM fixes it. Investigating disk IO is a red herring until memory is relieved.

Triage in 5 steps¶

Step 1 — Confirm signature (parallel commands)¶

ssh root@unraid 'echo "=== LOAD ==="; uptime
                 echo "=== VMSTAT ==="; vmstat 1 3
                 echo "=== FREE ==="; free -h
                 echo "=== IOSTAT ==="; iostat -xmN 2 2 | grep -E "^(avg-cpu|nvme0n1|loop0) \$|^(nvme0n1|loop0) "
                 echo "=== CGROUP IO ==="; cat /sys/fs/cgroup/io.stat'

Interpretation:

• vmstat columns of interest: r (runnable), b (blocked, D-state), in (interrupts), cs (context switches), bi/bo (block in/out), us/sy/id/wa (CPU split).
• r > 4×cores and b > 5 → system is saturated.
• sys > 50% + wa < 5% → CPU is burning in kernel (kswapd / kworker), not waiting on disk.
• cgroup io.stat shows cumulative rbytes/rios per device. 259:0 is nvme0n1. Cumulative reads > 10× physical RAM after < 3 days uptime = cache miss amplification.

Step 2 — Confirm memory exhaustion¶

ssh root@unraid 'free -h; echo "=== SWAP ==="; swapon --show; echo "=== TOP MEM ==="; ps aux --sort=-%mem | head -15'

Server has 31 GiB RAM. If used > 28 GiB and swap = 0 → memory exhausted → page cache evicted → thrashing confirmed.

Step 3 — Find OOM kills (proves memory was the constraint)¶

ssh root@unraid "dmesg -T | grep -iE 'oom|killed process' | tail -15"

OOM entries show: which process was killed, its total-vm vs anon-rss (real RAM), and oom_score_adj (was it deliberately prioritized for killing?).

Read carefully: total-vm is virtual address space, not RAM. Real RAM is anon-rss. Chromium reserves ~50 GB virtual per process; that's normal, not a leak. The OOM event itself is the signal.

Step 4 — Identify culprit(s)¶

ssh root@unraid 'ps aux --sort=-%mem | head -15
                 echo "=== DOCKER UNHEALTHY ==="
                 docker ps --filter health=unhealthy --format "{{.Names}} | {{.Status}}"
                 echo "=== FLOWCOLL/CHROMIUM/JVM ==="
                 ps aux | grep -iE "flowcoll|chromium|java" | grep -v grep | awk "{print \$2, \$4\"%\", \$6/1024\"MB\", \$11}" | head -20'

Common memory consumers on this server (2026-07 baseline):

Step 5 — Mitigate (decision tree)¶

graph TD
    Q1{"Q1: Page cache thrashing
signature present?"}
    Q1 -->|No| DISK["Real disk-bound workload
Use iotop -baoP to find reader"]
    Q1 -->|Yes| Q2{"Q2: Is swap configured?
(swapon --show)"}
    Q2 -->|No| ADDSWAP["Add swap on vortex (procedure below)
Immediate relief"]
    Q2 -->|Yes| Q3{"Q3: Is swap used > 75%?"}
    Q3 -->|No| WARM["Memory pressure already relieved
Wait 5 min for cache to warm"]
    Q3 -->|Yes| Q4{"Q4: Single process growing
monotonically?"}
    Q4 -->|Yes| LEAK["Leak. Limit container:
docker update --memory=Ng --memory-swap=Ng <name>
Or restart the leaking service and monitor"]
    Q4 -->|No| EXCEED["Workload genuinely exceeds RAM
Reboot or reduce running containers"]

Procedure: add swap on btrfs vortex¶

Swap on btrfs requires a NOCOW subvolume (CoW + swap = corruption). On this server, vortex is btrfs on nvme0n1p4.

ssh root@unraid 'set -e
# 1. Create NOCOW subvolume (one-time)
btrfs subvolume create /mnt/vortex/@swap
chattr +C /mnt/vortex/@swap

# 2. Create swapfile(s) — repeat per 16 GB chunk
truncate -s 0 /mnt/vortex/@swap/swapfile
chattr +C /mnt/vortex/@swap/swapfile   # must be set BEFORE fallocate
fallocate -l 16G /mnt/vortex/@swap/swapfile
chmod 600 /mnt/vortex/@swap/swapfile

# 3. mkswap + swapon with priority 100
mkswap /mnt/vortex/@swap/swapfile
swapon -p 100 /mnt/vortex/@swap/swapfile

# 4. Verify
swapon --show
free -h'

For 32 GB total, create swapfile2 with the same steps.

Make persistent across reboots¶

Append to /boot/config/go (always back up first: cp /boot/config/go /boot/config/go.bak.$(date +%Y%m%d-%H%M%S)):

# Activate swap files on vortex btrfs (NOCOW subvolume @swap)
if [ -f /mnt/vortex/@swap/swapfile ]; then
    swapon -p 100 /mnt/vortex/@swap/swapfile 2>/dev/null
fi
if [ -f /mnt/vortex/@swap/swapfile2 ]; then
    swapon -p 100 /mnt/vortex/@swap/swapfile2 2>/dev/null
fi

The subvolume @swap and the swapfiles persist (they're files on btrfs); only the swapon calls need to run at boot.

Procedure: limit a leaking container¶

When a specific container repeatedly OOMs and grows back:

# Inspect current limits
ssh root@unraid "docker inspect <name> --format '{{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}'"

# Apply hard limit (e.g., 4 GB)
ssh root@unraid "docker update --memory=4g --memory-swap=4g <name>"

# Restart to drop current bloat
ssh root@unraid "docker restart <name>"

To persist the limit, edit the stack TOML and redeploy via Komodo: add deploy.resources.limits.memory in the service definition.

Why this is NOT usually the disk¶

The IO on nvme0n1 during a thrashing event can hit 3.5 GB/s reads and 15k IOPS, looking exactly like a runaway reader. It isn't:

• iotop -baoP shows only small reads scattered across many processes (docker health-check exec's).
• The reads come from loop0 (/boot/bzmodules squashfs) — the OS binaries/libraries that get re-read on every exec because page cache is empty.
• Cumulative NVMe reads can exceed 40 TB in 65 h of uptime — pure cache-miss amplification.

Fix the memory → cache recovers → IO drops 20–30× within minutes.

Verification (post-mitigation)¶

After adding swap or freeing memory:

ssh root@unraid 'echo "=== AFTER ==="; uptime; free -h; swapon --show
                 echo "=== VMSTAT ==="; vmstat 1 3
                 echo "=== IOSTAT ==="; iostat -xmN 1 3 | grep -E "^(nvme0n1|loop0) "
                 echo "=== UNHEALTHY ==="; docker ps --filter health=unhealthy --format "{{.Names}}: {{.Status}}"'

Success criteria:

• Load avg drops below 2× core count within 5 min
• sys CPU < 10%
• Page cache grows back above 6 GB
• Unhealthy container count returns to 0
• vmstat bi < 10k/s with normal workload

Prevention checklist (post-incident)¶

• Swap configured (≥ 16 GB on vortex @swap NOCOW subvolume)
• Swap entry in /boot/config/go (survives reboot)
• flowcoll RSS monitored — alert if > 5 GB sustained (likely leak)
• JVM containers have explicit -Xmx (opensearch: 2g, others as appropriate)
• No container without restart: unless-stopped or equivalent
• Uptime Kuma tracks memory usage trend (or VictoriaMetrics alert: (1 - mem_cache/RAM) < 0.2 AND sys_cpu > 0.5)

Related¶

• Unraid Skill — ## Troubleshooting: Memory/IO Pressure section
• Array Stop Stuck Runbook — different failure mode, similar decision-tree pattern
• Incident 2026-07-06: 31 GB RAM, 0 swap, 60+ containers → load 308 → 12 unhealthy → fixed via 32 GB swap on vortex