After an upgrade to the next major version of OS on...

All posts Sergey's posts Post Back to profile

After an upgrade to the next major version of OS on my home server/box I started to observe that it freezes at some point. Iowait on some disks 100%, admin UI not responsive, apps start to die. Only restarting helps. Why this happens?

Context: OS is TrueNAS SCALE Dragonfish-24.04.0; 32GB RAM; 8TB RAIDz1 pool; 250GB nvme stripe for apps; 30GB boot drive SSD.

Like 4 May at 20:53 | Open on hachyderm.io

3 comments

Sergey Kibish

Maybe disks start to die? No. S.M.A.R.T. passes and other checks do not show that something is wrong.

Memory leak? Nope, all good. No app has stairway to heaven memory behaviour.

Swap is being used despite not all RAM is in use and there is free even free space. Why swap is being used?

4 May at 20:53 | Open on hachyderm.io

Sergey Kibish

Turns out that this version had fixed a problem where ZFS ARC cache would only use 50% of RAM on Linux, so now it tries to use as much as possible for cache and release when apps require memory.

Seems like an algorithm decides to use swap for everything else because ARC has higher priority.

Why system has swap in first place? Because by default TrueNAS creates swap of 2GB size on each disk which I left as is long time ago. But this is configurable and should be set to zero.

4 May at 20:53 | Open on hachyderm.io

Sergey Kibish

Now imagine performance of swap on ZFS raid array. It’s damn slow and if accessed frequently, it is very bad for performance and that’s why high iowait.

So, solution was to disable swap and all numbers went back to normal and server is healthy and live happily ever after.

The end

4 May at 20:53 | Open on hachyderm.io