Now imagine performance of swap on ZFS raid array. It’s damn slow and if accessed frequently, it is very bad for performance and that’s why high iowait.

So, solution was to disable swap and all numbers went back to normal and server is healthy and live happily ever after.

The end