Apparently my GPU is not really enough for Stable Diffusion, it only worked through one WebUI that had some specific optimizations optionally enabled.

Now that this specific webui doesn't work with NixOS I cannot even generate 512 x 512 image, it throws CUDA out of memory errors at me. I could use enviroment variables to force CUDA to fragment stuff it does, but it will probably slow down it a lot more than well tested optimizations, so I will just postpone deleting Fedora partition for that usecase, I guess? (Also, llama webUIs are not packaged for nixOS at all as for now)