Proxmox vs FreeBSD: Which Virtualization Host Performs...

Proxmox vs FreeBSD: Which Virtualization Host Performs Better?

Since migrating many servers from Proxmox to FreeBSD, we have consistently felt that the VMs are more responsive. It's time to conduct some concrete tests.

https://it-notes.dragas.net/2024/06/10/proxmox-vs-freebsd-which-virtualization-host-performs-better/

#FreeBSD #Proxmox #Linux #Virtualization #kvm #bhyve #IT #SysAdmin #ITNotes #NoteHUB

Like 10 June at 7:16 | Open on mastodon.bsd.cafe

24 comments

gyptazy

@stefano@bsd.cafe very interesting! Thanks for this blog post providing detailed information!

10 June at 7:21 | Open on gyptazy.ch

moozer

@stefano This is interesting. Thanks.

The file creation speed is very significant - going from 407 MB/s on proxmox to 1467 MB/s on freebsd.

Do you know why there is such a large difference?

10 June at 7:37 | Open on mastodon.social

Stefano Marinelli

@moozer Considering that the Proxmox host on ZFS goes at 968.64 MB/s and FreeBSD host on ZFS flies at 1625.67 MB/s, I'd dare to say that ZFS seems to be much more optimized on FreeBSD than on the Proxmox PVE Kernel.

10 June at 7:45 | Open on mastodon.bsd.cafe

cuddle

@stefano ZFS on Linux still not mature yet, and FreeBSD support ZFS out of the box, to the kernel level.

If ZFS had close integration with the Linux kernel, it would have much or less similar I/O performance, both are POSIX complaint systems after all.

10 June at 11:27 | Open on mastodon.bsd.cafe

DELETED

@cuddle @stefano ZFS is beast in general.

When i tinkered with poudriere, i had big difference in compilation time ZFS vs UFS. Rust is known for long compilation, and i don't remember by how much, but compilation was definitely faster with ZFS.

FreeBSD have better support for ZFS, no doubt.

10 June at 13:57 | Open on mastodon.bsd.cafe

cuddle

@sourcerer @stefano Yeah, that's true.

ZFS do a lot of memory caching to avoid read calls, and the structure is generally better than UFS2 so the performance is. It's also that when you use FreeBSD the only filesystem you'd most likely use is ZFS. UFS2 doesn't natively support journaling, I think you'd need geom journal module as the back.

Netflix seems also uses UFS2, but mainly because ZFS structure can't allow sendfile call as Netflix wants. So that's where UFS2 is in production.

FreeBSD definitely has better support for ZFS. I'm wondering about others too, such as OpenIndiana...

@sourcerer @stefano Yeah, that's true.

Expand text...

10 June at 14:23 | Open on mastodon.bsd.cafe

Paul Wilde :blobcatnim: :dontpanic_nobg:

@stefano Thanks,

I've recently moved some of my VMs from KVM to bhyve and have noticed a significant performance improvement.
Most noticeably with Nextcloud (Debian guest) where it feels really quick and snappy now where it felt clunky before. I guess this may be due to improved IO performance similar to what you have tested.

10 June at 7:48 | Open on notnull.space

Benjamin Kwiecień 🇵🇸

@paul @stefano bhyve sounds great

10 June at 10:09 | Open on kwiecien.us

Paul Wilde :blobcatnim: :dontpanic_nobg:

@ben to be fair it's probably ZFS doing a lot of the heavy lifting here, but @stefano 's testing has shown that even on ZFS on proxmox (KVM) the performance still isn't quite as good as ZFS on FreeBSD (bhyve) - which is likely due to the ZFS modules in FreeBSD being native over additional kernel modules in proxmox (or any other Linux distro that can use it)

bhyve is definitely worth looking into if you use VMs though, for sure!

10 June at 10:17 | Open on notnull.space

Phil Dennis-Jordan 😷

@paul @ben @stefano I don’t know, I could equally imagine the Qemu I/O paths not being ideal. Qemu is a beast of a code base, which even after years of effort hasn’t managed to get close to eliminating the big Qemu lock (BQL) for example; bhyve is tiny in comparison.

10 June at 14:52 | Open on mstdn.social

Stefano Marinelli

@paul Exactly the reason why I decided to perform those tests. Moving VMs to bhyve led people to think I've upgraded the hardware as they're "quicker". Glad to know you're having the same experience!

10 June at 10:11 | Open on mastodon.bsd.cafe

BitPirate

@stefano Rule of thumb: if ZFS consistently beats ext4 in a basic filesystem benchmark on Linux, there's either something wrong with your setup or your benchmark is flawed.

I'd switch from sysbench to fio for I/O related tests.

10 June at 8:02 | Open on mas.to

Stefano Marinelli

@bitpirate I'll perform some more tests with fio when possible.

10 June at 9:11 | Open on mastodon.bsd.cafe

oxy but high contrast

@stefano nice.

The results are interesting - I wouldnt have expected such a wide margin in some of the results.

That said, Ive never used Proxmox and it has been a while since I’ve used KVM at all (and at the time I was using Fedora/Centos on XFS - ext4 has eaten enough data for me to never ever want to use it again)

10 June at 9:42 | Open on mastodon.bsd.cafe

DELETED

@stefano Great work!

Numbers are just numbers, what really counts is " they contacted me a few days later to inquire if we had new hardware because they noticed better performance. "

i do not reject analysis, but in a lot of cases sometimes we need to trust our senses and real life.

Just opinion, i'm really happy with article because i feel i can trust my intuition when it comes to choices.

I aim for learning bhyve.
No wonder why distrowatch switched to FreeBSD.

@stefano Great work!

Numbers are just numbers, what really counts is " they contacted me a few days later to inquire if we had new hardware because they noticed better performance. "

i do not reject analysis, but in a lot of cases sometimes we need to trust our senses and real life.

Just opinion, i'm really happy with article because i feel i can trust my intuition when it comes to choices.

Expand text...

10 June at 12:05 | Open on mastodon.bsd.cafe

Stefano Marinelli

@sourcerer I agree: number can tell many things, but often impressions tell much more 🙂

10 June at 12:27 | Open on mastodon.bsd.cafe

Stefano Marinelli

@sourcerer PS: thank you!

10 June at 12:36 | Open on mastodon.bsd.cafe

DELETED

@stefano I thank You too!
For learning, being and space.

I think bsd.cafe can be great for getting community together.

Forums were great and they are still fine ... but i only check forums for technical reason, not for socializing.

Have a great day all!

10 June at 12:41 | Open on mastodon.bsd.cafe

Stefano Marinelli

@sourcerer thank you. And thank you for being a part of the BSD Cafe!

10 June at 12:53 | Open on mastodon.bsd.cafe

Lovis IX ♿ :exodus:

@stefano

Great post. Thanks for sharing your result. Always a source of inspiration.

10 June at 14:04 | Open on social.zdx.fr

Stefano Marinelli

@lovisix thank you!

10 June at 14:39 | Open on mastodon.bsd.cafe

Fritz Adalis

@stefano
This is excelent, thanks for sharing it. One other test that might be interesting would be to overprovision a bunch of VMs and see how it handles scheduling when resources are scarce. Of course it's easy for me to think tasks up when it's not my time.