Gregory's ServerThe very wonderful @aurynn wrote about what it takes to host cloudisland.nz which is hosted in New Zealand on Catalyst (and not AWS/Azure/GCP)
|
59 comments
@rastilin Sure, it's cheaper.... Until a hard drive fails at 3 am or you unexpectedly need to scale up. And of course once you've scaled up you can't trivially scale back down again because you may be stuck with a minimum lease/commitment. Not to mention all the administrative overhead of having to run a hypervisor for virtual machines and all that jazz yourself. There certainly is a point at which dedicated hardware makes sense, but for a professional setup imo it comes much later. @rastilin (After having read the blog post I do agree though that that particular instance is overengineered compared to the number of users it has to an almost comical degree) @Veticia @rastilin @xssfox @aurynn Yeah the way I see it, the cloud is for very small or very large workloads - attractive to get in because if you are not doing much it doesn't cost much, and attractive to large businesses because it costs more to pay your employees (not to mention dealing with headaches that can happen) to maintain infrastructure than to purchase that service externally. But the middle ground isn't so rosy.   @Firesphere @aurynn @xssfox entirely possible, and very definitely easy if you host domestically! I'm not aware of many local companies that transit them for _domestic_ as it's more expensive to do so. I don't know why Solarix are, but my guess would be that they want it for mitigation reasons. "Magic Transit" isn't cheap really vs normal IP transit. Either way, most of your stuff goes NZ ISP -> CloudFlare -> Solarix -> Catalyst from what Atlas traces tell me. @Firesphere @aurynn @xssfox didn't think I was going mad; I did mention it in the past (April). The RIPE Atlas link I shared has a traceroute from most ISP's in NZ and basically all - even Spark - transit CF to get to Solarix and then to Catalyst 🤷♂️ https://atlas.ripe.net/measurements/47551069/#probes the ℹ on the right-hand side shows the traces, if you want to check my checking 👍 @Firesphere @aurynn @xssfox to clarify, New Zealand Internet Exchange Inc is an NFP, not tied to CloudFlare or any one company. Everyone just peers there (and Megaport, which you may see in some other traces from atlas) hence it being so obvious in the trace. You can see the same from Voyager in the South Island as it traverses peering down there in CHC. @aurynn @xssfox screenshot above @Firesphere but also here's an Atlas run which will show from 70+ locations, https://atlas.ripe.net/measurements/47551069/#probes  @rbairwell @rastilin @xssfox @aurynn @Veticia @rbairwell @rastilin @xssfox yes. and I don't use Cloudflare because I don't want to support nazis. @aurynn @rbairwell @rastilin @xssfox Which they later retracted and kicked them out anyway. Or are you talking about another incident? @Veticia @rbairwell @rastilin @xssfox I am talking about the time they had to be pressured hard earlier this year to drop nazis, and kept refusing and kept refusing and kept refusing and are still unhappy that they had to drop the nazis because they're nazis and want to protect their own @aurynn @rbairwell @rastilin @xssfox @Veticia @rbairwell @rastilin @xssfox and also giving them more power over the internet at large if I did so, so, still no. As far as buying servers, I @aurynn @rbairwell @rastilin @xssfox But let me go back to the international traffic. Is using VPN common in NZ? To pretend all traffic is local? How big of a difference in price are we talking about? @Veticia @rbairwell @rastilin @xssfox Honestly this conversation makes me think you've never priced out actual server kit for actual production use with the full TCO and this is exactly the kind of irritating, unhelpful criticism that I was concerned about receiving. @aurynn @rbairwell @rastilin @xssfox @Veticia @rbairwell @rastilin @xssfox so because Cloud Island is something people rely on, it’s not one server I’d have to buy, it’s multiple, including multiple disk servers to live in geographically disparate DCs to ensure recoverability in the event of catastrophe. I’d have to get support contracts so that I have easy access to spare parts, and multiple servers so that the site isn’t offline for weeks while I wait for parts to arrive and for the DCs smart hands team to install. It adds up. @Veticia @rbairwell @rastilin @xssfox and this can be irritating, and I was snappy about it, because I don’t know that a lot of people know how much goes in to ensuring that you have a service that can be relied on to … well, be reliable. @Veticia @rbairwell @rastilin @xssfox and if I want my service to be reliable, I’m don’t feel that single servers in a single DC can provide it, whereas cloud lets me do a lot of things with a lot more capability than I would be able to otherwise, and I can abstract all of the depreciation and managing spares and maintaining DCs and everything away, and focus on making sure my users can rely on what I’m doing. It is a tradeoff in cost, though, yeah. @aurynn @Veticia @rbairwell @rastilin @xssfox go look at hetzner.com - much cheaper - incl. The vms (where they stay up/ get restarted on new machine) just like ec2 etc. On other clouds. We run all our Companys services on physical servers on hetzner, with HA provided by Kubernetes setup. And scaleup works well enough with vm combo @KlavsKlavsen @aurynn @Veticia @rbairwell @rastilin maybe I'm missing something but I didn't think hetzner hosted anything in New Zealand ? @xssfox @aurynn @Veticia @rbairwell @rastilin lol no. I would exoect you would have a nearby equivalent though, if latency to new zealand is highest prio. Hetzner is in germany. Great loco for Europe @KlavsKlavsen @Veticia @rbairwell @rastilin @xssfox you may be surprised to learn that Hertzner is not an Aotearoa New Zealand company and would require relinquishing data sovereignty @Veticia @rastilin @aurynn @xssfox it's still a single machine. Things fail other than disks. If that machine fails, can you quickly fail over to another machine? Aurynn can easily stand up and fail over to replacement instances if the hardware she's on fails. You're telling Aurynn that her service is over-engineered, without knowing the SLAs Aurynn is trying to work to. And I'll tell you now, they're SLAs that you cannot meet with a single physical machine. @rastilin @xssfox @aurynn given the historical stability of that network and the fact that no matter how fancy that server is, it is still a single point of failure at the Device, Datacenter and Network levels (admin domain, not talking about single NIC etc) |
@xssfox @aurynn
It's interesting to see how people organize things. TBH it looks overengeneered to me as well.
Ok, it's $1000 per month on the cloud. For that money you could go to Servers Australia and get a 48 core machine with 256GB of RAM and 2TB of Nvme storage and with 7TB of free bandwidth thrown in.
This is for a dedicated machine, not a VPS, so each processor would probably outperform a vCPU as well.
I think the cost to performance of cloud systems drops off sharply once you go beyond a small handful of small servers.
@xssfox @aurynn
It's interesting to see how people organize things. TBH it looks overengeneered to me as well.
Ok, it's $1000 per month on the cloud. For that money you could go to Servers Australia and get a 48 core machine with 256GB of RAM and 2TB of Nvme storage and with 7TB of free bandwidth thrown in.