Email or username:

Password:

Forgot your password?
xsspup's posts Post Back to profile
Top-level
rastilin

@xssfox @aurynn

It's interesting to see how people organize things. TBH it looks overengeneered to me as well.

Ok, it's $1000 per month on the cloud. For that money you could go to Servers Australia and get a 48 core machine with 256GB of RAM and 2TB of Nvme storage and with 7TB of free bandwidth thrown in.

This is for a dedicated machine, not a VPS, so each processor would probably outperform a vCPU as well.

I think the cost to performance of cloud systems drops off sharply once you go beyond a small handful of small servers.

2 Dec 2022 at 14:46 | Wall-to-wall | Open on aus.social
58 comments
lutoma :ohai:

@rastilin Sure, it's cheaper.... Until a hard drive fails at 3 am or you unexpectedly need to scale up. And of course once you've scaled up you can't trivially scale back down again because you may be stuck with a minimum lease/commitment.

Not to mention all the administrative overhead of having to run a hypervisor for virtual machines and all that jazz yourself.

There certainly is a point at which dedicated hardware makes sense, but for a professional setup imo it comes much later.

2 Dec 2022 at 15:49 | Open on ohai.social
lutoma :ohai:

@rastilin (After having read the blog post I do agree though that that particular instance is overengineered compared to the number of users it has to an almost comical degree)

2 Dec 2022 at 15:51 | Open on ohai.social
Veticia Misumena 🕷️🌺

@rastilin @xssfox @aurynn
$1000/month?! Just buy a physical machine and drop the rest of that money on an unlimited 1Gbps fiber internet and put cloudflare in front of it. Hell, for $1000 you can do that every month and build your own cloud.

2 Dec 2022 at 15:50 | Open on eldritch.cafe
saluk

@Veticia @rastilin @xssfox @aurynn Yeah the way I see it, the cloud is for very small or very large workloads - attractive to get in because if you are not doing much it doesn't cost much, and attractive to large businesses because it costs more to pay your employees (not to mention dealing with headaches that can happen) to maintain infrastructure than to purchase that service externally. But the middle ground isn't so rosy.

2 Dec 2022 at 16:57 | Open on peoplemaking.games
Aurynn Shaw
xsspup :blobhaj_hearttrans:

@Veticia @rastilin @aurynn I'm signed up to cloudisland specifically because it doesn't use CloudFlare 🙄

2 Dec 2022 at 20:14 | Open on cloudisland.nz
Alex

@aurynn @xssfox transits provided via CloudFlare 😕 at least domestically here in NZ as that’s who Catalysts upstreams use.

3 Dec 2022 at 1:35 | Open on mastodon.nz
Aurynn Shaw
Alex

@aurynn @xssfox I’ve mentioned before, I’ll go get the traces and share again one sec.

3 Dec 2022 at 1:36 | Open on mastodon.nz
Alex

@aurynn @xssfox here you go. AS13335 is CloudFlare, they are using their network transit offering via Solarix who are the upstream to Catalyst. I checked this a *lot* from every NZ network operator I could.
3 Dec 2022 at 1:42 | Open on mastodon.nz
Aurynn Shaw
Simon (a 🐮 in 🇳🇿)

@aurynn @alexs @xssfox ah, in that way, yes... It is indeed 😣. I thought you meant as core infra provider.
I'm curious if it is even possible to have any route that does not touch cf?

3 Dec 2022 at 1:44 | Open on cloudisland.nz
Alex replied to Simon (a 🐮 in 🇳🇿)

@Firesphere @aurynn @xssfox entirely possible, and very definitely easy if you host domestically! I'm not aware of many local companies that transit them for _domestic_ as it's more expensive to do so. I don't know why Solarix are, but my guess would be that they want it for mitigation reasons. "Magic Transit" isn't cheap really vs normal IP transit. Either way, most of your stuff goes NZ ISP -> CloudFlare -> Solarix -> Catalyst from what Atlas traces tell me.

3 Dec 2022 at 1:49 | Open on mastodon.nz
Simon (a 🐮 in 🇳🇿) replied to Alex

@alexs @aurynn @xssfox I knew solarix, I did not know about their use of cf...

3 Dec 2022 at 1:52 | Open on cloudisland.nz
Alex replied to Simon (a 🐮 in 🇳🇿)

@Firesphere @aurynn @xssfox didn't think I was going mad; I did mention it in the past (April). The RIPE Atlas link I shared has a traceroute from most ISP's in NZ and basically all - even Spark - transit CF to get to Solarix and then to Catalyst 🤷‍♂️ atlas.ripe.net/measurements/47 the ℹ on the right-hand side shows the traces, if you want to check my checking 👍

3 Dec 2022 at 2:08 | Open on mastodon.nz
Simon (a 🐮 in 🇳🇿) replied to Alex
Simon (a 🐮 in 🇳🇿) replied to Alex

@alexs @aurynn @xssfox I'm not at a pc, but that circled domain mentions ix.nz... Is nzix now maybe part of cf?

3 Dec 2022 at 2:15 | Open on cloudisland.nz
Alex replied to Simon (a 🐮 in 🇳🇿)
Alex replied to Alex

@Firesphere @aurynn @xssfox to clarify, New Zealand Internet Exchange Inc is an NFP, not tied to CloudFlare or any one company. Everyone just peers there (and Megaport, which you may see in some other traces from atlas) hence it being so obvious in the trace. You can see the same from Voyager in the South Island as it traverses peering down there in CHC.

3 Dec 2022 at 2:18 | Open on mastodon.nz
Simon (a 🐮 in 🇳🇿) replied to Alex
Simon (a 🐮 in 🇳🇿) replied to Simon (a 🐮 in 🇳🇿)

@alexs @aurynn @xssfox I knew it theoretically is nfp... But seeing the asn.ix domain just made me wonder "huh? What?"

3 Dec 2022 at 2:24 | Open on cloudisland.nz
Alex replied to Simon (a 🐮 in 🇳🇿)

@Firesphere @aurynn @xssfox They publish the stuff from AGM's no secrets really - ix.nz/2022/10/10/news-from-the - aside from that they have done huge amounts for uplifting peering domestically.

ix.asn.au is the Aussie side, they got ix.nz kick started over here.

3 Dec 2022 at 2:28 | Open on mastodon.nz
Ewen McNeill replied to Alex

@alexs @Firesphere @aurynn @xssfox FWIW from here (Vodafone HFC, ie DOCSIS FTTN) path is Vodafone (under several names of former purchases), 3 hops of random 198.41.237.0/24 nodes (~6 IPS returned per hop), and then Catalyst edge. All of 198.41.128.0/17 is CloudFlare.

I don’t see any Solarix hop at all in my path. The early 203.167.245.0/24 hops are Vodafone. First Catalyst named hop is 202.78.247.22, which is a Catalyst IP.
3 Dec 2022 at 2:30 | Open on cloudisland.nz
Alex replied to Ewen

@ewenmcneill @Firesphere @aurynn @xssfox More digging indicates Catalyst are seemingly directly up-streamed by CloudFlare also (BGP routes from Atlas) and your path indicates that also.

For your route, Vodafone private peers with CloudFlare so you'd see the most direct path.

Also, you can see the same in Aussie.
atlas.ripe.net/measurements/47

🤷‍♂️
3 Dec 2022 at 2:39 | Open on mastodon.nz
Alex

@aurynn @xssfox screenshot above @Firesphere but also here's an Atlas run which will show from 70+ locations, atlas.ripe.net/measurements/47

3 Dec 2022 at 1:44 | Open on mastodon.nz
Simon (a 🐮 in 🇳🇿) replied to Alex

@alexs @aurynn @xssfox well, fuck, that's decently hidden and outside my former scope...

3 Dec 2022 at 1:50 | Open on cloudisland.nz
Simon (a 🐮 in 🇳🇿)

@alexs @aurynn @xssfox ehm, I would usually know that, given my previous job... , and I don't know that

3 Dec 2022 at 1:42 | Open on cloudisland.nz
Simon (a 🐮 in 🇳🇿)

@alexs @aurynn @xssfox nevermind that, you meant in a different way than I thought you meant

3 Dec 2022 at 1:45 | Open on cloudisland.nz
Richard Bairwell

@Veticia @rastilin @xssfox @aurynn That is in New Zealand Dollars: after conversion 640 Freedom Eagle dollars: and a good chunk of that is bandwidth which is expensive in New Zealand for international traffic.

2 Dec 2022 at 20:15 | Open on mastodon.org.uk
Veticia Misumena 🕷️🌺

@rbairwell @rastilin @xssfox @aurynn
Wait, wait, wait... You pay different rates for domestic and international traffic?! 🤯

2 Dec 2022 at 20:33 | Open on eldritch.cafe
Aurynn Shaw

@Veticia @rbairwell @rastilin @xssfox yes.

and I don't use Cloudflare because I don't want to support nazis.

2 Dec 2022 at 20:35 | Open on cloudisland.nz
Veticia Misumena 🕷️🌺

@aurynn @rbairwell @rastilin @xssfox
You're talking about that time Cloudflare shielded neo-nazi website on the principle that they provide infrastructure and are not censors and therefore are on a moral obligation to not interfere.

Which they later retracted and kicked them out anyway.

Or are you talking about another incident?

2 Dec 2022 at 20:45 | Open on eldritch.cafe
Aurynn Shaw

@Veticia @rbairwell @rastilin @xssfox I am talking about the time they had to be pressured hard earlier this year to drop nazis, and kept refusing and kept refusing and kept refusing and are still unhappy that they had to drop the nazis

because they're nazis and want to protect their own

2 Dec 2022 at 20:46 | Open on cloudisland.nz
Veticia Misumena 🕷️🌺

@aurynn @rbairwell @rastilin @xssfox
Well, valid point to take. On the other hand, you can argue that as long as you don't pay them and use their infrastructure for free, you're harming them by using up their resources.

2 Dec 2022 at 20:53 | Open on eldritch.cafe
Aurynn Shaw

@Veticia @rbairwell @rastilin @xssfox and also giving them more power over the internet at large if I did so, so, still no.

As far as buying servers, I
- don't have >=$50k laying around to buy kit
- don't have relationships with any DCs to rack said kit
- don't have the time or interest in researching hardware monitoring
- don't have relationships to buy support contracts for said kit
- would lose a lot of flexibility
- have to deal with depreciation
- etc

2 Dec 2022 at 20:58 | Open on cloudisland.nz
Veticia Misumena 🕷️🌺 replied to Aurynn

@aurynn @rbairwell @rastilin @xssfox
I think the problems you describe are related to rather big scale servers. People run mastodons on raspberry pies. I don't think what we're talking about here is a $50k problem.

But let me go back to the international traffic. Is using VPN common in NZ? To pretend all traffic is local? How big of a difference in price are we talking about?

2 Dec 2022 at 21:07 | Open on eldritch.cafe
Aurynn Shaw replied to Veticia Misumena 🕷️🌺

@Veticia @rbairwell @rastilin @xssfox Honestly this conversation makes me think you've never priced out actual server kit for actual production use with the full TCO and this is exactly the kind of irritating, unhelpful criticism that I was concerned about receiving.

2 Dec 2022 at 21:08 | Open on cloudisland.nz
Veticia Misumena 🕷️🌺 replied to Aurynn

@aurynn @rbairwell @rastilin @xssfox
I did not have to deal with a rack mounted 48 cores 1TB ram type of machine, that's correct. Because I thankfully never had to. I usually run my servers on my old desktops I've replaced. My current one I've downgraded to an intel atom based decommissioned 1U server I got for free (saved from a landfill; downgraded to it to lower the power bills). Works perfectly fine on my home internet. I just had to script a cloudflare dns to go around my dynamic ip (updating cloudflate takes effect immediately so I don't have to wait for a dns'es to update; it prevents downtime).

@aurynn @rbairwell @rastilin @xssfox
I did not have to deal with a rack mounted 48 cores 1TB ram type of machine, that's correct. Because I thankfully never had to. I usually run my servers on my old desktops I've replaced. My current one I've downgraded to an intel atom based decommissioned 1U server I got for free (saved from a landfill; downgraded to it to lower the power bills). Works perfectly fine on my home internet. I just had to script a cloudflare dns to go around my dynamic ip (updating...

2 Dec 2022 at 21:17 | Open on eldritch.cafe
Aurynn Shaw replied to Veticia Misumena 🕷️🌺
Veticia Misumena 🕷️🌺 replied to Aurynn
Aurynn Shaw replied to Veticia Misumena 🕷️🌺

@Veticia @rbairwell @rastilin @xssfox so because Cloud Island is something people rely on, it’s not one server I’d have to buy, it’s multiple, including multiple disk servers to live in geographically disparate DCs to ensure recoverability in the event of catastrophe. I’d have to get support contracts so that I have easy access to spare parts, and multiple servers so that the site isn’t offline for weeks while I wait for parts to arrive and for the DCs smart hands team to install.

It adds up.

2 Dec 2022 at 21:28 | Open on cloudisland.nz
Aurynn Shaw replied to Aurynn

@Veticia @rbairwell @rastilin @xssfox and this can be irritating, and I was snappy about it, because I don’t know that a lot of people know how much goes in to ensuring that you have a service that can be relied on to … well, be reliable.

2 Dec 2022 at 21:29 | Open on cloudisland.nz
Aurynn Shaw replied to Aurynn

@Veticia @rbairwell @rastilin @xssfox and if I want my service to be reliable, I’m don’t feel that single servers in a single DC can provide it, whereas cloud lets me do a lot of things with a lot more capability than I would be able to otherwise, and I can abstract all of the depreciation and managing spares and maintaining DCs and everything away, and focus on making sure my users can rely on what I’m doing.

It is a tradeoff in cost, though, yeah.

2 Dec 2022 at 21:42 | Open on cloudisland.nz
KlavsKlavsen replied to Aurynn

@aurynn @Veticia @rbairwell @rastilin @xssfox go look at hetzner.com - much cheaper - incl. The vms (where they stay up/ get restarted on new machine) just like ec2 etc. On other clouds. We run all our Companys services on physical servers on hetzner, with HA provided by Kubernetes setup. And scaleup works well enough with vm combo

2 Dec 2022 at 22:41 | Open on fosstodon.org
xsspup :blobhaj_hearttrans: replied to KlavsKlavsen

@KlavsKlavsen @aurynn @Veticia @rbairwell @rastilin maybe I'm missing something but I didn't think hetzner hosted anything in New Zealand ?

2 Dec 2022 at 22:43 | Open on cloudisland.nz
KlavsKlavsen replied to xsspup

@xssfox @aurynn @Veticia @rbairwell @rastilin lol no. I would exoect you would have a nearby equivalent though, if latency to new zealand is highest prio. Hetzner is in germany. Great loco for Europe

2 Dec 2022 at 22:56 | Open on fosstodon.org
Aurynn Shaw replied to KlavsKlavsen

@KlavsKlavsen @Veticia @rbairwell @rastilin @xssfox you may be surprised to learn that Hertzner is not an Aotearoa New Zealand company and would require relinquishing data sovereignty

2 Dec 2022 at 22:50 | Open on cloudisland.nz
Sam Stephens

@Veticia @rastilin and what happens when that machine fails? What happens when that fiber connection fails?

@aurynn it's wild the number of people you have in your mentions who've clearly never owned real production systems and engineered for proper availability who feel they need to comment.

@xssfox

2 Dec 2022 at 21:50 | Open on mastodon.nzoss.nz
Veticia Misumena 🕷️🌺

@chopsstephens @rastilin @aurynn @xssfox
For redundancy I have every disk in raid 1 (with daily/weekly offsite backup) and it has 2 power supplies connected to a relatively big ups (it can take a few hours). As for internet I only had to deal with outages a few times in the last few years. I probably could get another connection from another provider, but I don't think it would help much. That one time someone cut the fiber, internet was down in entire city, so I guess all of them share a single point of failure anyway.

@chopsstephens @rastilin @aurynn @xssfox
For redundancy I have every disk in raid 1 (with daily/weekly offsite backup) and it has 2 power supplies connected to a relatively big ups (it can take a few hours). As for internet I only had to deal with outages a few times in the last few years. I probably could get another connection from another provider, but I don't think it would help much. That one time someone cut the fiber, internet was down in entire city, so I guess all of them share a single...

2 Dec 2022 at 21:58 | Open on eldritch.cafe
Sam Stephens

@Veticia @rastilin @aurynn @xssfox it's still a single machine. Things fail other than disks. If that machine fails, can you quickly fail over to another machine? Aurynn can easily stand up and fail over to replacement instances if the hardware she's on fails.

You're telling Aurynn that her service is over-engineered, without knowing the SLAs Aurynn is trying to work to. And I'll tell you now, they're SLAs that you cannot meet with a single physical machine.

2 Dec 2022 at 22:08 | Open on mastodon.nzoss.nz
Veticia Misumena 🕷️🌺

@chopsstephens @rastilin @aurynn @xssfox
Oh, I don't have anything to criticise Audrynn for. Her setup looks pretty nice actually.

But I also don't necessarily see a single machine setup to be inherently worse. Aurynn's setup currently works on 8 machines, each with its own distinct role. That's 8 points of failure. If one of them dies of if connection breaks between them, all of them can stop working (especially if that hits the database one). It's nice if someone's dealing with redundancy for you, but you have to trust them to do it right.

As for my setup, I can just throw those disks to another machine (I still have a few laying around) and it'll most likely keep working without changing anything.

But I have to agree, sometimes it's just better to pay someone else to deal with all of that for you. (Unless when it's too expensive to justify the cost.) But since Aurynn instance is a paid one I guess she can figure something out. She does look like a smart one.

@chopsstephens @rastilin @aurynn @xssfox
Oh, I don't have anything to criticise Audrynn for. Her setup looks pretty nice actually.

But I also don't necessarily see a single machine setup to be inherently worse. Aurynn's setup currently works on 8 machines, each with its own distinct role. That's 8 points of failure. If one of them dies of if connection breaks between them, all of them can stop working (especially if that hits the database one). It's nice if someone's dealing with redundancy for you,...

2 Dec 2022 at 22:24 | Open on eldritch.cafe
nick
@Veticia @chopsstephens it’s really frustrating seeing so many people jump in with the snooty “never run large infrastructure, huh?” attitude when that is objectively the opposite of how we should be building instances. The whole point of federation is many small parts of a larger whole so there aren’t huge single points of failure. It’s ok if your instance goes down for a bit, it’ll get caught back up when it comes back online. I think self hosting on used hardware is perfectly fine (it’s what I’m doing), if you do regular backups offsite, because if you’re going to outgrow that hardware you’re probably staring down an issue of maintaining effective moderation anyway. Don’t grow so big that you need big installation methods.
@Veticia @chopsstephens it’s really frustrating seeing so many people jump in with the snooty “never run large infrastructure, huh?” attitude when that is objectively the opposite of how we should be building instances. The whole point of federation is many small parts of a larger whole so there aren’t huge single points of failure. It’s ok if your instance goes down for a bit, it’ll get caught back up when it comes back online. I think self hosting on used hardware is perfectly fine (it’s what I’m...
2 Dec 2022 at 22:22 | Open on introvert.social
Aurynn Shaw
Craig Askings

@rastilin @xssfox @aurynn given the historical stability of that network and the fact that no matter how fancy that server is, it is still a single point of failure at the Device, Datacenter and Network levels (admin domain, not talking about single NIC etc)
I’d take the properly engineered option thanks.

3 Dec 2022 at 0:20 | Open on aus.social
Go Up