Gregory's ServerWow. so before the latest influx of users this server would process about 150k federated events a day (not just posts but all activities: follows, favorites, etc). Most of this week post-influx we floated around 280k. Yesterday jumped to 410k.
 41 comments | Expand all CWs
 @danbruno indirectly yes. a server of certain processing power can only reasonably handle so many events a day. But also I run a pretty low power server and not noticing any issues. And there's a queueing system so if there's a burst what happens is you just process events and backfill so the timeline might be behind by an hour or something for a little while before it catches up Wow. Stats on federated events consumed by our server this week: Nov 3-5: 280k events each day This is a sustained increase over the course of about a week. I have never seen anything like it. Usually what we see in an influx is a one or two day bump and then dropoff. I have never seen a week of solid rampup before. This feels very different.  @trendless that's what I am counting here! more or less each job is a single federated event -- a favorite, a follow, a post, etc @anarchistbicycleclub Hometown v1.0.6+3.5.2 running on a Linode "Dedicated CPU" 4GB plan as specified here https://www.linode.com/products/dedicated-cpu/ @darius how have you found that in terms of speed this last week, is federation slowing down a lot for you? @darius thank you! How have you found that in terms of speed this last week, is federation slowing down a lot for you? @anarchistbicycleclub @anarchistbicycleclub totally fine until today's huge jump. Now we have about 700 items floating the sidekiq queue (up from 0). Looking into ways to mitigate this (or if it even is really a problem. timelines seem to be realtime still (aka we're not getting posts minutes or hours after posted) @anarchistbicycleclub consider tweaking the settings in config/sidekiq.yml -- the number of queues for "push", the concurrency, and also the stuff in mastodon-sidekiq-push.service (DB_POOL and the "bundle exec sidekiq -c 10 -q push`, tweaking those values). I am not an expert in this stuff I just try different numbers and google to make sure I'm not being a total idiot @darius I came over from the bird site just a few days ago and I've spent maybe 15 minutes total back there since. Haven't been on it at all today. This is better. @darius Hi Darius. I'm curious about overall sustainability / scaling of the Mastodon / Fediverse network. Do you know whether anyone is looking at this in aggregate? Is anyone calculating the server costs (increase)? I haven't seen any mention of economics of #Fediverse, but my server's admin is already mentioned sudden, additional costs of running the instance. @tarkowski Costs will go up across the network for sure. The economics vary from software to software (Pleroma is a lot more efficient than Mastodon, for example). On my end, this unprecedented jump looks like it's going to cost us an extra $10/mo. My server hosts 50 active users. I think the economics ultimately work out better for small servers than big ones: individuals are more likely to contribute money when they belong to a tight community of shared principles. @tarkowski This of course assumes that web hosting costs stay more or less constant. This might not be the case, and could be a massive disaster. @darius What constitutes an event? Is it any interaction, such as a boost, favourite, bookmark, etc.? Does it also include posts, replies, server moves? Is a new account setup an event? @bookgaga basically anything is an event. Any interaction (posts, boost, favs, bookmarks, replies, literally anything), plus some behind the scenes things users don't see (but these make up the vast minority). I imagine a new account setup probably triggers some small number of events, less than 5.
[DATA EXPUNGED]
@andresmh it is basically every interaction that your server knows about in the entire universe of servers out there. Follows, favs, posts, bookmarks, etc etc. Basically that means that your server has seen 37,000 things happen on federated social media in the last (unspecified amount of time) @andresmh also this is not necessarily stuff addressed to your server. it could be a conversation happening between two people on two servers that your server know about. the way federation works, that's the still an event that gets delivered to you (otherwise how would you know to render the conversation, etc)  @objelisks I think we are going to have a bumpy period where Mastodon is going to need to seriously improve its overall performance. But I do think it will happen. it's correlated (significantly though not exclusively) with number of external follows for sure. I am imagining an economics of this where if someone on a small server has a million followers, maybe the admin has to be like "hey... pay more money for your fair share of the resources here or migrate elsewhere please" @objelisks my big fear is that if web hosting costs go up across the board significantly for whatever reason it will just be not viable to run a small server. Fortunately there are a lot of forces in place to prevent that (namely it would probably make billionaires less rich if that were to happen, lol)  > bigger servers are stuck retrying lots of messages @darius isn't there a back-off strategy ? I remember being frustrated that Mastodon doesn't respect when another fediverse service returns a "405 Method Not Allowed" to signal that it does not federate, but I was hoping that after a number of tries it stops. @mariusor yes there is a back off strategy but it still seems to clog up anyway for some reason @darius interesting—so the surge is passed but there’s a decent amount of folks who’ve stuck with it? @mattlaschneider yeah we essentially doubled our prior daily average in what seems like a sustained way |
@darius
I can feel it. Not the server struggling. Just the vibes.