Gregory's ServerHow Decentralized Is Bluesky Really? https://dustycloud.org/blog/how-decentralized-is-bluesky/
A technical deep-dive, since people have been asking me for my thoughts. I'll expand a bit on some of the key points here in a thread. ๐งต
 257 comments
 That said, let's get to the summary: Bluesky / ATProto are not decentralized or federated, according to my analysis. However, the "credible exit" goal is worth perusing, and does use decentralization techniques! But it is not decentralization/federation without moving the goalposts on those terms. Furthermore, I think Bluesky is providing something valuable: a lot of people are trying to leave X-Twitter *right now* because it has become a completely toxic place. The fact that Bluesky's team has managed to scale to receive such users is incredible, nearly feeling miraculous. On the fediverse we also see a lot of accusations of Bluesky being owned by Jack Dorsey, and this isn't true. My understanding is that Jay performed an impressive amount of negotiation to allow Bluesky to receive funding independently. These days Jack Dorsey is instead focusing on Nostr, which I can only describe as "a sequel to Secure Scuttlebutt with extremely bad vibes where bitcoin people talk about bitcoin" I participated a bit in the process of when Bluesky was Jack Dorsey and Parag Agrawal's personal project. I also believe Jack and Parag were sincere about Bluesky as a decentralized social network protocol that Twitter would adopt, which is the directive that Bluesky was given as an organization. When Jay Graber was awarded the position to lead Bluesky, I was not surprised. To me, Jay was the obvious choice to deliver what Bluesky was being directed, and I do think Jay is an excellent leader There is also something which Bluesky gets right which the fediverse does not. I mentioned that Bluesky uses decentralization *techniques*, and the most important of those is content-addressing. This allows content to exist even when a server goes down. This is a great decision and I have advocated that the fediverse do so as well. In fact several years ago I wrote a demo in @spritely's early days showing off how one could build a content-addressed ActivityPub in a spec-compatible way. So I have opened here with the things that Bluesky does well. As you may guess, we are about to move into critiques territory, and it's a lot of critiques from a *decentralization*/*federation* perspective. It doesn't erase the "credible exit" goals, which I think are good still. Let's dive in... A frequent way of describing Bluesky's decentralization, including by Bluesky's team, is "it's like a bunch of blogs (Personal Data Stores), and then the relay/appview/etc pieces are like search engines" This is a reasonable starting point for thinking about things, so let's run with it. In fact ATProto's own tutorial even says "Think of our app like a Google": https://atproto.com/guides/applications And indeed this is a good way to think about things. But it doesn't seem so bad, because we have Personal Data Stores like blogs, so probably things are fine, right? While most people would argue that blogs and websites are open, few would argue that *Google* is open. So this is a curious place to begin thinking, and yet structually, it is actually quite apt. PDS'es are like blogs, the rest is like Google. But relays/appviews/etc do a lot *more* than Google. Relays, AppViews, etc don't just index information. Blogs and their interactions are generally slow-moving, but social media is direct and responsive. Notifications and fast interactions are key. So search engines, yes, but we should also think of these components of doing much more. But let's stay on this blog/search engine analogy for a while before we unpack what it means on a *technical* level, which is interesting. Let's analyze for the moment from a power dynamics level. Building a web search engine is actually pretty easy these days, you can do so with off-the-shelf tools. And yet there are only a couple of search engines *really*, Google and Bing (DDG mostly uses Bing). And yet the information is right there. *Anyone* could run their own engine. Why don't they? Furthermore there is an interesting connection between blogs and social media: the death of blogs + feed aggregation directly aligns with the death of social media. How many of you were around for the birth and awkward death of blog engine feeds? Because I was! Oh, remember Google Reader?  @lopezsanchez @cwebber me too. But they are definitely at the margin. Many blogs died in that transition. Feed readers are also simple, and in fact they were even easy to self host, even on the desktop! But Google Reader came in and was such a good design that everyone used it. When it went away, blogs were still *there*. But blogging as a *syndication medium* died. One big player left, and it's gone. This was sad for me especially; my favorite medium on the internet ever was webcomics. Webcomics still exist, sort of, but the loss of independent publishing and aggregation meant that they had to change to survive. The shape of webcomics started to get shaped to the shape of Twitter's image box. This may seem like an enormous aside, but it isn't. The big sell currently is that "you don't need to run a relay because you can run your own PDS!" but as I have illustrated here, the distribution and syndication power dynamics matter a lot. So. It isn't enough to self-host your own PDS. Whether or not people can run their own relays/appviews/etc actually matters *a lot* if we want this stuff to survive. So, can we? How hard is it to run your own AppView/Relay/etc? Today, there is only one real organization running a Relay that really matters or an AppView that people use for anything other than fun aggregation of statistics. Nothing that resembles meaningful decentralization of the network. It's all run by one company: Bluesky. But could we change that? People are trying; most notably alice has done some great work recently: https://alice.bsky.sh/post/3laega7icmi2q So now someone *can* run their own Relay (not the AppView yet, but maybe soon), and we're getting a sense of the cost and scale. This is good news; we didn't know before. In fact we also have an idea of the rate of growth. Approximately 4 months prior, @bnewbold.net posted an article detailing how to run a Bluesky relay: https://whtwnd.com/bnewbold.net/entries/Notes%20on%20Running%20a%20Full-Network%20atproto%20Relay%20(July%202024) This is great. We need more people trying to do so to get a sense of how decentralized things can be. Just focusing on storage, in July @bnewbold.net estimated the amount of storage expected to run a Bluesky relay is approx 1 terabyte. In just 4 months at start of this month (November), alice estimates nearly 5 terabytes. This is a fast growth rate and this is *before* the big post-election influx. I tried estimating how much this would cost; as a lazy approximation I dumped a 5 terabyte machine into seeing what Linode would cost to self-host, and it was approximately 55k a year: https://bsky.app/profile/dustyweb.bsky.social/post/3lah5n3kld42q That's a lazy estimate, but that's also what many people make in the US every year However @bnewbold pointed out, correctly!, that there were cheaper options available. If we used even Linode's block storage, it would be cheaper (but still expensive) for the storage component, and this is true https://bsky.app/profile/dustyweb.bsky.social/post/3lah5n3kld42q In fact @bnewbold and alice had gotten the server down to just close to $200/month in their estimate, much much cheaper than I had, by choosing a dedicated server plan. Much cheaper! But there's a problem though; that's cheap because you've got a server that has a dedicated disk... Even if we look at the dedicated hosting provider that @bnewbold provided in June and scale the cost to the pre-election storage requirements, we are adding on a massive amount of cost every month, over $400/month more.  But worse, we have reached the limits of what is possible to do with a dedicated server. We *have to* move to abstracted storage from this point forward because we're starting to hit the limits of what's offered for cheap dedicated storage on one machine. And this number will only grow, and as said previously, is growing at an enormous rate.  @cwebber honestly blogs are a more 'credible exit' than a bsky pds .. I have spent a lot of time focusing on the cost of storage, but storage is only one cost required. These estimates have been done so far against servers that *nobody is actually using*. The cost of servers that people are using will be much higher, because more needs to happen than just store things. And that is not even to mention the challenges with administrating, dealing with takedown requests, illegal content, etc, which are probably much more serious. Let's take a break, the analysis of server costs is boring and I don't like doing it, and I'm sure people will throw numbers at me of the absolute race-to-the-bottom hosting numbers they can find to store and run all this stuff, but really that's not interesting to me. Let's do a comparison. Remember that the idea of "fully self-hosting" on Bluesky/ATProto at this point is primarily abstract; nobody is really doing it. But of course there's a place where tens of thousands of people are running their own servers for millions of users, and that's the fediverse/ActivityPub. As said, tens of thousands of people are self-hosting *today*. Fediverse software doesn't just scale up, it scales *down*. GotoSocial is cheap enough on resources where you can run it for family and friends on a raspberry pi or spare laptop you have sitting around.   @cwebber That's rather bad. Ideally a social media node would have a constant storage need, not one which needs to scale with the amount of users, the media, etc. Maybe ephemeral services like IRC were not such a bad idea to begin with. ยฏ\_(ใ)_/ยฏ The conclusion that we need big billionaire companies, just to fill our "need" to post cat videos, is a little bit silly. BUT when considering climate change, that fits the picture. We live in absurd times.  @cwebber Relevant discussion from earlier: https://bsky.app/profile/why.bsky.team/post/3lbjdux6ubc2f Non-archival relays solve some problems, introduce others.  @cwebber I use RSS to subscribe to my webcomics, but there's a surprisng lot where I can't  @cwebber Yep! I often cite that as an example of what I call "de-invention"   What exactly do you mean about content continuing to exist? Do you just mean to split into a content server and a regular server, where each had to be taken down separately?  For the chronically curious: are there publicly available details about the AP demo youโre referencing here?  @cwebber damn I thought he actually did own Bluesky but I looked it up and he just went back to Twitter where he follows Elon Musk and calls that hellsite "freedom technology" ๐   @cwebber@social.coop yeah i had a rant the other week about how fedi is a decentralized platform and bluesky is a distributed platform. the history nerd in me also went "bluesky is federated because a federation in government implies a central power which is the relay" and "ap is a confederacy because there is no central power". Of course nobody actually says we're on the confederate network because optics.   @solonovamax @cwebber @bnewbold literally no one on fedi thinks mastodon is "great". people either believe that fedi is just mastodon or realize that there are places outside of bad decision website boy's sphere of influence
 @cwebber @bnewbold i think stating upfront that you are trying to be kind and objective in your technical analysis, before your technical analysis, is important, because its so easy for readers to take things personally when you arent intending to do that. it's also great for your mental health, where if someone does give u an earful, its kind of on them to realize that you put in an effort to try and be kind, and that you even considered it in the first place. @cwebber I've been on Tribel, which is also decentralized. It is unfriendly, moderators blame victims for being attacked (IF they want to moderate), full of anti-LGBT+ posters, if a post is removed, it looks like YOU were the one at fault, and your entire page will be filled with '[poster] is a herpes infected muppet'. You can easily get banned for being trolled. Mastadon and Bluesky are very safe, though.  @cwebber Thanks for the analysis, really useful for non tech people but eager to learn what are the specific differences. Check out @clubmanhattan  @cwebber This is excellent, thank you for writing it up. It roughly matches my gut feeling from the little I knew about BlueSky's architecture, but of course with much more expertise and detail    @cwebber@social.coop I read many time the paper to be sure to not miss that point, but it seems you never consider the fact that a relay is not needed and that's explicit on the AT proto documentation here  @cwebber This is great. Thank you for taking the time to write it all up. One thing I like about this space is that people on all sides seem collegial. They acknowledge the issues with their own solution and praise the strengths of others. You don't get that much anymore.   @cwebber Thank You :D i am only at like 10% through and mostly know the "BSky is cosplaying decentralisation" article and a few other smaller things and i already learned a lot new or things that i did misunderstand or not know :)  @cwebber this is was a super interesting and insightful read. I learned a lot from it. Thank you!  Not in the slightest. They even defined their own AT protocol for that. On the other hand, i am on a really decentralized network, so i couldn't care less what they do.
BY THE WAY if you are reading this, I am dual-posting here and on the Bluesky simulataneously, dual-wielding social media "platforms" https://bsky.app/profile/dustyweb.bsky.social/post/3lbkkkj5mhs24 Why not follow the same thing in two places at once I guess  Yeah, it's definitely not federated and it's not distributed and it's not compatible with mastodon or threads. It's garbage and lies, just like Twitter.  @cwebber I'm just going to link this the next time you bait people to mansplain...just be like, oh this really clever person explain it all, you should really read it! ๐ @cwebber i am actually pretty curious , with spritely do you imagine that it could say scale to a global network? are you architecting it to never get that big? i like petnames, also a fan of mdns and just using sshfs to mount directories and just share files without any fancy mutable/immutable append only logs .. but i feel like my viewpoint is small. do you have any long term views for how p2p could actually scale? or will we always be in punctuated equilibrium between central vs decentral?   " Then along came Google Reader and... friends, if you are reading this and are of a certain age range, there is a strong chance you have feelings just seeing the phrase "Google Reader" mentioned." You got me. Great read, thank you.  Thank you for this article. I had a very general understanding of how Bluesky works, you helped me get a more coherent picture. It's even more centralized than I thought it was ๐ |
First of all, before I say anything else, my goal here is NOT to be mean to Bluesky's devs. I know there's a lot of fediverse-Bluesky rivalry, but I have enormous respect for Jay Graber and her team and I know they believe in their vision!
This started because I got some very kind encouragement by @bnewbold to write something. I'm trying to be technical in my analysis, not unkind. I hope that can be recognized, really and truly.