|
Top-level
 We're about halfway through this blogpost. There's a lot going on in my life. I am trying so hard to keep the organization I work for alive and moving forward. I am tired. I need rest. And I still need to drive two hours across the state tonight. We're going to resume tomorrow. But first... 52 comments
For tonight, I need to unwind, I need to put a label on a mailbox, I need to eat dinner, I need to drive across the state, I need to sleep. Maybe I appear ridiculous. I get it. I go pretty hardcore on this stuff. If you know me you know I tend to go all in. I am signing off for the night. Tomorrow we will analyze whether or not my assertion that "ATProto has explosive behavior as it approaches decentralization" problems. I'm not going to read notifications until I finish this. Maybe someone will prove me wrong before I get it done. I'll be oblivious. We will also analyze values, which maybe I care about more than anything. And there will be more secret goblins, hidden among the posts. For tonight, it's rest time. It's time for a === NO MORE LOOKING AT MY COMPUTER BREAK === Hello! I am back at my computer. Today we are going to talk about how ATProto does in terms of scaling. Yes, we know it scales up, and has done an impressive job of doing so! But what about scaling towards decentralization? Does it scale down? And does it scale wide? Let's look. Before we get deep into that, when we left last night I was extremely tired and had been working at my computer for over 14 hours. I then said I was going to drive two hours across the state that evening. Thankfully thanks to the support of people who love me, I did not do that foolish thing! So anyway, I am better rested, and also I woke up to the surprise that our fundraiser is doing a lot better, like by a lot, than it was yesterday, which is nice because I was extremely stressed out https://spritely.institute/donate/ So I am feeling much better and alive and today I remembered to eat lunch But you probably aren't here to hear about my lunch choices or how much sleep I got or whether or not I forgot to bring my ADHD medication with me (I did so now I am drinking a bunch of caffeine instead), you are probably here to hear the rest of the analysis about decentralization and Bluesky etc So let us get to it, let's talk about whether or not Bluesky can scale *down* in a meaningful way. In my last essay I made assertions that this was important for decentralization and said ATProto wasn't great for this, and this was one thing people challenged me on So let's take a look! When I say "scale down", what I generally mean is "small instances can generally participate on the network". (We'll talk about "scale wide" later.) But another useful possibility which has come up is "can you make a smaller, more isolated use-case and use the same protocol for it" This latter version of scale down does come up in Bryan's article: > A specific form of scale-down which is an important design goal is that folks building new applications (new Lexicons) can "start small", with server needs proportional to the size of their sub-network. (cotd) Strictly speaking, I agree, ATProto can scale down in this use case! For example, if you wanted to make a small specialized forum for collaborative storytelling, you could use ATProto for it, and that's true, you could do it But is it the right choice? In some ways we are talking about two different things here: extension of functionality (which you might want the same scale for) and having a smaller and more isolated community But regardless ATproto positions itself *specifically* as designed for not wanting to miss messages, and I talked previously about how ATProto's design requires a god's-eye view. It's a bit strange of a choice when you say "let's run a smaller community" Given that message passing systems handle small scale systems *beautifully*, and *still* allow for interactions with larger scale systems, it's a bit confusing to me *why* you'd choose ATProto for such use cases. What is the specific benefit you'd gain? Especially because it's actually lossier here At any rate, there's a bit of conflation here. "It scales down" by saying "you can have an isolated community/use case that's oblivious to the rest of the system" is categorically distinct from "it scales down" in terms of "a small node can meaningfully participate with the larger system" At any rate, the problem with "scaling down" is much clearer when it comes to the problem of "scaling wide". Or let me put it a different way: ATProto *explodes in complexity* when you try to scale it towards meaningful decentralization Yes that's right we're getting to the spicy part of this conversation. We did the warm-up, now it's time to talk about the real thing, whether or not decentralization in the way I believe people *think* that term means is reasonably possible with ATProto as it's currently designed But before we do that, I need to stretch and run to the bathroom So for those of you following along, if you found this, Secret Goblin #3, let me know: "👺" Oops wait actually we gotta talk about that one for a sec there's a reason I left it in scare quotes Why on earth is the textual descriptor for Unicode U+1F47A "JAPANESE GOBLIN", does anyone know? It's a Tengu, right? Despite being the only actually named "goblin" emoji, I feel awkward about this one because is it correct to call it a "JAPANESE GOBLIN" instead of just "TENGU"?!?! I don't know! If you have knowledge or OPINIONS about "👺", its name choice in unicode, or, for that matter, a white person just dropping it in the middle of a group chat WITHOUT putting it in quotes (I did tho), feel free to derail the comment thread Otherwise it's time for a === STRETCH BREAK === I'm back. It's time to talk about it: does Bluesky/ATProto suffer a "quadratic explosion" as we move from centralization towards *meaningful* decentralization? I claimed it did, but I was challenged on this. What did I mean? Am I right or wrong? It's time to find out! In the previous blogpost I said the following: > If this sounds infeasible to do in our metaphorical domestic environment, that's because it is. A world of full self-hosting is not possible with Bluesky. (cotd) Decentralized ATProto is quadratic quote, cotd: > In fact, it is worse than the storage requirements, because the message delivery requirements become quadratic at the scale of full decentralization: to send a message to one user is to send a message to all. Rather than writing one letter, a copy of that letter must be made and delivered to every person on earth. This was probably the thing I got the hardest pushback on from a team member of Bluesky, that it is not quadratic as we scale towards decentralization. Truth be told, I don't have a degree in CS. Most of what I know I learned from studying independently and community resources. Was I wrong? Just as a quick aside, regarding that comment about "agency", maximizing the agency of everyone (and more importantly, minimizing subjection!) sits at the heart of my ethical framework https://fossandcrafts.org/episodes/11-an-ethics-of-agency.html So I don't disagree on that part, but that's an aside! Now, I said I won't read replies until I am done summarizing things, and that's true, so maybe someone has gone out of their way and proven that I am wrong, that the claims in my article are factually incorrect and so on and so forth. I wouldn't know yet. But... I don't think I'm wrong. As said I'm very self-conscious about these things because I *don't* have formal CS training. But I do a lot of research and so I've tried to become knowledgeable about these things and this *seemed* like the correct analysis to me Because of that, I turned to people who actually knew more than me For one thing I derailed the entire Spritely morning standup by walking everyone through the scenario. I gave the story example, which I'll detail later. But @dthompson didn't find the story helpful, too much narrative detail. "I need to work through this example independently." So he did. @dthompson came back and laid it out in more formal terms and said I was right. But I was still nervous, so I called up one of my old MIT AI Lab type friends and rambled about it to them on a call. What did they think? "I think it's pretty clear immediately that it's quadratic. This is basic engineering considerations, the first thing you do when you start designing a system," they said. Well that's a relief, why isn't it clear to everyone else, I asked? So they suggested I lay it out to you as I did to them. Let's start with the following: - ATProto has positioned itself as "no compromises on centralized use cases". Well, in that case, let's say it can't do *worse* than eg ActivityPub. This includes with replies. You can't do *worse* than ActivityPub on replies and mentioning someone, etc. - We will interpret the most centralized system as one where there's only one provider for storage and distribution of all messages: the least amount of user participation - Just as blogging is decentralized but Google (and Google Reader) are not, it is not enough to have just PDS'es in Bluesky be self-hosted. When we say self-hosted, we really mean self-hosted: users are participating in the distribution of their content. - We will consider this a gradient. We can analyze the system from the greatest extreme of centralization which can "scale towards" the greatest degree of decentralization. - Finally, we will analyze both in terms of the load of a single participant on the network but also in terms of the amount of network traffic as a whole. Okay. That is the structure we will use for our analysis. Let's compare "message passing" vs ATProto-style "global public shared heap". So okay. Let's get the CS notation out of the way: "Message passing" at full decentralization: Okay, that's reasonable and what you'd expect "Public global no-missed-messages (or not worse than AP) shared-heap" ATProto style at full decentralization: Oof I'd better back this up because that ain't good! In other words, as our systems get more decentralized, message passing handles things fine. Individual nodes can participate in the network no matter how big it gets. The zoom-out for the network as a whole doesn't get more complicated as we add more users OR move more users towards self hosting. Things are NOT good, if I'm correct above, as we make things more decentralized in the atproto-public-shared-heap model. The more self-hosting and indeed the more "full nodes" join, the more it gets expensive for each of the nodes and the network EXPLODES! Truly self-hosted atproto is NOT POSSIBLE! And there is no solution to this without adding directed message passing. Another way to say this is: to fix a system like ATProto to allow for self-hosting, you have to ultimately fundamentally change it to be a lot more like a system like ActivityPub! Now I left more of the precise analytical explanation in my blogpost. But social media isn't great for that, so go check out my blogpost if you want to go through all that (eg if you're more like @dthompson and less like me, I'm a narrative person) https://dustycloud.org/blog/re-re-bluesky-decentralization/ Here's our story: Now just before you say "wait but ATProto isn't for DMs", yes, but one way this could happen is that eg Bob follows Alice, Carol follows Bob, etc. What I'm saying is, messages can have an "intended audience". That's what we're using here. Before we get into this, remember, the main difference between "message passing" and the "shared heap" is the former has directed and delivered messages, the latter does not. See prev blogpost for explainer. So, what happens in a day for both systems? Because that's what we really want to find out. Under message passing, Alice sends her message to Bob. Only Bob need *receive* the message. So on and so forth. - For an individual self-hosted node, messages passed per day: 1. That's about what we'd expect. Under the public-gods-eye-view-shared-heap model, each user must know of all messages to know what may be relevant. Each user must *receive* all messages. Sounds survivable with 26 users though, right? Message passing: Public gods-eye-view-shared-heap-model: Now, could we handle a million self hosted users? Is it possible? No problem in message passing. EXPLOSIVE with atproto. What if we had a million users and added just 5 more? How many more messages must the network bear? 5 new messages in message passing. "Christine that's ridiculous, we're not expecting a million self-hosted users" Well I think it would be nice! But regardless, ActivityPub has 27,000 servers on it, all meaningfully participating in the network. ATProto, in its current design, would be crushed to DEATH "But Christine", you may say, "I heard gossip might fix this!" No. It cannot. In fact, I was being more generous than a gossip network, and assumed you only *received* a message once. With gossip you might *receive* more than once. But you need to receive a message to know it. |
Gregory's Server
There's a reason I'm going really hard on this. I really care a lot about the shape of the internet. And tomorrow we're going to get into some more analysis and a talk about *values*, and one thing I like is that Bryan talked at length about Bluesky's values. And I think that part was really good.