Gregory's Server@nobodyzhome It's a distributed systems design interview question, meant to be solved at a high level, in the abstract. I think he literally thinks the implementation is therefore that easy.... 😂🤷 @hu_logic @rodhilton
|
36 comments
@reneestephen @nobodyzhome @rodhilton Imagine you're part of 10X Hardcore 2.0 skeleton crew and have been troubleshooting Sev1 production issue for a week. As a result, you haven't had any time to write any "code" so Elon fired you... Then, it turns out that Sev1 was caused by Elon randomly unplugging shit in the DC...  @reneestephen @nobodyzhome @rodhilton Somewhere, some CTO talking to his VP of infrastructure: "Sure, we have DR plans and architecture. But do we have Musk-proof DR plans and architecture? Make it happen!"  @reneestephen @hu_logic @nobodyzhome @rodhilton this is what I am wondering - at what point does that clock run out? @grrrr_shark that depends on the landlords he stiffs and the forbearance of various cloud providers to take a reputation and NRR hit. If they believe he won't turn it around any time soon they'll cut their losses and start proceedings, but... it might take awhile. @hu_logic @nobodyzhome @rodhilton @grrrr_shark @reneestephen @hu_logic @rodhilton @nobodyzhome He’s also running out the clock on several certifications; they’re either expiring now or in the next couple of months, and you just *know* that the people responsible for staying on top of this and who know which forms to send where got fired in that first purge. @nobodyzhome @hu_logic @rodhilton @reneestephen we’re assuming somebody didn’t point him at a rack of IDK, HVAC, tell him it was vital vital server equipment and turn it back on as soon as he left the room. Or equally likely, he’s just fried HA and yeah, everything still works… @nobodyzhome @hu_logic @rodhilton @reneestephen Or just a loss in surge capacity. The difference between NBD and OMG is often no more than when it happens. But fried HA means someone needs to spend time, maybe dollars, standing shit back up. Which is not always risk free either. @hu_logic @reneestephen @nobodyzhome @rodhilton Your screenshot was so ridiculous I just wasted time going to confirm that it was real. @nparikh @reneestephen @nobodyzhome @rodhilton NGL, I lost some braincells and faith in humanity reading his Tweet - "Twitter stack is so shit that I've unplugged half of its nodes and it still works! See? Total rewrite!" @hu_logic @reneestephen @nobodyzhome @rodhilton It is *literally* what a chaos monkey is. Usually the idea is the chaos monkey is a concept or tool, not an actual guy let alone the CEO. The line about Rube Goldberg is a pretty strong hint about what he doesn’t know about engineering, too. @nparikh @reneestephen @nobodyzhome @rodhilton Some people have a gift of taking a failing startup and turning it into a multi-billion-dollar enterprise. Elon has a gift of taking a $44B enterprise and turning it into a failing startup... @hu_logic @reneestephen @nobodyzhome @rodhilton Somebody trying to be *reverse* Steve Jobs is definitely a new thing to witness.  @rodhilton @nparikh @hu_logic @reneestephen @nobodyzhome He has the Shidas Touch. @nparikh @hu_logic @reneestephen @nobodyzhome @rodhilton @hu_logic @nparikh @reneestephen @nobodyzhome @rodhilton Yeah apparently the self-proclaimed “smartest person in the world” has never heard of redundancy or failover clusters. @MGoCoder @hu_logic @reneestephen @nobodyzhome @rodhilton Somehow he also doesn’t connect it to why it can feel like a bunch of Rube Goldberg machines. It’s like Lewis Black’s bit on “if it weren’t for my horse” — if you spend too much time thinking about it you’ll get an aneurysm. @nparikh @MGoCoder @reneestephen @nobodyzhome @rodhilton Somebody tried to explain "systems integration" to Elon. That person has been fired. @MGoCoder I mean, where would he have? For real making cars is all low-latency on-prem automation and MechEng, a lot of factories are deeply bad at BI data and observability, even newer ones. Same with rocketry. He also hasn't had people at Twitter whose job it was to explain this shit to him and steer him away from the worst ideas (or just ignore him/lie to him and do what should be done despite him), like in his other companies.  @reneestephen @MGoCoder @hu_logic @nparikh @nobodyzhome @rodhilton I've seen the stories about how at Tesla and SpaceX (and, presumably, Neuralink), they have a thick layer of bureaucracy to manage him specifically. Full self-driving aside, his main two companies manage because he's not doing the engineering, and just knows a bit of the jargon. Twitter, he's actually trying to do the work and showing off he doesn't know how. @hu_logic @reneestephen @nobodyzhome @rodhilton wait, is this screenshot real? it can't be real, can it?  @herid @hu_logic @reneestephen @nobodyzhome @rodhilton It IS real, sadly. This guy is definitely a moron and an idiot. My own screenshot 👇🏻  @hu_logic @reneestephen @nobodyzhome @rodhilton I thought the solution to all technical issues is to turn the machine off and turn it back on again. It worked in #Jurassic Park. Apparently #ElonMusk agrees. @hu_logic @reneestephen @nobodyzhome @rodhilton Lmao, this reminds me of when I was working at a place that had a data center with a storage farm. The director was taking some people on a tour, there were two tour groups, a morning and an afternoon one. To demonstrate the fault tolerance he showed the group that he could just pull a hard drive and things would keep humming along. He plugged it back in and it started recovery. @hu_logic @reneestephen @nobodyzhome @rodhilton Did the same thing for the afternoon group, only it was the same node and it was still recovering and didn't have enough redundancy left, so it took down the whole storage cluster for a while.  @flabberghaster we all wait for that not to happen with twitter.. Musk is doing a nice job with every week, every fortnight creating a new wave.. @hu_logic @reneestephen @nobodyzhome @rodhilton I'm willing to bet Musk doesn't understand fractals, Goldberg, or how Twitter works. @hu_logic @reneestephen @nobodyzhome @rodhilton @reneestephen @hu_logic @nobodyzhome @rodhilton I'm actually shocked nothing substantial hasn't happened to rock that system. Clearly, the team did a good job designing and building it out. But if he keeps messing with the system here and there, he's bound to crack it in half.  @reneestephen @nobodyzhome @hu_logic @rodhilton yeah it's fairly similar to a design question I got at my Google interview. Any engineer who works on or just understands robust distributed systems should be able to *sketch out* what Twitter's infrastructure would look like. (I'm primarily an application/library/tools person so for me it's actually a kind of spicy problem. I still answered well enough to get the job.) |
@reneestephen @nobodyzhome @rodhilton Twitter had to optimize and prioritize network routing between its DCs in order to scale properly. That's the kind of problems such implementation would have to solve at the grunt level.
To get it done right takes time, money and quality engineers.
Elon thinks he can do it with twenty sleep-deprived "10X Hardcore 2.0 SWEs" in 6 months on a shoestring budget... while shutting down several DCs in January.