Gregory's ServerI was forwarded this screenshot and it just is living rent free in my head right now.
 111 comments
 @sophieschmieg "We trained our neural net on thousands of images of lava lamps and can now generate an endless series of random numbers!" @wordshaper @sophieschmieg "quality is important to us so we paid for midjourney's mega plan subscription to generate our lava lamp picture dataset"  @sophieschmieg I know you have your methods, but if interested I've become the first only true random shitpost generator using my brain trained on naturally recurring shitposts with enough anthropy and randomness to defend against a whole quanta of warm tea    @sophieschmieg congratulations on your field having achieved enough fame to receive crank theories, heh @ireneista technically Bas got this one. But I do get my fair share of cranks, and have been for a while, cryptography just has a very high crank density. @sophieschmieg @ireneista if you've got enough tolerance to brush them off, the cryptography cranks are an amazing comedy source. I especially like the "revolutionary encryption algorithm" ones who have no formal training and put together some swiss cheese for public review @astraleureka @ireneista there is lots of fun to be had breaking their implementations. My favorite one so far is this one: @sophieschmieg @ireneista heck yes this is EXACTLY the type I was thinking of. 10/10 @astraleureka @sophieschmieg @ireneista We get those in compression too. Every now and then some eager amateur introduces their miraculous new compression algorithm with impressive claims and no clear explanation of how it works. I've made a few myself. Mine do work, they just don't work well enough to be any improvement on existing methods.  @sophieschmieg It's like those people who found a way to have a distributed data structure that's consistent, available and partition tolerant. Apparently the word "theorem" doesn't apply to them.  @sophieschmieg who would win: - random numbers from radioactive decay @amberage it isn't even a bad imitation of radioactive decay. True random numbers are random, i.e. there is literally, provably, mathematically no structure for a model to learn. The randomness in a Gen AI output is a (usually cryptographically insecure) classical random number generator. It's one of the few things we extremely well understand about GenAI, because we wrote that RNG to be part of it, it does not depend on the input data at all!  @sophieschmieg there's also the noise introduced by the GPU scheduler doing the matrix multiplies in a different order which produces different results because float is not associative. Surely they meant that... Right?... But also probably that isn't true random either.  @sophieschmieg also quantum computers can't fucking break rngs* *before the end of the universe He clearly stated "if interested". Which means if you're not interested there's no world-first quantum-proof random generator. Quite obvious. @sophieschmieg If LLMs are snake oil, this "AI RNG" is meta-snake oil. It's like expecting a homeopathy distillation of horse dewormer will cure Covid. It's so obviously fake that I can't even find a good metaphor to explain how bad it is.  @sophieschmieg@infosec.exchange Ironically most GenAI implementations have troubles on producing deterministic output due to floating point errors, inconsistent batching, etc. Not random enough for crypto, but random enough to create replication problems. It's what I call Murphy's Duality Law - In engineering, when a system can show both the property "A" and its negation "not A" depending on the specific context, it's always the opposite of what your application needs.  @sophieschmieg LMAO what??? There are ppl trying to use LLM output as RNG??? And thinking "I'm too stupid to understand how it works so that means it's secure!!!111" ??? 🤦 🤏 🎻 when they get pwned. I'm out of patience for the LLM fan 🤡 🚗 @sophieschmieg BTW not criticizing your choice of MT as an illustration because it's exactly the sort of thing these bozos would know by name, but it's utterly the worst choice of deterministic PRNG. Gratuitously large state, poor output quality. Even a 128bit or possibly even 64bit LCG throwing away lower bits is better.  @sophieschmieg Claude has a response for ya. "You're oversimplifying. While language models do use probabilistic token selection, reducing them to "fancy RNGs" is like calling a brain "just electrical signals." The learned probability distributions capture complex semantic relationships and patterns from human knowledge. That said, your skepticism about AI hype is fair - there are plenty of overinflated claims worth challenging."  @sophieschmieg "Let's generate low quality random numbers about as fast as a grandma knitting socks using terra-watts of power in billion dollar data centers." - said no one ev... Oh wait.   @sophieschmieg When you said the term, I just assumed they meant that they use an llm code generator o create a program that generates "cryptographically secure random numbers". I'm sure your standard llm can give you something that resembles this. It'll take about 5 minutes, and then you can spend the investment capital on more interesting things (like private jets).  @sophieschmieg My pal Derek's essay seems relevant here: https://www.tetragrammaton.com/content/making-heads-or-tails @sophieschmieg All of GenAI is the thought experiment "What if we did a shitty version that doesn't work and needs as much power as a city" except some bro did it for real, so this seems like a natural application. @sophieschmieg I'm waiting for the double-pun twist where they sample the black body radiation from the black box of the generative ai heating up a GPU, but I don't think it's coming... @sophieschmieg I’m wondering if I’m the only one who saw that and thought it looked like Star Trek Enterprise :-) I’m ignoring the AI horror aspect. Ain’t got brain cells to think of that…  @samantha42 wen I first saw the post, I thought the resemblance to the Enterprise was why it was living rent-free in @sophieschmieg 's head  @sophieschmieg oh god that's not how regression modeling works, that's not how any of this works :picardfacepalm:  @sophieschmieg Thank you for sharing the rent-free. And, yes, this person is real, and apparently believes he is doing something interesting. https://www.linkedin.com/in/eric-dresdale/ (don't have liquids in your mouth when reading…). @paulehoffman oh yeah, he founded a company, apparently. It doesn't say if he actually found anyone dumb enough to invest in it, but yeah. Remember, it's only fraud if you actually understand that what you're doing is nonsense! @sophieschmieg "actually" and “understand" both carry a lot of weight in that sentence… @sophieschmieg @paulehoffman It's funny that his website includes this quote: "Any sufficiently advanced technology is indistinguishable from magic." - Arthur C. Clarke -- Yep, yep, I know it seems like magic, but trust me it's not marketing scam with several evidently false claims, it's actually incredibly advanced technology that you are too stupid to understand  @sophieschmieg That is, quite possibly, the most over engineered version of the XKCD random number generator I’ve ever seen. @david_chisnall @sophieschmieg It kinda reminds me of that Bash.org quote about "our robot uses sophisticated sensors and input processing algorithms to generate an accurate and useful map of the world around it / then it discards all that and drives into walls"  @sophieschmieg "enough entropy and randomness to defend against quantum" I'm sorry, you'll need to pass the dressing if you expect me to swallow that word salad.  You'll be sorry when quantum is invading your home and eating all your food and leaving a mess everywhere and you have no defence Translation: "I am a fucking shitbag arsehole who will steal your money if you are stupid enough to believe a word I say. Also, quantum."  @sophieschmieg they had me at “software only TRNG”. But it just gets better and better. Chef’s kiss.  @sophieschmieg please tell me no one got investor money on "random number generator, AI edition" this is wild @sophieschmieg Ok, but is their “true" random number generator on the blockchain? /me ducks  @sophieschmieg that's cool that they did that! *stares blankly in excitement because he doesn't understand anything about encryption OR quantum*  @sophieschmieg Ugh I guess it was only a matter of time before the snake oil cryptographers discovered LLMs.  @sophieschmieg I think the best part of this is the plain and simple fact that it's ultimately using the system's random number generator anyway. Well, that's the second best part. The best part is that their random result I swear is just a ASCII art gen of a starship flying through space.     @sophieschmieg Mega Man riding a tractor, as that image seems to show, is pretty random in the colloquial sense.  @sophieschmieg So let me see if I understand this correctly. This person just dumped something like the output of the Cloud Flare lava lamps into a data set, threw it in a ML system and pulled peseudorandom data out of it?   @sophieschmieg Gold. Security professionals say “Hi”. The ‘enough’ part is wonderful. Who decided it was enough? Is it more than enough? Only just enough? How long will it be enough for? How did they get entropy from an AI model? Let me guess, they just threw a $reallySecureRng function in there. Good grief.   @sophieschmieg omg, that's hilarious I'm just sitting here trying to imagine how someone convinces themself that this would be either secure or efficient @sophieschmieg we are EXTREMELY scared, this is some of the biggest hell nope things we've seen AI bros come up with  @sophieschmieg  @sophieschmieg had a quick look at his feed and apparently he is a "pepe coin holder", so definitely the kind of technologist you want to put huge trust in @sophieschmieg I had to look at the full threat to understand that this guy is serious and not just making shit up. But then again, he is still on Xitter, so In guess I shouldn’t expect much of him in the first place…    @sophieschmieg I know you have your methods, but if interested we’ve created the first catgirl-only true random shitpost generator using generative ai trained on a naturally recurring phenomena (cat farts) with enough entropy and stupidness to defend against posts like this from living rent free in your head.  Hey AI can I have a random number 7 Hey AI can I have another random number 7 Hey AI will you always reply with 7 when I ask for a random number Yes, 7 is the most common number humans respond with when asked for a number from one to ten. |
@sophieschmieg @lexi thanks this is terrifying