Gregory's ServerWe accidentally invented computers that can lie to us and we can't figure out how to make them stop
 16 comments
 @simon a computer cannot deliberately lie because a computer cannot form intent. Sycophancy and sandbagging as you describe them here are emergent properties of an ML training regimen, things that you are training the model to do without the *humans* having the intent to do so, despite doing all the steps that predictably result in this behavior of the system @carlmjohnson @glyph I'm actually considering doubling down on "lying" as a term that's useful to use "ChatGPT lies to you" is a clear and important message for people learning to use these systems I'm not convinced the semantic debates over intent are genuinely helpful in getting this important message across @carlmjohnson @glyph "ChatGPT can hallucinate" is I think a much less useful message to people just starting to explore these tools @simon I think “lying” is punchier but it encourages anthropomorphism. At this point we need more public xenopomorphism though. LLMs are weird! Maybe in the future a human-like AI will have an LLM module, but today it’s more helpful to know about the token window and HFRL and whatnot. @carlmjohnson much as I dislike the anthropomorphism - I really wish ChatGPT didn't use "I" or answer questions about its own opinions - I feel like that's a lost battle at this point I'm happy to tell people "it has a bug where it will convincingly lie to you" while also emphasizing that it's just a mathematical language emulation, not an "AI" @simon @carlmjohnson I guess I also object to this term because it doesn’t really have a bug—it isn’t really “malfunctioning” as I put it either. The goal that it’s optimizing towards is “believability”. Sycophancy and sandbagging are not *problems*, they’re a logical consequence and a workable minimum-resource execution of the target being optimized. It bugs me that so much breathless prose is being spent on describing false outputs as defects when bullshit is *what LLMs produce by design* @simon @carlmjohnson if it accidentally wastes resources telling the truth where a more-compressible lie or error would have satisfied the human operator, that’s a failure mode! It will eventually be conditioned out in future iterations, although an endless game of whack-a-mole will ensue as they try to pin it down to *particular* “test” truths (which is exactly what “sycophancy” is) while all others decay @simon @carlmjohnson it worries me a little bit that I, with just like, a passing familiarity with what gradient descent is and how ML model training works, can easily predict each new PR catastrophe and “misbehavior” of these models and the people doing the actually phenomenally complex and involved work of building them seem to be constantly blindsided and confused by how the tools that *they are making* behave @simon @carlmjohnson @glyph Other people have pointed out that "bullshit" is more accurate than "lies", and honestly I think it's just as punchy and to the point.
@sketchytech @simon it seems that these LLM applications are now plausibly hallu-citating nonsense-generators, so they'll be putting political speechwriters out of business soon. @simon this feels related to the inner alignment problem Robert Miles described a few years back. I get the feeling it may be intractable. https://youtu.be/bJLcIBixGj8 @simon LLMs can’t lie, they can only ever output tokens according to statistical probability derived from their training. It responds to its input exactly as it was trained to do with zero understanding or agency. Please don’t fall into the anthropomorphism trap like so many others. This is a great, clear read on the differences between the ways in which humans think and LLMs predict, a short paper by Murray Shanahan https://arxiv.org/pdf/2212.03551.pdf @StuartGray I'm not convinced by that I think it's possible to use the term "lying" while also emphasizing that these are not remotely human-like entities |
(If you don't think it's possible for a computer to deliberately lie, take a look at "sycophancy" and "sandbagging" in the field of large language models! https://simonwillison.net/2023/Apr/5/sycophancy-sandbagging/ )