|
Trusting LLMs threatens your credibility. I read a bogus claim about GPU instruction sets, cited to GPT-4 and an anonymous "expert". This is my area of expertise, I know the claim is demonstrably false. And now I know the author is relying on bullshit generators. Now I doubt every other claim the author makes, because with egregious errors in the parts I know about, how could I trust the parts I don't? (Edit: narrowed the scope of the lead.) 82 comments
@mnl @alyssa I have to correct first years who were convinced ChatGPT got the right answers for their math homework, when it's consistently dead wrong. "But it gave a different answer" is really unconvincing with that in mind.
(Unrelated gripe - first years in CS, so writing a program to do it would be much less likely to fail.)  @alyssa Some publication or columnist confidently spouts absolute nonsense about stuff in my own field; and now I can never trust what they say about any other subject.  @alyssa Reminds me a bit of "journalism" in general. The amount of nonsense I read in credible newspapers, speaking about "tech(nology)". Makes me wonder how much nonsense they report in areas I'm not very knowledgeable about.  @sindastra @alyssa I have the same feeling. Basically LLMs are simply amplifying what has been and making it even more apparent. @alyssa what are your thoughts on isolated decision making behind a fixed input/output layer? For example imagine if Siri etc. Used a LLM to interpret and do language parsing but NOT to build the response  cool but you could make it better by getting rid of the bit where it does language parsing (serious answer, i cannot think of a justification to myself for why this is an improvement over just _not_ having that step) @erisceleste I mean, isn't one of the most common issues with virtual assistants that they misinterpret a query? For example, consider the following exchange with ChatGPT that Siri (and probably others??) would've completely failed at. Note that it properly parsed out the request AND corrected my spelling. perhaps i speak from too much of a cultural bubble when i say that groceries:append(apples, 1400) seems better to me than using english to describe a task that doesn't need to be described in english, in the first place this can be three button-presses and a swipe of a drop-down @erisceleste I appreciate why you would feel that way, but I hope you understand that the general population just wants to be able to make a request and have it understood, not following a syntax @alyssa It's a little disheartening that the mass response to public LLM availability seems to be "Well now I get to think EVEN LESS!". @omegajimes @alyssa They are an excellent tool for those who know how to use them, and unfortunately, misunderstood by most  @alyssa LLMs works best for popular texts, with huge corpus and data. For sure GPU drivers programming is faaar from it. They might be useful, as a tool, for helping in some tedious, repetitive work, but for sure not for such niche work. It should always be used as a tool - with big amount of scrutiny. Huge problem is, that if they „don’t know” something - LLMs are simply confabulating 🙃 Extremely dangerous for beginners or less ML-literate people 🫤  Even with a huge corpus of data, LLMs are useless, and here's why: They generate text which looks like, but is not, equivalent to a researched and reviewed paper. They will take bits and pieces from the entire set of articles and chunk them together into something which is functionally meaningless but looks acceptable at a casual glance. And I mean individual words! Sentence fragments! Syllables! They don't know ANYTHING. But they give the illusion of doing so. @theogrin @kkolakowski @alyssa LLMs don't necessarily need to generate stuff There are signs of promise for LLMs that avoid hallucination by paraphrasing and permutating instead. I recommend checking out perplexity.ai LLMs are also quite helpful for automation; the base training data is just to get the relations right in the first place, then constraints, checks, temperature, and human validation can help vet things out @theogrin @kkolakowski @alyssa on a slightly related note: the annoyances of hearing clearly-unqualified people tell you your job is worthless, your passion is not worth pursuing and your work is unnecessary because of LLMs and AIs in general... you know they're wrong, but it's disheartening still.  @ryanc be prepared to be very much more depressed. LLMs are very exciting to the Venture Capital set. @Seruko at least i'll be in school during the AI bubble, my condolences to everyone who's about to get/already got replaced by AI because of corporate https://www.cbsnews.com/news/eating-disorder-helpline-chatbot-disabled/ @alyssa Could LLMs be suitable for generating proofs which are then checked by a _sound_ proof checker that is secure against malicious input? If the proof is wrong, the checker will catch it, so no harm has been done. Depends how you define harm. The requirement for checkers becomes exponentially greater with the use of bots and large language models. Of course, one should perform one's best efforts to check the validity of any resource, but the need for more robust and cautious checks increases the time requirements greatly. Also, it feels like generating noise at random and then checking for anything which could be a sonnet. @alwayscurious @alyssa only if But even if those conditions can be met without undermining the viability of the product, it could only be used in contexts for which an ~infallible checker has been integrated, which could not include general use @alwayscurious @alyssa There are automated systems to generate mathematical proofs, but I don't think those work anything like LLMs. if you know you're trying to generate something specific like proofs, you probably don't need an overwhelmingly overpowered tool like GPT-4 to do that the use of LLMs is that, through gigantic amounts of wasted computing power, they can appear to emulate a near-infinity of simple tasks; but if you have a specific task in mind the chances are _extremely_ good that you don't need the Swiss-Army-Chainsaw to do it  @funky7monkey intriguing, thank you. In my experience as a JavaScript and Python dev, it is extremely helpful with those languages, plus things like ImageMagick and jq and command-line things, so I wonder if it’s better at those than game dev because of maybe more training data or whatever? I value others’ counterexamples so thank you for this. One of the best articles I've read isn't about specific types of highly specialized work, or even loosely specialized. It's something a five-year-old can typically explain: Tic-Tac-Toe. And ChatGPT is excellent at coming up with a seemingly convincing explanation for its tactics. Long-winded and verbose. But it's pants at playing, and I think that perfectly illustrates the difference between the illusion of intelligence, and actual brains. @alyssa LLMs are an excellent way of getting quick answers to some problems, and if you're open, learn a thing or two that you haven't considered. It's useful. But taking any LLM output at face value without double checking means you're foolish and naive. They're not actually useful for that either. They *look* useful for that, but they're actually just as garbage at that as they are at other tasks beyond "stringing together reasonable-seeming English text." Me: "In what year was the Battle of Hastings?" AI: "The Battle of Hastings took place on October 14, 1066." I'm sorry but it's your credibility on the subject that's suspect. Wild general statements like "LLMs are only for entertainment" are ridiculous. Here are some *reasonable* statements: "Be skeptical of everything you read, and when it really matters always verify." "Don't use a hammer to fix your plumbing. Every tool has an ideal use, choose wisely." @LotusHopper Because for questions that have a simple, deterministic answer LLMs are generally quite reliable and it's WAY FASTER than doing a web search. As I said, right tool for the job. A search engine isn't really the best tool for simple questions with a deterministic answer anymore. There's a new tool in town that's great if you use it right. @LotusHopper Questions that can be definitively answered and that require no interpretation nor change depending on your perspective. "What's 2+2?" or "What year was the Battle of Hastings" or "What is Miley Cyrus' birthday?" Questions like "what's the best pizza recipe" or "why is Iran/America/Russia so evil" are not well suited to LLMs. what year something happened is a statement that requires a lot of interpretation and presuppositions like, what is an event in the context? what calendar is used? what are the commonly agreed upon limits of the type of event? just because interpretation seems easy to you and you don't recognise that it happens, doesn't mean it isn't necessary there are no simple facts, because human language is quite complicated @troglodyt @LotusHopper Sorry Troglodyt but that's a whole pile of pseudo-intellectual horseshit. There is zero... ZERO... ambiguity to asking what Miley Cyrus' birthday is or what year man landed on the moon. Come back to earth space man. ok, looking at your feed and your nick i must make a jump and conclude that trying to make conversation with you is an absolute waste of time, you're much too gullible and already totally occupied by less sophisticated forces than my mind good luck with your faiths little fellow  @coupland lol. you're coming from crypto-dorks instance and have dot eth in display name.  @alyssa I find they can be occasionally useful if you want exactly what they generate, which is a regression to the mean of subjective answers, queries like “what is a typical naming scheme for a node in a kubernetes cluster” have a very low likelihood of causing actual harm, just don’t trust them for anything more complicated than incredibly general questions that don’t have wrong answers. Which to be clear, is most things. You shouldn’t trust them for most things   @alyssa I ‘almost’ told the parties recently that ‘no AI was used in the preparation of this (arbitration) Award,’ which was true, but I finally figured it would be worse to say it: most wouldn’t yet get it, and those who did would think I was making a jk, that might not land right. @alyssa Yeah. I've used it for daft things like "write me a dire straits song about Macbeth". I thought considering they did one about Romeo and Juliette it'd be interesting to see what it'd do. @alyssa Gosh, I think what you really need right now is an overconfident straight white guy who got his online learning certificate in GPT4 from LinkedIn to wade into the topic and “educate” you. 🤣 @alyssa “In a time of deceit telling the truth is a revolutionary act.” @alyssa This 1000%. ChatGPT should never be used for serious work, especially anything like legal defense. I'm sorry you have a bunch of ChatGPT apologists in your replies, so I'd like to let you know, there are a lot of us who fully agree with you They're also good for any time when the output is easily tested/verified. I've used GPT for little scripts in unfamiliar languages for example. They're good for manipulating information you give them, but can't be trusted to go out and find it in the first place.   @alyssa I use LLMs for research, for economics, for summarizing, and for programming. It's all fine. LLMs are highly useful tools. Publishing LLM-produced output without extensive checking and editing is dumb. @alyssa Yeah I had a friend mess with OpenAI to write a fake vulnerability report. It lied to him multiple times, and as he continued correcting it, ChatGPT started making dead links to non-existent vulnerabilities so he LOL’d hard at it because it was just citing unrelated CVEs in his troll report 😂 Be careful about that, even for POCs. https://www.securityweek.com/chatgpt-hallucinations-can-be-exploited-to-distribute-malicious-code-packages/ @bluewinds @alyssa I'm well aware, hence I try to only use the generated code when I fully understand it. The heads-up is still very much appreciated nevertheless <3 That's my real point: it's not actually good for anything. It's all hype. If you're understanding it fully before using it, you'd have been better off just doing the work yourself to begin with! Anything that chatgpt *seems* to be good at, it's more likely to be harmful than helpful.  @alyssa LLM is mansplaining-as-a-service - confidently given completely wrong answers 🤣 @alyssa Yeah I just use it like google. For some reason people know when they click a link on google they understand the information can be unreliable. With Chat GPT they implicitly trust it. Where do people think Chat GPT gets it's data from? In this case it's probably just quoting something from a stack overflow question from someone that didn't know what they were doing (which is why they were there). But sounding really confident while doing it. @alyssa I've heard some interesting arguments for using it to come up with project names or to expand on an email you're having trouble writing. Of course the asterisks to that are you need to edit it to your own after gpt goes at it and you need to verify everything contained within is true. @alyssa I mean I use copilot for coding and it's an LLM and in that context I find it very helpful. It's not doing the decision making though, mostly just remembering obscure syntax for me  @alyssa   @alyssa @feoh to me, the larger UX threat is the knowing misrepresentation of LLMs as expert systems for every use case. I do see the use-case @feoh is talking about, and I've given it a try a few times at the encouragement of others. It's… fine. But I agree that the overall effect of these systems is corrosive on trust, because as you say, it only takes one such failure to cast a shadow on everything else, even the stuff that isn't LLM output. @alyssa There are some LLMS specifically designed for permutative writing instead of generation, such as Perplexity.ai, which do show promise for being a viable AI search engine replacement, but I feel it AI is currently misunderstood and misused by the public, especially those who confuse chatbots for base models and general AI tldr; we need better AI education for the general public to solve for false credibility @alyssa maybe I’m wrong here, but I think it’s not bad to use LLMs to ask them questions as long as you order them to give you their sources, too. So that you check again what they tell you. @alyssa Had a similar experience a couple of times. ChatGPT straight up doesn't understand pointers - I was debugging some code and ran it through to see if it could find the error and it very confidently told me that actually I should allocate space for a double pointer to a data structure, and assign the result (a pointer) to a double pointer - in which case I'm returning a pointer to a double pointer and assigning that pointer to a double pointer and ending up with a triple pointer   @alyssa@social.treehouse.systems |
Gregory's Server
@alyssa in a way using llm for facts in earnest is going to expose a lot of people that benefitted from the doubt before. Now everybody is on the lookout, and the easiest to fool are the fools themselves