@jonny I had the great joy of seeing some folks at MIT present on using LLMs as they're meant to be used, which is to say as interfaces to more complex computational processes that just, convert otherwise difficult-to-parse data into more human-friendly language

god i hope the popular consciousness realizes that's what they're supposed to be sooner rather than later

@jonny My experience with ChatGPT as an aid in programming is that you end up tutoring ChatGPT no matter the language. Also ChatGPT is exquisitely bad at SQL. Maybe I should check how bad it is at VHDL. There is a potential for a very bad code here.

@jonny how did he ever come to believe that chatgpt was capable of doing something like that? it's absurd to think that!

@jonny

Oh dear god. I have a programmer coworker who just loves these LLMs for "coding assistance." I wonder how much time he spends banging his head against the wall fighting AI hallucinations.

@jonny so ... how did one become a code debugger, yet somehow come under the impression that ChatGPT isn't a LLM?

There seems to be an interesting story to uncover there. It genuinely intrigues me that a programmer would think that an LLM evaluates code for you, or that it isn't a LLM.

If we could figure out why a programmer thinks like that, maybe we could find a clue that'll help to rescue the rest of us.

@jonny #enshittification : #debugging edition

@jonny I had an interaction a few weeks back where somebody had asked in a public channel how to do something with ffmpeg. Another person gave a solution with a list of flags that was garbage, and only admitted that they'd "just fed that to the ol' GPT" after being told it didn't work.

When I called them out on it maybe not being a good idea to dispense LLM synthesised text as advice, they seemed surprised that a) the output wasn't helpful, and b) that what they'd done was objectionable.

@jonny on one of my projects I ended up using one of the unit testing frameworks that ChatGPT named when I asked it for suggestions but beyond that it's been about as helpful to me as a rubber duck or possibly an initially blank notes.txt.

@jonny also shows that naming this models AI suggests they do more than they can. My father, working at a biotech company, also asked a LLM to name the top 5 plasma products his company produces. He was astonished how wrong the answer was: 2 non plasma products, 3 from another company, one correct. Tried to explain to him that this stuff basically works like autocomplete - not fully correct but the analogy fits IMHO.

@jonny my only regret is not being able to boost more than once

@jonny If he doesn't understand what the bit shift operations are doing, why does he program! Let him go back to school.

@jonny well i can believe they're bad at programming tbh if they're learning off chatgpt

@jonny I've used LLM a handful of times when I've been absolutely, positively stuck on an issue, and it's helped a bit. But the thing is, you *have* to go in knowing there's a very, very high chance it's going to be either slightly or completely wrong. Therefore you should only use it to help reframe the issue in your mind, even if once in a while it does happen across a reasonably correct answer.

It's basically akin rubberducking with an overly confident friend.

@jonny Even people who should know better underestimate just how bad the results of those systems are: https://dl.acm.org/doi/10.1145/3558489.3559072

Quote: "Our results suggest that GitHub Copilot was able to generate valid code with a 91.5% success rate. In terms of code correctness, out of 164 problems, 47 (28.7%) were correctly, while 84 (51.2%) were partially correctly, and 33 (20.1%) were incorrectly generated."

More than 70% of the code generated is at least partly flawed.

@jonny now, sure people rely on it so clearly there is an issue, but there's a disclaimer as you open it: "Chatgpt may give inaccurate information. It's not intended to give advice". Then under the message box: "Chatgpt can make mistakes. Verify important information"

What else should they do, in your opinion, to make users understand that they shouldn't rely on it this way?

@jonny When programming anything interesting I always seem to spend much more time testing than coding. If this gets too repetitive, I have to figure out how to automate some of the tests. Eventually the whole thing needs rewriting due to changes in language support e.g. the change from Python 2 to Python 3. ChatGPT seems more suited to kicking out the bottom rungs of the learning ladder in this respect than a genuine productivity improvement.

@jonny It's sad really. LLMs do have a lot of potential. For example, the basic mechanism was just used as part of a multi-pronged effort to figure out how some bacteria are able to go into a sort of hibernation and survive extreme conditions for decades or longer. This may come in handy for finding a way to stop harmful ones. They can be useful when used right. It's really sad watching them be used wrongly instead. ChatGPT and similar just aren't AI and need to stop pretending to be.

@jonny I'm reminded of a principle I heard in the field of technical translation: you shouldn't try to translate something you couldn't have written (I can't remember the exact phrasing).

Perhaps the corollary is that there's no point in asking an LLM a question that you couldn't answer yourself. Which puts a pretty good upper bound on the usefulness of LLMs, tbh.

@jonny Playing with ChatGPT in the past, it even began to make up words that didn't exist. I never told it to do so.

@jonny LLMs can really help when you are trying to get up to speed with a particular interface, so long as you're aware they lie to you. If you are able to identify the lies and tell them about them, then sometimes they might even generate usable code. The problem comes as soon as you start trying to do something a little bit unusual, and the LLM steps up the pace from "bullshit" to "delusional fantasy".

Absolutely agree with the problem being the dishonest presentation of results.

@jonny I want to raise this point in particular: "Empowerment is learning how to figure things out, how to make tools for yourself and how to debug problems."

https://neuromatch.social/@jonny/111326984855400462

@jonny I think I'm not using it much as results are rarely useful straight away. There's extra work required to verify the results, or work out what's missing from the solution offered... so I might as well have not bothered wasting my time and the CPU cycles.

@jonny it's incredible to me that Certified Smart People I work with (you know, PhDs, experience in research, scientific publications, distinguished academic careers) have utterly bought into LLMs and seem to have no understanding of what they actually do.

@jonny

Thanks for your post and a special thank for calling out the gatekeeping (in the edit).

@jonny I hate these LLMs, and I hate the in-industry hypemonkeys that lend their clout to the absurd misconception that LLMs are any kind of intelligence, or that they are actually doing anythign else than autocorrect applied at scale.

@jonny My evaluation has ultimately been that the fundamental problem with these LLMs, at least in terms of the output the give, is that they are designed to give a satisfying answer to whatever is posed to them, even if they can't. So rather than say "I can't answer that" it will instead just invent something that sounds good. Because it may not know the answer, but it damn well knows what an answer *looks like*, and appearing to answer is preferable to giving a disappointing result.