@cstross
how do we explain to folks that 52% wrong doesn't mean 48% correct?

@cstross only 52%?

@cstross 52% sounds low.

@cstross Still 2% worse of my previous answer machine

@cstross@wandering.shop Color me soooooooooo shocked.

My last (as in both most recent, and final) tech employer went really whole hog into using ChatGPT to write code and DB queries.

@cstross

I worked for over 30 years as a geophysicist in the oil business interpreting seismic data. Companies always tried to sell us artificial intelligence software to allow the computer do the interpretation for us going back as far as the 90s. Up until my retirement about 7 years ago I found that geophysicists would spend more time correcting the "interpretation" the machine did than it would have taken to do the interpretation themselves. I do not trust this "artificial" intelligence.

@cstross Every piece of sample code ever provided to me by a project manager or non-technical co-worker that used functions that didn’t exist, had obvious syntax errors, or did things I considered insane…. Turned out to be from an LLM. Their attempt to show me how easy it was to write the necessary code turned into a lesson in why programmers should just be left alone to do the necessary thing.

@cstross I find it surprisingly good for rubber ducking or just getting search terms to find elsewhere, I really want to see these tools be a bit more open, is there anything that is good at that kind of problem that is properly open?

@cstross
Yeah, but that 48%....

@cstross

I am thinking about all the people I see on Hacker News raving about how EFFICIENT talking to AIs is making them and giggling.

I am also hoping I never have to deal with any system they were involved in building...

@cstross

Honestly, isn't it surprising it's that low?

I'd've thought it would bork a lot more questions than that.

@cstross so basically at best it is coin flip. No, thank you.

@cstross So like a real programmer then.

@cstross

So flipping a coin is more accurate? Awesome.

@cstross ChatGPT 5 will reduce that to 49%. Then everything will be fine.

@cstross If you ask it to write a program in #Rustlang it fails 90% of tasks :)

@cstross @lisamelton That’s CEO-level wrong, though.

@cstross that's the fun part. companies are going to go deep on ai coding only to absolutely fuck themselves over. the code ai can generate is often remedial code that i would never run on a production server. i've never seen it write code that isn't shit. companies think engineers are expensive, they're about to fuck around and find out.

@cstross In my (admittedly very limited) experience, I'm not even getting internally consistent answers, such that variables change name and other errors.

But the value, such that it is, has been in getting suggestions for new ways of solving something, which I then can do something with using actual reference documentation.

@cstross I'm legit shocked it's that low.

@cstross ...one way to be rid of AI garbage...
https://saturation.social/@clive/112493382329631597

@cstross …which is why I don’t use it for that purpose.

Of course, I’ve asked ChatGPT for nonessential programming, like making a Quine in 6502 assembly, and it succeeded in doing that. But for normal work, I don’t dare touch it because if it makes a mistake, then I will have no idea how to fix it.

@cstross i'm surprised it is only 52% and not much higher.

@cstross Hope healthcare, transportation aren't using AI for it because that's a whole lot of risk. They'd better lawyer up for the bugs that cause accidents and death.