Email or username:

Password:

Forgot your password?
Top-level
stephen ryner jr. 🦉

@danluu looks like a Ford Mach E

Why are car manufacturers so bad at software? Why are almost all hardware makers terrible at software?

81 comments
stephen ryner jr. 🦉

@danluu are hardware makers also terrible at hardware but it’s just harder for the average person to tell 🤔

Jacob Christian Munch-Andersen

@nuthatch @danluu I think it is about equally easy/difficult for the average person to tell, you just happen to be a software person.

Passenger

@nuthatch @danluu

A lot of software makers (especially, in my experience, for the enterprise market) are also terrible at software. We as a species are bad at software.

Passenger

@nuthatch @danluu

(The ghost of Edsger W. Dijkstra is standing right behind me when I say that, isn't he? He always finds a way to loom up in times like this.)

Dr.Nick

@passenger @nuthatch @danluu "simplicity is a prerequisite for reliability"

Scott Michaud

@passenger @nuthatch @danluu Yeah I was going to say that my "I need a walk" moments with third-party code doesn't seem to correlate with whether or not they're a hardware company.

Passenger

@scottmichaud @nuthatch @danluu

In fairness, the worst software I've ever used, without exception, has been internal-only stuff.

Iridium Zeppelin

@nuthatch @danluu

1. Car manufacturers do a pretty good job with their software, most of the time. None of my family's vehicles have had any major software problems. Everything just works as it is supposed to.

2. Hardware manufacturers are largely not terrible at software, you probably notice it more when they are.

3. Hardware is difficult to make. Most hardware manufacturers are actually pretty good at it.

Óscar Morales Vivó

@nuthatch @danluu well Ford sure isn’t famous for the reliability of their hardware products either.

On the other hand even Toyota can’t software their way out of a paper bag 🤷🏽‍♂️

antipode77

@nuthatch @danluu

I am inclined to think it is not their core competence.

IT systems bolted onto all kinds of subsystems seems to be the rule.

This problem has a known solution, but it was not implemented.

Ken Tindell

@nuthatch @danluu They’re not all terrible at software. A car is not a phone on wheels. A car is not a Windows PC on wheels. A car is not a web server on wheels. The problem domain is a lot more difficult than anything you’ve encountered in mainstream computing. For a start, if it goes wrong then people can be injured or die.

Avi

@kentindell @nuthatch @danluu But this is not an alien concept, right? Not all computing is end-user tablets. We use computing to fly to space, run trains, medical equipment. Do we see screens like this a lot in the operating room?

Ken Tindell

@ashmueli @nuthatch @danluu When you have hundreds and hundreds of millions of operating rooms left outside in the rain, let me know.

Avi

@kentindell @nuthatch @danluu Well, engineers are there to solve the problem based on its environment, not to let cars fail on software upload because it, maybe, rained. What you are REALLY saying is that the automotive software industry is choosing tradeoffs that will leave some people stranded.

Ken Tindell

@ashmueli @nuthatch @danluu You’ve seen one photograph of a screen yet you think you know the tradeoff between reliability and safety. Amazing.

Avi

@kentindell @nuthatch @danluu Are you saying there are no tradeoffs in engineering? Are you saying there are no tradeoffs in automotive engineering? Are you saying this screen, displaying an error message designed to fit the situation, is somehow not subject to tradeoffs in engineering?

Kneworldodor

@kentindell @ashmueli @nuthatch @danluu I would hope a problem with actual safety while moving is the root of this. Being stranded could be a safety issue. Vehicles have been computerized for decades and have always had a limp home backup for this reason. I would regard this as a design problem. My career covered analog to bus systems.

Avi

@kneworldodor @kentindell @nuthatch @danluu Exactly. It’s a design problem and, unlike, say, space flight, car designers have the option to allow for grounding the car. The question now is, what are the tradeoffs behind this particular case.

Kneworldodor

@ashmueli @kentindell @nuthatch @danluu the specifics are important and I might agree with a shutdown if I knew them. I see no reason that most systems can't shut down or resort to an open loop program 99% of the time though. If it's related to protecting from electrical fields during reflash or network access so it doesn't get bricked during update then it's a stupid mistake.

still can't work out who i am

@kentindell @ashmueli @nuthatch @danluu nd of course this could easily be a hardware fail and AFAIK cars don;t use secondary redundant hardware and evne if they do and a primary hardware fails, you'd still ground the car when stationary

Ken Tindell

@peterainbow @ashmueli @nuthatch @danluu There is often redundancy: ECC memory, lock-step CPU cores, multiple bus paths. The silicon has fault injection for built-in tests at start to check this.

still can't work out who i am

@kentindell @ashmueli @nuthatch @danluu that's not the case on the older gneration of car computers, but i'm guessing things have moved on, any pointers to where i can learn more ( jst for learning sake ), finding the search engine world pretty much borked again these days lol

Ken Tindell

@peterainbow @ashmueli @nuthatch @danluu Yes, search is grey goo now. But I can point you at a blog post about updating firmware in cars. kentindell.github.io/2023/04/1

still can't work out who i am replied to Ken

@kentindell @ashmueli @nuthatch @danluu oh i've done some of that from pi's and webcams to printers and even connecting to FIAT ECUs, but 2000ish models did not have ECU redundancy at all, just wondering if that's changed. obviously they probabky use the backup swtchover flash system, but if something fails (hardware) then that's it and as a firmware dev there are diminishing returns in showing the end user low level error info, better to go for the don;t panic general error screen as seen above

Ken Tindell

@Ooze @nuthatch @danluu It’s called safety critical software. You don’t develop it like normal software.

Ooze 𓁟

@kentindell @nuthatch @danluu Thank you for totally missing my point, completely not looking at the resource I provided in support of my point, and just restating your point.

Ken Tindell

@Ooze @nuthatch @danluu I’m very familiar with safety critical systems development. I didn’t think your naive take on it was worth engaging with.

Ooze 𓁟

@kentindell @nuthatch @danluu Ah the mainsplaining double, stubborn and rude.

Lee Fife

@Ooze @kentindell @nuthatch @danluu Immediately followed by the drive-by block... Don't have to worry about seeing his particular mix of hostility and self centeredness again

Oli

@kentindell @nuthatch @danluu well i guess a car that wont move is probably safe...but its hardly a graceful failure if the fall back is just to break everything

Its also a piss-poor user experience

Ken Tindell

@OliverNoble @nuthatch @danluu That’s how it’s supposed to work. Safety comes first, then comes reliability.

John Timaeus

@kentindell @OliverNoble @nuthatch @danluu

Part of safety is not pushing an update that breaks the system in the first place.

Plus when an update is performed >>NOT PUSHED<<, the system verifies that it received the update correctly, checks signatures, then runs the software.

If there is an error, it reverts to previous state >>WITHOUT USER INTERVENTION<<

A vehicle which fails to move because the Mfg pushed software is unsafe, not unreliable.

John Timaeus

@kentindell @OliverNoble @nuthatch @danluu

Not really. A quick dive into forums shows that Ford has a fully embuggerred software update system, either failing to install critical updates, or bricking vehicles. There seems to be a trend of incompetent/nonexistent dealer support and a lack of concern at the Mfg level.

I hadn't really thought about buying a Ford, and I've been hesitant to buy anything with push OTA update. Ten minutes of reading forums convinced me to avoid both.

Ken Tindell

@johntimaeus @OliverNoble @nuthatch @danluu If you think it’s easy then you don’t understand the problem. Like why cars need to be parked on level ground to be updated. Or why it entails new microcontroller silicon designs.

John Timaeus

@kentindell @OliverNoble @nuthatch @danluu

I understand the complexity. I don't understand _this_ particular problem because Ford hasn't paid me to do root cause and critical chain analysis. But I understand the domain of life safety & critical infra systems well enough to know that this is fail.

I never said it was easy, and would appreciate that you not put words in my mouth.

What I do say is; If you can't do it right you shouldn't be doing it and selling it to the public.

Bricking a $50-100k device by pushing a bad update is unacceptable. Saying that bricking is a safety feature is a cop out. Failing to address the problem is criminal.

@kentindell @OliverNoble @nuthatch @danluu

I understand the complexity. I don't understand _this_ particular problem because Ford hasn't paid me to do root cause and critical chain analysis. But I understand the domain of life safety & critical infra systems well enough to know that this is fail.

I never said it was easy, and would appreciate that you not put words in my mouth.

Ken Tindell

@johntimaeus @OliverNoble @nuthatch @danluu It’s not bricked: it needs to be connected to an authorized diagnostic tool. But clearly you don’t want to learn and I don’t want to hear uninformed opinion, so on that basis I’m muting this.

jesterchen42

@johntimaeus Yeah... I don't want OTA push, I don't want internet connectivity, I don't want ANY data sent to the manufacturer without explicit consent, I don't want a car that requires security updates, I don't want tons of sensors and helpers (except perhaps a camera in the back), I don't want "smart" or keyless, I don't want touchscreens everywhere.

But I do want an EV. Will there be something like this in the next decade (or 25 years)? 😔

Time for a better bike.

masukomi

@OliverNoble @kentindell @nuthatch @danluu it depends on what the car was doing when it stopped being drivable.

Its been shown multiple times that you can maliciously alter the software of a car that’s actively driving (multiple manufacturers), which means a lack of protections against changes at “bad” times

Ken Tindell

@masukomi @OliverNoble @nuthatch @danluu There are security threats to cars while driving eh? 🤔 I think you could be right.

Tim Ward ⭐🇪🇺🔶 #FBPE

@kentindell @nuthatch @danluu "if it goes wrong then people can be injured or die"

Only if you try hard to make it that bad. The aircraft I've flown, all the computers can fail, all the screens can go blank, and I'd still be able to land the thing safely. There's no obvious reason why cars should be any worse.

Ken Tindell

@TimWardCam @nuthatch @danluu And I bet if your pre-flight checks fail you don’t take off.

Tim Ward ⭐🇪🇺🔶 #FBPE

@kentindell @nuthatch @danluu Yup. I've rejected an aircraft more than once.

Once, for example, because I spotted a tiny dent in the tailplane that nobody else had seen. Presumably it had been bashed by something in the hangar. Had this been a hard enough bash to break or weaken something structural inside? - I didn't wish to find out, that's what engineers are for, not customers. The next time I rented that aircraft the dent was no longer there.

uis

@kentindell @nuthatch @danluu there is so much wrong with these statements.
1. PCs from 10 years ago compared to car on-board computer from this year are bleeding edge technology.
2. Why update of media computer should brick entire car? It's like if burned out light bulb brought down entire power grid.

uis

3. If problem domain so more difficult, then making updates not break system should be a breeze. Even if update is broken for some reason, entire High Avaliability world knows to boot from backup for more than 20 years.
4. It already went wrong. And given how wrong it went, I don't trust that people will not "be injured or die" as result of such incompetence.

Ken Tindell

@uis @nuthatch @danluu It’s not updating the “media computer”. Try to understand how cars work before making these assertions.

uis

Great, did it brick engine controller? So, do you still belive such incompetence will not result in injuries or deaths?

AT-AT Assault :verifiedtrans:

@nuthatch @danluu

Why is a two tone, DEADLY machine forced to receive unimportant updates that could render it inoperable?

maybenot

@atatassault
hypothetical:
a manufacturing defect is identified that makes the vehicles already in use dangerous, immediate fix is pushed, units that fail to complete the fix are preemptively immobilized, show this screen, wait for technican

AT-AT Assault :verifiedtrans:

@maybenot

Why is a mechanical machine engineered in such a way that software can make it dangerous?

maybenot

@atatassault it's cheaper this way?
an actuator and a microcontroller will cost less than a mechanical governor / feedback control mechanism (and be more reliable / require less maintenance)

also it allows for more advanced functionality, software controlled hardware
like more efficient engine control / advanced-driver-assists / etc
an extreme example would be tesla's infamous "full self driving"* as a hypothetical hardware fully controlled by software
*does not actually fully self drive

maybenot

@atatassault also also you could theoretically have a software hack/fix covering/patching a hardware flaw
(tell the computer to avoid the last 5% range on some actuator or something)

uis

They aren't bad at software, they are good at making an excuses for getting more your money.

Go Up