Email or username:

Password:

Forgot your password?
Dan Luu

"Unfortunately, a recent software update was not successful. Your vehicle cannot be driven.

Please call customer support"

225 comments
Elias Probst

@danluu Atomic updates and rollbacks seem to be foreign concepts to the car industry.

bttk

@eliasp @danluu it is likely that the design of the system makes it so that simply slapping on a second system partition for rollbacks is not enough. In fact it may be already present there.

The car runs a network of computers running a variety of systems. The display in the photo might be a QNX machine running an Android VM and these two systems need to coordinate their updates.

#BrokenByDesign

p6

@danluu Computer mit Reifen ... Warum soll der besser sein als der Computer ohne Reifen... Den ohne kann ich neu installieren... Aber wie fixed man das hier?

Peter Bindels

@danluu That has to be a photoshop right? Or a joke?

stephen ryner jr. 🦉

@danluu looks like a Ford Mach E

Why are car manufacturers so bad at software? Why are almost all hardware makers terrible at software?

stephen ryner jr. 🦉

@danluu are hardware makers also terrible at hardware but it’s just harder for the average person to tell 🤔

Jacob Christian Munch-Andersen

@nuthatch @danluu I think it is about equally easy/difficult for the average person to tell, you just happen to be a software person.

Passenger

@nuthatch @danluu

A lot of software makers (especially, in my experience, for the enterprise market) are also terrible at software. We as a species are bad at software.

Passenger

@nuthatch @danluu

(The ghost of Edsger W. Dijkstra is standing right behind me when I say that, isn't he? He always finds a way to loom up in times like this.)

Scott Michaud

@passenger @nuthatch @danluu Yeah I was going to say that my "I need a walk" moments with third-party code doesn't seem to correlate with whether or not they're a hardware company.

Passenger

@scottmichaud @nuthatch @danluu

In fairness, the worst software I've ever used, without exception, has been internal-only stuff.

Iridium Zeppelin

@nuthatch @danluu

1. Car manufacturers do a pretty good job with their software, most of the time. None of my family's vehicles have had any major software problems. Everything just works as it is supposed to.

2. Hardware manufacturers are largely not terrible at software, you probably notice it more when they are.

3. Hardware is difficult to make. Most hardware manufacturers are actually pretty good at it.

antipode77

@nuthatch @danluu

I am inclined to think it is not their core competence.

IT systems bolted onto all kinds of subsystems seems to be the rule.

This problem has a known solution, but it was not implemented.

Ken Tindell

@nuthatch @danluu They’re not all terrible at software. A car is not a phone on wheels. A car is not a Windows PC on wheels. A car is not a web server on wheels. The problem domain is a lot more difficult than anything you’ve encountered in mainstream computing. For a start, if it goes wrong then people can be injured or die.

Avi

@kentindell @nuthatch @danluu But this is not an alien concept, right? Not all computing is end-user tablets. We use computing to fly to space, run trains, medical equipment. Do we see screens like this a lot in the operating room?

Ken Tindell

@ashmueli @nuthatch @danluu When you have hundreds and hundreds of millions of operating rooms left outside in the rain, let me know.

Avi

@kentindell @nuthatch @danluu Well, engineers are there to solve the problem based on its environment, not to let cars fail on software upload because it, maybe, rained. What you are REALLY saying is that the automotive software industry is choosing tradeoffs that will leave some people stranded.

Ken Tindell

@ashmueli @nuthatch @danluu You’ve seen one photograph of a screen yet you think you know the tradeoff between reliability and safety. Amazing.

Avi

@kentindell @nuthatch @danluu Are you saying there are no tradeoffs in engineering? Are you saying there are no tradeoffs in automotive engineering? Are you saying this screen, displaying an error message designed to fit the situation, is somehow not subject to tradeoffs in engineering?

Kneworldodor

@kentindell @ashmueli @nuthatch @danluu I would hope a problem with actual safety while moving is the root of this. Being stranded could be a safety issue. Vehicles have been computerized for decades and have always had a limp home backup for this reason. I would regard this as a design problem. My career covered analog to bus systems.

Avi

@kneworldodor @kentindell @nuthatch @danluu Exactly. It’s a design problem and, unlike, say, space flight, car designers have the option to allow for grounding the car. The question now is, what are the tradeoffs behind this particular case.

still can't work out who i am

@kentindell @ashmueli @nuthatch @danluu nd of course this could easily be a hardware fail and AFAIK cars don;t use secondary redundant hardware and evne if they do and a primary hardware fails, you'd still ground the car when stationary

Ken Tindell

@Ooze @nuthatch @danluu It’s called safety critical software. You don’t develop it like normal software.

Ooze 𓁟

@kentindell @nuthatch @danluu Thank you for totally missing my point, completely not looking at the resource I provided in support of my point, and just restating your point.

Ken Tindell

@Ooze @nuthatch @danluu I’m very familiar with safety critical systems development. I didn’t think your naive take on it was worth engaging with.

Ooze 𓁟

@kentindell @nuthatch @danluu Ah the mainsplaining double, stubborn and rude.

Lee Fife

@Ooze @kentindell @nuthatch @danluu Immediately followed by the drive-by block... Don't have to worry about seeing his particular mix of hostility and self centeredness again

Oli

@kentindell @nuthatch @danluu well i guess a car that wont move is probably safe...but its hardly a graceful failure if the fall back is just to break everything

Its also a piss-poor user experience

Ken Tindell

@OliverNoble @nuthatch @danluu That’s how it’s supposed to work. Safety comes first, then comes reliability.

John Timaeus

@kentindell @OliverNoble @nuthatch @danluu

Part of safety is not pushing an update that breaks the system in the first place.

Plus when an update is performed >>NOT PUSHED<<, the system verifies that it received the update correctly, checks signatures, then runs the software.

If there is an error, it reverts to previous state >>WITHOUT USER INTERVENTION<<

A vehicle which fails to move because the Mfg pushed software is unsafe, not unreliable.

Tim Ward ⭐🇪🇺🔶 #FBPE

@kentindell @nuthatch @danluu "if it goes wrong then people can be injured or die"

Only if you try hard to make it that bad. The aircraft I've flown, all the computers can fail, all the screens can go blank, and I'd still be able to land the thing safely. There's no obvious reason why cars should be any worse.

Ken Tindell

@TimWardCam @nuthatch @danluu And I bet if your pre-flight checks fail you don’t take off.

Tim Ward ⭐🇪🇺🔶 #FBPE

@kentindell @nuthatch @danluu Yup. I've rejected an aircraft more than once.

Once, for example, because I spotted a tiny dent in the tailplane that nobody else had seen. Presumably it had been bashed by something in the hangar. Had this been a hard enough bash to break or weaken something structural inside? - I didn't wish to find out, that's what engineers are for, not customers. The next time I rented that aircraft the dent was no longer there.

AT-AT Assault :verifiedtrans:

@nuthatch @danluu

Why is a two tone, DEADLY machine forced to receive unimportant updates that could render it inoperable?

Gabriel Pettier

@danluu the future is awesome.

(full, i mean full).

Pyperkub

@danluu We have bricked your car. Pray we don't brick it further!

mah:~ $ :nixos:

@danluu we need the fedora people to step it up and make an automobile OS based on rpm-ostree.

Joe Fabisevich :verified:

@asmallteapot @danluu @kyleve The car looks electric so it’s a nightmare charge actually.

Kee Hinckley

@danluu Very very early in the history of TiVo, they put out an update and bricked the boxes. Fortunately there were not a lot out there (hundreds?) and all in the Bay Area. So they ended up going to every house and fixing them.

Needless to say, that never happened again.

Also, that was decades ago. There’s really no excuse now.

Charles J Gervasi ⚡🛡️🥥

@danluu "Press and hold the break pedal and accelerator pedal all the way down." 🤣
It had me going for a second.

Rupert

@danluu This is the future tech bros want.

Shimrra Shai

@danluu This is *excessive* computerization. A car from 50 years ago was *literally* more functional than this in the most basic of aspects.

Iridium Zeppelin

@1dalm @shimrrashai @danluu Indeed. Cars from 50 years ago were terrible in terms of reliability. My mum and dad regularly tell stories about her old cars and how they would fail if you just looked at them the wrong way.

EDIT: For the victim-blamers out there, they have always had their vehicles regularly maintained by professional mechanics.

Inkican

@danluu Have you tried turning it off and turning it back on?

mike805

@danluu Did it prompt for the update? Or just decide to do it without asking?

I do not want any car with a built-in cellular transceiver.

If the car has multiple computers, they should boot over the CAN bus from a central source. That would ensure you cannot brick the whole car, as only that one source has to have A and B partitions.

Whiskers

@danluu Just think of the geopolitical possibilities of this out there in the future before buying your #EV

@stevewfolds

@danluu Friends hybrid bricked by itself while parked last week “Catastrophic brake failure. Call a tow truck.” Dealer said we’ll have time next week.

Iridium Zeppelin

@stevewfolds @danluu Brakes fail. It happens. I regularly see and hear people operating cars with failing brakes here - if the software can tell if the brakes have failed, the car *should* be rendered inoperable, or perhaps only partially operable. It's not just dangerous for them, it's dangerous for everyone else on the road.

n3wjack

@danluu Holy crap. I'll never run a car update again when I'm not at home now ffs.

Martin Vermeer FCD

@n3wjack @danluu I have also been conservative in that way. You never know.

DELETED

@danluu there was never a good reason for any of the techie shit they've put in cars. This was always going to happen when people started trusting computers more than themselves and it was never a good idea.

Iridium Zeppelin

@SnepperStepper @danluu This is demonstrably false. ABS and TCS save huge numbers of lives every year in adverse conditions. Same with planes.

DELETED

@bananarama @danluu ABS and TCS are not an invention of the 21st century. They are also not needed if you actually know what you're doing. I'm horrified how little the people piloting the modern death machines actually know about them or what they're supposed to be doing with them. Ignorance is a weakness, and they have it in spades.

Iridium Zeppelin

@SnepperStepper @danluu That's not what you said. Don't move the goalposts.

DELETED

@bananarama that's what i'm talking about. Not my fault you don't understand the conversation and instead want to whine about it.

OddOpinions5

@danluu
I assume this is so well known that I'm just advertising my Noob Nerd status by posting it

snopes.com/fact-check/car-balk

Carolyn

@danluu Honestly, anti-lock brakes are as much smart as I want in a car.

Good luck.

DELETED

@danluu amazing how budget motherboards have had a way to safely restore a working firmware image in case of a failed update for years now, but a car costing tens of thousands of dollars can't possibly figure this out

Iridium Zeppelin

@rail @danluu It's not restoring a single piece of firmware. It's restoring *hundreds* of pieces of inter-operable firmware. Not to mention the difficulty of managing the hypervisor and it's guests.

Cars are not PCs.

Peter Relph

@danluu

Open & close the passenger front door twice whilst keeping the driver's door open & pressing the accelerator.

Kenner

@danluu hey, we messed up your car, good look doing everything you have to do in our city that is only defined to be done by car

w4tsn ~> :idle:

@danluu "I'm sorry dave. I'm afraid I can't do that"

Comrade Weez

@danluu That's disappointing.

My 1987 Hilux is still on Squirrel v1.0.

Monina6969

@danluu
Chain....weakest link.
In my experience once the car gets delivered to the car yard we enter the stage where no matter what happens absolutely no one there will be able to:
1. Fix it
2. Fix it.
Having an upgrade that fails and gives an answer like this is unacceptable. Having an upgrade that depends on a programmer being available is not a good business result.

More testing needed. (Repeat as necessary)

@danluu
Chain....weakest link.
In my experience once the car gets delivered to the car yard we enter the stage where no matter what happens absolutely no one there will be able to:
1. Fix it
2. Fix it.
Having an upgrade that fails and gives an answer like this is unacceptable. Having an upgrade that depends on a programmer being available is not a good business result.

Roy Brander

@danluu

At the very least, you should be able to poke a paperclip into a tiny hole in the dashboard, and reset to factory software from ROM.

Not off-topic: this is why we abandoned the whole concept of a "Kitchen Range" and filled in the same need (better and cheaper) with a modular cooking table of many smaller products that do NOT depend on chips.

In particular, the big job (today) of an oven, can be done by a $129 Roaster Oven whose control system is one rheostat.

brander.ca/range

monorail times

@danluu @lisamelton did you call the number? They will help you out, let me know if you can’t reach someone, I can escalate it.

DELETED

@danluu You should install Linux on it… 😉 /j

Nick Krichevsky

@danluu this can't be real... Can it? The future sucks

Raleigh Straight

@danluu @RobSF No rolling the update back?! You have to be kidding me. *puts Ford next to GM in the do-not-buy list*

Crispy

@danluu Rich people problems 😂😂😂😂😂😂

Peter Ludemann

@danluu
Which is more dangerous: a software engineer with a soldering iron or a hardware engineer writing code?

(And before you answer "hardware engineer", remember that your HDD or SSD has a cache, and that's all controlled by software)

Gothmog

@danluu The best part is that not only does your car sometimes just fail because it can't update software, but it's also spying on your sexual habits the rest of the time! Progress!

Felix Urbasik

@danluu Yeah sure, let's put computers in all cars for absolutely no fucking reason. What could go wrong?

Cass M 👋🏽🍁
@danluu So happy our vehicle gets updated at the shop via a physical connection.
Gavin Lux Enjoyer

@danluu this is why I want the lowest tech car as possible. Give me physical keys and an AUX cord. No Bluetooth, I’ll just use speaker phone if I have to talk while driving.

Dick Telder

@danluu
Decent handling of failure. Things do go wrong sometimes.

StaringAtClouds

@danluu BSOD takes on a new meaning when travelling at speed & your vehicle update fails

MegatronicThronBanks

@danluu Rise of the Machines would last less than a second before it choked on a fatal error.

tmyklebu

@danluu What's jarring to me is the amount of code that *is* working properly in order to tell you that the car can't do its job. At a minimum, the display is working, and it formatted text in a variable-width font, centred on the screen, in several different sizes.

Happy Thanksgiving! 🦃

@danluu really looking forward to my next automobile upgrade.

Facundo Olano

@danluu who would have thought that internet connected cars were a bad idea

Doug Grinbergs

@danluu Bricked Ford after failed software update.☹️😡 Your cue, rockstar software engineering management, to apply for Ford position? 😉

crazyeddie

@danluu My mother's Ford not too long ago just decided not to take input from the steering wheel. Luckily she was not on the highway at the time.

I work as a software developer and know way too much about what goes on behind the scenes to trust my life to the crap developed my most my peers.

DELETED

@danluu We actually have a serious competitor against Windows update for the most annoying update system it seem 😅

Bl4ck.V3n0m

@danluu

It must be a software made by Microsoft.

Heath Borders

@danluu oof. Is that a Mach-E or an F150 Lightning?

Arnan

@danluu Oh the times when cars just worked and didn’t run on software… 😩 What a ridiculous world we live in…

Pablo Rrrrrrr

@danluu Yeah, we’re gonna need to rethink those flying cars. Who wants to get a failed upgrade 500 feet in the air?

gudenau

@danluu Wait a second, those instructions aren't even specific to that car! This is so gross.

Bernard Sheppard

@danluu So much wrong with this:

1. The detailed instructions for the tow truck operator imply that there was sufficient knowledge that this was likely and that a choice between shipping the car with the broken update path and a pretty failed update screen, and a robust failed-update rollback process was available, and the choice (possibly forced by an announced launch date) was made to ship with the nice failure screen.

2. The update was pushed rather than a notification to schedule users to drive past a dealer to update via the same mechanism that will be used to fix the problem, avoiding the tow.

3. That someone signed off on this.

4. That maybe someone didn't sign off on it because it was thought unnecessary to escalate.

5. That whoever designed the hardware either chose not to have a robust OTA flashing process, or did, and whoever designed the firmware chose not to use it.

6. Whoever chose the hardware, or the firmware decided that robust updates were not required, but updates were required.

7. That something like updated navigation, wireless carplay, or more accurate DTE was considered more important than a percentage of users being able to drive their vehicle.

It has been nearly 30 years since I worked on embedded automotive systems (specifically ABS), so they've moved on a bit since then 🤣 but back when I did, every car that came off the line was tested (on a rolling dyno) to check if the ABS and all sensors were working, and all of the on-board computers were correctly interpreting the ABS signals. This included inducing a simulated skid (by de-clutching the rollers when up to speed and braking one of the rollers) to check that the stability control would attempt to correct the skid (by confirming that the ABS computer would signal to actuate the brakes on the opposite wheel). If not, that vehicle didn't leave the production line. Bad flash or bad wiring or incorrectly fitted sensors meant the system wasn't safe, and the car wasn't saleable.

@danluu So much wrong with this:

1. The detailed instructions for the tow truck operator imply that there was sufficient knowledge that this was likely and that a choice between shipping the car with the broken update path and a pretty failed update screen, and a robust failed-update rollback process was available, and the choice (possibly forced by an announced launch date) was made to ship with the nice failure screen.

Григорий Клюшников

If I'll ever buy a car, it would be the kind where updatable software doesn't control anything critical.

Lett Osprey

@danluu There is a reason why I pick "apply updates" when the car is parked nicely at home, and I have time to deal with it, should it fail... :P

Not a Ford / Mustang, though...

Atomic Fox

@danluu

Any vehicle which can undergo this event should be scrapped immediately.

David G. Smith

@danluu At least you got a nice error message. I got nothing on the big screen, was stuck in the garage, and it wouldn’t start at all. Just a terse shutdown message on the small odometer screen, with a red ring of death on the charger port. I had no idea how to get it into neutral. Thankfully, the tow truck operator did!

EVHaste

@danluu why is the view of the front windshield an isle in a Target but the reflection behind it a house

Scooter 🇺🇦 🇳🇿

@danluu imagine Just hope the guy who wrote the code doesn't work for Boeing now.

GhostOnTheHalfShell

@danluu

I want to know when a simple dutch bike, electrified or not becomes viral.

Electric everything is a fucking ponzi scheme.

Waseem

@danluu imagine having to go somewhere urgently and you are in this situation.
You've paid so much for the vehicle but you can't trust it because of this.

DELETED

@danluu I expect this garbage from Tesla, not Ford. Damn

Mathias, a walf ❄️🐺 :pansexual_flag:

@danluu

A bit of additional context for anyone reading the thread: this is definitely a Ferd thing, not an EV thing.

Michael

@danluu If this is the future, I'll stick with bikes.

Edbro

@danluu hope you weren’t in a hurry to begin with😅

Go Up