"Unfortunately, a recent software update was not successful. Your vehicle cannot be driven.
Please call customer support"
"Unfortunately, a recent software update was not successful. Your vehicle cannot be driven. Please call customer support" 225 comments
@eliasp @danluu it is likely that the design of the system makes it so that simply slapping on a second system partition for rollbacks is not enough. In fact it may be already present there. The car runs a network of computers running a variety of systems. The display in the photo might be a QNX machine running an Android VM and these two systems need to coordinate their updates. @danluu looks like a Ford Mach E Why are car manufacturers so bad at software? Why are almost all hardware makers terrible at software? @danluu are hardware makers also terrible at hardware but it’s just harder for the average person to tell 🤔 A lot of software makers (especially, in my experience, for the enterprise market) are also terrible at software. We as a species are bad at software. (The ghost of Edsger W. Dijkstra is standing right behind me when I say that, isn't he? He always finds a way to loom up in times like this.) @passenger @nuthatch @danluu Yeah I was going to say that my "I need a walk" moments with third-party code doesn't seem to correlate with whether or not they're a hardware company. @scottmichaud @nuthatch @danluu In fairness, the worst software I've ever used, without exception, has been internal-only stuff. 1. Car manufacturers do a pretty good job with their software, most of the time. None of my family's vehicles have had any major software problems. Everything just works as it is supposed to. 2. Hardware manufacturers are largely not terrible at software, you probably notice it more when they are. 3. Hardware is difficult to make. Most hardware manufacturers are actually pretty good at it. I am inclined to think it is not their core competence. IT systems bolted onto all kinds of subsystems seems to be the rule. This problem has a known solution, but it was not implemented. @nuthatch @danluu They’re not all terrible at software. A car is not a phone on wheels. A car is not a Windows PC on wheels. A car is not a web server on wheels. The problem domain is a lot more difficult than anything you’ve encountered in mainstream computing. For a start, if it goes wrong then people can be injured or die. @kentindell @nuthatch @danluu But this is not an alien concept, right? Not all computing is end-user tablets. We use computing to fly to space, run trains, medical equipment. Do we see screens like this a lot in the operating room? @kentindell @nuthatch @danluu Well, engineers are there to solve the problem based on its environment, not to let cars fail on software upload because it, maybe, rained. What you are REALLY saying is that the automotive software industry is choosing tradeoffs that will leave some people stranded. @kentindell @nuthatch @danluu Are you saying there are no tradeoffs in engineering? Are you saying there are no tradeoffs in automotive engineering? Are you saying this screen, displaying an error message designed to fit the situation, is somehow not subject to tradeoffs in engineering? @kentindell @ashmueli @nuthatch @danluu I would hope a problem with actual safety while moving is the root of this. Being stranded could be a safety issue. Vehicles have been computerized for decades and have always had a limp home backup for this reason. I would regard this as a design problem. My career covered analog to bus systems. @kneworldodor @kentindell @nuthatch @danluu Exactly. It’s a design problem and, unlike, say, space flight, car designers have the option to allow for grounding the car. The question now is, what are the tradeoffs behind this particular case. @kentindell @ashmueli @nuthatch @danluu nd of course this could easily be a hardware fail and AFAIK cars don;t use secondary redundant hardware and evne if they do and a primary hardware fails, you'd still ground the car when stationary @kentindell @nuthatch @danluu People get killed by software all the time. See https://archive.org/details/lca2019-Im_sorry_Dave_I_cant_do_that_Ethics_in_Software_Development @kentindell @nuthatch @danluu Thank you for totally missing my point, completely not looking at the resource I provided in support of my point, and just restating your point. @Ooze @kentindell @nuthatch @danluu Immediately followed by the drive-by block... Don't have to worry about seeing his particular mix of hostility and self centeredness again @kentindell @nuthatch @danluu well i guess a car that wont move is probably safe...but its hardly a graceful failure if the fall back is just to break everything Its also a piss-poor user experience @OliverNoble @nuthatch @danluu That’s how it’s supposed to work. Safety comes first, then comes reliability. @kentindell @OliverNoble @nuthatch @danluu Part of safety is not pushing an update that breaks the system in the first place. Plus when an update is performed >>NOT PUSHED<<, the system verifies that it received the update correctly, checks signatures, then runs the software. If there is an error, it reverts to previous state >>WITHOUT USER INTERVENTION<< A vehicle which fails to move because the Mfg pushed software is unsafe, not unreliable. @kentindell @nuthatch @danluu "if it goes wrong then people can be injured or die" Only if you try hard to make it that bad. The aircraft I've flown, all the computers can fail, all the screens can go blank, and I'd still be able to land the thing safely. There's no obvious reason why cars should be any worse. @TimWardCam @nuthatch @danluu And I bet if your pre-flight checks fail you don’t take off. @kentindell @nuthatch @danluu Yup. I've rejected an aircraft more than once. Once, for example, because I spotted a tiny dent in the tailplane that nobody else had seen. Presumably it had been bashed by something in the hangar. Had this been a hard enough bash to break or weaken something structural inside? - I didn't wish to find out, that's what engineers are for, not customers. The next time I rented that aircraft the dent was no longer there. Why is a two tone, DEADLY machine forced to receive unimportant updates that could render it inoperable? @danluu we need the fedora people to step it up and make an automobile OS based on rpm-ostree. @asmallteapot @danluu @kyleve The car looks electric so it’s a nightmare charge actually. @danluu Very very early in the history of TiVo, they put out an update and bricked the boxes. Fortunately there were not a lot out there (hundreds?) and all in the Bay Area. So they ended up going to every house and fixing them. Needless to say, that never happened again. Also, that was decades ago. There’s really no excuse now. @danluu "Press and hold the break pedal and accelerator pedal all the way down." 🤣 @danluu This is *excessive* computerization. A car from 50 years ago was *literally* more functional than this in the most basic of aspects. @1dalm @shimrrashai @danluu Indeed. Cars from 50 years ago were terrible in terms of reliability. My mum and dad regularly tell stories about her old cars and how they would fail if you just looked at them the wrong way. EDIT: For the victim-blamers out there, they have always had their vehicles regularly maintained by professional mechanics. @danluu Did it prompt for the update? Or just decide to do it without asking? I do not want any car with a built-in cellular transceiver. If the car has multiple computers, they should boot over the CAN bus from a central source. That would ensure you cannot brick the whole car, as only that one source has to have A and B partitions. @danluu Friends hybrid bricked by itself while parked last week “Catastrophic brake failure. Call a tow truck.” Dealer said we’ll have time next week. @stevewfolds @danluu Brakes fail. It happens. I regularly see and hear people operating cars with failing brakes here - if the software can tell if the brakes have failed, the car *should* be rendered inoperable, or perhaps only partially operable. It's not just dangerous for them, it's dangerous for everyone else on the road. @SnepperStepper @danluu This is demonstrably false. ABS and TCS save huge numbers of lives every year in adverse conditions. Same with planes. @bananarama @danluu ABS and TCS are not an invention of the 21st century. They are also not needed if you actually know what you're doing. I'm horrified how little the people piloting the modern death machines actually know about them or what they're supposed to be doing with them. Ignorance is a weakness, and they have it in spades. @bananarama that's what i'm talking about. Not my fault you don't understand the conversation and instead want to whine about it. @danluu Open & close the passenger front door twice whilst keeping the driver's door open & pressing the accelerator. At the very least, you should be able to poke a paperclip into a tiny hole in the dashboard, and reset to factory software from ROM. Not off-topic: this is why we abandoned the whole concept of a "Kitchen Range" and filled in the same need (better and cheaper) with a modular cooking table of many smaller products that do NOT depend on chips. In particular, the big job (today) of an oven, can be done by a $129 Roaster Oven whose control system is one rheostat. @danluu @lisamelton did you call the number? They will help you out, let me know if you can’t reach someone, I can escalate it. @danluu (And before you answer "hardware engineer", remember that your HDD or SSD has a cache, and that's all controlled by software) @danluu Yeah sure, let's put computers in all cars for absolutely no fucking reason. What could go wrong? @danluu this is why I want the lowest tech car as possible. Give me physical keys and an AUX cord. No Bluetooth, I’ll just use speaker phone if I have to talk while driving. @danluu BSOD takes on a new meaning when travelling at speed & your vehicle update fails @danluu Rise of the Machines would last less than a second before it choked on a fatal error. @danluu Bricked Ford after failed software update.☹️😡 Your cue, rockstar software engineering management, to apply for Ford position? 😉 @danluu My mother's Ford not too long ago just decided not to take input from the steering wheel. Luckily she was not on the highway at the time. I work as a software developer and know way too much about what goes on behind the scenes to trust my life to the crap developed my most my peers. @danluu Yeah, we’re gonna need to rethink those flying cars. Who wants to get a failed upgrade 500 feet in the air? If I'll ever buy a car, it would be the kind where updatable software doesn't control anything critical. @danluu There is a reason why I pick "apply updates" when the car is parked nicely at home, and I have time to deal with it, should it fail... :P Not a Ford / Mustang, though... @danluu At least you got a nice error message. I got nothing on the big screen, was stuck in the garage, and it wouldn’t start at all. Just a terse shutdown message on the small odometer screen, with a red ring of death on the charger port. I had no idea how to get it into neutral. Thankfully, the tow truck operator did! I want to know when a simple dutch bike, electrified or not becomes viral. Electric everything is a fucking ponzi scheme. A bit of additional context for anyone reading the thread: this is definitely a Ferd thing, not an EV thing. |
@danluu 😬😬😬