Email or username:

Password:

Forgot your password?
Foone🏳️‍⚧️

good lord. I pulled a microSD card out of a Raspi inside an IoT product and it appears they had some developer use a raspi to develop/test some software, and then they just yanked the SD card out of that machine and duped it on to all of their deployed products.

it's got .bash_history of the development process! there's git checkouts of private repos! WHY WOULD YOU DO THIS?

219 comments
maximemelian, enby fatale. :neocat_kisser:

@foone because they most probably don't know better. :neocat_googly_woozy:

Foone🏳️‍⚧️

I've also been able to de-stealth a "stealth startup" on linked in.
because this has commits from different users, and I can just look up on linkedin what stealth-startup all those people work/worked at and then look at the name on the IoT box I'm holding

gaytabase

@foone i mean it's very bad from a security perspective, but it's very convenient from a hacking perspective.

Foone🏳️‍⚧️

also, you punks are writing python 2 code in 2021? come on, who does that?

I mean, I do all the time, but I'm a known retrocomputerist. I run Windows 95 and MS-DOS regularly. of course I'm using a wildly outdated programming language. I'm not making a product I sell to customers!

Foone🏳️‍⚧️

oh cool you can pull the GPS history of a truck from azure without any login, you just need to know the device ID.

Foone🏳️‍⚧️

this might be UPS trucks. I should probably not query any of these GPS histories

Foone🏳️‍⚧️

also they're spamming 9 lines to syslog every minute.

this is a microsd card in a raspi, guys! you are going to fry your fucking card by running out of write cycles. That's not a good idea in any raspi application, especially not an IoT one

Gerhard D.

@foone Please stop. It's too damn depressing. Or wait .... SHARE IT!

Foone🏳️‍⚧️

oh sweet jesus they logged into slack from this machine('s image)

I have their chrome profile, with history and cookies and shit!

Foone🏳️‍⚧️

@cadey nah. I didn't get this device legitimately, so I can't really report any security holes in it.

Arne Babenhauserheide replied to Foone🏳️‍⚧️

@foone that’s why we need protection for people who report this stuff.

You may want to talk to folks from @CCC to find the right process to get this off the road before it causes bodily harm.

@cadey

Foone🏳️‍⚧️

this is deeply embarrassing. I have lists of their duckduckgo and google searches for the programming problems they were having building this product.

no programmer should ever have that personal shame shared with the world. let alone included on every microSD card your company ships!

Foone🏳️‍⚧️

oh sweet jesus

they automatically scp up some logs to a server somewhere. Did they set up keys so that authorized devices could log in automatically without passwords?

NOPE THEY USED SSHPASS

Foone🏳️‍⚧️

I have a file here with multiple lines like:

sudo sshpass -p PASSWORDHERE scp /path/system/network.log USERNAME@IPADDRESS:/home/manufacturing/

Foone🏳️‍⚧️

well I'm putting this away so I don't accidentally hack them.

Polychrome :clockworkheart: replied to Foone🏳️‍⚧️
@foone this thread just slowly became worse and worse as I was reading it :blobcatsweat:​
Foone🏳️‍⚧️ replied to Foone🏳️‍⚧️

this is one of the many reasons I'm not a security researcher.

it's a target rich environment.

Foone🏳️‍⚧️ replied to Foone🏳️‍⚧️

Also I'm a reverse engineer. There's no reverse engineering here!
I unscrewed the box, pulled out the raspi, pulled the SD card out, put it in my laptop, and it automounted. I then looked at some files while making a disgusted face.

That's not reverse engineering! That's just lookin'

Dasha Sierra replied to Foone🏳️‍⚧️

@foone You know, I never get bored of your tech adventures. Stay awesome. 💜

Esmé Ciredutemps replied to Foone🏳️‍⚧️

@foone i don't really understand completely what is explained but i read the thread as an excellent investigation and thriller movue. Thank you.
🍿

Haelwenn /элвэн/ :triskell: replied to Foone🏳️‍⚧️
@foone Kind of thing that is so bad I kind of wonder if it's a really weird attempt at an honeypot.
#wafuposting enjoyer replied to Foone🏳️‍⚧️
@foone LOG IN TO THEIR EMAIL AND START SENDING TONS OF SPAM YOU PUSSY
Glyph

@foone this is truly amazing. it's very unfortunate that you can't just post it all because the individual engineers will get blamed, rather than their incandescently irresponsible management

propapanda :verified:

@foone it would have made very little difference if they shipped a key instead of a password, but these people clearly don't know what they are doing

Foone🏳️‍⚧️ replied to propapanda

@panda could have individual keys per device and revoke them as the devices leave service. as it is, they can't do that without changing the passwords on every device

propapanda :verified: replied to Foone🏳️‍⚧️

@foone they totally could have and instead of ssh they could have used mqtt with some basic acls which would have prevent reading other people's location, but also save the overhead of the ssh handshake

Asta [AMP]

@foone@digipres.club ohhhhh no

no no no no

no to all of this but no to this

Randagodron

@foone lol better not talk about a company who sells emmbedded circuits for satellites, a friend checked two recently, boards where dirty as ***, covered in flux residue, lots of solder microballs hanging around (meaning they used too much solder paste and didn't follow their supplier's recommendations on solder masks) and bad solders making the board not functional without rework. Top space-grade shit, everything is fine 😅
If we find products this bad on aerospace devices, I am not surprised that it's even worse in IoT ...

@foone lol better not talk about a company who sells emmbedded circuits for satellites, a friend checked two recently, boards where dirty as ***, covered in flux residue, lots of solder microballs hanging around (meaning they used too much solder paste and didn't follow their supplier's recommendations on solder masks) and bad solders making the board not functional without rework. Top space-grade shit, everything is fine 😅
If we find products this bad on aerospace devices, I am not surprised that...

Foone🏳️‍⚧️

@randagodron they've got the excuse that no one will be able to tell, because their shit will be hiding in orbit

t3rminus

@foone holy dang that sounds… so bad. Just really, really bad. And Python 2??

Damn. Whatever you’re looking at I pray I don’t own one…

Sylvhem

@foone Hate it when my employer share my Slack session with hundred of customers.

embix

@foone "leave it so and mark the 'planned obsolescence' checkbox"

Phyxis

@foone Sometimes you have an investment in that codebase, and escape costs haven't crossed viable yet.

truh

@foone had a coworker rewrite my python3 app in python2 so it runs on google app engine, in 2020 or so.

Had to port it back to python3 later of course.

o76923

@foone@digipres.club
Southwest Airlines avoided the Crowdstrike outage today because their systems still run on Windows 3.1. Clearly these devs had the same idea and use python 2.x for security reasons since nobody is trying to find new exploits in it.

Jennifer 🏳️‍⚧️

@foone 😬

Utterly bad sys images check. Though, honestly, it doesn't surprise me that much now, seeing the number of times when we receive in production tools and deployment images that simply are very lacking of any field testing, and I work for a big IT support company, and for a big energy client, so you would think they would be extra careful, but they don't.

Pete Wildsmith

@foone One way to deal with “it works on my machine” is to make the developer write the code on the prod hardware, I suppose.

Sebastiaan Dammann

@weargoggles @foone "it works on my machine", "then we will just ship your machine"

666666t

@foone@digipres.club oh i've had this one happen before!!
booted up into the recovery shell of a cheap-ish modern arcade game that wasn't booting properly into x11 because of a corrupted video driver file, only to tap up a few times and start seeing various make invocations and thereabouts in the history

so... not the most uncommon practice, somehow

wakame

@syn @foone

Is there a .ssh directory? :blobcatgiggle:

syn

@wakame @foone answer: yes to both apparently :blobcatscared:

Shambolic Matter

@foone I work in video games and the products we make at my studio are generally considered to be held together with tape, glue, and broken promises. And then I look at people who are working on things like appliances and cars and their software is SO MUCH WORSE than our bad software.

a kilo of saucepans (rakslice)

@foone just an utter lack of people in the organization who knew what they were doing and I'm not even joking

a kilo of saucepans (rakslice)

@foone like the proverbial project done by one precocious somebody's-relative or resourceful generalist grad student who doesn't know what they don't know and everyone else in the organization responsible for delivering the project as a product knows even less

Markus Lindenberg

@foone Raspbian could and should be so much better imho. They should have build their distribution into something more of a Debian based read only firmware image building system, disable almost all logging to sd by default etc. and educate about best practices.
Now the internet is full of nice projects built on sd trashing ext4 r/w filesystems that should really be immutable firmware images.

Thea / sitharus

@foone it worked on the developer’s machine… so they shipped it!

Dr. hc.* Grober Unfug

@foone you botched a cool solution to a problem, Boss of course sells it and now you have to deliver something that an unwilling non Computer scientist can work with. NEXT WEEK.
Now the dev System Image goes on live production computers everyone agrees and swears to kill of the botchwork with the next Update.
Guess what. There won't be Updates cause production policies.
First year goes good, second year comes nothing happens. Everything is fine. And now you dismantle that botchwork! 🤪🤣

sebastian

@foone Actually not that uncommon. I've found similar stuff at least twice. My dayjob is to maintain an in-house embedded Linux distro for our IoT devices. I regularly get to mess around with the newest pi compute module base white label hardware platforms. So I know what they look like and how to get the SD card out or dump the emmc, if I find one in the wild.
Best thing that I found so far was an image with an IDE, git repo of the product and many other repos that person worked on still on the card.

Emily S

@foone "it'll be a like docker containers but physical"

kaitlin

@foone if it works on your machine we'll make a docker image of your machine and ship that

Foone🏳️‍⚧️

@thememesniper who needs a docker image? just clone the drive of the developer and ship that!

m0xEE

@foone
People forget to clean up all the time! And shit makes it into repos and sometimes even into production images as in your case.
This client was featured just yesterday on "This week in Fedi" and I'm not familiar with the whole Node.js ecosystem, but it seems to me that this is still just a log file that should have been added to .gitignore from the get go and thus never make it into the repo: github.com/Xyphyn/photon/blob/

Randagodron

@foone and a general rule of thumb is to NEVER EVER use a raspi in production. These boards are notoriously unreliable. I regularly hear about startups that have outage problems because their IoT devices have faulty raspis. These were designed for education purposes, nothing more.

punIssuer

@foone one too many exchanges of
"It works on my machine."
— "But we're not shipping your machine!"

vxo

@foone way back when I was trying to fix a first generation Harris Broadcast HD radio exporter and found a bunch of rather revealing .bash_history and home directory entries. I found it pretty hilarious. Rushed to market? Neeeeeverrrrr

The Hat Fox

@foone Quite a wild ride reading this thread! This strikes me as a perfect example of “knowing enough to be dangerous”. Whoever put this product together was able to get it functioning and out the door, but clearly had no idea how to do so properly and adopt best practices. Unfortunately there is a lot of that in the IoT space.

Григорий Клюшников

WHY WOULD YOU DO THIS?

Because you don't have enough experience in this field to know any better. And your self-esteem prevents you from looking for advice from those who do.

Andrew Elwell

I suspect @foone is running a stealth marketing campaign here - I'm now wondering if I can buy one of whatever this is *just* to see how badly they've fucked up!

Erik Ableson

@foone This is the low-tech completely clueless version of « it worked on my machine »

robelix

@foone

It works on Dave's computer.
OK, let's deploy Dave's computer.

Ryan Walmsley

@foone This is rather common and annoying.

RasPi actually did provide a tool to generate a fresh image with all of your changes too.

Henryk Plötz

@foone *shrug* "It works on my machine." "Ok, let's ship your machine."

Sam Burns

@foone I believe something similar happened with the footfall-cam/nanny-cam scandal. The SD card from the rpi had a Bruno Mars MP3 on it. On a CCTV camera.

NormanDunbar

@foone Good thread, thanks. Have you, by any chance, advised the company in question, and offered your services for a decent fee? 😉

Christopher Brown

@foone Thank you for writing this up. Even in the more abstract, it’s still instructional on what not to do.

jakob reading solarpunk

@foone That's my kind of open source. Not 'provided as is' but 'provided with an extensive audit trail'. :D

Kofi Loves Efia

@foone these are the same people who want to replace employees with auto complete. Checks out

Steveg58

@foone
Because Devs have absolutely no idea about building a release. They tend to give the job to the most junior person because "it isn't proper dev work" and "how hard can it be". I once went into a Medical Software company ans their first professional Configuration Manager. This is software with a high litigation value. The last release build by the devs took 3 days and they were not able to tell the testers what changes were included. I got it down to 3 hours including comprehensive release notes. Didn't stop one of the directors (an ex dev himself) telling me to "throw out your process we need this done as soon as possible".

@foone
Because Devs have absolutely no idea about building a release. They tend to give the job to the most junior person because "it isn't proper dev work" and "how hard can it be". I once went into a Medical Software company ans their first professional Configuration Manager. This is software with a high litigation value. The last release build by the devs took 3 days and they were not able to tell the testers what changes were included. I got it down to 3 hours including comprehensive release...

viq

@foone
There are jokes about "it works on my laptop" "we'll just ship your laptop to production then"
Well, apparently someone didn't realise it wasn't supposed to be serious advice.

Chris Samuel

@foone “look when we said it wanted to make this open source this wasn’t what we meant”

Kokos 💠

@foone Raspi? I have no clue. Nvidia Jetson? I totally understand.

Daniel Taylor

@foone it's the manager way around the "it works on MY machine " problem.

Now everything is that developer's machine

Fink :antifa:

@foone Features, features, Features! Customers ain't paying for this quality thingy!

Hamish The PolarBear

@foone Wow - I already boosted this post but the thread just got worse and worse - read and learn dev people!

gadgetoid

@foone this entire thread summarises why I try very hard not to deploy anything that isn’t CI automatable in a public git repo. I could *easily* be this incompetent!

HarJIT

@foone
"I don't care if it works on your machine; we're not shipping your machine!" is evidently now dead and buried.

Kodak Tgirl 400

@foone i definitely did this at a hardware company lol

King Calyo Delphi

@foone My guess is they didn't even know about .bash_history

Mx Autumn :blobcatpumpkin:

@foone its threads like this that keep my imposter syndrome in check.

Colin Cogle :verified:

@foone Too bad you’re a nice person and won’t dd a disk image and share it via BitTorrent for, let’s say, distributed research purposes.

This doesn’t sound like any IoT product I own, thankfully, but I hope this gets reported properly.

Tina H

@foone After THIS weeks "news", you ask THAT? ;)

"There's no procedure anymore. It's a fucking disgrace ... " - Torchwood.

Chris Bohn

@foone Maybe they took this idea and decided to skip the part where you have to configure Docker.

A young guy, holding back tears: "It works on my machine."

A compassionate-looking man: "Then we'll ship your machine."

The two of them hugging: "And that is how Docker was born."
poleguy

@foone I imagine the developer was being paid by the hour. The manager asked for a demo, saw that it worked and stopped paying the dev. Then he got his kid to duplicate the setup for him.

Go Up