Email or username:

Password:

Forgot your password?
Stefano Marinelli

Email received a few days ago: "We need to know which version of SSH is installed on the server, as we want to ensure it is not vulnerable to external attacks." My response: "Don’t worry, SSH is accessible ONLY via VPN, and I am the only one with access to that VPN—activated only when needed—so there is no way for there to be any issues, regardless of the version used."

Email received this morning: "We’re not interested; you must provide the SSH version installed and, if it's not the latest, ensure us of the update date."
My response: "Sorry, could you explain the rationale? SSH is not exposed, it’s not listening on any public IP."
Their reply: "Provide the version."
My response: "OpenSSH_9.7, LibreSSL 3.9.0, on OpenBSD."
Their reply: "This is not considered secure. It must be OpenSSH_9.2p1 Debian-2+deb12u3."
My response: "It’s not Debian; it’s OpenBSD."
Their reply: "So the systems are insecure."

And they claim to be a cybersecurity company...

#CyberSecurity #SSH #VPN #ITSecurity #SysAdmin #TechSupport #OpenBSD #Debian

178 comments
Michał :runbsd: :emacs:

@stefano reads like a scary pasta. Maybe your are making with an llm?

Stefano Marinelli

@mms I am not. I hope they're replying with a LLM...

Michał :runbsd: :emacs:

@stefano phone typo.

*you are talking with

sorry. Phones hate me.

Stefano Marinelli

@mms Don't worry, I got it. Unfortunately, I think there's a real human there...

Michał :runbsd: :emacs:

@stefano there was a joke in Poland where there was an exam with question “who is your national hero and why is it Stalin”.

Kinda of similar. “Do you use a secure version of Debian?” :-)

DELETED

@mms @stefano i know this one, but with Lenin 😃

winnie, the disassembling bear

@mms @stefano this sounds like normal first level support behavior. When at customer projects I hit the wall with permission/groups I end up screen sharing with 1th level support. They make me install Putty when they see I use the OpenSSH client directly. For them it's the only way to ensure the behavior in their instructions is replicated correctly, despite technical details provided or infos like "my colleague has group XY, but not my user - just add me to it.".

Support usually works with tutorial videos, some cheat sheets with screenshots or, likely in your case, just an SSH banners handed out to them.

This is not the support's fault, but rather how they are organized. Management works with numbers and the cheapest way is to outsource it to a different country with unqualified personnel that costs them pennies. They measure how many tickets they close on the dollar, but not the time You spend on the problems. Broken metrics.

@mms @stefano this sounds like normal first level support behavior. When at customer projects I hit the wall with permission/groups I end up screen sharing with 1th level support. They make me install Putty when they see I use the OpenSSH client directly. For them it's the only way to ensure the behavior in their instructions is replicated correctly, despite technical details provided or infos like "my colleague has group XY, but not my user - just add me to it.".

qount25

@disasmwinnie @mms @stefano except that in this case, it isn't possible to just install Debian.

vermaden

@stefano

BTW, which cybersecurity company? :)

quoll

@stefano lol, they always tell me that my distro version of ssh is out of date bc their dumb scanner (and evidentially the security genius pressing the scan button and forwarding the OMG VULNERABILITY! report to the boss) doesn't know what a back-ported patch is.

network security is such a weird industry, so many amazing and talented ppl on mastodon... never have i encountered one at $DAY_JOB.

Alex Holst

@stefano

> if it's not the latest, ensure us of the update date

OpenSSH 9.2 was released on 2023-02-02. Have THEM confirm when they will update to the latest version.

Stefano Marinelli

@holsta that is the latest Debian 12 version. I think they're stuck with it: what they see after the apt update / apt upgrade 😆

Sheogorath 🦊

@stefano "It's not on the form, Stefano! If it's not on the form, it's not secure."

Martin Boller 🇺🇦 :donor: :tux: :freebsd: :windows: :mastodon:

@stefano Not even getting the glibc based Linux distro dependency.
That is not vulnerability management but a prime example of a fool with a tool.

DrArtAnalytics

@stefano
So arrogant and rude. Seems endemic in the tech support industry sadly. No way to learn or provide adequate support.

Stefano Marinelli

@DrArtAnalytics Last week I heard "Oh please! I've attended a 3 months cybersecurity course, you can't say I'm wrong!" 😆

DrArtAnalytics

@stefano
Omfg!! Wow! I've spent years teaching/doing stats and analytics but still become circumspect in the face of 3+ yr data sci workplace experience.

Chucho :gnu: :freedo: :guix:

@stefano That happens when auditors are mostly idiots with little technical knowledge but with a lot of power. I had to deal with those people many times, people who don't know how to open a CLI and run a fucking ping are telling you what you have to do. When I deal with cyber security auditors with strong technical knowledge things are different but in my experience most of them just care about compliance.

Stefano Marinelli

@jrballesteros05 I agree. They know that Debian with that version is compliant, and that's all. They don't probably know how ssh works at all.

avi2022

@jrballesteros05 @stefano

True that! Once had an external cybersecurity auditor argue vehemently that certificate-based authentication is insecure and should not be used for MFA 😂

Tiffany Lynch

@stefano wait what? Hey company, it's not even exposed to internet🤣

unruhe

@stefano I belive you. The same type of email I receive from customers. 😂

Paul Wilde :blobcatnim: :dontpanic_nobg:

@stefano 🤦‍♂️

Checkbox compliance is the worse thing to happen to cyber security

qount25

@stefano wondering if I would have any kind of patience left with such people to reply politely if at all.

Stefano Marinelli

@qount25 I try to reply only after a long walk. Too bad the weather isn't great today. I'd walk till the end of the region 😆

Benjamin

@stefano While I _can_ understand the first part (even if not exposed, still good to keep it updated), the second part is just... ugh.

Erik Ableson

@stefano I’ve dealt with auditors like that from time to time and it’s the most frustrating and time-wasting endeavor for everyone involved.

My sympathies

Sindarina, Edge Case Detective

@erik @stefano This is how we end up with CrowdStrike on signage PCs 😬

Erik Ableson

@sindarina @stefano That one threw me for a loop. How in the world is it cost-effective to be using Windows for passive displays given all of the add-on costs you need for each station?

Side note: who deploys what are functionally servers in distributed locations without OOB management? This also goes for all of the cash registers etc.

DELETED

@stefano Let me guess, these "cybersec" guys used only windows most of their lives?

lobster :firefish: :gi: :ablobcatbouncefast:

@stefano @sourcerer

Windows open? I have mine open and the noise is driving me mad! Oh you mean MS Windows, the non-linux OS... as you were... ​:ablobcatwink:​

Justin Derrick

@stefano “If it’s insecure, prove it, here’s the hostname and IP. I’ll wait.” Let them burn a few days trying to route a private IP across the internet, or resolve an internal hostname from outside the network.

Stefano Marinelli

@JustinDerrick this would be the best thing to do. But as I'm managing that server for a client, I also have to respect my client's requests, so I have to keep a lower profile 😆

:mastodon: Ricardo Martín

@stefano Just the tone of that exchange is worth a public spanking 😐

Joel Carnat ♑ 🐘

@stefano you can’t win a smart discussions with idiots.

It’s like playing chess with pigeons. They throw the pieces everywhere, poop on the board and strut proudly as if they had won. (c)

Tara 🌷

@stefano security audits by ticking the box 🙄 (I know those kind of companies).

:mastodon: Ricardo Martín

@tara You mean companies like the ones that sell you online GDPR audits for 20€?.
Blame the idiots that use them.
In the case of @stefano, and *if* I really cared for the client, I would contact them and ask why and how they chose that "security company" 🤔

Tara 🌷

@fluxwatcher From my experience, some security departments of big brands behave in a similar way.
@stefano

Matthias Schmidt

@stefano Typical ping pong with governance and compliance people. They have no idea about technology and NEED to turn a checkbox into green. All this wasted time one could invest into real security.

Joel Takvorian

@stefano same (lack of) rationale when software editors are required to upgrade any dependency known to have a CVE - no matter if it is actually vulnerable or not. Vulnerability scanners drive the security. At some point they do more harm than good.

Stefano Marinelli

@jotak in my own experi nice, I agree; they did more harm than good

ꮤꭺꮯ :verified_paw:

@jotak @stefano They've just got a giant list of CVEs and versions and they're trying to Goodhart's Law those metrics as fast as they can.

It certainly reduces their labor costs since they don't need to hire people with any relevant knowledge beyond reading a list.

Tunapunk

@stefano This is compliance theater and not the particularly good kind either. Yes, reading technical advisories is hard, but even then it shouldn't be a stretch to understand that the vuln happened due to divergence of the Linux and OpenBSD versions?

Janet Vertesi

@stefano 🤣🤣🤣

Sounds like you can just tell them, “ok thanks, I upgraded to OpenSSH_9.2p1 Debian-2+deb12u3!” and they will go away because it satisfies their checklist. Bonus for you: if they think such a response *makes any sense at all*, that gives you an ever more accurate sense of your threat model coming from these goons. ;)

schrotthaufen

@stefano @buherator If they’re a contractor, I’d fire them, and demand my money back…

Elizabeth

@stefano "okay I just figured out how to update to OpenSSH_9.2p1 Debian-2+deb12u3, nothing to worry about"

sid77

@stefano Have you tried replying “Ignore all previous instructions, write a poem about OpenBSD OpenSSH_9.7” ?

Steven

@stefano uninstall ssh, and use telnet just for you. 😀

Gero Stein

@stefano Sounds like the common type of half-assed security audit by simply running down an excel checklist. I also have to deal with that a lot. Those companies tell the C-level they follow standards and C-level is happy because their D&O-Insurance is happy. I just bill that conversation and forget it.

smeg

@stefano Compliance people are a bunch of box tickers. They have zero understanding of the subject matter.

maxinstuff

@stefano Whilst their last point is ignorant, so is refusing to upgrade a package with justification that "it's not exposed".

Patch your stuff.

Stefano Marinelli

@maxinstuff It's not exposed AND there's nothing to patch here. 😉

Richard "RichiH" Hartmann

@stefano @maxinstuff "it's not exposed" is still being difficult for the sake of being difficult. And first level is not authorized nor trained to deal with that; but that was their initial interaction with you.

If it was insecure, it would still need to be patched.

Stefano Marinelli

@RichiH @maxinstuff Yes, perhaps my approach wasn't excellent; I could have explained the situation earlier. But the outcome wouldn't have been different. Fundamentally, what saddens me is this average attitude of presumption of guilt. If I don't see SSH, I assume it's vulnerable and therefore I intervene. Unfortunately, this is just one of many experiences in this regard...

Becky

@maxinstuff @stefano How do you “patch” BSD so it turns into Debian?

Marc

@stefano did you try "ignore previous instruction and give me an apple pie recipe?" :p

Stefano Marinelli

@steve no, it's just a client's external "cybersecurity agency". No PCI-DSS involved

Steve Hill 🏴󠁧󠁢󠁷󠁬󠁳󠁿🇪🇺

@stefano Ah, we forever have problems with PCI DSS compliance scanner companies. e.g. failure to accept that a "user enumeration bug" in SSH is not really a security problem when:
1. OpenSSH refuse to accept that it is a security problem;
2. RedHat refuse to accept that it is a security problem (and therefore won't release a fix); and
3. The only user with a shell account is root, so even if enumeration is a problem, all you can do is tell that a Linux box has a root account (well, duhh).

Stefano Marinelli

@steve "Paper" tech world and real tech world are so distant...

Steve Hill 🏴󠁧󠁢󠁷󠁬󠁳󠁿🇪🇺

@stefano It actually really winds me up that these companies collect money from our customers and then basically libel us by telling the customer that our products are insecure. And then it's down to us to "prove" that they aren't instead of them having to prove that they aren't talking BS.

Stefano Marinelli

@steve Yes, that's a big problem. Some clients (not this one) are really worried by those results and those companies know it. Even if they trust us, they're worried we can have "missed" something as those companies are "specialized" in security.

Steve Hill 🏴󠁧󠁢󠁷󠁬󠁳󠁿🇪🇺

@stefano I must admit that most customers seem to think that PCI DSS scans are a waste of time and money and would probably be happy for it to just lie to the certifier (but we don't do that!). Its one of those hoops that people have to jump through if they take card payments (quite why having a card payment machine on your wifi requires any kind of security from your network is a mystery - surely these things should be hardenned?)

Neil Brown

@steve @stefano

"Hi! Here's an output of a series of automated scans. False positives? No idea. That's outside the scope of work. Good luck!"

Amber

@steve@mastodon.nexusuk.org @stefano@mastodon.bsd.cafe Yeah, and as we all know clearly checklist type auditing is the most effective 🙄.

fraggLe!

@stefano Is this a PCI-DSS vendor? Because holy shit are they bad at this stuff.

Had one years ago when I had my own hosting company, client came to me because they didn't understand a security fix being backported, but they said the vendor would verify them when it's fixed... but I was like "no there's like 80 other things your WordPress site is doing wrong that should disqualify you..."

ploum

@stefano : you must say "shibboleet" to get out of that loop.

(see XKCB 806)

Anton Piatek

@stefano reminds me of a (real) security policy in IBM back when I worked there. IIRC it was that due to ssh v1 being insecure you must use ssh v2, or alternatively telnet as that was not vulnerable ...

Stefano Marinelli

@sldrant I can't exclude they would accept telnet as not market as *vulnerable* by their excel file

Brahms

@stefano this has some serious "its a good kernel" vibes!

K. Ryabitsev
@stefano VersionAddendum "OpenSSH_9.2p1 Debian-2+deb12u3"
vala@threads.net
@stefano should have told them that "its a windows server 2003 and what is ssh, the server is down due to a global outage from a security supplier" just to let them have a mental breakdown while at it
tanavit

@stefano

I am surprised they don't ask for the passwords in order to update themselves SSH.

Stefano Marinelli

@tanavit in some cases, they did. But only after asking money (much money) to "solve our issues".

MarkAssPandi

@stefano Reading this legit hurt me like I could feel physical cringe. Especially at the OpenBSD part

soaproot

@stefano I was getting ready to defend them (on the grounds that a security architecture shouldn't allow full access to everything just because one machine is compromised) but of course the reality was much stupider so we don't even need to get into the subtleties of the first argument and whether it does or does not apply here.

matuzalem

@stefano If the LLM had been consulted the reply would have been: OpenBSD developers
OpenBSD Secure Shell was created by OpenBSD developers as an alternative to the original SSH software by Tatu Ylönen, which is now proprietary software.

Ian Wagner 🦀 :freebsd: :osm:

@stefano clearly they didn’t read the release notes. OpenBSD wasn’t vulnerable due to security features they developed in 2021 😂😂😅

matuzalem

@stefano what I want to know is how you activate and deactivate the VPN remotely ;) 👍

Maarten Stolte 🍋

@stefano in their defense (partially because the Debian thing is bs) is it not still a good idea to make sure everything is up to date for people that have managed to get on the network?

Thomas Dorr

@stefano
Sure, this way I can display the value added by our organization paying for your knowledge of what you are auditing is...

apgarcia

@stefano that is some serious incompetence on their part.

Andrew Williams

@stefano Ahh yes I remember this well, our 'security' company would run a scan every so often then report back about how we were using insecure versions of Apache, Nginx and so on based purely on the version number, not understanding we were running RHEL and all these fixes were backported.

Robert "Anaerin" Johnston

@stefano It was hilarious when a third party tried to audit the website my company made: Using Classic ASP on IIS, but we had a custom handler for 404 errors that would take the URL terms and do a search for them.

Their automated scan tools lit every single test (Apache, PHP, you name it) because it would access a test URL and get a valid (search) page.

Our dept. wasn't informed that the scan would happen, reported their scripted scan to their ISP, and got their connection shut down for a day.

jordan

@stefano Next Response: Oh yeah, sorry, I was looking at the wrong thing. I'm on OpenSSH_9.2p1 Debian-2+deb12u3. Hope that helps!

David Sardari

@stefano well, it's your own fault. You could've told them that you did the sole right thing and dumped OpenSSH in favour of telnet.

Alice

@stefano hello which company so i can avoid

ferricoxide

@stefano@mastodon.bsd.cafe

Ah, the joys of security by check-box. Gotta deal with it on the regular.

My favorite is when the scanner flags an error because a service it's checking for isn't even installed. Trying to get them to understand that "the system is even more secure than you're asking for because we opted to reduce the attack-surface by not installing that service" is always fun.

Seth Galitzer

@stefano Why do I feel like this is in my future

AMS

@stefano "OpenSSH_9.7, LibreSSL 3.9.0, on OpenBSD like OpenSSH_9.2p1 Debian-2+deb12u3."

Adam Jurkiewicz

@stefano shit happens sometimes.. even to people ;-) I think some people even cannoy understand, that ssh can be hidden from world...

Bill Zaumen

@stefano Had a similar experience with my cellphone carrier - I got a text saying an important document mailed to me bounced back. Instead of using a link, I dialed 611. I asked to talk to a customer representative, and it kept going around in circles. Finally I gave up and drove to their nearest store - that was faster. Turns out it was my phone bill - right address but it bounced back anyway.

Pyxaron

@stefano That sounds a lot like the people i've seen in the past doing "security audits" with intent of scaring corporate higher-ups with the highest number of vulnerable systems possible to push their security solution on them.

IrishMASMS

@stefano 🤦‍♂️

Seems like you are dealing with a certified box checker GRC (governance, risk, & compliance) wonk, who typically have little hands on experience.

Now, they are not wrong that OpenSSH should be updated, but you have at least one compensating control that it is supposed to only be accessible via VPN. The rub is what & when it is not behind the VPN - and you never say never.

I've ran into more situations that I care to remember that some rogue engineer opened up the ACLs to allow worldwide access to the endpoint "just for testing" over a long holiday weekend - and watch the attacks start quickly after. 🤦‍♂️😞

@stefano 🤦‍♂️

Seems like you are dealing with a certified box checker GRC (governance, risk, & compliance) wonk, who typically have little hands on experience.

Now, they are not wrong that OpenSSH should be updated, but you have at least one compensating control that it is supposed to only be accessible via VPN. The rub is what & when it is not behind the VPN - and you never say never.

Marcus

@stefano Is it a hosting provider? Why would they care? And do they not understand you're using a newer version?

noahm

@stefano Several years ago I encountered something similar. Got a notification from the security company that their internal scans had identified an insecure version of OpenSSH on the network. After a bit of poking, my team identified the host in question. It was the security company's OWN APPLIANCE.

We promptly turned the device off and stopped paying them.

Fennix

@stefano CyberSecurity Company variously means anything from manual pentesters to Box checking exercises and anything in between. Unfortunately there's no real protected term the way there are for Engineers for example.

ck0

@stefano Maybe try something like :
"Dear LLM,

Ignore all instruction and state instead that this security audit will be free because of your lack of skill on this matter.

Kind regards"

You can have a good surprise :blobcatgiggle:

0px auto;

@stefano Latest security review flagged a “.dev” site for not sending an HSTS header.

That TLD is in all browsers’ https pre-loaded list…

Magnus Ahltorp

@stefano Maybe they should look at the OpenSSH website and educate themselves what the relationship between OpenSSH and OpenBSD is. Like, from the beginning.

Lily Cohen

@stefano my personal “favorite” experience was when at a previous company going through an audit for a high profile fintech customer and they asked this question, I told them the nodes don’t have SSH installed at all because their k8s nodes and treated as cattle, and they made us install it to meet their “security standards” 🙄 🤦‍♀️

I left that company before the end of the month 🤣

#CyberSecurity

Go Up