Email or username:

Password:

Forgot your password?
All posts Lesley Carhart :unverified:'s posts Post Back to profile
Lesley Carhart :unverified:

The number of cases I have that are "China" but are actually VTP,
Do not use VTP
Never use VTP
especially in your legacy industrial environments
somebody is gonna plug a switch in
send toot

20 March at 19:53 | Open on infosec.exchange
45 comments
Xavier Ashe :donor:
pasta la vida

@Xavier @hacks4pancakes someone misconfigured a VTP trunk? or spoofed one?

20 March at 19:55 | Open on tech.lgbt
Lesley Carhart :unverified:

@risottobias @Xavier nobody did either, somebody in a factory always ends up plugging in an old switch out of a closet

20 March at 19:57 | Open on infosec.exchange
Dave Wilburn :donor:

@hacks4pancakes @risottobias @Xavier lol, I was once accused of taking down the network for a major theater wide military exercise with an ARP storm. Turns out, someone took an unmanaged consumer hub and plugged it into the exercise network. And then someone took an Ethernet cable, plugged one end into the hub, and the other end into the same hub. It took the better part of an hour to finally track it down.

20 March at 20:26 | Open on infosec.exchange
Ash

@hacks4pancakes @risottobias @Xavier A lesson learned early on at University and at various jobs: Someone will definitely bring a wifi router configured to be a dhcp server in from home and plug it into the network to get better wifi reception on their laptop.

20 March at 20:51 | Open on hachyderm.io
Mr Midnigh7

@hacks4pancakes @risottobias @Xavier this is why I always backup and then clear a switch especially all vtp config before storing it in a closet. I love my vtp too much to not use it. 🙂

21 March at 5:03 | Open on infosec.exchange
Scott Hall
Alan

@Xavier @hacks4pancakes easiest way to loop up a network in existence, short of a physical loop.

20 March at 20:46 | Open on infosec.exchange
marek

@hacks4pancakes that reminds of the best lesson of my early IT life, I learned VTP very quickly, took me an hour or two to restore what blew up.
Good lessons, horrible protocol, should die on the lowest level of hell.
I feel sorry for people with big switching networks that do not read the fine print.

20 March at 20:04 | Open on infosec.exchange
Lesley Carhart :unverified:

@mlukaszuk This is why it is super important to have other experiences in systems and network administration going into security because stuff like this happens and they're like "switchports are flashing, network is dead, it's China"

20 March at 20:06 | Open on infosec.exchange
marek

@hacks4pancakes I reached security after few tech support roles. I can see that from my point of view this path is a huge benefit.
I know EXACTLY what you mean :)

20 March at 20:12 | Open on infosec.exchange
NightDice

@hacks4pancakes @mlukaszuk I feel like I should stop being surprised how prevalent "it can't be our own not-so-good decisions coming to haunt us, it must be [insert state-level-actor]" is.

20 March at 20:17 | Open on chaos.social
Bai Shen

@nightdice @hacks4pancakes @mlukaszuk That would imply that people were willing to admit their faults.

20 March at 20:35 | Open on mastodon.online
marek

@baishen @nightdice @hacks4pancakes we are going into a dark place here. I would like to start by saying that after being in various roles in IT for +20 years I honestly still have hope for people. Sometimes people act recklessly, sometimes they don't know better, sometimes they don't connect the dots quickly enough. People are mostly not malicious from my personal experience and in most cases they are willing to learn. As in any industry also in security there are assholes and as a (despite what the profile picture shows) white male I am fully aware of my privileged status which might affect the result here.
What I see in my personal experience is lack of willingness from people to invest in fundamentals how filesystems work, how network protocols work, how OSes run various processes in them, some basic (not language, level) coding, some basic concepts from cryptography. They are not required but they help a ton in understanding environments quickly and they allow to spot possible interesting problems between the boundaries of silos teams.

sorry for the wall of text, I was lurking here for a while already, but this topic struck that one note 🙂

@baishen @nightdice @hacks4pancakes we are going into a dark place here. I would like to start by saying that after being in various roles in IT for +20 years I honestly still have hope for people. Sometimes people act recklessly, sometimes they don't know better, sometimes they don't connect the dots quickly enough. People are mostly not malicious from my personal experience and in most cases they are willing to learn. As in any industry also in security there are assholes and as a (despite what...

20 March at 21:13 | Open on infosec.exchange
XenoPhage :verified:

@hacks4pancakes @mlukaszuk And it's just the poor guy frantically blinking those lights with morse code because he's locked in the boiler room and someone turned the heat up.

20 March at 20:30 | Open on infosec.exchange
Tindra

@hacks4pancakes @mlukaszuk there’s a reason why “…sometimes someone just tripped over a power cable” is a thing I say. Frequently.

20 March at 21:07 | Open on infosec.exchange
lj·rk

@hacks4pancakes @mlukaszuk Can confirm. I've worked on the dark side (IoT dev) before switching to the good team (audits of aforementioned stuff and others)

20 March at 21:39 | Open on todon.eu
h2onolan

@hacks4pancakes “hey we are just going to connect this new control panel to the plant network for testing”

20 March at 20:13 | Open on infosec.exchange
Wil

@hacks4pancakes STP was my big wake up. Plugged a new switch into the LAN that my employer’s call centre ran on so we could link up some PCs to play Quake II.

All the switches dipped off line while they calculated the spanning tree and I got shouted at a lot by my boss because nobody could take an order.

20 March at 20:24 | Open on infosec.exchange
Alan

@hacks4pancakes one of Cisco’s worst ideas, and that’s saying something.

20 March at 20:44 | Open on infosec.exchange
John Timaeus

@hacks4pancakes
Maybe expand this to:
Never run a vendor proprietary network protocol ?

20 March at 20:51 | Open on infosec.exchange
Audrey :v_trans:

@hacks4pancakes I always set the core to such a low number that you’d have to purposely want to take the network down.

20 March at 21:16 | Open on tech.lgbt
Lesley Carhart :unverified:
Audrey :v_trans:

@hacks4pancakes

My mentor when I was studying for my CCIE told me the horror stories of someone taking down the network by plugging in an old lab switch with a higher VLAN rev. I would even go in and delete the vlan.dat before putting a switch into the boneyard. That paranoia translated into me eventually going away from route/switch and into security space :)

21 March at 0:13 | Open on tech.lgbt
Adam Thompson

@AngryTransLady @hacks4pancakes Also do not intermingle PVST and RSTP in the same production network. Worst-case I've seen was a regional trauma center (hospital) sending all emergent patients out-of-country [1] thanks to those competing protocols not coexisting nicely.

Of course I've also flatlined a network for 6+ hrs while trying to fix it preventitively(?), so YMMV :-/

[1] only about two miles away across the border, but still a Really Big Problem. No-one died or had any resulting permanent major problems as a result, that I know of.

@AngryTransLady @hacks4pancakes Also do not intermingle PVST and RSTP in the same production network. Worst-case I've seen was a regional trauma center (hospital) sending all emergent patients out-of-country [1] thanks to those competing protocols not coexisting nicely.

Of course I've also flatlined a network for 6+ hrs while trying to fix it preventitively(?), so YMMV :-/

21 March at 2:37 | Open on bsd.network
QRSS_Test

@hacks4pancakes VTP is horrible. I inherited a network running it, and found a bug that would cause the switch to drop broadcast and multicast traffic on the trunk interface. Broke DHCP, routing protocols, and mdns.

20 March at 21:16 | Open on mastodon.sdf.org
Lockpick Extreme

@hacks4pancakes a few of my lab mates were doing load testing on some equipment. They plugged into the corporate networks to be able to reach their testbed on the admin. Guess how they discovered the corporate network had some of the same VLAN numbers in the testbed.
VTP and Portfast were banned from our lab configs.

20 March at 21:38 | Open on ioc.exchange
Programmer 832-529 🍅

@hacks4pancakes if you haven't been burnt by VTP at least once are you even doing networking?

I'm surprised it's still an issue now though.

20 March at 21:45 | Open on social.dropbear.xyz
David Gerhart

@smallsees @hacks4pancakes
Legacy rats nests are a primary reason I got out of the MSP business (2008) ... corporate under-investment in hardware/software and SECURITY are still big personal issues for technologists.

they decide, we wind up being at fault.

pet peeve.

21 March at 0:28 | Open on fosstodon.org
Sass, David

@smallsees @hacks4pancakes

I guess I'm removing that two years of network administrator experience from my CV...

21 March at 6:00 | Open on infosec.exchange
Programmer 832-529 🍅

@sassdawe @hacks4pancakes

You missed all the fun!

Why are all the switches lights blinking orange and nothing works anymore.

I did a write erase where are these vlans coming from and why are they the ones from the building the switch was removed from/the run up room?

21 March at 6:10 | Open on social.dropbear.xyz
Sass, David

@smallsees @hacks4pancakes

The "fun" I experienced was the CPU hitting 110% utilization on Windows because some moron thought that it's a good practical joke to connect two LAN ports to each other... You see this was a university dormitory with ~1300 endpoints across three buildings, and I had to track down the room and request keys to the doors while all those users were loosing their mind not having internet access.

21 March at 6:46 | Open on infosec.exchange
Regis - HTTP 1.1/418 Teapot

@hacks4pancakes ARGH. Fucking VTP. A while back I ended up going through a whole bunch of switches and rage-disabling VTP. Like, no, just fucking do what I tell you, switch, stop trying to be smart.

20 March at 22:26 | Open on infosec.exchange
Michael Kohne

@rmd1023 @hacks4pancakes I have that reaction to a LOT of things in recent years. If everything could just stop trying to outthink me and maintain a stable UI, I can actually do things faster if I could memorize how things work.

21 March at 0:52 | Open on mastodon.social
Ewen McNeill

@rmd1023 oh no, is the VTP that @hacks4pancakes was warning us about VLAN Trunking Protocol?! 😱

(Here was I assuming “VTP” was some acronym overload for something else in operational networks that people should worry about. Not a 1990s network ghost living on 😬)

cisco.com/c/en/us/support/docs

21 March at 3:46 | Open on cloudisland.nz
Jima :Compromise_bi_flag:

@rmd1023 @hacks4pancakes I need to better understand how to kill VTP. I somehow only had my first unfortunate run-in this week. 🤔

21 March at 6:56 | Open on mspsocial.net
AMS

@hacks4pancakes Probably even worse in modern industrial environments where every other cabinet has a switch in it.

20 March at 22:38 | Open on infosec.exchange
JW

@hacks4pancakes After reading this toot, would you say you like or really like VTP? :ablobcatnodfast:

21 March at 0:35 | Open on infosec.exchange
Katharta

@hacks4pancakes I've found myself at a small MSP responsible for a hospital with a single network engineer who really likes VTP. Please send help.

21 March at 0:47 | Open on quietzero.net
Interpipes 💙

@hacks4pancakes you mean people actually use it on purpose!?

I guess it might be the sort of thing that is maybe tempting to use in an enterprise environment if you haven't understood how it works and the resulting risks.

Fortunately I learned with "service provider mindset" as we acquired a SP network with some Cisco in it ~20 years ago and when I took the cisco press books on holiday to learn how to run my new network, the immediate determination on VTP was: "no, never that"

21 March at 1:17 | Open on thx.gg
Krux
The Psychotic Network Ferret

@hacks4pancakes VTP does not solve any problem any non cisco network has, but it sure does cause problems for cisco networks!

21 March at 1:39 | Open on infosec.exchange
The Psychotic Network Ferret

Fun fact.

The only time @hacks4pancakes and I have collaborated even remotely professionally was because of a VTP issue.

My advice was basically:
*Remove the problem switch
*Reboot _everything_ else
*Pray

VTP is quite possibly the most profound example of a technology with zero regression testing ever created.

21 March at 1:50 | Open on infosec.exchange
Go Up