Email or username:

Password:

Forgot your password?
Graham Spookyland🎃/Polynomial

if you've ever messed up a dimension or a hole position on something you're building, don't be too hard on yourself.

at least you're not the Cisco design engineer who caused an entire product line recall by placing the mode button (which resets the switch if held) directly above an RJ45 port.

227 comments
Till O'Rly :v8rified:

@gsuberland @13ma1 @thegpfury

Just imagine the person who had to write this:

"Workaround/Solution
There are three options used in order to address this problem:

* Use a snagless cable with a less-pronounced boot in Port 1.
* Trim the boot on the cable that is installed in Port 1.
...
"

John Cas

@13ma1 @thegpfury @mrgl @gsuberland file the reset switch back so it no longer protrudes and then cover it with tape.

Fernando :verified:

@moresunshine @13ma1 @thegpfury @mrgl @gsuberland Option 4. Place a rj45 port blocker so no one uses this port ever. 😂

Graham Spookyland🎃/Polynomial

@thefern inb4 the port blocker comes with a protective boot

Interiorce

@thefern lol if i paid $1500 for a 48-port switch, i would want to use all the 48 ports lol

Graham Spookyland🎃/Polynomial

@thegpfury 3650, 3650X, 3850, 3850X, and possibly a few others. It resulted in two separate recalls.

Graham Spookyland🎃/Polynomial

it's pretty impressive to fail so hard that you become a worldwide internal meme at a company the size of Cisco. back when I worked there, I saw this image used in slide decks and on posters in several different offices.

Graham Spookyland🎃/Polynomial

(I should've really said "Cisco design engineering team" rather than just one engineer; this is very much a combination of process & oversight failures)

Tom R

@gsuberland I was about to say, blaming one person is a bit rich. The fact that it made it to production indicated a total breakdown of their processes.

Still wild though ☺

Carl Liebold 🚀

@gsuberland This is a mistake by the Product Manager. But also, it should have been caught by the test team. That said, it's a PM mistake. Stuff like this happens, and then it's all about how the team responds. Oh, and I haven't checked lately, but I don't think those boots are part of the connector standard.

BMcRaeC@mastadon.social

@VintageVeloce @gsuberland it is terrible design regardless. Having a reset button not recessed AND in a place where clumsy fingers inserting the cable could accidentally push the button is just poor conception.

Carl Liebold 🚀

@Bmcraec @gsuberland
It probably happened because of the very limited front panel space. These things are designed to get as many ports as possible in the smallest possible space... leaving little room for stuff like switches. I agree it's a poor design...

Baloo Uriza

@gsuberland To be fair to that team, there's an extremely good chance that when the chassis was designed, it was still before widespread adoption of those protective boots.

Graham Spookyland🎃/Polynomial

@BalooUriza It was ~2013, so protective boots were very much a thing.

Baloo Uriza

@gsuberland Aah. I guess I didn't see them become ubiquitous until a couple years ago. I guess ethernet cables have long lives, since I started seeing them under desks before in datacenters.

Paul Schoonhoven 🍉 🍋

@gsuberland yes.. A lot of people didn't realise until the first accidents happen, I guess. 😇

B.

@gsuberland yes this is what impresses me the most. This passed the whole validation chain ! 🙄

Dr. Juande Santander-Vela

@gsuberland I was going to say just that: that this is not an individual contributors fault, is the whole process that failed. #BlamelessCulture works better for avoiding these kind of things.

JJ Krawczyk

@gsuberland Yeah how a QA process could miss this is hard to imagine.

Eat the Rich Jefferson

@gsuberland When was that? I was there from 1999-2014 (PGW-2200, BTS-10200, ISR-45xx) and never saw it... Ah, just saw the doc links- Catalyst a few years after that. Yikes. Up there with the Bevis and Butthead crash screen for sure.

Dustin [BusySignal]

@gsuberland what model / line was replaced or what was the true fix ?

I always struggle to believe that this was missed - and more - it was someone meeting a deadline and not delaying something...

Graham Spookyland🎃/Polynomial

@dustinfinn 48P models of 3650, 3650X, 3850, 3850X, and a few others.

Field Notices were issued to warn customers about the problem and provide workarounds (use a less pronounced boot, cut the boot back, or modify settings to disable the button). It wasn't a mandatory recall like you'd get with a safety issue; it's more like "grounds for RMA".

Adultxtraffic.com

@gsuberland let's give more credit to every person from the drawing board to quality control who saw this product and let it out the door. So many eyes, so many chances to stop it lol

Em McDonald 🏳️‍⚧️

@Adultxtraffic @gsuberland So much this. Sure, it's a big fuckup by the designer, but fuckups happen every day. We don't see most of them because of institutional process to catch and fix them.


@gsuberland great example of why QA testing is important :)

Timon 🛠

@gsuberland lol how did that pass QA tho
It's one thing to design a brain fart, another to actually qualify and mass manufacture it.

Graham Spookyland🎃/Polynomial

@timonsku I should've really said "design team" not "design engineer" - this was definitely a culmination of errors involving people and processes.

Kit Bashir

@timonsku @gsuberland this happened because all the cabling available to devs and QA is shitty old bootless RJs with the retaining clip snapped off.

Timon 🛠

@Unixbigot @gsuberland Even with that condition, the placement of both those USB ports is very questionable even without the hood on the cable that would cause issues.
If they considered hoods but not as long as the help article suggests, that makes it even more bizarre. You can read half the status lights or use the USB connectors or reach the button properly without unplugging cables.

Eddie Roosenmaallen

@timonsku @gsuberland
The way I heard the story, the machine that passed QA was a little different, then they got a last minute directive from the Branding Team on where the logo had to be.
That required moving the reset button, but "it's just moving the logo, not an engineering change" so they skipped the proper QA cycle and made history.

rag. Gustavino Bevilacqua

@silvermoon82 @timonsku @gsuberland

Branding teams and marketing teams must not be allowed to work inside the shipping box: they must be limited to fancy packaging (and sometimes they even mess with that).

HRH ginsterbusch

@silvermoon82
yes, that sounds so much like pretty lil "designer" brain, it hurts. Fremdschämen is a thing.
@timonsku @gsuberland

Michael Enger 💜

@silvermoon82 @timonsku @gsuberland Having worked on technical projects and been at the mercy of managers and marketers for over a decade, this is what I assumed had happened when I saw the picture. When stuff like this goes wrong its common for people to go "lol technician bad 😝" but in my experience it's almost always caused by some non-technical person pushing through a change.

Daniel Brockman

@timonsku My experience of silicon valley is They’ve mastered the art of inadvertently overlooking bugs in the product that everyone else sees unavoidably. Lincoln Spector wrote in the late 1970s of the cleverly designed Seppuku Mark 3 keyboard. To save precious space, the Mark 3 had the reset button adjacent to the backspace key. @gsuberland

Linh Pham

@gsuberland A reset is a form of a protective boot, right? 🙃

InstantArcade

@gsuberland this reminds me of the server case we had that had the entire top front as a very light touch power button.

We got a lot of calls about the server being down because there was a shelf behind it and people would accidentally turn it off when reaching for something.

Envizage

@gsuberland
I'll see your RJ245 and raise you a $125 million Mars Climate Orbiter .
Lost because nobody converted from Imperial to metric

fraggLe!

@gsuberland Woah, and I thought the home WAP a family member had was bad... the one where "WPS-PB" and "factory reset" were the same button, the latter just being what it did if you held it down for ~2 seconds.

Attie Grande

@gsuberland I love this one... it's so perfect that it's like it was designed specifically for this to be an intended use-case. 😂

Ben

@gsuberland dimensions are meaningless in a black hole

Philosphene

@gsuberland can confirm this causes the switch to restart

Seth Hanford 🐡

@gsuberland @hacks4pancakes I also heard people talking about how they changed the handsets on a generation of phones because the receiver audio holes would catch earrings. It amazed me how much our teams had to think about when shipping millions of a thing.

yakkoj 🦊

@gsuberland you know, I'm surprised CSCO recalled the product rather than instructing TAC to tell the customer to snip the protective boot with some scissors. Very impressive to not saddle us with more tech debt!

Shawn K. Quinn

@gsuberland Am I wrong for jokingly calling them "Crisco" all these years?

:mastodon: Edward Rosen :verified_paw:

@gsuberland I don't know if the team still exists, but in the 90s into the early 2000s, the Serviceability and Design team was the coolest there was. Sandwiched between TAC and Product Management, this team made sure syslogs covered various scenarios. They made sure MIBs were coded. They also looked after hardware things like this - all from a customer's perspective to try and make sure things like this didnt happen. The team was able to invoke a line stop and even hold it until problems were fixed. They worked as peers to PMs to make sure products were serviceable. It's rare to see this job function today. I don't know of any company that has engineering types in a customer advocacy role whose sole job it is to be sure a user can sufficiently troubleshoot a product.

@gsuberland I don't know if the team still exists, but in the 90s into the early 2000s, the Serviceability and Design team was the coolest there was. Sandwiched between TAC and Product Management, this team made sure syslogs covered various scenarios. They made sure MIBs were coded. They also looked after hardware things like this - all from a customer's perspective to try and make sure things like this didnt happen. The team was able to invoke a line stop and even hold it until problems were fixed....

Graham Spookyland🎃/Polynomial

@edrosen I met a few people who were in this kind of role, who knew every tiny secret detail about the switches from hardware to software. Unfortunately didn't get much of an opportunity to work with them outside of brief encounters - my job there was as part of an acquisition, which was a rather inauspicious story.

the vessel of morganna

@gsuberland cisco seems to have a trend of doing dumb shit like this. I've run into both the 3850 reset button switch and some older models where the boot gets caught inside the recessed bank of switchports and makes the cable impossible to remove (with the really cruddy cup shaped ones, not the type in your photo)

Dan Wallach

@mattblaze @gsuberland This is one of those "you only had one job" sorts of situations.

Clive Thompson

@mattblaze @gsuberland

Agreed -- the fail is so pure it's like an element as yet to be discovered

Cyper Bunk

@gsuberland even mechanical engineers pass unit tests but fail integration testing sometimes.

Brad Ganley
@gsuberland I'd nail that button every time I removed a cable from that position with our without the tab guard
Piotr Esden-Tempski

@gsuberland The pinpoint accuracy of this mistake is staggering. Is that how conspiracy theories are born? 🤔

Jake ✅ 🇺🇸 🇵🇱

@gsuberland Just crimp your own terminators. The plastic they use on those pre-fab cable hoods gets so hard I need a pair of pliers sometimes to get them unplugged. Usually once my fingertips are cooked. Still some of the best routers out there. Firewalls and Aeronet radios, too. Just don't ever use the graphical config interface. It's junk.

Karl Auerbach

@gsuberland - Perhaps its an editorial design element that reflects the designer's feelings about the device.

I've had network devices and computers that ought to have come with an attached sledge hammer to make needed "adjustments".

caleb

@gsuberland I have personally reset this fucking switch

Louis Mamakos

@gsuberland Back in the early Cisco days (mid 1990's), when they transitioned from AGS+ to 7000 boxes... The dimensions of the cabinet of the Cisco 7000 were precisely set to be the width of an open-frame 19" telco rack. With no margin for error.

We had our techs out in the telco colos with car jacks trying to wedge these things into racks. You gotta leave just a little margin -- floors are not levels, the racks are... racked slightly, etc.

It got better. We all learned a lot in those days.

Graham Spookyland🎃/Polynomial

@lmamakos reminds me of the phrase "trying to land the plane and run out of fuel just as the wheels touch down"

Louis Mamakos

@gsuberland It's all in the details. Early hot-swap linecards in cisco routers were like that. It'd work just fine.. MOST of the time. But that 1 out 50 attempts, you'd slide in the board, all the indicator lights would flash, the major alarm buzzer would sound and you knew it was bad news.

Don't even try to swap a line card with a different flavor board. Weird data structure problems you'd experence maybe hours later.

It was all new; Cisco and their ISP customers were learning together.

Graham Spookyland🎃/Polynomial

@lmamakos sounds about right tbh. I didn't join until 2015-ish when they acquired the company I worked for, but their initial foray into security services offerings was, uh, rather inauspicious. they just had no clue when it came to delivering those kinds of projects. took a lot of work from a few dedicated people to get it stumbling off the ground, several of whom ended up doing more than one job. sounds like they're doing better now though.

Nick

@gsuberland

The sort of thing that happens when there's no M.E.'s involved.....

Jon 404

@gsuberland I was affected by this "feature". Fun fun.

Jeff Enderwick

@gsuberland Having worked adjacent to the switching BU(s) of Cisco for years I struggle to believe this made it out the door. Which switch is this?

Yes, I though the 5800 was ridiculous.

preciousroy

@gsuberland

Damned if I don't remember a co-worker doing that in production. And calling tac to find out why the switch wouldn't keep a config.

skry

@gsuberland Something similar happened at Sun when a box was released with a new design: the CD drawer on the side. Turned out that if your box was in a rack, the drawer was inaccessible (all software was loaded that way, including the OS, because of bandwidth slowness and expense).

AFAIK it was not recalled, but many were returned. ancientelectronics.wordpress.c

Graham Spookyland🎃/Polynomial

@skry to be clear, in this context "recall" means "cause for return to manufacturer" rather than the kind of recall you'd do for a safety issue.

Graham Spookyland🎃/Polynomial

@skry there's a proper internal term for it but I didn't work on that side of the business and I have the memory of a goldfish

Joshua Small

@gsuberland The thing here was not that it happened, the thing was that it happened on kit that cost substantively more than most competitors, based on marketing itself as a premium product. I remember dealing with this at a time that affected switches cost 500% what the best alternative was.

Alan Campbell For Harris

@gsuberland I recall a monitor used in ATMs I used to work on that had what I thought was a poor design that placed the video PCB under a metal cage. Said cage had several tabs, one tab went inwards and under the PCB instead of outwards. That made them a right pain to service.

Ralf Lenz, BOFH Emeritus 🏴‍☠️

@gsuberland The 13 year old in me is really trying hard not to make a joke about messing up hole position.

UliBass

@gsuberland Hat Cisco eigentlich schonmal was gebaut, was sie dann nicht selbst unbenutzbar machen?! 😂
Mich fasziniert ja seit ca. einem Monat, wie die ihr Webex so verkrüppelt haben, dass es bei meinem Auftraggeber praktisch nicht mehr nutzbar ist...

virkon

@gsuberland OMG *rotfl* There is NO way the V&V people (if they have them at all) have missed that. How ignorant can a company be?

DELETED

@gsuberland HAH WOW
XD

new perspective successfully achieved

Toot Zi :mastodon:

@gsuberland I think the bigger design flaw is that the reset button is not embedded into the chassis. If it’s pointing out like that it’s possible to accidentally reset even if it requires some longer press. Simply working on the rack and leaning on the box would suffice.

Steinar Bang

@gsuberland

Disclosure: Mine: The cover for the kitchen fan.

Good to know I now can rest easy on that account!

Shy_sakura_

@gsuberland It is never a fault of one single person. One specified a requirement, one designed it, one built it, one tested it, one approved and released it.

Which one was the one who didn't do the job properly?

Nicolai

@gsuberland lost revenue opportunity for the port 1 cable. Proprietary and patented ofcourse.

Graham Spookyland🎃/Polynomial

@martinvermeer wasn't there an actual aircraft that did something like this? I forget what the actual toggle was (gear? fuel dump?) but it was right next to another commonly used control.

Chris Merle

@gsuberland I remember a computer lab at a college that has Macs and PCs. The Macs were PowerMacintosh 6100 series which had the power button next to the floppy drive. Students sitting down would stick their disk in the computer not realizing it wasn’t Windows until they got an error message it couldn’t read the disk. Macs didn’t have a disk eject button, so the students would push the power button turning it off. They’d turn it back on and the Mac would spit out their disk.

tto

@gsuberland not trying to defend an obvious design problem, but whoever plugs in that cable without realizing it presses that button should also rethink their career path

Сер Пітер Лорд

@gsuberland I was the UK serviceman for the Triumph-Adler Alphatronic micro back in the early 80s. This was before IBM standardised the industry. A rather unfortunate design oversight placed the CPU reset key just above the enter key: it certainly encouraged accurate touch-typing!

kaputtke

@gsuberland rj45 is one of the most cursed thing ever designed

Thomas Arend

@gsuberland

Every designer should test their product and assemble it 10,000 times before it is sold.

Jason Scurtu

@gsuberland wow, love it.. could be a feature :) But can’t only blame the designer, it goes much deeper then that.

Apicultor 🐝

@gsuberland Given that made it all the way through the design process and into production, it was not the engineer's fault.

elliot 🇩🇪 🇪🇺 🇬🇧 🏳️‍🌈

@gsuberland i think this says more about the product testing in place than about the engineer who designed it

Cosmic Tentacle

@gsuberland Are you sure that wasn't designed by politicians?

Michael

@gsuberland How did this ever pass the QA department?

Dietrich Feist

@gsuberland Hey, that's a user problem! Why do they insist on sticking 3rd party products into these ports? 😉

lertsenem

@gsuberland I'm definitely saving this for a poster in my own office. :D

Ksapp

@gsuberland
You got this totally wrong - this is cyber security- an unusable product can't do any harm ((-;
@Gargron

Paul H

@gsuberland

It's an attention to detail thing. For spectacular fails, look no further than
skyatnightmagazine.com/space-m

and

simscale.com/blog/nasa-mars-cl

This second one I use as a case study on why we use an internationally agreed set of units in science and engineering, and what can go wrong if we mess units up...

masterdan

@gsuberland It’s not the first time a switch goes in reset because of the mode button, no matter where is placed. Any good engineer with Cisco experience knows the trick: „no setup express“ in the config and oh wonder, you can now press the button for weeks and nothing happens.
I blame Cisco only to not set this as default.

Go Up