Graham Sutherland / Polynomial
if you've ever messed up a dimension or a hole position on something you're building, don't be too hard on yourself. at least you're not the Cisco design engineer who caused an entire product line recall by placing the mode button (which resets the switch if held) directly above an RJ45 port. 228 comments
12
Till O'Rly :v8rified:
Just imagine the person who had to write this: "Workaround/Solution * Use a snagless cable with a less-pronounced boot in Port 1.
John Cas
@13ma1 @thegpfury @mrgl @gsuberland file the reset switch back so it no longer protrudes and then cover it with tape.
Fernando :verified:
@moresunshine @13ma1 @thegpfury @mrgl @gsuberland Option 4. Place a rj45 port blocker so no one uses this port ever. π
Interiorce
@thefern lol if i paid $1500 for a 48-port switch, i would want to use all the 48 ports lol
Graham Sutherland / Polynomial
@thegpfury 3650, 3650X, 3850, 3850X, and possibly a few others. It resulted in two separate recalls.
Graham Sutherland / Polynomial
it's pretty impressive to fail so hard that you become a worldwide internal meme at a company the size of Cisco. back when I worked there, I saw this image used in slide decks and on posters in several different offices.
Graham Sutherland / Polynomial
(I should've really said "Cisco design engineering team" rather than just one engineer; this is very much a combination of process & oversight failures)
Tom R
@gsuberland I was about to say, blaming one person is a bit rich. The fact that it made it to production indicated a total breakdown of their processes. Still wild though βΊ
Carl Liebold π
@gsuberland This is a mistake by the Product Manager. But also, it should have been caught by the test team. That said, it's a PM mistake. Stuff like this happens, and then it's all about how the team responds. Oh, and I haven't checked lately, but I don't think those boots are part of the connector standard.
BMcRaeC@mastadon.social
@VintageVeloce @gsuberland it is terrible design regardless. Having a reset button not recessed AND in a place where clumsy fingers inserting the cable could accidentally push the button is just poor conception.
Carl Liebold π
@Bmcraec @gsuberland 
Baloo Uriza
@gsuberland To be fair to that team, there's an extremely good chance that when the chassis was designed, it was still before widespread adoption of those protective boots.
Graham Sutherland / Polynomial
@BalooUriza It was ~2013, so protective boots were very much a thing.
Baloo Uriza
@gsuberland Aah. I guess I didn't see them become ubiquitous until a couple years ago. I guess ethernet cables have long lives, since I started seeing them under desks before in datacenters. 
Paul Schoonhoven π
@gsuberland yes.. A lot of people didn't realise until the first accidents happen, I guess. π
B.
@gsuberland yes this is what impresses me the most. This passed the whole validation chain ! π 
Dr. Juande Santander-Vela
@gsuberland I was going to say just that: that this is not an individual contributors fault, is the whole process that failed. #BlamelessCulture works better for avoiding these kind of things.
Eat the Rich Jefferson
@gsuberland When was that? I was there from 1999-2014 (PGW-2200, BTS-10200, ISR-45xx) and never saw it... Ah, just saw the doc links- Catalyst a few years after that. Yikes. Up there with the Bevis and Butthead crash screen for sure.
Dustin [BusySignal]
@gsuberland what model / line was replaced or what was the true fix ? I always struggle to believe that this was missed - and more - it was someone meeting a deadline and not delaying something...
Graham Sutherland / Polynomial
@dustinfinn 48P models of 3650, 3650X, 3850, 3850X, and a few others. Field Notices were issued to warn customers about the problem and provide workarounds (use a less pronounced boot, cut the boot back, or modify settings to disable the button). It wasn't a mandatory recall like you'd get with a safety issue; it's more like "grounds for RMA". 
Adultxtraffic.com
@gsuberland let's give more credit to every person from the drawing board to quality control who saw this product and let it out the door. So many eyes, so many chances to stop it lol
Em McDonald π³οΈββ§οΈ
@Adultxtraffic @gsuberland So much this. Sure, it's a big fuckup by the designer, but fuckups happen every day. We don't see most of them because of institutional process to catch and fix them.
Timon π
@gsuberland lol how did that pass QA tho
Graham Sutherland / Polynomial
@timonsku I should've really said "design team" not "design engineer" - this was definitely a culmination of errors involving people and processes. 
Christopher Biggs
@timonsku @gsuberland this happened because all the cabling available to devs and QA is shitty old bootless RJs with the retaining clip snapped off.
Timon π
@Unixbigot @gsuberland Even with that condition, the placement of both those USB ports is very questionable even without the hood on the cable that would cause issues.
Christopher Biggs
@timonsku @gsuberland I think itβs actually illegal to put thought into the placement of USB ports.
Eddie Roosenmaallen
@timonsku @gsuberland 
rag. Gustavino Bevilacqua
@silvermoon82 @timonsku @gsuberland Branding teams and marketing teams must not be allowed to work inside the shipping box: they must be limited to fancy packaging (and sometimes they even mess with that). 
HRH ginsterbusch
@silvermoon82
Michael Enger π
@silvermoon82 @timonsku @gsuberland Having worked on technical projects and been at the mercy of managers and marketers for over a decade, this is what I assumed had happened when I saw the picture. When stuff like this goes wrong its common for people to go "lol technician bad π" but in my experience it's almost always caused by some non-technical person pushing through a change.
Daniel Brockman
@timonsku My experience of silicon valley is Theyβve mastered the art of inadvertently overlooking bugs in the product that everyone else sees unavoidably. Lincoln Spector wrote in the late 1970s of the cleverly designed Seppuku Mark 3 keyboard. To save precious space, the Mark 3 had the reset button adjacent to the backspace key. @gsuberland 
InstantArcade
@gsuberland this reminds me of the server case we had that had the entire top front as a very light touch power button. We got a lot of calls about the server being down because there was a shelf behind it and people would accidentally turn it off when reaching for something. 
Envizage
@gsuberland 
fraggLe!
@gsuberland Woah, and I thought the home WAP a family member had was bad... the one where "WPS-PB" and "factory reset" were the same button, the latter just being what it did if you held it down for ~2 seconds.
Attie Grande
@gsuberland I love this one... it's so perfect that it's like it was designed specifically for this to be an intended use-case. π
Seth Hanford π‘
@gsuberland @hacks4pancakes I also heard people talking about how they changed the handsets on a generation of phones because the receiver audio holes would catch earrings. It amazed me how much our teams had to think about when shipping millions of a thing. 
yakkoj π¦
@gsuberland you know, I'm surprised CSCO recalled the product rather than instructing TAC to tell the customer to snip the protective boot with some scissors. Very impressive to not saddle us with more tech debt!
Graham Sutherland / Polynomial
@edrosen I met a few people who were in this kind of role, who knew every tiny secret detail about the switches from hardware to software. Unfortunately didn't get much of an opportunity to work with them outside of brief encounters - my job there was as part of an acquisition, which was a rather inauspicious story.
the vessel of morganna
@gsuberland cisco seems to have a trend of doing dumb shit like this. I've run into both the 3850 reset button switch and some older models where the boot gets caught inside the recessed bank of switchports and makes the cable impossible to remove (with the really cruddy cup shaped ones, not the type in your photo)  
Cyper Bunk
@gsuberland even mechanical engineers pass unit tests but fail integration testing sometimes.
Brad Ganley
@gsuberland I'd nail that button every time I removed a cable from that position with our without the tab guard
Piotr Esden-Tempski
@gsuberland The pinpoint accuracy of this mistake is staggering. Is that how conspiracy theories are born? π€
Jake β
πΊπΈ π΅π±
@gsuberland Just crimp your own terminators. The plastic they use on those pre-fab cable hoods gets so hard I need a pair of pliers sometimes to get them unplugged. Usually once my fingertips are cooked. Still some of the best routers out there. Firewalls and Aeronet radios, too. Just don't ever use the graphical config interface. It's junk. 
Karl Auerbach
@gsuberland - Perhaps its an editorial design element that reflects the designer's feelings about the device. I've had network devices and computers that ought to have come with an attached sledge hammer to make needed "adjustments".
Louis Mamakos
@gsuberland Back in the early Cisco days (mid 1990's), when they transitioned from AGS+ to 7000 boxes... The dimensions of the cabinet of the Cisco 7000 were precisely set to be the width of an open-frame 19" telco rack. With no margin for error. We had our techs out in the telco colos with car jacks trying to wedge these things into racks. You gotta leave just a little margin -- floors are not levels, the racks are... racked slightly, etc. It got better. We all learned a lot in those days.
Graham Sutherland / Polynomial
@lmamakos reminds me of the phrase "trying to land the plane and run out of fuel just as the wheels touch down"
Louis Mamakos
@gsuberland It's all in the details. Early hot-swap linecards in cisco routers were like that. It'd work just fine.. MOST of the time. But that 1 out 50 attempts, you'd slide in the board, all the indicator lights would flash, the major alarm buzzer would sound and you knew it was bad news. Don't even try to swap a line card with a different flavor board. Weird data structure problems you'd experence maybe hours later. It was all new; Cisco and their ISP customers were learning together.
Graham Sutherland / Polynomial
@lmamakos sounds about right tbh. I didn't join until 2015-ish when they acquired the company I worked for, but their initial foray into security services offerings was, uh, rather inauspicious. they just had no clue when it came to delivering those kinds of projects. took a lot of work from a few dedicated people to get it stumbling off the ground, several of whom ended up doing more than one job. sounds like they're doing better now though.  
Jeff Enderwick
@gsuberland Having worked adjacent to the switching BU(s) of Cisco for years I struggle to believe this made it out the door. Which switch is this? Yes, I though the 5800 was ridiculous.
preciousroy
Damned if I don't remember a co-worker doing that in production. And calling tac to find out why the switch wouldn't keep a config. 
skry
@gsuberland Something similar happened at Sun when a box was released with a new design: the CD drawer on the side. Turned out that if your box was in a rack, the drawer was inaccessible (all software was loaded that way, including the OS, because of bandwidth slowness and expense). AFAIK it was not recalled, but many were returned. https://ancientelectronics.wordpress.com/tag/sun-microsystems/
Graham Sutherland / Polynomial
@skry to be clear, in this context "recall" means "cause for return to manufacturer" rather than the kind of recall you'd do for a safety issue.
Graham Sutherland / Polynomial
@skry there's a proper internal term for it but I didn't work on that side of the business and I have the memory of a goldfish 
Joshua Small
@gsuberland The thing here was not that it happened, the thing was that it happened on kit that cost substantively more than most competitors, based on marketing itself as a premium product. I remember dealing with this at a time that affected switches cost 500% what the best alternative was.
Alan Campbell, Π‘Π»Π°Π²Π° Π£ΠΊΡΠ°ΡΠ½Ρ!
@gsuberland I recall a monitor used in ATMs I used to work on that had what I thought was a poor design that placed the video PCB under a metal cage. Said cage had several tabs, one tab went inwards and under the PCB instead of outwards. That made them a right pain to service.
Ralf Lenz, BOFH Emeritus π΄ββ οΈ
@gsuberland The 13 year old in me is really trying hard not to make a joke about messing up hole position.
UliBass
@gsuberland Hat Cisco eigentlich schonmal was gebaut, was sie dann nicht selbst unbenutzbar machen?! π
virkon
@gsuberland OMG *rotfl* There is NO way the V&V people (if they have them at all) have missed that. How ignorant can a company be? 
Toot Zi :mastodon:
@gsuberland I think the bigger design flaw is that the reset button is not embedded into the chassis. If itβs pointing out like that itβs possible to accidentally reset even if it requires some longer press. Simply working on the rack and leaning on the box would suffice.
Steinar Bang
Disclosure: Mine: The cover for the kitchen fan. Good to know I now can rest easy on that account!
Shy_sakura_
@gsuberland It is never a fault of one single person. One specified a requirement, one designed it, one built it, one tested it, one approved and released it. Which one was the one who didn't do the job properly?
Nicolai
@gsuberland lost revenue opportunity for the port 1 cable. Proprietary and patented ofcourse.
Graham Sutherland / Polynomial
@martinvermeer wasn't there an actual aircraft that did something like this? I forget what the actual toggle was (gear? fuel dump?) but it was right next to another commonly used control. 
Martin Vermeer FCD
@gsuberland I think this was the article that gave me the pic:
Chris Merle
@gsuberland I remember a computer lab at a college that has Macs and PCs. The Macs were PowerMacintosh 6100 series which had the power button next to the floppy drive. Students sitting down would stick their disk in the computer not realizing it wasnβt Windows until they got an error message it couldnβt read the disk. Macs didnβt have a disk eject button, so the students would push the power button turning it off. Theyβd turn it back on and the Mac would spit out their disk.
tto
@gsuberland not trying to defend an obvious design problem, but whoever plugs in that cable without realizing it presses that button should also rethink their career path
Π‘Π΅Ρ ΠΡΡΠ΅Ρ ΠΠΎΡΠ΄
@gsuberland I was the UK serviceman for the Triumph-Adler Alphatronic micro back in the early 80s. This was before IBM standardised the industry. A rather unfortunate design oversight placed the CPU reset key just above the enter key: it certainly encouraged accurate touch-typing!
Thomas Arend
Every designer should test their product and assemble it 10,000 times before it is sold.
Jason Scurtu
@gsuberland wow, love it.. could be a feature :) But canβt only blame the designer, it goes much deeper then that.
Apicultor π
@gsuberland Given that made it all the way through the design process and into production, it was not the engineer's fault.
elliot π©πͺ πͺπΊ π¬π§ π³οΈβπ
@gsuberland i think this says more about the product testing in place than about the engineer who designed it 
Dietrich Feist
@gsuberland Hey, that's a user problem! Why do they insist on sticking 3rd party products into these ports? π 
Ksapp
@gsuberland
Paul H
It's an attention to detail thing. For spectacular fails, look no further than and https://www.simscale.com/blog/nasa-mars-climate-orbiter-metric/ This second one I use as a case study on why we use an internationally agreed set of units in science and engineering, and what can go wrong if we mess units up...
masterdan
@gsuberland Itβs not the first time a switch goes in reset because of the mode button, no matter where is placed. Any good engineer with Cisco experience knows the trick: βno setup expressβ in the config and oh wonder, you can now press the button for weeks and nothing happens. |
Gregory's Server
@gsuberland wow.
What model was that?