I work for a hardware vendor and their customers are basically the same as Cisco's. So here is what happened with us - we had some hardcoded root password in the equipment during early development stages. And it was also known to some of the customers support staff for debugging. Then some of our tech leads raised this issue and discussed that we had to change the root password because it was a correct thing to do and don't wait for some breach in the field. After some usual months of bureaucracy someone did the push and actually changed the password in all versions.
Can you already guess what has happened next? :)
The biggest customer threw a gigantic hysterical fit and practically demanded to return the old root password which they knew. We just shrugged and complied. It's not like it will be ours network which can be breached. If they want to operate insecurely, they are welcome.
PS: just saying, no need to involve NSA in something that is better and more often explained by stupidity. :)
> The biggest customer threw a gigantic hysterical fit and practically demanded to return the old root password which they knew. We just shrugged and complied. It's not like it will be ours network which can be breached. If they want to operate insecurely, they are welcome.
Did a PM or sales engineer from your org ever sit down with the customer to figure out what they were doing with root access that your solution lacked?
My naive read here is that the customer saw it as easier to keep root access because your product didn't expose what they wanted or did it in a cumbersome way.
Not OP, but over here, PM and sales usually understand that what's important to me and our other engineers (building a really great machine, with a well-architected, secure, extensible backend) is completely immaterial to my end customer.
They're trying to build a car or a chair or an appliance. To them, anything that gets in the way of that goal is bad, anything that supports it is good. Sure, the customer's engineering department had a grand vision of skilled shift leads empowered to look up the fault diagnostics, call over a QC engineer, and get two-party authorization to ignore the fault and continue...but once it hit the floor, and getting QC over to the machine took a whopping 4 minutes, the operators all memorized their lead's and QC's passwords and were rewarded for cutting TAKT times by 3 minutes and 50 seconds. Yes, bad product containment is a huge deal in the boardroom, but there's almost no will to enforce it on the production floor.
They were providing root either way, the only change I assume was switching from the same root on all machines to a per-machine root password.
I can understand how that would be a pain in the ass if your legacy architecture (e.g. administration and reporting tooling, etc…) is predicated upon a unified root, you probably don’t even have a way to report the information so that machines and passwords could be paired.
Assuming a good way to provision keys is provided, I’d make the insecure unified root password into a paid option (with lots of warnings) the price of which would increase every year. This way the existing workflow of the customer keeps working short-term, but you CY the hell out of your A in case of breach at the customer’s, and at one point the line item will get large enough that the customer allocates time & money to fix their shit.
Charging for insecure systems incentivizes selling those same systems to customers.
Soon you’ll have sales guys regaling how this unified root password system is extremely convenient and you’re one of the few companies who provides this single-signature-security-system (S4 compliance). Next thing you know someone will gain an software patent on this to ensure no one else can provide it.
The solution is trivial: no commission on this. Possibly even negative commission, can't convince your client not to take this? You get less money for the sale.
This way sales are incentivised to only talk about this if not doing so would lose them the sale entirely.
For b2b or b2c your customers don’t want to sit with you to make your product better.
Yes they want your product better.
Myself as a customer I don’t spend time filling in customer reviews. I just want to go on with my life.
I see the same in company I work for, our customers don’t have time - they have stuff to be done - either our software helps them achieve their goals or they leave.
Yeah, they did want to maintain root access because they had started to rely on it, I'm not very familiar with exact details. We have an ongoing process of exposing different debug info in the restricted CLI, but it is a never ending task so they always want something which is only visible to the root.
Oh, I'm pretty sure we can just say whatever, even "we have decided and it will be so". There is only a tiny problem when a customer is bigger (and richer) than the vendor by an order of magnitude or more :) . They tend to control a lot, with such power imbalance.
Honestly doesn't get better in the reverse either. At Microsoft I'd sit in on calls with banks and other "big names", watching them pound the table and argue their way, usually involving something dangerous and borderline malicious use of the product, all the while thinking "we could buy your company with our cash on hand".
I think the key was that they're used to being the big dog in the room, and Microsoft made a point of not correcting them when that behavior was seen.
Do you mean can't be changed on a live system? It's very easy to change a hard-coded password, just recompile the source and flash. That could definitely be what OP is referring to if they pushed OTA firmware upgrades.
There is nothing as permanent as a temporary fix...and worse people will become dependent on the specific inner workings of the temporary fix. Once given...
Or instead of changing out of the blue what exists today and is probably soaked into whole following ecosystem around the product and would be massive PITA to change it as well, you could have create parallel way for login (i.e. via certificate) and tell customer that password access will be deprecated in 1-2 years, so they have time to update their tooling.
Can confirm the hardware vendor that I work for does the same. root password is given and customers are asked to change. No one wants to. Heck I would prefer if they disabled root login completely and used keys instead. But thats now it is. Some customers even enable login from web with those passwords. The stupidity is crazy.
Would be better to have such default creds parameterized by your build system. So every dev can choose theirs and you can choose unique ones per customer.
Do I understand this correctly? Are you saying you removed that root password, such that it wasn't possible to log in with another (but different) root password?
Or are you saying that you merely changed it, to one the customer no longer knew, but was the same password across all of your install base, for every customer?
It was at that moment a simple change of the password. The benefit was that people outside of our company wouldn't have access (thus not sharing it further), and then as a future improvement we could have implemented some better key scheme. But while customers want to have root access it is not an option.
Theoretically we could implement a scheme with individual passwords or keys per device (as someone suggested in the comments above), but such thing is not being prioritized by our company, so that's our fault.
What’s their complaint? Do they have a workflow that requires the root password all the time and it creates too much of a management to not have QWERTY1234 as root password?
Our lead developers tried, and they are way smarter than me). No luck, for some reason customers don't consider this a vulnerability. Even ones who take security more seriously and do some external pentesting on our equipment.
Smart people explain things with big mental steps they deem obvious (true or not), and less smart people disagree with these steps but are unable to point the finger on them.
In this case, only one version out of three (the middle one) had this vuln, so it's very likely it was introduced by mistake. But your scenario happens all the time too.
I hope you guys got an indemnification agreement or something else that make you not liable when the inevitable happens but, knowing how these things work, I doubt it.
Daydream: A new law puts legal liability for the consequences of hard-coded passwords on the manufacturers and their distribution chains. (The latter so Amazon can't sell root-holed crap from Lucky Good Fast Router Co. this week, then that company vanishes and Amazon sells amazingly similar crap from Good Lucky Fast Router Co. next week, then that company vanishes and ...)
Not going to happen. Most online services require applications that access them to embed hard-coded, obfuscated passwords in the binaries. That doesn't really work for open-source software, and Thunderbird's hard-coded passwords are easily viewed:
I don't quite understand how we ended up in this situation. These passwords are more like user-agent strings. But they are still passwords in the sense that without any of them, you can't access (the authentication providers for) those online services.
Possession of OAuth Client Secrets doesn't grant you access to any private data, it's a non-issue and it's unrelated to the linked Cisco vulnerability:
> A vulnerability in Cisco Emergency Responder could allow an unauthenticated, remote attacker to log in to an affected device using the root account, which has default, static credentials that cannot be changed or deleted.
I understand these cases are structurally different. I just don't see how a quasi-ban of hardcoded passwords could work, given current industry practices.
And I'm not sure these client-side secrets/group passwords are entirely meaningless. Some people somewhere must ascribe some security-related property to them.
I think this is just a case of the APIs here supporting client ids and secrets so that users can control who uses the API on their behalf… but it being up to the API user to decide how much they care about who they share that client secret with.
If Thunderbird shared their Netflix password in here, that wouldn’t say anything about the quality of Netflix’s security implementation or the security value of Netflix passwords in general - it tells you Thunderbird don’t care about how their Netflix account gets used.
And in terms of the structural difference - this is of course a hardcoded credential used to identify this client to a service - the case at hand is about a hardcoded credential authentication embedded in a service, used to authenticate clients. We can distinguish those cases regulatorily.
> I don't quite understand how we ended up in this situation. These passwords are more like user-agent strings. But they are still passwords in the sense that without any of them, you can't access (the authentication providers for) those online services.
Blame anti-cybercrime laws and weird interpretations of these for that one. The idea was that, as application level secrets (client_id/client_secret) in OAuth are scoped to a specific legal entity requesting them, site operators like Facebook, Twitter and others could request a takedown of third party apps at app stores for using the secrets of other apps.
> root-holed crap from Lucky Good Fast Router Co. this week, then that company vanishes and Amazon sells amazingly similar crap from Good Lucky Fast Router Co.
Even cheap routers start putting passwords on them that look randomized.
Look, it shouldn't be that hard. One would expect that since manufacturers surely can produce unique serial numbers and MAC addresses and put them both into hardware and on the sticker on the bottom, a unique pair of credentials should be a piece of cake. Right? Right?
But this was a dev account that ended up in prod, not an account you should use as it shouldn't exist. The device has a set up wizard where you set up your real account, like most Cisco products.
Cisco is a very large company, that mostly acquires existing products and employees. They do make an effort to train new hires on their SDLC and security practices. But at the end of the day, security is up to the individual product development team. There's lots of those teams, and most of them are clueless about security. So it's honestly kind of a miracle when there isn't a big security vulnerability in any given product.
I've worked for a lot of large enterprises, and they all are the same. Nothing special about Cisco or any others. Security is hard, and it does not provide a good return to invest in good security. You can stop using hardcoded passwords, but it won't make you any more money, and customers clearly aren't leaving because of them. The stock price is the only thing that matters. As long as that's safe, it's business as usual.
Another thing to consider is how many development resources a given product/team has available. Emergency Responder is fundamentally a supporting product[0] to their PBX software. For the organizations that need it, it is something they have to have, but nobody that buys PBXes[1] picks a PBX on the basis of the security of the E911 add on package.
Now, if you're a big vendor, do you give more resources to the PBX team that builds features which sells PBX customers, or the add on team used by a fraction of a percent of the worldwide PBX install base? I'm not asking what you should do, but rather what your inner MBA would do...
0 - the fundamental purposes of Emergency Responder are to a) provide enhanced location information about a caller, say the general location within a building, and b) to allow emergency responders to call back the person who placed an emergency call, even if the phone they called from does not have a directly callable phone number. It does some other things, too, but those are the "primary" features.
Cisco, to it's credit, does a good job of internal quality auditing, so bad behavior of these individual teams gets caught and corrected. But there is just so much bad behavior...
Sadly, it's true. There are a lot of rebranded products with completely different stacks. And some started as pretty gross, thrown together startups.
If you wanted to make some juice at the cost of a lot of squeeze, legacy support is always needing self-abusive people who can keep fragile crap going.
Not only do they have standard security practices and training programs, they have one of the industry's best known security research teams. rest of comment redacted due to too much honesty/whisky
I do get why they have hardcoded passwords in place: to save people who have misplaced their credentials (or SSO config) from having to wipe their devices and start anew, because that's what a lot of the support calls will be.
What I don't get is why these passwords are baked into the firmware instead of a "split secret" scheme - basically, a user in trouble would read some sort of identifier from a label on the inside of the hardware in question to the support, who'd then use a secret on their side together with the identifier from the customer to generate the backdoor password.
That way they'd avoid someone brute-forcing or leaking the backdoor password and compromising fleets of devices in the field, while still creating a paper trail of who has access to that password.
That would require someone with physical access to the device. It's not a home network, the person responsible for fixing it could be in another country, and you could have hundreds of devices to fix.
A company could scan this as part of the onboarding process, and even if the network admins are in another country, in 99% of cases there's some on-site IT support staff that could read it out for you in an emergency.
A lot gets made of the inflation of CVSS scores but this is an interesting low ball of a score. It's a bit of a bold move to mark this one as a 9.8. If root access available to remote, unauthenticated users via the simplest means possible, unrevokable creds, isn't a perfect 10 then I don't know what is. I tried to find the mitigating factor that pushes it down that 0.2 but couldn't.
When I was at Cisco (over 10 years ago), one of the clauses in the PSB (Product Security Baseline) was "No default credentials". Back then, most of the issues arose in products that were produced by companies that Cisco subsequently purchased. I can't find the PSB that I was familiar with, but this is a variant that Cisco uses for Linux: https://www.cisco.com/c/dam/en_us/about/security/intelligenc...
Cisco isn't perfect by any stretch of the imagination, but the corporation cares about security even if some of the individual employees don't care to the same extent.
Imagine if this were found in a Huawei product. The warpigs would be calling this a "CCP Backdoor" and proof that Chinese networking products are intentionally insecure and should not be allowed on your network on one end, and calling for an all out cyberwar on the other.
I kinda hoped, before clicking the link, to find some reasoning (compatibility blablabla, poor Cisco not having resources to make it right, etc, any reason at all, doesn't have to be a good one) behind this, but was disappointed that it's just another instance of business as usual.
Field Technicians/Engineers love them. It's so much easier to use a never changing default at a new site under construction/finalization.
Compared to other manufacturers that have default passwords printed on the device or something made from the serial number. When that hardware could be shoved in behind a ceiling panel somewhere (even chances of that hardware not being where it's supposed to be).
_____
It's clearly not the "right answer", but it is an answer.
The funny thing is that Cisco doesn't use this for support; to get remote support from them there is a user-initiated challenge-response workflow that authorizes Cisco TAC to connect to a box.
Conspiracy people always gonna conspiracy though, and sometimes they're right, and whether it's an intentional backdoor or not, it is a backdoor.
Both. Neither. It might not be an intention of the management but it surely is a consequence of their outsourcing decisions.
There is a team working for Cisco who does not care for quality who decided it is a good idea to have a baked in password and who do not learn from past mistakes.
The core of the problem is outsourcing your core product to people who are neither competent nor interested in the quality of the product.
Every time I see any company thinking they can push their core responsibility to somebody else who's interests are not aligned, I can confidently say it is going to go downhill and fail at some point.
Regardless of outsourcing: you have to do at least some inbound QA and some checks on your product's firmware. Especially if you have had that exact bug before. But the paranoid part of me simply wonders whether this is so they can claim plausible deniability and 'accident' rather than malice.
You do outsourcing for one and one reason only: you need to throw warm bodies onto a problem for cheap. Everything else is expensive, and any other reason is sugar coating. Having inbound QA? Expensive! Checks on firmware? Expensive! Exhaustive testing before shipping firmware so that edge cases are cared for before the machine is in the field? Expensive! Get rid of it all! Just look at our bottom line grow.
You know what else is expensive? You need to troubleshoot some gear but you can't get in easily. Sophisticated procedures for getting into a router take time and skill to implement and to use, so they are, yes, you guessed it right: Expensive! What is not expensive? Having a surefire set of shared credentials that will work on bloody anything, that's what!
Short term cheap, but long term very expensive. I stopped using CISCO after the first time, but I will never know how my current vendor deals with these things either because it's not as if this gear comes with full source code attached and even if it did I wouldn't have the resources to audit it.
But inside CISCO even the most basic code review should have caught this, besides that even for test purposes they should have never ever implemented this.
Inside the MBA ideology, that "long term very expensive" doesn't exist. Lowering costs is good long term, because it grows the market and gives the company more money to invest into competitive advantage and grow its brand.
I don't care if it is a prevalent ideology. If the ideology calls for providing poor service to the customer, somebody will find a way to provide better service and will boot you out of the market entirely.
Getting your company disrupted or getting your market share diminished is extremely expensive and this is my understanding of what Jacques meant.
That sounds nice but I don't think it actually happens. Shittier but cheaper service seems to always fare better than the better but more expensive one.
Fully disagree, and especially with "always". Fortunately, it is trivial to show it is false.
What kind of shiny rectangle do you have in your pocket. Is it the cheapest option? Which company in that market makes the most expensive product? Which company in the market fares best?
And Cisco itself, for decades, was not only the most expensive network equipment in multiple markets but also bringing highest margins and biggest bucks.
> You do outsourcing for one and one reason only: you need to throw warm bodies onto a problem for cheap. Everything else is expensive, and any other reason is sugar coating.
Do people who post these silly hyperboles sincerely believe them? There’s plenty of other reasons to outsource something, including many where it’s known from the outset that it would be the more expensive option.
I don't have anything against outsourcing. There is plenty of reasons to outsource, true. But there is one that is most important and IT IS NOT THE COST. I am not saying the cost is not important, I am just saying it is not the most important.
The most important reason is that you want to focus on your core product and not on everything else. If your core product that you sell to your customers are routers, if you are the CEO of the company, you probably do not want to oversee a division that produces accounting software or remote desktop software. If you need these things, you either buy them or outsource them.
You want to spend all your focus producing the best product for your customers, and this is routers. Not anything else that does not directly go into making better experience for your costumers.
It usually is or at least, in principle, should be cheaper. Because for some reason if somebody cares for the product, the total cost to produce the product at a given quality level will be lower.
But then your core product is the routers, and you outsource their firmware, you also outsource a very good chunk of hardware they are made of, and this is where what you're saying kind of falls apart.
Because from the looks of it, if I assume what you're saying is 100% true, the product is sales contracts for the routers and not the routers themselves.
I wouldn't know if it is malice. We won't know unless somebody catches them red handed with an email to prove it.
But I don't think it is malice. The engineer in me says that there are just too many eyes looking at the software and that it is just too easy for somebody to identify a password and that the consequences of it are too costly for the company.
And I can think of a zillion ways to leave a backdoor that is much more difficult to detect.
Of course, this does not absolve them from responsibility in any way. The core responsibility, the core product they provide is security. And not only they are showing they can't provide a secure product, they prove they simply do not learn and do not care to fix it.
Lots of companies thrive without trying to be cheapest in the market. In most markets there is "expensive", but quality option and then there is a cheap option that everybody knows may be less quality. Companies make these choices all the time and those more expensive options are still successful.
And Cisco is actually positioned as an expensive option. For decades in many segments of the market, Cisco was the top option that you had to pay for through the nose.
If they wanted to add a backdoor for feds there are better ways to do it. This is just stupid, lazy engineering. Not everything has to be some grand conspiracy.
I dunno if its still the case but stratasys 3d printers used to have a hard coded root password. I had cause to crack it in order to repair something and did so with a couple gpu hours.
Doesn't Cisco have a Vice President Of Security In Charge Of Knowing His Ass From A Hole In The Ground who they can ask for his professional opinion of whether or not it's a brilliant idea to ship products with hard-coded passwords, before they ship products with hard-coded passwords AGAIN and AGAIN and AGAIN?
Ok, so they chat to the C-suite about what to do. The result will probably be "saying our security is flawed will make us look weak and undermine our marketing, just leave it". Then VP will either leave, because they have integrity; or stay and tell themselves they tried whilst enjoying their large financial compensation. Result, nothing changes.
Market forces just don't seem to work on these things.
I don't know what the answer is but putting the fate of millions of customers in the hands of one person's conscience, because of a knowledge imbalance, just doesn't seem right.
I can imagine a technocracy having a licensing scheme for suppliers, where Cisco would be told, this is substandard we'll pull your license. But, that sorry if structure just seems like it would fail to corruption so easily.
How do we improve standards over small, but impactful, parts of life that most politicians wouldn't understand; things that the market doesn't begin to address?
Like the fable of the Dog and the Wolf, do we need to forgo some liberty to acquire the comfort of progress?
The problem is sloppy processes lacking protected networks for staging and deployment that properly configures said network gear with a unique password or TACACS+ auth.
I humbly disagree. However, this is much worse, anyway - it's a static root password:
> A vulnerability in Cisco Emergency Responder could allow an unauthenticated, remote attacker to log in to an affected device using the root account, which has default, static credentials that cannot be changed or deleted.
> This vulnerability is due to the presence of static user credentials for the root account that are typically reserved for use during development.
You're mischaracterizing what I was talking about: I was talking about the general case of a hard-reset default password that is user-configurable.
I don't care about Cisco, so they can continue footgunning themselves in obviously stupid ways. It sounds like such a Cisco user should or must have an ability to be disabled. If not, that's fucking moronic. But really, what kind of morons put what should be their internal management network on the fucking internet and don't have TACACS+?
> You're mischaracterizing what I was talking about
I didn't mean to imply you was talking about something else - I tried to make two separate points:
1) I don't think default passwords are "no problem"
2) This Cisco thing is anyway much worse than that.
As for what you bring up:
3) But really, what kind of morons put what should be their internal management network on the fucking internet and don't have TACACS+?
Indeed. Sadly 3) doesn't surprise me; it makes both 1 and 2 much worse (1 because the people that give us 3 aren't likely to change the default - even if they could - and with 2 they can't).
Yup. For example Symantec's "A tough day as leaders" blog post (of course no longer available shortly after they wrote it but it's on the various archives) is basically this formula. "Oh dear, a bad thing happened, we'll fire a junior employee but change nothing".
Can you already guess what has happened next? :)
The biggest customer threw a gigantic hysterical fit and practically demanded to return the old root password which they knew. We just shrugged and complied. It's not like it will be ours network which can be breached. If they want to operate insecurely, they are welcome.
PS: just saying, no need to involve NSA in something that is better and more often explained by stupidity. :)