First, my apologies for the outage. I consider our package infrastructure as critical infrastructure, both for the free and commercial versions of Docker. It's true that we offer better support for the commercial version (it's one if its features), but that should not apply to fundamental things like being able to download your packages.
The team is working on the issue and will continue to give updates here. We are taking this seriously.
Some of you pointed out that the response time and use of communication channels seem inadequate, for example the @dockerststus bot has not mentioned the issue when it was detected. I share the opinion but I don't know the full story yet; the post-mortem will tell us for sure what went wrong. At the moment the team is focusing on fixing the issue and I don't want to distract them from that.
Once the post-mortem identifies what went wrong, we will take appropriate corrective action. I suspect part of it will be better coordination between core engineers and infrastructure engineers (2 distinct groups within Docker).
> At the moment the team is focusing on fixing the issue
> and I don't want to distract them from that.
That might be ok for feature teams, but for infrastructure tools/services, it's very frustrating for users (devs) to be kept in the dark on the progress of the fix.
At work, the incident response starts with identifying Investigators (to find and fix the problem) and a Communicator (to update channel topics, send the outage email & periodic updates, field first-line questions about the incident, and to contact those most affected by the incident so they don't get surprised/try to fix it themselves). The person who starts the incident is the Coordinator, who assigns the roles, escalates if more help is needed, tries to unblock investigations, and turns facts from the investigators into status updates for the communicator.
i will provide an opposing viewpoint which i'm sure many people do not agree with.
if a service i use is down, all i want is an acknowledgement and that "we are working on it right now with high priority". i want all available resources to be fixing the problem.
my anxiety over powerlessness in relying on others during a crisis manifests in other ways, like figuring out why i'm at the mercy of this thing in the first place, and putting alternatives in place.
but during the crisis i'll just go do something else for an hour and then read the post mortem when it comes out.
Updates that are communicated are not the details of what's happening in the investigation, but things that are expected to be useful to users/clients. They are things such as the estimated time for problem resolution, updates on the scope of the problem (e.g. "this is an instrumentation problem" vs "this is an actual outage") or mitigation steps that could be applied by the clients. It is often very useful to know such things during an outage.
i don't think anyone, anywhere would have given you an accurate estimated figure of nearly 5 hours to fix this problem.
furthermore, even if you somehow could divine the future, telling the customer that you think an outage will last over half a working day is going to turn an extremely shitty situation into something even worse.
sometimes the resources available are suitable for different problems.
I consider high quality communicators to be a significant resource, and the problem to solve is to make sure everyone affected has the right information to make their next decision to mitigate the issue.
These communicators might not know how to solve the actual technical problem, and it'd be like putting monkeys on a typewriter to tell them to "all hands in fixing a configuration issue with the web server" for example.
I'd say apply the right people to the right problems.
I agree with you and we have a similar process at Docker. Part of what went wrong in this particular case is precisely that our infrastructure incident response process did not kick in. I am guessing that one conclusion of the post-mortem will be "make sure to handle open-source package distribution infrastructure like the rest of our infrastructure, including incident response checklist".
I think this is fine.
If you use a 3rd party repository, you can almost expect it to fail. If you rely on repositories so heavily, run Apt Cacher or mirror them on s3 or the like, its pretty simple.
This is a pretty common practice in any professional environment to contain all your dependencies - binaries, debs etc locally mirrored...
Can't wait for dockerhub to go down now... :troll:
This isn't doing devops, this is release management, which is something that traditional sysadmins do all the time. Making sure that the image file and repository information match up is pretty basic, and that a release is deployed correctly is pretty basic. A project like this, I'm surprised they don't have tools like nagios constantly checking to make sure downloads are working and that checksums, etc all match up, preferably on the servers before whatever load balancing system you have points at them. Deployment can, and should, be as atomic and possible, regardless of who is pushing the go button.
Docker's messaging is specifically anti-devops. They market containers as a "separation of concerns" between dev and ops, where dev is responsible for everything that's in the container, and ops is responsible for deploying the black box.
It's true that Docker helps separate dev concerns from ops concerns. But it doesn't prevent dev and ops teams from collaborating, or the same team to wear both hats - the most common aspects of a "devops" methodology in my experience.
In fact separation of concern makes collaboration more efficient, because everybody knows who is responsible for what. So you could argue that Docker actually facilitates "doing devops" if that's the methodology you choose.
You make a good point that "doing devops" is a methodology that you can choose to use or not. There is no moral hazard in not using it.
That said, having ops excluded from the architecture decisions that go into building containers is absolutely antithetical to "doing devops."
As for your assertion that people know who is responsible for what in your model, I argue that devs are usually not thinking ahead that they're going to be the ones on-call to fix whatever breaks in production in the middle of the night, because they're the only ones who know what's inside the container. Ops can't be responsible for fixing whatever is inside an artifact that it had no role in creating.
We're going to to have the inverse of the little-girl-smiling-at-the-house-fire meme. "Kubernetes is Running Fine, Dev Problem Now."
> That said, having ops excluded from the architecture decisions that go into building containers is absolutely antithetical to "doing devops".
My point precisely. Just because there is clean separation of concerns doesn't mean anyone needs to be excluded.
The methodology I've seen work best is one where people are not divided by skillset (dev/ops) but instead by area of responsibility (app/infrastructure). Then you embed people from different functional areas into app teams: devs of course, but also security engineer, operations specialist, and various domain experts. From the point of view of IT, you're influencing the development of the app before development to make sure it follows best practices.
A second important point is that, just because you're running a container built by someone else doesn't mean you can't enforce good operations practices. For example, you can mandate that all the containers on your production swarm expose health information at a specific url prefix and pass CVE scanning - or they will not be deployed.
DevOps isn't (just) Dev doing Ops and Ops doing dev, though. It's about understanding each teams domain and facilitating communication. Nothing about Docker limits that, per se.
e.g:
> I argue that devs are usually not thinking ahead that they're going to be the ones on-call to fix whatever breaks in production in the middle of the night
That's not a Docker issue. That's just a DevOps culture that is incomplete.
Maybe you are being downvoted because your comment is too short, but it's something that crossed my mind when reading the end of the parent comment.
The best thing is, this might end up being the best proof of why you need to embrace devops methodologies and maybe take advantage of tools like Docker while doing so :)
I am copying it below:
<<< Hi everyone. I work at Docker.
First, my apologies for the outage. I consider our package infrastructure as critical infrastructure, both for the free and commercial versions of Docker. It's true that we offer better support for the commercial version (it's one if its features), but that should not apply to fundamental things like being able to download your packages.
The team is working on the issue and will continue to give updates here. We are taking this seriously.
Some of you pointed out that the response time and use of communication channels seem inadequate, for example the @dockerststus bot has not mentioned the issue when it was detected. I share the opinion but I don't know the full story yet; the post-mortem will tell us for sure what went wrong. At the moment the team is focusing on fixing the issue and I don't want to distract them from that.
Once the post-mortem identifies what went wrong, we will take appropriate corrective action. I suspect part of it will be better coordination between core engineers and infrastructure engineers (2 distinct groups within Docker).
Thanks and sorry again for the inconvenience. >>>