A sad thing happened earlier this week to the (d)vcs hosting service Code Spaces. An unauthorized person gained access to their Amazon controlpanel, and after an attempt to extort money out of Code Spaces to stop a DDoS on their services, (s)he started deleting their Amazon instances. Everything. At the point where the Code Spaces people were able to stop this from happening, nearly everything was gone already.

This hacker rendered Code Spaces dead. According to the statement published on the Code Spaces website:

Code Spaces will not be able to operate beyond this point, the cost of resolving this issue to date and the expected cost of refunding customers who have been left without the service they paid for will put Code Spaces in a irreversible position both financially and in terms of on going credibility.

This is extremely sad for all those working for Code Spaces. Obviously, it's not good for the Code Spaces customers, but they'll replace the Code Spaces services with something else. The people involved with Code Spaces itself just lost their job, their company, in the period of about 12 hours. That is horrible.

What can we learn?

At this point, I can only speculate about how this hacker gained access to the Code Spaces Amazon account. But however (s)he did, we can learn from this. The hacker did not start deleting instances until after it turned out Code Spaces was trying to regain access to the control panel. There were backup accounts created already, so even though the main account had been restored, the backup accounts were still there. When trying to recover from such a hack, always check the accounts section to make sure there are no backup accounts present.

The most important lesson...

The most important lesson is actually not one of security. It is one of redundancy. And it's not just a lesson we learn for our own infrastructure, it's actually one we need to learn for everything. For all of our software projects.

This time, it was Code Spaces, a relatively small player in the (d)VCS hosting market. Next time, it may be Github or Bitbucket. Sure, I am convinced something like this could not as easily happen to these, but let's just say this would've happened to Bitbucket. How would you have deployed a new version of your software? How would you have run your continuous integration? Every composer install or update would fail horribly, because we rely on the Github infrastructure for that. A huge part of the (PHP) software development would simply stall. I've seen it happen before when Github was down. And I've seen Github go down just as we were deploying an application. That is a horrible situation to be in. You surely don't want that.

We need more redundancy. If you run an open source project, it might be wise to publish it to more than one channel (perhaps use Bitbucket next to Github). Again, I don't expect something to happen to Github, but you may never know. Shit happens, and we better be prepared.

Perhaps Composer (or Packagist) needs to be changed for this as well. Although, perhaps not. Just yesterday Jordi blogged about Toran Proxy which, amongst other things, protects against GitHub outages and such.

Most importantly, we need to be aware. We take things for granted way too much. Things may change, services may disappear. We need to be prepared for this.