Distributed Thinking

As we attempt to provide round-the-clock IT services without employing out-of-hours staff, our worst nightmare for an "incident" is one that would affect the whole machine room, starting after everyone has left work on a Friday evening, with the whole weekend until normal work resumes on Monday morning.  It would be even worse if this were to occur just before the start of a new academic year. So guess what happened last weekend? One of the power supplies in our main machine room caught fire as a result of an electrical fault.  Although the fire was quickly contained, the emergency services shut off all power to the machine room as a precaution.  Several of our major services became unavailable and staff had to be called in over the weekend to fix everything. The good news was that those  services that are designed to automatically fail-over to the backup machine room did so. Also, the support team had all our top priority services back up by six o'clock on the...