Skip to main content

Small fire, no-one dead

As we attempt to provide round-the-clock IT services without employing out-of-hours staff, our worst nightmare for an "incident" is one that would affect the whole machine room, starting after everyone has left work on a Friday evening, with the whole weekend until normal work resumes on Monday morning.  It would be even worse if this were to occur just before the start of a new academic year.

So guess what happened last weekend?

One of the power supplies in our main machine room caught fire as a result of an electrical fault.  Although the fire was quickly contained, the emergency services shut off all power to the machine room as a precaution.  Several of our major services became unavailable and staff had to be called in over the weekend to fix everything.

The good news was that those  services that are designed to automatically fail-over to the backup machine room did so. Also, the support team had all our top priority services back up by six o'clock on the Saturday.  Our disaster recovery plan aims to have them restored within 24 hours so this was a good result.  (Technically, this wasn't a "disaster" in the terms of our DR plan because we were able to use the main site once the power was restored, but it seems to me that the result still stands).

Even though the overall result was not too shabby, there are a lot of things that our support teams will learn from this experience.  I'll be interested to see the results of the post-event analysis.

Comments

Post a Comment

Popular posts from this blog

Project Leadership

When I look back over recent projects that I've been involved in, it seems that one key to making a project successful is having someone on the team who really drives it forward: someone who is invested in the project as a whole and not just their own part in it. We (by which I mean the University's Applications division) have a well-defined project process, with defined roles, required milestones, deployment standards, and so forth.    All these are useful, but if the team doesn't have a leader, it seems a project can lose its way, perhaps not responding to changing circumstances, getting stuck on a technical problem, or not securing a needed resource in time to meet some external constraint. The leader can be any member of the team - it could be a developer, a project manager, or the sponsor, or someone in another role.  A team can include several people who are this committed to the project; it doesn't have to be a single person.   As an example, one of...
Post a Comment
Read more

Business Model Canvas

A Business Model Canvas is a tool for mapping the core functions and capabilities of an organisation.  Compared to the Core Diagrams that I described in an earlier post , the business model canvas attempts to present more aspects of the business, starting with the value proposition – a statement of what the organisation offers to its users (in the business world, to its customers).  It shows the activities and resources, as Core Diagrams do, but also shows user relationships & channels, and also benefits and costs.  I’m not aware of any universities that have used this tool but you can find examples from elsewhere on the web. We are considering business model canvases as a tool for mapping the strategic capabilities of units at the University of Edinburgh.  Phil Taylor, our EA contractor, sketched an outline of what a business model canvas might begin to look like for HR: This is only intended to be suggestive: the real canvas would need to result from in-de...
2 comments
Read more

"That's not Grid!" - A cautionary tale

I don't usually attempt humour here, but see if you like this... Once upon a time, in the far-off world of Computerland, a great guru arose and declared a vision. "I see a future when all computers will be linked together and people will run their programs without knowing which computers are running them. People do not need to know where their jobs run; they just need the results. I call this Grid Computing!" The people of Computerland were excited by the guru's vision. They went away and worked to make it happen. When they were ready, they returned to the guru and said: "Oh great guru! We have implemented your vision. We can run our programs on whichever processor is free at the time, making sure that all programs can run and making best use of all our processors. No longer does one computer sit idle while another one is overloaded. Your vision is a great success!" "No, no, no!", said the guru, "That's not Grid!" The guru exp...
3 comments
Read more