Skip to main content

Grids, data centres and reliability

In my work with the Grid Computing Now! Knowledge Transfer Network, I talk about "virtualisation" and "service-oriented architecture" just as much as "grid" itself. People sometimes ask what is the difference between these concepts. My first answer is perhaps rather glib - I say that I don't care as long as the technology gets the job done. Although this is not a straight answer, those of us on the GCN! team believe it is important to put the business answers before any notion of technological purity.

But if we turn to the question as stated, I think that as long as a solution includes the key concepts of virtualised resources and dynamic allocation of applications across those resource, then that to me is enough to call the system a grid. But, of course, we can go further.
A recent conversation reminded me of the important point that distributed systems typically have to manage failure. As systems scale to many machines and many sites, then some of those are going to fail some of the time. The systems have to be resilient enough to adapt and recover. Systems also have to cope with additions and deletions from the set of available resources.

This is most obvious in cycle-stealing grids, which use spare power of desktop PCs, and of scientific grids, which link many research sites across the world. The interesting question is whether this also applies to the data centre. That seems to depend to some extent on how the system is designed. For example, Google is built specifically around this approach; they have always used lots of generic systems and just replaced resources when they fail. I believe Ebay's massive server farms use the same dynamic approach.

This question arose in a conversation I had with Liam Newcombe, an independent consultant. We were supposed to be talking about Green IT (of which more another time), but our discussion wandered to include all sorts of ideas. Liam is working on an open source model of data centre reliability and performance. He believes that reliability is best achieved by adopting this approach of explicitly allowing for it within the software - rather than, for example, attempting to make the hardware itself ultra-reliable.

A key question must be how high up the stack does this awareness have to extend? Can we write applications without worrying about this or does every application have to have some potential adaptability built in? It's a fascinating topic and I look forward to reading the book the Liam is co-authoring, in due course.

Comments

Popular posts from this blog

Project Leadership

When I look back over recent projects that I've been involved in, it seems that one key to making a project successful is having someone on the team who really drives it forward: someone who is invested in the project as a whole and not just their own part in it. We (by which I mean the University's Applications division) have a well-defined project process, with defined roles, required milestones, deployment standards, and so forth.    All these are useful, but if the team doesn't have a leader, it seems a project can lose its way, perhaps not responding to changing circumstances, getting stuck on a technical problem, or not securing a needed resource in time to meet some external constraint. The leader can be any member of the team - it could be a developer, a project manager, or the sponsor, or someone in another role.  A team can include several people who are this committed to the project; it doesn't have to be a single person.   As an example, one of...

Business Model Canvas

A Business Model Canvas is a tool for mapping the core functions and capabilities of an organisation.  Compared to the Core Diagrams that I described in an earlier post , the business model canvas attempts to present more aspects of the business, starting with the value proposition – a statement of what the organisation offers to its users (in the business world, to its customers).  It shows the activities and resources, as Core Diagrams do, but also shows user relationships & channels, and also benefits and costs.  I’m not aware of any universities that have used this tool but you can find examples from elsewhere on the web. We are considering business model canvases as a tool for mapping the strategic capabilities of units at the University of Edinburgh.  Phil Taylor, our EA contractor, sketched an outline of what a business model canvas might begin to look like for HR: This is only intended to be suggestive: the real canvas would need to result from in-de...

"That's not Grid!" - A cautionary tale

I don't usually attempt humour here, but see if you like this... Once upon a time, in the far-off world of Computerland, a great guru arose and declared a vision. "I see a future when all computers will be linked together and people will run their programs without knowing which computers are running them. People do not need to know where their jobs run; they just need the results. I call this Grid Computing!" The people of Computerland were excited by the guru's vision. They went away and worked to make it happen. When they were ready, they returned to the guru and said: "Oh great guru! We have implemented your vision. We can run our programs on whichever processor is free at the time, making sure that all programs can run and making best use of all our processors. No longer does one computer sit idle while another one is overloaded. Your vision is a great success!" "No, no, no!", said the guru, "That's not Grid!" The guru exp...