Distributed Thinking

The secret of good estimation

2012-02-05T22:57:00.001Z

At this time of year, the various departments of the university are planning their activities and budgets for next year. Applications Division provide the IT component of many new projects, so we are asked to estimate how much time and money will be required or a wide range of proposals. Most of these proposals are at preliminary stages, with only outline ideas of what will be required. Given this uncertainty, You might ask what is the secret that lets us produce accurate estimates for all these proposals?

The truth is: we don't. Of course, we look at our project record to see how much effort was required for similar projects in the past. This data gives us a guide to the costs of different types of project, such as software procurements, infrastructure upgrades, and software development. The data also helps us allocate the expected effort across the different teams that will be involved. But we often don't know exactly what a project will entail and there is usually a large element of uncertainty.

The key is that these estimates should be the start of a conversation, rather than the end of a process. Although they give people a sense of the likely scale of a project and help us plan our future work, they cannot be set in stone.

For most projects, the work of the project is an ongoing process of knowledge elicitation. Everyone involved will be aiming to improve their understanding of what the IT system needs to achieve and how it will help people achieve their actual goals. New features may be proposed, previous assumptions may be overturned, and technical problems may occur. Often the project team will face a choice: either increase the budget to achieve more features, or reduce the project scope to fit the budget.

So the budget that accompanies an initial proposal is always provisional, always contingent. When the project starts, so does the process of refining the estimate as we learn more about the special attributes of that particular project.

Time out: co-mentoring

2012-01-22T18:29:00.001Z

A year ago I completed a year-long course run by the Leadership Foundation for Higher Education. This course brought together 18 IT and library managers from universities across the UK and Ireland, made us look at who we are, what we value and how we work, and then gave us lots of pointers for how to lead our teams and projects more effectively.

This week we all gathered for a reunion and for meetings of our "action learning sets". These sets are groups who help each other with problems we encounter in our jobs. Sometimes these are practical problems; sometimes they are about relationships with other people; sometimes we just ask the others to help us understand our own thoughts and feelings.

I'm blogging about this because everyone in my action learning set has found it hugely valuable to have someone we can talk to who will give us their honest response, in an atmosphere of trust. The fact that we are all from different institutions mean that our reactions are not tinged by concerns about how we will interact outside the group. This trust allows us to challenge each other as well as to provide mutual support: we don't get an easy ride from the other members. Our meetings tend to be both intense and exhilirating.

I've found it very worthwhile to have someone I can talk to like this, outside of work and family. Several other people have also told me that they have benefited from mentors of some form or another. So if an opportunity arises for anyone reading this blog to get this sort of advice and feedback, I'd definitely recommend that you give it a go.

Preparing for a post-PC world?

2012-01-15T15:13:00.000Z

Martin Hamilton has posted a thoughtful piece about the impact of the current IT trends away from the desktop PC and towards cloud computing. I've been musing about how these changes might affect us as service providers.

The move towards staff and students using their own devices to access our services is well established. We have a policy of providing services via the web whenever we can, giving people access to our services "anywhere, anytime". The EASE system provides a single authentication point for common access and is integrated with Shibboleth as well. There are still university systems that don't fit this model, perhaps because the vendor doesn't provide the option or because a university department has bought their own system without considering the wider picture, but we are moving in the right direction.

In some cases, the move towards outsourcing services can make this harder. The big players all have their own authentication systems and want users to move into their world. It isn't necessarily in the vendors' interest to have interoperable authentication, just as vendors of desktop systems before didn't necessarily want interoperable systems. This is something we will have to keep pressing for. Universities should keep working together on this - as we have with Shibboleth adoption.

Obviously a key aspect of the post-PC world is the expansion in the range of devices that people are using - phones, tablets, laptops, consoles, thin-clients, and so forth. So far, we have the U@Ed service for smartphones and provide mail/diary services on phones too. It seems to me that this is an area where we are likely to become more dependent on vendors, simply because of the support implications. We simply can't test systems on all the different devices, with their different browsers and screen sizes, let alone different "apps". We can write systems to comply with standards, but standards are usually open to some interpretation. If one of our systems doesn't work on a certain device, the best we can do is address problems as they arise, if we have the budget to do so.

So that's one side of the post-PC world: far fewer users using (or wanting) traditional PCs. The other side is the use of cloud computing. We already outsource some of our systems and we are looking at this option for others. Some people tell us that all our services will end up being provided in the cloud. I'm sure that many will, but it's interesting to compare this with the current situation where we procure a lot of applications to run locally.

Even though we are using more third-party systems than before (either from vendors or open-source projects), we are still asked to build local systems, or to substantially reconfigure third-party systems. It seems that the market does not address all the needs of a complex university (or in some cases, not at an affordable price). It will be interesting to see whether cloud-based commercial systems will address more of our needs, or whether the scale required will mean even more space for local niche applications.

Basically, when it comes to providing services, we are cloud providers, as far as our users are concerned. They connect to our services via the web just as they do with commercial services such as Google or Microsoft. In some cases, we know that we can provide these services cheaper than the commercial options. So our decision will be whether to run our "cloud" services in-house, to outsource them (while still branding them as university services), or to stop providing them and let our staff and students use the commercial ones. Factors affecting this will include data protection considerations, freedom of information obligations, branding, consistency of user experience, support for multiple devices, and of course cost.

Martin's post: http://blog.martinh.net/2011/11/post-pc-dark-clouds-and-silver-linings.html

Reboot: three years on

2012-01-09T17:54:00.002Z

I've decided to revive this blog, after a three-year hiatus, to post news and thoughts about my current role as Head of Development Services, in the IS Applications Division, at the University of Edinburgh. My aim is to give some visibility to what goes on "behind the scenes" (or should that be "behind the screens"?) and to reveal the sort of issues that face IT in a Russell Group university.

My current job is quite different from my previous role at the National e-Science Centre, which was the topic of the previous incarnation of this blog. Some issues, such as virtualisation and cloud, are common to all IT jobs these days, but the emphasis of my current job is on support and administration systems rather than research. One important difference is that I must stress that all the posts here will not represent the official position of the University of Edinburgh in any way.

To start off, let me point you at an interesting comparison of the "student experience" at three European universities. This posting doesn't mention IT at all - which is a little humbling. On the other hand, perhaps this is how it should be. People come to university to learn, not to use the IT. People tend to notice IT most when something goes wrong, so no news can be good news. Thanks for Guillermo Rein for noting this via LinkedIn.
http://nickybartlett.blogspot.com/2012/01/per-usual-i-must-give-recap-of-my.html .

Moving on....

2008-10-28T22:06:00.002Z

I started a new job at the beginning of the month. I'm no longer working at the National e-Science Centre and that means that I won't be involved with Grid Computing Now!. I was granted a day from my new job to host the webinar on cloud computing with Ross Cooney, but that was my last commitment for GCN!.

It's been almost 6 years since I joined NeSC and it was about this time 4 years ago that we were writing the proposal for the Knowledge Transfer Network that became GCN!. Looking back, I the experience has taught me a lot as we evolved the KTN to be most effective. It took a lot of effort to get ourselves recognised - to "build the brand" in marketing speak - and I think the work paid off.

Now I have the chance to actually practice some of what I preached. My new position is Head of Development Services in the Information Systems Group of the University of Edinburgh, and one of our goals for the next year or so is to roll out a service-oriented architecture for the university. Other part of the ISG are deploying virtualised servers and my erstwhile colleagues at NeSC are working towards a campus grid. With good planning, we might even get these initiatives to work together.

I will post here when I have progress to report, but I will be blogging even less often than I have been until now. In the meantime, I wish everyone involved with Grid Computing Now! all the best for the future.

Entering the Era of the Cloud

2008-10-24T22:16:00.003+01:00

The Grid Computing Now! webinar on Cloud computing is now available on the GCN web site. We had to make some last-minute changes because Alan Williamson was unable to join us; so after Ross Cooney finished his presentation, he and I had an extended discussion, including several questions sent in be the audience. It went very well; a couple of times I wondered whether I should bring the broadcast to an early close, only to receive new questions from the audience that kept the debate going.

We covered many issues, but perhaps the key issue was when to use cloud and when to keep provision in-house. This depends on measurement and requirements (doesn't everything?). In the case of EMailCloud, Ross estimates that if a server will be kept well utilised for more than 8 hours a day, it is cheaper to run that machine in-house, while using the cloud for peak loads, disaster recovery, and so forth. We went into more detail than that - if you're interested, watch the webinar!

We also covered the issue of interoperability. This is vital for users, because cloud services do fail. Ross develops his applications for two vendors, which is an overhead for him. The industry needs some common interfaces. In my opinion, these will only come about if users and smaller vendors push for them. The big vendors will each be hoping to be the next Google and will see commonality as a threat.

Take a look!

Synching the 2.0 web

2008-10-15T21:50:00.002+01:00

Advocates of web 2.0 suggest that we can access nearly all of the services we need from web suppliers. We can edit our documents, store our photos or company data, and run our applications. It sounds great - but what happens when the web is unavailable? Over the last few years I have travelled quite a bit and I've often found myself in places with no wifi connectivity - or at least none at a price I'm willing to pay. So I value having a copy of my data on my laptop, so that I can carry on working.

I've put forward this argument at a couple of events recently. At an excellent session on Web 2.0 and science at the UK e-Science All Hands Meeting, the response was that 3G coverage will soon be sufficient to give us access almost everywhere. The next generation will take it for granted, the way they take GSM talk coverage for granted already. I have to admit that this scenario seems quite likely, although of course there are still places that don't even have talk coverage.

Nevertheless, there are still problems. Cloud services are far from 100% reliable, at least as yet. The word from companies using cloud computing for their business is that we should expect failure and deploy applications on multiple providers. I believe we should do the same with our data. In addition to guarding against technical failures, it would protect us from vendors who go out of business or close down a service. It would would also prevent vendors from taking advantage of "lock-in" to increase their prices.

So, we need systems that can replicate data from one data store to another. Fortunately, we know how to do this, whether via Grid or via P2P technologies. Unfortunately, we seem no nearer achieving standards for interoperability, so we will need to build systems that interface to the variety of proprietary systems out there.

Ideally, the data should be self-describing, so that two copies can be synchronised by a different application from the one that actually created the copies. I'm put in mind of the apparently simple problem of syncing my calendar between my PDA and my PC. When I migrated my PC calendar to a new application, the next synchronisation created two copies of each event. You'd have thought that the iCalendar format would tag each event with a UUID so that multiple copies could be easily reconciled, but it seems that this doesn't happen. Let's make this a ground rule for storing data in the cloud.

I'll leave the last word to a panellist at the Cloud Computing event in Newcastle. When I explained that I wanted my data on my laptop so that I could work on the plane, he suggested that perhaps I'd be better using the time to read a good book.

Cloud Computing Panel

2008-09-26T00:40:00.001+01:00

I’ve just attended a panel session on Cloud Computing in Newcastle, which gave several points of view on the uptake and applicability of Cloud. The discussion covered Sofware as a Service (e.g; SalesForce, EMailCloud), Platform as a Service (e.g. Google App Engine, Arjuna) and Infrastructure as a Service (e.g. Flexiscale, Amazon EC2).

The optimistic view, taken by the majority of the panel, was that we are on a journey towards cloud computing becoming the norm for business computing. Duncan Mactear of 4Projects sounded a more cautious note; his company provides SaaS for the construction industry but does not use cloud; instead their servers are hosted in a third-party data centre. To which Tony Lucas of Flexiscale pointed out that 10 years ago, similar companies weren’t even using hosting services.

Sarat Pedirela of Hedgehog Lab, an ISV, pointed out that the appropriate infrastructure will depend on the type of application. Currently, Hedgehog use cloud for non-critical applications where the load varies greatly, such as downloading media files or running software configuration tests. Steve Caughey of Arjuna suggested that Cloud will be useful for any application where you can’t predict the demand. Ross Cooney of Rozmic agreed and added another example, of intense computations that can be sped up by running them in parallel.

All the panellists agreed that startup companies can particularly benefit, because they don’t need to install and run their own IT infrastructure. By way of contrast, Tony explained that some large companies have financial systems that can’t cope with variable IT costs; they want a fixed bill known in advance.

Trust was raised as a key issue. Several panellists opined that interoperability was the best answer to this; then if your provider has problems, you can switch your application to another. Rozmic run their EmailCloud application on both Amazon and Flexiscale, switching between them when one has problems. The downside of this is that it is currently expensive to implement applications for multiple providers, although some companies (such as CohesiveFT and Rightscale) are providing systems to aid this process.

All in all, it was an interesting and useful discussion. The session was arranged by Codeworks, with sponsorship from Flexiscale and Amazon.

Webinar: Powering your business with Cloud Computing

2008-09-23T12:27:00.003+01:00

On October 14th, I will be hosting a Grid Computing Now!
web seminar on the topic of Cloud Computing. We have lined up two very interesting speakers who are using Cloud now to make businesses work.

Ross Cooney had a good technological solution to sell but couldn't make it economic until Cloud Computing allowed him to pay for his computation only when he needed it. He will discuss the instant benefits and long term impact of cloud computing to the development, competitiveness and scalability of your application.

Alan Williamson created the BlueDragon Java CFML runtime engine that powers MySpace.com. He advises several businesses and will give an overview of the different types of services available and how to avoid being locked-in to a single supplier.

You can register for this event here.

Technology Strategy Board: Information Day, 22nd October

2008-09-17T09:53:00.002+01:00

I've been asked to publicise the following event.

The Technology Strategy Board has arranged an Information Day for Wednesday 22nd October to outline the various R & D Competitions being planned over the next 9 months.

This Information Day will provide delegates with an opportunity to find out about the activities of the Technology Strategy Board and gain an understanding of the application process for Collaborative R&D Competitions as well as find out about other Technology Strategy Board activities.

The event, being held at the Hyatt Regency Hotel in Central Birmingham, will open at 09:30 for a 10:00 start and will close at approximately 16:30; a full agenda will be available shortly.

To register for this event please click on the following link and complete the on-line registration form

For more information on the Technology Strategy Board please visit their web site

Competition: Grid Solutions for a Greener Planet

2008-09-14T22:57:00.002+01:00

This is a reminder that Grid Computing Now! is running a competition to find uses of grid technology to reduce human impact on climate change. The competition is open to anyone who is 18 or over and resident in the UK. So get your thinking caps on and submit your best ideas! The topic is deliberately wide, as is the interpretation of "grid", to allow a wide scope for proposals.

The deadline for entries has been extended to Friday October 17. This extension is particularly intended to give more time to university staff and students who wish to enter.

The initial proposal just requires 1,000 words describing the proposed solution. See the competition web page for background information and details of how to enter. There are two tracks, one for IT professionals and the other for everyone else (including students).

AHM 2008

2008-09-12T23:21:00.002+01:00

I was pleased by our workshop on research opportunities this week. Our speakers met several people who were interested in their work and might contribute to taking it further. It's hard to measure the outcomes of these events, because the collaborations that we are aiming to catalyse may take months to firm up and then may take much longer to produce actual results, but the first impressions are positive.

Some of the networking happened outside the workshop itself, of course. That is the advantage of face-to-face meetings; sometimes all you need is to bring the right people together for the first few minutes. Also, you can follow serendipitous links, such as when a colleague pointed me at the workshop on declarative data centres that Microsoft Research Cambridge and HP Labs organised earlier this year. I think the UK is building a critical mass in data centre management and I hope this can be encouraged to the point where it becomes a viable industry.

Beyond our workshop, there was plenty of interest in the wider AHM conference. I chaired one "regular" session in which most of the papers were about the use of social networking tools and similar systems for supporting science teams and similar projects. There is a lot of potential here and several people are taking advantage of it.

One limitation of this year's event was simply the large number of parallel sessions, which meant that individual sessions were quite small and that you couldn't follow even half of what was going on. But I think this was offset by he bringing together of several different communities, which should bear fruit in later years.

Workshop on Research Opportunities

2008-09-07T11:13:00.002+01:00

This week will see the annual conference for UK e-Science, which for historical reasons is called the e-Science All-Hands Meeting. I have organised a knowledge transfer workshop for the Tuesday afternoon, with the aims of presenting research opportunities for e-Science in the UK commercial and public sectors. We have four excellent speakers lined up.

Mark Ferrar is the Director of Infrastructure Architecture for NHS Connecting for Health in England. Mark is interested in opportunities for using the processing power available to the NHS to improve clinical outcomes, for example by running HPC models and diagnosis applications.

Liam Newcombe is tackling the question of "Green IT" in data centres. This is a big topic in the industry, because energy prices are rising and carbon accounting is being deployed. Liam has developed an open-source integrated model of data centres for the BCS and the Carbon Trust. He is looking for collaborators to further improve this model.

Alexander Efimov will give us a presentation about the exploitation of e-science technologies. Alex is from the CERN and ESA UK Technology Transfer Office. He is working with Constellation Technologies to build a business around the provision of services using the open-source gLite software.

Finally, Chris Greenwell of Durham University will present a new collaborative R&D project. The NIMES project is using e-science to directly improve the performance and reduce the environmental damage of the oil drilling industry, via a combination of computational modelling, visualisation and computation steering. This shows the opportunities that arise from the continuing improvement of e-science technologies.

The workshop will conclude with a panel discussion. Full details can be found here.

Greening the desktop

2008-09-05T22:40:00.002+01:00

I attended an interesting workshop this week. It was one of the series that Peter James has put together for his SusteIT project; this one focussed on desktop PCs. The talks and panels looked at measurement procurement of energy use, procurement options, desktop grids, power management and thin clients. In any sizeable organistion, Desktop PCs use a large amount of electricity and there are many options available for reducing this consumption - and saving money too. This is the second time that I've seen a British university do the sums and expect to save £250,000 a year.

The panel on power management tools was interesting. These seem to be coming of age at last. Operating systems have had support for managing individual computers but a large organisation needs a system for managing thousands of PCs, with different policies for different groups, and of course the important facility to wake up in time for the distribution of updates.

James Osborne gave an interesting analysis of the FLOPS/Watt providing by Cardiff University's Condor pool, as compared to their old cluster (inefficient) and the one they installed last year using modern cooling systems. The new cluster is much more efficient, so it seems that if you have the funds to install modern dedicated technology for high-throughput computing, that may be the way to go. Of course, if you don't have that sort of money, a desktop grid will make good use of your existing infrastructure.

Unless, of course, you go for the thin client option. As Queen Margaret University have shown, a thin client approach can dovetail very neatly with energy-efficient buildings. Conversely, Merrill Lynch found that they used more energy in the server room than they saved on the desktop. The moral, I believe, is that you have to measure carefully the gains and losses for your particular installation. Case studies from elsewhere can give hints about what to look for and approaches to consider, but they are no substitute for performing your own analysis.

A cloud + a fringe = a silver lining?

2008-08-31T22:51:00.002+01:00

The Edinburgh Festival Fringe is over for another year. Hundreds of performers have given thousands of shows to a huge number of culture seekers. Meanwhile, the IT industry has notched up another very public failure. The new box office system failed to cope with the demand for tickets on the first day and had to be patched hurriedly. It struggled along after that but was unable to implement the special offer that had been planned for last few days of the festival.

It must be a challenge to cope with the huge interest on the first day of ticket sales and then to manage the varying load fot the next three months. For the other nine months of the year, of course, the system isn't needed at all.

Does this sound like a candidate for cloud computing? I don't know how the existing system is implemented, but if it doesn't already use cloud to react to large swings in demand, perhaps the service providers should consider this option.

A holistic use of thin clients

2008-08-22T21:32:00.002+01:00

Yesterday saw a workshop on Sustainable IT at the new Queen Margaret University campus in Musselburgh. The workshop was ostensibly about "new ways of working" but another major focus was on how the adoption of thin clients allowed the architects to design an more environmentally friendly campus. Thin clients use less power on the desktop than PCs, which means less heat is generated in the classrooms, labs and open working areas. For QMU, this meant that the building can use natural cooling and ventilation, saving considerably more energy in addition to the saving from the terminals themselves.

The switch to thin clients was also used to introduce the use of virtual desktops. These let staff and students access their work from anywhere on the campus and from home. Staff are encouraged to work from home when it suits them. On campus, staff now work in open plan areas rather than offices; overall, the extra freedom seems to outweigh any disadvantage from the change.

The energy saving aspect of the QMU deployment contrasts with the experience of Merrill Lynch, who found that the servers and associated equipment used at least as much extra energy as was saved on the desktop. QMU's holistic approach means that the reduction on the desktop is leveraged to gain further savings overall. This is a very sensible and productive use of thin clients.

This workshop was led by the SusteIT project, which is running a series of such workshops. I have been helping them by finding speakers and providing material on data centres, grid and virtualisation. Back in June, we ran a workshop on data centre cooling and power supplies, and this is one area where QMU could improve. Their server room is fairly small by industry standards but even so they could almost certainly cut their power bills further by adopting the latest technology. It seems the quest for energy efficiency can always find further improvements.

Progress with our "Green IT" theme

2008-07-25T14:54:00.002+01:00

We've been running our Green IT theme for several months and I'm pleased with the progress we've made. We've run two webinars and spoken at several events, including the ITU symposium on ICTs and Climate Change and Oxford University's conference on Low Carbon ICT As an example sector, we are working with the JISC-funded project on Sustainable IT in Higher Education- partly because this is a sector in which we can publicise progress without too many strings attached.

We helped Peter James organise the first SusteIT workshop, which was held in Cardiff on June 19th. This has already caused quite a stir; Cardiff have done a good job of procuring an energy-efficient machine hall and other education establishments seem to be looking to this as an examplar. Later this year, we will be helping Peter with two more workshops that fall under our remit.

Green IT has to consider all aspects of running an IT service. As such, it is a broader topic than we can cover. We attempt to address the following aspects:

- Modelling the power and cooling systems in data centres
- Virtualisation
- The use of Cloud Computing to outsource IT provision
- The impact of new processor architectures

The last of these is only beginning to appear on people's radar. Our industry has grown used to new hardware being both more powerful than the previous generation yet remaining largely compatible with it. This is likely to change: the era of "simple scaling" is drawing to a close - and is being helped on its way by increasing energy prices. The mass market won't fade away but those customers who have specialist needs - including high-throughput computing - will be able to take advantage of non-standard machines. Chip-makes are investing heavily in supercomputers composed of many GPUs - the graphics processors found on PC graphics boards - which can perform multiple instructions for certain calculations faster and using much less energy than conventional processors. Other possibilities include more use of FPGAs, for similar reasons.

Anyone interested in these topics may like to attend LTG workshop. This isn't one of our events but we're happy to advertise it - it looks good.

Competition: Grid solutions for a greener planet

2008-07-08T16:51:00.002+01:00

Grid Computing Now! is running a competition to find uses of grid technology to reduce human impact on climate change. The competition is open to anyone who is 18 or over and resident in the UK. So get your thinking caps on and submit your best ideas! The topic is deliberately wide, as is the interpretation of "grid", to allow a wide scope for proposals.

There will be three stages to the competition: an initial proposal of 1,000 words; an apprentice workshop, where finalists can speak to grid architects and academics to develop their ideas; and the final presentation at The British Computer Society on December 1st 2008.

The deadline for registering your interest is July 31st, but the initial submission is not required until September 1st. See the competition web page for background information and details of how to enter. There are two tracks, one for IT professionals and the other for everyone else (including students).

Combining BOINC and BitTorrent

2008-06-06T11:38:00.002+01:00

A busy schedule and an unusually flakey wi-fi setup have conspired to limit my blogging activity from OGF23 this week. This is just a quick post to note a rather neat idea that was demonstrated by one part of the CoreGrid project.

BOINC is the infrastructure used by volunteer computing projects such as SETI@home and ClimatePrediction.net. The system sends out jobs to be run on people's home computers and collects the result. Handling all this network traffic puts quite a load on the central server. What the CoreGrid demo has done is to combine BOINC with the BitTorrent peer-to-peer data distribution system, so that the load is distributed.

This may not be news to some of you, as the paper was published last year. I like it; it's a simple idea to solve an immediate problem. I've heard of other projects looking at the potential of BitTorrent as well.

Alex Hardisty on HC2008

2008-05-08T11:44:00.002+01:00

Alex Hardisty, of Cardiff University and the Healthcare@Home project, has noted his observations on HC2008. In particular, he notes some trends which he plans to address in his research.

Cross-Enterprise Document Sharing

2008-05-07T20:52:00.000+01:00

At HC2008 I was introduced to the XDS standard for Cross-Enterprise Document Sharing (not to be confused with several other uses of the XDS acronym). XDS is a profile of the ebXML standards for registries and other related standards to specify a system for sharing medical documents. The important point is that XDS is supported by several major vendors and has been deployed in clinical health systems in several countries. As far as I'm aware, it has not made any impact in the grid world.

The basic XDS standard can be extended to address particular use cases or to add functionality. One popular extension handles DICOM files (a format widely used in medical imaging). Another (XCA) supports federated registries, removing the single point of failure of the basic model.

I think it is worth investigating for other e-Science applications. It may be simpler to leverage this work than to reinvent it. Also, it might be possible to apply our work on e-infrastructure security to the federation scenarios in XCA, which does not have adequate security mechanisms of its own.

PGC08 to discuss Green HPC

2008-05-01T17:43:00.002+01:00

GridToday reports that the Platform Global Conference will include a panel on whether HPC data centres can "go green". Among the strategies they will discuss include energy-directed scheduling, i.e. dynamically allocating workload to minimise electricity consumption.

It so happens that I am organising a workshop at OGF23 on exactly this subject. We will be looking at the details of what is needed to make this a reality and what steps are needed to make this idea a reality.

Ian Foster finds an interesting take on Green IT

2008-04-29T11:31:00.002+01:00

Ian Foster notes a project at University of Notre Dame near Chicago which is distributing research computing facilities in order to provide heat to campus buildings. This is an interesting trade-off: on the one hand, the distribution of resources means that the waste heat is put to good use; on the other hand, it's possible that each distributed installation is less efficient that a good-quality centralised machine hall.

This is another example of where we need good quality models and well-measured example deployments to help us decide which approaches give the best results.

Grids & e-Health

2008-04-25T15:52:00.004+01:00

This week, I attended Healthcare Computing 2008 to get an update on the current state of e-health in the UK and to explore how grid technology can contribute. Health Informatics is a broad subject and it isn't possible to engage with the whole field, but I see three main areas of potential engagement.

The path is most followed by the academic community to date is that of linking together data used in clinical trials or in health research. This is a natural fit for the e-science community as it extends existing work on secure access to distributed research data. The medical world imposes more security constraints, which adds academic interest, but is otherwise familiar to the e-scientists. It is also of a scale that is manageable in research projects. Successful projects include Psygrid, which is now deployed across the NHS research centres in mental health and in bioinformatics. So this is the first area of engagement.

A natural question is whether this experience with research data grids is also applicable to clinical systems. Clinical practice is a much larger enterprise. Major initiatives exist to share data such as patient records and medical images. These are either provided by small-scale systems implemented by individual health trusts, or by specialist commercial solutions. These systems face the same problems as data grids - single sign-on, access control, metadata management, data vocabularies - but engagement with the grid world seems virtually non-existent. There is probably scope for mutual learning between the two communities, but the healthcare community is understandably focussed on providing solutions now. So I regard the chances of successful engagement here as slight.

A more likely area is the computing aspect of grids. Although this has also made little impression on the clinical community to date, there are successful R&D projects in the areas of data analysis and diagnosis support, including GIMI and Healthcare at Home. These use pattern matching technologies to detect suspicious lumps in medical images or to warn of possible problems in patient monitoring systems. These are rather specialised systems, which would make it easier to arrange pilot deployments to text their clinical effectiveness.

The third area of promise is the most straightforward. The NHS is one of the world's biggest organisations and runs a lot of IT systems. As a result, there are plenty of opportunities for deploying modern IT infrastructure techniques, with virtualisation as an easy quick win. The NHS has yet to consider the effect of CO2 emissions and electricity costs from its myriad systems and it will have to do so soon.

Taking this to a further stage, I believe that the NHS would benefit greatly from adopting service-oriented techniques. There is currently a developing interest in lean management techniques in NHS hospitals, with great opportunities for improving processes for patients and employees while saving money for hard-pressed hospital trusts. Such process optimisations can be delivered more easily if the IT layer is equally flexible. The business management should lead, but the IT infrastructure must be agile to follow the dance.

Garbled Grid Hype

2008-04-16T22:08:00.003+01:00

There has been some rather confused coverage in the press about the grid infrastructure that supports the Large Hadron Collider at CERN. The Times has an article that cast the grid as a "superfast internet", with the emphasis on the high-bandwidth links that have been laid to support the LHC data dispersal. It also talks about the numbers of servers connected to the LHC grid, but without clarifying the distinction between bandwidth and processing power. It also implies that the LHC grid is the only grid, which perhaps we can forgive the journalists for, as plenty of technical people still refer to "the grid" as if there were only one.

A Yahoo article, taken from Sky News, goes rather further, claiming that "the internet, as we know it, could be obsolete within a decade". The phrase, "as we know it", lends a wonderful vagueness to the claim. The article goes on to say that the Grid was the brainchild of CERN, which of course is an exaggeration. On the technical side, this article is possibly better, describing the LHC grid as a linked network of computers and describing some of the applications that it has been used for. There's just one paragraph where the illustration of bandwidth has been taken rather literally and expounded as if every household will have such connectivity, which is a separate issue.

I'm sure that none of the inaccuracies are the fault of the people mentioned in the articles. Journalists need to tell a story snappily and with an "angle" that will interest their audience. But the grid community need to do a better job of telling people what grids are about. We are doing our bit by producing some new material for Grid Computing Now!, which should be on the web site shortly.

Grid infrastructure for clinical trials

2008-04-08T16:43:00.002+01:00

Ian Foster pointed readers of his blog at a good article about CaBIG, the Cancer Biomedical Informatics Grid. This major project is led by the US National Cancer Institute and also involves Cancer Research Labs in the UK, with the aim to share data between cancer researchers. The UK's OGSA-DAI system is a major component of the deployed system.

It's particularly interesting to note that the system includes support for clinical trials. Clinical trials are time-consuming and expensive, so many people want better systems for managing them and a number of grid projects are tackling this area.

PsyGrid is one such project in the UK. They're not exactly blowing their own trumpet about this, but their system is being used as the studies and trials platform for the Mental Health Research Network, and has been selected by the National Institute of Health Research (NIHR) as the platform for providing electronic data collection services to support studies and trial across the Biomedical Research Centres.

May 7 Webinar: Energy efficient data centres

2008-04-07T11:56:00.002+01:00

I am very pleased with the line-up for our webinar on May 7th. We will continue our investigation into best practice for running energy efficient data centres, following our successful webinar last October. This time we will have an emphasis on the impact of virtualisation and on how one can model and measure the effectiveness of proposed improvements.

Nic Barnes will explain what Merrill Lynch observed when they applied virtualisation on the desktop and in the server room. They found that the benefits were in some cases partially offset by losses elswhere. Nic will demonstrate the importance of measuring real gains in practice.

Liam Newcombe will describe the work of the BCS in developing a model for predicting data centre efficiency. This approach will allow managers to plan and evaluate designs in advance of their implementation. Liam will show that the choice of metrics requires careful analysis.

As always, viewers will be encouraged to ask questions. Any that we can't answer during the seminar itself will be replied to on a web page after the event.

This webinar will run from 2.30-3.30 GMT on Wednesday May 7th. To register, please go to http://mediazone.brighttalk.com/event/gridcomputingnow/0c0a756691-1388-intro

Virtualisation, HPC and Cloud Computing

2008-04-01T17:32:00.002+01:00

Virtualisation has obvious benefits for much commercial IT, where existing servers often have utilisation rates of 10% or less. It's less clear whether virtualisation is so useful for High-Performance Computing (HPC), where systems are often kept running at utilisation rates above 90%. Nevertheless, there are potential benefits from adopting some aspects of virtualisation in these environments. The question is, do the benefits outweigh the cost in performance?

This was the topic of yesterday's workshop on System Level Virtualisation for HPC (which was part of EuroSys 2008). The workshop was rather small but I did learn quite a bit.

A couple of talks investigated the performance hits of using virtualised OS's instead of addressing the hardware directly. This varied, depending on the type of application; if IO was minimal, the slowdown was minimal too. An "average" slowdown seemed to be on the order of 8%.

Samuel Thibault of Xensource looked at ways of using the flexibility of a hypervisor to implement just those parts of an operating system that are absolutely necessary - ignoring more general-purpose facilities such as virtual memory, pre-emptive threads and the like. Essentially, this was using Xen as a "micro-kernel done right".

Larry Rudolph, who works for VMWare but was here speaking with his MIT hat, gave a scenario where virtualisation might benefit the HPC user. Some problems require massive machines that researchers only get access to maybe twice a year. The example Larry gave was modelling global ocean currents. If the researchers can save the messages sent between regions of this model, they can later run more detailed analyses on one particular region on a much smaller machine, using the recorded messages to playback the effect of the rest of the simulation. This gets complicated if the new micro-simulation gets out of sync with the orginal and in this case the system may have to switch other regions from record mode to full simulation. The point is that it is easier to handle this mixture of recorded and actual simulations using virtual machines than having to control it all "by hand".

The discussion that ended the workshop brought all these points together. Virtualisation makes grid or cloud computing more feasible, which may make HPC resources available to more people. New CPUs have explicit support for virtualisation that will make it almost cost-free for the data centre, and there is a good chance that these advances will also reduce the overhead for some HPC applications too. The key problem seems to be predicting which applications will run smoothly on a virtualised system and which will suffer noticeable slowdowns. As always, more research is needed!

Webinar on licensing, virtualisation and grid

2008-03-24T20:16:00.004Z

Licensing has been an ongoing issue for virtualising and grid-enabling applications. Many licensing models do not transfer easily to a world where applications and virtual servers can be run on many processors and moved from machine to machine. Vendors want to ensure that they are paid for the full use of their applications, while IT managers want to know how much they can expect to pay in a given accounting period. Everyone is aware of the problem and several people have been trying to work out ways forward, but the issue is still unresolved.

Grid Computing Now! are bringing together some of the key players in the industry for a webinar on Thursday 10 April at 14:00 GMT:

David Gittins of Capgemini, who has been working with the public sector on this challenge
Neil Sanderson, Product Manager for Virtualisation at Microsoft
Mark Cresswell of Scalable Solutions, a leader in the provision of tools for monitoring and reporting usage.

Ian Osborne, Project Director of Grid Computing Now! will chair the session and invite those attending to an online conference to discuss the topic following the broadcast. Register for the webinar here.

Grid Computing Now! is working closely with FAST, the Federation against Software Theft, to bring insight and understanding to this issue for IT managers in all sectors of the industry.

Low Carbon ICT Conference

2008-03-20T15:03:00.002Z

Yesterday I attended the Low Carbon ICT Conference organised by Oxford University, with funding from JISC. This was a good an useful day, with speakers covering a range of topics, and a small number of exhibits. Some of the speakers were familiar - Zafar Chaudry gave his excellent talk on the benefits of virtualisation in one of our webinars, while Liam Newcombe will be speaking in our next-but-one webinar on May 7th. Others were new to me.

The speakers' talks covered office ITC equipment, virtualisation, data centres, PC manufacturing, sustainability entrepreneurship and enterprise planning. I was more at home with the techie end of things, but this is not a problem that can be tackled by technology alone so it was good to see the broad participation of this conference.

I was aware of most of the technical issues, although I was surprised at how much energy is used by desktop PCs and peripherals (see below). I was also interested to hear Juergen Heidegger recommend the use of thin clients, as our case study from Merrill Lynch suggests that the saving here is less than might be expected. Juergen has sent me a paper that backs up his claim, which I look forward to reading.

Nick Eyre and Daniel Curtis are from Oxford University's Environmental Change Institute. They mainly addressed the question of emissions from office PCs and related equipment, rather than data centres. Nick set the context by reminding us that the government is aiming for a cut in greenhouse gas emissions of 25-30% by 2020, while current non-domestic ITC equipment uses 17,000 GWh/yr in the UK. This is 7% of non-domestic electricity consumption and rising. For contrast, the power consumption of servers and data centres are estimated at 5,000 GWh/yr. So even simple schemes such as configuring PCs to power down when not in use can save significant sums; Daniel produced figures that estimated the university could save £250,000 and 1,445 tonnes CO2 each year.

Liam Newcombe expanded on the new regulations that large organisations will face, mentioning UK cap-and-trade and the EU Emissions Trading Scheme. He also pointed out that being "green" is already an important aspect of brand value. Universities are included in this - People and Planet have produced a league table to help applicants choose their preferred place of study. Liam went on to present the BCS's work on data centre modelling, as introduced in our October 2007 webinar by Zahl Limbuwala.

Juergen Heidegger of Fujitsu Siemens and Peter Waggett of IBM told the conference about their companies' internal work to reduce energy consumption. Juergen focussed on Fujitsu Siemens' production of PCs with low energy usage and which can be largely recycled at the end of their lifespan. Peter gave a broader presentation of areas where IBM is looking to use less power.

A contrast from all this tech talk was provided by Martin Chilcott of Meltwater Ventures. Martin is an entrepreneur and marketeer who sees sustainability as the "fourth revolution" (following the agrarian, industrial and information revolutions"), and hence sees the chance for people with the right ideas to make very successful businesses.

The JISC project that funded this conference aims to improve energy efficiency of IT use across Oxford University and disseminate the results so that other universities and businesses can learn from the experience. Personally, I think this is only one part of the puzzle; the HE funding agencies need to take the whole "green IT" issue on board when buying equipment and I see no sign that they have done this.

A Research Strategy for the Century of Information

2008-03-14T22:25:00.002Z

"This is the Century of Information" - G. Brown, November 2007.

This week, I attended a think-a-thon about a strategy document that the e-Science community in the UK is developing. The goal is to put in place the right mechanisms for ensuring that UK research can make the most of new computing technologies and methods. We have many success stories from the e-science programme; the question is, how do we build on those successes and make the techniques available to everyone?

One point that arose from the workshop is that we need different types of successes. Most of the examples put forward were of good research enabled in a range of domains (GeoSciences, BioScience, Chemistry, Physics, Social Science, etc.). We also found examples of advances in Computer Science itself, rather than just using CS to support other fields; this is essential if we are to engage CS academics.

Beyond academia, we need examples of knowledge transfer to industry. This is where the Grid Computing Now! KTN can help and I will be working on this strategy document in the next couple of weeks to flesh out this story. We also need to engage the public and the schools, so that students coming into university or industry know about the existence of our techniques.

When we looked at the barriers between us and this vision, we were perhaps less imaginative; the usual social and technical issues were listed. I would have liked to seen more about technology transfer in both directions between academia and industry; if nothing else, the academics should be interested in this because it will ensure support for their research from the politicians and funding agencies.

We also had an interesting discussion about Green IT. Although some of us felt very strongly about the need to tackle the problem (Greenhouse gas emissions from university IT is doubling every four years), it became apparent that this won't be seen as a barrier until it impinges on day-to-day behaviour. The research funding structure means that researchers often don't see the full cost of the facilities they use, because they are funded centrally. It's not clear whether it would be practical to change that structure. Perhaps when politicians come under pressure to cut CO2 emissions, perhaps academic computing will seem an easy target, which would make the researchers take notice. Until then, energy efficiency seems more relevant to the infrastructure providers rather than the end users.

Big Green

2008-02-05T22:20:00.000Z

With the twin growth of supercomputing demand on the one hand and energy cost on the other, it makes sense for companies to produce supercomputers that use less power. IBM started the trend with the Blue Gene series, of course, although that project was nore about building the fastest computer while keeping energy consumption somewhat reasonable. More recently, others have joined the fold. One such is SiCortex, who claim to have designed their systems from the chip level up to minimise power consumption. Their SC5832 machine offers 5832 1GFlops 64-bit processors for 20kW, while for the SC648 they claim half a teraflop powered from a standard wall socket.

That's not the only approach, of course. Floating-point accelerator boards and Graphics Processing Units are being used to boost computing power for specific applications while keeping costs low. The Register has a good overview article from November's SuperComputing conference. Meanwhile, in a talk at the Mardi Gras Conference last week, Satoshi Matsuoka explained how the Japanese are building a specialist supercompuiting facility that combines all these elements to maximise computing power and minimise energy usage. His slides are here (8.3MB). Incidentally, they also show an example of how to design machine room to make efficient use of air cooling, rather than the all-too-common practice of sticking a load of machines in a giant airconditioned room.

Green Broadband can save the planet?

2008-01-30T21:56:00.000Z

This blog posting is an oddity - Bill St. Arnaud suggests a micropayment scheme whereby consumers pay for broadband services by increasing the cost of their household electricity (or car mileage) for a certain duration. The utility companies then pay this amount to the broadband service provider. So the service provider gets income from a source other than advertising; the consumer gets reasonably priced product, and the consumer reduces their energy usage because the price has gone up for that period.

I don't see it happening, myself, but it's an interesting piece of lateral thinking.

More on "green" data centres

2008-01-26T21:34:00.000Z

In order to expand our "community of practice" on energy-efficient data centres, we have visited several people or groups who have an interest in the topic.

Just over a week ago, some NeSC colleagues and I visited BRE's Scottish office. BRE used to be known as the Buildings Research Establishment and the folk at their East Kilbride office are particularly interested in sustainable development. So far they've been mainly working with housing and small businesses, but they seem potentially interested in modelling larger establishments. They are also interested in multi-level modelling of heat flow within buildings; something our e-science connections could definitely help with. We also had a good conversation about DC power circuits.

This week, I was invited to speak at a meeting of the Russell University Group IT directors (RUGIT) - i.e. the people responsible for ICT at many of our leading universities. I presented some of the outcomes from the HTC week in November and GCN's Green IT webinar, both of which led to stimulating conversations.

My colleagues in GCN have also been talking to interesting people. The most recent of these was someone who has retrofitted fresh-air cooling to existing data centres. I wish I'd known about that before the RUGIT meeting, as this was one of the ideas that was most heavily discussed! I will have to settle for passing on the information now.

I'd like to see these discussions and relationships lead to a "best practices" document, perhaps beginning with the university sector, as the universities are comparatively open about their businesses. We've a way to go before this happens.

In the meantime, I'm putting together ideas for another webinar. I'll blog here if this becomes a reality.

Data Protection, TPM and Grids

2008-01-11T15:31:00.000Z

This week, the e-Science Institute launched a new research theme which should be of great relevance to industry as well as scientists - in fact, it may even help ordinary consumers to protect our own privacy online. The theme is about "Trust and Security in Virtual Communities". Andrew Martin, the theme leader, explained its aim in a webcast talk.

The problem that Andrew is exploring is how we can trust a grid infrastructure to protect our sensitive data. In addition, how can we trust the results that we get back from running a job on "the computing cloud"?

To give one concrete example, Andrew was involved in the climateprediction.net project, which encouraged people to contributed their PC's spare cycles to run climate modelling simulations. This raised several security issues. From the users' point of view, could they trust that the climateprediction.net program would not hijack their PC? Conversely, could the scientists trust that the data sets returned were run by their model and not by some hacked (or "improved") version?

Commercial examples are easy to find. Much industrial data is valuable and/or sensitive, which limits the trust that companies have in sharing it with potential collaborators. And in e-health, patients want to be sure that their personal medical records are only seen by relevant people in appropriate situations.

In everyday life, one example arises when we put photos on a web site. We may put them there for family and friends but may want to stop even close relatives from copying and pasting them elsewhere.

In all these cases, we want to attach policy statements to the data that control who may do what with that data. For this to work, the data must then only be viewed by applications that we trust to "do the right thing". This is a hard problem - how can we trust software running on someone else's machine?

Fortunately, the computing industry is developing tools to do some of this work. Central to this is the Trusted Platform Module (TPM), which can uniquely identify hardware. The new eSI theme will look at ways that this technology can benefit scientists, companies and citizens. For more information, take a look at this green paper.

Innovation and Knowledge Transfer Networks

2007-12-21T15:38:00.000Z

I'm very pleased to say that our Knowledge Transfer Network, Grid Computing Now!, will continue to operate for at least another year. We have just received final confirmation from the newly-reconstituted Technology Strategy Board. This is welcome news; it means that we can continue our plans to bring users, vendors and academics together to address real problems in several sectors. So it seems a good moment to reflect on the state of KTNs and how they might develop.

Innovate07 was the showcase for all 22 Knowledge Transfer Networks. This was my first time at Innovate and I was impressed by the range of technologu areas and delegates. It was also a good opportunity for networking between KTNs, which has led to some joint initiatives.

This range of KTNs is in part a branding exercise, as some KTNs had previous existences as Faraday Institutes or other institutes. So we at GCN are in the odd position of being one of the first KTNs to be set up and at the same time among the ones that have been operating for the shortest period! For once, I think the branding works well, presenting a breadth of expertise available to UK industry.

In the GCN KTN, our activities are gradually moving from general outreach into more focused engagement in specific projects. In our first two years, we were mainly building our network of contacts, collecting our set of case studies and producing webinars. Over the past year, we've begun to engage more in specific sectors and issues such as tranaport modelling, software licencing and Green IT. This is a deliberate move that we will continue in the new year by setting up specialist interest groups within the KTN. This brings our modus operandi closer to that of some of the other KTNs.

Even so, it's important that our contacts and our funders realise that different KTNs work in different ways. For example, the Applied Maths KTN runs an excellent series of intensive workshops in which mathematicians work on industrial problems, often making substantial progress in the course of that week. That approach works well for mathematics but is not suitable when it comes to building new IT infrastructures!

The aim of all this, of course, is to stimulate innovation. By its very nature, innovation is hard to codify; the best we can do is to bring people together in an environment that stimulates new ideas and rewards their exploitation. So one thing I'd like the TSB to do is to set up KTNs in less established areas - a key example being alternative energy. Within the existing KTNs, it's our job to look for the new ideas to pass on to potential exploiters, and to look for business problems that can be solved with innovative ideas. It's not easy but it is certainly interesting!

Scotland's first eco-powered data farm

2007-12-11T17:58:00.000Z

I delighted to see that Internet Villages International are building a major data centre to be powered entirely by "green" energy in Scotland. This is exactly the kind of development that some of us have been arguing for. The centre will be built next to a source of renewable energy and the waste heat is intended to be used for a new village and for local horticulture.

This makes sense in many ways. Data centres do not need to be near the businesses that use them (think of how often you use Google and where their data centres are). Placing them near sources of renewable energy saves on transmission losses; indeed Google, Amazon and other firms are already doing this in the USA. Also, data centres produce a lot of heat and it makes sense to use this rather than waste it.

Let's hope that the operators minimise the energy they do use as well, for example by running virtualised servers, building modular UPS and cooling systems, and using external air intakes rather than cooling recycled air.

I have no idea who is behind Internet Villages International are, although this URL suggests they are related to Cygnet. Whoever they are, I wish them every success.

High Throughput Computing week

2007-11-30T20:51:00.000Z

We;ve just finished a week (well, four days) of talks, tutorials and discussion about High Throughput Computing. The event was opened by Miron Livny, leader of the Condor team, who gave an excellent introduction - the key point is that HTC is about the number of tasks that can be completed in a given time, whereas "traditional" High Performance Computing is about how much computing power can be brought to bear at a given time. As Miron puts it, Floating Operations per Year is not necessarily 60*60*24*7*52 Floating Operations per Second (FLOPS).

We've hosted events by the Condor team in the past, but for HTC week we extended our range. In particular, John Powers and Dan Ciruli of Digipede flew over from the Bay Area to tell us about their product. A day of hands-on tutorials allowed delegates to compare the strengths of Digipede and Condor, and the evening discussions included ways the systems could be used together.

Scheduled discussions looked at requirements for HTC in academia and in business, at Green IT, data handling and policy management. As one outcome of the discussions, we're looking to capture HTC design patterns, publish them on the web and incorporate them into training materials. On the academic side, we are planning to write a report to explain the policy issues to university heads of research computing.

There is talk of running this event again next year. We would like to extend the range of participants again, e.g. by looking to UnivaUD, DataSynapse or Platform. For more vendors to attend, we will need more commercial users, and vice versa, so we need to start encouraging people now. I'm wondering whether a small exhibition area might be useful to the commercial vendors and delegates - not a major trade show (with the concomitant expense) but enough to be an effective market place.

Watch this space - and get in touch if you're interested!

Web 2.0, e-Science and Innovation

2007-11-27T22:06:00.000Z

Last week the e-Science Institute organised a "think tank" to review the state of e-Science and suggest opportunities for research. A major emphasis of the debate was the recent trend to use Web 2.0 tools to support scientists. Dave de Roure gave several examples he saw at recent conferences, including wikis and blogs such as Open Wetware and Useful Chemistry, as well as various data mashups. Tony Hey gave a public lecture on e-Science and Digital Scholarship which presented a similar story, including the use of utility computing (which now seems to be called cloud computing - you've got to love the constantly changing buzzwords in IT). Among the discussions, people mentioned the combination of Web 2.0 tools with semantic web technology, and the combination of structured queries and semi-structured information as in DBpedia.

This growth of e-Science 2.0 (to coin a buzzword of my own) has mainly seems to have occurred largely in the life sciences, perhaps because they're not centrally organised like the physicists and astronomers. Perhaps they also have more smaller-sclae experiments. One interesting point is that the adoption of these tools has been driven by the scientists making use of commonly-available tools. It isn't the result of existing "road maps" for e-science. It's another case of innovation occurring at the boundaries between communities; where ideas meet and produce new ideas. Sometimes this sort of innovation can be planned (or at least encouraged); sometimes it happens anyway.

This suggests to me that whenever an organisation runs a roadmapping exercise, it pays to include people with a broad mix of backgrounds and some wild ideas. Many of the ideas produced won't germinate, but some of them may grow into something useful.

Centralisation, security, and 25m personal account details

2007-11-21T13:59:00.000Z

The media and online world are buzzing with the news that HMRC have lost discs containing financial details of 25m people. My particular interest is to what extent the centralisation of the database contributed to the problem. If we consider the NPfIT programme for storing medical records, would they be safer in distributed data stores? At first glance, one might think that a security breach in one store would at least be limited to the set of data held there. But those distributed stores would have to be networked and to allow remote queries; would this increase security (by checking for mass requests) or decrease it (because people wouldn't know which remote requests to distrust)?

In the meantime, here are a few URLs to comments that I found interesting. Philip Virgo worries (as do I) about those organisations that cover up their breaches rather than report them. David Lacey argues that certification of security practices is required to make sure policies are followed. On a related note, the UK's information commisionner is calling for his office to have powers to inspect and fine organistions for failing to properly protect personal data. Finally, Ross Anderson's rather trenchant comment includes several links that are relevant to security practices in the UK government.

The Business Case and Methods for the Green Data Centre

2007-11-08T15:25:00.000Z

"Green IT" is a hot topic at the moment. In the UK, data centres contribute nearly 2% of the country's CO2 emissions, a figure which is similar to that of the much-vilified airline sector. We recently broadcast a webinar on this subject which went very well - several people have said that it was our best webinar yet.

We were fortunate to have two excellent speakers. The first was Zahl Limbuwala of the BCS Specialist Group on Data Centres. This SG has developed an open source model for measuring server room efficiency, which will be published in January; Zahl presented the case for why energy efficiency is relevant and talked a little about this model.

The second speaker was Kate Craig-Wood, who runs a carbon-neutral hosting company. Kate gave a lot of low-level practical suggestions for improving the energy efficiency, which her company has used in their new data centre. There are obvious advantages in making such improvements, both economic and environmental.

The "Green IT" label covers many more issues, including the disposal of old equipment and power management within the office. Even when considering data centres, there are many avenues to approach, from building design, via power systems, to virtualisation software. We covered many of these topics.

The full title of the webinar was, "The Business Case and Methods for the Green Data Centre". You can see a Windows Media Player recording of the webinar here - just register and login to see it. A flash version will be provided in due course.

Virtual autopsies

2007-10-14T15:40:00.000+01:00

One of the best talks at AHM2007 was on medical imaging, by Prof. Anders Ynnerman of Linkoping University in Sweden. Imaging researchers always have an advantage when giving talks because they can show much of their work in pictures; Anders took full advantage. The immediacy of visualisation in communicating information meant that some viewers found his medical pictures a little too gruesome.

The computer science aspect of his talk was about techniques for 3-D reconstruction of body images from the 2-D “slices” taken by medical CAT and MRI scanners. As these scanning devices increase in resolution, so the size of data to be processed increases. In order to process the data, ZZX is using intelligent compression techniques, based on whether a voxel represents bone, blood or tissue.

Aside from the technical content, I was very interested in the uses XXX has found for this work in addition to the usual pre-operative medical briefing. His team are now able to help the police by performing virtual autopsies. They can receive a corpse in a body bag, scan it, analyses the results on a computational cluster and give the resulting images to the pathologies a few hours later.

This technique is particular useful for spotting broken bones, for example, or detecting the path of a bullet through a body. This latter example was used to clear a police officer of misconduct charges after a fatal shooting – the team were able to show that the office had aimed at the target’s legs as per orders but the bullet had ricocheted upwards off a railing, killing the victim. So the death was an accident rather than murder.

At present, this system requires careful tuning by the expert team for each case, but it seem eminently suitable to develop further into production use.

e-Science and Industry

2007-09-25T20:27:00.000+01:00

AHM2007 saw several examples of the industrial exploitation of e-Science technologies. I organised a session on this theme for Grid Computing Now! (GCN), which was attended by some 300 people. Jim Austin and Yike Guo gave talks about their contrasting approaches to commercialising their research.

Jim has built his company, Cybula, from income, opting for the advantage of control and accepting slower growth. Yike has chosen an investment-based approach for Inforsense, achieving the fast growth required by his funders. Both explained how they found markets for their technologies. They also reflected on how to maintain both academic and commercial careers, using each to benefit the other while balancing their sometimes contrasting demands.

I followed with a short talk that gave a high-level outline of the state of advanced IT infrastructure in industry, drawing on some of the presentations given our Grids Mean Business track at OGF20. Of course, IT infrastructure is only one aspect of e-Science; there are also application-level advances that are suitable for take-up by industry.

Later in the week, Thomas Hartkens of Ixico presented their experience of exploiting e-Science infrastructure. Derek Hill had given an excellent contribution to a GCN webinar about their business case earlier back in March. In contrast, Thomas focussed more on the management details, particularly the quality management required to meet regulatory approval in medical informatics. This level of project management is new to many academics, so this talk was very informative for anyone considering taking the plunge into the commercial world.

AHM2007 also had many demonstrations of e-science projects that involve industrial partners. On our stand, which was one out of many, we hosted demos from 6 of the collaborative R&D projects on inter-enterprise computing funded by the Technology Strategy Board.

In summary, there are many good links between parts of the e-Science community and some companies. One of our tasks at GCN is to encourage more of these successes.

Grids, data centres and reliability

2007-08-17T21:42:00.000+01:00

In my work with the Grid Computing Now! Knowledge Transfer Network, I talk about "virtualisation" and "service-oriented architecture" just as much as "grid" itself. People sometimes ask what is the difference between these concepts. My first answer is perhaps rather glib - I say that I don't care as long as the technology gets the job done. Although this is not a straight answer, those of us on the GCN! team believe it is important to put the business answers before any notion of technological purity.

But if we turn to the question as stated, I think that as long as a solution includes the key concepts of virtualised resources and dynamic allocation of applications across those resource, then that to me is enough to call the system a grid. But, of course, we can go further.
A recent conversation reminded me of the important point that distributed systems typically have to manage failure. As systems scale to many machines and many sites, then some of those are going to fail some of the time. The systems have to be resilient enough to adapt and recover. Systems also have to cope with additions and deletions from the set of available resources.

This is most obvious in cycle-stealing grids, which use spare power of desktop PCs, and of scientific grids, which link many research sites across the world. The interesting question is whether this also applies to the data centre. That seems to depend to some extent on how the system is designed. For example, Google is built specifically around this approach; they have always used lots of generic systems and just replaced resources when they fail. I believe Ebay's massive server farms use the same dynamic approach.

This question arose in a conversation I had with Liam Newcombe, an independent consultant. We were supposed to be talking about Green IT (of which more another time), but our discussion wandered to include all sorts of ideas. Liam is working on an open source model of data centre reliability and performance. He believes that reliability is best achieved by adopting this approach of explicitly allowing for it within the software - rather than, for example, attempting to make the hardware itself ultra-reliable.

A key question must be how high up the stack does this awareness have to extend? Can we write applications without worrying about this or does every application have to have some potential adaptability built in? It's a fascinating topic and I look forward to reading the book the Liam is co-authoring, in due course.

"That's not Grid!" - A cautionary tale

2007-07-11T13:18:00.000+01:00

I don't usually attempt humour here, but see if you like this...

Once upon a time, in the far-off world of Computerland, a great guru arose and declared a vision.

"I see a future when all computers will be linked together and people will run their programs without knowing which computers are running them. People do not need to know where their jobs run; they just need the results. I call this Grid Computing!"

The people of Computerland were excited by the guru's vision. They went away and worked to make it happen. When they were ready, they returned to the guru and said:

"Oh great guru! We have implemented your vision. We can run our programs on whichever processor is free at the time, making sure that all programs can run and making best use of all our processors. No longer does one computer sit idle while another one is overloaded. Your vision is a great success!"

"No, no, no!", said the guru, "That's not Grid!"

The guru explained, "What you have implemented is just cluster computing. Grid computing is about linking machines that are administered separately".

The people of Computerland were excited by the guru's vision. They went away and worked to make it happen. When they were ready, they returned to the guru and said:

"Oh great guru! We have implemented your vision. We can run our jobs on all the desktop PC's of the world. We have run the biggest climate model ever and we have used it to find new drugs against deadly diseases. Your vision is a great success!"

"No, no, no!", said the guru, "That's not Grid!"

The guru explained, "What you have implemented is just cycle-stealing. Grid computing is about making computing power available on demand; a utility similar to water or electricity. It's not just about using spare capacity".

The people of Computerland were excited by the guru's vision. They went away and worked to make it happen. When they were ready, they returned to the guru and said:

"Oh great guru! We have implemented your vision. People can buy processor time and disk space when they need it. Our computer resources are available to anyone who can pay. Your vision is a great success!"

"No, no, no!", said the guru, "That's not Grid!"

The guru explained, "What you have implemented is just utility computing. Grid computing is about linking many distributed resources to address a single problem".

The people of Computerland were excited by the guru's vision. They went away and worked to make it happen. When they were ready, they returned to the guru and said:

"Oh great guru! We have implemented your vision. People provide their programs as services using standard protocols. Then users can combine these services in new and unforeseen ways using scripts or workflows. Our businesses are more efficient and agile. Our scientists have made new discoveries. Your vision is a great success!"

"No, no, no!", said the guru, "That's not Grid!"

The guru explained, "What you have implemented is just service-oriented architecture. Grid computing is about common information and management protocols so distributed resources can be tightly co-ordinated".

"Oh!", said the people, "But we have already revolutionised our business methods, increased our agility and resource utilisation, put our previously wasted computer power to good use, provided computer power to people who need it, produced new science and found new ways of working together. We don't need anything more".

With that, the people stopped the guru's research grant and forced him into a life of giving keynote speeches on the conference circuit.

Moral: Don't define yourself out of success.

(Disclaimer: Given the opportunities for misunderstanding on the internet, I feel I have to stress that this is not aimed at any particular individuals).

Research grids and industrial data

2007-07-09T21:35:00.000+01:00

What happens when industry collaborates with academics, using the grid to share data? This was one of the main issues that we discussed today in a meeting of the NanoCMOS project. The industrial partners were clear that they would have to be convinced that their valuable data will be adequately protected before they allow their academic colleagues to use it on the grid.

The NanoCMOS project is looking at the impact of variability on the design and production of next-generation microchips. It is funded by the EPSRC and involves several leading electronics companies. The aim is to make circuit designs more resistent to the variations in the yield and performance of microchips; such variability is increasing as transistors get smaller and smaller. In a multi-billion dollar industry, it is clear that the companies involved do not want information about the design or performance of their products to go AWOL.

In the B.G. world (Before Grids), companies license their data to certain academics for them to use at their institutions. The academics are responsible for the use or misuse of this data and their institutions can be held to account in the courts.

In the world of grids, the licensing situation becomes more complicated. When scientists in different institutions use a grid to collaborate, all of them have to be bound into a licence agreement. In addition, the data providers must also trust the underlying technology and the people who use and maintain it. This requires advances in the state of the art of both the technology and in writing licences.

So far, the NanoCMOS project has focussed on the technology. Richard Sinnott's group at NeSC Glasgow are using Shibboleth to manage remote authentication and authorisation. They have developed appropriate authorisation roles, which include the authority to access particular software packages or particular data sets. Users can also be given the authority to delegate some of their roles to other people.

This apporach should work; it will allow data owners to restrict access to named individuals. The more taxing question is who manages the creation and assignment of roles. Ultimately this policy must be determined by the licensing organisation. They may install Shibboleth themselves and require all attempts to access a data set to seek authorisation from their server. Alternatively, they could delegate this right to the lead academic, who would then be responsible for managing the allocation of access rights.

Underlying this, the implementation, deployment and management of the technology must be trustworthy. The system administrators at the various sites will have the opportunity to misconfigure a system (whether deliberately or not). Additionally, of course, each deployment must be secure in itself. This will require a system of checklists and audits. Finally, each system must keep a secure log, so that they can demonstrate they have satisfied the licence agreements.

The NanoCMOS project should provide an excellent opportunity to test this in practice. The industrialists want to contribute real data and will only do so if we can get all the details right.

Interlude: New opportunities for user interfaces

2007-06-26T14:05:00.001+01:00

In my previous entry, I promised some thoughts on presentations at OGF20. Since then you will have seen nothing from me. This is not because I lost interest; unfortunately I've been in hospital and then recuperating. Now I'm well again, back to work and the proud owner of a heart pacemaker. I still plan to blog some thoughts arising from OGF20, but for this entry I will deal with a different topic.

I've just read and been inspired by Chris Mairs' 2006 Turing Lecture on Inclusion and Exclusion in the Digital World. Chris takes what could be a dull (albeit important) topic and makes it very interesting by showing the big picture. He's talking about accessbility, which previously I tended to associate with detailed guidelines about which colours to use on web pages and how to tag HTML images with meaningful "Alt" tags. I did consider it important - I'm now old enough to hate small fonts - but I didn't find it inspiring.

Chris's lecture looks both at technologies that help to overcome disabilities (including pacemakers!) and at technologies that are hard to use for disabled people. He gives mobile phones as an example of the latter, as their use of soft keys means that they were practically unusable by blind people - of which Chris is one. He notes that this is not just a matter of a particular technology but has major social implications, as people now tend to use mobiles to navigate their social lives. Similar concerns apply to the World Wide Web.

Now, however, Chris sees many opportunities for improved interfaces - to benefit all sorts of users, disabled people included. To continue with the example of mobile phones, these are now powerful computers with many input and output devices (including microphones, speakers, displays, cameras, GPS, ...). Just as important as the hardware is the software environment: most phones are now extensible platforms with development kits available. This allows third parties to develop specialist interfaces. The next step is to send the data in multiple forms, so that different interfaces can process it differently. For example, a phone choice menu might be sent both in sound ("Press 1 for new sales, press 2 for...") and in XML (" 1: New sales ... ").

What I found inspiring was Chris's optimism and sense of opportunities. The use of open platforms and cool technology could benefit lots of us in many different ways. I recommend the article. Unfortunately you may have to pay to read it - it's online at http://comjnl.oxfordjournals.org/cgi/content/abstract/50/3/274.

Webinar: The Semantic Web in Industry

2007-05-15T18:12:00.000+01:00

Our next webinar will look at a technology that is only just beginning to be deployed in industrial applications. The Semantic Web allows meanings to be attached to data and text and users to look for content by querying these annotations. At its simplest this should mean no more scrolling through pages of search results. More sophisticated uses include enabling service-oriented markets and automating aspects of data integration.

This seminar, which will take place on Thursday May 24th, will describe the principles of the semantic web and show how it can already be applied to real industry use cases.

John Davies of BT will give a brief introduction to Semantic Web technology and show how it can be applied in industry, focussing on four application areas: knowledge management, information integration, service-oriented environments and applications in the health sector.

Paul Walsh of Segala will show how semantic Content Labels improve trust when browsing, by letting users discover which sites are making assertions about compliance with standards or codes of conduct without having to actually visit each site.

As always, I'll be taking questions during the broadcast and we’ll have time for discussion at the end. You're also welcome to send me questions in advance as well. The broadcast will be at 2:30pm GMT on Thursday May 24th and you can register to attend at http://mediazone.brighttalk.com/event/gridcomputingnow/38913e1d6a-517-intro.

Initial reflections on OGF20 / EGEE User Forum

2007-05-14T17:58:00.000+01:00

So that was OGF20 and the 2nd EGEE User Forum. I was so busy there that I didn't have time or energy to blog. So much happened in such a short time that I'd have been hard put to keep up.

This turned out to be the best-attended GGF/OGF meeting ever, narrowly beating GGF5 (which was held in Edinburgh in 2002). We had over 900 people attend during the course of the week and a significant number stayed for all or most of the week. I haven't had feedback from all the sessions but I believe the workshops were well attended and I know the Grids Mean Business and the EGEE-specific sessions went very well.

A notable aspect of the week was the good interaction between the different communities. In particular, the colocation of the OGF and EGEE events has helped to show both sides where standards can apply or are needed. The commercial delegates helped to guide the requirements for standardisation work as well as sharing information on best practice. The exhibition space had a particularly good showing of EC-funded R&D projects as well as the EGEE demos, which certainly gave me a better picture of what these projects aim to achieve, in addition to stands from industry sponsors and UK e-Science.

Tony Hey and Peter Coveney gave stimulating keynote speeches, each worthy of its own blog entry. Mark Linesch, Ian Bird and Mario Campolargo gave overviews of OGF, EGEE and of the latest round of EU funding opportunities. Many other presentations raised interesting ideas and experiences. I was (of course) particularly pleased with the Grids Mean Business sessions, which were well attended and full of interested presentations, even though I say so myself. Gillian Law has written reports of these sessions which you can find on the Grid Computing Now! web site.

Grids Mean Business, Day 1

2007-05-08T23:15:00.000+01:00

It's the end of the first day of Grids Mean Business, the industry track that we've organised at the joint meeting of OGF20 and The EGEE User Forum. I'm too tired to write a detailed report but I'm very happy with today's sessions. We've had some great talks and really interesting discussions. Here's hoping that tomorrow is just as successful.

BBC Horizon programme on FireGrid

2007-04-19T11:56:00.000+01:00

Next Tuesday, April 24th, the BBC will be showing an episode of Horizon all about the FireGrid project. The vision behind FireGrid is that we can use the grid and HPC computing to model the spread of fire in buildings, to improve building design, firefighters' training and actual response in emergencies.

This is one of a number of grid projects looking at the use of real-time modelling to improve emergency response. Others are focussing on floods, earthquakes or the release of noxious substances into the atmosphere. In each case, the idea is to use lots of sensors to provide input to complex HPC models, which in turn provide forecasts to an emergency response control room.

There are many technical issues that need to be solved before such systems can be deployed live. We don't yet know the best way to guide simulations as new sensor input becomes available. We need better ways of allocating priority jobs to grid resources - it's clear that a traditional queue system won't give the response we need. The models themselves need to be adapted to work in this new computational environment. And of course there are security issues to consider. But this is interesting research with great potential benefit.

You can find more details about the BBC programme at http://www.bbc.co.uk/sn/tvradio/programmes/horizon/broadband/tx/firegrid/.

Webinar: Virtualisation and Service-Oriented Architecture - Building a cutting edge IT Infrastructure

2007-04-15T12:14:00.000+01:00

On Thursday (April 19th) I will be hosting another in our series of webinars. This week's seminar will feature case studies of two core technologies for building a cutting-edge infrastructure. Zafar Chaudry will describe how he used virtualisation to consolidate his disparate servers and storage provision into a manageable and scalable infrastructure. Then Mark Simpson will show how he deployed a service-oriented architecture to give a major financial institution far more control and scalability over its business processes.

Zafar and Mark come from very different sectors. Zafar is at the Liverpool Women's Hospital while Mark works for the business IT consultants Griffiths-Waite. I find it fascinating to see just how broad is the uptake of these new techniques - they really seem to be applicable across most sectors. This will be one issue that I'll explore in the discussion on Thursday. I'll also ask Mark and Zafar to comment on how virtualisation and SOA interact in more complex infrastructures - my belief is that complement each other well.

I'll be taking questions during the broadcast but you're also welcome to send me any in advance as well. The broadcast will be at 2:30 on Thursday and you can register to attend at http://mediazone.brighttalk.com/event/gridcomputingnow/7eacb53257-419-intro .

OGF20 & Grids Mean Business

2007-04-01T23:42:00.000+01:00

Much of the reason that I haven't blogged much recently is that I've been up to my eyes in organising OGF20 (colocated with the EGEE User Forum) and in particular the Grids Mean Business industry track. With just over a month to go, I'm pleased to report that we have well over 500 delegates registered and the programme of speakers is rather good. In addition, we have a range of workshops which will look at new developments in Grid middleware and operations.

Our two keynote speakers for OGF20 will be Tony Hey and Peter Coveney. Tony is VP of technical computing at Microsoft and will be giving us a new talk on the Social Grid. Peter is professor of computational chemistry at University College London and has stretched the use of Grid infrastructures to perform massive simulations of chemical processes. Both are excellent speakers and will give fascinating talks. In addition, Mark Linesch and Bob Jones will introduce the activities of OGF and EGEE respectively, while Mario Campolargo will present the European Commission's support for Grid infrastructure.

The Grids Mean Business programme will explore the various stages of uses of Grid in industry. In the first session, Paul Strong of Ebay and William Fellows & Steve Wallage of the 451 Group will set the scene, giving the motivation for and overview of grid. Following sessions will look at Paths for Adoption, Grid Markets, Scaling Up To The Enterprise Level and Collaborative Grids. Speakers will include John van Uden of Citigroup, Chris Swan of Credit Suisse and Brooklin Gore of Micron.

These and other speakers will share their experience from many sectors, including finance, the creative industries and engineering. We will also see some insights into future grid technologies for business. In addition, Ian Osborne and Nick Selby will lead discussions of issues in licensing and security, addressing the problems that IT managers must address when deploying grids.

OF20 and the EGEE User Forum will be at Manchester Central from May 7-11. The Grid Means Business track will be on May 8-9 and there is a special Enterprise Programme registration that is cheaper than that for the full week. If you haven't signed up already, the URL to visit is http://www.ogf.org/OGF20/events_regstrtn_ogf20.php.

Webinar: Distributed Systems in e-Health

2007-03-15T21:24:00.000Z

Last week I hosted a webinar on distributed systems in e-health. The speakers were Derek Hill, CEO of Ixico, and Michael Rigby, Professor of Health Information Strategy at Keele University. Derek launched Ixico three years ago as a spin-off of the UK e-Science Programme and now it is selling services internationally. Michael's project is still in the research stages but in my opinion it shows a more robust approach to managing health records than the current approach being implemented by the English National Health Service.

Take a look and see what you think - follow the links at http://grid.globalwatchonline.com/epicentric_portal/site/GRID/menuitem.c4f9e41660ec9e9b08a38510eb3e8a0c/.

Coping in a parallel world

2007-02-23T21:38:00.000Z

The heat produced by modern CPUs means that chip designers can no longer crank up the clock speed. But they can fit more processing cores on a chip. 2-core and 4-core x86 chips are already standard; Sun's SPARC chips have 8 or 16 cores, and Intel recently announced an 80-core prototype.

To get the best performance out of multi-core chips, especially once you get beyond 2 or 4 cores, you either have to write applications to make best use of those cores or write compilers and dynamic optimising systems that automatically transform your code to get that performance. The problem is, most programmers aren't very good at this and most of the tools they use aren't brilliant either.

I think this has immediate policy implications. CS departments should immediately, if they haven't already done so, make the teaching of parallel and distributed programming a required component of their undergraduate courses. Companies and professional bodies should encourage their staff to retrain. For this retraining, someone has to prepare and give the courses. Some government funding might kickstart the process and so ensure that the UK doesn't fall behind.

Several pundits, including Tony Hey of Microsoft, have suggested that experience from the supercomputer world could help, because supercomputers have been massively parallel for years. There is some truth in this, and organisations such as EPCC are already teaching parallel programming techniques. But there are differences too. The economics of the supercomputer world are rather old fashioned; there, the computer is still the expensive part of the system while programmers' time is relatively cheap. So programmers often tune each application for each new machine.

You can see this philosophy in much of the scientific grid world: people submit jobs that require a certain number of processors to run. Think about that. They're not asking for a certain amount of processing power, nor for their jobs to run in a certain time or for a certain cost. Their applications are programmed to run on a specific number of processors. To anyone outside that community, this approach is clearly crazy, and it certainly won't transfer to a world where multi-core processors are plentiful and programmers' time is expensive.

So the distributed (grid) world and the multicore CPU world face similar problems. Both need new programming models and tools. Fortunately there are many computer scientists who are investigating better ways to program distributed and parallel machines. I can't attempt to represent the field but some of my friends and colleagues are working on such problems. For example, Murray Cole at the University of Edinburgh has developed a model for transforming parallel programs on to different numbers of processors. In 2002, a team from Microsoft Research in Cambridge incorporated modern concurrency abstractions into a research version of C# - replacing the usual locks, semaphores and critical regions that Tony Hoare invented 40 year ago.

Here at the e-Science Institute, we are about to launch a theme - a series of workshops and visitors - on Distributed Programming Abstractions. I certainly hope that this will address some of the questions raised above and that soome of the people reading this will contribute. Another useful resource is this wiki at Berkeley, which also includes an interesting white paper.

OGF20 Registration is open

2007-02-22T23:07:00.000Z

Much of the reason I haven't blogged here more frequently is that I am programme chair of OGF20 and this is taking much of my time. So I'm pleased to report that registration for OGF20 is now open. Here is the official announcement. Note that early registration rates are available until March 15th.

Some aspects of the programme have been confirmed; others are still being finalised. I'll post news here over the next few weeks.

Registration is now open for OGF20 and the EGEE 2nd User Forum being held May 7-11 in Manchester, UK. Register on-line by visiting http://www.ogf.org/OGF20/events_regstrtn_ogf20.php

This event will feature:
• Keynote and Plenary presentations by leading grid luminaries
• Chartered Group Sessions including Standards Working Group Sessions and BoFs
• Enterprise Track including Requirements Alignment, Best Practices and Adoption Sessions
• e-Science Track featuring Community Workshops
• ‘Grids Means Business’ Industry Program showcasing business solutions and case studies
• Vendor Showcase and Exhibit Hall
• Demonstration and Poster Session Area
• Networking and Social Activities

About OGF20 and the EGEE User Forum
OGF20, presented by the Open Grid Forum and co-located with EGEE's 2nd User Forum, is the premier grid technologies event of 2007. At OGF20/EGEE UF, the global Grid community will gather to develop Grid standards, showcase real-world applications, workshop Enterprise and eSciences best practices and present business case studies and solutions. OGF20 is hosted by UK e-Science and the University of Manchester. Grids Means Business Program is sponsored by Grid Computing Now!

Webinar - Case Studies: IT Infrastructure for inter-enterprise collaboration

2007-02-14T20:51:00.000Z

Tomorrow (Thursday) I'll be hosting the next web seminar run by Grid Computing Now! , at 2:30pm UK time. We'll be looking at how grid technologies can help businesses collaborate on joint projects.

Mike Boniface of IT Innovation will explain how the SIMDAT project has enabled pharmaceutical and automotive companies to collaborate on product design. Tom Jackson of York University will describe how the BROADEN project is enabling Rolls-Royce to monitor after-sales performance.

Both projects use grid to manage distributed data and computer assets belonging to multiple organisations. The presentations will show how this leads to real business benefits.

You can join the seminar at http://mediazone.brighttalk.com/event/gridcomputingnow/a4f23670e1-260-intro.

Storage meets the Grid

2007-02-05T23:40:00.000Z

I'm just back from OGF19, which was very productive in a number of ways. One strand of interest is the continuing dialogue between the storage industry (represented by SNIA) and the grid world. This conversation has been developing slowly over the last year. I was on a panel at MSST 2006 that explored some aspects of this. SNIA were also present at GGF18 to explore where the two concerns meet. Both sides are still learning about each other, as they are both complex and changing technologies.

One obvious area of overlap is that of data replication. Many grid projects maintain replicas of data, to improve access times and/or to guard against loss. The classic example is the LHC Grid, but there are many other examples, particularly in the world of data librarines. Meanwhile, the storage industry supply replication systems for commerical data, specialising in backups and disaster recovery.

The two technologies work at different levels. Storage systems copy data from one disk block to another, with no knowledge about the data being copied. Grid systems copy files or database tables (in fact these two are themselves very different technologies), while maintaining application-specific metadata. Storage systems are usually deployed over LANs while grids are deployed over WANs, but this is not a hard and fast rule.

It remains to be seen which approach is best in which circumstances. Certainly there seem to be standards and tools from the storage industry that might be useful in the grid world, and vice versa. This will be an interesting conversation for some time.

ClimatePrediction.net on the BBC

2007-01-18T08:43:00.000Z

ClimatePrediction.net will feature on the BBC again this Sunday, in a television programme fronted by Sir David Attenborough. "Climate Change: Britain Under Threat" will broadcast the results of a BBC-sponsored experiment using ClimatePredication.net with a focus on how climate change is and will affect the UK.

The results will appear on the BBC Climate Change website after thebroadcast (and will also hopefully be in a Nature paper in a little while, pending peer review!)

Service Modeling Language

2007-01-16T23:11:00.000Z

Last night Heather Kreger gave an overview of the Service Modeling Language (SML) to members of the OGF's OGSA working group. The high-level view that I took away from this is that SML is a modelling effort based purely on XML (particularly XML Schema), rather than initiatives that map CIM or UML into XML. The advantage of this is that it has a cleaner rendering; for example, it doesn't have to translate the CIM or UML use of inheritance (which XML Schema doesn't directly support).

In practice, models can be rather large. SML also uses Schematron to enable models to be split across multiple documents, while still enabling validation against the models. A related activity, CML, is producing some core models in SML.

SML is backed by BEA, BMC, CA, Cisco, Dell, EMC, HP, IBM, Intel, Microsoft, and Sun, so certainly has industry support. They intend to submit the specification to a standards body in a few months time, although it will be a while after that before it is published as a recommended specification.

My personal opinion is that this may well be useful down the line. Anyone working on models now should not wait for this but should continue with whatever language and system you are currently using. SML won't be ready for production use for quite a while and will need to build a user community and some momentum before it sees widespread use. But with this range of backers, it might just happen.

Channel 4 news: The Big Bang Machine

2006-12-21T23:31:00.000Z

This seven minute report is mostly about the Large Hadron Collider at CERN, but any discussion of the LHC must mention the Grid. From our point of view, the novelty is not the technology - that's what we do - but seeing it explained on the national news.

OGF20 - May in Manchester (Call For Proposals)

2006-12-15T13:13:00.000Z

I'm very pleased to report that OGF20 will be held in the UK: to be precise, at the Manchester International Convention Centre from May 7-11 next year. It will be co-located with the 2nd EGEE User Forum. Grid Computing Now! are organising a two-day industry outreach track.

This is something I've been working on for almost a year. At first my main interest was to get the event in the UK and run the GCN! part of the programme. This got me involved in choosing the venue and general planning; I am now programme chair of the whole shebang and devoting quite a bit of time to planning the sessions.

We have just announced a Call For Proposals for presentations, panels and workshops. This call covers all aspects of industry and academic use of Grids, including the GCN! track. Full details are available on the OGF web site. I would encourage everyone to submit suggestions; the deadline for submission is February 9th. Obviously I'd be happy to discuss this with anyone who is interested.

I will post further updates here as they happen.

Virtualisation for green IT centres?

2006-12-12T18:34:00.000Z

Power consumption is already on the minds of IT managers, having replaced space as the main (non-staff) cost of running IT centres. The recent publication of the Stern report on the economic effects of climate change [full report; BBC online coverage] gives extra incentive to look at ways of reducing power use.

Virtualisation and Grid seem obvious tools to investigate for this end. Case studies have shown they can more than double the utilisation rates of existing servers. The Grid Computing Now! team are looking into this approach and we're keen to hear your stories. Please get in touch if you have experience of increasing server utilisation - or if you need to control your power bills!

Webinar: The business case for next generation IT architecture

2006-12-11T10:59:00.000Z

Last week we recorded and broadcast another Grid Computing Now! webinar. This featured Steve Wallage of The 451 Group and Shahid Mohammed of Marsh talking about the business case for Grid. Steve presented a distillation of the 451 Group's findings from talking to Grid users across a wide range of sectors. Shahid then gave us the benefit of his experience applying Grid in an e-commerce business. Both are good presenters and I recommend their talks.

You can see the webinar at http://brighttalk.com/comm/gridcomputingnow/3b03d97eee-2473-632-2319. This uses Flash and so should be compatible with most browsers.

Viewers submitted a good range of questions. A couple of people asked about the type of businesses - small, medium or large; finance, pharma or other - that can benefit from Grid. Shahid's talk demonstrated that reasonably small firms in a general commercial sector can benefit. Another question focussed on security. Both Steve and Shahid explained that Grid can be deployed within the enterprise, which can greatly reduce concerns about security.

We didn't discuss the definition of Grid - there are people who would maintain that an infrastructure deployed entirely within one organisation "isn't really Grid". As always at Grid Computing Now!, we are less interested in what the technology should be called than in what it can do for UK businesses. It is clear from these talks that the combination of virtualisation, dynamic resource allocation and service-oriented deployment can achieve significant business gains - and that is what matters most.

VO workshop report

2006-11-29T15:28:00.000Z

I've already mentioned last week's workshop on Virtual Organisations and Grids. You can now read my review of the event on the Grid Computing Now! web site.

GRIDtoday on UK e-Science at SC06

2006-11-29T15:14:00.000Z

I am gratified to see that GRIDtoday's summary of SC06 gave prominence to two UK projects - the social science simulation of population trends and the Grid monitor based on Google Earth. (I've mentioned both of these on this blog). The UK e-Science stand was only one of many in a large exhibition, so it's good to see that we made an impression.

VO 2.0?

2006-11-26T13:11:00.000Z

We just held an interesting and I think successful workshop on Virtual Organisations and Grids, in which we looked at the limitations of the current state of the art and what is needed to support industrial requirements. A short summary is in the works and will appear on the Grid Computing Now! web site in a couple of days, with a longer report to follow in due course.

What I want to raise here is a more speculative notion that arose during the discussions - what would be the Web 2.0 implementation of a Virtual Organisation (VO)? Could users easily create their own VO, invite their colleagues to join, actually manage distributed access in a distributed fashion, etc.? Obviously, being Web 2.0, the legal and contractual issues would have to be minimal, so this would apply more to open-source projects.

The nearest model I can think of is more the SourceForge approach, in which anyone can create a project and invite others to join. The SourceForge (or equivalent) software provides core functionality to support the process. I haven't particularly thought of SourceForge in the Web 2.0 model before and it does seem to fit the idea of people forming networks - the people are developers rather than "end users" and the network has a particular purpose, but the basic approach is the same. I wonder whether it would work for VOs?

Grid in the mass media?

2006-11-25T22:14:00.000Z

One last comment on SC06. We sent out a press release about Imperial College's new visual monitor of the EGEE Grid, which uses Google Earth to show the state of jobs submitted around the world and made a rather eye-catching display on the UK e-Science stand. This news even found its way into a paragraph on the science page of the Metro (the freesheet that is read by commuters on public transport across the UK).

This is how they rendered it:

GOOD GRID

The first map of the world's most powerful computer grid is to be unveiled this week. Nine of the largest grids are featured in the display at the Supercomputing festival in Tampa, Florida. The map uses Google Earth to pinpoint more than 300 sites on six continents. Grids are made up of hundreds or thousands of PCs, linked together to create a supercomputer. They are vital for scientists who need extra computing power to process large amounts of data.

As you can see, that's not quite right - Grids in general and EGEE in particular are not necessarily comprised of PCs - but its not too bad for a five-sentence summary of a complicated system.

ClimatePrediction.net

2006-11-21T18:32:00.000Z

I'm back from SC06 and almost recovered from jetlag. We're in the middle of a workshop on Virtual Organisations, which I'll report on in due course. Right now I want to highlight another of the projects we demonstrated on the UK e-Science stand at SC06.

ClimatePrediction.net uses home enthusiasts to run climate models on their PCs, inspired by SETI@Home and similar projects. This has the advantage that it can run multiple versions of models with slightly varying parameters and look at the most likely results.

It's good and worthwhile science but what makes the project particularly worth noting is the work they've done with education and the media. They have created a schools pack so that the system can be used in school teaching (that's pre-college school in USA terms). As a result they have several schools running models on their IT labs.

The project has also been fortunate to attract the interest of the BBC, who included it in a documentary on climate change and invited viewers to join in by running their own models. We played this documentary on our stand. The BBC are producing another programme to present the results, to be broadcast early next year.

At SC06, Carl Christensen of ClimatePrediction.net and David Anderson of BOINC and SETI@home gave a good discussion of what's required to run a successful project of this kind. The social questions of how to get a good user base have to be addressed as well as the technological problems. Several techniques exist but getting the BBC on your side must count as a particular useful one.

Progress on Grid licensing

2006-11-16T03:10:00.000Z

One of the events on our SC06 stand was a vendors' round-table discussion on Grid licensing. Representatives from Platform, NAG, Visual Numerics, Allinea, among others, contributed their thoughts on how we can produce more flexible licensing schemes for Grid use cases. There are two levels of concern: technical and business.

At the technical level, some vendors expressed frustration that the licensing systems that are currently available are too restrictive. So even though they want to offer more flexible business models, the underlying technology prevents them from doing so. There was broad agreement that a usage-based monitoring technology would provide a more flexible substrate.

There were several scenarios offerred at the business level. One of the simplest is to charge directly based on use. Ohio State University want to do this for local SME's who are just setting out down the road to Grid adoption; these SMEs don't want to commit to a large upfront cost when they are just experimenting with the technology. More complex use cases were suggested that would give more predictable costs for users and more predictable revenue streams for vendors. These would take advantage of the usage tracking technology but would not be constrained by it.

What made this discussion rather different from similar panels before is that there is a plan to work towards possible solutions. A first step will be to produce sample use cases for Grid licensing, together with outlines of what licenses would be required for each. This should help to frame the discussion and set requirements on any usage monitoring system.

My thanks to David Wallom and Laura McGinnis for organising this event and choosing UK e-Science to host it. I look forward to observing developments. I am certain that we will develop the discussion further at OGF20 in Manchester next year.

Politics, identity and privacy

2006-11-15T22:26:00.000Z

Kim Cameron's Identity weblog is worth keeping an RSS eye on. In this article he reports an Australian politician who is seeking to limit the uses of medical identity cards by law. I wish our UK politicians showed this level of understanding.

"Sim City for real"

2006-11-14T15:34:00.000Z

I'm particularly taken by one of the projects we're demonstrating on the UK e-Science Stand at SC06. "e-Social Science in action: a prototype geo-simulation portal" is, as the name suggests, applying Grid computing to the social sciences. It is simulating the interactions of individual households across an entire city to produce estimates of various factors such as health, housing, and car ownership. The simulations are based on UK census data. Users can then model the effects of various policy changes on these factors.

Under the hood, this is using standard Grid systems such as a compute cluster for running the simulations and SRB for managing the data. The system is easily scalable, so we might see more detailed simulations or see the system applied to larger cities such as London. There are some interesting research challenges left - I'm especially interested in the secure management of sensitive data - but the potential is clear.

The UK is at the leading edge of e-science in the social sciences and I'm pleased to have this demo on our stand at SC06.

SC06 - Supercomputing

2006-11-13T15:19:00.000Z

This week I'm at the SC06 Conference in Tampa, Florida, where I'm organising the content for the UK e-Science booth in the exhibition. This exhibition is large by the standards of most conferences, with booths from all the major vendors (IBM, Sun, HP, Microsoft, etc.) and research labs. It's a good opportunity to catch up with developments around the world.

Our booth has 10 demonstrations running continously. Researchers will explain to visitors what their science is all about and how the computing infrastructure helps them to achieve it. The demos range from physics (astronomy & particle physics) through to the social sciences, taking in the life sciences and computer science too.

We also have some interesting talks, including panel sessions on topics such as licensing and volunteer computing. We will also show a BBC horizon programme about climate change, which features one of our demos (climateprediction.net).

For more info, see http://www.nesc.ac.uk/events/sc06/.

ETSI take on Grid standards

2006-10-26T09:26:00.000+01:00

ETSI recently announced they were entering the Grid standardisation space. This week I got the chance to learn more about this, thanks to Mike Fisher of BT, who leads the relevant ETSI Technical Committee. He explained that ETSI has a protocol and testing competence centre, along with the methods and experience to take specifications written in English and write conformance test suites that allow precise checks of interoperability. This is certainly a facility that the OGF does not have; indeed, the OGSA working group has carefully framed its work to avoid making any claim about conformance. So if the ETSI initiative can agree use cases and find sufficient consensus on standards, perhaps building on OGF work, it could add a valuable facility to the Grids standards community.

Next Generation Networks and Grids

2006-10-24T19:46:00.000+01:00

This week I attended a meeting at the ITU in Geneva on the topic of Next Generation Networks (NGNs) and Grids. NGNs are developed by the telco industry to provide a range of services over IP networks. The key to their design is the separation of service provision from transport mechanisms. Traffic may be routed over different types of network, each of which provides a common interface. The same traffic may in turn implement all sorts of services, including voice, television, videoconferencing, data transfer, multiplayer games, and whatever application designers come up with in the future.

So NGNs have a virtualised infrastructure providing a range of user services. They are using Web Services technology and are tackling issues such as security (authentication, authorisation, audit), accounting & billing, service description and deployment. Which all sounds familiar from the Grid world. That's what this workshop was about. In particular, it was to investigate opportunites for the ITU (International Telecommunications Union) and the OGF (Open Grid Forum) to work together. Of necessity, much of the time was spent on mutual education - I certainly knew little about NGNs before (although I knew something of BTs incarnation of NGNs, their "21st Century Network").

I was particularly interested in the presentations about Quality of Service (QoS). This is absolutely crucial to NGNs, because different transport mechanisms have different performance characteristics, while different services have different requirements. For example, data services usually demand zero loss of information, while voice can be more relaxed about packet loss provided that enough arrive in time. IPTV has very stringent requirements on both packet loss and arrival time. So the NGN architecture details how QoS requirements can be passed from services to transport mechanisms via a central abstraction.

QoS hasn't received so much attention in the Grid world, but I believe that it will be vital there. Currently Grids are still being installed by experts and used in circumstances where "best effort" performance is satisfactory. (Within a single organisation, "best effort" may be very good quality). There is research that addresses how to specify and establish service-level agreements, but fpr the most part this hasn't been deployed in production grids or entered the standardisation process.

The workshop finished with a discussion of how the ITU and OGF could co-operate on producing standards that address some well-chosen use cases. This seems a potentially valuable partnership. Certainly the ITU can bring expertise in many areas to the Grid world, including audit and accounting. Perhaps the partnership will also advance the state of QoS specification too.

Web Seminar

2006-10-06T20:36:00.000+01:00

This week I chaired a web seminar on Grid Markets. We had two excellent speakers: Dennis Kehoe from the AIMES Centre in Liverpool and John Darlington from the Imperial College Internet Centre. Between them they covered what is possible for businesses now and what we might expect in the future. After the presentations we had a fruitful discussion and answered some questions from the viewers.

This was the first time I've taken part in a web seminar and I know I made some mistakes - such as not introducing myself! A little more briefing by the hosting company would have helped but at least I'll know better for next time. Perhaps I'll even dare to watch myself on the stored video. In the meantime, if you would like to watch the seminar, click to http://brighttalk.com/event/gridcomputingnow/9b8619251a-130-intro-2-1.

Mashing in the UK?

2006-09-30T15:17:00.000+01:00

In the last two weeks, several people have suggested to me that the UK could take more advantage of data collected here. Public agencies such as the Met. Office, Ordnance Survey, Transport for London, the British Crime Survey, the Land Registry and others hold data on various aspects of our lives. Private companies such as Experian and major retailers collect much more. If these were available as web services (in the broadest sense), inventive developers could mix and match them in new ways, displaying the results in meaningful ways (such as in map form).

Such activities are already popular in the USA, where more similar data sets are available. A widely-quoted example is Chicago Crime and there are many more. The question is, how can the UK catch up?

A few example projects are already underway. The DEWS project is combining Met Office data with health information and (separately) the coastguard. The Ordnance Survey is holding a Mashups day. Paul Longley's team at UCL caught the public eye with their Surname Profiler and eSociety Profiler. The Department for Transport has funded similar work - e.g. travel time maps and a data mashing workshop.

Despite these examples, "mashing" has yet to really take off in the UK. A major block is that some of the data is only available commercially - for example, there is an ongoing debate about access to Ordnance Survey data. I'm not against people making money - far from it - but the UK could benefit from providing would-be entrepeneurs with better access to data sets for experimental use. Licences could then be charged on commercial ventures that arise as a result, benefitting both the entrepeneurs and the data curators.

We also need to increase the provision of data sets in the first place, preferably using a small set of standard access mechanisms. Web services seem a good candidate, especially where secure access may be an issue. Alternatively, plain Web 2.0 interfaces may be good enough to experiment with.

The whole area could benefit from some joined-up thinking and collaboration across a range of agencies. Provided, of course, that this hurries things along rather than holding it up while people argue.

2006-09-21T12:45:00.000+01:00

This year's e-Science All-Hands Meeting has had a really good buzz going on. The conference is the same size as last week's OGF/GridWorld meeting in Washington, in a friendlier venue and with lots going on. This year's talks & demos seem more about applications than technology, which suggests that e-Science is maturing and becoming viable for more widespread use.

I'm here mainly wearing my Grid Computing Now! hat. We have a booth in which we present the DTI-funded R&D projects alongside our industry case studies. We've had quite a good interest from the delegates. From my point of view it's been good to see what the R&D projects are doing now that they are up and running. We had excellent demos from BROADEN, DEWS and Healthcare@Home.

Healthcare@Home is developing a system whereby diabetes patients can monitor key indicators (such as glucose level in the blood) at home, with readings transmitted to a server via mobile phone. This enables clinicians to track their progress and risk analysis software to detect problems.

DEWS is linking weather forecast data with health data and coastguard services. One application is to give better assistance to search and rescure operations, allowing for the effect of wind and tide. Another is to forecast weather-related health problems. The key idea here is to make better use of available data - a theme I;ll return to in a later posting.

BROADEN is looking at the analysis of vibration data from aircraft engines in order to detect problems before they become critical. This is a development of the earlier DAME e-Science project. The same technology can be used for other types of data - there's another possible health link here as heartbeats are a possible application.

Far more happened than I can summarise in one blog entry. The hard thing has been finding time to follow up all the discussions I've had during the week - whenever I've sat down to make notes there's always been someone else interesting to talk to.

OGF misses a chance with Pharma & EDA

2006-09-14T17:01:00.000+01:00

GGF18 included a series of sessions that were intended to capture requirements from the Pharmaceutical and Electronic Design Automation industries, with the intended output of guiding and prioritising the standards work in OGF. These started well. A session on EDA produced a small set of priotised requirements. Then the Pharma folks had their turn and produced a similar set, with some overlaps.

Unfortunately things went wrong after that. These requirements were added to a veritable soup of other requirements, some of which were reasonable and some of which were high-level and vague. Then people made an attempt to order all these, which ran out of steam because the list was too long. The original focus from the industry speakers was lost.

The goal of these sessions was highly desirable but this time it wasn't achieved. Afterwards I heard industry representatives bemoaning the outcome. They received the impression that the OGF is rather an academic organisation that is not focussed on the needs of industry. I know that isn't the case but this is the image that was generated.

I wouldn't want the OGF to give up on this. The dedicated sessions worked well; the OGF just needs a more direct mechanism for acknowledging and addressing the issues raised.

UK e-Science wins GridToday award

2006-09-14T13:52:00.000+01:00

UK e-Science won an award in the first GridToday Readers' and Editors' Choice awards - see http://www.gridtoday.com/grid/888759.html.

First GCN! Webinar

2006-09-13T22:18:00.000+01:00

I'm very pleased to say that the Grid Computing Now! KTN (see sidebar) will run our first web seminar on October 4. The title is The next-generation internet: Business opportunities and challenges for Grid markets.

This webinar will describe new mechanisms for business enabled by the next generation internet and computing infrastructures. Prof. Dennis Kehoe, Director of AiMeS (Advanced Internet Methods and Emergent Systems) will talk about current opportunities and challenges for utility computing and software services. He will be followed by Prof. John Darlington, Imperial College London, with a look at a future for Grid markets.

We expect this to be of interest to MDs of SMEs, CIOs, CTOs, business analysts and consultants. For more details, please contact tara.kelly@intellectuk.org.

OGSA + EGA = ?

2006-09-13T20:40:00.000+01:00

What does the merger of the Global Grid Forum and the Enterprise Grid Alliance mean for the people actually developing Grid standards? This has been addressed in a couple of sessions here at GGF18. The first indications is that their efforts mesh quite well - or at least they avoid much conflict.

The EGA have produced a reference model, a security model and use cases, concentrating on the management of data centres. Their focus on the provisioning of data servers should nicely complement the existing OGSA work, which addresses provisioning of compute servers rather than data and considers only higher-level data services. Similarly the EGA reference model describes the components of a data centre at a higher level than current CIM work, which is where OGSA is currently concentrating its efforts. And of course the Enterprise requirements have a major focus on SLAs, QoS, policy management, billing and chargeback - long recognised by OGSA but not something they've got around to addressing. Even the glossaries produced by the two bodies hardly clash - so we can't use the excuse that we don't understand each other!

One area where the two approaches currently differ is in the area of management. In OGSA, each service is typically self-managed, whereas the EGA typically has one service managing multiple others. This will require some work to reconcile. In the meantime, the two groups have agreed to compare their provisioning models to see whether they can produce a common solution.

So the initial prognosis is hopeful. As always, the devil will be in the detail. It will be interesting to see what comes out of the detailed discussions.

GGF18: Grid in the Enterprise?

2006-09-11T21:11:00.000+01:00

This morning's sessions at GGF18 in Washington showed the interesting contrast between the academic and commercial ideas of what "Grid computing" means. The academic view came from Ian Foster's classic paper, The Anatomy of the Grid, which was referenced by Dan Atkins of the USA National Science Foundation in his keynote speech. This view stresses the importance of a Virtual Organisation - a group of people from different organisations working together on a project or task. The term "Virtual Organisation" comes from business economics, so is not unknown to the commercial world - classic examples are the supply chain of a manufacturing process or the various organisations working together on a civil engineering project - but in the Grid world, VOs are more commonly seen in science Grids.

The commercial view was presented during the session on Enterprise Grid Requirements. Paul Strong of Ebay put it clearly: the focus is on removing silos between sub-organisations and replacing dedicated servers with virtualised resources. The approach is the same as the science use case: virtualisation of resources, heterogeneity, dynamic mapping of application instances, hot deployment of services - all the technologies that form what I call Computing as commodity. The difference is that these might all be deployed within a single enterprise; virtual organisations are possible but not necessary.

I believe these differences arise from the contrasting natures of the communities. In the academic world it is now common to use Beowulf clusters as high-performance compute resources and these really don't bring much new to the IT world. They take "jobs" from a queue and run them on multiple processors in much the same way as "traditional" supercomputers. So Foster's definition stresses the difference from such systems, focusing on collaboration between different organisations. The commercial world often has multiple machines and servers running within a single organisation. Virtualising these is qualitatively different from running a Beowulf cluster and so the Grid metaphor is more appropriate.

The OGSA Data Architecture

2006-09-03T22:21:00.000+01:00

One of my many hats is as co-chair of the OGSA Data working group of the Open Grid Forum. We work on the data-oriented aspects of the Open Grid Services Architecture (see also here), covering the description, movement, access, replication, federation and storage of data in a Grid environment. Our aim is to provide a framework that links existing standards with those in development, and to show gaps that are not currently covered by standards work.

We had a fruitful face-to-face meeting here in Edinburgh last week, in which we concentrated on the descriptions of the interfaces in the architecture. The architecture is really a toolbox of data services that can be composed in various ways to address a range of use cases. As I expected, this focus helped us to to describe in greater detail how the various components will work together.

I believe we should have a draft of the architecture ready for public comment soon. I'm looking forward to actually completing this work. It's been a slow process, as we have had to bring various distributed groups on board throughout, but that's the only way to create a standard that will actually have any effect in the real world.

Computing as a commodity

2006-08-31T21:45:00.000+01:00

There is a key concept that links Utility Computing, Grid, Service-Oriented Architecture, Virtualisation and several more slippery terms as well. That concept is of computing as a commodity. This is not the same as computers being a commodity - we're all used to that, whether we're buying a home PC by mail order or building even mid-range servers from similar machines. Commodity computing lets us use computing resources or services when we need them and only when we need them. It takes several forms.

The most obvious example is Utility Computing, where a vendor (such as Sun or Amazon) provide you with CPU cycles or storage space and you pay for what you use. Web hosting companies operate on a similar basis, just slightly higher up the software stack. But other examples show that the same basic idea can operate within a company.

Trader Media are a classic example of how computing can be a virtual commodity within a company. They started with the classic setup of separate servers for web, database, business logic & development for each web product, with a complete set of duplicates for backup and failover. They now have a common set of servers with services deployed over them as necessary. Their server utilisation has risen from the industry average of 10% to about 50% and they are aiming for higher rates still. Larger industries can supply services to whichever departments need them, using internal accounting mechanisms much as in the inter-enterprise cases. So we can have a separation between servers and services.

Similarly, companies can provide services themselves as commodities. These can become parts of larger systems stretching across multiple companies. This is a small market as yet but there are enough examples out there to prove the concept.

All these scenarios are underpinned by common technological ideas. So although one term may become unfashionable and another take its place, or while one vendor's system may lose favour and another become more popular, we can see that the key concept that links them all remains the same. Computing is becoming a commodity.

Welcome

2006-08-30T23:42:00.000+01:00

Welcome to Distributed Thinking. In this blog I will present my thoughts and experiences in the world of distributed computing, Grid, e-Science, utility computing, service-oriented architectures, virtualisation, Web 2.0 and other similar technologies. Via my work at the National e-Science Centre, I'm involved in several projects and other activities which have snippets of interest to a broader community. Some of these appear in more formal forums, including www.gridcomputingnow.org; this blog gives me the chance to mention smaller items, to discuss things of interest and to express the occasional personal opinion. I hope that you enjoy reading it.