Skip to main content

Coping in a parallel world

The heat produced by modern CPUs means that chip designers can no longer crank up the clock speed. But they can fit more processing cores on a chip. 2-core and 4-core x86 chips are already standard; Sun's SPARC chips have 8 or 16 cores, and Intel recently announced an 80-core prototype.

To get the best performance out of multi-core chips, especially once you get beyond 2 or 4 cores, you either have to write applications to make best use of those cores or write compilers and dynamic optimising systems that automatically transform your code to get that performance. The problem is, most programmers aren't very good at this and most of the tools they use aren't brilliant either.

I think this has immediate policy implications. CS departments should immediately, if they haven't already done so, make the teaching of parallel and distributed programming a required component of their undergraduate courses. Companies and professional bodies should encourage their staff to retrain. For this retraining, someone has to prepare and give the courses. Some government funding might kickstart the process and so ensure that the UK doesn't fall behind.

Several pundits, including Tony Hey of Microsoft, have suggested that experience from the supercomputer world could help, because supercomputers have been massively parallel for years. There is some truth in this, and organisations such as EPCC are already teaching parallel programming techniques. But there are differences too. The economics of the supercomputer world are rather old fashioned; there, the computer is still the expensive part of the system while programmers' time is relatively cheap. So programmers often tune each application for each new machine.

You can see this philosophy in much of the scientific grid world: people submit jobs that require a certain number of processors to run. Think about that. They're not asking for a certain amount of processing power, nor for their jobs to run in a certain time or for a certain cost. Their applications are programmed to run on a specific number of processors. To anyone outside that community, this approach is clearly crazy, and it certainly won't transfer to a world where multi-core processors are plentiful and programmers' time is expensive.

So the distributed (grid) world and the multicore CPU world face similar problems. Both need new programming models and tools. Fortunately there are many computer scientists who are investigating better ways to program distributed and parallel machines. I can't attempt to represent the field but some of my friends and colleagues are working on such problems. For example, Murray Cole at the University of Edinburgh has developed a model for transforming parallel programs on to different numbers of processors. In 2002, a team from Microsoft Research in Cambridge incorporated modern concurrency abstractions into a research version of C# - replacing the usual locks, semaphores and critical regions that Tony Hoare invented 40 year ago.

Here at the e-Science Institute, we are about to launch a theme - a series of workshops and visitors - on Distributed Programming Abstractions. I certainly hope that this will address some of the questions raised above and that soome of the people reading this will contribute. Another useful resource is this wiki at Berkeley, which also includes an interesting white paper.


I wholeheartedly agree with the challenge that Dave offers here. We are building a largescale, distributed computing infrastructure capable of wholesale parallel operation for both transactional and computational computing. It should be a matter of course that significant applications are designed with multiple-thread operation in mind. This is not a new problem in CS, its just that we've never had much opportunity in the past to do this for real!!
Dave Berry said…
Intel have a white paper than makes these points and many more, at

Thoams Sterling addresses these issues too. His recent keynote points out some of the differences as well as the similarities. A major difference is the bottleneck in memory access on multi-core CPUs. See

Popular posts from this blog

Presentation: Putting IT all together

This is a presentation I gave to an audience of University staff: 

In this seminar, I invite you to consider what the University’s online services would be like, if we worked together to design them from the perspective of the student or member of staff who will use them, instead of designing them around the organisational units that provide them. I’ll start with how the services might appear to that student or member of staff, then work back from there to show what this implies for how we work, how we manage our data, and how we integrate our IT systems. It might even lead to changes in our organisational structure.

Our online services make a vital and valued contribution to the work of our students and staff. I argue that with better integration, more consistent user interfaces, and shared data, this contribution could be significantly enhanced.

This practice is called “Enterprise Architecture”. I’ll describe how it consults multiple organisational units and defines a framework …

Service Excellence, Digital Transformation and Enterprise Architecture

Our University Secretary has sponsored a major review of the University’s administrative processes, coining the banner “Service Excellence”.  The aim is to look at the services we provide to staff and students with a fresh eye, making them more effective, more efficient, and focussed on the user rather than administrative convenience.

Our CIO is sponsoring a similar programme called “Digital Transformation”. This will replace old paper-based processes, starting with the question of what would processes look like if we designed them afresh for the modern connected world.  The aim is to make processes that are more focussed on the user and hence more effective and efficient.

Both of these ambitious programmes will need an effective enterprise architecture, if they are to succeed.  Digital Transformation is intrinsically about using opportunities provided by new technology to improve services and, as such, it requires effective technology services to make data available when needed, to pro…

Not so simple...

A common approach to explaining the benefits of Enterprise Architecture is to draw two diagrams: one that shows a complicated mess of interconnections, and one that shows a nicely layered set of blocks. Something like this one, which came from some consultants:

I've never felt entirely happy with this approach.  Yes, we do want to remove as much of the needless complexity and ad-hoc design that litters the existing architecture.  Yes, we do want to simplify the architecture and make it more consistent and intelligible.  But the simplicity of the block diagram shown here is unobtainable in the vast majority of real enterprises.  We have a mixture of in-house development and different third-party systems, some hosted in-house, some on cloud infrastructure and some accessed as software-as-a-service.  For all the talk of standards, vendors use different authentication systems, different integration systems, and different user interfaces.

So the simple block diagram is, basically, a l…