And the two most exciting happenings:

1. I entered two drawings and won twice! (an X-box Elite from Microsoft *YAY* (thanks Fab!) and some USB stick from teragrid (I hope they’ll send it to me))
2. I was complaining about the low threading support (only four) in the Power 7 and the random guy next to me started to explain why. It turned out that this guy was Burton Smith! He entered my personal hall of fame after his keynote at SPAA 2008. This man knows exactly what he talks about and we chatted a long time (until the show closed) about network topologies and routing. It was surprising to me that he mentioned the same fundamental insight in topologies that I had about a month ago independently. He also studied Cayley graphs and friends … five years ago (D’oh, I’m too young!).

This week’s MPI Forum was very interesting! Marc Snir presented his convincing hybrid proposal. It’s really nice and orthogonal to the current standard. It needs some minor polishing and an implementation and seems ready to go.

We had some incremental discussions in the collectives working group but nothing very exciting. I think it is time to look for applications/systems that can benefit from the sparse collective proposal. Sameer Kumar sent me a very interesting paper which seems to be what we need! We also assimilated the Topology chapter into the collectives working group (now called collectives and topology working group — short colltop ).

The RMA discussions were helpful this time and motivated me to summarize all ideas that floated in my head into a patch to the MPI-2.2 standard document during the weekend. I’ll post it to the RMA list and will see what happens. I think the RMA interface in MPI-2.0 is rather elegant and only needs some minor tweaks and some semantic changes to make it useful.

The MPI-3 discussions were going in circles (again). We went back and forth if we should call our next release (which contains nonblocking collectives and probably support for hybrid environments) MPI 2.3 or MPI 3.0 draft. We didn’t come to any conclusion. The only decision we made (I think, we didn’t vote though) is that we don’t want to break source compatibility in the next revision yet. I’d like to call it 2.3 then because having a 3.0 draft means that 3.0 will be a similar release and we would probably break compatibility in 3.1 which doesn’t seem to useful. 2.3 also gives the user a better impression that it’s not a revolutionary new thing (e.g., fault tolerant). However, I don’t have a too strng opinion, I just have some users who want nonblocking collectives in a release that is at least source compatible.

Another really annoying thing is the whole MPI_Count story. I have to admit that I was in favor of it at the beginning because abstraction seems right to me, however, I am now really against it due to several reasons: (1) the workaround is trivial and causes negligible overhead, (2) it breaks source compatibility which is a total no-go, and (3) it causes all kinds of Fortran 77 problems (it seems that this is the reason why int was selected in the first place). Could we just withdraw the ticket please?

• Mondrian’s “A resource optimized remote-memory-access architecture for low-latency communication” (I need to talk to those guys (I did )
• Argonne’s “Improving Resource Availability By Relaxing Network Allocation Constraints on the Blue Gene/P” (I need to read the paper because I missed the talk due to chaotic re-scheduling, but Narayan’s 5-minute elevator pitch summary seemed very interesting)
• Prof. Resch’s keynote on “Simulation Performance through Parallelism -Challenges and Options” (he even mentioned the German Pirate party which I really enjoyed!)
• Brice work with Argonne on “Cache-Efficient, Intranode Large-Message MPI Communication with MPICH2-Nemesis”
• Argonne’s “End-to-End Study of Parallel Volume Rendering on the IBM Blue Gene/P” (yes, another excellent Argonne talk right before my presentation )

Here are some nice pictures:

My talk at the last day was a real success (very well attended, even though it was the last talk in the conference)! It’s good to have friends (and a good talk from Argonne right before mine ). Btw. two of the three talks in the (only) “Information Retrieval” session were completely misplaced and had nothing to do with it, weird …

My co-author, friendly driver, and camera-man and me in front of the parliament.

The conference was nice, I learned about formal verification of MPI programs in the first day’s tutorial. This technique seems really nice for non-deterministic MPI programs (how many are there?) but there are certainly some open problems (similar to the state explosion of thread-checkers). The remainder of the conference was very nice and it feels good to meet the usual MPI suspects again. Some highlights were in my opinion:

• Edgar’s “VolpexMPI: an MPI Library for Execution of Parallel Applications on Volatile Nodes” (indeterminism is an interesting discussion in this context)
• Rusty’s keynote on “Using MPI to Implement Scalable Libraries” (which I suspect could use collectives)
• Argonne’s “Processing MPI Datatypes Outside MPI” (could be very very useful for LibNBC)
• and Steven’s invited talk on “Formal Verification for Scientific Computing: Trends and Progress” (an excellent overview for our comunity)

The whole crowd:

Unfortunately, I had to leave before the MPI Forum information session to catch my flight.
Videos of many talks are available at Videos. All-in-all, it was worth to attend. Next year’s EuroMPI (yes, the conference was finally renamed after the second year in a row without a PVM paper) will be in Stuttgart. So stay tuned and submit papers!

Here are the results of the official vote on the main document:
yes: 14
no: 1
abstain: 2 (did not participate)

The votes by chapter will be online soon.

The feature-set of the standard did not change. I posted it earlier here and Jeff also. But it’s official now! Implementors should now get everything implemented so that all users can enjoy the new features.

Here is a local copy (mirror) of the official document: mpi-report-2_2.pdf (the creation date might change)

One downside is that we already have errata items for things that were discovered too late in the process. This seems odd, however, we decided that we should not break our own rules. And even if the standard says that an MPI_BOOL is 4 bytes, we had to close the door for changes at some point. The errata (MPI_BOOL is one byte) will be voted on and posted soon on the main webpage.

Rolf will publish the MPI 2.2 book (like he did for MPI 2.1) and it will be available at Supercomputing 2009. I already ordered my copy .

And now we’re moving on to MPI 3, so stay tuned (or participate in the forum)!

Some tickets that I think are worth discussing are:

Add a local Reduction Function - this enables the user to use MPI reduction operations locally (without communication). This is very useful for library implementors (e.g., implementing new collective routines on top of MPI) – PASSED!

Regular (non-vector) version of MPI_Reduce_scatter - this addresses a kind of missing functionality. The current Reduce_scatter should be Reduce_scatterv … but it isn’t. Anyway, if you ever asked yourself why the heck should I use Reduce_scatter then think about parallel matrix multiplication! An example is attached to the ticket. – PASSED!

Add MPI_IN_PLACE option to Alltoall – nobody knows why this is not in MPI-2. I suppose that it seemed complicated to implement (an optimized implementation is indeed NP hard), but we have a simple (non-optimal, linear time) algorithm to do it. It’s attached to the ticket . – PASSED!

Fix Scalability Issues in Graph Topology Interface – this is in my opinion the most interesting/important addition in MPI-2.2. The graph topology interface in MPI-2.1 is horribly broken in that every process needs to provide the *full* graph to the library (which even in sparse graphs leads to $\Omega(P)$ memory *per node*). I think we have an elegant fix that enables fully distributed specification of the graph as well as each node specifies its neighbors. This will be even more interesting in MPI-3, when we start to use the topology as communication context. – PASSED!

Extending MPI_COMM_CREATE to create several disjoint sub-communicators from an intracommunicator -Neat feature that allows you to create multiple communicators with a single call! – PASSED!

Add MPI_IN_PLACE option to Exscan – again, don’t know why this is missing in MPI-2.0. The rationale that is given is not convincing. PASSED!

Define a new MPI_Count Datatype – MPI-2.1 can’t send more than 2^31 (=2 Mio) objects on 32-bit systems right now – we should fix that! However, we had to move this to MPI-3 due to several issues that came up during the implementation (most likely ABI issues) POSTPONED! It feels really good to have this strict implementation requirement! We will certainly have this important fix in MPI-3!

Add const Keyword to the C bindings – most discussed feature I guess – I am not sure about the consequences yet, but it seems nice to me (so far). – POSTPONED! We moved this to MPI-3 because some part of the Forum wasn’t sure about the consequences. I am personally also going back and forth, the issue with strided datatypes seems really worrysome.

Allow concurrent access to send buffer – most programmers probably did not know that this is illegal, but it certainly is in MPI<=2.0. For example:

int sendbuf;

MPI_Request req[2];
MPI_Isend(&sendbuf, 1, MPI_INT, 1, 1, MPI_COMM_WORLD, &req[0]);

MPI_Isend(&sendbuf, 1, MPI_INT, 2, 1, MPI_COMM_WORLD, &req[1]);
MPI_Waitall(2, &req);

is not valid! Two threads are also not allowed to concurrently send the same buffer. This proposal will allow such access. - PASSED!

MPI_Request_free bad advice to users – I personally think that MPI_Request_free is dangerous (especially in the context of threads) and does not provide much to the user. But we can’t get rid of it. … so let’s discourage users to use it! – PASSED!

Deprecate the C++ bindings – that’s funny, isn’t it? But look at the current C++ bindings, they’re nothing more then pimped C bindings and only create problems. Real C++ programmers would use Boot.MPI (which internally uses the C bindings ), right? – PASSED (even though I voted against it )

Something odd happened to New Predefined Datatypes. We found a small typo in the ticket (MPI_C_BOOL should be 1 instead of 4 bytes). However, it wasn’t small enough that we could just change it (the process doesn’t allow significant changes after the first vote). It was now voted in with this bug (I abstained after the heavy discussion though) and it’s also too late to file a new ticket to fix this bug. However, we will have an errata item that will clarify this. It might sound strange, but I’m very happy that we stick to our principles and don’t change anything without proper reviews (these reviews between the meetings where vendors could get user-feedback have influences tickets quite a lot in the past). But still PASSED!

For all tickets and votes, see MPI Forum votes!

I’m very satisfied with the way the Forum works (Bill Gropp is doing a great job with MPI-2.2), I hear about other standardization bodies and have to say that our rules seem very sophisticated. I think MPI-2.2 will be a nice new standard which is not only a bugfix but offers new opportunities to library developers and users (see the tickets above). We are also planning to have a book again (perhaps with an editorial comment addressing the issue in ticket 18 (MPI_C_BOOL)!

Then, we decided to go for MPI-2.2 which might change something but doesn’t break anything! We’re still unsure if we allow ABI changes though. But MPI-2.2 will certainly be source-code compliant (so a recompile might be required – which seems not that bad to me). So the MPI-2.2 process is supposed to guarantee quality. We use the trac system here at IU to manage the changes. Each “ticket” represents a change which has to be reviewed unofficially by at least four members of the Forum. Then, it can be read in front of the whole Forum at any meeting. Then, we have a first and a second vote and each successful ticket has to pass both. At the end, we vote for the inclusion of each chapter in MPI-2.2. Each ticket must go through this procedure and only a single state change is allowed during each meeting. This gives the Forum and the public a long time (>8 months) to review the proposals carefully. We also require an (open-source) implementation of each proposed change.

We’re discussing MPI-2.2 since several meetings – but the last (April’09) meeting was an important milestone! Since we plan to release MPI-2.2 at this year’s EuroPVM, we had to close the door. This means effectively, all tickets that have not been read in this meeting are postponed to MPI-3. I think we did pretty well and we’re within our schedule.

Some tickets that I think are interesting are:

Add a local Reduction Function - this enables the user to use MPI reduction operations locally (without communication). This is very useful for library implementors (e.g., implementing new collective routines on top of MPI)

Regular (non-vector) version of MPI_Reduce_scatter - this addresses a kind of missing functionality. The current Reduce_scatter should be Reduce_scatterv … but it isn’t. Anyway, if you ever asked yourself why the heck should I use Reduce_scatter then think about parallel matrix multiplication!An example is attached to the ticket.

Add MPI_IN_PLACE option to Alltoall – nobody knows why this is not in MPI-2. I suppose that it seemed complicated to implement (an optimized implementation is indeed NP hard), but we have a simple (non-optimal, linear time) algorithm to do it. It’s attached to the ticket .

Fix Scalability Issues in Graph Topology Interface – this is in my opinion the most interesting/important addition in MPI-2.2. The graph topology interface in MPI-2.1 is horribly broken in that every process needs to provide the *full* graph to the library (which even in sparse graphs leads to $\Omega(P)$ memory *per node*). I think we have an elegant fix that enables fully distributed specification of the graph as well as each node specifies its neighbors. This will be even more interesting in MPI-3, when we start to use the topology as communication context.

Extending MPI_COMM_CREATE to create several disjoint sub-communicators from an intracommunicator -Neat feature that allows you to create multiple communicators with a single call!

Add MPI_IN_PLACE option to Exscan – again, don’t know why this is missing. The rationale that is given is not convincing.
Define a new MPI_Count Datatype – MPI-2.1 can’t send more than 2^31 (=2 Mio) objects on 32-bit systems right now – we should fix that!
Add const Keyword to the C bindings – most discussed feature I guess – I am not sure about the consequences yet, but it seems nice to me (so far).

Allow concurrent access to send buffer – most programmers probably did not know that this is illegal, but it certainly is. For example:

int sendbuf;

MPI_Request req[2];
MPI_Isend(&sendbuf, 1, MPI_INT, 1, 1, MPI_COMM_WORLD, &req[0]);

MPI_Isend(&sendbuf, 1, MPI_INT, 2, 1, MPI_COMM_WORLD, &req[1]);
MPI_Waitall(2, &req);

is not valid! Two threads are also not allowed to concurrently send the same buffer. This proposal will allow such access.
MPI_Request_free bad advice to users – I personally think that MPI_Request_free is dangerous (especially in the context of threads) and does not provide much to the user. But we can’t get rid of it. … so let’s discourage users to use it!

Deprecate the C++ bindings – that’s funny, isn’t it? But look at the current C++ bindings, they’re nothing more then pimped C bindings and only create problems. Real C++ programmers would use Boot.MPI (which internally uses the C bindings ), right?

We made also some progress regarding MPI-3 where we can add more complex features that might (!) change the interface (but not break backwards compatibility).So we voted on Nonblocking Collective Operations (#109 my hobbyhorse) – and it passed unanimously!

For all votes, see votes.

[click for full-size picture]
Now, when it gets interesting – I should probably start an MPI blog about new features for MPI-2.2 or MPI-3 .

What’s wrong in this picture? Me or the Microsoft logo?