Handling Multiple MPI Implementations

1.  Introduction

Motivation

There are numerous MPI (Message Passing Interface) implementations in the portage tree, and far more in general. While it would be nice if everyone could settle on just one, there are a number of reasons this isn't going to happen anytime soon. For instance, binary-only applications that rely on a specific implementation/version, early support for new hardware, and, of course, developer preference.

The goal of empi is to ease the administrator burden inherent in managing numerous HPC applications and libraries, each of which support some subset of the available MPI implementations. Also, empi strives to ensure that any user of an HPC system can happily work with their favorite implementation and supporting applications, while another user may utilize an entirely different implementation, with potentially overlapping applications.

If you are only concerned with quickly getting up and running, you may skip the rest of this section as it only serves to provide a description of how empi works behind the scenes.

Empi

Heavily based on sys-devel/crossdev, empi is a script wrapping portage that uses a small bit of trickery to emerge MPI implementations and any dependent packages to unique directories in /usr/lib/mpi/. Quickly explained, this is done by making a local overlay, creating new categories that define a set consisting of a single version of one MPI implementation and any packages that utilize this implementation, and then using that category to find the correct install locations. The bulk of the work involving portage is handled by mpi.eclass.

Empi itself can handle creating the overlay directory structure, merging definitions in package.use and package.keywords, and calling emerge for the required packages. The overlay is created using symlinks, so any update to the portage tree will be replicated in the empi overlay.

mpi.eclass

The eclass uses the package category to decide where a package should be installed inside of /usr/lib/mpi/. Only packages inheriting this eclass will ever be installed in this separate root. The advantage over using ROOT=/some/where/ emerge stuff being we only need to install packages that actually require MPI or are implementations into this separate root directory, everything else is provided by the standard system.

Note that it is required that any ebuild will have to be ported to use mpi.eclass, just inheriting in anything but the simplest ebuild will not work automatically. Currently autotools based packages can be ported with minimal effort. Sadly, these are in the minority in the HPC package space, so mpi.eclass also provides mechanisms for getting install paths as well as wrappers around the standard ebuild utilities, do* and new*, with the prefix mpi_. For instance, if an ebuild needs to call dobin, it should be ported to use mpi_dobin instead.

The eclass should be transparent if empi is not being used; packages can still be emerged normally onto the system without any trace of mpi.eclass being used.

eselect-mpi

Recall that the grand unified goal is to enable users to easily work with different implementations at the same time on the same system. Therefore, unlike the majority (perhaps all) of the other eselect modules, eselect-mpi is designed to administer a users' personal MPI environment. This is done by writing the required environment variables and other implementation specific details to ${HOME}/.env.d/mpi.sh and ${HOME}/.env.d/mpi.csh which the user must remember to source if they plan on utilizing the implementation. For the curious, all we're really doing is providing the environment variables such as PATH, MANPATH and LD_LIBRARY_PATH in this file.

A user may also use eselect-mpi to unselect an implementation. Then, after re-sourcing the appropriate file in ${HOME}/.env.d/, their environment is returned to its previous (empi-less) state and they may continue on with their work using the system MPI implementation (if installed).

2.  Using empi

Installing empi

The above introduction makes a lot more sense once we actually get some implementations installed and start working with them. To begin, add the science overlay, which is where empi and the related ebuilds are currently being maintained.

emerge app-portage/layman
layman -a science
emerge sys-cluster/empi

Adding an Implementation and Packages

To get started, we'll add a single implementation based on sys-cluster/openmpi as well as the sys-cluster/hpl package that will be built using openmpi and installed into the correct root.

/usr/bin/empi --create --implementation mpi-openmpi \
    =sys-cluster/openmpi-1.2.6-r1

Empi will emerge mpi-openmpi/openmpi-1.2.6-r1 and not sys-cluster/openmpi-1.2.6-r1. By changing the category, empi is alerting mpi.eclass that it needs to handle separating this install from the standard system root. This new package can be manipulated using portage via all the normal mechanisms. Next, we install sys-cluster/hpl using the mpi-openmpi implementation.

/usr/bin/empi --implementation mpi-openmpi --add sys-cluster/hpl

Both openmpi and hpl are now installed in the /usr/lib/mpi/mpi-openmpi root. However, a user still needs to actually select this implementation set in order to start using it.

$ eselect mpi set mpi-openmpi
$ eselect mpi list
Available MPI implementations:
  mpi-openmpi               Enabled
$ source ~/.env.d/mpi.sh
$ eselect mpi list
Available MPI implementations:
  mpi-openmpi               Enabled, In Use

Other implementations and packages can be emerged and used in the same manner as show above. Assuming a bash shell, it would probably be useful to add the following to ${HOME}/.bashrc

for i in ${HOME}/.env.d/*.sh; do
    source ${i}
done

3.  References

Links

• Gentoo Overlays: Users' Guide
• Cross-Compile Howto
• Gentoo Science Overlay

The contents of this document are licensed under the Creative Commons - Attribution / Share Alike license.