Installing BEM++ from source

The developers of BEM++ are very active and maintain the code very well. They also try to make the installation of the framework as easy as possible, and release a Docker installer for Windows, MacOS and Linux. Unfortunately, this cannot be used on the computing cluster of Delft University of Technology, because of the lack of root permissions.

The system administrators do not support installing third party software, so I had to go the long way round and do a complete installation from source code myself. From the Google group for BEM++ it seems more people want to run the code under similar circumstances. Below you will find the steps I took to get BEM++ installed on our cluster, and solutions for problems I ran into. You will probably not be able to just copy the steps one-by-one, but I hope it may serve as inspiration to get it going in your environment. A bit of knowledge about linux and compiling will help you a lot.

Step 1: Virtual environment

To have full control over the Python installation required to run BEM++, we need to run Python in a virtual environment that separates it from the system installation. This also allows us to install libraries without running into policy issues that restrict installing Python libraries. To do this, we use virtualenvwrapper and install it for the local user only.

$ pip --user install virtualenvwrapper
$ export PATH=$PATH:~/.local/bin

We then make a new folder that we will use as our main folder, and enter it.

$ mkdir foobar
$ cd foobar

The virtual environment is created by invoking the virtualenvwrapper. It will automatically create a separate folder, and install a clean Python environment inside it for you. Once installed, you can activate it using the source command, after which all python commands are linked to the Python installation in the virtual environment. In this instance, the virtual environment will be named foobarve.

$ virtualenv foobarve
$ source foobarve/bin/activate

BEM++ requires a few Python dependencies. These can be installed as you are accustomed to using pip. For multithreading, BEM++ uses the MPI platform. You may have to enable MPI libraries on your system, for example by loading the appropriate modules. This is very much cluster specific, so talk to the system administrator if you do not know how to do this. On the TUD systems, this is done by enabling the right modules. We need to install numpy, cython and mpi4py.

$ module load intel/2016
$ module load mpi/openmpi-1.8.8-intel
$ pip install numpy scipy cython mpi4py IPython

Step 2: Installing gcc from source (optional)

BEM++ uses Dune as a dependency, and this requires c++ standard 14 to compile correctly. Our cluster still runs on older version of c++, version 4.8.5, that does not support this standard in full. To test if your c++ will run standard 14 correctly, try executing the following. If it fails, you will have to install a new version of c++.

$ g++ -std=c++14

Because you do not have root permissions, you will have to install c++ from source for the local user. Reserve a few hours to a day for this, because it is a long and tedious process. Depending on the hardware you are doing this on it will take from a few hours to over a day.

To install gcc, download the source first from the gnu website or one of the mirrors. In this case, we will install version 8.2.0, but this will work equally fine for any other version. These instructions can be found on Github page of Arthur Lui.

$ cd ~
$ wget "https://ftp.gnu.org/gnu/gcc-8.2.0/gcc-8.2.0.tar.gz"
$ tar xzfv gcc-8.2.0.tar.gz
$ cd ./gcc-8.2.0
$ ./contrib/download_prerequisites
$ cd ..
$ mkdir objdir
$ cd objdir
$ $PWD/../gcc-8.2.0/configure --prefix=$HOME/gcc-8.2.0 --enable-languages=c,c++,fortran,go
$ make
$ make install

Depending on your system, the configuration step starting with $PWD.. might fail because the system does does not support 32 bit installations. You then have to add the –disable-multilib flag to the end of that line to perform a 64 bit only installation.

Unpacking the tarball can easily take up to an hour. At 108 MB it is not big at all, but it contains a few tens of thousands small files. I recommend you therefore run the command with the verbose flag on to see it is still progressing. If you don’t want the verbose output and speed up the process a bit, you can use the xzf flag instead of the xzfv flag. Just be patient then.

The make and make install steps can take a few hours in total. On a single node with a 2.4 GHz processor and ample of memory (320 GB), it took about 9 hours from download to finished installation for c, c++, gfortran and go.

Step 3: Installing Patchelf (optional)

BEM++ has Patchelf as one of its dependencies and will try to build it from source if it is not installed yet. Unfortunately, in the downloading stage, it failed for me because the MD5 hash of the download did not match the check hash. I therefore had to install it manually for the local user.

For this, I followed the simple plan outlined here, which downloads the tarball for version 0.8 (required by BEM++), unwraps it and installs it for the local user.

$ cd ~
$ wget http://nixos.org/releases/patchelf/patchelf-0.8/patchelf-0.8.tar.bz2
$ tar xf patchelf-0.8.tar.bz2
$ cd patchelf-0.8/
$ ./configure --prefix="$HOME/.local"
$ make install
$ strip ~/.local/bin/patchelf
$ gzip -9 ~/.local/share/man/man1/patchelf.1

To make sure it will be found, we have to add the location to the path variable, it if isn’t already on there.

$ export PATH=$HOME/.local/bin:$PATH

Step 4: Installing Threading Building Blocks (TBB, optional)

Because of a C++ version issue, I had problems getting the BEM++ installer to build against the version of TBB it downloads and installs itself. Therefore, I had to build TBB from source as well and make sure that the installer uses the new version instead from that point forward.

You can download the source code for the latest stable release from the TBB Github page. Browse to the version you want, and download the Source code (tar.gz). For example, for version 2018, update 5:

$ cd ~
$ mkdir tbbsrc
$ cd tbbsrc/
$ wget https://github.com/01org/tbb/archive/2018_U5.tar.gz
$ tar xvzf 2018_U5.tar.gz

Enter the folder, and run make to compile the source. This step will take a while. For me it took roughly 20 to 30 minutes.

$ cd tbb-2018_U5/
$ make all

Once it is finished, we need to tell BEM++ where to find the new installation of TBB. For this, we have to set two environment variables. The actual paths will depend on your build location and the version you are installing, as well as the C++ compiler, operating system, etc. TBB_LIBRARY_DIR will have to be set to the folder that contains the actual libraries. You will find it in the folder you just built in: <build folder/build/<os_version_kernel_version_release>.  The TBB_INCLUDE_DIR contains the header files for the C++ compiler. These are stored at a higher level in the file tree at <build folder>/../include/tbb. For example:

$ export TBB_LIBRARY_DIR=/home/rjfbijster/FSC2018/tbb/tbb-2018_U5/build/linux_intel64_gcc_cc8.2.0_lib2.17_kernel3.10.0_release
$ export TBB_INCLUDE_DIR=/home/rjfbijster/FSC2018/tbb/tbb-2018_U5/include/tbb

Step 5: Install a local version of Eigen3 (optional)

The TU Delft cluster seems to feature an older version of the Eigen3 libraries that what is required for a proper BEM++ compilation. The newest version can be downloaded from the Eigen3 Github mirror, and does not need a make step. This is a header-only library. 

$ cd ~
$ git clone https://github.com/eigenteam/eigen-git-mirror.git

Step 6: Installing BEM++

We just passed the most time consuming part. For this, make sure you are still in the virtual environment we’ve setup before. Change the directory to where you want to place the BEM++ source files. Once you are there, go to the BEM++ Bitbucket page, log in and find the link to clone the repository. It will look something like this:

$ cd ~
$ git clone $ https://yname@bitbucket.org/bemppsolutions/bempp.git

Go back to the terminal and run this command to clone the git repository. CD into the created directory and run the setup.py script:

$ cd bempp/
$ python setup.py build

This script controls CMake in the background to compile BEM++ from source code. If it runs without errors, you can then run

$ python setup.py install

to install BEM++ on your system locally.  If all goes well, you should be all done. If not, and it crashes on an error with Eigen3, you will have to make a change to the CMakeCache.txt file to have it use the newly installed version over the system version of Eigen3.

$ cd build2/
$ nano CMakeCache.txt

Scroll down the file, and find variable EIGEN3_INCLUDE_DIR and change its value to the folder where you just installed Eigen3, for example /home/rjfbijster/eigen-git-mirror

To test if BEM++ has installed correctly, open up Python and try to import the bempp library.

$ python
$ <inside the python runtime compiler>
$ import bempp

I hope these tips and tricks are useful. Good luck!