Preview

Table of Content

[[TOC]]

1. General remarks

The installation instructions will use the following environment variables

WM_PROJECT_DIR: directory, where OpenFOAM source code provided by The OpenFOAM Foundation is installed
WM_PROJECT_USER_DIR: OpenFOAM user directory, where user's code or addons can be installed
WM_PROJECT_SITE: OpenFOAM site directory, where addons can be installed
WM_PROJECT_VERSION: OpenFOAM version, e.g., dev, 6, 9, 9-s.1

2. Compiling from sources

2.1. Compiling of OpenFOAM

To compile the HZDR Multiphase Addon for OpenFOAM an installation of OpenFOAM provided by The OpenFOAM Foundation is required. The source code is available at the OpenFOAM-dev repository at GitHub.

Note, if you have downloaded the HZDR Multiphase Addon for OpenFOAM from Rossendorf Data Repository (RODARE) an archive with the required commit of the OpenFOAM release source code is available for download. Obtaining the source code from GitHub or the provided archive is equal, instead of cloning the repository unpack the archive to the required folder and follow the instructions.

On an Ubuntu Linux system install the following general packages for OpenFOAM

sudo apt-get install build-essential flex bison git-core cmake zlib1g-dev \
     libboost-system-dev libboost-thread-dev libopenmpi-dev openmpi-bin \
     gnuplot libreadline-dev libncurses-dev libxt-dev ca-certificates libfl-dev

Create a new directory for OpenFOAM

mkdir -p $HOME/OpenFOAM
cd $HOME/OpenFOAM

and enter the newly created directory. Now, clone the ThirdParty sources from GitHub and reset the sources to the required commit sha

git clone https://github.com/OpenFOAM/ThirdParty-dev.git ThirdParty-9-s.1
(cd ThirdParty-9-s.1 && git reset --hard 588fb48a605b80819d371735fba4de2aa3d69ae6)

The same has to be done for the OpenFOAM sources, clone from GitHub repository and reset to the required commit sha

git clone https://github.com/OpenFOAM/OpenFOAM-dev.git OpenFOAM-9-s.1
(cd OpenFOAM-9-s.1 && git reset --hard 01f0753ac7bafff1c39df37b5259adf4df2f380e)

Once all repositories are successfully cloned and reset, the compilation of OpenFOAM can be started as normal

sed -i -e "s/\(WM_PROJECT_VERSION=\).*$/\19-s.1/" OpenFOAM-9-s.1/etc/bashrc
sed -i -e "s/\(WM_PROJECT_VERSION=\).*$/\19-s.1/" OpenFOAM-9-s.1/etc/cshrc
source $HOME/OpenFOAM/OpenFOAM-9-s.1/etc/bashrc
(cd $WM_PROJECT_DIR && ./Allwmake -j)

Once the compilation process is finished successfully, load the OpenFOAM environment and test the installation

source $HOME/OpenFOAM/OpenFOAM-9-s.1/etc/bashrc
foamVersion

For more details concerning other Linux operating systems, additional software packages or more read the installation instructions for OpenFOAM-dev from Source Repository.

2.2. Compiling of HZDR Multiphase Addon

There are two possible ways to obtain the source code for the HZDR Multiphase Addon for OpenFOAM:

gitlab.hzdr.de: For Helmholtz and Friends via Helmholtz AAI or using a HZDR guest account
Rossendorf Data Repository (RODARE): For everybody

For usage of the GPU (graphic processor unit) accelerated population balance method a NVIDIA GPU with CUDA support is required as well as an installation of the NVIDIA Cuda Toolkit with the NVCC compiler.

2.2.1 Source Code from gitlab.hzdr.de

The HZDR Multiphase Addon repository includes content that is versioned using git-lfs

sudo apt-get install git-lfs

to store large binary files in the repository.

Then create a directory for OpenFOAM user developments, clone and compile the addon

source $HOME/OpenFOAM/OpenFOAM-9-s.1/etc/bashrc
mkdir -p $WM_PROJECT_USER_DIR
git clone --single-branch git@gitlab.hzdr.de:openfoam/fwdc/Developments.git $WM_PROJECT_USER_DIR
(cd $WM_PROJECT_USER_DIR && ./Allwmake -j)

2.2.2. Source code from Rossendorf Data Repository (RODARE)

Download tar archive for HZDR Multiphase Addon for OpenFOAM from RODARE, unpack into OpenFOAM user directory and compile the addon

source $HOME/OpenFOAM/OpenFOAM-9-s.1/etc/bashrc
mkdir -p $WM_PROJECT_USER_DIR
tar -xzf HZDR-Multiphase-Addon-<version>.tgz -C $WM_PROJECT_USER_DIR
(cd $WM_PROJECT_USER_DIR && ./Allwmake -j)

3. Using Docker Images

Docker is a popular software for Operating-System-Level virtualisation and can be used to deliver software in packages and a pre-defined environment. Those packages are called Docker images and a running instance of such an image is called a Docker container. Those containers are isolated from each other and from the host system. For Helmholtz and Friends the HZDR Multiphase Addon for OpenFOAM together with a pre-installed working environment for OpenFOAM can be provided as Docker image. The Docker images in the corresponding Docker registry feature support for graphical desktop applications, such as Paraview. Supported are Intel and NVIDIA graphic cards, but currently no AMD graphic cards.

Optional: A useful tool for defining, deploying and managing Docker applications is docker compose. The HZDR Multiphase Addon for OpenFOAM contains a set of docker compose files to build and launch the desired Docker images.

3.1. Prerequisites

Follow the installation instructions for Docker and your operating system on the Docker homepage. For security reasons, it is highly recommended to install Docker in rootless mode.

Optional: If the CUDA functionality of the HZDR Multiphase Addon for OpenFOAM is of interest, the Nvidia Container Toolkit has to be installed. Also, if docker compose is of interest, follow the installation instructions for docker compose V2+.

3.2. Available Docker Images at gitlab.hzdr.de

The following table contains a list of available Docker images in the project registry including a pre-compiled OpenFOAM (openfoam*, recommended for developers) or also a pre-compiled HZDR Multiphase Addon (hzdr*, recommended for users).

Docker image tag	OpenFOAM	HZDR Multiphase Addon	X11	Cuda	Compiler	Compile Option
hzdrLinuxGccOpt	:white_check_mark:	:white_check_mark:	:white_check_mark: Intel	:x:	Gcc	Optimised
hzdrNvidiaGccOpt	:white_check_mark:	:white_check_mark:	:white_check_mark: Nvidia	:x:	Gcc	Optimised
hzdrCudaGccOpt	:white_check_mark:	:white_check_mark:	:x:	:white_check_mark:	Gcc, Nvcc	Optimised
openfoamLinuxGccOpt	:white_check_mark:	:x:	:white_check_mark: Intel	:x:	Gcc	Optimised
openfoamLinuxClangOpt	:white_check_mark:	:x:	:white_check_mark: Intel	:x:	Clang	Optimised
openfoamLinuxGccDebug	:white_check_mark:	:x:	:white_check_mark: Intel	:x:	Gcc	Debugging
openfoamNvidiaGccOpt	:white_check_mark:	:x:	:white_check_mark: Nvidia	:x:	Gcc	Optimised
openfoamNvidiaGccDebug	:white_check_mark:	:x:	:white_check_mark: Nvidia	:x:	Gcc	Debugging
openfoamCudaGccOpt	:white_check_mark:	:x:	:x:	:white_check_mark:	Gcc, Nvcc	Optimised

The source code of the OpenFOAM Foundation release is installed inside the docker image in $HOME/OpenFOAM/ and the HZDR Multiphase Addon for OpenFOAM is located in $WM_PROJECT_SITE.

3.2.1. Launch image with Docker

docker login registry.hzdr.de

# No X11 support
docker run -it registry.hzdr.de/openfoam/fwdc/developments/9-s.1:<image> bash

# X11 support for Intel
xhost +
docker run -it --device=/dev/dri:/dev/dri --env DISPLAY=$DISPLAY --volume /tmp/.X11-unix:/tmp/.X11-unix registry.hzdr.de/openfoam/fwdc/developments/9-s.1:<image> bash

# X11 support for nvidia
xhost +
docker run -it --gpus all --env DISPLAY=$DISPLAY --volume /tmp/.X11-unix:/tmp/.X11-unix registry.hzdr.de/openfoam/fwdc/developments/9-s.1:<image> bash

# No X11, Cuda support
xhost +
docker run -it --gpus all registry.hzdr.de/openfoam/fwdc/developments/9-s.1:<image> bash

Note, you have to replace <image> with an image tag.

3.2.2. Launch image with foamDockerRun.sh

For a more comfortable usage a bash script foamDockerRun.sh is provided to run an OpenFOAM Docker image.

docker login registry.hzdr.de

# X11 support for Intel
foamDockerRun.sh registry.hzdr.de/openfoam/fwdc/developments/9-s.1:<image>

# X11 support for nvidia
foamDockerRun.sh --gpus all registry.hzdr.de/openfoam/fwdc/developments/9-s.1:<image>

# No X11 but Cuda support
foamDockerRun.sh --gpus all registry.hzdr.de/openfoam/fwdc/developments/9-s.1:<image>

For more information and options run foamDockerRun.sh --help or docker run --help. Note, you have to replace <image> with an image tag.

3.2.3. Launch image with `docker compose`

To use docker compose, first download the pre-configured compose and environment variable files. Alternatively, you can clone the repository and run docker-compose from inside the directory.

docker compose uses services to run a pre-defined Docker environment. The following table contains a list of the available services and the Docker images that are used. From within the directory containing the compose and environment variable files, run

docker compose --profile '*' config --services

to get a full list of all available services.

Service	OpenFOAM	HZDR Multiphase Addon	X11	Cuda	Compiler	Compile Option
hzdrLinux	:white_check_mark:	:white_check_mark:	:white_check_mark: Intel	:x:	Gcc	Optimised
hzdrNvidia	:white_check_mark:	:white_check_mark:	:white_check_mark: Nvidia	:x:	Gcc	Optimised
hzdrCuda	:white_check_mark:	:white_check_mark:	:x:	:white_check_mark:	Gcc, Nvcc	Optimised
openfoamLinux	:white_check_mark:	:x:	:white_check_mark: Intel	:x:	Gcc	Optimised
openfoamLinuxDebug	:white_check_mark:	:x:	:white_check_mark: Intel	:x:	Gcc	Debugging
openfoamLinuxProf	:white_check_mark:	:x:	:white_check_mark: Intel	:x:	Gcc	Profiling
openfoamNvidia	:white_check_mark:	:x:	:white_check_mark: Nvidia	:x:	Gcc	Optimised
openfoamNvidiaDebug	:white_check_mark:	:x:	:white_check_mark: Nvidia	:x:	Gcc	Debugging
openfoamNvidiaProf	:white_check_mark:	:x:	:white_check_mark: Nvidia	:x:	Gcc	Profiling
openfoamCuda	:white_check_mark:	:x:	:x:	:white_check_mark:	Gcc, Nvcc	Optimised

To launch a service from the table above do

docker login registry.hzdr.de

# Launch service in detached mode (background)
docker compose up -d <service>

# execute command in running container
docker compose exec <service> bash

# stop service
docker compose down <service>

Note, you have to replace <service> with a service name. The source code of the OpenFOAM Foundation release is installed inside the docker image in $HOME/OpenFOAM/ and the HZDR Multiphase Addon for OpenFOAM is located in $WM_PROJECT_SITE.

3.3. Running Docker image from Rodare

This is currently not supported.

3.4. System and Git Configuration within Docker

3.4.1. General Linux Configurations

Most software packages on Linux systems store there configurations in ~/.config. To persistently store and share configurations for software packages, such as Paraview, this directory should be defined and created as a Docker volume, and shared between Docker containers.

# create docker volume
docker volume create config

# mount volume when launching container
docker run -it --volume config:/root/.config <image> bash

When using foamDockerRun.sh this volume will be created and shared automatically. When using docker compose the volume has to be created manually, but will be shared automatically.

3.4.2 Git Configuration

The Git version control system typically stores its configuration in ~/.gitconfig. It is recommended to copy the git configuration into the Docker containers instead of binding the file as a volume to it. When using pure Docker the following commands will copy the git configuration file into the container

CONTAINERID=$(docker create -it <image>)
docker cp $HOME/.gitconfig $CONTAINERID:/root/.gitconfig
docker start -i $CONTAINERID

For docker compose at least docker compose V2+ is required, then use the following commands

docker compose up -d <service>

docker compose cp $HOME/.gitconfig <service>:/root/.gitconfig

docker compose exec <service> bash

The script foamDockerRun.sh will copy the git configuration into the container automatically. Note, you have to replace <image> or <service> with a image tag or a service name, respectively.

4. Building Docker Images

The required Dockerfile for building the Docker images with the required OpenFOAM Foundation version and the HZDR Multiphase Addon for OpenFOAM is available in etc/docker. The Dockerfile is a multi-stage Dockerfile with the following two stages:

openfoam - build Openfoam image without HZDR Multiphase Addon
hzdraddon - build Openfoam image and HZDR Multiphase Addon

4.1. Building with Docker

For building the Docker images use the following command

docker build --target <openfoam|hzdraddon> --tag <image> --build-arg ... -f etc/docker/Dockerfile .

The available variables for build-arg are defined by the Dockerfile and the default options are listed in the following table

Name	Default	Options	Description
FROMIMAGE	ubuntu:20.04	nvidia/opengl:base-ubuntu20.04, nvidia/cuda:11.2.0-devel-ubuntu20.04	base image with operating system
WM_PROJECT_VERSION	dev	-	OpenFOAM version, e.g. dev, 9, 9-s.1, ...
WM_COMPILE_OPTION	Opt	Debug, Prof	Compile Option for OpenFOAM
WM_COMPILER	Gcc	Clang	Compiler for OpenFOAM
OFREPOSITORY	https://github.com/OpenFOAM/OpenFOAM-dev.git	-	Git repository for OpenFOAM-dev
TPREPOSITORY	https://github.com/OpenFOAM/ThirdParty-dev.git	-	Git repository for ThirdParty-dev
WM_PROJECT_COMMIT	Head	-	Valid Git commit sha for OpenFOAM-dev
THIRDPARTY_COMMIT	Head	-	Valid Git commit sha for ThirdParty-dev
PARAVIEW	No	Yes	Build Paraview

Note, that in particular WM_PROJECT_COMMIT and THIRDPARTY_COMMIT have to be set to the correct commit sha, otherwise the HZDR Multiphase Addon will not build correctly. The most recent values can be found in README.md.

4.2. Building with Docker Compose

Building the images with docker compose is the most easy way and requires almost no configuration. Use the following commands to build the Docker images by yourself

# current sprint cycle release
docker compose -f docker-compose.yml -f etc/docker-compose/deployment.yml build <service>

# current OpenFOAM-dev release
docker compose --env-file etc/docker-compose/.env-dev build -f docker-compose.yml -f etc/docker-compose/deployment.yml <service>

4.3. Customised Docker image

The Docker images provide an OpenFOAM environment and the HZDR Multiphase Addon for OpenFOAM. However, additional software like editors or graphical diff tools are not installed as default to keep the size of the Docker images small. If additional software packages are required, the Docker images can be customised.

4.3.1. Customised Docker image with `docker commit`

The simplest way is to run the desired Docker image

docker login registry.hzdr.de
# No X11 support
docker run -it registry.hzdr.de/openfoam/fwdc/developments/9-s.1:<image> bash

and install all additional software packages inside the container

apt-get update
apt-get install ADD_UBUNTU_PACKAGE_HERE

Once all software packages are installed successfully, leave the container running and open a new terminal. Use docker ps to extract the container id of the running container and create a new image from it

docker commit <container id> registry.hzdr.de/openfoam/fwdc/developments/9-s.1:<custom image>

Finally, stop the container and run the new custom image

# No X11 support
docker run -it registry.hzdr.de/openfoam/fwdc/developments/9-s.1:<custom image> bash

Note, you have to replace <image> with an image tag.

4.3.2. Customised Dockerfile

The recommended way is to create your own Dockerfile. An example is shown below

# Build variables required for docker image
ARG FROMIMAGE
FROM $FROMIMAGE

# Install custom ubuntu packages
RUN apt-get update \
    && apt-get install -qy --fix-missing --no-install-recommends \
       ADD_UBUNTU_PACKAGE_HERE \
       ADD_UBUNTU_PACKAGE_HERE \
       ADD_UBUNTU_PACKAGE_HERE

# Slim down packages
RUN apt-get clean autoclean && apt-get autoremove -y

To build your custom image with the selected software packages run

docker build --build-arg FROMIMAGE=registry.hzdr.de/openfoam/fwdc/developments/9-s.1:<image> .

Note, you have to replace <image> with an image tag.

4.3.3. Customised docker-compose.override.yml

It is also possible to build a custom image with docker compose. Therefore, a Dockerfile as shown above is required as well as a docker-compose.override.yml in the project repository folder. Each service refers to a specific Docker image than can be customised according to

version: "3.9"

services:
  <service>:
    build:
      dockerfile: Dockerfile
      args:
        FROMIMAGE: "${REGISTRY}/9-s.1:<image>"

Note, you have to replace <image> and <service> with an image tag and with a service name, respectively. Finally, run the service

# Build customised docker image
docker compose build <service>

# Launch service in deattached mode (background)
docker compose up -d <service>

# execute command in running container
docker compose exec <service> bash

For further information about the structure of the Compose file and its syntax, refer to the general information about Docker Compose as well as the Compose Specification.

1. General remarks

2. Compiling from sources

2.1. Compiling of OpenFOAM

2.2. Compiling of HZDR Multiphase Addon

2.2.1 Source Code from gitlab.hzdr.de

2.2.2. Source code from Rossendorf Data Repository (RODARE)

3. Using Docker Images

3.1. Prerequisites

3.2. Available Docker Images at gitlab.hzdr.de

3.2.1. Launch image with Docker

3.2.2. Launch image with foamDockerRun.sh

3.2.3. Launch image with docker compose

3.3. Running Docker image from Rodare

3.4. System and Git Configuration within Docker

3.4.1. General Linux Configurations

3.4.2 Git Configuration

4. Building Docker Images

4.1. Building with Docker

4.2. Building with Docker Compose

4.3. Customised Docker image

4.3.1. Customised Docker image with docker commit

4.3.2. Customised Dockerfile

4.3.3. Customised docker-compose.override.yml

3.2.3. Launch image with `docker compose`

4.3.1. Customised Docker image with `docker commit`