# Installation Instructions

[[_TOC_]]

## 1. General remarks

The software development is based on sprint cycles of the software releases by
OpenFOAM Foundation. The sprint version number is constructed according to
semantic versioning[^7] as follows:

- **first digit** represents stable release of the OpenFOAM Foundation software
- **s** is an abbreviation for sprint cycle
- **last digit** represents the current sprint cycle with respect to the first digit
<!-- markdownlint-disable MD011 -->
- **hzdr** is short for custom developments
<!-- markdownlint-enable MD011 -->
- **third digit** represents the current code repository version with
  respect to the current sprint cycle

The installation instructions will use the following environment variables

- `WM_PROJECT_DIR`: source code of the OpenFOAM Foundation Software [^2]
- `WM_PROJECT_SITE`: addon source code
- `WM_PROJECT_VERSION`: version of the OpenFOAM Foundation Software[^2],
  e.g., dev, 6, 9, 13-s.1

## 2. Installation of Debian Packages

A Debian package is provided for Ubuntu 24.04 or newer. For installation
download the package from `Packages & Registries`[^1] menu and run

```shell
sudo apt update
sudo apt install ./multiphase-code-repository-by-hzdr-13-s.1-hzdr.1-amd64.deb
```

Once the installation is finished add

```shell
source /opt/OpenFOAM/OpenFOAM-13-s.1/etc/bashrc WM_PROJECT_SITE=/opt/OpenFOAM/site
```

to your `~/.bashrc`.

Optionally, install Paraview

```shell
sudo apt install paraview
```

and use

```shell
paraFoam -builtin
```

to run Paraview.

## 3. Compiling from sources

### 3.1. Prerequisites and Source Code of the OpenFOAM Foundation Software

To compile the source code in the repository an installation of the software
released by the OpenFOAM Foundation[^2] is required. The required source code is
available at the OpenFOAM-dev[^3] repository at GitHub. To download the source
code a Git installation is required, which is typically the case on Ubuntu
Linux systems.

Note, if you have downloaded the source code from Rossendorf Data Repository
(RODARE) an archive with the required commit of the software released by the
OpenFOAM Foundation is available for download. Obtaining the source code from
GitHub or the provided archive is equal, instead of cloning the repository
unpack the archive to the required folder and follow the instructions.

On an Ubuntu Linux system install the following packages

```shell
sudo apt install build-essential flex bison git-core cmake zlib1g-dev \
     libboost-system-dev libboost-thread-dev libopenmpi-dev openmpi-bin \
     gnuplot libreadline-dev libncurses-dev libxt-dev ca-certificates libfl-dev
```

Create and enter a new directory

```shell
mkdir -p $HOME/OpenFOAM
cd $HOME/OpenFOAM
```

Now, clone the ThirdParty repository from GitHub and reset it to the required
commit sha

```shell
git clone https://github.com/OpenFOAM/ThirdParty-dev.git ThirdParty-13-s.1
(cd ThirdParty-13-s.1 && git reset --hard 190027ccd6ce49ba05c19c01cf2ef651437b031b)
```

The same has to be done for the OpenFOAM-dev repository; clone it from GitHub
and reset to the required commit sha

```shell
git clone https://github.com/OpenFOAM/OpenFOAM-dev.git OpenFOAM-13-s.1
(cd OpenFOAM-13-s.1 && git reset --hard a0fd65c0051fc11ef041f94e0e5aa41a1de589a6)
```

**Mac OS:** As a default Mac OS uses the Apple File System (APFS) and this is
not case-sensitive. Before cloning any source code a new volume with
case-sensitive APFS is required[^6].

### 3.2. Prerequisites and Source Code for Addon

The source code can be obtained via:

- **Helmholtz Codebase**[^1]: For Helmholtz and Friends via Helmholtz ID[^5]
  or using a HZDR guest account
- **Rossendorf Data Repository (RODARE)**[^18]: For everybody

For usage of the GPU (graphic processor unit) accelerated population balance
method a NVIDIA GPU with CUDA support is required as well as an installation of
the NVIDIA Cuda Toolkit[^4] with the NVCC compiler.

On an Ubuntu Linux system the following additional packages are required

```shell
sudo apt install qt5-qmake qtchooser qtbase5-dev libqt5charts5-dev
```

#### 3.2.1 Source Code from Helmholtz Code Base

For cloning the source code, first add the public ssh key[^7] of your local
machine to your Gitlab account. The git repository includes content that is
versioned using `git-lfs`[^8] to store large binary files in the repository

```shell
sudo apt install git-lfs
```

Then create the directory for the addon and clone the source code

```shell
mkdir -p $HOME/OpenFOAM/site/13-s.1
git clone git@codebase.helmholtz.cloud:fwdc/multiphase/code.git $HOME/OpenFOAM/site/13-s.1
```

#### 3.2.2. Source code from Rossendorf Data Repository (RODARE)

Download and extract the tar archive[^18] from RODARE into the addon directory

```shell
mkdir -p $HOME/OpenFOAM/site/13-s.1
tar -xzf multiphase-code-repository-by-hzdr-13-s.1.tgz -C $HOME/OpenFOAM/site/13-s.1
```

### 3.3 Compiling the Source Code

Firstly, apply the provided patches to the installation of the software released
by the OpenFOAM Foundation

```shell
cp $HOME/OpenFOAM/site/13-s.1/patches/*.sprint.tgz $HOME/OpenFOAM/OpenFOAM-13-s.1
cd $HOME/OpenFOAM/OpenFOAM-13-s.1
for f in *.sprint.tgz; do if [[ -f $f ]]; then tar xzf "$f"; fi; done
for f in *.sprint; do if [[ -f $f ]]; then git am "$f"; fi; done
rm -f *.sprint *.sprint.tgz
```

Configure and load the software environment

```shell
sed -i -e "s/\(WM_PROJECT_VERSION=\).*$/\113-s.1/" $HOME/OpenFOAM/OpenFOAM-13-s.1/etc/bashrc
sed -i -e "s/\(WM_PROJECT_VERSION \).*$/\113-s.1/" $HOME/OpenFOAM/OpenFOAM-13-s.1/etc/cshrc
source $HOME/OpenFOAM/OpenFOAM-13-s.1/etc/bashrc WM_PROJECT_SITE=$HOME/OpenFOAM/site
```

Compile the source code by

```shell
cd $WM_PROJECT_DIR
./Allwmake -j
```

Once the compilation process is finished successfully, reload the software
environment and test the installation

```shell
source $HOME/OpenFOAM/OpenFOAM-13-s.1/etc/bashrc WM_PROJECT_SITE=$HOME/OpenFOAM/site
foamVersion
```

For more details concerning other Linux operating systems, additional software
packages or more, read the installation instructions for OpenFOAM-dev from
Source Repository[^9].

Finally, compile the source code in the site directory

```shell
cd $HOME/OpenFOAM/site/13-s.1
./Allwmake -j
```

## 4. Using Container Images

By means of Operating-System-Level virtualisation, software can be delivered in
packaged form together with all its dependencies, providing a pre-defined and
portable environment. Those packages are called images and a running instance of
such an image is called a container. For Helmholtz and Friends ready-to-run
installations are provided in the form of images. The images in the
corresponding container registry[^1] are available for the platform x86_64.
The software in the images is compiled with the Gcc compiler, optimized
for production environments, and features support for graphical desktop
applications, such as Paraview. Supported are Intel and NVIDIA graphic cards,
but currently no AMD graphic cards. The source code of the OpenFOAM Foundation
Software is installed inside the docker image in `/opt/OpenFOAM/` and the source
code from this repository is located in `/opt/OpenFOAM/site/13-s.1`.

Running containers requires a container environment such as Docker[^10] or
Apptainer[^11]. We recommend Docker for developing code and running simulations
locally on a PC. Apptainer is an easy-to-use alternative for users who only wish
to run simulations. Apptainer can also be used on High Performance Computer
(HPC) clusters where Docker is usually not available.

### 4.1. Using Docker

#### 4.1.1. Prerequisites

Follow the installation instructions for Docker and your operating system on the
Docker homepage[^12]. If the CUDA functionality is of interest, the Nvidia
Container Toolkit[^13] has to be installed.

Login to the container registry

```shell
docker login registry.hzdr.de
```

using your Helmholtz Codebase user name and

- password if you have an HZDR account
- personal access token with `read_registry` scope[^14] if you login via
  Helmholtz ID[^5]

#### 4.1.2. Launch container with Docker

Launch a container by

```shell
docker run -it registry.hzdr.de/fwdc/multiphase/code/13-s.1:multiphase-code-repository-by-hzdr bash
```

or with CUDA support

```shell
docker run -it --gpus all registry.hzdr.de/fwdc/multiphase/code/13-s.1:multiphase-code-repository-by-hzdr bash
```

#### 4.1.3. Launch container with mpydocker

If you have access to Helmholtz Code Base [^1], you can use a more comfortable
way to launch and customize our docker images. A script named `mpydocker` is
provided with the `multiphasepy` package[^17]. To install it,
follow the corresponding installation instructions.

The tool also provides a comfortable way to customize the docker
container with your own software, e.g., by defining build instructions in its
configuration file. Use `mpydocker --help` and `mpydocker --print_config` for
more information.

### 4.2. Using Apptainer

#### 4.2.1. Prerequisites

Notes on installing Apptainer on PC's and HPC systems are provided below. If you
wish to run distributed simulations on an HPC system, make sure to fulfill all
prerequisites. The minimum supported Apptainer version is 1.3.0.

Login to the container registry

```shell
apptainer registry login --username <user> docker://registry.hzdr.de
```

using your Helmholtz Codebase user name and

- password if you have an HZDR account
- personal access token with `read_registry` scope[^14] if you login via
  Helmholtz ID[^5]

##### Installation on a PC

Follow the installation instructions for Apptainer and your operating system on
the Apptainer homepage[^15]. On an Ubuntu Linux system you can

```shell
sudo add-apt-repository ppa:apptainer/ppa
sudo apt update
sudo apt install apptainer-suid
```

##### Installation on a HPC cluster

On HPC clusters, Apptainer may be available by default. Test by

```shell
apptainer --version
```

If this is not the case, it may be available as a loadable module, e.g.

```shell
module load apptainer
```

For running simulations distributed across compute nodes, make sure that
Apptainer is automatically loaded during login. If your system uses the Bash
shell, check to see if any of the following files are present, in the order
listed

- `~/.bash_profile`
- `~/.bash_login`
- `~/.profile`

and add the following lines to it:

```shell
if [ -z "$BASHRC_READ" ]; then
  export BASHRC_READ=1
  module load apptainer
fi
```

If none of these files exist, create `~/.profile` and add the lines.

Add the submit host as a known host to your SSH configuration

```shell
ssh-keyscan -H $(hostname) >> ~/.ssh/known_hosts
```

Simulation setups for the OpenFOAM Foundation Software usually contain an
`Allrun` shell script which gathers all commands for pre-processing, solving and
post-processing. Make sure that these scripts are equipped with a Bash Shebang
`#!/bin/bash` and source the correct run functions (template in
`etc/template/case`).

Note that the software for network communication between compute nodes is
supplied with the container image. Currently, only clusters that use the Slurm
workload manager and InfiniBand interconnects are supported.

#### 4.2.2. Launch container with Apptainer

Launch and enter a container by

```shell
apptainer run oras://registry.hzdr.de/fwdc/multiphase/code/13-s.1:multiphase-code-repository-by-hzdr-amd64.sif
```

Note that Apptainer mounts your home and the current working directory to the
container by default. Other paths need to be bind mounted in order to be
available in the container by

```shell
apptainer run -B /net,/scratch oras://registry.hzdr.de/fwdc/multiphase/code/13-s.1:multiphase-code-repository-by-hzdr-amd64.sif
```

A comma-separated list of paths is accepted.

From within the container environment, graphical applications such as ParaView
can be started without further options.

Commands can also be executed directly without entering the container, e.g.

```shell
apptainer exec oras://registry.hzdr.de/fwdc/multiphase/code/13-s.1:multiphase-code-repository-by-hzdr-amd64.sif ./Allrun
```

It is also possible to download an Apptainer image and store it locally by

```shell
apptainer pull oras://registry.hzdr.de/fwdc/multiphase/code/13-s.1:multiphase-code-repository-by-hzdr-amd64.sif
```

## 5. Building Docker Images

The required `Dockerfile` for building the Docker images is available in
`etc/docker`. The `Dockerfile` is a multi-stage `Dockerfile` with
the following stages:

- **ubuntu:** operation system and required packages
- **openmpi:** communication libraries
- **paraview:** visualisation toolkit
- **openfoam:** image for OpenFOAM Foundation Software
- **user:** user configuration for deployment
- **python:** multiphasepy package with python code and dependencies
- **addon:** source code for Multiphase Code Repository by HZDR

### 5.1. Building with Docker

For building the Docker images use the following command

```shell
docker build --tag multiphase-code-repository-by-hzdr -f etc/docker/Dockerfile .
```

The image can be modified with docker build arguments. The available variables
for  `--build-arg` are defined by the `Dockerfile` and the default options are
listed in the following table. It is possible to build the image only to a
certain stage by using the option `--target <stage>`.

| Name                 | Default                                        | Options          | Description                             |
|----------------------|------------------------------------------------|------------------|-----------------------------------------|
| FROMIMAGE            | ubuntu:24.04                                   | -                | Base image with operating system        |
| OPENFOAM_ROOT        | /opt/OpenFOAM                                  | -                | Installation path                       |
| WM_PROJECT_VERSION   | 13-s.1                                         | dev              | Version of OpenFOAM Foundation software |
| WM_COMPILE_OPTION    | Opt                                            | Debug, Prof      | Compile option                          |
| WM_PROJECT           | OpenFOAM                                       | -                | Name of the project                     |
| WM_COMPILER          | Gcc                                            | -                | Compiler                                |
| WM_ARCH              | linux64                                        | linuxArm64       | Compiler Platform                       |
| WM_CC                | gcc                                            | -                | ThirdParty C compiler                   |
| WM_CFLAGS            | -m64 -fPIC                                     | -fPIC            | ThirdParty C compiler flags             |
| WM_COMPILER_LIB_ARCH | 64                                             | -                | ThirdParty compiler architecture        |
| WM_CXX               | g++                                            | -                | ThirdParty Cpp compiler                 |
| WM_CXXFLAGS          | -m64 -fPIC -std=c++0x                          | -fPIC -std=c++0x | ThirdParty Cpp compiler flags           |
| WM_LDFLAGS           | -m64                                           | -                | ThirdParty flags for linking            |
| OFREPOSITORY         | <https://github.com/OpenFOAM/OpenFOAM-dev.git> | -                | Url for OpenFOAM-dev git repository     |
| WM_PROJECT_COMMIT    | a0fd65c0051fc11ef041f94e0e5aa41a1de589a6       | HEAD             | Valid Git commit sha for OpenFOAM-dev   |
| PARAVIEW             | No                                             | Yes              | Build Paraview                          |
| PARAVIEW_ROOT        | /opt/paraview                                  | -                | Installation path for Paraview          |
| PARAVIEW_VERSION     | 5.11.1                                         | -                | Paraview version                        |
| OMPI_VERSION         | 4.0.4                                          | -                | OpenMPI library version                 |
| USERNAME             | docker                                         | -                | Default user name in docker container   |
| GROUPNAME            | docker                                         | -                | Default group name in docker container  |
| USERID               | 1042                                           | -                | Default user id in docker container     |
| GROUPID              | 1042                                           | -                | Default group id in docker container    |

### 5.2. Customised Docker image

To keep the image size small only a limited amount of software is installed in
the docker image. Furthermore, the software provided by the OpenFOAM Foundation
requires a user, which is not `root` to be used in the docker image for security
reasons. Hence, the docker image used a predefined user named `docker`. When
binding volumes to the container, this can cause conflicts with write
permissions, as typically the id of the `docker` user does not match the id of
the host user. The only way to ensure consistent write permissions is to build a
custom docker image and add the host user, e.g.,

```docker
RUN groupadd -g "<gid>" "<groupname>" \
    && useradd -l -u "<uid>" -g "<groupname>" -m "<username>" -s "/bin/bash" \
    && echo "<username>" ALL=\(root\) NOPASSWD:ALL > "/etc/sudoers.d/<username>" \
    && chmod 0440 "/etc/sudoers.d/<username>"
```

The name and the id of the user and the group should match the host user and its
primary group. The corresponding values can be obtained by, e.g., `id -u`,
`id -un`, `id -g`, and `id -gn`. Create your own `Dockerfile`[^16], with the
required additional packages and build it with `docker build`.

---
[^1]: <https://codebase.helmholtz.cloud/fwdc/multiphase/code>
[^2]: <https://openfoam.org>
[^3]: <https://github.com/OpenFOAM/OpenFOAM-dev>
[^4]: <https://developer.nvidia.com/cuda-toolkit>
[^5]: <https://hifis.net/aai>
[^7]: <https://docs.gitlab.com/ee/user/ssh.html>
[^6]: <https://support.apple.com/guide/disk-utility>
[^8]: <https://git-lfs.github.com>
[^9]: <https://openfoam.org/download/source/>
[^10]: <https://www.docker.com>
[^11]: <https://apptainer.org>
[^12]: <https://docs.docker.com/engine/installation>
[^13]: <https://github.com/NVIDIA/nvidia-container-toolkit>
[^14]: <https://codebase.helmholtz.cloud/help/user/packages/container_registry/authenticate_with_container_registry.html>
[^15]: <https://apptainer.org/docs/user/latest/>
[^16]: <https://docs.docker.com/engine/reference/builder>
[^17]: <https://doi.org/10.14278/rodare.3093>
[^18]: <https://doi.org/10.14278/rodare.767>
