# Installation Instructions [[_TOC_]] ## 1. General remarks The software development is based on sprint cycles of the software releases by OpenFOAM Foundation. The sprint version number is constructed according to semantic versioning[^7] as follows: - **first digit** represents stable release of the OpenFOAM Foundation software - **s** is an abbreviation for sprint cycle - **last digit** represents the current sprint cycle with respect to the first digit - **hzdr** is short for custom developments - **third digit** represents the current code repository version with respect to the current sprint cycle The installation instructions will use the following environment variables - `WM_PROJECT_DIR`: source code of the OpenFOAM Foundation Software [^2] - `WM_PROJECT_SITE`: addon source code - `WM_PROJECT_VERSION`: version of the OpenFOAM Foundation Software[^2], e.g., dev, 6, 9, 13-s.1 ## 2. Installation of Debian Packages A Debian package is provided for Ubuntu 24.04 or newer. For installation download the package from `Packages & Registries`[^1] menu and run ```shell sudo apt update sudo apt install ./multiphase-code-repository-by-hzdr-13-s.1-hzdr.2-amd64.deb ``` Once the installation is finished add ```shell source /opt/OpenFOAM/OpenFOAM-13-s.1/etc/bashrc WM_PROJECT_SITE=/opt/OpenFOAM/site ``` to your `~/.bashrc`. Optionally, install Paraview ```shell sudo apt install paraview ``` and use ```shell paraFoam -builtin ``` to run Paraview. ## 3. Compiling from sources ### 3.1. Prerequisites and Source Code of the OpenFOAM Foundation Software To compile the source code in the repository an installation of the software released by the OpenFOAM Foundation[^2] is required. The required source code is available at the OpenFOAM-dev[^3] repository at GitHub. To download the source code a Git installation is required, which is typically the case on Ubuntu Linux systems. Note, if you have downloaded the source code from Rossendorf Data Repository (RODARE) an archive with the required commit of the software released by the OpenFOAM Foundation is available for download. Obtaining the source code from GitHub or the provided archive is equal, instead of cloning the repository unpack the archive to the required folder and follow the instructions. On an Ubuntu Linux system install the following packages ```shell sudo apt install build-essential flex bison git-core cmake zlib1g-dev \ libboost-system-dev libboost-thread-dev libopenmpi-dev openmpi-bin \ gnuplot libreadline-dev libncurses-dev libxt-dev ca-certificates libfl-dev ``` Create and enter a new directory ```shell mkdir -p $HOME/OpenFOAM cd $HOME/OpenFOAM ``` Now, clone the ThirdParty repository from GitHub and reset it to the required commit sha ```shell git clone https://github.com/OpenFOAM/ThirdParty-dev.git ThirdParty-13-s.1 (cd ThirdParty-13-s.1 && git reset --hard 190027ccd6ce49ba05c19c01cf2ef651437b031b) ``` The same has to be done for the OpenFOAM-dev repository; clone it from GitHub and reset to the required commit sha ```shell git clone https://github.com/OpenFOAM/OpenFOAM-dev.git OpenFOAM-13-s.1 (cd OpenFOAM-13-s.1 && git reset --hard a0fd65c0051fc11ef041f94e0e5aa41a1de589a6) ``` **Mac OS:** As a default Mac OS uses the Apple File System (APFS) and this is not case-sensitive. Before cloning any source code a new volume with case-sensitive APFS is required[^6]. ### 3.2. Prerequisites and Source Code for Addon The source code can be obtained via: - **Helmholtz Codebase**[^1]: For Helmholtz and Friends via Helmholtz ID[^5] or using a HZDR guest account - **Rossendorf Data Repository (RODARE)**[^18]: For everybody For usage of the GPU (graphic processor unit) accelerated population balance method a NVIDIA GPU with CUDA support is required as well as an installation of the NVIDIA Cuda Toolkit[^4] with the NVCC compiler. On an Ubuntu Linux system the following additional packages are required ```shell sudo apt install qt5-qmake qtchooser qtbase5-dev libqt5charts5-dev ``` #### 3.2.1 Source Code from Helmholtz Code Base For cloning the source code, first add the public ssh key[^7] of your local machine to your Gitlab account. The git repository includes content that is versioned using `git-lfs`[^8] to store large binary files in the repository ```shell sudo apt install git-lfs ``` Then create the directory for the addon and clone the source code ```shell mkdir -p $HOME/OpenFOAM/site/13-s.1 git clone git@codebase.helmholtz.cloud:fwdc/multiphase/code.git $HOME/OpenFOAM/site/13-s.1 ``` #### 3.2.2. Source code from Rossendorf Data Repository (RODARE) Download and extract the tar archive[^18] from RODARE into the addon directory ```shell mkdir -p $HOME/OpenFOAM/site/13-s.1 tar -xzf multiphase-code-repository-by-hzdr-13-s.1.tgz -C $HOME/OpenFOAM/site/13-s.1 ``` ### 3.3 Compiling the Source Code Firstly, apply the provided patches to the installation of the software released by the OpenFOAM Foundation ```shell cp $HOME/OpenFOAM/site/13-s.1/patches/*.sprint.tgz $HOME/OpenFOAM/OpenFOAM-13-s.1 cd $HOME/OpenFOAM/OpenFOAM-13-s.1 for f in *.sprint.tgz; do if [[ -f $f ]]; then tar xzf "$f"; fi; done for f in *.sprint; do if [[ -f $f ]]; then git am "$f"; fi; done rm -f *.sprint *.sprint.tgz ``` Configure and load the software environment ```shell sed -i -e "s/\(WM_PROJECT_VERSION=\).*$/\113-s.1/" $HOME/OpenFOAM/OpenFOAM-13-s.1/etc/bashrc sed -i -e "s/\(WM_PROJECT_VERSION \).*$/\113-s.1/" $HOME/OpenFOAM/OpenFOAM-13-s.1/etc/cshrc source $HOME/OpenFOAM/OpenFOAM-13-s.1/etc/bashrc WM_PROJECT_SITE=$HOME/OpenFOAM/site ``` Compile the source code by ```shell cd $WM_PROJECT_DIR ./Allwmake -j ``` Once the compilation process is finished successfully, reload the software environment and test the installation ```shell source $HOME/OpenFOAM/OpenFOAM-13-s.1/etc/bashrc WM_PROJECT_SITE=$HOME/OpenFOAM/site foamVersion ``` For more details concerning other Linux operating systems, additional software packages or more, read the installation instructions for OpenFOAM-dev from Source Repository[^9]. Finally, compile the source code in the site directory ```shell cd $HOME/OpenFOAM/site/13-s.1 ./Allwmake -j ``` ## 4. Using Container Images By means of Operating-System-Level virtualisation, software can be delivered in packaged form together with all its dependencies, providing a pre-defined and portable environment. Those packages are called images and a running instance of such an image is called a container. For Helmholtz and Friends ready-to-run installations are provided in the form of images. The images in the corresponding container registry[^1] are available for the platform x86_64. The software in the images is compiled with the Gcc compiler, optimized for production environments, and features support for graphical desktop applications, such as Paraview. Supported are Intel and NVIDIA graphic cards, but currently no AMD graphic cards. The source code of the OpenFOAM Foundation Software is installed inside the docker image in `/opt/OpenFOAM/` and the source code from this repository is located in `/opt/OpenFOAM/site/13-s.1`. Running containers requires a container environment such as Docker[^10] or Apptainer[^11]. We recommend Docker for developing code and running simulations locally on a PC. Apptainer is an easy-to-use alternative for users who only wish to run simulations. Apptainer can also be used on High Performance Computer (HPC) clusters where Docker is usually not available. ### 4.1. Using Docker #### 4.1.1. Prerequisites Follow the installation instructions for Docker and your operating system on the Docker homepage[^12]. If the CUDA functionality is of interest, the Nvidia Container Toolkit[^13] has to be installed. Login to the container registry ```shell docker login registry.hzdr.de ``` using your Helmholtz Codebase user name and - password if you have an HZDR account - personal access token with `read_registry` scope[^14] if you login via Helmholtz ID[^5] #### 4.1.2. Launch container with Docker Launch a container by ```shell docker run -it registry.hzdr.de/fwdc/multiphase/code/13-s.1:multiphase-code-repository-by-hzdr bash ``` or with CUDA support ```shell docker run -it --gpus all registry.hzdr.de/fwdc/multiphase/code/13-s.1:multiphase-code-repository-by-hzdr bash ``` #### 4.1.3. Launch container with mpydocker If you have access to Helmholtz Code Base [^1], you can use a more comfortable way to launch and customize our docker images. A script named `mpydocker` is provided with the `multiphasepy` package[^17]. To install it, follow the corresponding installation instructions. The tool also provides a comfortable way to customize the docker container with your own software, e.g., by defining build instructions in its configuration file. Use `mpydocker --help` and `mpydocker --print_config` for more information. ### 4.2. Using Apptainer #### 4.2.1. Prerequisites Notes on installing Apptainer on PC's and HPC systems are provided below. If you wish to run distributed simulations on an HPC system, make sure to fulfill all prerequisites. The minimum supported Apptainer version is 1.3.0. Login to the container registry ```shell apptainer registry login --username docker://registry.hzdr.de ``` using your Helmholtz Codebase user name and - password if you have an HZDR account - personal access token with `read_registry` scope[^14] if you login via Helmholtz ID[^5] ##### Installation on a PC Follow the installation instructions for Apptainer and your operating system on the Apptainer homepage[^15]. On an Ubuntu Linux system you can ```shell sudo add-apt-repository ppa:apptainer/ppa sudo apt update sudo apt install apptainer-suid ``` ##### Installation on a HPC cluster On HPC clusters, Apptainer may be available by default. Test by ```shell apptainer --version ``` If this is not the case, it may be available as a loadable module, e.g. ```shell module load apptainer ``` For running simulations distributed across compute nodes, make sure that Apptainer is automatically loaded during login. If your system uses the Bash shell, check to see if any of the following files are present, in the order listed - `~/.bash_profile` - `~/.bash_login` - `~/.profile` and add the following lines to it: ```shell if [ -z "$BASHRC_READ" ]; then export BASHRC_READ=1 module load apptainer fi ``` If none of these files exist, create `~/.profile` and add the lines. Add the submit host as a known host to your SSH configuration ```shell ssh-keyscan -H $(hostname) >> ~/.ssh/known_hosts ``` Simulation setups for the OpenFOAM Foundation Software usually contain an `Allrun` shell script which gathers all commands for pre-processing, solving and post-processing. Make sure that these scripts are equipped with a Bash Shebang `#!/bin/bash` and source the correct run functions (template in `etc/template/case`). Note that the software for network communication between compute nodes is supplied with the container image. Currently, only clusters that use the Slurm workload manager and InfiniBand interconnects are supported. #### 4.2.2. Launch container with Apptainer Launch and enter a container by ```shell apptainer run oras://registry.hzdr.de/fwdc/multiphase/code/13-s.1:multiphase-code-repository-by-hzdr-amd64.sif ``` Note that Apptainer mounts your home and the current working directory to the container by default. Other paths need to be bind mounted in order to be available in the container by ```shell apptainer run -B /net,/scratch oras://registry.hzdr.de/fwdc/multiphase/code/13-s.1:multiphase-code-repository-by-hzdr-amd64.sif ``` A comma-separated list of paths is accepted. From within the container environment, graphical applications such as ParaView can be started without further options. Commands can also be executed directly without entering the container, e.g. ```shell apptainer exec oras://registry.hzdr.de/fwdc/multiphase/code/13-s.1:multiphase-code-repository-by-hzdr-amd64.sif ./Allrun ``` It is also possible to download an Apptainer image and store it locally by ```shell apptainer pull oras://registry.hzdr.de/fwdc/multiphase/code/13-s.1:multiphase-code-repository-by-hzdr-amd64.sif ``` ## 5. Building Docker Images The required `Dockerfile` for building the Docker images is available in `etc/docker`. The `Dockerfile` is a multi-stage `Dockerfile` with the following stages: - **ubuntu:** operation system and required packages - **openmpi:** communication libraries - **paraview:** visualisation toolkit - **openfoam:** image for OpenFOAM Foundation Software - **user:** user configuration for deployment - **python:** multiphasepy package with python code and dependencies - **addon:** source code for Multiphase Code Repository by HZDR ### 5.1. Building with Docker For building the Docker images use the following command ```shell docker build --tag multiphase-code-repository-by-hzdr -f etc/docker/Dockerfile . ``` The image can be modified with docker build arguments. The available variables for `--build-arg` are defined by the `Dockerfile` and the default options are listed in the following table. It is possible to build the image only to a certain stage by using the option `--target `. | Name | Default | Options | Description | |----------------------|------------------------------------------------|------------------|-----------------------------------------| | FROMIMAGE | ubuntu:24.04 | - | Base image with operating system | | OPENFOAM_ROOT | /opt/OpenFOAM | - | Installation path | | WM_PROJECT_VERSION | 13-s.1 | dev | Version of OpenFOAM Foundation software | | WM_COMPILE_OPTION | Opt | Debug, Prof | Compile option | | WM_PROJECT | OpenFOAM | - | Name of the project | | WM_COMPILER | Gcc | - | Compiler | | WM_ARCH | linux64 | linuxArm64 | Compiler Platform | | WM_CC | gcc | - | ThirdParty C compiler | | WM_CFLAGS | -m64 -fPIC | -fPIC | ThirdParty C compiler flags | | WM_COMPILER_LIB_ARCH | 64 | - | ThirdParty compiler architecture | | WM_CXX | g++ | - | ThirdParty Cpp compiler | | WM_CXXFLAGS | -m64 -fPIC -std=c++0x | -fPIC -std=c++0x | ThirdParty Cpp compiler flags | | WM_LDFLAGS | -m64 | - | ThirdParty flags for linking | | OFREPOSITORY | | - | Url for OpenFOAM-dev git repository | | WM_PROJECT_COMMIT | a0fd65c0051fc11ef041f94e0e5aa41a1de589a6 | HEAD | Valid Git commit sha for OpenFOAM-dev | | PARAVIEW | No | Yes | Build Paraview | | PARAVIEW_ROOT | /opt/paraview | - | Installation path for Paraview | | PARAVIEW_VERSION | 5.11.1 | - | Paraview version | | OMPI_VERSION | 4.0.4 | - | OpenMPI library version | | USERNAME | docker | - | Default user name in docker container | | GROUPNAME | docker | - | Default group name in docker container | | USERID | 1042 | - | Default user id in docker container | | GROUPID | 1042 | - | Default group id in docker container | ### 5.2. Customised Docker image To keep the image size small only a limited amount of software is installed in the docker image. Furthermore, the software provided by the OpenFOAM Foundation requires a user, which is not `root` to be used in the docker image for security reasons. Hence, the docker image used a predefined user named `docker`. When binding volumes to the container, this can cause conflicts with write permissions, as typically the id of the `docker` user does not match the id of the host user. The only way to ensure consistent write permissions is to build a custom docker image and add the host user, e.g., ```docker RUN groupadd -g "" "" \ && useradd -l -u "" -g "" -m "" -s "/bin/bash" \ && echo "" ALL=\(root\) NOPASSWD:ALL > "/etc/sudoers.d/" \ && chmod 0440 "/etc/sudoers.d/" ``` The name and the id of the user and the group should match the host user and its primary group. The corresponding values can be obtained by, e.g., `id -u`, `id -un`, `id -g`, and `id -gn`. Create your own `Dockerfile`[^16], with the required additional packages and build it with `docker build`. --- [^1]: [^2]: [^3]: [^4]: [^5]: [^7]: [^6]: [^8]: [^9]: [^10]: [^11]: [^12]: [^13]: [^14]: [^15]: [^16]: [^17]: [^18]: