Note (2024-06-03):
We are now recommending loading software modules through the EESSI project whenever possible! The software is read from CVMFS, which is more performant than our local BeeGFS file system. If the software is not available their, you can still read it from BeeGFS as usual.
Software: Modules
We use LMod environment modules to provide software installations to our users (also see hpc-wiki.info/hpc/Modules). Please have a look at the LMod user guide.
Preparing the EESSI Environment
The EESSI project provides many modules through a community driven “streaming service” for scientific software. We recommend trying this approach first, since you can benefit from two optimizations:
- EESSI has optimized software builds for many hardware architectures (selected automatically)
- EESSI is distributing its software via CVMFS, which is very performant when reading software
The very first access to a software package might take a short while to download in the background (seconds). Every subsequent access is served by local caches and therefore very fast.
Feel free to contact us if you run into one of the following issues with EESSI:
- The software you want to use is not be available on EESSI, yet (we may be able to change that!)
- Software for GPUs (e.g. built on CUDA) might need more work before it is available (licensing reason)
You can use EESSI on all of our cluster systems. You can verify this with a simple check:
less /cvmfs/software.eessi.io/README.eessi
If the file is available, you can load software modules through EESSI through a single command:
$ source /cvmfs/software.eessi.io/versions/2023.06/init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/intel/haswell
Using x86_64/intel/haswell as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/haswell/.lmod/lmodrc.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/haswell/modules/all as the directory to be added to MODULEPATH.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/haswell/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/haswell/.lmod/SitePackage.lua
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/haswell/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!
{EESSI 2023.06} $ module avail # run the usual module commands
Be aware, that more recent versions of the whole software stack might be available in the /cvmfs/software.eessi.io/versions directory (here 2023.06).
You can find more information about EESSI on hpc-wiki.info and the EESSI documentation.
Finding Modules
After login, you can use the module command to find and load software:
$ module --help
Usage: module [options] sub-command [args ...]
Options:
-h -? -H --help This help message
.
.
.
For example, if you need a recent version of GCC and CMake and want to see available versions, you can execute:
$ module spider GCC CMake
-------------------------------------------------------------------------------
GCC:
-------------------------------------------------------------------------------
Description:
The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...).
Versions:
GCC/4.9.3-2.25
GCC/8.3.0
GCC/9.3.0
GCC/10.2.0
GCC/10.3.0
Other possible modules matches:
GCCcore
-------------------------------------------------------------------------------
CMake:
-------------------------------------------------------------------------------
Description:
CMake, the cross-platform, open-source build system. CMake is a family of tools designed to build, test and package software.
Versions:
CMake/3.15.3
CMake/3.16.4
CMake/3.18.4
CMake/3.20.1
-------------------------------------------------------------------------------
To find other possible module matches execute:
$ module -r spider '.*GCC.*'
-------------------------------------------------------------------------------
For detailed information about a specific "CMake" package (including how to load the modules) use the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:
$ module spider CMake/3.20.1
-------------------------------------------------------------------------------
Use the specific versions of GCC and CMake, to see details on how to load these modules:
$ module spider GCC/10.3.0 CMake/3.20.1
-------------------------------------------------------------------------------
GCC: GCC/10.3.0
-------------------------------------------------------------------------------
Description:
The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...).
You will need to load all module(s) on any one of the lines below before the "GCC/10.3.0" module is available to load.
2021a
2021a-norpath
Help:
Description
===========
The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada,
as well as libraries for these languages (libstdc++, libgcj,...).
More information
================
- Homepage: https://gcc.gnu.org/
-------------------------------------------------------------------------------
CMake: CMake/3.20.1
-------------------------------------------------------------------------------
Description:
CMake, the cross-platform, open-source build system. CMake is a family of tools designed to build, test and package software.
You will need to load all module(s) on any one of the lines below before the "CMake/3.20.1" module is available to load.
2021a GCCcore/10.3.0
2021a-norpath GCCcore/10.3.0
Help:
Description
===========
CMake, the cross-platform, open-source build system. CMake is a family of
tools designed to build, test and package software.
More information
================
- Homepage: https://www.cmake.org
To finally load the software you can call module load 2021a GCC/10.3.0 CMake/3.20.1:
$ gcc --version
gcc (GCC) 10.3.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ which gcc
/beegfs/Tools/easybuild/stacks/rome/2021a/software/GCCcore/10.3.0/bin/gcc
$ which cmake
/beegfs/Tools/easybuild/stacks/rome/2021a/software/CMake/3.20.1-GCCcore-10.3.0/bin/cmake
You would use such a module load command, whenever you prepare your environment for interactive work, as well as in batch job scripts.
Background Information: Hierarchical Naming Scheme
The module visibility is following a hierarchical naming scheme. It means that software only becomes visible to the module available command, after it’s dependencies are loaded. 2021a represents a “meta module”, acting as an entry point into a given version of the software stack. A new software stack is released about every 6 months and older stacks will be phased out over time. To ensure reproducibility, you can still access old software stacks by calling module use /beegfs/Tools/easybuild/old-meta-module/, but we recommend to use the most recent versions whenever possible.
The hierarchical naming scheme can make it difficult to discover available software. If you look for something specific, just use module spider <software> to get an overview and module spider <software>/<version> to learn what dependencies have to be loaded. You can also look into /beegfs/Tools/easybuild/stacks/rome/2021a/software/ to get a general idea of the available software (as of 2021-11-18):
$ ls /beegfs/Tools/easybuild/stacks/rome/2021a/software/
Anaconda3 flatbuffers h5py libiconv numactl SciPy-bundle
ATK flatbuffers-python HarfBuzz libjpeg-turbo NVHPC Score-P
at-spi2-atk flex HDF5 libpciaccess OPARI2 SCOTCH
at-spi2-core FlexiBLAS help2man libpng OpenBLAS SDL2
Autoconf fmt hwloc libreadline OpenFOAM SIONlib
Automake fontconfig hypothesis LibTIFF OpenMPI snappy
Autotools foss ICU libtool OpenSSL spdlog
Bazel freeglut iimpi libunwind OTF2 SQLite
binutils freetype imkl libxml2 Pango Szip
Bison FriBidi impi libyaml PAPI Tcl
Boost GCC intel likwid Paraver TensorFlow
bzip2 GCCcore intel-compilers LLVM ParaView Theano
cairo GDB intltool LMDB PCRE typing-extensions
CFITSIO Gdk-Pixbuf ISL Lua PCRE2 UCX
CGAL gettext JasPer lz4 PDT UnZip
Clang giflib Java M4 Perl util-linux
CMake git jbigkit makeinfo pixman Valgrind
CubeGUI GL2PS JsonCpp Mako pkgconfig wxWidgets
CubeLib glew JUBE MCFM pkg-config X11
CubeWriter GLib Keras Mesa PMIx x264
CUDA GMP LAME Meson protobuf x265
cURL gnuplot libarchive METIS protobuf-python xorg-macros
DB GObject-Introspection libcerf MPC pybind11 XZ
DBus gompi libdrm MPFR Python Yasm
double-conversion gperf libepoxy NASM PyYAML Z3
Doxygen groff libevent ncurses Qt5 Zip
Eigen GSL libfabric netCDF re2c zlib
elfutils GST-plugins-base libffi Ninja ROOT zstd
expat GStreamer libgd NSPR Rust
FFmpeg GTK3 libGLU NSS ScaLAPACK
FFTW gzip libglvnd nsync Scalasca
Background Information: Module Build
We build the software stacks with EasyBuild. All modules have been built on our wn21 nodes and therefore optimized for the AMD Rome architecture.
If you are missing a software, please check if it is available in EasyBuilds list of supported software. In this case it might be trivial for us to provide it as well. Otherwise, we hope you understand that building unsupported software may require some time or might not be feasible for us. If you have made your own modules, e.g. in /beegfs/<user>/modules, you can add the path via module use /path/to/my/modules.
Troubleshooting
Occasionally you are in a situation where you know a new module just has been made available, but the module command claims it is not. In this case you can try module --ignore-cache spider, since available module locations might be cached in your user directory.
If you load many modules, you might experience an increased latency in your interactive shell commands. This is caused by a growing LD_LIBRARY_PATH environment variable, when modules from the 2021a (and any -norpath) stack are loaded. The growing LD_LIBRARY_PATH is causing frequent search operations on our /beegfs, which can be slow if many file system operations occur in parallel, e.g. through other users jobs.
We are working on mitigations of this issue and as a workaround you can try to use the software from the 2021a-rpath stack instead. Here, the whole software stack is built with RPATH, i.e. lookup paths of library dependencies are compiled into all binaries and the lookup through LD_LIBRARY_PATH is not necessary anymore. There are a couple of drawbacks though:
- The
2021a-rpathstack is not necessarily suited to build your own software, e.g. by loading GCC and CMake. In this case you would have to consider using RPATH in your build as well. For simplicity, try to use the-rpathstack only if you intend to use the application, but not build further software against it. - The
2021a(-norpath) and2021a-rpathstack might be out of sync. Please tell us if you need a certain module that is available in one, but no the other.
Alternative to Modules: LCG
The SFT project at CERN provides multiple LCG releases that bundle many useful software packages. This also contains different versions of the gcc or clang compilers. If CVMFS is available, you can set up a specific release via
source /cvmfs/sft.cern.ch/lcg/views/LCG_99/x86_64-centos7-gcc8-opt/setup.sh
CVMFS is available on all login and worker nodes of PLEIADES.
If you are only interested in a single tool, e.g. gcc, you can source the corresponding setup.sh, e.g. in
/cvmfs/sft.cern.ch/lcg/releases/gcc/8.1.0/x86_64-centos7/setup.sh
Please check if the version of the operating system (here centos7) matches the architecture in that path. For more info refer to the LCG info web page or the README files in /cvmfs/sft.cern.ch/lcg/