Skip to content
GitLab

Admin guide · Changes

Page history
ear-5.1 authored by Oriol Vidal Teruel's avatar Oriol Vidal Teruel
Hide whitespace changes
Inline Side-by-side
Admin-guide.md
View page @ eda70abc
...@@ -15,81 +15,250 @@ For a more detailed information about EAR components, visit the [Architecture](A ...@@ -15,81 +15,250 @@ For a more detailed information about EAR components, visit the [Architecture](A
# Quick Installation Guide # Quick Installation Guide
This section provides a, summed up, step by step installation and execution guide for EAR. For a more in depth explanation of the necessary steps see the [Installation from source](Installation from source) page or the [Installing from RPM](#installing-from-rpm) section, following the [Configuration](Configuration) guide, or contact us at ear-support@bsc.es This section provides summary of needed steps for compiling and installing EAR.
The complete guide is left into [another section](Installation from source), but **it is recommended to read first this section** since it contains useful information about how and what gets compiled and installed.
## EAR Requirements Check out first whether your system satisfies all [requirements](EAR-requirements), then check that you have Autoconf version 2.69 or later.
You can then bootstrap the build system:
You need at least a modern **C compiler**. ```bash
Other requirements to compile EAR with all of its features are: autoreconf -i
- MPI compiler and headers for supporting MPI applications. Intel MPI and OpenMPI are the most used and tested implementations for EAR development team. ```
- A [CUDA](https://developer.nvidia.com/cuda-toolkit) installation if you want support for NVIDIA GPUs.
- A [Likwid](https://hpc.fau.de/research/tools/likwid/) installation for retrieving performance metrics through this interface.
- A [FreeIPMI](https://www.gnu.org/software/freeipmi/) installation if you are going to read power and energy through IPMI specification.
- [GSL](https://www.gnu.org/software/gsl/) is needed by some tools for coefficient computation.
SLURM must also be present if the SLURM plugin wants to be used. Since current EAR version only supports automatic execution of applications with EAR Library using the SLURM plug-in, it must be running when EARL wants to be used (not needed for the most basic node monitoring service). As commented in the overview, the EAR Library might be loaded along with MPI applications thanks to the EAR Loader library.
The latter detects the application symbols at runtime and loads the right Library.
Therefore, you should compile at least two versions of the EAR Library:
To install EAR from **rpm** (only binaries) all these dependencies have been removed except *mysqlclient*. However, they are needed when running EAR components. - One for MPI applications (using one of the [MPI implementations supported](EAR-requirements#compilation).
- One for non-MPI applications.
Lastly, but not less important: This is an example to configure EAR to be compiled for both versions:
- The drivers for CPU frequency management (*acpi-cpufreq*) and Open IPMI must be present and loaded in compute nodes.
- *msr kernel* module must be loaded in compute nodes.
- mariaDB or postgress server must be up and running if using these services for storing EAR data.
- Hardware counters must be accessible for normal users. Set */proc/sys/kernel/perf\_event\_paranoid* to 2 (or less). Type `sudo sh -c "echo 2 > /proc/sys/kernel/perf_event_paranoid"` in compute nodes.
Run `./configure --help` to see all the flags and options. ```bash
# Configure EAR to compile the non-MPI version of the EAR Library
./configure --disable-mpi \
MPICC=mpicc MAKE_NAME=nompi
## Compiling and installing EAR # Configure EAR to compile the MPI version of the Library
./configure MPICC=mpicc MPICC_FLAGS="-O2 -g" MAKE_NAME=impi
```
Once downloaded the code from repository, execute: > **The above example assumes your MPI Library is Intel MPI.**
- `autoreconf -i`. > If you want to compile EARL for another MPI flavour check out [this section](supporting-more-than-one-mpi-implementation).
``` EAR currently does not support GNU make parallel builds, so the above example must be run in the source code root directory.
./configure --prefix=ear-install-path \ For the same reason, the `configure` script support a variable called `MAKE_NAME`, so it generates a Makefile called `Makefile.<MAKE_NAME variable value>`.
EAR_TMP=ear-tmp-path \ Therefore you can call `make` program targeting each configuration Makefile generated program targeting each configuration Makefile generated.
EAR_ETC=ear-etc-path \
CC=c-compiler-path \ The flag `--disable-mpi` is used for configuring the non-MPI version of the EAR Library.
MPICC=mpi-compiler-path \ > **Note** that even when configuring for this use case, `MPICC` variable is also set.
CC_FLAGS=c-flags-compiler \ > This is because the EAR Loader still needs MPI headers for checking whether the application being running is MPI, and `configure` finds out these by checking this variable.
MPICC_FLAGS=mpi-flags \
--with-cuda=path-to-cuda \
MAKE_NAME=make_extension`
```
Additionally to the Makefile, `MAKE_NAME` forces to copy the generated Makefile with the name Makefile._make\_extension_. After completing the previous steps, you can compile and install EAR by targeting each of the generated Makefiles.
It simplifies the fact of having multiple configurations (1 for each library version needed). More relevant options are: the following example takes the Makefile suffixes used in the previous one:
- The option `--disable-mpi` must be set to generate a configuration for non-MPI version of the library.
- Use `MPI_VERSION=ompi` for OpenMPI compatible version.
Before running `make`, review the Makefile and the configuration log to validate all the requirements of your installation have been automatically detected. ```bash
In particular, if you need to use some specific library such likwid, freeipmi or CUDA. If CUDA path is specified, EAR will be compiled with GPU support. Check also that MySQL ot PostgreSQL paths have been detected. # Compile and install EAR. The EAR Library version installed
You can use options `USER` and `GROUP` if you want to install EAR with a special USER/GROUP. # will be for supporting non-MPI applications.
make -f Makefile.nompi
make -f Makefile.nompi install
The following shows how to configure EAR to be compiled with Intel MPI: # sysconfdir installation needs another target
make -f Makefile.nompi etc.install
make -f Makefile.nompi doc.install
# Compile and install just the
# MPI version of the EAR Library
make -f Makefile.impi full
make -f Makefile.impi earl.install
``` ```
In the above example, some non-standard targets are used.
`etc.install` target is needed for installing all configuration, module and service files to be used later when configuring EAR.
The `full` target is the equivalent of calling first `clean` and then `all` targets.
Finally, `earl.install` is used for installing the EAR Library, since we are just compiling again because we want another version of the Library installed along with the previous one.
EAR Makefiles include a specific target for each [component](#ear-components), supporting full or partial updates:
| Target | Description |
|------------------ |------------------------------------------------- |
| `install` | Reinstalls all the files except `etc` and `doc`. |
| `earl.install` | Reinstalls only the EARL. |
| `eard.install` | Reinstalls only the EARD. |
| `earplug.install` | Reinstalls only the EAR SLURM plugin. |
| `eardbd.install` | Reinstalls only the EARDBD. |
| `eargmd.install` | Reinstalls only the EARGMD. |
| `reports.install` | Reinstalls only report plugins. |
Here is an example of a bash script summarizing the information provided until now, compiling and installing EAR with two versions of the Library: one supporting Intel MPI applications and other one supporting any non-MPI application:
```bash
#!/bin/bash
# This script bootstraps, configures, compiles and installs
# EAR with two versions of the Library: one supporting Intel MPI applications
# and other one supporting any non-MPI application
#
# Requirements:
# - GNU Autoconf
# - GNU make
# - A modern C compiler
# - Intel MPI compiler and Library
EAR_INSTALL_PATH= # Set the root location of your installation
EAR_TMP= # Set the location of temporary directories and files
EAR_ETC= # Set the location of configuration and services files.
my_CFLAGS="-O2 -g"
# Bootstrap the configure script
autoreconf -i autoreconf -i
./configure --prefix=/opt/ear CC=icc MPICC=mpiic MAKE_NAME=impi
make -f Makefile.impi # Configure EAR to compile the non-MPI version of the EAR Library
make -f Makefile.impi install ./configure --disable-mpi --prefix=$EAR_INSTALL_PATH \
make -f Makefile.impi doc.install MPICC=mpicc CC=gcc CC_FLAGS="$my_CFLAGS" \
make -f Makefile.impi etc.install EAR_TMP=$EAR_TMP EAR_ETC=$EAR_ETC \
MAKE_NAME=nompi
# Compile and install EAR. The EAR Library version installed
# will be for supporting non-MPI applications.
make -f Makefile.nompi
make -f Makefile.nompi install
# sysconfdir installation needs another target
make -f Makefile.nompi etc.install
make -f Makefile.nompi doc.install
# Configure EAR to compile the MPI version of the Library.
./configure --prefix=$EAR_INSTALL_PATH \
MPICC=mpicc MPICC_FLAGS="$my_CFLAGS" \
CC=gcc CC_FLAGS="$my_CFLAGS" \
EAR_TMP=$EAR_TMP EAR_ETC=$EAR_ETC \
MAKE_NAME=impi
# Compile and install just the
# MPI version of the EAR Library
make -f Makefile.impi full
make -f Makefile.impi earl.install
``` ```
At this point the EAR binaries will be installed including one version of the After compiling and installing following the previous step, you should have the following directories under `configure`'s `--prefix` flag used path:
EAR Library for MPI (default), EAR documentation, EAR service files for EAR
daemons and templates for `ear.conf` files and SLURM plug-in. The configure - `bin`: Including commands and tools.
tool tries to automatically detect paths for mysql and/or postgress, scheduler
sources, etc. It is mandatory to detect the scheduler path, by default SLURM is
assumed. After the configure, check in the Makefile all the options have been
detected. After the `make install`, you should have the following folders in the
`prefix` path:
- `bin`: Includes commands and tools.
- `sbin`: Includes EAR services binaries. - `sbin`: Includes EAR services binaries.
- `etc`: Includes templates and examples for EAR service files, the `ear.conf` file, the EAR module and so. - `etc`: Includes templates and examples for EAR service files, the `ear.conf` file, the EAR module and so.
- `lib`: Includes all libraries and plugins. - `lib`: Includes all libraries and plugins.
- `include` - `include`
- `man`: Man pages. - `man`: Man pages.
Inside `lib` directory, apart from plug-ins, you should see at least three files.
- `libearld.so`: This is the EAR Loader.
- `libear.so`: This is the EAR Library compiled with Intel MPI symbols. See the [next section](supporting-more-than-one-mpi-implementation) if you need support for other MPI implementations.
- `libear.gen.so`: This is the EAR Library compiled without MPI symbols. The `.gen` extension is added automatically when setting `--disable-mpi` flag.
## Supporting more than one MPI implementation
Many systems have different MPI implementations installed, so users can choose which one fits better for their applications.
Even all of them provide the same interface, each one has some specific symbols not specified in the standard.
Therefore you need to install an EAR Library version for each MPI flavor you want to support.
In order to help the EAR Loader to load the proper Library version, coliving libraries must be named different.
This is accomplished by providing `MPI_VERSION` variable to `configure`.
This variable sets an extension of the `libear.so` shared object compiled, so when the EAR Loader detects the MPI version of the application, it can easily load the proper Library.
You need to set a specific value to variable value depending on the MPI implementation you are going to compile following this table:
| **Implementation** | **MPI_VERSION value** | **EARL Name** |
|:------------------:|----------------------- |--------------------- |
| Intel MPI | not required | libear.so (default) |
| MVAPICH | not required | libear.so (default) |
| OpenMPI | ompi | libear.ompi.so |
| Fujitsu MPI | fujitsu | libear.fujitsu.so |
| Cray MPI | cray | libear.cray.so |
Note that in the example used until now this variable was not used.
This is because for this MPI version the EAR Loader does not find for an extension, and it is the continuation of the first EARL design and it was not changed.
So, if you would like to add to your previous EAR installation the support for, let's say, OpenMPI, you should type the following:
```bash
# Configure EAR to compile Library supporting OpenMPI applications
# Note: mpicc must point to an OpenMPI installation
./configure MPICC=mpicc MPICC_FLAGS="-O2 -g" MAKE_NAME=openmpi MPI_VERSION=ompi
make -f Makefile.openmpi full
# The below line assumes you already have installed all other components,
# i.e., `make -f Makefile.<extension> install`.
make -f Makefile.openmpi earl.install
```
This is an example of a bash script which summarizes the configuration, compilation and installation of EAR providing support for multiple MPI implementations:
```bash
#!/bin/bash
# This script bootstraps, configures, compiles and installs
# EAR with two versions of the Library: one supporting Intel MPI applications
# and other one supporting any non-MPI application
#
# Requirements:
# - GNU Autoconf
# - GNU make
# - A modern C compiler
# - Intel MPI compiler and Library
EAR_INSTALL_PATH= # Set the root location of your installation
EAR_TMP= # Set the location of temporary directories and files
EAR_ETC= # Set the location of configuration and services files.
my_CFLAGS="-O2 -g"
# Bootstrap the configure script
autoreconf -i
# Replace with an Intel MPI module
module load intel-mpi-module
# Configure EAR to compile the non-MPI version of the EAR Library
./configure --disable-mpi --prefix=$EAR_INSTALL_PATH \
MPICC=mpicc CC=gcc CC_FLAGS="$my_CFLAGS" \
EAR_TMP=$EAR_TMP EAR_ETC=$EAR_ETC \
MAKE_NAME=nompi
# Compile and install EAR. The EAR Library version installed
# will be for supporting non-MPI applications.
make -f Makefile.nompi
make -f Makefile.nompi install
# sysconfdir installation needs another target
make -f Makefile.nompi etc.install
make -f Makefile.nompi doc.install
# Configure EAR to compile the MPI version of the Library.
./configure --prefix=$EAR_INSTALL_PATH \
MPICC=mpicc MPICC_FLAGS="$my_CFLAGS" \
CC=gcc CC_FLAGS="$my_CFLAGS" \
EAR_TMP=$EAR_TMP EAR_ETC=$EAR_ETC \
MAKE_NAME=impi
# Compile and install just the
# MPI version of the EAR Library
make -f Makefile.impi full
make -f Makefile.impi earl.install
# Configure EAR to compile Library supporting OpenMPI applications
# Note: mpicc must point to an OpenMPI installation
module unload intel-mpi-module
module load openmpi-module
./configure --prefix=$EAR_INSTALL_PATH \
MPICC=mpicc MPICC_FLAGS="$my_CFLAGS" \
CC=gcc CC_FLAGS="$my_CFLAGS" \
EAR_TMP=$EAR_TMP EAR_ETC=$EAR_ETC \
MAKE_NAME=openmpi MPI_VERSION=ompi
make -f Makefile.openmpi full
make -f Makefile.openmpi earl.install
```
## Deployment and validation ## Deployment and validation
...@@ -97,8 +266,7 @@ detected. After the `make install`, you should have the following folders in the ...@@ -97,8 +266,7 @@ detected. After the `make install`, you should have the following folders in the
**Prepare the configuration** **Prepare the configuration**
Either installing from sources or rpm, EAR installs a template for `ear.conf` file Either installing from sources or rpm, EAR installs a template for `ear.conf` file in `$EAR_ETC/ear/ear.conf.template` and `$EAR_ETC/ear/ear.conf.full.template`.
in `$EAR_ETC/ear/ear.conf.template` and `$EAR_ETC/ear/ear.conf.full.template`.
The full version includes all fields. Copy only one as `$EAR_ETC/ear/ear.conf` and update The full version includes all fields. Copy only one as `$EAR_ETC/ear/ear.conf` and update
with the desired configuration. Go to the [configuration](Configuration) section to see how to do it. with the desired configuration. Go to the [configuration](Configuration) section to see how to do it.
The `ear.conf` is used by all the services. It is recommended to have in a shared folder to simplify the changes in the configuration. The `ear.conf` is used by all the services. It is recommended to have in a shared folder to simplify the changes in the configuration.
...@@ -110,10 +278,11 @@ You can add ear module whan it is not in standard path by doing `module use $EAR ...@@ -110,10 +278,11 @@ You can add ear module whan it is not in standard path by doing `module use $EAR
`module load ear`. `module load ear`.
**EAR Database** **EAR Database**
Create EAR database with `edb_create`, installed at `$EAR_INSTALL_PATH/sbin`. Create EAR database with `edb_create`, installed at `$EAR_INSTALL_PATH/sbin`.
The `edb_create -p` command will ask you for the DB root password. The `edb_create -p` command will ask you for the DB root password.
If you get any problem here, check first whether the node where you are running the If you get any problem here, check first whether the node where you are running the
command can connect to the DB server. In case problems persists, execute edb_create -o to report the specific SQL command can connect to the DB server. In case problems persists, execute `edb_create -o` to report the specific SQL
queries generated. In case of trouble, contact with ear-support@bsc.es or open in issue. queries generated. In case of trouble, contact with ear-support@bsc.es or open in issue.
**Energy models** **Energy models**
...@@ -227,75 +396,28 @@ For the coefficients to be active, restart daemons. ...@@ -227,75 +396,28 @@ For the coefficients to be active, restart daemons.
> **Important** Reloading daemons will NOT make them load coefficients, restarting the service is the only way. > **Important** Reloading daemons will NOT make them load coefficients, restarting the service is the only way.
## EAR Library versions: MPI vs. Non-MPI
As commented in the overview, the EAR Library is loaded next to the user MPI
application by the EAR Loader.
The Library uses MPI symbols, so it is compiled by using the includes provided
by your MPI distribution. The selection of the library version is automatic at runtime,
but it is not required during the compilation and installation steps.
Each compiled library version has its own file name that has to be defined by the
`MPI_VERSION` variable during the `./configure` or by editing the
root Makefile.
The name list per distribution is exposed in the following table:
| **Distribution** | **Name** | **MPI_VERSION value** |
|:----------------: |--------------------- |----------------------- |
| Intel MPI | libear.so (default) | not required |
| MVAPICH | libear.so (default) | not required |
| OpenMPI | libear.ompi.so | ompi |
If different MPI distributions share the same library name, it means their
symbols are compatible between them, so compiling and installing the library
one time will be enough.
However, if you provide different MPI distributions to users,
you will have to compile and install the library multiple times.
EAR makefiles include a specific target for each [EAR component](#ear-components),
supporting full or partial updates:
| Command | Description |
|--------------------------------------------------- |------------------------------------------------- |
| `make -f Makefile.make_extension install` | Reinstall all the files except `etc` and `doc`. |
| `make -f Makefile.make_extension earl.install` | Reinstall only the EARL. |
| `make -f Makefile.make_extension eard.install` | Reinstall only the EARD. |
| `make -f Makefile.make_extension earplug.install` | Reinstall only the EAR SLURM plugin. |
| `make -f Makefile.make_extension eardbd.install` | Reinstall only the EARDBD. |
| `make -f Makefile.make_extension eargmd.install` | Reinstall only the EARGMD. |
| `make -f Makefile.make_extension reports.install` | Reinstall only report plugins. |
Before compiling new libraries you have to install by typing `make install`.
Then you can run the `./configure` again, changing the `MPICC`, `MPICC_FLAGS`
and `MPI_VERSION` variables, or just opening the root Makefile and edit the
same variables and `MPI_BASE`, which just sets the MPI installation root path.
Now type `make full` to perform a clean compilation and `make earl.install`,
to install only the new version of the library.
If your MPI version is not fully compatible, please contact ear-support@bsc.es.
See the [User guide](User guide) to check the use cases supported and how to submit jobs with EAR.
# Installing from RPM # Installing from RPM
EAR includes the specification files to create an rpm from an already existing installation. EAR includes the specification files to create an rpm **from an already existing installation**.
The spec file is placed at `etc/rpms`. Once created, it can be included in the compute nodes images.
To create the RPM it is needed a valid installation from source. It is recommened only when no more changes are expected on the installation or when your compute fleet has ephimeral storage and EAR is installed in a non-shared file system.
The spec file is placed at `etc/rpms/specs/ear.spec` and it is generated from `etc/rpms/specs/ear.spec.in` at configuration time.
The RPM can be part of the system image. The RPM can be part of the system image.
Visit the [Requirements](#rpm-requirements) page for a quick overview of the requirements. Visit the [Requirements](#rpm-requirements) page for a quick overview of the requirements.
Execute the `rpmbuild.sh` script to create the EAR rpm file. Execute the `rpmbuild.sh` script to create the EAR rpm file.
Once created, it can be included in the compute nodes images. It is recommened This is script is located at `etc/rpms` and it is created from `etc/rpms/rpmbuild.sh.in` at configuration time.
only when no more changes are expected on the installation. **Run it from its location**.
Once you have the rpm file, execute the following steps: The rpm file will be located at `$HOME/rpmbuild/RPMS`.
- Before the installation, make sure the installation path is accessible by all the computing nodes. Do the same in the folder where you want to set the temporary files (it will be called `$(EAR_TMP)` in this guide for simplicity). You can install it by typing:
- Default paths are `/usr` and `/etc`.
- Run `rpm -ivh --relocate /usr=/new/install/path --relocate /etc=/new/etc/path ear.version.rpm`.
> You can also use the `--nodeps` if your dependency test fails. ```bash
rpm -ivh <ear_rpm_filename>.rpm
```
- During the installation the configuration files `*.in` are compiled to the ready to use version, replacing tags for correct paths. You will have more information of those files in the following pages. Check the [next section](#installation-content) for more information. > You can also use the `--nodeps` if your dependency test fails.
- Type `rpm -e ear.version` to uninstall. > Type `rpm -e <ear_rpm_filename>` to uninstall.
## Installation content ## Installation content
......